3DFlex Dependencies; Building pycuda

Install, Configure and Update
1

Hi,

I am attempting to install 3DFlex dependencies but failing to build pycuda. Stderr runs 3000+ lines but I’m not sure which bits may be informative. I’ll send the entirety by email.

Reverting to the original state with cryosparcw forcedeps after a botched attempt seems to work.

Cheers,
Yang

$ uname -a
Linux <redacted> 4.18.0-372.32.1.el8_6.x86_64 #1 SMP Tue Oct 25 05:53:57 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugs.almalinux.org/ --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --disable-libmpx --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 8.5.0 20210514 (Red Hat 8.5.0-10) (GCC)
$ echo $PATH | grep cuda
$ echo $LD_LIBRARY_PATH | grep cuda
$ pwd
<redacted>/cryosparc2/cryosparc_worker
$ ./bin/cryosparcw install-3dflex
...
3 Likes
2

Hello,

We are experiencing a similar problem on our single-workstation configuration with centos7. Pycuda breaks and throws a really long error message regarding gcc during the building stage, then csparc is able to launch jobs but becomes unable to run any jobs. Reverting back works for us too.

$ uname -a
Linux xxxxxx 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/opt/rh/devtoolset-7/root/usr/libexec/gcc/x86_64-redhat-linux/7/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/opt/rh/devtoolset-7/root/usr --mandir=/opt/rh/devtoolset-7/root/usr/share/man --infodir=/opt/rh/devtoolset-7/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --enable-plugin --with-linker-hash-style=gnu --enable-initfini-array --with-default-libstdcxx-abi=gcc4-compatible --with-isl=/builddir/build/BUILD/gcc-7.3.1-20180303/obj-x86_64-redhat-linux/isl-install --enable-libmpx --enable-gnu-indirect-function --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 7.3.1 20180303 (Red Hat 7.3.1-5) (GCC)

Best,
Kookjoo

1 Like
3

Same problem here also with centos 7. Seems to be the building wheel for pycuda step that fails with the command ‘gcc’ failed with exit status 1 message at the end of a very long error message.

2 Likes
4

Same problem here.
The install crashes at the level of the PyCuda installation.

Building wheels for collected packages: pycuda
Building wheel for pycuda (setup.py) … error
(followed by a long error message).

We are on Ubuntu 22.04.
Output gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: …/src/configure -v --with-pkgversion=‘Ubuntu 11.3.0-1ubuntu1~22.04’ --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)

1 Like
5

Tried on a second system (this time Ubuntu, not CentOS) - same error. Last lines of error here:

    In file included from /usr/include/c++/9/memory:80,
                     from bpl-subset/bpl_subset/boost/config/no_tr1/memory.hpp:21,
                     from bpl-subset/bpl_subset/boost/smart_ptr/shared_ptr.hpp:27,
                     from bpl-subset/bpl_subset/boost/shared_ptr.hpp:17,
                     from src/cpp/cuda.hpp:30,
                     from src/wrapper/wrap_cudadrv.cpp:3:
    /usr/include/c++/9/bits/unique_ptr.h:53:28: note: declared here
       53 |   template<typename> class auto_ptr;
          |                            ^~~~~~~~
    src/wrapper/wrap_cudadrv.cpp: In function ‘void init_module__driver()’:
    src/wrapper/wrap_cudadrv.cpp:997:26: error: ‘CU_TARGET_COMPUTE_20’ was not declared in this scope; did you mean ‘CU_TARGET_COMPUTE_90’?
      997 |     .value("COMPUTE_20", CU_TARGET_COMPUTE_20)
          |                          ^~~~~~~~~~~~~~~~~~~~
          |                          CU_TARGET_COMPUTE_90
    src/wrapper/wrap_cudadrv.cpp:1000:26: error: ‘CU_TARGET_COMPUTE_21’ was not declared in this scope; did you mean ‘CU_TARGET_COMPUTE_61’?
     1000 |     .value("COMPUTE_21", CU_TARGET_COMPUTE_21)
          |                          ^~~~~~~~~~~~~~~~~~~~
          |                          CU_TARGET_COMPUTE_61
    In file included from bpl-subset/bpl_subset/boost/function/detail/maybe_include.hpp:23,
                     from bpl-subset/bpl_subset/boost/function/function2.hpp:11,
                     from bpl-subset/bpl_subset/boost/python/object/function_object.hpp:8,
                     from bpl-subset/bpl_subset/boost/python/make_function.hpp:14,
                     from bpl-subset/bpl_subset/boost/python/data_members.hpp:15,
                     from bpl-subset/bpl_subset/boost/python/class.hpp:17,
                     from bpl-subset/bpl_subset/boost/python.hpp:18,
                     from src/cpp/cuda.hpp:36,
                     from src/wrapper/wrap_cudadrv.cpp:3:
    bpl-subset/bpl_subset/boost/function/function_template.hpp: In instantiation of ‘void pycudaboost::detail::function::basic_vtable2<R, T0, T1>::assign_functor(FunctionObj, pycudaboost::detail::function::function_buffer&, mpl_::true_) const [with FunctionObj = pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >; R = bool; T0 = const pycudaboost::python::detail::exception_handler&; T1 = const pycudaboost::function0<void>&; mpl_::true_ = mpl_::bool_<true>]’:
    bpl-subset/bpl_subset/boost/function/function_template.hpp:602:13:   required from ‘bool pycudaboost::detail::function::basic_vtable2<R, T0, T1>::assign_to(FunctionObj, pycudaboost::detail::function::function_buffer&, pycudaboost::detail::function::function_obj_tag) const [with FunctionObj = pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >; R = bool; T0 = const pycudaboost::python::detail::exception_handler&; T1 = const pycudaboost::function0<void>&]’
    bpl-subset/bpl_subset/boost/function/function_template.hpp:492:45:   required from ‘bool pycudaboost::detail::function::basic_vtable2<R, T0, T1>::assign_to(F, pycudaboost::detail::function::function_buffer&) const [with F = pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >; R = bool; T0 = const pycudaboost::python::detail::exception_handler&; T1 = const pycudaboost::function0<void>&]’
    bpl-subset/bpl_subset/boost/function/function_template.hpp:936:7:   required from ‘void pycudaboost::function2<R, T1, T2>::assign_to(Functor) [with Functor = pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >; R = bool; T0 = const pycudaboost::python::detail::exception_handler&; T1 = const pycudaboost::function0<void>&]’
    bpl-subset/bpl_subset/boost/function/function_template.hpp:722:7:   required from ‘pycudaboost::function2<R, T1, T2>::function2(Functor, typename pycudaboost::enable_if_c<pycudaboost::type_traits::ice_not<pycudaboost::is_integral<Functor>::value>::value, int>::type) [with Functor = pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >; R = bool; T0 = const pycudaboost::python::detail::exception_handler&; T1 = const pycudaboost::function0<void>&; typename pycudaboost::enable_if_c<pycudaboost::type_traits::ice_not<pycudaboost::is_integral<Functor>::value>::value, int>::type = int]’
    bpl-subset/bpl_subset/boost/python/exception_translator.hpp:20:39:   required from ‘void pycudaboost::python::register_exception_translator(Translate, pycudaboost::type<Target>*) [with ExceptionType = pycuda::error; Translate = void (*)(const pycuda::error&)]’
    src/wrapper/wrap_cudadrv.cpp:691:74:   required from here
    bpl-subset/bpl_subset/boost/function/function_template.hpp:566:49: warning: placement new constructing an object of type ‘pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >’ and size ‘16’ in a region of type ‘char’ and size ‘1’ [-Wplacement-new=]
      566 |           new (reinterpret_cast<void*>(&functor.data)) FunctionObj(f);
          |                                         ~~~~~~~~^~~~
    In file included from bpl-subset/bpl_subset/boost/function/detail/prologue.hpp:17,
                     from bpl-subset/bpl_subset/boost/function/function_template.hpp:13,
                     from bpl-subset/bpl_subset/boost/function/detail/maybe_include.hpp:13,
                     from bpl-subset/bpl_subset/boost/function/function0.hpp:11,
                     from bpl-subset/bpl_subset/boost/python/errors.hpp:13,
                     from bpl-subset/bpl_subset/boost/python/handle.hpp:11,
                     from bpl-subset/bpl_subset/boost/python/args_fwd.hpp:10,
                     from bpl-subset/bpl_subset/boost/python/args.hpp:10,
                     from bpl-subset/bpl_subset/boost/python.hpp:11,
                     from src/cpp/cuda.hpp:36,
                     from src/wrapper/wrap_cudadrv.cpp:3:
    bpl-subset/bpl_subset/boost/function/function_base.hpp: In instantiation of ‘static void pycudaboost::detail::function::functor_manager_common<Functor>::manage_small(const pycudaboost::detail::function::function_buffer&, pycudaboost::detail::function::function_buffer&, pycudaboost::detail::function::functor_manager_operation_type) [with Functor = pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >]’:
    bpl-subset/bpl_subset/boost/function/function_base.hpp:364:56:   required from ‘static void pycudaboost::detail::function::functor_manager<Functor>::manager(const pycudaboost::detail::function::function_buffer&, pycudaboost::detail::function::function_buffer&, pycudaboost::detail::function::functor_manager_operation_type, mpl_::true_) [with Functor = pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >; mpl_::true_ = mpl_::bool_<true>]’
    bpl-subset/bpl_subset/boost/function/function_base.hpp:412:18:   required from ‘static void pycudaboost::detail::function::functor_manager<Functor>::manager(const pycudaboost::detail::function::function_buffer&, pycudaboost::detail::function::function_buffer&, pycudaboost::detail::function::functor_manager_operation_type, pycudaboost::detail::function::function_obj_tag) [with Functor = pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >]’
    bpl-subset/bpl_subset/boost/function/function_base.hpp:440:20:   required from ‘static void pycudaboost::detail::function::functor_manager<Functor>::manage(const pycudaboost::detail::function::function_buffer&, pycudaboost::detail::function::function_buffer&, pycudaboost::detail::function::functor_manager_operation_type) [with Functor = pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >]’
    bpl-subset/bpl_subset/boost/function/function_template.hpp:934:13:   required from ‘void pycudaboost::function2<R, T1, T2>::assign_to(Functor) [with Functor = pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >; R = bool; T0 = const pycudaboost::python::detail::exception_handler&; T1 = const pycudaboost::function0<void>&]’
    bpl-subset/bpl_subset/boost/function/function_template.hpp:722:7:   required from ‘pycudaboost::function2<R, T1, T2>::function2(Functor, typename pycudaboost::enable_if_c<pycudaboost::type_traits::ice_not<pycudaboost::is_integral<Functor>::value>::value, int>::type) [with Functor = pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >; R = bool; T0 = const pycudaboost::python::detail::exception_handler&; T1 = const pycudaboost::function0<void>&; typename pycudaboost::enable_if_c<pycudaboost::type_traits::ice_not<pycudaboost::is_integral<Functor>::value>::value, int>::type = int]’
    bpl-subset/bpl_subset/boost/python/exception_translator.hpp:20:39:   required from ‘void pycudaboost::python::register_exception_translator(Translate, pycudaboost::type<Target>*) [with ExceptionType = pycuda::error; Translate = void (*)(const pycuda::error&)]’
    src/wrapper/wrap_cudadrv.cpp:691:74:   required from here
    bpl-subset/bpl_subset/boost/function/function_base.hpp:318:54: warning: placement new constructing an object of type ‘pycudaboost::detail::function::functor_manager_common<pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > > >::functor_type’ {aka ‘pycudaboost::_bi::bind_t<bool, pycudaboost::python::detail::translate_exception<pycuda::error, void (*)(const pycuda::error&)>, pycudaboost::_bi::list3<pycudaboost::arg<1>, pycudaboost::arg<2>, pycudaboost::_bi::value<void (*)(const pycuda::error&)> > >’} and size ‘16’ in a region of type ‘char’ and size ‘1’ [-Wplacement-new=]
      318 |             new (reinterpret_cast<void*>(&out_buffer.data)) functor_type(*in_functor);
          |                                           ~~~~~~~~~~~^~~~
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/tmp/pip-req-build-_l7dp5ir/setup.py'"'"'; __file__='"'"'/home/tmp/pip-req-build-_l7dp5ir/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /home/tmp/pip-record-6hs1zd7g/install-record.txt --single-version-externally-managed --compile --install-headers /home/user/software/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m/pycuda Check the logs for full command output.
1 Like
6

Same error:

Building wheel for pycuda (setup.py) ... error
ERROR: Command errored out with exit status 1:
   command: /home/spuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-jaj9i4s1/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-jaj9i4s1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-zamats68
       cwd: /tmp/pip-req-build-jaj9i4s1/
  Complete output (2812 lines):
  *************************************************************
  *** I have detected that you have not run configure.py.
  *************************************************************
  *** Additionally, no global config files were found.
  *** I will go ahead with the default configuration.
  *** In all likelihood, this will not work out.
  ***
  *** See README_SETUP.txt for more information.
  ***
  *** If the build does fail, just re-run configure.py with the
  *** correct arguments, and then retry. Good luck!
  *************************************************************
  *** HIT Ctrl-C NOW IF THIS IS NOT WHAT YOU WANT
  *************************************************************

Many lines later:

    error: command 'gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/spuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-jaj9i4s1/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-jaj9i4s1/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-u_zstxdc/install-record.txt --single-version-externally-managed --compile --install-headers /home/spuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m/pycuda Check the logs for full command output.
1 Like
7

Sorry to add to the pile on, but same here:

All three systems I tried to install were Linux Mint 20.3.

Output from gcc -v

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7.5.0-3ubuntu1~18.04' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
1 Like
8

Same issue here, Ubuntu 20.04.

cryosparc@troll:/home/users/posert$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.4.0-1ubuntu1~20.04.1' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-Av3uEd/gcc-9-9.4.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)
1 Like
9

Not to pile on, but we’re having the same issue. Ubuntu 22.04:

Building wheels for collected packages: pycuda
  Building wheel for pycuda (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-kydhq6hf/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-kydhq6hf/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-er4trb0z
       cwd: /tmp/pip-req-build-kydhq6hf/
  Complete output (3111 lines):
  *************************************************************
  *** I have detected that you have not run configure.py.
  *************************************************************
  *** Additionally, no global config files were found.
  *** I will go ahead with the default configuration.
  *** In all likelihood, this will not work out.
  ***
  *** See README_SETUP.txt for more information.
  ***
  *** If the build does fail, just re-run configure.py with the
  *** correct arguments, and then retry. Good luck!
  *************************************************************
  *** HIT Ctrl-C NOW IF THIS IS NOT WHAT YOU WANT
  *************************************************************

gcc -v output:

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.3.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)
1 Like
10

Same issue here, RHEL 8.5

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/8/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-redhat-linux
Configured with: …/configure --enable-bootstrap --enable-languages=c,c++,fortran,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --with-isl --disable-libmpx --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 8.5.0 20210514 (Red Hat 8.5.0-15) (GCC)

Building wheels for collected packages: pycuda
Building wheel for pycuda (setup.py) … error
ERROR: Command errored out with exit status 1:
command: /apps/cryosparc4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c ‘import io, os, sys, setuptools, tokenize; sys.argv[0] = ‘"’"’/tmp/pip-req-build-riub75yd/setup.py’"’"’; file=’"’"’/tmp/pip-req-build-riub75yd/setup.py’"’"’;f = getattr(tokenize, ‘"’"‘open’"’"’, open)(file) if os.path.exists(file) else io.StringIO(’"’"‘from setuptools import setup; setup()’"’"’);code = f.read().replace(’"’"’\r\n’"’"’, ‘"’"’\n’"’"’);f.close();exec(compile(code, file, ‘"’"‘exec’"’"’))’ bdist_wheel -d /tmp/pip-wheel-to6x7bpc
cwd: /tmp/pip-req-build-riub75yd/
Complete output (3926 lines):

*** I have detected that you have not run configure.py.

*** Additionally, no global config files were found.
*** I will go ahead with the default configuration.
*** In all likelihood, this will not work out.

*** See README_SETUP.txt for more information.

*** If the build does fail, just re-run configure.py with the
*** correct arguments, and then retry. Good luck!

*** HIT Ctrl-C NOW IF THIS IS NOT WHAT YOU WANT

Continuing in 10 seconds…
Continuing in 9 seconds…
Continuing in 8 seconds…
Continuing in 7 seconds…
Continuing in 6 seconds…
Continuing in 5 seconds…
Continuing in 4 seconds…
Continuing in 3 seconds…
Continuing in 2 seconds…
Continuing in 1 seconds…
/apps/cryosparc4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: ‘test_requires’
warnings.warn(msg)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/pycuda
copying pycuda/init.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/_cluda.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/_mymako.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/autoinit.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/characterize.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/compiler.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/cumath.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/curandom.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/debug.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/driver.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/elementwise.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/gpuarray.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/reduction.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/scan.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/tools.py → build/lib.linux-x86_64-3.7/pycuda
creating build/lib.linux-x86_64-3.7/pycuda/gl
copying pycuda/gl/init.py → build/lib.linux-x86_64-3.7/pycuda/gl
copying pycuda/gl/autoinit.py → build/lib.linux-x86_64-3.7/pycuda/gl
creating build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/init.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/cg.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/coordinate.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/inner.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/operator.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/packeted.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/pkt_build.py → build/lib.linux-x86_64-3.7/pycuda/sparse
creating build/lib.linux-x86_64-3.7/pycuda/compyte
copying pycuda/compyte/init.py → build/lib.linux-x86_64-3.7/pycuda/compyte
copying pycuda/compyte/array.py → build/lib.linux-x86_64-3.7/pycuda/compyte
copying pycuda/compyte/dtypes.py → build/lib.linux-x86_64-3.7/pycuda/compyte
running egg_info
writing pycuda.egg-info/PKG-INFO
writing dependency_links to pycuda.egg-info/dependency_links.txt
writing requirements to pycuda.egg-info/requires.txt
writing top-level names to pycuda.egg-info/top_level.txt
reading manifest file ‘pycuda.egg-info/SOURCES.txt’
reading manifest template ‘MANIFEST.in’
warning: no files found matching ‘doc/source/_static/.css’
warning: no files found matching 'doc/source/_templates/.html’
warning: no files found matching ‘.cpp’ under directory ‘bpl-subset/bpl_subset/boost’
warning: no files found matching '.html’ under directory ‘bpl-subset/bpl_subset/boost’
warning: no files found matching ‘.inl’ under directory ‘bpl-subset/bpl_subset/boost’
warning: no files found matching '.txt’ under directory ‘bpl-subset/bpl_subset/boost’
warning: no files found matching ‘.h’ under directory ‘bpl-subset/bpl_subset/libs’
warning: no files found matching '.ipp’ under directory ‘bpl-subset/bpl_subset/libs’
warning: no files found matching ‘*.pl’ under directory ‘bpl-subset/bpl_subset/libs’
adding license file ‘LICENSE’
writing manifest file ‘pycuda.egg-info/SOURCES.txt’
creating build/lib.linux-x86_64-3.7/pycuda/cuda
copying pycuda/cuda/pycuda-complex-impl.hpp → build/lib.linux-x86_64-3.7/pycuda/cuda
copying pycuda/cuda/pycuda-complex.hpp → build/lib.linux-x86_64-3.7/pycuda/cuda
copying pycuda/cuda/pycuda-helpers.hpp → build/lib.linux-x86_64-3.7/pycuda/cuda
copying pycuda/sparse/pkt_build_cython.pyx → build/lib.linux-x86_64-3.7/pycuda/sparse
running build_ext
building 'driver’ extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/src
creating build/temp.linux-x86_64-3.7/src/cpp
creating build/temp.linux-x86_64-3.7/src/wrapper
creating build/temp.linux-x86_64-3.7/bpl-subset
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/converter
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/smart_ptr
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/smart_ptr/src
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/system
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/system/src
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/thread
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/thread/src
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/thread/src/pthread
gcc -pthread -B /apps/cryosparc4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/compiler_compat -Wl,–sysroot=/ -Wsign-compare -fwrapv -Wall -O3 -DNDEBUG -fPIC -DBOOST_ALL_NO_LIB=1 -DBOOST_THREAD_BUILD_DLL=1 -DBOOST_MULTI_INDEX_DISABLE_SERIALIZATION=1 -DBOOST_PYTHON_SOURCE=1 -Dboost=pycudaboost -DBOOST_THREAD_DONT_USE_CHRONO=1 -DPYGPU_PACKAGE=pycuda -DPYGPU_PYCUDA=1 -DHAVE_CURAND=1 -Isrc/cpp -Ibpl-subset/bpl_subset -I/apps/cryosparc4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include -I/apps/cryosparc4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include -I/apps/cryosparc4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m -c src/cpp/cuda.cpp -o build/temp.linux-x86_64-3.7/src/cpp/cuda.o
In file included from bpl-subset/bpl_subset/boost/smart_ptr/shared_ptr.hpp:32,
from bpl-subset/bpl_subset/boost/shared_ptr.hpp:17,
from src/cpp/cuda.hpp:30,
from src/cpp/cuda.cpp:4:
bpl-subset/bpl_subset/boost/smart_ptr/detail/shared_count.hpp:284:33: warning: ‘template class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
explicit shared_count( std::auto_ptr & r ): pi( new sp_counted_impl_p( r.get() ) )
^~~~~~~~
In file included from /usr/include/c++/8/memory:80,
from bpl-subset/bpl_subset/boost/config/no_tr1/memory.hpp:21,
from bpl-subset/bpl_subset/boost/smart_ptr/shared_ptr.hpp:27,
from bpl-subset/bpl_subset/boost/shared_ptr.hpp:17,
from src/cpp/cuda.hpp:30,
from src/cpp/cuda.cpp:4:
/usr/include/c++/8/bits/unique_ptr.h:53:28: note: declared here
template class auto_ptr;
^~~~~~~~
In file included from bpl-subset/bpl_subset/boost/shared_ptr.hpp:17,
from src/cpp/cuda.hpp:30,
from src/cpp/cuda.cpp:4:
bpl-subset/bpl_subset/boost/smart_ptr/shared_ptr.hpp:146:65: warning: ‘template class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
template< class T, class R > struct sp_enable_if_auto_ptr< std::auto_ptr< T >, R >
^~~~~~~~
In file included from /usr/include/c++/8/memory:80,
from bpl-subset/bpl_subset/boost/config/no_tr1/memory.hpp:21,
from bpl-subset/bpl_subset/boost/smart_ptr/shared_ptr.hpp:27,
from bpl-subset/bpl_subset/boost/shared_ptr.hpp:17,
from src/cpp/cuda.hpp:30,
from src/cpp/cuda.cpp:4:
/usr/include/c++/8/bits/unique_ptr.h:53:28: note: declared here
template class auto_ptr;
^~~~~~~~
In file included from bpl-subset/bpl_subset/boost/shared_ptr.hpp:17,
from src/cpp/cuda.hpp:30,
from src/cpp/cuda.cpp:4:
bpl-subset/bpl_subset/boost/smart_ptr/shared_ptr.hpp:285:30: warning: ‘template class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
explicit shared_ptr(std::auto_ptr & r): px(r.get()), pn()
^~~~~~~~
In file included from /usr/include/c++/8/memory:80,
from bpl-subset/bpl_subset/boost/config/no_tr1/memory.hpp:21,
from bpl-subset/bpl_subset/boost/smart_ptr/shared_ptr.hpp:27,
from bpl-subset/bpl_subset/boost/shared_ptr.hpp:17,
from src/cpp/cuda.hpp:30,
from src/cpp/cuda.cpp:4:
/usr/include/c++/8/bits/unique_ptr.h:53:28: note: declared here
template class auto_ptr;
^~~~~~~~
In file included from bpl-subset/bpl_subset/boost/shared_ptr.hpp:17,
from src/cpp/cuda.hpp:30,
from src/cpp/cuda.cpp:4:
bpl-subset/bpl_subset/boost/smart_ptr/shared_ptr.hpp:329:34: warning: ‘template class std::auto_ptr’ is deprecated [-Wdeprecated-declarations]
shared_ptr & operator=( std::auto_ptr & r )
^~~~~~~~
In file included from /usr/include/c++/8/memory:80,

1 Like
11

I’ve had a lot of problems in the past with cuda (nvcc) and gcc version mismatch. The cuda compiler (nvcc) shipped with older cuda versions might not be compatible with the version of gcc that your system is using (e.g. nvcc wants gcc vesion <7, but ubuntu has default gcc >9).
I cannot find a suitable table of compatibilities, but you can always try to use a lower version of gcc / g++ and see if that works.

I have had success installing multiple gcc (and g++) versions, e.g.
sudo apt install gcc-9 g++9
sudo apt install gcc-8 g++8

and then using update-alternatives to manage which version is used at any given time (update-alternatives(1) - Linux manual page)

e.g.
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++9 1
update-alternatives --install /usr/bin/g++ g++ /usr/bin/g+±8 1

#and then select the one you need with
update-alternatives --config g++

Do the same for gcc, then try to build pycuda again.
Do not forget to configure the gcc / g++ back to the default version once you are done.

1 Like
12

Hi, just spitballing here, but I’ve had similar issues when we installed our GTX 4090 and had to reinstall pycuda in our existing cryosparc installation.
‘CU_TARGET_COMPUTE_90’ is what the 4090 wants, but not all cuda versions even know that compute capabilities “90” exist (cuda > 11.7 I think?)

Maybe cryosparc is referencing compute 90 but the pycuda version that is compatible with your cuda does not know what cryosparc is talking about.

When I had these problems with the new 4090, I solved them by installing cuda 11.8 and then forcing cryosparc to reinstall cuda with

cryosparc_worker/bin/cryosparcw newcuda <path_to_cuda>

which forces dependency reinstall as well.

Worth a shot?

1 Like
13

We’re using cuda 11.8 on our devices already since we’re using RTX A5000 and A4500 GPUs. Same error occurs across all of them.

It almost seems like there’s a configuration error with pytorch. Like cryosparc is using a non-standard configuration and it’s making pycuda freak out.

1 Like
14

Testing the 3D flex refine install on a system which has just finished a big run (and project is stored safely…) Ubuntu 22.04, with CUDA 11.8.

Update to 4.1.0 went smoothly. 3D flex refine install went smoothly until it hit PyCUDA, then did the now recognisable PyCUDA build fail.

During the 3D flex install, it seems to be downloading a mix of CUDA 11.7 and CUDA 12. CUDA 12 currently causes a world of pain for all of the 3DEM programs I’ve actually tried it with.

The PyCUDA build references very old compute targets, which might have been deprecated in CUDA 12?

Would anyone else please check whether they see cuda-cudart-12, cuda-cudart-dev-12 etc in their install list?

Testing a manual PyCUDA install now, but unlike getting it working for Arch, this is just sitting on “Solving environment” indefinitely.

Can provide logs if necessary. Updates if the manual PyCUDA install works.

Update: manually installing PyCUDA eventually throws dozens of compatibility errors (but anaconda took about three hours to work it all out). Will revert to non-3D Flex install for now.

1 Like
15

Hey folks,

Had the same issue and figured out how to fix it. I think @rbs_sci is right and it’s a mix of cuda version which is the issue.

I simply change the line of cryosparcw from

conda install -y cuda-nvcc=11.7 cuda-toolkit=11.7 -c nvidia

to

conda install -y cuda-nvcc=11.7 cuda-toolkit=11.7 -c “nvidia/label/cuda-11.7.0”

This force the usage of 11.7 for all packages and it seems to work afterwards

Hope that helps.

10 Likes
16

Hey @scaiola what do you mean exactly?

I simply change the line of cryosparcw from xxxxxxxx

Could you provide a little more elaborate step description?
Thanks!

1 Like
17

Can confirm this works for us too, thank you!!

@stavros you need to edit the cryosparcw script (used to install the 3D-flex dependencies), located in the worker/bin directory, editing the conda install line in the way that @scaiola showed works. Be careful when making the edit if you are pasting the new line in - there are multiple types of quotation marks, and those that get pasted can have formatting that you don’t want, so I would recommend typing rather than copy-pasting.

Cheers
Oli

EDIT:
This worked for the Ubuntu system. For the CentOS one it is not quite there:

Requirement already satisfied: pytools>=2011.2 in ./deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages (from pycuda==2020.1) (2020.4.4)
Requirement already satisfied: decorator>=3.2.0 in ./deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages (from pycuda==2020.1) (4.4.2)
Requirement already satisfied: appdirs>=1.4.0 in ./deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages (from pycuda==2020.1) (1.4.4)
Requirement already satisfied: mako in ./deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages (from pycuda==2020.1) (1.1.6)
Requirement already satisfied: numpy>=1.6.0 in ./deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages (from pytools>=2011.2->pycuda==2020.1) (1.19.5)
Requirement already satisfied: MarkupSafe>=0.9.2 in ./deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages (from mako->pycuda==2020.1) (2.0.1)
Building wheels for collected packages: pycuda
  Building wheel for pycuda (setup.py) ... done
  Created wheel for pycuda: filename=pycuda-2020.1-cp37-cp37m-linux_x86_64.whl size=615783 sha256=329190f5458cf4d0e958ca89ab1226de960ff4bc47c9032d7f90bbe8a2d2aad5
  Stored in directory: /home/tmp/pip-ephem-wheel-cache-01nqtq71/wheels/fe/c9/2f/377db1b07f46ef88920cd6e533c2f0e1d0d0e3a5dbac1997bb
Successfully built pycuda
Installing collected packages: pycuda
Successfully installed pycuda-2020.1
PyTorch not installed correctly, or NVIDIA GPU not detected.

EDIT2:
Fixed! Had to unset LD_LIBRARY_PATH first.

Well… almost fixed. On our Ubuntu machine all runs fine. On CentOS it says dependencies installed successfully, but any 3D-flex job “terminates abnormally”, with no useful info in the joblog.

EDIT3: OK now it is working on CentOS, but I have no idea why or what changed…

2 Likes
18

Thanks @stavros great spotting.
The double quotation marks are not needed and it also compiles with cuda-11.7.1 label:

conda install -y cuda-nvcc=11.7 cuda-toolkit=11.7 -c nvidia/label/cuda-11.7.1

But would 3Dflex not work out of the box if one just did a normal worker install, compiled against external cuda-11.7
Followed by:

pip install torch ninja

And why suddenly the need for internal cuda env, when the rest of cryosparc uses system cuda env?

Best,
Jesper

EDIT: I just tested this. 3DFlex does not need the internal CUDA env in anaconda. It can be build against system CUDA installation. Just run normal update from worker directory:

eval $(bin/cryosparcw env)
bin/cryosparcw update
pip install torch ninja

Now pycuda in cryosparc is build against system CUDA defined in config.sh and it even runs.

1 Like
19

This gets us through the installation process as well. Brilliant.

1 Like
20

changing the line of cryosparcw from:

conda install -y cuda-nvcc=11.7 cuda-toolkit=11.7 -c nvidia

to

conda install -y cuda-nvcc=11.7 cuda-toolkit=11.7 -c “nvidia/label/cuda-11.7.0”

worked for us.

Thanks @scaiola

1 Like