Pycuda install error

Hi,

I have been trying to get my CryoSPARC to work for the past 5 days now and cant seem to get past installing pycuda successfully for whatever reason. I am getting the following error when either trying “pip install pycuda” and just trying to connect the worker to the master by running “./cryosparcw connect --master dt-exxact-01 --worker dt-exxact-01 --ssdpath /mnt/data”

Error from command line usage (shortened because of character limit)

Preparing metadata (setup.py) … done
Skipping wheel build for pycuda, due to binaries being disabled for it.
Installing collected packages: pycuda
Running setup.py install for pycuda … error
ERROR: Command errored out with exit status 1:
command: /usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c ‘import io, os, sys, setuptools, tokenize; sys.argv[0] = ‘"’"’/tmp/pip-req-build-3nygm_zw/setup.py’"’"’; file=’"’"’/tmp/pip-req-build-3nygm_zw/setup.py’"’"’;f = getattr(tokenize, ‘"’"‘open’"’"’, open)(file) if os.path.exists(file) else io.StringIO(’"’"‘from setuptools import setup; setup()’"’"’);code = f.read().replace(’"’"’\r\n’"’"’, ‘"’"’\n’"’"’);f.close();exec(compile(code, file, ‘"’"‘exec’"’"’))’ install --record /tmp/pip-record-iq5vckgp/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m/pycuda
cwd: /tmp/pip-req-build-3nygm_zw/
Complete output (1020 lines):
*************************************************************
*** I have detected that you have not run configure.py.
*************************************************************
*** Additionally, no global config files were found.
*** I will go ahead with the default configuration.
*** In all likelihood, this will not work out.
***
*** See README_SETUP.txt for more information.
***
*** If the build does fail, just re-run configure.py with the
*** correct arguments, and then retry. Good luck!
*************************************************************
*** HIT Ctrl-C NOW IF THIS IS NOT WHAT YOU WANT
*************************************************************
Continuing in 10 seconds…
Continuing in 9 seconds…
Continuing in 8 seconds…
Continuing in 7 seconds…
Continuing in 6 seconds…
Continuing in 5 seconds…
Continuing in 4 seconds…
Continuing in 3 seconds…
Continuing in 2 seconds…
Continuing in 1 seconds…
/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/distutils/dist.py:274: UserWarning: Unknown distribution option: ‘test_requires’
warnings.warn(msg)
running install
/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/setuptools/command/install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
setuptools.SetuptoolsDeprecationWarning,
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/pycuda
copying pycuda/init.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/scan.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/tools.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/cumath.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/compiler.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/reduction.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/debug.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/_cluda.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/_mymako.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/characterize.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/curandom.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/gpuarray.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/driver.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/autoinit.py → build/lib.linux-x86_64-3.7/pycuda
copying pycuda/elementwise.py → build/lib.linux-x86_64-3.7/pycuda
creating build/lib.linux-x86_64-3.7/pycuda/gl
copying pycuda/gl/init.py → build/lib.linux-x86_64-3.7/pycuda/gl
copying pycuda/gl/autoinit.py → build/lib.linux-x86_64-3.7/pycuda/gl
creating build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/init.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/operator.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/coordinate.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/packeted.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/cg.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/inner.py → build/lib.linux-x86_64-3.7/pycuda/sparse
copying pycuda/sparse/pkt_build.py → build/lib.linux-x86_64-3.7/pycuda/sparse
creating build/lib.linux-x86_64-3.7/pycuda/compyte
copying pycuda/compyte/init.py → build/lib.linux-x86_64-3.7/pycuda/compyte
copying pycuda/compyte/dtypes.py → build/lib.linux-x86_64-3.7/pycuda/compyte
copying pycuda/compyte/array.py → build/lib.linux-x86_64-3.7/pycuda/compyte
running egg_info
writing pycuda.egg-info/PKG-INFO
writing dependency_links to pycuda.egg-info/dependency_links.txt
writing requirements to pycuda.egg-info/requires.txt
writing top-level names to pycuda.egg-info/top_level.txt
reading manifest file ‘pycuda.egg-info/SOURCES.txt’
reading manifest template ‘MANIFEST.in’
warning: no files found matching ‘doc/source/_static/.css’
warning: no files found matching 'doc/source/_templates/
.html’
warning: no files found matching ‘.cpp’ under directory ‘bpl-subset/bpl_subset/boost’
warning: no files found matching '
.html’ under directory ‘bpl-subset/bpl_subset/boost’
warning: no files found matching ‘.inl’ under directory ‘bpl-subset/bpl_subset/boost’
warning: no files found matching '
.txt’ under directory ‘bpl-subset/bpl_subset/boost’
warning: no files found matching ‘.h’ under directory ‘bpl-subset/bpl_subset/libs’
warning: no files found matching '
.ipp’ under directory ‘bpl-subset/bpl_subset/libs’
warning: no files found matching ‘*.pl’ under directory ‘bpl-subset/bpl_subset/libs’
adding license file ‘LICENSE’
writing manifest file ‘pycuda.egg-info/SOURCES.txt’
creating build/lib.linux-x86_64-3.7/pycuda/cuda
copying pycuda/cuda/pycuda-complex-impl.hpp → build/lib.linux-x86_64-3.7/pycuda/cuda
copying pycuda/cuda/pycuda-complex.hpp → build/lib.linux-x86_64-3.7/pycuda/cuda
copying pycuda/cuda/pycuda-helpers.hpp → build/lib.linux-x86_64-3.7/pycuda/cuda
copying pycuda/sparse/pkt_build_cython.pyx → build/lib.linux-x86_64-3.7/pycuda/sparse
running build_ext
building ‘_driver’ extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/src
creating build/temp.linux-x86_64-3.7/src/cpp
creating build/temp.linux-x86_64-3.7/src/wrapper
creating build/temp.linux-x86_64-3.7/bpl-subset
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/object
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/python/src/converter
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/system
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/system/src
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/smart_ptr
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/smart_ptr/src
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/thread
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/thread/src
creating build/temp.linux-x86_64-3.7/bpl-subset/bpl_subset/libs/thread/src/pthread
gcc -pthread -B /usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/compiler_compat -Wl,–sysroot=/ -Wsign-compare -fwrapv -Wall -O3 -DNDEBUG -fPIC -DBOOST_ALL_NO_LIB=1 -DBOOST_THREAD_BUILD_DLL=1 -DBOOST_MULTI_INDEX_DISABLE_SERIALIZATION=1 -DBOOST_PYTHON_SOURCE=1 -Dboost=pycudaboost -DBOOST_THREAD_DONT_USE_CHRONO=1 -DPYGPU_PACKAGE=pycuda -DPYGPU_PYCUDA=1 -DHAVE_CURAND=1 -Isrc/cpp -Ibpl-subset/bpl_subset -I/usr/local/cuda/include -I/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/numpy/core/include -I/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m -c src/cpp/cuda.cpp -o build/temp.linux-x86_64-3.7/src/cpp/cuda.o
In file included from /usr/include/stdlib.h:235:0,
from /usr/local/cuda/include/cuda.h:53,
from src/cpp/cuda.hpp:14,
from src/cpp/cuda.cpp:4:
/usr/include/locale.h:152:8: error: ‘__locale_t’ does not name a type
extern __locale_t newlocale (int __category_mask, const char *__locale,
error: command ‘gcc’ failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python3.7 -u -c ‘import io, os, sys, setuptools, tokenize; sys.argv[0] = ‘"’"’/tmp/pip-req-build-3nygm_zw/setup.py’"’"’; file=’"’"’/tmp/pip-req-build-3nygm_zw/setup.py’"’"’;f = getattr(tokenize, ‘"’"‘open’"’"’, open)(file) if os.path.exists(file) else io.StringIO(’"’"‘from setuptools import setup; setup()’"’"’);code = f.read().replace(’"’"’\r\n’"’"’, ‘"’"’\n’"’"’);f.close();exec(compile(code, file, ‘"’"‘exec’"’"’))’ install --record /tmp/pip-record-iq5vckgp/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.7m/pycuda Check the logs for full command output.

Error from CryoSPARC failed 3D Job:

Traceback (most recent call last):
File “/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7f221c01f090>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/run.py”, line 93, in cryosparc_compute.run.main
File “cryosparc_worker/cryosparc_compute/jobs/class3D/run.py”, line 203, in cryosparc_compute.jobs.class3D.run.run_class_3D
File “cryosparc_worker/cryosparc_compute/engine/newengine.py”, line 2980, in cryosparc_compute.engine.newengine.reconstruct
File “cryosparc_worker/cryosparc_compute/engine/newengine.py”, line 2992, in cryosparc_compute.engine.newengine.reconstruct
File “cryosparc_worker/cryosparc_compute/engine/newengine.py”, line 1728, in cryosparc_compute.engine.newengine.EngineThread.allocate_backproject_accumulators
File “/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py”, line 549, in fill
func = elementwise.get_fill_kernel(self.dtype)
File “”, line 2, in get_fill_kernel
File “/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 433, in context_dependent_memoize
result = func(*args)
File “/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 498, in get_fill_kernel
“fill”)
File “/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 163, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File “/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 149, in get_elwise_kernel_and_types
keep, options, **kwargs)
File “/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 76, in get_elwise_module
options=options, keep=keep, no_extern_c=True)
File “/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 291, in init
arch, code, cache_dir, include_dirs)
File “/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File “/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 137, in compile_plain
stderr=stderr.decode(“utf-8”, “replace”))
pycuda.driver.CompileError: nvcc compilation of /tmp/tmpex7k8phk/kernel.cu failed
[command: nvcc --cubin -arch sm_75 -I/usr/local/CryoSPARC/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda kernel.cu]
[stderr:
/usr/include/locale.h(152): error: identifier “__locale_t” is undefined
/usr/include/locale.h(153): error: identifier “__locale_t” is undefined
/usr/include/locale.h(187): error: identifier “__locale_t” is undefined
/usr/include/locale.h(187): error: identifier “__locale_t” is undefined
/usr/include/locale.h(191): error: identifier “__locale_t” is undefined
/usr/include/locale.h(198): error: identifier “__locale_t” is undefined
/usr/include/locale.h(198): error: identifier “__locale_t” is undefined
/usr/include/string.h(162): error: identifier “__locale_t” is undefined
/usr/include/string.h(163): error: attribute “nonnull” references nonpointer parameter
/usr/include/string.h(166): error: identifier “__locale_t” is undefined
/usr/include/string.h(166): error: attribute “nonnull” references nonpointer parameter
/usr/include/string.h(441): error: identifier “__locale_t” is undefined
/usr/include/string.h(544): error: identifier “__locale_t” is undefined
/usr/include/string.h(545): error: attribute “nonnull” references nonpointer parameter
/usr/include/string.h(548): error: identifier “__locale_t” is undefined
/usr/include/string.h(549): error: attribute “nonnull” references nonpointer parameter
/usr/include/time.h(226): error: identifier “__locale_t” is undefined
/usr/include/time.h(232): error: identifier “__locale_t” is undefined
/usr/include/stdlib.h(241): error: identifier “__locale_t” is undefined
/usr/include/stdlib.h(241): error: attribute “nonnull” references nonpointer parameter
/usr/include/stdlib.h(245): error: identifier “__locale_t” is undefined
/usr/include/stdlib.h(246): error: attribute “nonnull” references nonpointer parameter
/usr/include/stdlib.h(251): error: identifier “__locale_t” is undefined
/usr/include/stdlib.h(252): error: attribute “nonnull” references nonpointer parameter
/usr/include/stdlib.h(257): error: identifier “__locale_t” is undefined
/usr/include/stdlib.h(258): error: attribute “nonnull” references nonpointer parameter
/usr/include/stdlib.h(261): error: identifier “__locale_t” is undefined
/usr/include/stdlib.h(262): error: attribute “nonnull” references nonpointer parameter
/usr/include/stdlib.h(265): error: identifier “__locale_t” is undefined
/usr/include/stdlib.h(266): error: attribute “nonnull” references nonpointer parameter
/usr/include/stdlib.h(270): error: identifier “__locale_t” is undefined
/usr/include/stdlib.h(271): error: attribute “nonnull” references nonpointer parameter
32 errors detected in the compilation of “/tmp/tmpxft_00002fe2_00000000-6_kernel.cpp1.ii”.
]

I have tried already reinstalling gcc, c++, python-devel, and essentially every stackoverflow solution that comes with the gcc exit error 1. I really don’t know what else to do at this point. Please help!

Does this topic concern the same system as that topic: Cryosparc ping: socket: Operation not permitted
If so, has the cryosparc_master component been successfully installed?

In that case, please can you:

  • describe how the cuda toolkit was installed on this system
  • redo the cryosparc_worker install.sh
    • post the full install.sh command line
    • post any error messages
  • post output of
    ls -al /usr/local/CryoSPARC
  • if cryosparc_worker/install.sh was successful, please provide output of:

Hi wtempel,

Thank you for suggestions. I ended up just downgrading to the stable CryoSPARC V3.4.0 and everything is running as expected. I really wanted to try the new 3D classification feature but I will need to wait until a stable V4 is released.

To answer your questions,
I was able to get past the issue I had in my previous submitted topic so cryosparc_master does successfully install.

cudatoolkit was installed by downloading cuda-10.1 (cuda_10.1.243_418.87.00_linux.run), running it and checking all options except the nvidia-driver installation.

Output of ls -al /usr/local/CryoSPARC

(base) [exx@dt-exxact-01 CUDA_Install_Files]$ ls -al /usr/local/CryoSPARC
total 2687716
drwxr-xr-x 5 exx root 4096 Oct 12 10:44 .
drwxr-xr-x. 38 exx root 4096 Oct 11 17:37 …
drwxrwxr-x 3 exx exx 12288 Oct 12 12:19 cryosparc_database
drwxrwxr-x 15 exx exx 4096 Oct 12 10:39 cryosparc_master
-rw-rw-r-- 1 exx exx 831226943 Oct 12 09:12 cryosparc_master.tar.gz
drwxrwxr-x 8 exx exx 4096 Oct 12 10:41 cryosparc_worker
-rw-rw-r-- 1 exx exx 1920814924 Oct 12 09:28 cryosparc_worker.tar.gz
-rw-rw-r-- 1 exx exx 1107 Oct 12 09:31 License_exx1.txt
-rw-rw-r-- 1 exx exx 4227 Jul 6 07:37 Make 2D Classes (use icli).ipynb
-rw-rw-r-- 1 exx exx 126694 Aug 31 16:32 Save 2D Class Images.ipynb
(base) [exx@dt-exxact-01 CUDA_Install_Files]$

I have tried the troubleshooting guidelines posted by stephan before and nothing ever appeared out of unordinary. Regardless, here are the outputs:

(base) [exx@dt-exxact-01 CUDA_Install_Files]$ eval $(/usr/local/CryoSPARC/cryosparc_worker/bin/cryosparcw env)
(base) [exx@dt-exxact-01 CUDA_Install_Files]$ echo $CRYOSPARC_CUDA_PATH
/usr/local/cuda
(base) [exx@dt-exxact-01 CUDA_Install_Files]$ ${CRYOSPARC_CUDA_PATH}/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
(base) [exx@dt-exxact-01 CUDA_Install_Files]$ python -c “import pycuda.driver; print(pycuda.driver.get_version())”
(10, 1, 0)
(base) [exx@dt-exxact-01 CUDA_Install_Files]$ uname -a && free -g && nvidia-smi
Linux dt-exxact-01 3.10.0-1160.66.1.el7.x86_64 #1 SMP Wed May 18 16:02:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
total used free shared buff/cache available
Mem: 125 15 0 1 109 107
Swap: 9 0 9
Wed Oct 12 12:17:51 2022
±----------------------------------------------------------------------------+
| NVIDIA-SMI 515.76 Driver Version: 515.76 CUDA Version: 11.7 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce … Off | 00000000:19:00.0 Off | N/A |
| 45% 74C P2 204W / 250W | 3307MiB / 11264MiB | 69% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 NVIDIA GeForce … Off | 00000000:1A:00.0 Off | N/A |
| 31% 45C P8 22W / 250W | 4MiB / 11264MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 2 NVIDIA GeForce … Off | 00000000:67:00.0 Off | N/A |
| 33% 41C P8 4W / 250W | 4MiB / 11264MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 3 NVIDIA GeForce … Off | 00000000:68:00.0 On | N/A |
| 32% 45C P8 9W / 250W | 357MiB / 11264MiB | 11% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 4588 C python 3301MiB |
| 3 N/A N/A 2105 G /usr/bin/X 136MiB |
| 3 N/A N/A 5564 G …cal/Chimera/bin/python2.7 36MiB |
| 3 N/A N/A 50930 G /usr/bin/gnome-shell 34MiB |
| 3 N/A N/A 77757 G /usr/lib64/firefox/firefox 143MiB |
±----------------------------------------------------------------------------+
(base) [exx@dt-exxact-01 CUDA_Install_Files]$