All to cuStreamCreate results in CUDA_ERROR_OPERATING_SYSTEM after upgrading to v4.5.1

HamishGBrown · May 15, 2024, 11:47pm

Encountering the following error on 3 different nodes of a master-worker install (Linux Mint 20.1) during motion correction in cryosparc live (v4.5.1)

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py", line 381, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.rtp_worker
  File "cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py", line 450, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.process_movie
  File "cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py", line 566, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.do_patch_motion
  File "cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py", line 567, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.do_patch_motion
  File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 314, in cryosparc_master.cryosparc_compute.gpu.gpucore.EngineBaseThread.__init__
  File "/home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/devices.py", line 232, in _require_cuda_context
    return fn(*args, **kws)
  File "/home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/api.py", line 330, in stream
    return current_context().create_stream()
  File "/home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 1480, in create_stream
    handle = driver.cuStreamCreate(flags)
  File "/home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 348, in safe_cuda_api_call
    return self._check_cuda_python_error(fname, libfn(*args))
  File "/home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py", line 408, in _check_cuda_python_error
    raise CudaAPIError(retcode, msg)
numba.cuda.cudadrv.driver.CudaAPIError: [CUresult.CUDA_ERROR_OPERATING_SYSTEM] Call to cuStreamCreate results in CUDA_ERROR_OPERATING_SYSTEM

Other bits requested by the discussion forum guidelines:

$ uname -a && free -g
Linux Freyja 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
              total        used        free      shared  buff/cache   available
Mem:            187          12           0           1         173         171
Swap:             1           1           0```
$ nvidia-smi
Thu May 16 09:44:35 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:18:00.0 Off |                  N/A |
| 50%   53C    P2   134W / 350W |  11096MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:3B:00.0 Off |                  N/A |
| 30%   38C    P8    23W / 350W |      8MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  Off  | 00000000:86:00.0 Off |                  N/A |
| 30%   37C    P8    16W / 350W |      8MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  Off  | 00000000:AF:00.0 Off |                  N/A |
| 30%   34C    P8    16W / 350W |      8MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2257      G   /usr/lib/xorg/Xorg                  4MiB |
|    0   N/A  N/A   2347863      C   ...nvs/relion-5.0/bin/python    11086MiB |
|    1   N/A  N/A      2257      G   /usr/lib/xorg/Xorg                  4MiB |
|    2   N/A  N/A      2257      G   /usr/lib/xorg/Xorg                  4MiB |
|    3   N/A  N/A      2257      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+

Are you able to assist with this issue?

wtempel · May 16, 2024, 3:31pm

Please can you add the following details:

From which version of CryoSPARC did you upgrade?
Did CryoSPARC GPU-accelerated jobs run properly before the most recent CryoSPARC upgrade?
Did any other system changes (such as upgrade of the nvidia driver) coincide with the CryoSPARC upgrade?
How (file names, commands) was the nvidia driver installed?
Outputs of the commands on the worker node where CUDA_ERROR_OPERATING_SYSTEM occurred:

/home/bio21em2/Software/Cryosparc/cryosparc_worker/bin/cryosparcw env | grep PATH
/sbin/ldconfig -p | grep libcu
/home/bio21em2/Software/Cryosparc/cryosparc_worker/bin/cryosparcw call numba -s
/home/bio21em2/Software/Cryosparc/cryosparc_worker/bin/cryosparcw call python -c "from numba import cuda; cuda.cudadrv.libs.test()"

HamishGBrown · May 16, 2024, 10:26pm

Hi @wtempel,

Thanks for your response, see below:

From which version of CryoSPARC did you upgrade?

v4.4.1

Did CryoSPARC GPU-accelerated jobs run properly before the most recent CryoSPARC upgrade?

Yes, though please note that all jobs except the live ones seem to be running normally and, after doing a bit of digging, it seemed that the issue only affected on the of the workers, the other worker thread seemed to process movies normally. The “Test GPU” jobs also work on all the nodes (I do this routinely after an upgrade).

Did any other system changes (such as upgrade of the nvidia driver) coincide with the CryoSPARC upgrade?

No system changes in the past month before upgrade

How (file names, commands) was the nvidia driver installed?

We use the Linux Mint Driver Manager to install nvidia drivers.

Outputs of the commands on the worker node where CUDA_ERROR_OPERATING_SYSTEM occurred:

$ /home/bio21em2/Software/Cryosparc/cryosparc_worker/bin/cryosparcw env | grep PATH
export "CRYOSPARC_PATH=/home/bio21em2/Software/Cryosparc/cryosparc_worker/bin"
export "CRYOSPARC_CUDA_PATH=/usr/local/cuda-11.2"
export "PATH=/home/bio21em2/Software/Cryosparc/cryosparc_worker/bin:/home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/condabin:/home/bio21em2/.local/bin:/home/bio21em1/Scripts:/home/bio21em2/Software/Cryosparc/cryosparc_master/bin:/home/bio21em1/.aspera/connect/bin:/home/bio21em1/Software/mediaflux-data-mover/bin:/home/bio21em1/anaconda3/condabin:/home/bio21em1/Software/relion/build/bin:/usr/local/ccpem-1.5.0/bin:/usr/local/phenix-1.19.2-4158/build/bin:/usr/local/Particle/bin:/usr/local/IMOD/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"
export "LD_LIBRARY_PATH=/public/em/RELION/relion/lib:"
export "PYTHONPATH=/home/bio21em2/Software/Cryosparc/cryosparc_worker"
export "NUMBA_CUDA_INCLUDE_PATH=/home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include"

$ /sbin/ldconfig -p | grep libcu
	libcusparse.so.10 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcusparse.so.10
	libcusparse.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcusparse.so
	libcusolverMg.so.10 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcusolverMg.so.10
	libcusolverMg.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcusolverMg.so
	libcusolver.so.10 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcusolver.so.10
	libcusolver.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcusolver.so
	libcurl.so.4 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcurl.so.4
	libcurl-gnutls.so.4 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcurl-gnutls.so.4
	libcurand.so.10 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcurand.so.10
	libcurand.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcurand.so
	libcupti.so.10.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcupti.so.10.1
	libcupti.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcupti.so
	libcupsimage.so.2 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcupsimage.so.2
	libcupsfilters.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcupsfilters.so.1
	libcups.so.2 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcups.so.2
	libcuinj64.so.10.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuinj64.so.10.1
	libcuinj64.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuinj64.so
	libcufftw.so.10 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcufftw.so.10
	libcufftw.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcufftw.so
	libcufft.so.10 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcufft.so.10
	libcufft.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcufft.so
	libcudart.so.10.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcudart.so.10.1
	libcudart.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcudart.so
	libcudadebugger.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcudadebugger.so.1
	libcuda.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so.1
	libcuda.so.1 (libc6) => /lib/i386-linux-gnu/libcuda.so.1
	libcuda.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so
	libcuda.so (libc6) => /lib/i386-linux-gnu/libcuda.so
	libcublasLt.so.10 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcublasLt.so.10
	libcublasLt.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcublasLt.so
	libcublas.so.10 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcublas.so.10
	libcublas.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcublas.so

$ /home/bio21em2/Software/Cryosparc/cryosparc_worker/bin/cryosparcw call numba -s
System info:
--------------------------------------------------------------------------------
__Time Stamp__
Report started (local time)                   : 2024-05-17 08:24:42.748822
UTC start time                                : 2024-05-16 22:24:42.748826
Running time (s)                              : 3.869798

__Hardware Information__
Machine                                       : x86_64
CPU Name                                      : skylake-avx512
CPU Count                                     : 48
Number of accessible CPUs                     : 48
List of accessible CPUs cores                 : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
CFS Restrictions (CPUs worth of runtime)      : None

CPU Features                                  : 64bit adx aes avx avx2 avx512bw
                                                avx512cd avx512dq avx512f avx512vl
                                                bmi bmi2 clflushopt clwb cmov
                                                crc32 cx16 cx8 f16c fma fsgsbase
                                                fxsr invpcid lzcnt mmx movbe
                                                pclmul pku popcnt prfchw rdrnd
                                                rdseed rtm sahf sse sse2 sse3
                                                sse4.1 sse4.2 ssse3 xsave xsavec
                                                xsaveopt xsaves

Memory Total (MB)                             : 192067
Memory Available (MB)                         : 175605

__OS Information__
Platform Name                                 : Linux-5.4.0-166-generic-x86_64-with-glibc2.31
Platform Release                              : 5.4.0-166-generic
OS Name                                       : Linux
OS Version                                    : #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023
OS Specific Version                           : ?
Libc Version                                  : glibc 2.31

__Python Information__
Python Compiler                               : GCC 12.3.0
Python Implementation                         : CPython
Python Version                                : 3.10.13
Python Locale                                 : en_AU.UTF-8

__Numba Toolchain Versions__
Numba Version                                 : 0.59.0
llvmlite Version                              : 0.42.0

__LLVM Information__
LLVM Version                                  : 14.0.6

__CUDA Information__
CUDA Device Initialized                       : True
CUDA Driver Version                           : 12.0
CUDA Runtime Version                          : 11.8
CUDA NVIDIA Bindings Available                : True
CUDA NVIDIA Bindings In Use                   : True
CUDA Minor Version Compatibility Available    : False
CUDA Minor Version Compatibility Needed       : False
CUDA Minor Version Compatibility In Use       : False
CUDA Detect Output:
Found 4 CUDA devices
id 0    NVIDIA GeForce RTX 3090                                                                                              [SUPPORTED]
                      Compute Capability: 8.6
                           PCI Device ID: 0
                              PCI Bus ID: 24
                                    UUID: GPU-14613529-d676-130d-0560-f69ad0eec811
                                Watchdog: Enabled
             FP32/FP64 Performance Ratio: 32
id 1    NVIDIA GeForce RTX 3090                                                                                              [SUPPORTED]
                      Compute Capability: 8.6
                           PCI Device ID: 0
                              PCI Bus ID: 59
                                    UUID: GPU-f44d55a5-c8e6-9c34-8ff4-00192183ee93
                                Watchdog: Enabled
             FP32/FP64 Performance Ratio: 32
id 2    NVIDIA GeForce RTX 3090                                                                                              [SUPPORTED]
                      Compute Capability: 8.6
                           PCI Device ID: 0
                              PCI Bus ID: 134
                                    UUID: GPU-1ced1ef8-7949-36aa-01e3-c1d824406646
                                Watchdog: Enabled
             FP32/FP64 Performance Ratio: 32
id 3    NVIDIA GeForce RTX 3090                                                                                              [SUPPORTED]
                      Compute Capability: 8.6
                           PCI Device ID: 0
                              PCI Bus ID: 175
                                    UUID: GPU-74950c4a-1706-8c44-212d-e1c2f3883415
                                Watchdog: Enabled
             FP32/FP64 Performance Ratio: 32
Summary:
	4/4 devices are supported

CUDA Libraries Test Output:
Finding driver from candidates:
	libcuda.so
	libcuda.so.1
	/usr/lib/libcuda.so
	/usr/lib/libcuda.so.1
	/usr/lib64/libcuda.so
	/usr/lib64/libcuda.so.1
Using loader <class 'ctypes.CDLL'>
	Trying to load driver...	ok
		Loaded from libcuda.so
	Mapped libcuda.so paths:
		/usr/lib/x86_64-linux-gnu/libcuda.so.525.147.05
Finding nvvm from Conda environment (NVIDIA package)
	Located at /home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/nvvm/lib64/libnvvm.so.4.0.0
	Trying to open library...	ok
Finding nvrtc from Conda environment (NVIDIA package)
	Located at /home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/libnvrtc.so.11.8.89
	Trying to open library...	ok
Finding cudart from Conda environment (NVIDIA package)
	Located at /home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/libcudart.so.11.8.89
	Trying to open library...	ok
Finding cudadevrt from Conda environment (NVIDIA package)
	Located at libcudadevrt.a
	Checking library...	ERROR: failed to find cudadevrt:
libcudadevrt.a not found
Finding libdevice from Conda environment (NVIDIA package)
	Located at /home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/nvvm/libdevice/libdevice.10.bc
	Checking library...	ok


__NumPy Information__
NumPy Version                                 : 1.22.4
NumPy Supported SIMD features                 : ('MMX', 'SSE', 'SSE2', 'SSE3', 'SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX', 'F16C', 'FMA3', 'AVX2', 'AVX512F', 'AVX512CD', 'AVX512VL', 'AVX512BW', 'AVX512DQ', 'AVX512_SKX')
NumPy Supported SIMD dispatch                 : ('SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX', 'F16C', 'FMA3', 'AVX2', 'AVX512F', 'AVX512CD', 'AVX512_KNL', 'AVX512_KNM', 'AVX512_SKX', 'AVX512_CLX', 'AVX512_CNL', 'AVX512_ICL')
NumPy Supported SIMD baseline                 : ('SSE', 'SSE2', 'SSE3')
NumPy AVX512_SKX support detected             : True

__SVML Information__
SVML State, config.USING_SVML                 : False
SVML Library Loaded                           : False
llvmlite Using SVML Patched LLVM              : True
SVML Operational                              : False

__Threading Layer Information__
TBB Threading Layer Available                 : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available              : True
+-->Vendor: GNU
Workqueue Threading Layer Available           : True
+-->Workqueue imported successfully.

__Numba Environment Variable Information__
NUMBA_CUDA_INCLUDE_PATH                       : /home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include
NUMBA_CUDA_USE_NVIDIA_BINDING                 : 1
NUMBA_CUDA_MAX_PENDING_DEALLOCS_COUNT         : 0

__Conda Information__
Conda Build                                   : not installed
Conda Env                                     : 24.1.2
Conda Platform                                : linux-64
Conda Python Version                          : 3.10.14.final.0
Conda Root Writable                           : True

__Installed Packages__
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   2.1.0                    pypi_0    pypi
aom                       3.5.0                h27087fc_0    conda-forge
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
astunparse                1.6.3                    pypi_0    pypi
bcrypt                    3.2.2           py310h5764c6d_1    conda-forge
blinker                   1.7.0              pyhd8ed1ab_0    conda-forge
blosc                     1.21.5               h0f2a231_0    conda-forge
brotli                    1.0.9                h166bdaf_9    conda-forge
brotli-bin                1.0.9                h166bdaf_9    conda-forge
brotli-python             1.0.9           py310hd8f1fbe_9    conda-forge
brunsli                   0.1                  h9c3ff4c_0    conda-forge
bzip2                     1.0.8                hd590300_5    conda-forge
c-ares                    1.27.0               hd590300_0    conda-forge
c-blosc2                  2.13.2               hb4ffafa_0    conda-forge
ca-certificates           2024.2.2             hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                5.3.3                    pypi_0    pypi
certifi                   2024.2.2           pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py310h2fee648_0    conda-forge
cfitsio                   4.2.0                hd9d235c_0    conda-forge
charls                    2.3.4                h9c3ff4c_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
contourpy                 1.2.0           py310hd41b1e2_0    conda-forge
cryptography              42.0.5          py310h75e40e8_0    conda-forge
cuda-cudart               11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-nvrtc                11.8.89                       0    nvidia/label/cuda-11.8.0
cuda-python               11.8.3                   pypi_0    pypi
cuda-version              11.8                 h70ddcb2_3    conda-forge
cycler                    0.12.1             pyhd8ed1ab_0    conda-forge
cython                    3.0.9           py310hc6cd4ac_0    conda-forge
dav1d                     1.2.1                hd590300_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
dnspython                 2.6.1              pyhd8ed1ab_1    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
executing                 2.0.1              pyhd8ed1ab_0    conda-forge
fftw                      3.3.10          nompi_hc118613_108    conda-forge
filelock                  3.13.1                   pypi_0    pypi
flask                     2.2.3              pyhd8ed1ab_0    conda-forge
flask-jsonrpc             0.3.1                    pypi_0    pypi
flask-pymongo             2.3.0                    pypi_0    pypi
flatbuffers               24.3.7                   pypi_0    pypi
fonttools                 4.49.0          py310h2372a71_0    conda-forge
freetype                  2.12.1               h267a509_2    conda-forge
fsspec                    2024.2.0                 pypi_0    pypi
gast                      0.5.4                    pypi_0    pypi
giflib                    5.2.1                h0b41bf4_3    conda-forge
gmp                       6.3.0                h59595ed_1    conda-forge
google-auth               2.28.2                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.46.3          py310hba10ccf_0    conda-forge
h5py                      3.10.0          nompi_py310h65828d5_101    conda-forge
hdf5                      1.14.3          nompi_h4f84152_100    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
imagecodecs               2022.9.26       py310h543e91f_4    conda-forge
imageio                   2.34.0             pyh4b66e23_0    conda-forge
importlib-metadata        7.0.2              pyha770c72_0    conda-forge
ipython                   8.22.2             pyh707e725_0    conda-forge
itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h0b41bf4_3    conda-forge
jxrlib                    1.1                  hd590300_3    conda-forge
keras                     2.8.0                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.5           py310hd41b1e2_1    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
lazy_loader               0.3                pyhd8ed1ab_0    conda-forge
lcms2                     2.14                 h6ed2654_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libaec                    1.1.2                h59595ed_1    conda-forge
libavif                   0.11.1               h8182462_2    conda-forge
libblas                   3.9.0           21_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_9    conda-forge
libbrotlidec              1.0.9                h166bdaf_9    conda-forge
libbrotlienc              1.0.9                h166bdaf_9    conda-forge
libcblas                  3.9.0           21_linux64_openblas    conda-forge
libclang                  16.0.6                   pypi_0    pypi
libcufft                  10.9.0.58                     0    nvidia/label/cuda-11.8.0
libcurand                 10.3.0.86                     0    nvidia/label/cuda-11.8.0
libcurand-dev             10.3.0.86                     0    nvidia/label/cuda-11.8.0
libcurl                   8.5.0                hca28451_0    conda-forge
libcusolver               11.4.1.48                     0    nvidia/label/cuda-11.8.0
libcusparse               11.7.5.86                     0    nvidia/label/cuda-11.8.0
libdeflate                1.14                 h166bdaf_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgfortran-ng            13.2.0               h69a702a_5    conda-forge
libgfortran5              13.2.0               ha4646dd_5    conda-forge
libgomp                   13.2.0               h807b86a_5    conda-forge
liblapack                 3.9.0           21_linux64_openblas    conda-forge
libllvm14                 14.0.6               hcd5def8_4    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnsl                    2.0.1                hd590300_0    conda-forge
libopenblas               0.3.26          pthreads_h413a1c8_0    conda-forge
libpng                    1.6.43               h2797004_0    conda-forge
libprotobuf               3.19.4               h780b84a_0    conda-forge
libsqlite                 3.45.2               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_5    conda-forge
libtiff                   4.4.0                h82bc61c_5    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxcrypt                 4.4.36               hd590300_1    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
libzopfli                 1.0.3                h9c3ff4c_0    conda-forge
llvmlite                  0.42.0          py310h1b8f574_1    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
markdown                  3.5.2                    pypi_0    pypi
markupsafe                2.1.5           py310h2372a71_0    conda-forge
matplotlib-base           3.8.3           py310h62c0568_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mpmath                    1.3.0                    pypi_0    pypi
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.4                  h59595ed_2    conda-forge
networkx                  3.2.1              pyhd8ed1ab_0    conda-forge
numba                     0.59.0          py310h7dc5dd1_1    conda-forge
numpy                     1.22.4          py310h4ef5377_0    conda-forge
oauthlib                  3.2.2              pyhd8ed1ab_0    conda-forge
openjpeg                  2.5.0                h7d73246_1    conda-forge
openssl                   3.2.1                hd590300_0    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 24.0               pyhd8ed1ab_0    conda-forge
pandas                    1.5.3           py310h9b08913_1    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pbzip2                    1.1.13               h1fcc475_2    conda-forge
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.2.0           py310h454ad03_3    conda-forge
pip                       24.0               pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.42             pyha770c72_0    conda-forge
protobuf                  3.19.4          py310h122e73d_0    conda-forge
psutil                    5.9.8           py310h2372a71_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pyasn1                    0.5.1                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pycryptodome              3.20.0          py310hb0f0acc_0    conda-forge
pyfftw                    0.12.0          py310h0885ff1_3    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pyjwt                     2.8.0              pyhd8ed1ab_1    conda-forge
pylibtiff                 0.4.2           py310he9d7d2b_7    conda-forge
pymongo                   4.6.2           py310hc6cd4ac_0    conda-forge
pyparsing                 3.1.2              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.13         hd12c33a_1_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-slugify            5.0.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    4_cp310    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pywavelets                1.4.1           py310h1f7b6fc_1    conda-forge
pyyaml                    6.0.1           py310h2372a71_1    conda-forge
readline                  8.2                  h8228510_1    conda-forge
requests                  2.29.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1              pyhd8ed1ab_0    conda-forge
requests-toolbelt         0.10.1             pyhd8ed1ab_0    conda-forge
rsa                       4.9                      pypi_0    pypi
scikit-image              0.22.0          py310hcc13569_2    conda-forge
scikit-learn              1.4.1.post1     py310h1fdf081_0    conda-forge
scipy                     1.12.0          py310hb13e2d6_2    conda-forge
semver                    2.13.0             pyh9f0ad1d_0    conda-forge
setuptools                69.2.0             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleef                     3.5.1                h9b69904_2    conda-forge
snappy                    1.1.10               h9fff704_0    conda-forge
sniffio                   1.3.1              pyhd8ed1ab_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
sympy                     1.12                     pypi_0    pypi
tabulate                  0.9.0              pyhd8ed1ab_1    conda-forge
tensorboard               2.8.0                    pypi_0    pypi
tensorboard-data-server   0.6.1           py310h600f1e7_4    conda-forge
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.8.4                    pypi_0    pypi
tensorflow-estimator      2.8.0                    pypi_0    pypi
tensorflow-io-gcs-filesystem 0.36.0                   pypi_0    pypi
termcolor                 2.4.0                    pypi_0    pypi
text-unidecode            1.3                pyhd8ed1ab_1    conda-forge
threadpoolctl             3.3.0              pyhc1e730c_0    conda-forge
tifffile                  2022.10.10         pyhd8ed1ab_0    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
torch                     2.1.2+cu118              pypi_0    pypi
traitlets                 5.14.2             pyhd8ed1ab_0    conda-forge
triton                    2.1.0                    pypi_0    pypi
typing-extensions         4.10.0               hd8ed1ab_0    conda-forge
typing_extensions         4.10.0             pyha770c72_0    conda-forge
tzdata                    2024a                h0c530f3_0    conda-forge
unicodedata2              15.1.0          py310h2372a71_0    conda-forge
unidecode                 1.3.8              pyhd8ed1ab_0    conda-forge
urllib3                   1.26.18            pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
werkzeug                  2.3.6              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
wrapt                     1.16.0          py310h2372a71_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zfp                       1.0.1                h59595ed_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zlib-ng                   2.0.7                h0b41bf4_0    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge

No errors reported.


No warnings reported.


$ /home/bio21em2/Software/Cryosparc/cryosparc_worker/bin/cryosparcw call python -c "from numba import cuda; cuda.cudadrv.libs.test()"
Finding driver from candidates:
	libcuda.so
	libcuda.so.1
	/usr/lib/libcuda.so
	/usr/lib/libcuda.so.1
	/usr/lib64/libcuda.so
	/usr/lib64/libcuda.so.1
Using loader <class 'ctypes.CDLL'>
	Trying to load driver...	ok
		Loaded from libcuda.so
	Mapped libcuda.so paths:
		/usr/lib/x86_64-linux-gnu/libcuda.so.525.147.05
Finding nvvm from Conda environment (NVIDIA package)
	Located at /home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/nvvm/lib64/libnvvm.so.4.0.0
	Trying to open library...	ok
Finding nvrtc from Conda environment (NVIDIA package)
	Located at /home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/libnvrtc.so.11.8.89
	Trying to open library...	ok
Finding cudart from Conda environment (NVIDIA package)
	Located at /home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/libcudart.so.11.8.89
	Trying to open library...	ok
Finding cudadevrt from Conda environment (NVIDIA package)
	Located at libcudadevrt.a
	Checking library...	ERROR: failed to find cudadevrt:
libcudadevrt.a not found
Finding libdevice from Conda environment (NVIDIA package)
	Located at /home/bio21em2/Software/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/nvvm/libdevice/libdevice.10.bc
	Checking library...	ok

HamishGBrown · May 19, 2024, 11:41pm

Hi @wtempel, The problem seems to have gone away (without even a system or cryosparc restart ), which is perplexing. I’ll repost if it re-occurs.