Error in GPU task

Troubleshooting
1

BS"D

During submission of any GPU based task, the following error is received.

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 301, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction File
“cryosparc2_compute/engine/init.py”, line 8, in from engine import * File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 11, in init cryosparc2_compute.engine.engine File “cryosparc2_worker/cryosparc2_compute/engine/gfourier.py”,
line 6, in init cryosparc2_compute.engine.gfourier File “/data/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py”, line 13, in from . import cufft File “/data/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”,
line 238, in _libcufft.cufftSetCompatibilityMode.restype = int File “/data/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/ctypes/init.py”, line 379, in getattr func = self.getitem(name) File “/data/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/ctypes/init.py”,
line 384, in getitem func = self._FuncPtr((name_or_ordinal, self)) AttributeError: /usr/local/cuda/lib64/libcufft.so: undefined symbol: cufftSetCompatibilityMode

We have 4 GTX 1080 GPU’s on a standalone workstation, using Cuda 9.2 (patched), Centos 7.3

Nvidia driver 390.77

Thanks

2

I’m having basically the same problem:

AttributeError: /usr/local/cuda/lib64/libcufft.so: undefined symbol: cufftSetCompatibilityMode

but on an Ubuntu 18.04 system with CUDA 9.2. I’m not sure the Nvidia driver matters much, but it’s 396.54 (I thought CUDA 9.2 was incompatible with the 390.xx drivers?).

3

The call

cufftSetCompatibilityMode

seems to have been dropped between CUDA 9.1 and 9.2.

On our system (Ubuntu 18.04, Nvidia driver 396.54), this problem could be fixed by installing CUDA 9.0 in an additional directory, which can be defined during CryoSparc installation (in our case: /usr/bin/cuda-9.0). Please note that you might need to symlink gcc5 and g++5 in the CUDA 9.0 bin folder.

Best,
Ben

Edit: