AttributeError: undefined symbol: cufftSetCompatibilityMode (V2)

Troubleshooting
1

I’ve been stuck on this error for 3 weeks, any type of help or even acknowledgement of this problem would be nice…

Here’s the output from a 2D classification job:

OverviewInputs and ParametersMetadata
Launching job on lane default target dgl-Precision-7920-Tower …

License is valid.

Running job on master node hostname dgl-Precision-7920-Tower

Project P1 Job J3 Started

Master running v2.1.0, worker running v2.1.0

Running on lane default

Resources allocated:

Worker: dgl-Precision-7920-Tower

CPU : [0, 1]

GPU : [0]

RAM : [0, 1]

SSD : True

Importing job module for job type class_2D…

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 69, in cryosparc2_compute.run.main
File “cryosparc2_compute/jobs/jobregister.py”, line 260, in get_run_function
runmod = importlib.import_module("…"+modname, name)
File “/home/dgl/cryosparc2_worker/deps/anaconda/lib/python2.7/importlib/init.py”, line 37, in import_module
import(name)
File “cryosparc2_worker/cryosparc2_compute/jobs/class2D/run.py”, line 15, in init cryosparc2_compute.jobs.class2D.run
File “cryosparc2_compute/engine/init.py”, line 8, in
from engine import *
File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 11, in init cryosparc2_compute.engine.engine
File “cryosparc2_worker/cryosparc2_compute/engine/gfourier.py”, line 6, in init cryosparc2_compute.engine.gfourier
File “/home/dgl/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py”, line 13, in
from . import cufft
File “/home/dgl/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”, line 238, in
_libcufft.cufftSetCompatibilityMode.restype = int
File “/home/dgl/cryosparc2_worker/deps/anaconda/lib/python2.7/ctypes/init.py”, line 379, in getattr
func = self.getitem(name)
File “/home/dgl/cryosparc2_worker/deps/anaconda/lib/python2.7/ctypes/init.py”, line 384, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /usr/local/cuda/lib64/libcufft.so: undefined symbol: cufftSetCompatibilityMode

2

Hi @Bruk sorry for the delay with this issue.
It appears that NVIDIA has made a breaking change in CUDA as of version 9.2

It seems the only solution currently will be to use a CUDA version 9.1 or older.
It will be easiest to re-install the cryosparc2_worker and point it to an older CUDA version.

Ali

3

Thanks for the reply. I will need detailed instructions then.

nvcc --version returns:
nvcc: NVIDIA ® Cuda compiler driver
Copyright © 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

  1. uninstalling cryosparc2_worker is same as deleting that directory?
  2. Do I need to roll back to an earlier version of CUDA, if so, how? Will this create problems with the cryosparc2_master install?
  3. How do I point the cryosparc2_worker re-install to the older CUDA version? command i use is below

cd /home/dgl/
export LICENSE_ID=“xxx”
curl -L https://get.cryosparc.com/download/worker-latest/$LICENSE_ID > cryosparc2_worker.tar.gz
tar -xf cryosparc2_worker.tar.gz
cd cryosparc2_worker
./install.sh --license $LICENSE_ID --cudapath /usr/local/cuda

4

Hi @Bruk,

I can help you out with a few things:

Yes, you can uninstall cryosparc2_worker by deleting the directory. Also, what @apunjani meant by “point it to an older CUDA version” is specify a different CUDA path in the args for the cryosparc2_worker install script.

./install.sh --license $LICENSE_ID --cudapath /usr/local/older-cuda

Also, cryosparc2_master doesn’t need to know about the CUDA path, so there will be no problems with downgrading.

5

alright, looks like things are working. thanks for your help!