Jobs failing after updating to 2.6.4, same error after downgrading

We were running v.2.4.2 and wanted to update to 2.4.6 for the GCTF bug fix; however after updating to 2.6.4 we are geting this error: “Traceback (most recent call last): File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main File “cryosparc2_worker/cryosparc2_compute/jobs/class2D/run.py”, line 310, in cryosparc2_compute.jobs.class2D.run.run_class_2D File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 867, in cryosparc2_compute.engine.engine.process File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 24, in cryosparc2_compute.engine.cuda_core.initialize Error: cuInit failed: unknown error”

downgrading back to 2.4.2 still gives the same error.

os: centos 7.5
cuda: 8.0

Hi @nis_it,

If you run “nvidia-smi” on your machine, what is the output?

it shows the driver version 390.87 and that all 4 of my gpus are idle.

The first time i upgraded to 2.4.6 there was this error: ““Traceback (most recent call last): File “cryosparc2_worker/cryosparc2_compute/run.py”, line 69, in cryosparc2_compute.run.main File “cryosparc2_compute/jobs/jobregister.py”, line 275, in get_run_function runmod = importlib.import_module(”…”+modname, name) File “/home/cryosparc_user/cryosparc2_worker/deps/anaconda/lib/python2.7/importlib/init.py”, line 37, in import_module import(name) File “cryosparc2_worker/cryosparc2_compute/jobs/class2D/run.py”, line 15, in init cryosparc2_compute.jobs.class2D.run File “cryosparc2_compute/engine/init.py”, line 8, in from engine import * File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 12, in init cryosparc2_compute.engine.engine File “cryosparc2_worker/cryosparc2_compute/engine/gfourier.py”, line 6, in init cryosparc2_compute.engine.gfourier File “/home/cryosparc_user/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py”, line 19, in from . import misc File “/home/cryosparc_user/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/misc.py”, line 38, in from . import cusolver File “/home/cryosparc_user/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cusolver.py”, line 290, in _libcusolver.cusolverDnCreateSyevjInfo.restype = int File “/home/cryosparc_user/cryosparc2_worker/deps/anaconda/lib/python2.7/ctypes/init.py”, line 379, in getattr func = self.getitem(name) File “/home/cryosparc_user/cryosparc2_worker/deps/anaconda/lib/python2.7/ctypes/init.py”, line 384, in getitem func = self._FuncPtr((name_or_ordinal, self)) AttributeError: /usr/local/cuda-8.0/lib64/libcusolver.so: undefined symbol: cusolverDnCreateSyevjInfo"

but now any version from 2.2.0 to 2.4.6 gives the cuInit error. should i uninstall the nvidia driver and cuda and reinstall?

I uninstalled and reinstalled the nvidia driver, still using 390.87, and now 2.4.6 is working!

I had a similar problem, but the error was different. Only by downgrading to version 2.1.0 I could start to run jobs. I am now waiting to upgrade back to 2.4.6 again.

Cheers