cublasNotInitialized error (skcuda) after upgrading to V2.12.4

Troubleshooting
1

I recently upgraded my cryosparc to V2.12.4 (cuda 10.1) and when I tried to run a refinement job and I got the following error:

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 69, in cryosparc2_compute.run.main
File “cryosparc2_compute/jobs/jobregister.py”, line 329, in get_run_function
runmod = importlib.import_module("…"+modname, name)
File “/home/czhao/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/importlib/init.py”, line 37, in import_module
import(name)
File “cryosparc2_worker/cryosparc2_compute/jobs/local_refine/run.py”, line 15, in init cryosparc2_compute.jobs.local_refine.run
File “cryosparc2_compute/engine/init.py”, line 8, in
from engine import *
File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 12, in init cryosparc2_compute.engine.engine
File “cryosparc2_worker/cryosparc2_compute/engine/gfourier.py”, line 6, in init cryosparc2_compute.engine.gfourier
File “/home/czhao/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py”, line 20, in
from . import misc
File “/home/czhao/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/misc.py”, line 25, in
from . import cublas
File “/home/czhao/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cublas.py”, line 292, in
_cublas_version = int(_get_cublas_version())
File “/home/czhao/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cublas.py”, line 285, in _get_cublas_version
h = cublasCreate()
File “/home/czhao/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cublas.py”, line 203, in cublasCreate
cublasCheckStatus(status)
File “/home/czhao/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cublas.py”, line 179, in cublasCheckStatus
raise e
cublasNotInitialized

I have seen a similar post however there is no solution posted. I am just wondering what I could do to get rid of this error.

Just FYI, when I tried V12.2.0, I did not have such error. However, after upgrading to V2.12.4, even if I delete all cryosparc V12.2.4 files and reinstall the V12.2.0 using my original tar files, I still got this error. I am not 100% confident that this is the case, but I am more than 99% confident :slight_smile:

Any suggestions will be much appreciated.

2

Hi @chenzhao,

Can you post full system specs? (OS, GPUS, CUDA Toolkit Version, NVIDIA Driver Version, CPU, RAM)?
Can you also ensure nvidia-smi works?

3

Hi @stephan,

Thank you so much for your reply! After I ran nvidia-smi, I realized that I was running relion jobs on all GPUs for weeks that I even did not remember that I started those jobs… I am so stupid haha. But I would expect an out of memory error or something similar so I did not check this. But anyway the problem is solved now and I appreciate your help!