cudaErrorInsufficientDriver from 2D Classification (CUDA 10.0, v2.9.0)

Troubleshooting
1

Hi guys,
Cryosparc 2.9.0 throws up the following error at the beginning of 2D classification. I am using cuda 10.0 ? Is it cuda to driver incompatibility? Will changing to Cuda 8 or 9 solve the issue and is there a way to do this without having to re-install cryosparc? Any help would be greatly appreciated.
Thanks a lot,
Arka

Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 830, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 110, in cryosparc2_compute.engine.cuda_core.GPUThread.run (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:4625)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 111, in cryosparc2_compute.engine.cuda_core.GPUThread.run (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:4576)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 991, in cryosparc2_compute.engine.engine.process.work (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:27291)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 109, in cryosparc2_compute.engine.engine.EngineThread.load_image_data_gpu (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:5742)
  File "cryosparc2_worker/cryosparc2_compute/engine/gfourier.py", line 33, in cryosparc2_compute.engine.gfourier.fft2_on_gpu_inplace (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/gfourier.c:1866)
  File "/net/nfs1/public/EM/cryoSPARCv2/cryosparcV2p9p0/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py", line 101, in __init__
    capability = misc.get_compute_capability(misc.get_current_device())
  File "/net/nfs1/public/EM/cryoSPARCv2/cryosparcV2p9p0/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/misc.py", line 254, in get_current_device
    return drv.Device(cuda.cudaGetDevice())
  File "/net/nfs1/public/EM/cryoSPARCv2/cryosparcV2p9p0/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cudart.py", line 766, in cudaGetDevice
    cudaCheckStatus(status)
  File "/net/nfs1/public/EM/cryoSPARCv2/cryosparcV2p9p0/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cudart.py", line 564, in cudaCheckStatus
    raise e
cudaErrorInsufficientDriver
2

Hi @arka,

Please double check your CUDA configuration. Are you able to successfully run nvidia-smi on the machine you’re trying to run this job on? It looks like you probably need to re-install CUDA with the proper driver.

3

Hi Stephan,
Thanks a lot for your reply. This is a machine that we regularly use for relion gpu jobs and nividia-smi returns details as it should. I will try reinstalling CUDA with the proper driver.
Cheers,
Arka

4

Run into this issue before. Make sure that the path of both the worker and master point to the right cuda10 directory. Stop cryosparcm. and go to cryosparc2_worker directory and run

./install.sh --license XXXXXXX --cudapath /usr/lib/cuda-10.1

This should fix the worker path. Similarly, as per the installation instructions, do the same for the master path.

5

Hi Stephan, Stav,
I updated the driver on the machine and now it throws up a different error as stated below. I am using Cuda 10.0. Do you think moving to Cuda 10.1 could solve the issue?

Thanks!
Arka

Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 830, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 110, in cryosparc2_compute.engine.cuda_core.GPUThread.run (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:4625)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 111, in cryosparc2_compute.engine.cuda_core.GPUThread.run (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:4576)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 1053, in cryosparc2_compute.engine.engine.process.work (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:28374)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 308, in cryosparc2_compute.engine.engine.EngineThread.compute_resid_pow (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:11165)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 293, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:9489)
LogicError: cuMemHostAlloc failed: OS call failed or operation not supported on this OS
6

Hi @arka,

Its worth a try. To re-compile cryoSPARC with a new version of CUDA, use the command cryosparcw newcuda <path_to_cuda> e.g. cryosparcm newcuda /usr/local/cuda-10.1

Also, what are your system’s details (OS, GPUs, CUDA Driver version)?
And what are the parameters of your 2D Classification job (# particles, # classes)?

7

Hi Stephan,
Thanks a lot. I changed the driver to be compatible with CUDA 10.1.
NVIDIA-SMI 418.56 Driver Version: 418.56 CUDA Version: 10.1
OS is Scientific Linux, GPUS GTX 1080. For 2D classification I am using 50 classes with 440000 particles and a box size of 322 pix.
Oddly, cryosparcm does not recognize the newcuda command!
Could you please comment on that?
Cheers,
Arka

8

My apologies, please refer to my earlier post for detailed instructions: