pycuda._driver.Error: cuMemGetInfo failed: unknown error from NU Refinement of v3.1.0

Hi All,

I got the following error at the bottom during NU Refinement of v3.1.0. After this error, the nvidia-smi command can’t show any GPU and had this message:
Unable to determine the device handle for GPU 0000:67:00.0: GPU is lost. Reboot the system to recover this GPU

I have GeForce RTX 2080 Ti and CUDA release 10.1, V10.1.105.
Could the Cuda version 10.1 cause the problem? Please help. Thanks.

File “cryosparc_worker/cryosparc_compute/”, line 84, in
File “cryosparc_worker/cryosparc_compute/jobs/nonuniform_refine/”, line 434, in
File “cryosparc_worker/cryosparc_compute/engine/”, line 2069, in cryosparc_compute.engine.newengine.process
File “cryosparc_worker/cryosparc_compute/engine/”, line 1886, in cryosparc_compute.engine.newengine.get_current_GPU_memory
File “cryosparc_worker/cryosparc_compute/engine/”, line 1887, in cryosparc_compute.engine.newengine.get_current_GPU_memory
pycuda._driver.Error: cuMemGetInfo failed: unknown error

Hi All,

I re-imported the same particles from a different JoinStar file of Relion and imported a different cryosparc map as initial model. Failed again with the new error:

[CPU: 26.97 GB] Traceback (most recent call last):
File “/data/donghua/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/”, line 1726, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_worker/cryosparc_compute/engine/”, line 130, in
File “cryosparc_worker/cryosparc_compute/engine/”, line 131, in
File “cryosparc_worker/cryosparc_compute/engine/”, line 1027, in
File “cryosparc_worker/cryosparc_compute/engine/”, line 106, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
File “cryosparc_worker/cryosparc_compute/engine/”, line 33, in cryosparc_compute.engine.gfourier.fft2_on_gpu_inplace
File “/data/donghua/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/”, line 127, in init
onembed, ostride, odist, self.fft_type, self.batch)
File “/data/donghua/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/”, line 742, in cufftMakePlanMany
File “/data/donghua/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/”, line 117, in cufftCheckStatus
raise e

Hi All,

Good news! I have updated my CUDA to V10.2.89 and CryoSPARC v3.1.0 seemed to be working fine after the CUDA update. Thanks.