CUDA error (pycuda._driver.LogicError) - Patch Motion Correction

[eMKiso]

Dear all,

this is a somehow similar issue to this old one:
https://discuss.cryosparc.com/t/cryosparc-unable-to-run-any-2d-or-3d-job/4391

@Rafael-Ayala

Dear all,

We have recently installed cryosparc v 2.14.2 on a 4 GPU node. However, we are unable to successfully run any job type other than CTF estimation, particle picking and particle extraction.

The other job types (2D and all sorts of 3D jobs) crash before even starting with the following error:

Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 110, in cryosparc2_compute.engine.cuda_core.GPUThread.run
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 111, in cryosparc2_compute.engine.cuda_core.GPUThread.run
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 991, in cryosparc2_compute.engine.engine.process.work
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 101, in cryosparc2_compute.engine.engine.EngineThread.load_image_data_gpu
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_kernels.py", line 1803, in cryosparc2_compute.engine.cuda_kernels.prepare_real
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 362, in cryosparc2_compute.engine.cuda_core.context_dependent_memoize.wrapper
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_kernels.py", line 1707, in cryosparc2_compute.engine.cuda_kernels.get_util_kernels
  File "/datalocal/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/compiler.py", line 294, in __init__
    self.module = module_from_buffer(cubin)
LogicError: cuModuleLoadDataEx failed: device kernel image is invalid - error   : Binary format for key='0', ident='' is not recognized

Any advice on how to solve the issue would be greatly appreciated!

Thanks a lot in advance!

Best wishes,

Rafa

but in our case it happens in the Patch Motion Correction jobs and only at certain pixel sizes. We can change the pixel size for 0.01 or 0.001 A and the it finishes successfully.

cryoSPARC version: 4.2.1
Cluster setup with SLURM
CUDA version: not entirely sure, I think it is 11.8 (can I check which CUDA is in use?)

The error we get looks like this:

Error occurred while processing J393/imported/003179231194746464381_FoilHole_21570445_Data_21569014_21569016_20230422_110242_Fractions.tif
Traceback (most recent call last):
  File "/d/hpc/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 60, in exec
    return self.process(item)
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 177, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 180, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 182, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 255, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 668, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/cuda_kernels.py", line 811, in cryosparc_compute.jobs.motioncorrection.cuda_kernels.do_unbend_gpu
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 414, in cryosparc_compute.engine.cuda_core.context_dependent_memoize.wrapper
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/cuda_kernels.py", line 797, in cryosparc_compute.jobs.motioncorrection.cuda_kernels.get_unbend_gpu
  File "/d/hpc/software/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/compiler.py", line 294, in __init__
    self.module = module_from_buffer(cubin)
pycuda._driver.LogicError: cuModuleLoadDataEx failed: device kernel image is invalid - error   : Binary format for key='0', ident='' is not recognized

Marking J393/imported/003179231194746464381_FoilHole_21570445_Data_21569014_21569016_20230422_110242_Fractions.tif as incomplete and continuing...

...

Movies from the same cryo-TEM microscope on a different cluster have no issues. This CUDA issue appears only on this one location of the 3 cryoSPARC locations that we have access to.

Any ideas?

Best!