CUDA error (pycuda._driver.LogicError) - Patch Motion Correction

Dear all,

this is a somehow similar issue to this old one:

but in our case it happens in the Patch Motion Correction jobs and only at certain pixel sizes. We can change the pixel size for 0.01 or 0.001 A and the it finishes successfully.

cryoSPARC version: 4.2.1
Cluster setup with SLURM
CUDA version: not entirely sure, I think it is 11.8 (can I check which CUDA is in use?)

The error we get looks like this:

Error occurred while processing J393/imported/003179231194746464381_FoilHole_21570445_Data_21569014_21569016_20230422_110242_Fractions.tif
Traceback (most recent call last):
  File "/d/hpc/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/", line 60, in exec
    return self.process(item)
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/", line 177, in
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/", line 180, in
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/", line 182, in
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/", line 255, in
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/", line 668, in
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/", line 811, in
  File "cryosparc_master/cryosparc_compute/engine/", line 414, in cryosparc_compute.engine.cuda_core.context_dependent_memoize.wrapper
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/", line 797, in
  File "/d/hpc/software/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/", line 294, in __init__
    self.module = module_from_buffer(cubin)
pycuda._driver.LogicError: cuModuleLoadDataEx failed: device kernel image is invalid - error   : Binary format for key='0', ident='' is not recognized

Marking J393/imported/003179231194746464381_FoilHole_21570445_Data_21569014_21569016_20230422_110242_Fractions.tif as incomplete and continuing...


Movies from the same cryo-TEM microscope on a different cluster have no issues. This CUDA issue appears only on this one location of the 3 cryoSPARC locations that we have access to.

Any ideas?