Particle extraction error (pycuda._driver.LogicError: cuModuleLoadDataEx failed: device kernel image is invalid - error)

Hi, I imported movies, did motion correction, “Manually Curate Exposures” and particle picking in cryosparc v4.3.0 then did “Inspect Picks”.
But when I extract particles with total ~6k micrographs and ~500k particles, there are errors (see below) and no particle is extracted when the job is done.

[CPU: 543.8 MB Avail: 498.95 GB]
Error occurred while processing micrograph J740/motioncorrected/014490630867895961341_20231226-edi2804_100_007_X+1Y+0-3_patch_aligned_doseweighted.mrc
Traceback (most recent call last):
File “/apps/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 60, in exec
return self.process(item)
File “/apps/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/extract/run.py”, line 505, in process
result = extraction_gpu.do_extract_particles_single_mic_gpu(mic=mic, bg_bin=bg_bin,
File “/apps/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/extract/extraction_gpu.py”, line 192, in do_extract_particles_single_mic_gpu
cu.do_fcrop_gpu(ET.output_f_gpu, ET.patches_f_gpu, stream = stream)
File “/apps/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/extract/cuda_kernels.py”, line 127, in do_fcrop_gpu
fcrop_gpu = get_fcrop_gpu()
File “/apps/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/extract/cuda_kernels.py”, line 116, in get_fcrop_gpu
return SM(fcrop_cu).get_function(‘fcrop’)
File “/apps/software/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/compiler.py”, line 294, in init
self.module = module_from_buffer(cubin)
pycuda._driver.LogicError: cuModuleLoadDataEx failed: device kernel image is invalid - error : Binary format for key=‘0’, ident=‘’ is not recognized

Marking J740/motioncorrected/014490630867895961341_20231226-edi2804_100_007_X+1Y+0-3_patch_aligned_doseweighted.mrc as incomplete and continuing…

@jianhaoc I am not sure what causes this error. To eliminate certain error sources, you could try, if your worker’s GPU devices are supported by v520 or newer nvidia driver:

  1. ensure the nvidia driver on the CryoSPARC worker(s) is v520 or newer.
  2. update CryoSPARC to the latest version
  3. build and run a new (as opposed to cloning the failed job) particle extraction job

Does this help?

Thank you! I updated both cryosparc and GPU driver and it works.