Blob Picker cufftAllocFailed error

Hi, I am processing some data taken with a K3 camera, and the majority of the micrographs are failing on the blob picker job with the following error:

File "cryosparc_worker/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main
File "cryosparc_master/cryosparc_compute/jobs/template_picker_gpu/run.pu", line 61, in cryosparc_compute.jobs.template_picker_gpu.run.run
File "cryosparc_master/cryosparc_compute/jobs/template_picker_gpu/run.py", line 249, in cryosparc_compute.jobs.template_picker_gpu.run.do_pick
File "cryosparc_master/cryosparc_compute/jobs/template_picker_gpu/run.py", line 376, in cryosparc_compute.jobs.template_picker_gpu.run.do_pick
File "/app/cryosparc_worker/cryosparc_compute/skcuda_internal/fft.py", line 134, in __init__ onembed, ostride, odist, self.fft_type, self.batch)
File "/app/cryosparc_worker/cryosparc_compute/skcuda_internal/cufft.py", line 749, in cufftMakePlanMany cufftCheckStatus(status)
File "/app/cryosparc_worker/cryosparc_compute/skcuda_internal/cufft.py", line 124, in cufftCheckStatus raise e

cryosparc_compute.skcuda_internal.cufft.cufftAllocFailed

I am using cryosparc v3.2.0+210817, micrographs from a K3 camera, and Geforce 2080Ti gpus. Interestingly, the blob picker job has run successfully on ~3000 of the micrographs, but has failed on all the others (~10,000). I have tried splitting these into smaller and smaller sets to find a corrupt file, even tried running on single micrographs, with no success.

Thanks!

Welcome to the forum @gnye8.

Your GPU appears to have run out of memory. This may happen due to large values of certain input parameters (what was the extraction box size?) or due to another job that was also running on the same GPU. Were there non-CryoSPARC GPU tasks running at the time the error occurred?

Hmmm the gpus had not run out of memory and there was no extraction box size assigned yet (blob picker job), but there were non-gpu tasks running at the same time - does this affect the gpu tasks? here is a picture with job parameters

Here is the info on the gpus - I have been assigning to 2 and 3