cufftAllocFailed during frame motion correction

Troubleshooting
1

Hi there, I recently worked on cryosparc and had a problem like this, I have ~600 movies and only processed 100 of them before the job failed.

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 387, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 388, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 215, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 234, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
File “/hpcdata/lmiv_hpisv/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py”, line 115, in init
onembed, ostride, odist, self.fft_type, self.batch)
File “/hpcdata/lmiv_hpisv/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”, line 222, in cufftPlanMany
cufftCheckStatus(status)
File “/hpcdata/lmiv_hpisv/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”, line 110, in cufftCheckStatus
raise cufftExceptions[status]
cufftAllocFailed

Is there anyone could provide some suggestions overcoming this problem?

Thanks

2

@ruiruigo can you describe the movies? How many frames, which camera, what size, superresolution or not, etc?
Could there have been any other GPU processes running at the time (non-cryoSPARC)? The memory requirements for motion correction should be constant for all movies in the set of 600, so it’s strange that 100 worked but the next one failed.

3

Hi, thanks, problem solved. it turns out someone is using the node too.