cufftAllocFailed during frame motion correction

Hi there, I recently worked on cryosparc and had a problem like this, I have ~600 movies and only processed 100 of them before the job failed.

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 387, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 388, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 215, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 234, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction
File “/hpcdata/lmiv_hpisv/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py”, line 115, in init
onembed, ostride, odist, self.fft_type, self.batch)
File “/hpcdata/lmiv_hpisv/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”, line 222, in cufftPlanMany
cufftCheckStatus(status)
File “/hpcdata/lmiv_hpisv/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py”, line 110, in cufftCheckStatus
raise cufftExceptions[status]
cufftAllocFailed

Is there anyone could provide some suggestions overcoming this problem?

Thanks

@ruiruigo can you describe the movies? How many frames, which camera, what size, superresolution or not, etc?
Could there have been any other GPU processes running at the time (non-cryoSPARC)? The memory requirements for motion correction should be constant for all movies in the set of 600, so it’s strange that 100 worked but the next one failed.

Hi, thanks, problem solved. it turns out someone is using the node too.