cufftAllocFailed during frame motion correction



Hi there, I recently worked on cryosparc and had a problem like this, I have ~600 movies and only processed 100 of them before the job failed.

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/”, line 78, in
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/”, line 387, in
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/”, line 388, in
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/”, line 215, in
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/”, line 234, in
File “/hpcdata/lmiv_hpisv/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/”, line 115, in init
onembed, ostride, odist, self.fft_type, self.batch)
File “/hpcdata/lmiv_hpisv/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/”, line 222, in cufftPlanMany
File “/hpcdata/lmiv_hpisv/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/”, line 110, in cufftCheckStatus
raise cufftExceptions[status]

Is there anyone could provide some suggestions overcoming this problem?



@ruiruigo can you describe the movies? How many frames, which camera, what size, superresolution or not, etc?
Could there have been any other GPU processes running at the time (non-cryoSPARC)? The memory requirements for motion correction should be constant for all movies in the set of 600, so it’s strange that 100 worked but the next one failed.


Hi, thanks, problem solved. it turns out someone is using the node too.