I always meet a err like this ‘skcuda.cufft.cufftAllocFailed’ in many kind of jobs.It can fix when I restart my station.But I will meet this err a day late.Is there any suggestions?My GPU are 3090,always rtx 8000.Thank very much for any suggestions.


1 Like

Dear @wonderful, I’m going to direct message you instructions to update to a version of cryoSPARC that potentially fixes this issue for CentOS machines.

@spunjani, I got the same error when I was running 3D variability (v3.1.0, GeForce RTX 2080 Ti, CentOS Linux release 7.7.1908 (Core)) last night. Not sure how to fix this issue completely. Thanks.

Start iteration 0 of 20
[CPU: 6.53 GB] batch 3567 of 3567
[CPU: 7.38 GB] Done. Solving…
[CPU: 8.34 GB] diagnostic: min-ev 2.88105224609375
[CPU: 10.25 GB] diagnostic: num bad voxels 0
[CPU: 8.17 GB] batch 2529 of 3567
[CPU: 6.91 GB] Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/”, line 84, in
File “cryosparc_worker/cryosparc_compute/jobs/var3D/”, line 526, in
File “cryosparc_worker/cryosparc_compute/jobs/var3D/”, line 312, in
File “cryosparc_worker/cryosparc_compute/engine/”, line 312, in cryosparc_compute.engine.newengine.EngineThread.load_models_rspace
File “cryosparc_worker/cryosparc_compute/engine/”, line 153, in cryosparc_compute.engine.newgfourier.rfft3_on_gpu_inplace
File “cryosparc_worker/cryosparc_compute/engine/”, line 72, in cryosparc_compute.engine.newgfourier.get_plan_R2C_3D
File “/data/donghua/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/”, line 127, in init
onembed, ostride, odist, self.fft_type, self.batch)
File “/data/donghua/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/”, line 742, in cufftMakePlanMany
File “/data/donghua/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/”, line 117, in cufftCheckStatus
raise e

Hi, can you please also send me the instructions? I’m having the same issues, also CentOS.

Hi @mmiotto,

If you’re on the latest version of cryoSPARC, the fix is included.
If you encounter cuMemHostAlloc failed errors, there is an additional step you need to run:
add export CRYOSPARC_NO_PAGELOCK=true to the cryosparc_worker/ file, then re-run the job.

Thank you Stephan, actually I have cryosparc_compute.skcuda_internal.cufft.cufftAllocFailed during Extract from micrographs. I will try it to check.

Hi @stephan - we are still seeing this on our CentOS system with the latest version. Heterogeneous refinements are commonly failing with a cryosparc_compute.skcuda_internal.cufft.cufftAllocFailed error, even though when I check using nvidia_smi they don’t seem anywhere close to exceeding the capabilities of the cards (RTX-3090s). Also sometimes a hetero refine job will run to completion, and sometimes the same job will fail midway through with this error (and I have checked for other processes, so it isn’t that). Thoughts?


EDIT: here is the job log in case it is useful:

========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
**custom thread exception hook caught something
**** handle exception rc
set status to failed
========= main process now complete.
========= monitor process now complete.

Hi @olibclarke,

We are more or less at a loss on this unfortunately - are there any other patterns you can detect? Anything to do with box size, number of classes, etc?
Sorry we can’t be more helpful!