NU refinement cufftAllocFailed error

closed

(Omid Haji-Ghassemi) #1

Dear cryosparc community,

I keep experiencing the error below during NU refinement. Would really appreciate some advice.
4 GPU RTX2080 workstation. Using Cuda 10 and Cryosparc 2.8.0. The error occured during dynamic masking.

Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 830, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 109, in cryosparc2_compute.engine.cuda_core.GPUThread.run (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:4599)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 110, in cryosparc2_compute.engine.cuda_core.GPUThread.run (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:4550)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 990, in cryosparc2_compute.engine.engine.process.work (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:27207)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 108, in cryosparc2_compute.engine.engine.EngineThread.load_image_data_gpu (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:5658)
  File "cryosparc2_worker/cryosparc2_compute/engine/gfourier.py", line 33, in cryosparc2_compute.engine.gfourier.fft2_on_gpu_inplace (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/gfourier.c:1866)
  File "/data/CRYOSPARC/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py", line 126, in __init__
    onembed, ostride, odist, self.fft_type, self.batch)
  File "/data/CRYOSPARC/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py", line 741, in cufftMakePlanMany
    cufftCheckStatus(status)
  File "/data/CRYOSPARC/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py", line 116, in cufftCheckStatus
    raise e
cufftAllocFailed

Cheers,
Omid


(Stephan Arulthasan) #2

Hi @Omid,

Thanks for reporting this. Is it possible if you can let us know what parameters you used for this job?