2D Classification Failure: `cufftInternalError`

2D Classification
1

Hi I am running 2D classification with ~2.8 M particles of 562 pixel box size and I am keep getting this error after 19th iteration and start of 20th iteration. I am wondering about the cause of this error, and how to resolve it. Any help is appreciated.

[CPU: 18.32 GB]  Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1844, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
  File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
  File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 1028, in cryosparc_compute.engine.engine.process.work
  File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 107, in cryosparc_compute.engine.engine.EngineThread.load_image_data_gpu
  File "cryosparc_worker/cryosparc_compute/engine/gfourier.py", line 32, in cryosparc_compute.engine.gfourier.fft2_on_gpu_inplace
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/fft.py", line 134, in __init__
    onembed, ostride, odist, self.fft_type, self.batch)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cufft.py", line 749, in cufftMakePlanMany
    cufftCheckStatus(status)
  File "/home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cufft.py", line 124, in cufftCheckStatus
    raise e
cryosparc_compute.skcuda_internal.cufft.cufftInternalError

Anamika

2

If it’s happening at the start of the full-dataset iteration, it’s likely you’re running out of memory.

You can try running it with “2D zeropad factor” set to 1, rather than the default of 2, or (higher?) binned particles, or splitting the full set into, say, four or five and running them separately, then selected good classes, recombining and doing a final 2D to clean up a little more.