When I turned off "Force Max over poses/shifts", 2D classification expressed error

Troubleshooting
1

Hi Everyone,

I tried to turn off “Force Max over poses/shifts” option.
Cryosparc showed this error and stopped.

I was processing 1.7 millions particles with 450 box size.
Is that to big to apply that option?

Anyone can helps?

  File "cryosparc2_compute/jobs/runcommon.py", line 1678, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc2_master/cryosparc2_compute/engine/cuda_core.py", line 128, in cryosparc2_compute.engine.cuda_core.GPUThread.run
  File "cryosparc2_master/cryosparc2_compute/engine/cuda_core.py", line 129, in cryosparc2_compute.engine.cuda_core.GPUThread.run
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 991, in cryosparc2_compute.engine.engine.process.work
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 109, in cryosparc2_compute.engine.engine.EngineThread.load_image_data_gpu
  File "cryosparc2_worker/cryosparc2_compute/engine/gfourier.py", line 33, in cryosparc2_compute.engine.gfourier.fft2_on_gpu_inplace
  File "/home/exx/MySoftware/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py", line 127, in __init__
    onembed, ostride, odist, self.fft_type, self.batch)
  File "/home/exx/MySoftware/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py", line 742, in cufftMakePlanMany
    cufftCheckStatus(status)
  File "/home/exx/MySoftware/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py", line 117, in cufftCheckStatus
    raise e
cufftAllocFailed

  File "cryosparc2_compute/jobs/runcommon.py", line 1678, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc2_master/cryosparc2_compute/engine/cuda_core.py", line 128, in cryosparc2_compute.engine.cuda_core.GPUThread.run
  File "cryosparc2_master/cryosparc2_compute/engine/cuda_core.py", line 129, in cryosparc2_compute.engine.cuda_core.GPUThread.run
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 991, in cryosparc2_compute.engine.engine.process.work
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 109, in cryosparc2_compute.engine.engine.EngineThread.load_image_data_gpu
  File "cryosparc2_worker/cryosparc2_compute/engine/gfourier.py", line 33, in cryosparc2_compute.engine.gfourier.fft2_on_gpu_inplace
  File "/home/exx/MySoftware/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/fft.py", line 127, in __init__
    onembed, ostride, odist, self.fft_type, self.batch)
  File "/home/exx/MySoftware/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py", line 742, in cufftMakePlanMany
    cufftCheckStatus(status)
  File "/home/exx/MySoftware/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/skcuda/cufft.py", line 117, in cufftCheckStatus
    raise e
cufftAllocFailed

Jinseo

2

sorry I don’t know the answer. but you can use “particle sets tool” to very quickly split the particle file into smaller ones and see if the job will run. Maybe try 100,000 particles as a quick test. It will also be very easy to join them back together after selecting the appropriate 2D classes.

3

Hi @Jinseo,

It looks like you might not have enough GPU memory available to support that combination of inputs and parameters. I just have a few questions:

  • What is the number of classes used?
  • Does the job work when Force Max over poses/shifts option is true?

Disabling this option should only consume slightly more GPU memory. If you want to run the job with Force Max over poses/shifts off, you may consider downsampling the input particles. This can be done using a “Downsample Particles” job, and by manually specifying a smaller box size by setting the Fourier crop to box size (pix) parameter. You may also consider the suggestion from @CryoEM1. The Particle Sets Tool can split your particle stack into “batches” by setting the Split batch size parameter, and running separate 2D classifications as suggested may also help reduce GPU memory usage.

Best,
Michael

4

Hi Michael,

Thanks for suggestion,

Now I am working at 150 classes with 1.9m particles.
Force Max over poses/shifts on worked with that particles.
After several rounds of 2D classifications, my particles is 0.7m particles. But “off” option didn’t worked. So as you suggested, I am now doing “Downsample”.
I will update further progress when “Downsample” finished.
Thanks a lot.

Jinseo