skcuda.cufft.cufftAllocFailed in 3D Variability (v3.0.0)

Troubleshooting
1

Hi,

Thanks for the very exiting updates to cryoSPARC v3.0.
I have just tested the new version and found a few issues with it.
Some have already been reported but this one have not I think?

After updating to v3.0, I seem to have problems running 3D variability.
It starts up fine and goes through the Initial reconstruction fine, but fails at iteration 0 with this error:

> [CPU: 5.16 GB]   Traceback (most recent call last):
>   File "cryosparc_worker/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main
>   File "cryosparc_worker/cryosparc_compute/jobs/var3D/run.py", line 526, in cryosparc_compute.jobs.var3D.run.run
>   File "cryosparc_worker/cryosparc_compute/jobs/var3D/run.py", line 312, in cryosparc_compute.jobs.var3D.run.run.E_step
>   File "cryosparc_worker/cryosparc_compute/engine/newengine.py", line 304, in cryosparc_compute.engine.newengine.EngineThread.load_models_rspace
>   File "cryosparc_worker/cryosparc_compute/engine/newgfourier.py", line 153, in cryosparc_compute.engine.newgfourier.rfft3_on_gpu_inplace
>   File "cryosparc_worker/cryosparc_compute/engine/newgfourier.py", line 72, in cryosparc_compute.engine.newgfourier.get_plan_R2C_3D
>   File "/opt/bioxray/programs/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/fft.py", line 127, in __init__
>     onembed, ostride, odist, self.fft_type, self.batch)
>   File "/opt/bioxray/programs/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/cufft.py", line 742, in cufftMakePlanMany
>     cufftCheckStatus(status)
>   File "/opt/bioxray/programs/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/skcuda/cufft.py", line 117, in cufftCheckStatus
>     raise e
> skcuda.cufft.cufftAllocFailed

I have tried recompile pycuda with CUDA version 10.0, 10.1, 10.2, 11.0 and 11.1 but non of them seems to do the trick.
This is done on a GTX 1080 GPU.

Best regards,
Jesper

2

Hey @jelka, try this

  1. Set the CUDA version to the newest one you have available (11.1)
  2. Navigate to the cryosparc_worker directory and run this command:
    bin/cryosparcw forcedeps

Then running the job again. Also can you tell me the box size of your particles and how many variability modes you’re trying to solve?

3

Thanks @nfrasser,

But “bin/cryosparcw forcedeps” enden up removing the pucuda module.

I have also tried total reinstall of the worker. This end up with the original error.

This is just the T20S tutorial dataset (empiar_10025_subset) I am trying on. Boxsize 448pix and standard settings (3 modes).
All other jobs have worked flawless.

//Jesper

4

@nfrasser your hint to box size was indeed the problem. I reduced box size a bit and the job ran through.
I guess I need a GPU with more memory…

Thanks for the help

//Jesper

3 Likes