I have successfully run CryoSPARC on a few different data sets. Last few days cryoSPARC has been crashing with the following error during refinement (Ab-initio is fine):
Traceback (most recent call last):
File “/data1/progs/cryosparc/cryosparc-compute/sparc/streamlog.py”, line 321, in run_with_except_hook
File “/data1/progs/cryosparc/cryosparc-compute/engine/cuda_core.py”, line 86, in run
self.target(self.args, dev=self.dev, thidx=self.thidx)
File “/data1/progs/cryosparc/cryosparc-compute/engine/engine.py”, line 619, in work
File “/data1/progs/cryosparc/cryosparc-compute/engine/engine.py”, line 113, in load_image_data_gpu
File “/data1/progs/cryosparc/cryosparc-compute/engine/gfourier.py”, line 45, in fft2_on_gpu_inplace
File “/data1/progs/cryosparc/anaconda2/lib/python2.7/site-packages/skcuda/fft.py”, line 115, in init
onembed, ostride, odist, self.fft_type, self.batch)
File “/data1/progs/cryosparc/anaconda2/lib/python2.7/site-packages/skcuda/cufft.py”, line 222, in cufftPlanMany
File “/data1/progs/cryosparc/anaconda2/lib/python2.7/site-packages/skcuda/cufft.py”, line 110, in cufftCheckStatus
Can you give some info about box-size? And the version number of cryosparc you’re running?
Any chance another process is using the GPUs? Did you only start getting the errors after updating? Nothing has really changed in the FFT parts since 0.3.9.
Updated from version 0.3.9. Running the current version, 0.4.1. The box size is 386. I have only about 8000 particle.
Tested a refinement that I ran using the older version and it completes without any errors.
Sorry for the long delay in getting back to you.
Our testing on GTX1080 hasn’t revealed a similar issue. Can you try running the refinement that is failing, but set the refinement box size (parameter in the refinement section) to a smaller value, and see if the memory error disappears?
Can you also check via nvidia-smi that nothing else is using up RAM on the GPU?
Finally, you can force downgrade the cryosparc version back to 0.3.9 like so:
When I reduced the box size the refinement worked. However, after running couple of those jobs I am getting the hardware not registered error. Not sure whether it is related issue or not. Restarting cryosparc doesn’t resolve the issue.
@apunjani, Deactivating and activating resolves the hardware is not registered error. Everything seems to be working fine.
However, I haven’t played with different box sizes to see what may be the size limit for my data set.