I keep getting the same error message when I run 2D classification on ~3M particles. After about 5-10 iterations, I get the following traceback error:
Blockquote Traceback (most recent call last):
File “/usr/local/CryoSPARC/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 1726, in run_with_except_hook
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 130, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1096, in cryosparc_compute.engine.engine.process.work
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 500, in cryosparc_compute.engine.engine.EngineThread.cull_candidates
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 319, in cryosparc_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
pycuda._driver.MemoryError: cuMemHostAlloc failed: out of memory
It generally crashes once CPU memory reaches 8GB. My workstation is GPU optimized with 4x Nvidia GeForce RTX 2080. Nvidia-smi shows that I am using version 460.39 and CUDA version 11.2, although I specified --cudapath to /usr/local/cuda-10.0. It seems that the 2D classification job is using CPU memory and not GPU memory. Is there a workaround or fix to this issue?
I am running the most recent version of CryoSPARC.
Thanks in advance,