CUDA memory error during 2D classification

Hi all,
I keep getting the same error message when I run 2D classification on ~3M particles. After about 5-10 iterations, I get the following traceback error:

Blockquote Traceback (most recent call last):
File “/usr/local/CryoSPARC/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 1726, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 130, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 1096, in cryosparc_compute.engine.engine.process.work
File “cryosparc_worker/cryosparc_compute/engine/engine.py”, line 500, in cryosparc_compute.engine.engine.EngineThread.cull_candidates
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 319, in cryosparc_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
pycuda._driver.MemoryError: cuMemHostAlloc failed: out of memory

It generally crashes once CPU memory reaches 8GB. My workstation is GPU optimized with 4x Nvidia GeForce RTX 2080. Nvidia-smi shows that I am using version 460.39 and CUDA version 11.2, although I specified --cudapath to /usr/local/cuda-10.0. It seems that the 2D classification job is using CPU memory and not GPU memory. Is there a workaround or fix to this issue?

I am running the most recent version of CryoSPARC.

Thanks in advance,
Karl

1 Like

@kherbine, thanks for posting. Which OS are you running?

1 Like

@spunjani Hi, I am running CentOS Linux release 7.9.2009 (Core).

Hi @kherbine,

Thanks for reporting. I’m going to direct message you instructions to update to a version of cryoSPARC that potentially fixes this issue for CentOS machines.

1 Like

Hi, I have the same issue when running 2D classification and ab-initio reconstruction on CentOS. I have the latest version of Cyrosparc. Could you please help me with that?

@Cwuz, are you currently running v3.2?

Yes. Current cryoSPARC version: v3.2.0

Thanks @Cwuz. CryoSPARC v3.2 contains an option to work around the bug in CUDA on CentOS 7 that causes cuMemHostAlloc failed errors in multiple job types. To engage this, please add export CRYOSPARC_NO_PAGELOCK=true to the cryosparc_worker/config.sh file.