Non-uniform refinement failed

[xtalee]

Dear cryoSPARC experts,

Would any of you be so kind and advise me how to figure the ‘out of memory’ error out in non-uniform refinement? CS version working in my quad-GPU workstation (RTX 2080Ti, 128GB ECC DDR4 RAM, 2x Intel E5-2620V4 16 cores) is 2.5.0. Dynamic mask was used.

Thank you in advance.
JH

Traceback (most recent call last): File “cryosparc2_compute/jobs/runcommon.py”, line 747, in run_with_except_hook run_old(*args, **kw) File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 101, in cryosparc2_compute.engine.cuda_core.GPUThread.run File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 102, in cryosparc2_compute.engine.cuda_core.GPUThread.run File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 987, in cryosparc2_compute.engine.engine.process.work File “cryosparc2_worker/cryosparc2_compute/engine/engine.py”, line 82, in cryosparc2_compute.engine.engine.EngineThread.load_image_data_gpu File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 242, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 110, in cryosparc2_compute.engine.cuda_core.allocate_cpu MemoryError: cuMemHostAlloc failed: out of memory

[xtalee]

Dear all,

I’m glad to share that I figured out this issue by reducing refinement box size (from 572 to 560). It’s hard to believe, but apparently 16 cores and 128GB RAM were not ‘good enough’ to tolearte my earlier setting in the non-uniform refinement. At any rate, it would be great to allocate/use multiple GPUs for 3D refinement
in CS.

Cheers,
JH

[xtalee]

Dear all,

Here is the update. Reducing the refinement box size actually didn’t work. During iteration#3 in non-uniform refinement, RAM/swap goes all the way up, and then GPU (0) allocation is suddenly removed. The refinement doesn’t move on anymore. Weird.

Cheers,
JH

[xtalee]

Upgrading GPU workstation computer with 512GB DDR4 RAM eventually figured this issue out. It‘s likely that non-uniform refinement uses more resources in each iteration (e.g., over 300GB RAM usage to refine
600 box size in my case).

John