Patch motion correction cuMemAlloc failed: out of memory

Troubleshooting
1

Hi alI,
Error occurred while processing:

J1/imported/009546207670462818115_14sep05c_00024sq_00003hl_00002es.frames.tif
Traceback (most recent call last):
  File "/home/cyhuang/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 60, in exec
    return self.process(item)
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 190, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 193, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 195, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 255, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 601, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 353, in cryosparc_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
  File "/home/cyhuang/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py", line 210, in __init__
    self.gpudata = self.allocator(self.size * self.dtype.itemsize)
pycuda._driver.MemoryError: cuMemAlloc failed: out of memory

Marking J1/imported/009546207670462818115_14sep05c_00024sq_00003hl_00002es.frames.tif as incomplete and continuing...

Could anyone please help me? Many thanks.

2

Welcome to the forum @hanxue

This job may have run out of VRAM. Please could you share:

  • file format of the movies
  • number of frames per file, number of pixels along horizontal and vertical frame edges
  • non-default import and motion correction parameters
  • how you ensure that no other GPU-intensive tasks may be competing for the same GPU resources when cryoSPARC jobs are running
  • your computational environment
1 Like