Cryosparc v2.15 patch motion correction fails with out of memory message

Hi,

I am trying to run patch motion correction to K3 images. I am keeping getting the error below. I tried applying the current patch for v2.15 and running it with the option “reduce GPU memory usage” but I still get the same error.

When increasing the maximum alignment resolution to values greater than 6, the error disappears but the run time gets very slow and the smoothed curves seem to fit the trajectories poorly.

[CPU: 846.3 MB] Traceback (most recent call last): File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook run_old(*args, **kw) File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "cryosparc2_compute/jobs/pipeline.py", line 165, in thread_work work = processor.process(item) File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5332) File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5228) File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 164, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5137) File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 641, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction_nofix (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:35926) File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 790, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction_nofix (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:29885) File "cryosparc2_master/cryosparc2_compute/engine/cuda_core.py", line 330, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/engine/cuda_core.c:10407) File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py", line 210, in __init__ self.gpudata = self.allocator(self.size * self.dtype.itemsize) MemoryError: cuMemAlloc failed: out of memory

[CPU: 166.7 MB] Outputting partial results now...

[CPU: 166.7 MB] Traceback (most recent call last): File "cryosparc2_worker/cryosparc2_compute/run.py", line 85, in cryosparc2_compute.run.main File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 362, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.c:14887) AssertionError: Child process with PID 47485 has terminated unexpectedly!

Hi @lalmagor,

I can look into this for you. Could you please let us know the following:

  • What model GPU are you using?
  • How many frames per movie
  • Total electron dose

Thanks

– Harris

Hi Harris,

Thanks for your quick response.

The GPU hardware is: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1). We have four of those.

There are 80 frames per movie.

Total exposure dose is 65 e/A^2.

Best,
Lior

Hi Lior,

Interesting, the 1080 Ti should definitely have enough memory… Could you also provide the following parameters from the corresponding import movies job, as well as any parameters in the patch motion job that are not on their default settings?

  • Raw pixel size
  • Accelerating voltage
  • Spherical aberration

Thanks,
Harris

Raw pixel size = 1.111 A
Voltage = 300 kV
Spherical abberation = 2.7
All default parameters

Any Idea about this?

How much physical memory do you have? And what size of the movies? I have met similar problems on K3 movies. Using F-crop to bin movies will help.