Cryosparc v2.15 patch motion correction fails with out of memory message

Hi,

I am trying to run patch motion correction to K3 images. I am keeping getting the error below. I tried applying the current patch for v2.15 and running it with the option “reduce GPU memory usage” but I still get the same error.

When increasing the maximum alignment resolution to values greater than 6, the error disappears but the run time gets very slow and the smoothed curves seem to fit the trajectories poorly.

[CPU: 846.3 MB] Traceback (most recent call last): File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook run_old(*args, **kw) File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "cryosparc2_compute/jobs/pipeline.py", line 165, in thread_work work = processor.process(item) File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5332) File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5228) File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 164, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5137) File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 641, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction_nofix (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:35926) File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 790, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction_nofix (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:29885) File "cryosparc2_master/cryosparc2_compute/engine/cuda_core.py", line 330, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/engine/cuda_core.c:10407) File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py", line 210, in __init__ self.gpudata = self.allocator(self.size * self.dtype.itemsize) MemoryError: cuMemAlloc failed: out of memory

[CPU: 166.7 MB] Outputting partial results now...

[CPU: 166.7 MB] Traceback (most recent call last): File "cryosparc2_worker/cryosparc2_compute/run.py", line 85, in cryosparc2_compute.run.main File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 362, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi (/fast5/userhome/nfrasser/cryosparc2/cryosparc2_package/cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.c:14887) AssertionError: Child process with PID 47485 has terminated unexpectedly!

Hi @lalmagor,

I can look into this for you. Could you please let us know the following:

  • What model GPU are you using?
  • How many frames per movie
  • Total electron dose

Thanks

– Harris

Hi Harris,

Thanks for your quick response.

The GPU hardware is: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1). We have four of those.

There are 80 frames per movie.

Total exposure dose is 65 e/A^2.

Best,
Lior

Hi Lior,

Interesting, the 1080 Ti should definitely have enough memory… Could you also provide the following parameters from the corresponding import movies job, as well as any parameters in the patch motion job that are not on their default settings?

  • Raw pixel size
  • Accelerating voltage
  • Spherical aberration

Thanks,
Harris

Raw pixel size = 1.111 A
Voltage = 300 kV
Spherical abberation = 2.7
All default parameters

Any Idea about this?

How much physical memory do you have? And what size of the movies? I have met similar problems on K3 movies. Using F-crop to bin movies will help.

We have 192GB of physical memory (please see the attached configuration).

The files are around 850GB each. In my hands F-corp did not help to avoid this error. When I increase the resolution values to > 6A the error message disappear but the output plots for the motion correction look suspicious, with jagged lines that are not well smoothed.

1 Like

Hi Lior,

Sorry for taking a while to get back to you on this. I ran some K3 super resolution movies with 80 frames and your settings and I can confirm that the job will not fit into 11 GB of GPU memory. Unfortunately, the output f-crop factor won’t help in this case because of where in the processing the failure occurs. However, we’re working on an optional feature which will cause motion correction to use less memory but go slower - a good compromise in cases like this. The feature will most likely be available in the next release of CryoSPARC. I’ll post again when it has been released.

Harris

Hi Harris,

Thanks for your reply. I am glad that this is going to be resolved in the next release. Do you have an idea when is a new release coming?

In the meantime, to avoid this error, I am processing using only a subset of the frames in each movie. I successfully use 62 frames to get a signal of above 50 e/A2. One thing is that when looking at the motion correction results, the plots show movement with no particular tendentiousness (see picture). I wonder if this is a genuine problem, or am I just confused by the plots’ small scale?

Another suggestion was to merge every two subsequent frames in each movie, generating 40 rather than 80 frames movies, before running motion correction. Is there any tool in Cryosparc that can be used for that?

Best,
Lior

Hi Lior,

We’re going to release the change that I spoke of as a patch. I don’t have an exact timeline for the release at the moment, but I will let you know when it’s available for download. It should be fairly soon.

Regarding your issue you described with patch motion correction, those plots do look a bit odd. I could look into it for you - would you be willing to send us (confidentially) a sample of the data? We would need the movie that you used to generate these plots, as well as the un-modified (80 frame) corresponding movie. Let me know if you’d be willing to do so and I’ll get in touch with transfer instructions.

Harris

Hi Harris,

We might have found the source for the weird motion correction results. The pixel size value we were using was twice larger then the actual one (because the pixel size in raw images is half of the physical (“real") pixel size). Reducing the pixel size value to 0.5555 enables us to run motion correction with the complete 80 frame movie resulting with plots that make sense. CTF estimation that beforehand gave also nonsense results now works perfectly.

I am sorry for occupying you with this while it is all about a wrong parameter assignment problem. This detector is newly installed and hence there is not a large experience around us in processing its data.

Thank you,
Lior

Hi Lior,

No problem. I’m glad you got it sorted out. Ironically, the patch is now available, and can be downloaded as per the instructions found here: https://guide.cryosparc.com/setup-configuration-and-management/software-updates#apply-patches

Regards,
Harris