Memory error during full-frame motion correction

closed

(Shi Feng) #1

Hi everyone,

I came across a memory error when I ran full-frame motion correction. I imported 437 gain-corrected tif movies (~940 M/movie) and try to run full-frame motion correction, and cryosparc kept giving me errors like this:

Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/run.c:3954)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 387, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.c:13604)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 388, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.c:13462)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 216, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.c:12078)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 241, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.c:11497)
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 304, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:9741)
File “/local/storage/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 210, in init
self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory

Later I tried to import only 10 movies, even 1 movie, and tried to run on single or multiple GPU, neither worked, same errors. I also checked the quality of my movies, they are clear and you can see particles on them. However, I ran full-frame motion correction on tutorial dataset, T20S, it worked (~400M/movies).

My server has 8 GeForce RTX 2080 GPU (8GB memory each), and 80 cores, 384GB RAM. The cryosparc I used is v2.8. This is a new server, so I never run any real dataset on it before. Maybe there is any configuration problem? The movies used are gained already, so I didn’t give them gain reference when I imported them. Maybe that is a problem? My GPUs have 8GB memory, maybe that is not enough?

Any suggestion would be appreciated.


(Stephan Arulthasan) #2

Hi @Feng10,

Can you try motion correcting these movies using Patch Motion (multi)?


(Shi Feng) #3

Hi @sarulthasan,

Unfortunately, it does not work. Different amount of GPUs and micrographs were tried, but it gave the similar error.

Traceback (most recent call last):
File “cryosparc2_compute/jobs/runcommon.py”, line 830, in run_with_except_hook
run_old(*args, **kw)
File “/local/storage/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “cryosparc2_compute/jobs/pipeline.py”, line 53, in stage_target
work = processor.process(item)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 142, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5032)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 145, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4927)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 146, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4834)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py”, line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:18557)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py”, line 225, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:12909)
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 304, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:9741)
File “/local/storage/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 210, in init
self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory


(Stephan Arulthasan) #4

Hi @Feng10,

It might be possible that your GPUs don’t have enough memory to process your movies. What camera was used to capture these movies?


(Shi Feng) #5

Hi @sarulthasan,

Yeah I think GPU is the problem. The camera I used is K3


(Shi Feng) #6

Update: our GPUs are GeForce RTX 2080 Ti 10.9 GB. When I used full-frame/patch motion correction, the memory of GPU is not enough for the job. However, if I pointed cryosparc2 to cuda 10.0 rather than 10.1, patch motion correction works.