I came across a memory error when I ran full-frame motion correction. I imported 437 gain-corrected tif movies (~940 M/movie) and try to run full-frame motion correction, and cryosparc kept giving me errors like this:
Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 78, in cryosparc2_compute.run.main (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/run.c:3954)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 387, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.c:13604)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.py”, line 388, in cryosparc2_compute.jobs.motioncorrection.run.run_rigid_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run.c:13462)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 216, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.c:12078)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.py”, line 241, in cryosparc2_compute.jobs.motioncorrection.motioncorrection.motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/motioncorrection.c:11497)
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 304, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:9741)
File “/local/storage/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 210, in init
self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory
Later I tried to import only 10 movies, even 1 movie, and tried to run on single or multiple GPU, neither worked, same errors. I also checked the quality of my movies, they are clear and you can see particles on them. However, I ran full-frame motion correction on tutorial dataset, T20S, it worked (~400M/movies).
My server has 8 GeForce RTX 2080 GPU (8GB memory each), and 80 cores, 384GB RAM. The cryosparc I used is v2.8. This is a new server, so I never run any real dataset on it before. Maybe there is any configuration problem? The movies used are gained already, so I didn’t give them gain reference when I imported them. Maybe that is a problem? My GPUs have 8GB memory, maybe that is not enough?
Unfortunately, it does not work. Different amount of GPUs and micrographs were tried, but it gave the similar error.
Traceback (most recent call last):
File “cryosparc2_compute/jobs/runcommon.py”, line 830, in run_with_except_hook
run_old(*args, **kw)
File “/local/storage/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “cryosparc2_compute/jobs/pipeline.py”, line 53, in stage_target
work = processor.process(item)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 142, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5032)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 145, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4927)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 146, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4834)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py”, line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:18557)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py”, line 225, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:12909)
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 304, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:9741)
File “/local/storage/software/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 210, in init
self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory
Update: our GPUs are GeForce RTX 2080 Ti 10.9 GB. When I used full-frame/patch motion correction, the memory of GPU is not enough for the job. However, if I pointed cryosparc2 to cuda 10.0 rather than 10.1, patch motion correction works.
I just changed to Cuda 10.0 and it was fixed. I cannot promise 10.0 definitely works for you, because my current workstation is using 10.2 and full-frame is working fine. Maybe try 10.1 or 10.2 to see which one works.
Also, please make sure your GPU has at least around 12G memory. You can also try patch-motion correction, or full-frame motion correction with multiple GPUs, or full-frame correction with less frames.
Thanks for your suggestions. Unfortunately my GPUs have 8GB memory only. I tried both patch motion correction and full-frame motion correction using multiples GPUs. I also reduced frames but no luck so far. I guess, I need GPU with higher memory.
I’m wondering if your nvidia-smi (driver?) version was not matching your cryosparc cuda version? In @Feng10’s case, that would mean that your driver version was 10.0 but the cryosparc2 version was originally 10.1, so which you changed to 10.0, it started to work?
That makes more sense to me than 10.1 not working for some reason.
I am hoping that I am right, because I am experiencing similar errors with a GeForce RTX 2080 Ti, but in the reverse direction.