Patch motion correction fails after 5 movies

[nikydna]

HI
I am running patch motion correction (multi) using cryosparc version 2.15.0, cuda version CUDA Version 10.1.243.

I am getting the following error

Launching job on lane default target rrserver ...
Running job on master node hostname rrserver
[CPU: 90.4 MB]   Project P1 Job J9 Started
[CPU: 90.5 MB]   Master running v2.15.0, worker running v2.15.0
[CPU: 90.7 MB]   Running on lane default
[CPU: 90.7 MB]   Resources allocated: 
[CPU: 90.7 MB]     Worker:  rrserver
[CPU: 90.7 MB]     CPU   :  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35]
[CPU: 90.7 MB]     GPU   :  [0, 1, 2, 3, 4, 5]
[CPU: 90.7 MB]     RAM   :  [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
[CPU: 90.8 MB]     SSD   :  False
[CPU: 90.8 MB]   --------------------------------------------------------------
[CPU: 90.8 MB]   Importing job module for job type patch_motion_correction_multi...
[CPU: 165.1 MB]  Job ready to run
[CPU: 165.2 MB]  ***************************************************************
[CPU: 165.5 MB]  Job will process this many movies:  10
[CPU: 165.5 MB]  parent process is 5594
[CPU: 133.3 MB]  Calling CUDA init from 5627
[CPU: 133.3 MB]  Calling CUDA init from 5629
[CPU: 133.3 MB]  Calling CUDA init from 5630
[CPU: 133.3 MB]  Calling CUDA init from 5628
[CPU: 133.3 MB]  Calling CUDA init from 5632
[CPU: 133.3 MB]  Calling CUDA init from 5631
[CPU: 261.6 MB]  -- 5.0: processing 0 of 10: J4/imported/FoilHole_8848755_Data_8822150_8822152_20200610_020109_fractions.tif
        loading /run/media/nxb274/83de12da-670b-49ac-adfa-dba429684ef0/06092020/cryosparc/P1/J4/imported/FoilHole_8848755_Data_8822150_8822152_20200610_020109_fractions.tif
        Loading raw movie data from J4/imported/FoilHole_8848755_Data_8822150_8822152_20200610_020109_fractions.tif ...
        Done in 6.15s
        Loading gain data from J4/imported/gain.mrc ...
        Done in 0.28s
        Processing ...
[CPU: 261.6 MB]  -- 3.0: processing 1 of 10: J4/imported/FoilHole_8848755_Data_8822164_8822166_20200610_020123_fractions.tif
        loading /run/media/nxb274/83de12da-670b-49ac-adfa-dba429684ef0/06092020/cryosparc/P1/J4/imported/FoilHole_8848755_Data_8822164_8822166_20200610_020123_fractions.tif
        Loading raw movie data from J4/imported/FoilHole_8848755_Data_8822164_8822166_20200610_020123_fractions.tif ...
        Done in 7.49s
        Loading gain data from J4/imported/gain.mrc ...
        Done in 0.23s
        Processing ...
[CPU: 261.6 MB]  -- 2.0: processing 2 of 10: J4/imported/FoilHole_8848756_Data_8822150_8822152_20200610_020206_fractions.tif
        loading /run/media/nxb274/83de12da-670b-49ac-adfa-dba429684ef0/06092020/cryosparc/P1/J4/imported/FoilHole_8848756_Data_8822150_8822152_20200610_020206_fractions.tif
        Loading raw movie data from J4/imported/FoilHole_8848756_Data_8822150_8822152_20200610_020206_fractions.tif ...
        Done in 6.27s
        Loading gain data from J4/imported/gain.mrc ...
        Done in 0.31s
        Processing ...
[CPU: 261.6 MB]  -- 0.0: processing 3 of 10: J4/imported/FoilHole_8848756_Data_8822164_8822166_20200610_020220_fractions.tif
        loading /run/media/nxb274/83de12da-670b-49ac-adfa-dba429684ef0/06092020/cryosparc/P1/J4/imported/FoilHole_8848756_Data_8822164_8822166_20200610_020220_fractions.tif
        Loading raw movie data from J4/imported/FoilHole_8848756_Data_8822164_8822166_20200610_020220_fractions.tif ...
        Done in 6.29s
        Loading gain data from J4/imported/gain.mrc ...
        Done in 0.30s
        Processing ...

[CPU: 261.6 MB]  -- 1.0: processing 4 of 10: J4/imported/FoilHole_8848811_Data_8822150_8822152_20200610_020302_fractions.tif
        loading /run/media/nxb274/83de12da-670b-49ac-adfa-dba429684ef0/06092020/cryosparc/P1/J4/imported/FoilHole_8848811_Data_8822150_8822152_20200610_020302_fractions.tif
        Loading raw movie data from J4/imported/FoilHole_8848811_Data_8822150_8822152_20200610_020302_fractions.tif ...
        Done in 5.78s
        Loading gain data from J4/imported/gain.mrc ...
        Done in 0.39s
        Processing ...
[CPU: 261.6 MB]  -- 4.0: processing 5 of 10: J4/imported/FoilHole_8848811_Data_8822164_8822166_20200610_020316_fractions.tif
        loading /run/media/nxb274/83de12da-670b-49ac-adfa-dba429684ef0/06092020/cryosparc/P1/J4/imported/FoilHole_8848811_Data_8822164_8822166_20200610_020316_fractions.tif
        Loading raw movie data from J4/imported/FoilHole_8848811_Data_8822164_8822166_20200610_020316_fractions.tif ...
        Done in 6.22s
        Loading gain data from J4/imported/gain.mrc ...
        Done in 0.30s
        Processing ...
[CPU: 950.7 MB]  Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 165, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 161, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 396, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 312, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py", line 210, in __init__
    self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory
[CPU: 950.5 MB]  Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 165, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 161, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 396, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 312, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py", line 210, in __init__
    self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory
[CPU: 950.4 MB]  Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 165, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 161, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 396, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 312, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py", line 210, in __init__
    self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory
[CPU: 950.4 MB]  Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 165, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 161, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 396, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 312, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py", line 210, in __init__
    self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory
[CPU: 950.5 MB]  Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 165, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 161, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 394, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 312, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py", line 210, in __init__
    self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory
[CPU: 950.5 MB]  Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 165, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 161, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 396, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 312, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
  File "/home/nxb274/applications/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py", line 210, in __init__
    self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory

full-frame motion correction runs without error.

Let me know if you need more information.

Don’t know how to fix this.

Thanks,
Nikhil

[nfrasser]

Hi Nikhil, this error indicates that the GPU ran out of memory while running the job. This is a known issue with Patch Motion which we’ve addressed in a patch. You can install the patch with my instructions here:

AssertionError in non-uniform refinement

Hi @Nitesh, thanks for reporting this, we’ve identified the issue and have a patch available. You can get it in cryoSPARC v2.15 with cryosparcm patch: https://guide.cryosparc.com/setup-configuration-and-management/software-updates#apply-patches After you do that, please also refresh cryoSPARC so that it registers the patch files cryosparcm cli "reload()" cryosparcm cli "refresh_job_types()" Let me know if you run into any trouble with that. Nick

Then create a new Patch Motion job and enable the new “Reduce GPU memory usage” option in the Job Builder.

Let me know if that works.

Nick

[nikydna]

Thank you so much for quick response.

Patch solved the problem.

Nikhil

[jamon]

Dear NiKil, I installed 2.15, but did not find “Reduce GPU memory usage” option in a new Patch Motion job. Where should I be looking? Thanks,

[jamon]

I am sorry. It showed up only after applying the patch. It is working well now. Thanks a lot.