Errors during motion correction step

chen12 · November 2, 2020, 3:35pm

After successfully imported all the movies, I was trying to run Motion Correction (full frame multi-patch) and encountered the following errors (failed). Any idea on what I could to to move forward? Really appreciated!

Launching job on lane default target localhost ...

License is valid.

Running job on master node hostname localhost

Project P7 Job J4 Started

Master running v2.9.0, worker running v2.9.0

Running on lane default

Resources allocated: 

  Worker:  localhost

  CPU   :  [0, 1, 2, 3, 4, 5]

  GPU   :  [0]

  RAM   :  [0, 1]

  SSD   :  False

--------------------------------------------------------------

Importing job module for job type patch_motion_correction_multi...

Job ready to run

***************************************************************

Job will process this many movies:  2970

parent process is 52202

Calling CUDA init from 52256

-- 0.0: processing J1/imported/new__00028.tif
        loading /data/new__00028.tif
        Loading raw movie data from J1/imported/new__00028.tif ...
        Done in 7.36s
        Loading gain data from J1/imported/new__00028.tif ...
        Done in 0.33s
        Processing ...
Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 830, in run_with_except_hook
    run_old(*args, **kw)
  File "/mnt/ssd/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 53, in stage_target
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 142, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5032)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 145, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4927)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 146, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4834)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:18557)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 371, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:14676)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 364, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction.do_align_patch_spline (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:7019)
  File "/mnt/ssd/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 193, in fmin_l_bfgs_b
    **opts)
  File "/mnt/ssd/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 321, in _minimize_lbfgsb
    isave, dsave, maxls)
error: failed in converting 4th argument `u' of _lbfgsb.setulb to C/Fortran array

spunjani · November 4, 2020, 5:42pm

Hi @chen12,
Thanks for reporting. I notice that you are running v2.9.0. I would highly recommend first updating to the current version (v2.15) and seeing if that resolves the issue. Please follow the update instructions here: https://guide.cryosparc.com/setup-configuration-and-management/software-updates

chen12 · November 4, 2020, 9:16pm

Thank you so much! Yes we just updated to version 2.15, and the error did go away. However, we are now seeing this when running the motion correction job:

[CPU: 1.19 GB] Traceback (most recent call last):
File “cryosparc2_compute/jobs/runcommon.py”, line 1685, in run_with_except_hook
run_old(*args, **kw)
File “/mnt/ssd/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “cryosparc2_compute/jobs/pipeline.py”, line 165, in thread_work
work = processor.process(item)
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 161, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py”, line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py”, line 394, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py”, line 312, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
File “/mnt/ssd/cryosparc/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 210, in init
self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory

spunjani · November 5, 2020, 7:38pm

Hi @chen12, what kind of GPUs do you have?

If your GPU has 11 GB of VRAM or more, then the next thing to check would be the settings for your import movies job - if the pixel size or electron dose is incorrect during the import movies stage, it can have a dramatic effect on the GPU memory requirements. Could you please post:

Raw pixel size (A)
Total exposure dose (e/A^2)
Frame dimensions (pixels)
Number of frames per movie.

chen12 · November 5, 2020, 7:48pm

Thanks again for your help!

Raw pixel size (A):; 0.438 A
Total exposure dose (e/A^2): 1.45
Frame dimensions (pixels): 11520x8184 (K3, super-res, bin 0.5)
Number of frames per movie: 50

chen12 · November 5, 2020, 7:52pm

the dose i entered was dose per frame (1.45).

should I enter total dose for the movie stack here?

hsnyder · November 24, 2020, 3:33pm

Hi @chen12,

Yes, it should be the total dose for the movie stack. Also, what GPU are you using? we have found that GPUs with less than 11GB of VRAM often experience this issue on K3 superres data…

Harris

chen12 · November 24, 2020, 4:51pm

Thanks Harris!

we have 8 1080Ti cards in our GPU server. Any recommendations on settings that would allow cryosparc to run more smoothly on our setup? really appreciated!

-james

hsnyder · December 2, 2020, 3:46pm

Hi @chen12,

Just to confirm - did you change the electron dose to be the total movie dose? That would be the first thing to do… Beyond that, try using the memory-use reduction switches in the patch motion correction job builder.

Harris

chen12 · December 4, 2020, 8:12pm

Thanks Harris. yes i have tried correcting for the total dose, but ended up with the same error messages. could you tell me how to access the memory-use reduction switch? I can’t seem to find it on the interface in the patch motion correction building.

chen12 · December 4, 2020, 8:21pm

okay, I’ve located an old thread about this and have tried the following without success:

Create a patch-motion job with the K3 superres movies as input
set the F-crop factor to 1/2
toggle the advanced parameters switch (top right of the job builder)
set the Max alignment resolution to 8A or 10A
set the override knots X to 5, override knots Y to 3, override knots Z to 3

chen12 · December 4, 2020, 8:24pm

also tried F-crop 1/4, same error.

hsnyder · December 11, 2020, 3:55am

Hi @chen12,

I’m sorry for the delayed response!

The memory-use reduction switches were made available in a patch to v2.15. They are located in the job builder, right under the parameter called “only process this many movies”. If you’ve upgraded to version 2.15 but didn’t apply the latest patch to that version, that would explain why you’re not seeing those options. But actually, rather than worry about the patch, I’d recommend you upgrade to v3.0, released just yesterday.

In version 3.0, patch motion uses less memory than before by default (no switch needed), and there’s an optional feature that can bring the memory use down even more, but at the cost of a little bit of performance.

Please let me know if you continue to experience this issue in version 3.0

Harris