Patch motion correction failed

Motion Correction
1

I have ran the T20S project successfully with my current installation. I have failed many times when ran my own data with jon ‘patch motion correction’. I thought it was memory issue and installed the patches as suggested in other thread. But it still failed. Could anyone please help me? Many thanks.

Log file:

Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
**** handle exception rc
set status to failed
Traceback (most recent call last):
  File "cryosparc2_worker/cryosparc2_compute/run.py", line 85, in cryosparc2_compute.run.main
  File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 363, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi
AssertionError: Child process with PID 5005 has terminated unexpectedly!
========= main process now complete.
========= monitor process now complete.




JOB OVERVIEW RED ERROR:

[CPU: 867.6 MB]  Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 165, in thread_work
    work = processor.process(item)
  File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 164, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 84, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 243, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc2_master/cryosparc2_compute/engine/cuda_core.py", line 330, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
  File "/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/pycuda/gpuarray.py", line 210, in __init__
    self.gpudata = self.allocator(self.size * self.dtype.itemsize)
MemoryError: cuMemAlloc failed: out of memory


[CPU: 173.3 MB]  Outputting partial results now...

[CPU: 172.7 MB]  Traceback (most recent call last):
  File "cryosparc2_worker/cryosparc2_compute/run.py", line 85, in cryosparc2_compute.run.main
  File "cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 363, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi
AssertionError: Child process with PID 5005 has terminated unexpectedly!
2

Hi @yy314,

I’m sorry for taking so long to address this. This is definitely a memory issue. What model of GPU are you using?

If your GPU has 11 GB of VRAM or more, then the next thing to check would be the settings for your import movies job - if the pixel size or electron dose is incorrect during the import movies stage, it can have a dramatic effect on the GPU memory requirements. Could you please post:

  • Raw pixel size (A)
  • Total exposure dose (e/A^2)
  • Frame dimensions (pixels)
  • Number of frames per movie.

Thanks for your patience,
Harris

1 Like
3

Thanks for your reply.

We run CS on clusters. Here is the set up of our clusters. We have 8GB GPU memory on each of 12-16 nodes.

Compute nodes triton-[1-5] are fitted with 2x Nvidia Geforce 1060Ti cards offering 768 CUDA cores and 4GB of GPU memory each.
Compute nodes triton-[6-10] are fitted with 2x Nvidia Quadro P1000 cards offering 640 CUDA cores and 4GB of GPU memory each.
Compute nodes triton-[12-16] are fitted with 1x Nvidia Tesla P4 cards offering 2560 CUDA cores and 8GB of GPU memory.

  • Raw pixel size (A) 0.83 (super-resolution 0.415)
  • Total exposure dose (e/A^2) 48.47
  • Frame dimensions (pixels): 5760 x 4092 (11520 x 8184 in super-resolution mode)
  • Number of frames per movie:47

Kind regards,
Yahui

4

what’s the pixel size you use in the import job ? If it’s K3 super resolution you should use 0.415

5

Thanks!
Here is part of the oversiew showing the inout parameters.

[CPU: 173.7 MB] ===========================================================

[CPU: 173.7 MB] Loaded 2613 movies.

[CPU: 173.7 MB] Common fields:

[CPU: 173.7 MB] mscope_params/accel_kv : set([300.0])

[CPU: 173.7 MB] mscope_params/cs_mm : set([2.7])

[CPU: 173.7 MB] mscope_params/total_dose_e_per_A2 : set([48.470001])

[CPU: 173.7 MB] mscope_params/exp_group_id : set([2])

[CPU: 173.7 MB] mscope_params/phase_plate : set([0])

[CPU: 173.7 MB] mscope_params/neg_stain : set([0])

[CPU: 173.7 MB] movie_blob/psize_A : set([0.41499999])

[CPU: 173.7 MB] movie_blob/shape : [ 47 8184 11520]

[CPU: 173.8 MB] movie_blob/is_gain_corrected : set([0])

[CPU: 173.8 MB] ===========================================================

[CPU: 173.8 MB] Making example plots…

6

if it’s really a problem of memory, if you specify you want to run the job only on 1 movie, does it work ?

7

I set it up to preocess 1 movie. Still got the same error in the Overview window.

[CPU: 90.1 MB] Project P2 Job J45 Started

[CPU: 90.1 MB] Master running v2.15.0+200710, worker running v2.15.0+200710

[CPU: 90.4 MB] Running on lane tesla

[CPU: 90.4 MB] Resources allocated:

[CPU: 90.4 MB] Worker: tesla

[CPU: 90.4 MB] CPU : [0, 1, 2, 3, 4, 5]

[CPU: 90.4 MB] GPU : [0]

[CPU: 90.4 MB] RAM : [0, 1]

[CPU: 90.4 MB] SSD : False

[CPU: 90.4 MB] --------------------------------------------------------------

[CPU: 90.4 MB] Importing job module for job type patch_motion_correction_multi…

[CPU: 166.1 MB] Job ready to run

[CPU: 166.1 MB] ***************************************************************

[CPU: 166.9 MB] Job will process this many movies: 1

[CPU: 166.9 MB] parent process is 12740

[CPU: 134.9 MB] Calling CUDA init from 12761

[CPU: 287.3 MB] – 0.0: processing 0 of 1: J42/imported/FoilHole_4672037_Data_4673551_4673553_20200605_220605_fractions.tiff
loading /mnt/argo/homes/yy314/20200605_GAPif2/P5/J42/imported/FoilHole_4672037_Data_4673551_4673553_20200605_220605_fractions.tiff
Loading raw movie data from J42/imported/FoilHole_4672037_Data_4673551_4673553_20200605_220605_fractions.tiff …
Done in 5.53s
Loading gain data from J42/imported/20200603-K3-18210048GainRef.x1.m1.mrc …
Done in 3.33s
Processing …

[CPU: 886.0 MB] Traceback (most recent call last):
File “cryosparc2_compute/jobs/runcommon.py”, line 1685, in run_with_except_hook
run_old(*args, **kw)
File “/usr/local/em/cryosparc/yy314/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py”, line 754, in run
self.__target(*self.__args, **self.__kwargs)
File “cryosparc2_compute/jobs/pipeline.py”, line 165, in thread_work
work = processor.process(item)
File “cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 157, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 160, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 164, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.py”, line 84, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/patchmotion.py”, line 535, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc2_master/cryosparc2_compute/engine/cuda_core.py”, line 215, in cryosparc2_compute.engine.cuda_core.transfer_ndarray_to_cudaarray
MemoryError: cuArrayCreate failed: out of memory

[CPU: 167.1 MB] Outputting partial results now…

[CPU: 167.1 MB] Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 85, in cryosparc2_compute.run.main
File “cryosparc2_master/cryosparc2_compute/jobs/motioncorrection/run_patch.py”, line 363, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi
AssertionError: Child process with PID 12761 has terminated unexpectedly!

8

It may not be possible to process super-resolution movies with 4 and 6 GB cards. Try running your jobs on the 8 GB cards only, and make sure that both “reduce memory usage” switches (from the recent patches) are active.

Harris