Patch Motion (M) Error

open

#1

Hi,

After the recent update of our cryosparc2 to version 2.8.3, I have been getting a new error message in Patch motion correction which did not appear in 2.8.2.

I have attached the log from a succesful v2.8.2 job and a failed v2.8.3 job.

Here is a succesful run with v2.8.2:

-------- Cluster Job ID: 
35572

-------- Queued at 2019-06-17 01:31:16.539201

-------- Job status at 2019-06-17 01:31:16.550214
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             35572       gpu cryospar cryospar PD       0:00      1 (None)

Project P36 Job J294 Started

Master running v2.8.2, worker running v2.8.2

Running on lane SNOKES-SHARED (GPU)

Resources allocated: 

  Worker:  SNOKES-SHARED (GPU)

  CPU   :  [0, 1, 2, 3, 4, 5]

  GPU   :  [0]

  RAM   :  [0, 1]

  SSD   :  False

--------------------------------------------------------------

Importing job module for job type patch_motion_correction_multi...

Job ready to run

***************************************************************

Job will process this many movies:  7059

parent process is 57036

Calling CUDA init from 57057

-- 0.0: processing J5/imported/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226.mrc
        loading /emcc/users/an0n/cryosparc2/P36/J5/imported/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226.mrc
        Loading raw movie data from J5/imported/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226.mrc ...
        Done in 2.77s
        Loading gain data from J5/imported/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226.mrc ...
        Done in 0.28s
        Processing ...
        Done in 19.15s
        Completed rigid and patch motion with (Z:6,Y:7,X:7) knots
        Writing non-dose-weighted result to J294/motioncorrected/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226_patch_aligned.mrc ...
        Done in 0.05s
        Writing dose-weighted result to J294/motioncorrected/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226_patch_aligned_doseweighted.mrc ...
        Done in 0.05s
        Writing background estimate to J294/motioncorrected/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226_background.mrc ...
        Done in 0.01s
        Writing motion estimates...
        Done in 0.00s

-- 0.0: processing J5/imported/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227.mrc
        loading /emcc/users/an0n/cryosparc2/P36/J5/imported/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227.mrc
        Loading raw movie data from J5/imported/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227.mrc ...
        Done in 2.44s
        Loading gain data from J5/imported/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227.mrc ...
        Done in 0.00s
        Processing ...
        Done in 9.89s
        Completed rigid and patch motion with (Z:6,Y:7,X:7) knots
        Writing non-dose-weighted result to J294/motioncorrected/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227_patch_aligned.mrc ...
        Done in 0.05s
        Writing dose-weighted result to J294/motioncorrected/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227_patch_aligned_doseweighted.mrc ...
        Done in 0.05s
        Writing background estimate to J294/motioncorrected/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227_background.mrc ...
        Done in 0.01s
        Writing motion estimates...
        Done in 0.00s
...

Here is a copy of the relevant section of the log for v 2.8.3:

-------- Cluster Job ID: 
36391

-------- Queued at 2019-06-24 10:26:58.689007

-------- Job status at 2019-06-24 10:26:58.702112
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
             36391       gpu cryospar cryospar PD       0:00      1 (None)

Project P50 Job J50 Started

Master running v2.8.3, worker running v2.8.3

Running on lane SNOKES-SHARED (GPU)

Resources allocated: 

  Worker:  SNOKES-SHARED (GPU)

  CPU   :  [0, 1, 2, 3, 4, 5]

  GPU   :  [0]

  RAM   :  [0, 1]

  SSD   :  False

--------------------------------------------------------------

Importing job module for job type patch_motion_correction_multi...

Job ready to run

***************************************************************

Job will process this many movies:  2542

parent process is 54586

Calling CUDA init from 54614

-- 0.0: processing J2/imported/FoilHole_27450479_Data_27449836_27449837_20190528_0220-38570.mrc
        loading /emcc/users/an0n/cryosparc2/P50/J2/imported/FoilHole_27450479_Data_27449836_27449837_20190528_0220-38570.mrc
        Loading raw movie data from J2/imported/FoilHole_27450479_Data_27449836_27449837_20190528_0220-38570.mrc ...
        Done in 3.10s
        Loading gain data from J2/imported/FoilHole_27450479_Data_27449836_27449837_20190528_0220-38570.mrc ...
        Done in 0.25s
        Processing ...

Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 830, in run_with_except_hook
    run_old(*args, **kw)
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 53, in stage_target
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 142, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5032)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 145, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4927)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 146, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4834)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:18557)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 371, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:14676)
  File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 364, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction.do_align_patch_spline (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:7019)
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 193, in fmin_l_bfgs_b
    **opts)
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 321, in _minimize_lbfgsb
    isave, dsave, maxls)
error: failed in converting 4th argument `u' of _lbfgsb.setulb to C/Fortran array

Kind regards,
Jeppe


#3

Hi @jan,

Somehow, it seems like K_Z, K_Y, or K_X was zero when computing the patches.
What was the pixel size/dose rate? Can you try overriding the K_Z/K_X/K_Y with 5/5/5?
We’ve updated the code to ensure this doesn’t end up happening. It’ll be released in the next version of cryoSPARC.