Hi,
After the recent update of our cryosparc2 to version 2.8.3, I have been getting a new error message in Patch motion correction which did not appear in 2.8.2.
I have attached the log from a succesful v2.8.2 job and a failed v2.8.3 job.
Here is a succesful run with v2.8.2:
-------- Cluster Job ID:
35572
-------- Queued at 2019-06-17 01:31:16.539201
-------- Job status at 2019-06-17 01:31:16.550214
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
35572 gpu cryospar cryospar PD 0:00 1 (None)
Project P36 Job J294 Started
Master running v2.8.2, worker running v2.8.2
Running on lane SNOKES-SHARED (GPU)
Resources allocated:
Worker: SNOKES-SHARED (GPU)
CPU : [0, 1, 2, 3, 4, 5]
GPU : [0]
RAM : [0, 1]
SSD : False
--------------------------------------------------------------
Importing job module for job type patch_motion_correction_multi...
Job ready to run
***************************************************************
Job will process this many movies: 7059
parent process is 57036
Calling CUDA init from 57057
-- 0.0: processing J5/imported/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226.mrc
loading /emcc/users/an0n/cryosparc2/P36/J5/imported/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226.mrc
Loading raw movie data from J5/imported/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226.mrc ...
Done in 2.77s
Loading gain data from J5/imported/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226.mrc ...
Done in 0.28s
Processing ...
Done in 19.15s
Completed rigid and patch motion with (Z:6,Y:7,X:7) knots
Writing non-dose-weighted result to J294/motioncorrected/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226_patch_aligned.mrc ...
Done in 0.05s
Writing dose-weighted result to J294/motioncorrected/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226_patch_aligned_doseweighted.mrc ...
Done in 0.05s
Writing background estimate to J294/motioncorrected/FoilHole_7402854_Data_7403063_7403064_20190208_1637-12226_background.mrc ...
Done in 0.01s
Writing motion estimates...
Done in 0.00s
-- 0.0: processing J5/imported/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227.mrc
loading /emcc/users/an0n/cryosparc2/P36/J5/imported/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227.mrc
Loading raw movie data from J5/imported/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227.mrc ...
Done in 2.44s
Loading gain data from J5/imported/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227.mrc ...
Done in 0.00s
Processing ...
Done in 9.89s
Completed rigid and patch motion with (Z:6,Y:7,X:7) knots
Writing non-dose-weighted result to J294/motioncorrected/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227_patch_aligned.mrc ...
Done in 0.05s
Writing dose-weighted result to J294/motioncorrected/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227_patch_aligned_doseweighted.mrc ...
Done in 0.05s
Writing background estimate to J294/motioncorrected/FoilHole_7402854_Data_7403131_7403132_20190208_1638-12227_background.mrc ...
Done in 0.01s
Writing motion estimates...
Done in 0.00s
...
Here is a copy of the relevant section of the log for v 2.8.3:
-------- Cluster Job ID:
36391
-------- Queued at 2019-06-24 10:26:58.689007
-------- Job status at 2019-06-24 10:26:58.702112
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
36391 gpu cryospar cryospar PD 0:00 1 (None)
Project P50 Job J50 Started
Master running v2.8.3, worker running v2.8.3
Running on lane SNOKES-SHARED (GPU)
Resources allocated:
Worker: SNOKES-SHARED (GPU)
CPU : [0, 1, 2, 3, 4, 5]
GPU : [0]
RAM : [0, 1]
SSD : False
--------------------------------------------------------------
Importing job module for job type patch_motion_correction_multi...
Job ready to run
***************************************************************
Job will process this many movies: 2542
parent process is 54586
Calling CUDA init from 54614
-- 0.0: processing J2/imported/FoilHole_27450479_Data_27449836_27449837_20190528_0220-38570.mrc
loading /emcc/users/an0n/cryosparc2/P50/J2/imported/FoilHole_27450479_Data_27449836_27449837_20190528_0220-38570.mrc
Loading raw movie data from J2/imported/FoilHole_27450479_Data_27449836_27449837_20190528_0220-38570.mrc ...
Done in 3.10s
Loading gain data from J2/imported/FoilHole_27450479_Data_27449836_27449837_20190528_0220-38570.mrc ...
Done in 0.25s
Processing ...
Traceback (most recent call last):
File "cryosparc2_compute/jobs/runcommon.py", line 830, in run_with_except_hook
run_old(*args, **kw)
File "/opt/bioxray/programs/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "cryosparc2_compute/jobs/pipeline.py", line 53, in stage_target
work = processor.process(item)
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 142, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:5032)
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 145, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4927)
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.py", line 146, in cryosparc2_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/run_patch.c:4834)
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 77, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:18557)
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 371, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:14676)
File "cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.py", line 364, in cryosparc2_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction.do_align_patch_spline (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/motioncorrection/patchmotion.c:7019)
File "/opt/bioxray/programs/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 193, in fmin_l_bfgs_b
**opts)
File "/opt/bioxray/programs/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/site-packages/scipy/optimize/lbfgsb.py", line 321, in _minimize_lbfgsb
isave, dsave, maxls)
error: failed in converting 4th argument `u' of _lbfgsb.setulb to C/Fortran array
Kind regards,
Jeppe