Reference motion correction crashes after first iteration

[mathewmclaren]

Hi all,

I’m trying to run the new polishing job on an existing dataset already processed using cryoSPARC and I’m coming across a crash.
This is the error I’m getting:

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py”, line 322, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py”, line 1033, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.mainfn_param_optimize
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/pyplot.py”, line 2862, in scatter
__ret = gca().scatter(
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/init.py”, line 1442, in inner
return func(ax, *map(sanitize_sequence, args), **kwargs)
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/_axes.py”, line 4602, in scatter
self._parse_scatter_color_args(
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/_axes.py”, line 4400, in _parse_scatter_color_args
and isinstance(cbook._safe_first_finite(c), str)))
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/cbook/init.py”, line 1715, in _safe_first_finite
return next(val for val in obj if safe_isfinite(val))
StopIteration

Looking in the log output file, I see a few other suspect lines from the same job that are connected and are repeated a lot:

refmotion worker 0 (NVIDIA GeForce RTX 2080 Ti)
scale (alpha): -nan
noise model (sigma2): -nan

BFGS problem in opt_traj_gpu: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH
BFGS problem in opt_traj_gpu: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH
BFGS problem in opt_traj_gpu: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH

My pipeline has been: the micrographs have been motion corrected with the full frame motion correction job and particle is a filament so has been processed with the helical refinement job.

I have tried using different memory limits for the RAM and I’ve tried but fast and balanced hyperparameter searching. Does anyone have any ideas about what the cause is?

[mathewmclaren]

Just a little update on this. I decided to rerun the motion correction as a patch job just in case the full-frame motion correction was the problem and it seems to be working fine now. I was rereading the info page for the job and I saw this:

As inputs, the job requires movies, particles, and one or more reference volumes. The connected movies must have rigid motion estimates and background estimates. A patch motion correction job provides these estimates.
This doesn’t explicitly mention that full-frame motion correction doesn’t work, but it would definitely be good to know for future reference whether or not this is the case.

[hsnyder]

Hi @mathewmclaren, I’m glad you found a workaround. In our testing, Reference motion has worked with full-frame motion correction as the input as well, so I’m wondering if there’s something specific about your dataset that is causing this issue to arise. Do the full-frame motion estimates look unusual?

[mathewmclaren]

I had a look at the plots and they seem to be fine. The total motion is around 1-16 pixels and the curvature is 1-3. The micrographs I’ve looked at seem normal as well. I didn’t want to try again with any other datasets processed with full-frame correction just in case any further issues arise, so I’ve just repeated motion correction using patches instead. The full-frame motion correction was done using an older version of cryoSPARC, probably 3.x - could this be a potential issue? Otherwise, I can’t think of anything off-hand that might be causing this.

[hsnyder]

Hi @mathewmclaren, I can’t think of any obvious reason why full-frame motion from an older version of cryosparc would trigger this issue, but do let me know if you find any pattern in this regard. I’m glad that running those datasets through patch motion correction is a good workaround. For what it’s worth, we do always recommend patch motion over full-frame motion unless there’s some obscure detail or goal related to a dataset that really indicates the use of full-frame motion correction.

[eugene_pichkur]

Hi!

I’m dealing with the same problem:

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py”, line 322, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py”, line 1033, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.mainfn_param_optimize
File “/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/pyplot.py”, line 2862, in scatter
__ret = gca().scatter(
File “/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/init.py”, line 1442, in inner
return func(ax, *map(sanitize_sequence, args), **kwargs)
File “/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/_axes.py”, line 4602, in scatter
self._parse_scatter_color_args(
File “/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/_axes.py”, line 4400, in _parse_scatter_color_args
and isinstance(cbook._safe_first_finite(c), str)))
File “/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/cbook/init.py”, line 1715, in _safe_first_finite
return next(val for val in obj if safe_isfinite(val))
StopIteration

Im my case there is no difference between patch motion correction and full frame motion correction as an input, but I know for sure that some movies contain dark or partially illuminated first and (sometimes) last frames. Movies were collected using Falcon 2 (integrating mode only), so there is no gain reference as it’s already applied.
My cryoSPARC version is 4.4.1, CUDA 12.3. RBMS works fine for K3 and Falcon 4 data.

[eugene_pichkur]

To add detail, this problem first occurred when I used patch-based motion correction as input with the options to discard the first and last frames. I then tried full-frame motion correction without discarding those frames, re-run the refinement, but the RBMS job still failed. Then I tried skipping parameter estimation using already known values from K3 data, but still nothing worked, at the second dose estimation step I got:

assertion failed: h->magic == TALLOC_HEADER_MAGIC

I then tried to re-run the patch motion correction with all frames, but this resulted in meaningless motion tracks as expected.

[hsnyder]

Hi @eugene_pichkur,

This is a known bug, as explained in the linked topic, though unfortunately if your first frame is bad, then the proposed workaround won’t work for you: Reference based motion correction fails with "h->magic == TALLOC_HEADER_MAGIC" - #5 by hsnyder

Sorry that you’re encountering it as well. This will be fixed in a future release.

–Harris