Reference motion correction crashes after first iteration

Hi all,

I’m trying to run the new polishing job on an existing dataset already processed using cryoSPARC and I’m coming across a crash.
This is the error I’m getting:

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/”, line 95, in
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/”, line 322, in
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/”, line 1033, in
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/”, line 2862, in scatter
__ret = gca().scatter(
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/”, line 1442, in inner
return func(ax, *map(sanitize_sequence, args), **kwargs)
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/”, line 4602, in scatter
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/”, line 4400, in _parse_scatter_color_args
and isinstance(cbook._safe_first_finite(c), str)))
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/cbook/”, line 1715, in _safe_first_finite
return next(val for val in obj if safe_isfinite(val))

Looking in the log output file, I see a few other suspect lines from the same job that are connected and are repeated a lot:

refmotion worker 0 (NVIDIA GeForce RTX 2080 Ti)
scale (alpha): -nan
noise model (sigma2): -nan

BFGS problem in opt_traj_gpu: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH
BFGS problem in opt_traj_gpu: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH
BFGS problem in opt_traj_gpu: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH

My pipeline has been: the micrographs have been motion corrected with the full frame motion correction job and particle is a filament so has been processed with the helical refinement job.

I have tried using different memory limits for the RAM and I’ve tried but fast and balanced hyperparameter searching. Does anyone have any ideas about what the cause is?

Just a little update on this. I decided to rerun the motion correction as a patch job just in case the full-frame motion correction was the problem and it seems to be working fine now. I was rereading the info page for the job and I saw this:

As inputs, the job requires movies, particles, and one or more reference volumes. The connected movies must have rigid motion estimates and background estimates. A patch motion correction job provides these estimates.
This doesn’t explicitly mention that full-frame motion correction doesn’t work, but it would definitely be good to know for future reference whether or not this is the case.

Hi @mathewmclaren, I’m glad you found a workaround. In our testing, Reference motion has worked with full-frame motion correction as the input as well, so I’m wondering if there’s something specific about your dataset that is causing this issue to arise. Do the full-frame motion estimates look unusual?

I had a look at the plots and they seem to be fine. The total motion is around 1-16 pixels and the curvature is 1-3. The micrographs I’ve looked at seem normal as well. I didn’t want to try again with any other datasets processed with full-frame correction just in case any further issues arise, so I’ve just repeated motion correction using patches instead. The full-frame motion correction was done using an older version of cryoSPARC, probably 3.x - could this be a potential issue? Otherwise, I can’t think of anything off-hand that might be causing this.

Hi @mathewmclaren, I can’t think of any obvious reason why full-frame motion from an older version of cryosparc would trigger this issue, but do let me know if you find any pattern in this regard. I’m glad that running those datasets through patch motion correction is a good workaround. For what it’s worth, we do always recommend patch motion over full-frame motion unless there’s some obscure detail or goal related to a dataset that really indicates the use of full-frame motion correction.