I am encountering this weird issue in reference-based motion correction where the job finished successfully. However, there was no plot for either the trajectory activity or the empirical dose weights. In addition, the final output particles showed no motion at all. Please see the attached figures. The log indicates that the Optimize hyperparameters step “Exiting early, all search paths have reached the ~zero trajectory regime”.
@wtempel Below is the final portion of the Metadata|Log:
========= sending heartbeat at 2024-06-14 03:51:00.976034
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH
refmotion worker 2 (NVIDIA RTX A6000)
BFGS iterations: 0
scale (alpha): -nan
noise model (sigma2): -nan
TIME (s) SECTION
0.001516221 sanity
1.379050303 read movie
0.000004389 get gain, defects
0.015041977 read bg
0.000174456 read rigid
0.780070844 prep_movie
1.763310240 extract from frames
0.178361597 extract from refs
0.000090303 adj
0.000000102 bfactor
0.881658365 rigid motion correct
0.013423073 get noise, scale
0.337022049 optimize trajectory
0.640824690 shift_sum patches
0.021017806 ifft
0.024218020 unpad
0.001301263 fill out dataset
0.045755815 write output files
6.082841515 --- TOTAL ---
<string>:1: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
========= sending heartbeat at 2024-06-14 03:51:10.986981
9747549296399457272 (1.094 GiB) -> cache in memory
3635259252510360727 (1.141 GiB) -> cache in memory
8877874629219603501 (1.057 GiB) -> cache in memory
3748397017681197109 (1.236 GiB) -> cache in memory
7695884367936023843 (1.066 GiB) -> cache in memory
14036087667688387001 (1.358 GiB) -> cache in memory
17032332713817824721 (1.255 GiB) -> cache in memory
11595796904852081874 (0.651 GiB) -> cache in memory
272017425062071178 (1.066 GiB) -> cache in memory
353204565015503645 (1.104 GiB) -> cache in memory
12589193062579340792 (1.104 GiB) -> cache in memory
10912498378188119395 (0.991 GiB) -> cache in memory
3385745243335628908 (1.217 GiB) -> cache in memory
8042063966413826256 (0.962 GiB) -> cache in memory
3649944384377477201 (0.915 GiB) -> cache in memory
5642416485674044799 (0.264 GiB) -> cache in memory
4358653517423970364 (0.717 GiB) -> cache in memory
11415622915099925060 (0.953 GiB) -> cache in memory
3132653515688032378 (1.141 GiB) -> cache in memory
3163043630109554593 (0.774 GiB) -> cache in memory
18399841443150836179 (0.924 GiB) -> cache in memory
17112377282916077641 (1.057 GiB) -> cache in memory
4688830289112608865 (0.679 GiB) -> cache in memory
1532324345890157832 (1.189 GiB) -> cache in memory
2219471573186483867 (1.340 GiB) -> cache in memory
2939291246483953498 (1.236 GiB) -> cache in memory
4660769303830208845 (1.028 GiB) -> cache in memory
14390485587257269874 (1.160 GiB) -> cache in memory
789017676653990202 (0.783 GiB) -> cache in memory
6536090176841851645 (1.038 GiB) -> cache in memory
17184064480892926252 (1.311 GiB) -> cache in memory
9087185805438650695 (1.057 GiB) -> cache in memory
6583598244354239294 (1.047 GiB) -> cache in memory
8327201403500339698 (1.424 GiB) -> cache in memory
1798995984778101813 (1.057 GiB) -> cache in memory
6262708356385750841 (1.207 GiB) -> cache in memory
5187922715372048096 (0.915 GiB) -> cache in memory
3922692389523349892 (0.953 GiB) -> cache in memory
9744887392173616604 (1.075 GiB) -> cache in memory
17116961794744661107 (1.066 GiB) -> cache in memory
13788282207591185046 (1.075 GiB) -> cache in memory
2625700893024534059 (1.019 GiB) -> cache in memory
11593851965591011507 (0.726 GiB) -> cache in memory
17142677017780462292 (0.906 GiB) -> cache in memory
1328012370682502791 (0.821 GiB) -> cache in memory
11861605238711561656 (0.726 GiB) -> cache in memory
1879845505691957013 (1.189 GiB) -> cache in memory
1287845918721126759 (0.906 GiB) -> cache in memory
10417288987908097403 (1.292 GiB) -> cache in memory
14645748530722100276 (0.811 GiB) -> cache in memory
1170193806931120665 (1.170 GiB) -> cache in memory
13446554095643616459 (0.538 GiB) -> cache in memory
11722213884605583784 (0.887 GiB) -> cache in memory
1843187345183815768 (1.236 GiB) -> cache in memory
17505309684692265292 (0.896 GiB) -> cache in memory
2373430977192685999 (0.849 GiB) -> cache in memory
8578596502033121192 (1.255 GiB) -> cache in memory
829911688001626833 (0.868 GiB) -> cache in memory
5451724862529873113 (0.943 GiB) -> cache in memory
14126905630778741159 (0.764 GiB) -> cache in memory
8289083996677193100 (1.207 GiB) -> cache in memory
8383697403415656035 (0.962 GiB) -> cache in memory
6604153660660539855 (1.009 GiB) -> cache in memory
3081264685893047047 (1.141 GiB) -> cache in memory
17407898819707560094 (1.094 GiB) -> cache in memory
9217409599554839981 (1.132 GiB) -> cache in memory
4663989040908363142 (1.047 GiB) -> cache in memory
2905666071177533235 (1.009 GiB) -> cache in memory
3510273936603927726 (0.915 GiB) -> cache in memory
5649738215691261558 (1.000 GiB) -> cache in memory
4693007942737294676 (1.123 GiB) -> cache in memory
4547931621925621178 (1.207 GiB) -> cache in memory
10293217900470943052 (0.887 GiB) -> cache in memory
14383468593178732686 (0.943 GiB) -> cache in memory
9187090258974671404 (0.934 GiB) -> cache in memory
17304596655196727683 (1.123 GiB) -> cache in memory
15220107178190532708 (1.113 GiB) -> cache in memory
18433772407493530105 (0.868 GiB) -> cache in memory
9228626799084725654 (1.094 GiB) -> do not cache
5929684470581278066 (1.217 GiB) -> do not cache
11196283967324345748 (1.085 GiB) -> do not cache
5712863555824352827 (0.877 GiB) -> do not cache
7768216038223992263 (0.972 GiB) -> do not cache
8519961722024738292 (0.943 GiB) -> do not cache
6863462380164539841 (1.028 GiB) -> do not cache
14720527958266160201 (0.972 GiB) -> do not cache
7874498751476752989 (1.283 GiB) -> do not cache
9215914444322904060 (1.085 GiB) -> do not cache
4564440585562450585 (1.066 GiB) -> do not cache
13655320122326438758 (0.981 GiB) -> do not cache
6452419797163831878 (1.000 GiB) -> do not cache
343578828104043284 (1.066 GiB) -> do not cache
16852355183414087420 (0.972 GiB) -> do not cache
18158041600732891514 (1.151 GiB) -> do not cache
10200072267082044279 (0.991 GiB) -> do not cache
5003934190017334537 (1.151 GiB) -> do not cache
7957187802958711295 (1.207 GiB) -> do not cache
6457150656015951648 (1.434 GiB) -> do not cache
9508634503276445226 (1.189 GiB) -> do not cache
9756044410768145984 (0.896 GiB) -> do not cache
15258612848868049535 (0.877 GiB) -> do not cache
5398636720755207148 (1.443 GiB) -> do not cache
3215077087366045653 (0.972 GiB) -> do not cache
6227260187761005619 (0.915 GiB) -> do not cache
10280898427373664825 (1.141 GiB) -> do not cache
14254262703017908796 (0.783 GiB) -> do not cache
9983984310051115898 (1.123 GiB) -> do not cache
2618972044896693344 (1.075 GiB) -> do not cache
15848315741578094327 (1.047 GiB) -> do not cache
12841674678941095483 (1.132 GiB) -> do not cache
4465693161443338922 (1.066 GiB) -> do not cache
14353504729905609224 (0.953 GiB) -> do not cache
9513923960677179190 (1.330 GiB) -> do not cache
210574262316189147 (1.066 GiB) -> do not cache
1106555094605024079 (0.481 GiB) -> do not cache
HOST ALLOCATION FUNCTION: using numba.cuda.pinned_array
SEARCH RANGES:
zs:
4.6052
6.2146
8.0064
9.7981
thetas:
-1.9373
-2.0420
-2.1468
-2.2515
-2.3562
-2.4609
-2.5656
-2.6704
-2.7751
r start:
0.1000
r end:
10.0000
r step:
0.4950
***************************************************************
========= main process now complete at 2024-06-14 03:51:16.670584
========= main process now complete at 2024-06-14 03:51:21.002971.
========= monitor process now complete at 2024-06-14 03:51:21.008757.```
Thanks.
Yang
Thanks for posting those logs. The fact that the scale is ‘-nan’ means that there might be something wrong with the raw movies (though that’s not the only possibility). Could you show some example plots from your original patch motion correction job?
By the way, while waiting for suggestions and advice from the cryoSPARC team and the community regarding this issue, I processed the dataset in RELION from the beginning. The Bayesian Polishing in RELION worked quite well for this dataset and led to a noticeable improvement in resolution. Does it mean this issue is specific to the RBMC job in cryoSPARC?
Thanks for posting those, @YYang. The high movement in the late frames suggests the could be something wrong with or different about those frames, but the motion in the plots isn’t so extreme as to be an obvious giveaway of a problem. Do you have an easy way of downloading a movie file and inspecting the individual frames? If not, and if you’re at liberty to share the data, I could have a look - let me know if that’s possible. Alternatively, you could just try dropping the last 7 frames and trying again, but unfortunately there’s no way, yet, to drop frames directly in RBMC, meaning you’d have to restart from patch motion correction if you wanted to try that.
