Reference motion correction crashes after first iteration

Motion Correction
1

Hi all,

I’m trying to run the new polishing job on an existing dataset already processed using cryoSPARC and I’m coming across a crash.
This is the error I’m getting:

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py”, line 322, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py”, line 1033, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.mainfn_param_optimize
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/pyplot.py”, line 2862, in scatter
__ret = gca().scatter(
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/init.py”, line 1442, in inner
return func(ax, *map(sanitize_sequence, args), **kwargs)
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/_axes.py”, line 4602, in scatter
self._parse_scatter_color_args(
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/_axes.py”, line 4400, in _parse_scatter_color_args
and isinstance(cbook._safe_first_finite(c), str)))
File “/usr/local/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/cbook/init.py”, line 1715, in _safe_first_finite
return next(val for val in obj if safe_isfinite(val))
StopIteration

Looking in the log output file, I see a few other suspect lines from the same job that are connected and are repeated a lot:

refmotion worker 0 (NVIDIA GeForce RTX 2080 Ti)
scale (alpha): -nan
noise model (sigma2): -nan

BFGS problem in opt_traj_gpu: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH
BFGS problem in opt_traj_gpu: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH
BFGS problem in opt_traj_gpu: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH

My pipeline has been: the micrographs have been motion corrected with the full frame motion correction job and particle is a filament so has been processed with the helical refinement job.

I have tried using different memory limits for the RAM and I’ve tried but fast and balanced hyperparameter searching. Does anyone have any ideas about what the cause is?

1 Like
2

Just a little update on this. I decided to rerun the motion correction as a patch job just in case the full-frame motion correction was the problem and it seems to be working fine now. I was rereading the info page for the job and I saw this:

As inputs, the job requires movies, particles, and one or more reference volumes. The connected movies must have rigid motion estimates and background estimates. A patch motion correction job provides these estimates.
This doesn’t explicitly mention that full-frame motion correction doesn’t work, but it would definitely be good to know for future reference whether or not this is the case.

3

Hi @mathewmclaren, I’m glad you found a workaround. In our testing, Reference motion has worked with full-frame motion correction as the input as well, so I’m wondering if there’s something specific about your dataset that is causing this issue to arise. Do the full-frame motion estimates look unusual?

4

I had a look at the plots and they seem to be fine. The total motion is around 1-16 pixels and the curvature is 1-3. The micrographs I’ve looked at seem normal as well. I didn’t want to try again with any other datasets processed with full-frame correction just in case any further issues arise, so I’ve just repeated motion correction using patches instead. The full-frame motion correction was done using an older version of cryoSPARC, probably 3.x - could this be a potential issue? Otherwise, I can’t think of anything off-hand that might be causing this.

5

Hi @mathewmclaren, I can’t think of any obvious reason why full-frame motion from an older version of cryosparc would trigger this issue, but do let me know if you find any pattern in this regard. I’m glad that running those datasets through patch motion correction is a good workaround. For what it’s worth, we do always recommend patch motion over full-frame motion unless there’s some obscure detail or goal related to a dataset that really indicates the use of full-frame motion correction.

6

Hi!

I’m dealing with the same problem:

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py”, line 322, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/refmotion.py”, line 1033, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.refmotion.mainfn_param_optimize
File “/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/pyplot.py”, line 2862, in scatter
__ret = gca().scatter(
File “/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/init.py”, line 1442, in inner
return func(ax, *map(sanitize_sequence, args), **kwargs)
File “/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/_axes.py”, line 4602, in scatter
self._parse_scatter_color_args(
File “/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/_axes.py”, line 4400, in _parse_scatter_color_args
and isinstance(cbook._safe_first_finite(c), str)))
File “/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/cbook/init.py”, line 1715, in _safe_first_finite
return next(val for val in obj if safe_isfinite(val))
StopIteration

Im my case there is no difference between patch motion correction and full frame motion correction as an input, but I know for sure that some movies contain dark or partially illuminated first and (sometimes) last frames. Movies were collected using Falcon 2 (integrating mode only), so there is no gain reference as it’s already applied.
My cryoSPARC version is 4.4.1, CUDA 12.3. RBMS works fine for K3 and Falcon 4 data.

7

To add detail, this problem first occurred when I used patch-based motion correction as input with the options to discard the first and last frames. I then tried full-frame motion correction without discarding those frames, re-run the refinement, but the RBMS job still failed. Then I tried skipping parameter estimation using already known values from K3 data, but still nothing worked, at the second dose estimation step I got:

assertion failed: h->magic == TALLOC_HEADER_MAGIC

I then tried to re-run the patch motion correction with all frames, but this resulted in meaningless motion tracks as expected.
J76_patch_motion_for_011436195794184589356_foilhole_17333287_data_17334477_17334478_20210917_1754_frames

8

Hi @eugene_pichkur,

This is a known bug, as explained in the linked topic, though unfortunately if your first frame is bad, then the proposed workaround won’t work for you: Reference based motion correction fails with "h->magic == TALLOC_HEADER_MAGIC" - #5 by hsnyder

Sorry that you’re encountering it as well. This will be fixed in a future release.

–Harris

9

In v4.5, RBMC generates zero trajectories on the same Falcon 2 dataset and parameters search doesn’t actually do anything, stopping after the first interaction. Truncating the first and the last frame doesn’t help either.

10

Hi @eugene_pichkur, could you have a look in the job text log for error messages? You can access it in the UI by going to the metadata tab and then choosing “log”, or by running cryosparcm joblog Pxx Jyy on the command line

11

Hi! Sorry for the delayed response. Here is a part of the log file.

================= CRYOSPARCW =======  2024-06-05 13:37:47.862834  =========
Project P9 Job J568
Master cryosparc Port 39002
===========================================================================
MAIN PROCESS PID 408904
========= now starting main process at 2024-06-05 13:37:47.863321
motioncorrection.run_reference_motion cryosparc_compute.jobs.jobregister
/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
MONITOR PROCESS PID 408906
========= monitor process now waiting for main process
========= sending heartbeat at 2024-06-05 13:37:48.533150
***************************************************************
Running job on hostname %s cryosparc
Allocated Resources :  {'fixed': {'SSD': False}, 'hostname': 'cryosparc', 'lane': 'default', 'lane_type': 'node', 'license': True, 'licenses_acquired': 1, 'slots': {'CPU': [4, 5, 6, 7, 8, 9], 'GPU': [1], 'RAM': [3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]}, 'target': {'cache_path': '/srv/pcache', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 1, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 2, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 3, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}], 'hostname': 'cryosparc', 'lane': 'default', 'monitor_port': None, 'name': 'cryosparc', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47], 'GPU': [0, 1, 2, 3], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]}, 'ssh_str': 'cryoem@cryosparc', 'title': 'Worker node cryosparc', 'type': 'node', 'worker_bin_path': '/opt/cryoem/cryosparc_worker/bin/cryosparcw'}}
FSC No-Mask...         ========= sending heartbeat at 2024-06-05 13:37:58.548081
0.143 at 30.634 radwn. 0.5 at 23.181 radwn. Took 9.783s.
FSC With Mask...      ========= sending heartbeat at 2024-06-05 13:38:08.562012
0.143 at 43.136 radwn. 0.5 at 33.117 radwn. Took 9.484s.
<string>:1: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
========= sending heartbeat at 2024-06-05 13:38:18.576437
gpufft: creating new cufft plan (plan id 0   pid 408904) 
	gpu_id  1 
	ndims   3 
	dims    512 512 512 
	inembed 512 512 514 
	istride 1 
	idist   134742016 
	onembed 512 512 257 
	ostride 1 
	odist   67371008 
	batch   1 
	type    R2C 
	wkspc   automatic 
	Python traceback:

/opt/cryoem/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/nvrtc.py:257: UserWarning: NVRTC log messages whilst compiling kernel:

kernel(963): warning #177-D: variable "Nb2p1" was declared but never referenced


  warnings.warn(msg)

refmotion worker 0 (NVIDIA GeForce RTX 3090)
scale (alpha):        -nan
noise model (sigma2): -nan
     TIME (s)  SECTION
  0.000022994  sanity
  3.335733615  read movie
  0.000003487  get gain, defects
  0.092959357  read bg
  0.000102589  read rigid
  0.480308550  prep_movie
  0.563904728  extract from frames
  0.000752688  extract from refs
  0.000007184  adj
  0.000000130  bfactor
  0.013949821  rigid motion correct
  0.000165340  get noise, scale
  4.487910482  --- TOTAL ---

BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              

refmotion worker 0 (NVIDIA GeForce RTX 3090)
Min BFGS iterations:  0
Max BFGS iterations:  0
 AVG TIME (s)  SECTION
  0.000000000  Cache IO
  0.013677160  Optimize trajectories
  0.000628146  Compute cross-validation
  0.000091230  Save trajectory
  0.014396536  --- TOTAL (Cros-Val) ---


refmotion worker 1 (NVIDIA GeForce RTX 3090)
scale (alpha):        -nan
noise model (sigma2): -nan
     TIME (s)  SECTION
  0.000020851  sanity
  3.514260015  read movie
  0.000002555  get gain, defects
  0.006901684  read bg
  0.000061118  read rigid
  0.476125691  prep_movie
  0.380864717  extract from frames
  0.000710025  extract from refs
  0.000005020  adj
  0.000000130  bfactor
  0.011862199  rigid motion correct
  0.000150081  get noise, scale
  4.390964086  --- TOTAL ---

BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              

refmotion worker 1 (NVIDIA GeForce RTX 3090)
Min BFGS iterations:  0
Max BFGS iterations:  0
 AVG TIME (s)  SECTION
  0.000000000  Cache IO
  0.012036821  Optimize trajectories
  0.000571479  Compute cross-validation
  0.000114661  Save trajectory
  0.012722961  --- TOTAL (Cros-Val) ---

========= sending heartbeat at 2024-06-05 13:38:28.590646

refmotion worker 1 (NVIDIA GeForce RTX 3090)
scale (alpha):        -nan
noise model (sigma2): -nan
     TIME (s)  SECTION
  0.000011192  sanity
  3.505306919  read movie
  0.000002966  get gain, defects
  0.010982936  read bg
  0.000206811  read rigid
  0.533659093  prep_movie
  0.375072222  extract from frames
  0.000598809  extract from refs
  0.000007104  adj
  0.000000150  bfactor
  0.013733693  rigid motion correct
  0.000152034  get noise, scale
  4.439733928  --- TOTAL ---

BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                
========= sending heartbeat at 2024-06-05 14:00:30.307391
BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              

refmotion worker 0 (NVIDIA GeForce RTX 3090)
BFGS iterations:      0
scale (alpha):        -nan
noise model (sigma2): -nan
     TIME (s)  SECTION
  0.000010721  sanity
  6.209569568  read movie
  0.000002124  get gain, defects
  0.106970186  read bg
  0.000037933  read rigid
  0.441303145  prep_movie
  1.179054230  extract from frames
  0.000747818  extract from refs
  0.000009268  adj
  0.000000140  bfactor
  0.035843484  rigid motion correct
  0.000372040  get noise, scale
  0.020325226  optimize trajectory
  0.456935984  shift_sum patches
  0.004998088  ifft
  0.016113801  unpad
  0.000019267  fill out dataset
  0.025494485  write output files
  8.497807509  --- TOTAL ---

BFGS problem while attempting to estimate trajectories: (iters: 0, fn evals: 21) ABNORMAL_TERMINATION_IN_LNSRCH                              

refmotion worker 1 (NVIDIA GeForce RTX 3090)
BFGS iterations:      0
scale (alpha):        -nan
noise model (sigma2): -nan
     TIME (s)  SECTION
  0.000010420  sanity
  6.584119729  read movie
  0.000002254  get gain, defects
  0.020686305  read bg
  0.000030930  read rigid
  0.532058875  prep_movie
  0.906994742  extract from frames
  0.000502192  extract from refs
  0.000003357  adj
  0.000000140  bfactor
  0.019272647  rigid motion correct
  0.000246397  get noise, scale
  0.013766236  optimize trajectory
  0.314869211  shift_sum patches
  0.003252808  ifft
  0.010311937  unpad
  0.000019067  fill out dataset
  0.014941742  write output files
  8.421088989  --- TOTAL ---

<string>:1: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
17759899408047537129 (0.045 GiB) -> cache in memory
6338579942102911542 (0.043 GiB) -> cache in memory
17895439457370695783 (0.040 GiB) -> cache in memory
16676411176888399008 (0.044 GiB) -> cache in memory
15049069864001104569 (0.037 GiB) -> cache in memory
11931243749759286639 (0.040 GiB) -> cache in memory
15663738474946531102 (0.024 GiB) -> cache in memory
11085399655876863566 (0.037 GiB) -> cache in memory
14760531268354910877 (0.054 GiB) -> cache in memory
3137158999644402350 (0.049 GiB) -> cache in memory
2584242116799678402 (0.034 GiB) -> cache in memory
17241399646709987481 (0.053 GiB) -> cache in memory
6333405462770530498 (0.041 GiB) -> cache in memory
5065968626686323626 (0.042 GiB) -> cache in memory
37053212132028609 (0.036 GiB) -> cache in memory
2825862036207507820 (0.042 GiB) -> cache in memory
14337212201131067949 (0.046 GiB) -> cache in memory
4875098917682695412 (0.040 GiB) -> cache in memory
8121405977702554561 (0.047 GiB) -> cache in memory
11783317076477610337 (0.033 GiB) -> cache in memory
4871536605908497728 (0.047 GiB) -> cache in memory
11869631441993347766 (0.046 GiB) -> cache in memory
12564191883571071317 (0.004 GiB) -> cache in memory
13558141013104793321 (0.047 GiB) -> cache in memory
11586612311403417194 (0.038 GiB) -> cache in memory
17169740072656081265 (0.050 GiB) -> cache in memory
17688256495867781329 (0.046 GiB) -> cache in memory
433813041761589427 (0.052 GiB) -> cache in memory
12544972823819154874 (0.039 GiB) -> cache in memory
13082398772540685134 (0.037 GiB) -> cache in memory
18061521751409115070 (0.049 GiB) -> cache in memory
7974438742359703571 (0.049 GiB) -> cache in memory
10792707984443117180 (0.048 GiB) -> cache in memory
16233928096160118762 (0.038 GiB) -> cache in memory
16546409167939866318 (0.056 GiB) -> cache in memory
12882832501224248651 (0.035 GiB) -> cache in memory
14334161572565213209 (0.040 GiB) -> cache in memory
3944049454032363101 (0.043 GiB) -> cache in memory
11273910526151665254 (0.035 GiB) -> cache in memory
12946541647311372970 (0.048 GiB) -> cache in memory
13889466711805816244 (0.029 GiB) -> cache in memory
16951008346553708989 (0.050 GiB) -> cache in memory
13879921996192107580 (0.049 GiB) -> cache in memory
18272922326032167209 (0.043 GiB) -> cache in memory
18261558092192269688 (0.038 GiB) -> cache in memory
8905881028513268650 (0.046 GiB) -> cache in memory
18128971698582771728 (0.040 GiB) -> cache in memory
7791381216208083839 (0.035 GiB) -> cache in memory
17183049941020705830 (0.049 GiB) -> cache in memory
3382558405193260716 (0.049 GiB) -> cache in memory
7888425891418398667 (0.043 GiB) -> cache in memory
1544628690901812628 (0.038 GiB) -> cache in memory
2448065133024760531 (0.037 GiB) -> cache in memory
3411261104753748753 (0.025 GiB) -> cache in memory
53271613460935311 (0.040 GiB) -> cache in memory
14063975853710158958 (0.054 GiB) -> cache in memory
11648482015726379064 (0.043 GiB) -> cache in memory
8174804590332892896 (0.053 GiB) -> cache in memory
18149244831418112125 (0.037 GiB) -> cache in memory
5276985052807465670 (0.037 GiB) -> cache in memory
3535155049752180865 (0.031 GiB) -> cache in memory
7355324687906657768 (0.042 GiB) -> cache in memory
6414625393421694093 (0.046 GiB) -> cache in memory
10844778876473744100 (0.034 GiB) -> cache in memory
10558012486122626891 (0.039 GiB) -> cache in memory
12491964733508030720 (0.040 GiB) -> cache in memory
600215084111216693 (0.043 GiB) -> cache in memory
776114183447395061 (0.054 GiB) -> cache in memory
2045795438414411886 (0.035 GiB) -> cache in memory
10889731293925324102 (0.050 GiB) -> cache in memory
9666263895435306574 (0.040 GiB) -> cache in memory
10108310265824559698 (0.037 GiB) -> cache in memory
3540353916464390830 (0.036 GiB) -> cache in memory
14869969952980312027 (0.046 GiB) -> cache in memory
2692406647394684092 (0.044 GiB) -> cache in memory
8223499126913286967 (0.045 GiB) -> cache in memory
3197204831633690521 (0.046 GiB) -> cache in memory
547318657994055369 (0.053 GiB) -> cache in memory
11760556920178836830 (0.050 GiB) -> cache in memory
5284145630995969354 (0.015 GiB) -> cache in memory
4361763239795562531 (0.037 GiB) -> cache in memory
6927758216088118049 (0.048 GiB) -> cache in memory
4320633124972784924 (0.045 GiB) -> cache in memory
902825595680957936 (0.048 GiB) -> cache in memory
8921267222444633088 (0.038 GiB) -> cache in memory
6366073996145615403 (0.045 GiB) -> cache in memory
12148777701408994859 (0.048 GiB) -> cache in memory
7682041139475436935 (0.050 GiB) -> cache in memory
3558005849431273609 (0.039 GiB) -> cache in memory
5342903397584388865 (0.037 GiB) -> cache in memory
955020920483008977 (0.043 GiB) -> cache in memory
15977908966233115461 (0.043 GiB) -> cache in memory
15886038643724476878 (0.053 GiB) -> cache in memory
2853169665936933650 (0.039 GiB) -> cache in memory
2085290006650085983 (0.048 GiB) -> cache in memory
10328774708840113191 (0.045 GiB) -> cache in memory
2504895449701718261 (0.042 GiB) -> cache in memory
651648017212638903 (0.043 GiB) -> cache in memory
15138111384206103191 (0.041 GiB) -> cache in memory
18137358999961898500 (0.054 GiB) -> cache in memory
2280226939266801886 (0.040 GiB) -> cache in memory
10801823503039594973 (0.045 GiB) -> cache in memory
6611733800687920277 (0.037 GiB) -> cache in memory
5319981878311135588 (0.037 GiB) -> cache in memory
15017819343968300616 (0.046 GiB) -> cache in memory
17621057073746265017 (0.042 GiB) -> cache in memory
16031253708068827633 (0.039 GiB) -> cache in memory
21238843451050474 (0.043 GiB) -> cache in memory
14968707496543366874 (0.039 GiB) -> cache in memory
2677579439073709807 (0.040 GiB) -> cache in memory
17806642861127306683 (0.040 GiB) -> cache in memory
1220900522261476807 (0.041 GiB) -> cache in memory
7705832250223220062 (0.038 GiB) -> cache in memory
7632139629640782930 (0.035 GiB) -> cache in memory
7861313695753278902 (0.043 GiB) -> cache in memory
1575078004237444020 (0.032 GiB) -> cache in memory
10131700436896801636 (0.043 GiB) -> cache in memory
7562899840940790984 (0.033 GiB) -> cache in memory
16995215661048920277 (0.043 GiB) -> cache in memory
2254512877192144018 (0.037 GiB) -> cache in memory
16836622511552371372 (0.042 GiB) -> cache in memory
8590297896409488949 (0.044 GiB) -> cache in memory
1754992518901349536 (0.042 GiB) -> cache in memory
842425628018737680 (0.031 GiB) -> cache in memory
2818585104729335786 (0.034 GiB) -> cache in memory
HOST ALLOCATION FUNCTION: using numba.cuda.pinned_array
SEARCH RANGES:
zs: 
	6.2146
thetas: 
	-2.0420
	-2.1468
	-2.2515
	-2.3562
	-2.4609
	-2.5656
	-2.6704
r start:
	0.1000
r end:
	10.0000
r step:
	0.6600
***************************************************************
========= main process now complete at 2024-06-05 14:00:37.931890
========= main process now complete at 2024-06-05 14:00:40.320725.
========= monitor process now complete at 2024-06-05 14:00:40.323458.
12

Hi @eugene_pichkur,

Ah, as I suspected, the trajectory optimization is failing. Usually this means there’s something wrong with the data (blank frames, for example). Are you excluding the early/late dark frames in this run? If not I’d recommend trying that.

– Harris