NaN error in refinement

strawb · April 27, 2025, 5:20am

I ran a Non-Uniform refinement job with a particle set a couple months ago, and it ran fine, no problems. I’m now revisiting this dataset to perform subsequent refinements. Submitting any subsequent non-uniform or local refinements with the same particle set gives an NaN error screenshotted below. In addition, when I cloned the same previous Non-Uniform refinement job that had run successfully and re-ran it, I encountered the same NaN issue.

I checked for corrupt particles and all particles are fine as expected. I also re-extracted the particles (400px box size) and continue to get the same error. Where could the issue be coming from? Any help would be appreciated.

leetleyang · April 27, 2025, 5:45pm

Hi,

Probably the same (yet-to-be-fixed) issue described in other threads related to per-particle scale minimization [1] [2] [3]. Perhaps try re-setting per particle scales and/or disabling minimisation to see if that helps?

Cheers,
Yang

wtempel · April 28, 2025, 7:03pm

What were the CryoSPARC versions of the

original, successful job from which the clone was generated
cloned job that failed

?

strawb · May 8, 2025, 6:58pm

Thanks for the help – sorry for the slow reply, was out of town. The current version is 4.6.2 where all refinements are failing. The previous version where all processing steps were successfully done was 4.4.1.

I have tried re-picking, re-extracting, checking integrity of particles and micrographs, excluding the “problematic” particles and micrographs, and still I am getting the same NaN issue in refinement. Your help would be much appreciated, especially if it is an issue related to version update.

wtempel · May 12, 2025, 10:07pm

Thanks @strawb

For the Check Particles and NU-Refine jobs, please can you post the outputs of these commands

csprojectid="P99" # substitute actual project ID
csjobid="J999" # substitute actual job ID for Check Particles, then NU-Refine job
cryosparcm cli "get_job('$csprojectid', '$csjobid', 'type', 'version', 'status', 'params_spec', 'started_at', 'input_slot_groups')"
cryosparcm eventlog $csprojectid $csjobid | tail -n 30

strawb · May 13, 2025, 5:33am

thanks, here you go:

check particles:

cryosparcm cli "get_job('$csprojectid', '$csjobid', 'type', 'version', 'status', 'params_spec', 'started_at', 'input_slot_groups')"
{'_id': '681a4fad71290da2634660a3', 'input_slot_groups': [{'connections': [{'group_name': 'particles', 'job_uid': 'J432', 'slots': [{'group_name': 'particles', 'job_uid': 'J432', 'result_name': 'blob', 'result_type': 'particle.blob', 'slot_name': 'blob', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J432', 'result_name': 'location', 'result_type': 'particle.location', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J432', 'result_name': 'alignments3D', 'result_type': 'particle.alignments3D', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J432', 'result_name': 'pick_stats', 'result_type': 'particle.pick_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J432', 'result_name': 'ctf', 'result_type': 'particle.ctf', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J432', 'result_name': 'ml_properties', 'result_type': 'particle.ml_properties', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Particles to check. Multiple stacks will be concatenated.', 'name': 'particles', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'blob', 'optional': False, 'title': 'Particle data blobs', 'type': 'particle.blob'}], 'title': 'Particles', 'type': 'particle'}], 'params_spec': {}, 'project_uid': 'P63', 'started_at': 'Tue, 06 May 2025 18:06:49 GMT', 'status': 'completed', 'type': 'check_corrupt_particles', 'uid': 'J440', 'version': 'v4.6.2'}


cryosparcm eventlog $csprojectid $csjobid | tail -n 30

[Tue, 06 May 2025 19:49:53 GMT] [CPU RAM used: 640 MB] This job outputted results ['blob']
[Tue, 06 May 2025 19:49:53 GMT] [CPU RAM used: 640 MB]   Loaded output dset with 651973 items
[Tue, 06 May 2025 19:49:53 GMT] [CPU RAM used: 640 MB] Passthrough results ['location', 'alignments3D', 'pick_stats', 'ctf', 'ml_properties']
[Tue, 06 May 2025 19:50:03 GMT] [CPU RAM used: 1000 MB]   Loaded passthrough dset with 652217 items
[Tue, 06 May 2025 19:50:04 GMT] [CPU RAM used: 1013 MB]   Intersection of output and passthrough has 651973 items
[Tue, 06 May 2025 19:50:36 GMT] [CPU RAM used: 1015 MB]   Output dataset contains:  ['ml_properties', 'alignments3D', 'ctf', 'pick_stats', 'location']
[Tue, 06 May 2025 19:50:36 GMT] [CPU RAM used: 1015 MB]   Outputting passthrough result location
[Tue, 06 May 2025 19:50:36 GMT] [CPU RAM used: 1015 MB]   Outputting passthrough result alignments3D
[Tue, 06 May 2025 19:50:36 GMT] [CPU RAM used: 1015 MB]   Outputting passthrough result pick_stats
[Tue, 06 May 2025 19:50:36 GMT] [CPU RAM used: 1015 MB]   Outputting passthrough result ctf
[Tue, 06 May 2025 19:50:36 GMT] [CPU RAM used: 1015 MB]   Outputting passthrough result ml_properties
[Tue, 06 May 2025 19:50:36 GMT] [CPU RAM used: 851 MB] Checking outputs for output group particles
[Tue, 06 May 2025 19:50:46 GMT] [CPU RAM used: 998 MB] Done
[Tue, 06 May 2025 19:50:46 GMT] [CPU RAM used: 998 MB] **** Job marked as completed (ID: 65f51a78b06be222c6e87a17) ****
[Tue, 06 May 2025 19:50:46 GMT] [CPU RAM used: 998 MB] Finalizing Job...
[Tue, 06 May 2025 19:50:46 GMT] [CPU RAM used: 998 MB] Passing through outputs for output group particles from input group particles
[Tue, 06 May 2025 19:50:46 GMT] [CPU RAM used: 1053 MB] This job outputted results ['blob']
[Tue, 06 May 2025 19:50:46 GMT] [CPU RAM used: 1053 MB]   Loaded output dset with 651973 items
[Tue, 06 May 2025 19:50:46 GMT] [CPU RAM used: 1053 MB] Passthrough results ['location', 'alignments3D', 'pick_stats', 'ctf', 'ml_properties']
[Tue, 06 May 2025 19:50:48 GMT] [CPU RAM used: 1198 MB]   Loaded passthrough dset with 652217 items
[Tue, 06 May 2025 19:50:49 GMT] [CPU RAM used: 1217 MB]   Intersection of output and passthrough has 651973 items
[Tue, 06 May 2025 19:51:13 GMT] [CPU RAM used: 1217 MB]   Output dataset contains:  ['ml_properties', 'alignments3D', 'ctf', 'pick_stats', 'location']
[Tue, 06 May 2025 19:51:13 GMT] [CPU RAM used: 1217 MB]   Outputting passthrough result location
[Tue, 06 May 2025 19:51:13 GMT] [CPU RAM used: 1217 MB]   Outputting passthrough result alignments3D
[Tue, 06 May 2025 19:51:13 GMT] [CPU RAM used: 1217 MB]   Outputting passthrough result pick_stats
[Tue, 06 May 2025 19:51:13 GMT] [CPU RAM used: 1217 MB]   Outputting passthrough result ctf
[Tue, 06 May 2025 19:51:13 GMT] [CPU RAM used: 1217 MB]   Outputting passthrough result ml_properties
[Tue, 06 May 2025 19:51:14 GMT] [CPU RAM used: 1053 MB] Checking outputs for output group particles
[Tue, 06 May 2025 19:51:21 GMT] [CPU RAM used: 1038 MB] Done
[Tue, 06 May 2025 19:51:21 GMT] [CPU RAM used: 1038 MB] **** Job marked as completed (ID: 65f51a78b06be222c6e87a17) ****

NU refine job

cryosparcm cli "get_job('$csprojectid', '$csjobid', 'type', 'version', 'status', 'params_spec', 'started_at', 'input_slot_groups')"
{'_id': '681a503f71290da263469ca0', 'input_slot_groups': [{'connections': [{'group_name': 'particles', 'job_uid': 'J440', 'slots': [{'group_name': 'particles', 'job_uid': 'J440', 'result_name': 'blob', 'result_type': 'particle.blob', 'slot_name': 'blob', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J440', 'result_name': 'ctf', 'result_type': 'particle.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J440', 'result_name': 'alignments3D', 'result_type': 'particle.alignments3D', 'slot_name': 'alignments3D', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J440', 'result_name': 'location', 'result_type': 'particle.location', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J440', 'result_name': 'pick_stats', 'result_type': 'particle.pick_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J440', 'result_name': 'ml_properties', 'result_type': 'particle.ml_properties', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Particle stacks to use. Multiple stacks will be concatenated.', 'name': 'particles', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'blob', 'optional': False, 'title': 'Particle data blobs', 'type': 'particle.blob'}, {'description': '', 'name': 'ctf', 'optional': False, 'title': 'Particle ctf parameters', 'type': 'particle.ctf'}, {'description': '', 'name': 'alignments3D', 'optional': True, 'title': 'Particle 3D alignments (optional)', 'type': 'particle.alignments3D'}], 'title': 'Particle stacks', 'type': 'particle'}, {'connections': [{'group_name': 'volume', 'job_uid': 'J246', 'slots': [{'group_name': 'volume', 'job_uid': 'J246', 'result_name': 'map', 'result_type': 'volume.blob', 'slot_name': 'map', 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J246', 'result_name': 'map_sharp', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J246', 'result_name': 'map_half_A', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J246', 'result_name': 'map_half_B', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J246', 'result_name': 'mask_refine', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J246', 'result_name': 'mask_fsc', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J246', 'result_name': 'mask_fsc_auto', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'volume', 'job_uid': 'J246', 'result_name': 'precision', 'result_type': 'volume.blob', 'slot_name': None, 'version': 'F'}]}], 'count_max': 1, 'count_min': 1, 'description': '', 'name': 'volume', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'map', 'optional': False, 'title': 'Initial volume raw data', 'type': 'volume.blob'}], 'title': 'Initial volume', 'type': 'volume'}, {'connections': [], 'count_max': 1, 'count_min': 0, 'description': '', 'name': 'mask', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'mask', 'optional': False, 'title': 'Static mask', 'type': 'volume.blob'}], 'title': 'Static mask', 'type': 'mask'}], 'params_spec': {'refine_dynamic_mask_start_res': {'value': 1}, 'refine_num_final_iterations': {'value': 2}, 'refine_res_init': {'value': 15}, 'refine_scale_min': {'value': True}, 'refine_symmetry': {'value': 'C1'}}, 'project_uid': 'P63', 'started_at': 'Tue, 06 May 2025 19:50:52 GMT', 'status': 'failed', 'type': 'nonuniform_refine_new', 'uid': 'J441', 'version': 'v4.6.2'}


cryosparcm eventlog $csprojectid $csjobid | tail -n 30
[Tue, 06 May 2025 21:55:21 GMT] [CPU RAM used: 12033 MB]   Estimated Bfactor: -143.8
[Tue, 06 May 2025 21:55:21 GMT] [CPU RAM used: 12033 MB]   Plotting..
[Tue, 06 May 2025 21:55:27 GMT]  Real Space Slices Iteration 003
[Tue, 06 May 2025 21:55:28 GMT]  Fourier Space Slices Iteration 003
[Tue, 06 May 2025 21:55:29 GMT]  Real Space Mask Slices Iteration 003
[Tue, 06 May 2025 21:55:29 GMT]  FSC Iteration 003
[Tue, 06 May 2025 21:55:30 GMT]  cFSCs (Half-angle: 20°) Iteration 003, with tight mask
[Tue, 06 May 2025 21:55:35 GMT]  Guinier Plot Iteration 003
[Tue, 06 May 2025 21:55:35 GMT]  Noise Model Iteration 003
[Tue, 06 May 2025 21:55:37 GMT]  Viewing Direction Distribution Iteration 003
[Tue, 06 May 2025 21:55:39 GMT]  Posterior Precision Directional Distribution Iteration 003
[Tue, 06 May 2025 21:55:41 GMT]  Per particle scale factors 003
[Tue, 06 May 2025 21:55:41 GMT] [CPU RAM used: 12068 MB]     Done in 19.810s.
[Tue, 06 May 2025 21:55:41 GMT] [CPU RAM used: 12068 MB]   Outputting files..
[Tue, 06 May 2025 21:58:16 GMT] [CPU RAM used: 11981 MB]     Done in 155.287s.
[Tue, 06 May 2025 21:58:16 GMT] [CPU RAM used: 11981 MB] Done iteration 3 in 3229.738s. Total time so far 7636.246s
[Tue, 06 May 2025 21:58:17 GMT] [CPU RAM used: 11981 MB] ----------------------------- Start Iteration 4
[Tue, 06 May 2025 21:58:17 GMT] [CPU RAM used: 11981 MB]   Using Max Alignment Radius 75.227 (3.658A)
[Tue, 06 May 2025 21:58:17 GMT] [CPU RAM used: 11981 MB]   Using Full Dataset (split 325912 in A, 326061 in B)
[Tue, 06 May 2025 21:58:20 GMT] [CPU RAM used: 11980 MB]   Minimizing over per-particle scale factor
[Tue, 06 May 2025 21:58:20 GMT] [CPU RAM used: 11980 MB] -- THR 0 BATCH 500 NUM 12500 TOTAL 37.354285 ELAPSED 118.71109 --
[Tue, 06 May 2025 22:00:20 GMT] [CPU RAM used: 13591 MB] Traceback (most recent call last):
  File "/home/cryosparc4/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 2304, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/cryosparc4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 2730, in cryosparc_master.cryosparc_compute.engine.newengine.process.work
  File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 2809, in cryosparc_master.cryosparc_compute.engine.newengine.process.work
  File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 1534, in cryosparc_master.cryosparc_compute.engine.newengine.EngineThread.compute_error
ValueError: Detected NaN values in newengine.compute_error. 162567 NaNs in total, 1 particles with NaNs.

wtempel · May 13, 2025, 8:55pm

Thanks for posting the additional information.
Please can you

clone the Check Particles job J432
in the new job enable Check for NaN values
run the new job

after job completion, post the outputs of these commands

csprojectid="P63" 
csjobid="J999" # substitute actual job ID for Check Particles with NaN check enabled
cryosparcm cli "get_job('$csprojectid', '$csjobid', 'type', 'version', 'status', 'params_spec', 'started_at', 'input_slot_groups')"
cryosparcm eventlog $csprojectid $csjobid | tail -n 30