Ab-initio error

3D Reconstruction
1

Hi I’m getting this error when try to run ab-initio on NS data set:
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 115, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/abinit/run.py”, line 316, in cryosparc_master.cryosparc_compute.jobs.abinit.run.run_homo_abinit
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1194, in cryosparc_master.cryosparc_compute.engine.engine.process
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1195, in cryosparc_master.cryosparc_compute.engine.engine.process
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1136, in cryosparc_master.cryosparc_compute.engine.engine.process.work
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 421, in cryosparc_master.cryosparc_compute.engine.engine.EngineThread.compute_error
ValueError: Detected NaN values in engine.compute_error. 10445760 NaNs in total, 90 particles with NaNs.

I have run corrupt particle check on the same particles set and no corruption found.
The 2D look OK

image
image696×96 12.9 KB

Best,

2

@Elad What version of CryoSPARC do you use?

3

I’m using CryoSPARC 4.5.0

4

Thanks @Elad. Please can you post the output of the command

cryosparcm cli "get_job('P99', 'J199', 'job_type', 'version', 'instance_information', 'status',  'params_spec', 'errors_run')"

where P99 and J199 are replaced by the failed job’s project and job IDs, respectively.

5

{‘_id’: ‘66fd90e261e050ac966f3069’, ‘errors_run’: [{‘message’: ‘Detected NaN values in engine.compute_error. 10445760 NaNs in total, 90 particles with NaNs.’, ‘warning’: False}], ‘instance_information’: {‘CUDA_version’: ‘11.8’, ‘available_memory’: ‘493.12GB’, ‘cpu_model’: ‘Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz’, ‘driver_version’: ‘12.4’, ‘gpu_info’: [{‘id’: 0, ‘mem’: 11539054592, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’, ‘pcie’: ‘0000:02:00’}, {‘id’: 1, ‘mem’: 11539054592, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’, ‘pcie’: ‘0000:03:00’}, {‘id’: 2, ‘mem’: 11539054592, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’, ‘pcie’: ‘0000:83:00’}, {‘id’: 3, ‘mem’: 11539054592, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’, ‘pcie’: ‘0000:84:00’}], ‘ofd_hard_limit’: 4096, ‘ofd_soft_limit’: 1024, ‘physical_cores’: 36, ‘platform_architecture’: ‘x86_64’, ‘platform_node’: ‘mamba.csb.vanderbilt.edu’, ‘platform_release’: ‘3.10.0-1160.76.1.el7.x86_64’, ‘platform_version’: ‘#1 SMP Wed Aug 10 16:21:17 UTC 2022’, ‘total_memory’: ‘503.79GB’, ‘used_memory’: ‘9.20GB’}, ‘job_type’: ‘homo_abinit’, ‘params_spec’: {}, ‘project_uid’: ‘P31’, ‘status’: ‘failed’, ‘uid’: ‘J103’, ‘version’: ‘v4.5.0’}

6

Thanks @Elad for posting this information.
Does ValueError: Detected NaN values in engine.compute_error occur

  • every time you run a clone of the failed job
  • when inputting particles from a different extraction, downsampling or restacking job in the same project
  • in other projects

?
If possible, can you test if the error occurs after downgrading the CryoSPARC worker’s nvidia driver to version 525 and subsequently restarting the CryoSPARC worker.

7

*Yes, every time that I tried it failed

  • different extraction, from non NS flag micrographe, solved it.
  • it is work in other project
8

I have the same error when running ab-initio.

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 115, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/abinit/run.py”, line 329, in cryosparc_master.cryosparc_compute.jobs.abinit.run.run_homo_abinit
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1194, in cryosparc_master.cryosparc_compute.engine.engine.process
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1195, in cryosparc_master.cryosparc_compute.engine.engine.process
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1136, in cryosparc_master.cryosparc_compute.engine.engine.process.work
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 421, in cryosparc_master.cryosparc_compute.engine.engine.EngineThread.compute_error
ValueError: Detected NaN values in engine.compute_error. 52228800 NaNs in total, 90 particles with NaNs.

Thanks,
Wendy