Ab-initio error

3D Reconstruction
1

Hi I’m getting this error when try to run ab-initio on NS data set:
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 115, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/abinit/run.py”, line 316, in cryosparc_master.cryosparc_compute.jobs.abinit.run.run_homo_abinit
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1194, in cryosparc_master.cryosparc_compute.engine.engine.process
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1195, in cryosparc_master.cryosparc_compute.engine.engine.process
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1136, in cryosparc_master.cryosparc_compute.engine.engine.process.work
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 421, in cryosparc_master.cryosparc_compute.engine.engine.EngineThread.compute_error
ValueError: Detected NaN values in engine.compute_error. 10445760 NaNs in total, 90 particles with NaNs.

I have run corrupt particle check on the same particles set and no corruption found.
The 2D look OK

image
image696×96 12.9 KB

Best,

2

@Elad What version of CryoSPARC do you use?

3

I’m using CryoSPARC 4.5.0

4

Thanks @Elad. Please can you post the output of the command

cryosparcm cli "get_job('P99', 'J199', 'job_type', 'version', 'instance_information', 'status',  'params_spec', 'errors_run')"

where P99 and J199 are replaced by the failed job’s project and job IDs, respectively.

5

{‘_id’: ‘66fd90e261e050ac966f3069’, ‘errors_run’: [{‘message’: ‘Detected NaN values in engine.compute_error. 10445760 NaNs in total, 90 particles with NaNs.’, ‘warning’: False}], ‘instance_information’: {‘CUDA_version’: ‘11.8’, ‘available_memory’: ‘493.12GB’, ‘cpu_model’: ‘Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz’, ‘driver_version’: ‘12.4’, ‘gpu_info’: [{‘id’: 0, ‘mem’: 11539054592, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’, ‘pcie’: ‘0000:02:00’}, {‘id’: 1, ‘mem’: 11539054592, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’, ‘pcie’: ‘0000:03:00’}, {‘id’: 2, ‘mem’: 11539054592, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’, ‘pcie’: ‘0000:83:00’}, {‘id’: 3, ‘mem’: 11539054592, ‘name’: ‘NVIDIA GeForce RTX 2080 Ti’, ‘pcie’: ‘0000:84:00’}], ‘ofd_hard_limit’: 4096, ‘ofd_soft_limit’: 1024, ‘physical_cores’: 36, ‘platform_architecture’: ‘x86_64’, ‘platform_node’: ‘mamba.csb.vanderbilt.edu’, ‘platform_release’: ‘3.10.0-1160.76.1.el7.x86_64’, ‘platform_version’: ‘#1 SMP Wed Aug 10 16:21:17 UTC 2022’, ‘total_memory’: ‘503.79GB’, ‘used_memory’: ‘9.20GB’}, ‘job_type’: ‘homo_abinit’, ‘params_spec’: {}, ‘project_uid’: ‘P31’, ‘status’: ‘failed’, ‘uid’: ‘J103’, ‘version’: ‘v4.5.0’}

6

Thanks @Elad for posting this information.
Does ValueError: Detected NaN values in engine.compute_error occur

  • every time you run a clone of the failed job
  • when inputting particles from a different extraction, downsampling or restacking job in the same project
  • in other projects

?
If possible, can you test if the error occurs after downgrading the CryoSPARC worker’s nvidia driver to version 525 and subsequently restarting the CryoSPARC worker.

7

*Yes, every time that I tried it failed

  • different extraction, from non NS flag micrographe, solved it.
  • it is work in other project
8

I have the same error when running ab-initio.

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 115, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/abinit/run.py”, line 329, in cryosparc_master.cryosparc_compute.jobs.abinit.run.run_homo_abinit
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1194, in cryosparc_master.cryosparc_compute.engine.engine.process
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1195, in cryosparc_master.cryosparc_compute.engine.engine.process
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1136, in cryosparc_master.cryosparc_compute.engine.engine.process.work
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 421, in cryosparc_master.cryosparc_compute.engine.engine.EngineThread.compute_error
ValueError: Detected NaN values in engine.compute_error. 52228800 NaNs in total, 90 particles with NaNs.

Thanks,
Wendy

9

Hi @Wendy,

Have you tried running the check for corrupt particles job? If you turn on the parameter “Check for NaN values”, it will report whether or not any particles on disk contain NaNs. I recommend running that job, just to see if there are in fact NaN values on-disk. The job doesn’t fail if there are NaN values, it just emits a warning, which can be easy to overlook. There’s also a summary printed near the end of the job’s stream log which will list all files that contained NaNs. The job will output a subset of the input particles that is chosen to avoid all the files that contain NaNs, which may allow you to continue processing the data by using the cleaned particle subset.

–Harris

10

Hi @Wendy,

Just following up regarding my previous post. Did you try the check for corrupt particles job, and if so what was the outcome. Your input would be valuable to us as we try to diagnose the cause of these NaN value errors.

–Harris

11

I have try it and no corrupt particles.
It was happened again in a different project.
In both cases it is a NS data set that the particles do not picked with the blob picked [picking mostly non particle] so I use the exp tool to switch to non-ns and then it pick well bur give this error.

12

Hi Harris,

I tried the check for corrupt particles job . No corruption detected( see attached picture).

I can resolve this issue by reselecting the 2D classes and using the reselected classes to run an ab initio job. This approach often solves the problem.

image
image1624×746 166 KB

Thanks,
Wendy

1 Like