Bogus NaN particle warnings in refinement

3D Refinement
1

Hi, I’m seeing what appear to be bogus corrupt particle warnings during NU-refine with 4.5.1, e.g.:

Traceback (most recent call last):
  File "/home/exx/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 2294, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 134, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
  File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 135, in cryosparc_master.cryosparc_compute.gpu.gpucore.GPUThread.run
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 1136, in cryosparc_master.cryosparc_compute.engine.engine.process.work
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 421, in cryosparc_master.cryosparc_compute.engine.engine.EngineThread.compute_error
ValueError: Detected NaN values in engine.compute_error. 33976503 NaNs in total, 209 particles with NaNs.

When I use check corrupt particles on the input stack, no NaN containing particles are detected.

This only seems to happen on large particles ~>900px, and the same particle stack when downsampled to a smaller boxsize does not give any issues. When I try to run homogeneous reconstruction on the same stack, it doesn’t give any warnings, but the FSC & map slices look weird:

image
image1072×644 72 KB

image
image1748×594 67 KB

Cheers
Oli

2

Hey @olibclarke –

This looks similar to another bug we’re trying to track down related to per-particle minimization. Do these failing jobs have pp scale minimization on? If so, could you try to re-run them with it off and see if they complete without issue?

Thanks!
Valentin

1 Like
3

I think they might have been - will check and get back to you!

4

It did have scale minimization on, but repeating with it off (& scales reset to 1.0) produced the same error.

5

Ok good to know, thanks!

6

@olibclarke – a couple more questions/requests when you get the chance! Could you please try:

  1. Randomly subsampling the set of 900px particles to a (much) smaller set (maybe ~5K particles) and re-running the homogeneous reconstruction on this set.

  2. Using a fixed mask (not from the same upstream NU-Refine) within homogeneous reconstruction.

Let us know if these two changes produce different results.

Thanks again,
V

7

Hi @vperetroukhin, just to clarify, this initial error was obtained from NU-refine, not homogeneous reconstruction - do these suggestions still pertain to this case? I will try a smaller subset for homogeneous reconstruction and see if the empty map slices go away

8

Understood, yep. Based on the reconstruction plots you shared originally, we’re now suspicious that perhaps the half-maps are fine but mask generation is somehow to blame (since the no mask/spherical FSCs are not NaNs). Getting at this via homo reconstruct is just faster, which is why I asked for that first.

1 Like
9

Gotcha - will test, thanks!

10

I think you might be right that mask generation is an issue - the FSC mask generated during homogeneous reconstruction visually looks ok, but looking at the header it has some quite negative values:

image
image1120×584 83 KB

11

Very strange! Can you send us this mask mrc file? Sending you a DM shortly.

1 Like