NaN error when processing binned movies

Motion Correction
1

Hi,

I am trying to generate some binned movies for a workshop (to speed up motion correction & reduce disk usage).

I can make an lzw-compressed 2x binned tiff using IMOD (newstack -bi 2 -->mrc followed by mrc2tif -c 5). The gain is also binned using newstack.

I can import the gain and movie into CS 4.7 and everything looks fine, but motion correction fails with this error:

Error occurred while processing J5518/imported/007816666275979851502_lzw_test.tif
Traceback (most recent call last):
  File "/home/exx/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 59, in exec
    return self.process(item)
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 357, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
  File "/home/exx/cryosparc/cryosparc_worker/cryosparc_compute/blobio/mrc.py", line 252, in write_mrc
    ioengine.write_mrc(
RuntimeError: Detected NaN/INF in 1 / 1 particles or frames: /home/exx/processing/cryosparc_projects/francesca/P2/J5519/motioncorrected/007816666275979851502_lzw_test_background.mrc


Marking J5518/imported/007816666275979851502_lzw_test.tif as incomplete and continuing...

Here is the header of the background mrc:

 RO image file on unit   1 : 007816666275979851502_lzw_test_background.mrc     Size=       4097 K

 Number of columns, rows, sections .....    1024    1024       1
 Map mode ..............................    2   (32-bit real)
 Start cols, rows, sects, grid x,y,z ...    0     0     0    1024   1024      1
 Pixel spacing (Angstroms)..............   6.640      6.640      6.640
 Cell angles ...........................   90.000   90.000   90.000
 Fast, medium, slow axes ...............    X    Y    Z
 Origin on x,y,z .......................    0.000       0.000       0.000
 Minimum density .......................    -Infinity
 Maximum density .......................     Infinity
 Mean density ..........................   531.48
 tilt angles (original,current) ........   0.0   0.0   0.0   0.0   0.0   0.0
 Space group,# extra bytes,idtype,lens .        0        0        0        0

Header of movie:

 RO image file on unit   1 : lzw_test.tif     Size=      50227 K

                    This is a TIFF file.

 Number of columns, rows, sections .....    2880    2046      50
 Map mode ..............................    0   (byte)
 Start cols, rows, sects, grid x,y,z ...    0     0     0    2880   2046     50
 Pixel spacing (Angstroms)..............   1.000      1.000      1.000
 Cell angles ...........................   90.000   90.000   90.000
 Fast, medium, slow axes ...............    X    Y    Z
 Origin on x,y,z .......................    0.000       0.000       0.000
 Minimum density .......................   0.0000
 Maximum density .......................   4.0000
 Mean density ..........................   2.0000
 tilt angles (original,current) ........   0.0   0.0   0.0   0.0   0.0   0.0
 Space group,# extra bytes,idtype,lens .        0        0        0        0

     0 Titles :

Header of gain:

 RO image file on unit   1 : bin2_gain.mrc     Size=      23018 K

 Number of columns, rows, sections .....    2880    2046       1
 Map mode ..............................    2   (32-bit real)
 Start cols, rows, sects, grid x,y,z ...    0     0     0    2880   2046      1
 Pixel spacing (Angstroms)..............   2.000      2.000      1.000
 Cell angles ...........................   90.000   90.000   90.000
 Fast, medium, slow axes ...............    X    Y    Z
 Origin on x,y,z .......................    0.000       0.000       0.000
 Minimum density .......................  0.78800
 Maximum density .......................   1.7497
 Mean density ..........................   1.0034
 tilt angles (original,current) ........   0.0   0.0   0.0   0.0   0.0   0.0
 Space group,# extra bytes,idtype,lens .        0        0        0        0

     1 Titles :
NEWSTACK: Images copied                                  9-May-25  14:14:08

If I look at the motion corrected micrograph with relion_display it looks fine, but the outputs are not usable due to this error. What is going on here?

Here are the test stack and gain so the error can be reproduced (Cs=2.7, 300kV, Apix=1.66, dose=58.2):

stack: https://www.dropbox.com/scl/fi/bfl74x7bxlzmu524wt38o/lzw_test.tif?rlkey=t0elv8sixm6137b39ujpv2aez&dl=0

gain: https://www.dropbox.com/scl/fi/cxzlr54579z75z5us01v5/bin2_gain.mrc?rlkey=p23f7o4bbhjfovj4079he2gfx&dl=0

Cheers
Oli

2

Hi @olibclarke, sorry for the delay on this. Were you trying to save the outputs to float16? I’ve seen background files overflow on float16 (though there’s a guard against it in CS 4.7), but I’d be very surprised to see a case of overflowing float32. I’ll have a look at the test stack and get back to you, but it would be very useful to know whether you were trying to save the results in float16 or not.
–Harris

3

Hi Harris - yes I think it was fp16! As I was trying to minimize space requirements for a workshop

I should say that the output micrographs are actually fine - they just can’t be used directly (or used for e.g. denoise micrographs), but they can be reimported as micrographs

4

And was this instance on cs 4.7?

5

Yes it was on 4.7 (must be at least 20 characters)

6

Yeah I was just typing “sorry I realize you already said that”. Can you send me the full job report, if you still have it? DM me for upload instructions if required.

If float16 overflows on write, 4.7 is supposed to fall back to float32 for that file. Again I’d be very surprised to see a float32 overflow, but float16 background overflows are relatively common…

7

Yep I have it - will send later!

1 Like
8

Actually it fails with either fp16 or fp32 - here are the job reports:

fp16:
https://www.dropbox.com/scl/fi/y307g0ickg1ch8skaesy4/job_report_P5_J46_2025_05_20T15_35_46.382Z_fp16.zip?rlkey=n33fjkhaz1znpj17ua847fdvx&dl=0

fp32:
https://www.dropbox.com/scl/fi/uyzo2yj53aqn5yr3l5hiw/job_report_P5_J47_2025_05_20T15_36_08.861Z_fp32.zip?rlkey=1i5mxhyejm3jhobmc32ebeuju&dl=0

9

@olibclarke thanks for sharing. it’s definitely overflowing a float32. I would need to see the actual movie to debug this further. If you can provide it that would be great. Feel free to DM me if you need upload instructions.
– Harris

10

Hi Harris - it was in the first post, along with the gain

11

Apologies! I see it.

1 Like
12

Hi @olibclarke, I looked into this and was able to determine the cause. I’ll spare you the details, but it’s ultimately a silly numerical bug that is very unlikely to ever crop up, but can. Thanks for making us aware, we’ll fix this in an upcoming release.
–Harris

2 Likes