Reference based motion correction fails when patch motion was run on parts of movie

Motion Correction
1

Due to an issue with Patch Motion and Falcon3 movies, I have to skip the last frame during motion correction. When I try to run a Reference based motion correction job with motion correction on truncated movies, I get an error:

assertion failed: rigid_frame_end <= io_movie.nz

Could you please add the option in Reference based motion correction to set an “End frame” similarily to Patch Motion or alternatively just include those frames for which there is rigid frame information.

2

Hi @daniel.s.d.larsson, does the RBMC job fail right away with that error message, or does it get partway through? It’s possible you’ve found a bug, but RBMC has been successfully tested on datasets where frames were excluded in patch motion in the manner you’re saying. I’m wondering if perhaps not all of your movies have the same frame count, in the actual files? If that’s the case I would expect this message to appear partway through processing rather than right away. Do you have an easy way to check the actual frame count in the movies on disk?

3

I tested to import the movies again, disabling “Skip Header Check” and I get lots of errors of the type:

Traceback (most recent call last):
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/jobs/imports/run.py", line 967, in run_import_movies_or_micrographs
    result.get()
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/jobs/imports/run.py", line 887, in header_check_worker
    assert cshape == datashape, "All data files must be of the same shape but file %s with shape %s did not match common shape %s" % (abs_path, str(cshape), str(datashape))
AssertionError: All data files must be of the same shape but file /home/data/storage/project/Images-Disc1/GridSquare_17798669/Data/FoilHole_18820260_Data_17803095_17803097_20231118_013500_Fractions.tif with shape (7, 4096, 4096) did not match common shape (4, 4096, 4096)

The dimensions of the data vary wildly, between e.g. (4, 4096, 4096) and (20, 4096, 4096).

The data was collected with 20 dose fractions using EPU 3.5 with a Falcon 3EC camera. Less than 10% of the movies were successfully imported as 4-frame movies. Originally, I processed the data using CryoSPARC Live with the “Last frame” parameter set to 19.

4

You may want to try @mathewmclaren’s script here to identify the culprits: https://github.com/mathewmclaren/cryoem-stuff/blob/main/bad_frames_finder.sh

1 Like
5

Thanks for the tip Oliver, but the problem is that a majority of movies are identified with fewer than the nominal number of frames, so it is not just about finding a handful of broken files. It seems like CryoSPARC jobs (import movies, patch motion and RBMC) are “hypersensitive” to have a valid TIFF header.

1 Like
6

Ah yes I see now - sorry I misread, I thought 10% were bad, but now I see it’s the other way around! That is unusual, I have not seen this with Falcon data… do the headers read ok with other programs, e.g. if you check them with IMOD do they look normal?

7

I could confirm using IMOD that the data set was corrupt with corrupted headers. I was able to perform reference-based motion correction for another data set collected on the same system and with the “End Frame” option enabled, as described in the original post.

1 Like