Set frame range for RBMC [Feature request]

Hi,

With reference to the issue discussed here, it would be useful to be able to set a frame range to use during reference-based motion correction, so that later frames of the movie can be excluded.

Cheers
Oli

3 Likes

(It would also be useful to be able to calculate frame range reconstructions without recalculating trajectories each time, and with/without dose weights - this would be useful for analysis of radiation damage)

1 Like

Hi Oli. Thanks for the feature request(s). Good points, we’ve noted the request and we will consider this for a future release. In the meantime, reference motion re-uses the frame range from patch motion (because it uses the rigid motion trajectory computed in patch motion), so as an (admittedly inconvenient) workaround, you could use multiple patch motion jobs to control which frames are used in reference motion.

– Harris

3 Likes

@hsnyder

Could we get this behavior without re-running patch motion by changing the frame count of a set of exposures via cryosparc-tools?

@DanielAsarnow Maybe, but I’ve never tried it. You’d also have to chop the corresponding frames off of the rigid motion estimate that patch motion produces, though.

You mean in the .npy files themselves, or just the shape metadata fields?

@DanielAsarnow sorry for the delay. You’d have to do both.

I just tried the “re-run Patch Motion Correction with defined frame count” method (as I forgot and got all the way to RBMC before it crashed with:
AssertionError: All movies must have the same number of frames

So I re-ran Patch Motion Correction defining a maximum number of frames to use, got to RBMC again (and gained a little resolution also!) but RBMC is still crashing with:
AssertionError: All movies must have the same number of frames

At this point, I think I might just process the movies so that only the first n frames are present for the whole dataset, and see if that works. If CryoSPARC actually did obey the PMC input instructions, that should work? If it read information concerning the extra frames, or didn’t obey the frame limit input, I guess it’ll crash. Asking it to ignore movies with incorrect frame counts makes it throw away about 70% of the dataset, despite PMC being set to use 80 frames (0-79, which was the lowest common denominator for all datasets).

Hi @rbs_sci, are you sure that the dataset didn’t unintentionally have some moves which actually contain fewer frames than they’re supposed to? If you have, for example, four movies, three of which have 30 frames and one of which has 24, and you ask for frames 0-26, PMC will happily accept that and you’ll end up with three mics processed with 27 frames and one with 24. RBMC would still complain in this case.

– Harris

Lowest frame count was 80, but I’ll double check.

edit: Checked. Looped through all micrographs, lowest framecount was 80, highest was 132. I sliced out the first 80 frames into a new set, moved the old set and replaced then with the 80 frame set, and a new RBMC error appears:

All filenames are identical to original variable frame count stack.

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 129, in cryosparc_master.cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py", line 242, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
  File "/home/cryosparcer/bin/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py", line 286, in append_many
    assert len(all_uids) == len(n.unique(all_uids)), "Cannot append datasets that contain the same UIDs."
AssertionError: Cannot append datasets that contain the same UIDs.

This is with three volumes input to RBMC; if I set just one, it’s back to:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 129, in cryosparc_master.cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_reference_motion.py", line 302, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_reference_motion.run_reference_motion_correction
AssertionError: All movies must have the same number of frames

So I presume Patch Motion is writing total frame count out in some of its output, even if a maximum number of frames are set during Patch Motion Correction.

I’ve got an idea I want to check. Will update again when I know one way or t’other.

OK, that’s not it. Guess I’ll redo Patch Motion with the 80-frame pre-processed movies. Third time lucky…?

Hi @rbs_sci, my apologies for the delayed response. I assume you’ve got this worked out by now, but I can add some comments on a couple of points.

First, I can confirm that in recent versions of CryoSPARC, patch motion jobs write out the actual frame count they saw when processing the file, even if import movies got it wrong (usually due to skip header check).

Second, and you probably know this but it’s worth stating for the benefit of others who may read this thread, CS decides whether two particles/micrographs/movies are the same UID (not the same filename or any other such property). UIDs are randomly assigned when the object is created (so at import time or picking time for movies/particles respectively). UID-matching allows you to do things like run two parallel refinements, and take for example the poses from one and the refined CTF fits from another for a downstream job (probably a poor example but you get the point). But yes, CS won’t let you double-connect duplicates of the same movies/particles to the same job, by design.

–Harris