I am trying to get ahead on processing some data as it is slowly transferring from the microscopy core. What I’d like to do is import all .tif files from a directory (say,
J1) and begin motion correction (
J2). Then, say an hour or two later, launch
J3 which is a clone of
J1. Then take the difference of
J4) and input that into a new motion correction job on
J5) to avoid duplicating work.
I thought that using the
exposure sets job would let me do this, but taking the intersection and looking at
B_minus_A, it seems to me that the
exposure sets job operates on
exposure objects, not the paths of those
B_minus_A have the size of
B respectively, even though they both contain several of the same micrographs (precisely,
A is a superset of
B, so I would expect
B_minus_A to have zero
I searched some for an answer to this question but didn’t find anything, I apologize if I missed something.
This is actually one of the main use cases we built cryoSPARC Live for. In a cryoSPARC Live session, you’re able to specify a directory that cryoSPARC should watch to find new movies.
As new exposures are found, a preprocessing worker will pick them up and complete Patch Motion Correction, CTF Estimation, blob/template picking and particle extraction all in one go. You can then continue processing in the main cryoSPARC application, or continue onto streaming 2D Classification and 3D Refinement.
In the meantime, if you’d still like to use the Exposure Sets tool to only process the remainder of movies, I’ve modified the job to use the movie’s path rather than the unique identifier to complete the operations.
You can get the new job (compatible with cryoSPARC v3.0+) by downloading it here:
Make a backup of the existing run_sets.py and copy the new file on the master node into the folder:
You can then re-run the job and it will use the new code.
We will release the option to choose the paths field rather than the unique identifier in the next version of cryoSPARC. Until then, the new job will only use paths when using the “intersection” mode in both the Particle Sets job and Exposure sets job.
Wow thank you! I was halfway through trying to do this with bash scripting, this is a much better solution given my abysmal skill in bash.
Great! Let me know how the modified job works out for you.
Note you can manipulate .cs files pretty easily using python. We have a tutorial with an example on how to do this here:
Your request is now a feature in the Exposure Sets Tool and Particle Sets Tool jobs in v3.3.0 (released December 1, 2021) available via the “Field to Intersect” parameter: