I was wondering if any advanced cryoSPARC user could suggest any tips or warnings for re-using particles.blob and particles.ctf subcomponents.
For example, as @olibclarke kindly explained here, one can replace the particles.blob subcomponent of a stack of binned particles to avoid having to re-extract particles prior to Non-Uniform Refinement.
I can see this could be useful for some more complex cases but I’m worried I might be misusing this approach. For example, I have a somewhat convoluted pipeline for processing a C3 pseudo-symmetric homotrimer. After NU Refinement, I usually use the Volume Alignment Tools to align my volume and particles to the C3 symmetry axis followed by local refinements on the individual protomers, as well as symmetry expansion followed by local refinement, 3D classification, and further downstream local refinements on the individual classes.
What if, say, I later wanted to re-run my NU Refinement after Global/Local CTF Refinement and then re-run my various downstream refinements using the newly refined particles (to avoid having to re-run 3D classification etc.)? Can I/should I replace the particles.blob and particles.ctf of the input to Local Refinement with those from the new NU refinement? (N.B. I presumably want to retain the original particles.alignments3D as the starting volume is now aligned to the symmetry axis). Maybe I can run a new symmetry expansion job to provide CTF-refined particles for the jobs downstream of the original symmetry expansion? Or does each symmetry expansion job create entirely new particle IDs?
Hi @LouisWilson! The particles.blob result refers to the images on the disk. So the only thing you need to avoid is using the particles.blob result from one Extract Micrographs with refinement / CTF results that were calculated using a different Extract Micrographs job.
Hi @rwaldo - thanks very much for your reply! Thanks also for the documentation - I was looking for this but couldn’t find it.
I’m a bit confused though - you mean that we shouldn’t mix and match a blob result derived from one particle extraction with ctf / alignments3D results from refinements derived from another particle extraction?
Does this mean that what I have done below is ‘wrong’? I extracted with binning (J18) and after several rounds of Het Ref, re-extracted a subset of these particles in a new extraction job without binning (J22, cloned from J18). Then, I did a further Het Ref still on the binned particles (J26), before combining everything from the best class of J26 (apart from the blob) with the blob from J22 into the input of a NU Refinement (J29). Therefore, the blob comes from the J22 extraction, while everything else (ctf, alignments3D, and Passthrough) is from the J18 extraction. Nevertheless, this seemed to work out fine (unless I’m missing something?). Or you meant more that we shouldn’t combine them if the particle sets are entirely different (i.e. different locations)?
I should have said that using blobs and other results from different extractions may cause a problem. The main issue is that if you re-extract particles with the aligned shifts (which is the default behavior and almost always what you want), then the poses from other extractions for the “same particles” will be slightly wrong, since the images themselves (the blobs) are different.
In practice, though, this only matters if you immediately perform a job which re-uses existing poses (Local Refinement, 3D Classification) without performing a global refinement in between. So in your case, there’s no problem, because both Heterogeneous Refinement and Non-Uniform Refinement discard existing poses.
Ahah! Thanks for clarifying, this is very helpful!
Then if I can ask one more question - how does the symmetry expansion data structure work in this context? If I have a refinement (e.g. J39 in the example below) whose particles I went on to duplicate with symmetry expansion (followed by 3D classification and local refinements), and I then go back and apply CTF refinement to the original particles, is there some way to carry the ctf result from the CTF-refined particles through to the refinements downstream of symmetry expansion without re-doing the 3D classification etc? I already tried running a new symmetry expansion job on the CTF-refined particles and forwarded their ctf and blob results into one of the older local refinements (e.g. a clone of J60), but it seems like it didn’t work as expected (I ended up with fewer particles in the refinement than there should have been). I guess particles created by the second symmetry expansion have new IDs and hence these don’t match up with the previous particle subcomponents?
Just bumping this in case anyone can help! I have a lot of projects like this where I’d like to apply CTF refinement to pre-existing local refinements without re-doing 3D classification. In some cases I can’t get the same results when I do a new 3D classification so I’m a bit stuck.
Hi, sorry for the delay @LouisWilson. The way the low level results work under the hood is that particles’ metadata is combined using their UIDs. As you realized, symmetry expansion creates new particles with new UIDs, so you cannot currently use the low-level results to combine those.
If you must use existing, expanded results with new CTF parameters from unexpanded particles, you can use cryosparc-tools. You can match the source particles’uid field to the destination particles’sym_expand/src_uid field.
