Hello, I have a problem using on the fly Defocus- and Global CTF refinement: I have 4 relatively small datasets, which I try to merge in order to increase the number of particles for the final refinements. They were collected in the same microscope over the period of two months, same protein prep, some are on quantifoil some ultraufoil, some with detergent, but pixel size, dose, etc were all the same (I think total number of frames may differ slightly between them). They were imported and went separetely through preprocessing, picking, 2D classification and homogeneous refinement, with CTF refinement. Each show the same single 3D class, and give a reasonable resolution for the 30-70k particles used in the end (3.5-3.6 NU refinement in C1). After thiw, when merged, initially they show a very modest increase in resolution (using NUrefinement). When trying to continue refinement applying on the fly Defocus- and Global CTF refinement, it fails with this message: “Inconsistent values detected in input exposure groups. Please run homogeneous or non-uniform refinement without global CTF refinement enabled, then reset and refine the CTF parameters separately using the standalone Global CTF Refinement job.“ . When I follow this recommendation, resetting Tilt and Trefoil and either re-fitting them OR not, resolution seems to get worse afterwards, when trying to repeat homogeneous refinement with on the fly Defocus- and Global CTF refinement. From what I’ve read on the forum I understand this may be an issue with the groups from each dataset and the way I’ve merged them, after each was processed separately. But I fail to see how I can revert this. Any suggestions?
Hi @cmau,
Even if the data were collected on the same microscope, higher-order aberrations (beam tilt, trefoil, etc.) drift over time and can differ between grids/conditions. After merging datasets that were collected and processed separately, cryoSPARC sometimes ends up with inconsistent values within the same exposure group (due to the differences in aberrations on the different collection days), which triggers the “inconsistent exposure groups” error when Global CTF refinement is enabled.
Global CTF refinement in cryoSPARC fits these aberrations per exposure group, so the fix is to reassign exposure groups on the merged particle set (e.g., one group per dataset/session/grid) using Exposure Group Utilities. Mixing heterogeneous datasets in one group can also explain why resetting/refitting tilt/trefoil made the resolution worse (it drives the refinement toward a compromise that’s suboptimal for each subset).
@OleUns thank you for your answer. As far as I could tell (might be wrong though) particles still seem to be correctly discriminated by their exposure groups. I was expecting cryosparc would then correctly refine per-group CTF parameters for each group independently of the others. This is not actually how it works?
In that case, supposing they all do represent a single 3D class, could I use the best 3D map obtained, and separate each group of particles coming from a different dataset, refine their parameters without the other groups of particles present, and perhaps afterwards try to merge them, without ever again refining per-group parameters?
When trying to continue refinement applying on the fly Defocus- and Global CTF refinement, it fails with this message: “Inconsistent values detected in input exposure groups. Please run homogeneous or non-uniform refinement without global CTF refinement enabled, then reset and refine the CTF parameters separately using the standalone Global CTF Refinement job.“ .
How did you merge them? This error indicates that there is variation of (for example) beam tilt within a single exposure group, so exposure groups must have become merged at some point.
Agree with @olibclarke. Remember that the different datasets were originally assigned exposure group IDs - merging them will not change the IDs of them so that they are still independent. It will just merge exposure group 0 from dataset 1 with exposure group 0 from dataset 2, and so on.
If you did any Global CTF refinement at all, this will nearly certainly result in one exposure group having particles with two different beam tilts, magnification anisotropies (etc) internally of the one group - hence the error.
You can either reset all CTF parameters to default (toggles in the job) (not recommended!) or change the exposure group assignments of the second dataset so that they IDs do not overlap.
To revert, run an exposure groups job on the first dataset and split how you wish (e.g. into 9 exposure groups) starting at ID 0 (groups 0-8). Then run a second job on the second dataset, setting the starting ID at 9 (for groups 9-17). Then feed the resulting particle groups to a single homogeneous/non-uniform refinement, then continue Global CTF refinement from there.