Exposure group utilities

I have got good results from relion 3.1 ctf refinement correcting high-order aberrations and anisotropic magnification and I want to try the same thing in cryosparc. However, currently I am struggling to split exposure groups based on the micrograph number in ‘blob/path’. The dataset was collected by serialEM with the micrograph names having the format of mic_???.tif (??? is a numer with 4 digits), and the data collection was done by 3x3 beam tilt. In relion, what I did was that I wrote a script to assign optical groups based on the modulus of ??? (micrograph number) divided by 9. However, I didn’t find a clear tutorial of the exposure group utilities in cryosparc regarding how to achieve a similar function.

Meanwhile, in cryosparc, I have also tried to manually change the ‘ctf/exp_group_id’ values of all particles with a script based on the same idea as I did for relion. However, if I use this set of particles for a global CTF refinement job, the output still has only 1 exposure group. Does cryosparc reset everything before ctf refinement?

I appreciate any input!

Chen

Hi @chenzhao,

Currently, we have a brief tutorial on how exposure groups are identified/created in cryoSPARC here:

I have also documented every parameter available in the Exposure Groups Utility Job here:

If the exposure group utilities job doesn’t suit your needs, the method you mentioned (editing the .cs file directly) is definitely the best way. A short tutorial on how to do that is included in the first link I posted.

Related to this @stephan, does/will cryosparc support import of optics groups or rlnBeamTiltClass columns from star files? This would make import of exposure groups defined outside cryoSPARC more straightforward

Cheers
Oli

Hi @olibclarke,

That’d be nice! Is it possible if you can point me towards an example of this column being used in a .star file? Or even if theres any documentation on how to use/set it?

Hi @stephan - sure - I can DM you an example star file for both the relion 3 and relion 3.1 cases if that would be helpful? The implementation of optics groups in relion 3.1 is described in section 1.1. of the tutorial: ftp://ftp.mrc-lmb.cam.ac.uk/pub/scheres/relion31_tutorial.pdf

1 Like

Hi @stephan,

Thank you for your reply and sorry that I have missed the tutorials. However, I am still a little bit confused now. After roughly going through all methods, I still prefer to modify the .cs file directly. However, as I mentioned in the post, I have already tried to change the ‘ctf/exp_group_id’ in .cs, but even so after global CTF refinement I still get only one exposure group. In the tutorial you sent, it looks like ‘ctf/exp_group_id’ is for micrograph dataset and ‘location/exp_group_id’ is for particle dataset. Do I have the wrong metafile label? (Should I use ‘location/exp_group_id’ instead?) If so, how should I get the correct label?

Thanks a lot,
Chen

Hi @chenzhao,

So ctf/exp_group_id is the “global” exposure group indicator. mscope_params/exp_group_id is specific to exposures (movies, micrographs), and location/exp_group_id is specific to particles. You should ensure both ctf/exp_group_id and the dataset-specific field stay consistent.

Hi @stephan,

I am sorry but I do not have ‘location/exp_group_id’ in my .cs file. How should I import the particles to incorporate this field? I only have one dataset, but want 9 exposure groups within the same dataset.

Thanks a lot and sorry for my confusions.

Best,
Chen

Hi @chenzhao,

Not to worry if you don’t have the field. The location slot is only added if the particle locations were included in the .star file or if the particles themselves were extracted using cryoSPARC. The ctf/exp_group_id field should be the proper field to change. If you’d like to post a bit more details about your micrograph naming convention, I’d be happy to help.

Also, please note that the Exposure Groups Utility job has an “info_only” mode that will let you view the current dataset’s exposure group values easily (instead of using the 6-8th iteration of the CTF Refinement job to tell you!).

Hi @stephan,

Thanks a lot for pointing out the “info_only” option in Exposure Groups Utility job. Indeed when I ran this job I got only 1 exposure group, although in the .cs file I have 9 exposure group IDs. What I did was that I modified the particle.cs for the last iteration after refinement, but it looks like this action is not sufficient for cryosparc to recognize the exposure group IDs. What are the other things that I have to change?

Finally I have time to take another look at this problem and I realized that I should change the exposure group id in the passthrough particles instead of the refined particles. Thanks a lot for everyone’s help!

@chenzhao In SerialEM > Setup > Record > File options there is a checkbox for “add navigator item.” In future collections, you should check this box, then each of your images will have a name like Base_NavItem_0001 to Base_NavItem_0009. This makes things a lot easier. You might also rename the images you have according to the modulus to apply the existing Exposure Utilities, though it seems you’ve already figured out another way.

@DanielAsarnow Thanks a lot for the tip and currently I am just using the micrograph number because I have never ended data collection in a way such that it stopped in the middle of take the multiple shots based on the preset 3x3 matrix. So in this case, I just have the micrograph names as Base_0001.mrc to Base_0009.mrc. However even so I did not figure out how to use the exposure utility tools in cryosparc to parse out the numbering. Maybe I just missed that part in the tutorial that explains this…

If there are just underscore separated positions, with one of them being each of the 9 image shift locations, then you do this:

But change the split group index to the 0-based position of your image shift number.

Maybe one of the other SerialEM automatic naming features besides NavItem also works - did you include the time or anything else in your file name? Because otherwise the numbers should go up to the number of images taken, e.g. 900. I thought this is what you meant by setting the group by the micrograph number mod 9.

Hi @DanielAsarnow, thank you so much for your help, and this works! I was just being stupid and I thought that if I say split by ‘_’ I will still have ‘XXXX.mrc’ as the last field. I guess the program just ignored .mrc or .whatever by default… I shouldn’t have made so many assumptions and should just try it out!

Thanks a lot again!

Best,
Chen

Ah, I see. Actually it matches the group “index” as a string, so they just have to be unique. If the extension wasn’t cut you would end up with groups named “0001.mrc,” “0002.mrc” and so forth, but still correctly separated.

1 Like

Got it! I never thought about it in this way…