Importing Optic Groups from EPU AFIS

Hi guys,

For importing the Optics groups when the collection has been done with EPU AFIS strategy (https://forum.scilifelab.se/t/creating-optics-groups-from-epu-afis-data-and-more/122) first one should run the EPU_GROUP_AFIS tool (https://github.com/DustinMorado/EPU_group_AFIS)
The output is a .star file for using Relion/Scipion. Here I show how do I do to import the correct exposure group info in to Cryosparc.

Step 1 RUN EPU_AFIS

$python EPU_Group_AFIS.py --xml_dir '<PATH_TO_XML> ’ --apix 0.86 --ftype tiff --movie_dir '<PATH_TO_movies> ’ --output_fn '<PATH_TO_outputfolder> /movies.star' #--algorithm hac

Step 2 curate output

I enter the output file and eleminate the header then sort data for better comprehension runing in the terminal
$sort -k2,2 movies.star > movies.good.sorted`

Then I enter with NEDIT the sorted file. IS important to use Nedit as it has the posibility of select in columns (Ctrl+SHIFT)

nedit '<PATH_TO_outputfolder> /movies.good.sorted '

Then I change the text of EACH line to have the following example text you can do it selecting the text in colmuns (ctrl+shift+mouse click), then REPLACING the text with the replace tool to do all lines at the same time.

Example of one line:

micrographs.data[‘ctf/exp_group_id’] [micrographs.data[‘movie_blob/path’] == ‘J1/imported/FoilHole_4113390_Data_3092199_3092201_20201204_100228_fractions.tiff’] = 20
#In BOLD the formula text that you have to include in between the micrographs paths and the optic group
#In ITALICS the adres of the micrographs that you found using micrographs.data[‘movie_blob/path’] and the actual optic group.

Save the file with the name: ctf.py

Duplicate the ctf.py file and in the ducplicate change the marked field for all the lines:

Example of one line:

micrographs.data[‘mscope_params/exp_group_id’] [micrographs.data[‘movie_blob/path’] == ‘J1/imported/FoilHole_4113390_Data_3092199_3092201_20201204_100228_fractions.tiff’] = 20
#In BOLD the substitution

Save the file name: mscope_params.py

Step 3 run Pymol to change the optics group using the 2 files form step 2.

In a terminal open the file ‘*_passthrough_exposures_accepted.cs’ of the accepted exposures after the exposure curation job in PYTHON using the following:

$cryosparcm icli

Now we are in a Python terminal then I write teh following

$import numpy as n

$import sys

$from cryosparc_compute import dataset

$dataset_path = '<PATH_TO_JOB> ’ /*_passthrough_exposures_accepted.cs'

$micrographs = dataset.Dataset().from_file(dataset_path)

Then I check everything is correctly loaded.

$micrographs.data['movie_blob/path']

I run then the two files using

$exec(open('<PATH_TO_outputfolder> /ctf .py').read())
$exec(open('<PATH_TO_outputfolder> /mscope_params.py').read())

Then I check that they groups are changed in the Python terminal

$n.set_printoptions(threshold=sys.maxsize)
$micrographs.data['ctf/exp_group_id']
I shouold see a long matrix with the numbers of the neww optic groups

$micrographs.data['mscope_params/exp_group_id']

I shouold see a long matrix with the numbers of the neww optic groups

to save the changes in the ‘*_passthrough_exposures_accepted.cs’ type in the python terminal:

$micrographs.to_file(dataset_path) #particles.to_file(dataset_path)

I close the Python terminal

In cryoparc I use the job Import result group
Loading the .csg file that is related to the ‘<PATH_TO_JOB>/*_passthrough_exposures_accepted.cs’ that we modified using python.

This: ‘<PATH_TO_JOB>/*_passthrough_exposures_accepted.csg’

For checking In cryosparc I use the job exposure Group Utilities
input: result from importing
action: info_only

Work with this exposures.

3 Likes

Hi @PabloGallego Thank you for this useful post!

1 Like