CryoSPARC to Relion Particle Re-extraction

Poko18 · November 23, 2023, 9:10am

Hello,

While there have been several discussions on similar topics, I couldn’t find any recent solutions to my specific issue.

I used Cryosparc to do import-2D classification and subsequently aimed to re-extract particles in Relion on my local machine.
I used csparc2star.py to prepare .star file. Since extraction job also need micrographs.star, I performed motioncorr and ctffind in Relion, adjusted the paths, and initiated the extraction process.

While the extraction itself was successful, I encountered a peculiar problem – the resulting 2D classes appeared suboptimal, and I obtained fewer particles compared to the scenario where I uploaded particles from CryoSPARC and performed 2D classification. I even employed disparticle.py, and the picks seemed accurate.

2 questions:
1.) Is there a way of generating both micrographs.star and particles.star files when converting .cs files? In that way I could just do re-extraction in relion without doing motioncorr and ctffind again

2.) Does anyone have any experience with this? Why would re-extraction of the same particles in relion yield worse 2D classes?

leetleyang · November 23, 2023, 9:57am

Hi,

Using csparc2star.py with the --movies flag can generate a corrected_micrographs.star file as part of its process. See Daniel’s write-up for more information.

For clarity, is this subsequent 2D classification performed in RELION or in cryoSPARC? Is it possible to rationalise the differences you see by the points Daniel has collated towards the bottom of this page?

Cheers,
Yang

Poko18 · November 24, 2023, 8:42am

Hi,

thanks for the quick answer!

I tried it adding --movies flag, but am getting an “MicrographOriginalPixelSize” error.

Traceback (most recent call last):
  File "/home/perun-d12/anaconda3/envs/pyem2/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3653, in get_loc
    return self._engine.get_loc(casted_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'rlnMicrographOriginalPixelSize'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/media/longstorage/Tadej/cryocloud/datasets/enc_raw/cs/csparc2star.py", line 179, in <module>
    sys.exit(main(parser.parse_args()))
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/longstorage/Tadej/cryocloud/datasets/enc_raw/cs/csparc2star.py", line 59, in main
    data_general = metadata.cryosparc_2_cs_movie_parameters(cs, passthroughs=pts, trajdir=trajdir, path=args.micrograph_path)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/longstorage/Tadej/cryocloud/cs2star/pyem/pyem/metadata/cryosparc2.py", line 224, in cryosparc_2_cs_movie_parameters
    data_general[star.Relion.MICROGRAPHPIXELSIZE] / data_general[star.Relion.MICROGRAPHORIGINALPIXELSIZE]
                                                    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/perun-d12/anaconda3/envs/pyem2/lib/python3.11/site-packages/pandas/core/frame.py", line 3761, in __getitem__
    indexer = self.columns.get_loc(key)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/perun-d12/anaconda3/envs/pyem2/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3655, in get_loc
    raise KeyError(key) from err
KeyError: 'rlnMicrographOriginalPixelSize'

For the second question. I’m doing both 2D classifications in RELION. One is with particles extracted in CryoSparc, and for the second one, I re-extract the same particles in Relion. In theory, extracted particles should be the same, however, I am getting very different results.

2D classification of particles from Cryosparc:

2D classification of particles from Relion (same particle locations!):

Not sure what seems to be the problem. Thanks for all the help!

DanielAsarnow · November 24, 2023, 7:40pm

The missing pixel size error is because .cs files with that information for the exposures wasn’t provided. You’ll need another passthrough or something.

However, if you just want the micrographs file and don’t care about polishing in Relion with the cryoSPARC initial motion estimates, instead of using --movies you can just run star.py --to-micrographs on your particles .star file.

The disparticle.py image does look good. We expect that if you use the same micrographs in Relion and cryoSPARC then csparc2star.py coordinates are correct by default, but if you use micrographs from Patch Motion in CS and then in Relion use micrographs from a re-run with MotionCor3, RelionCorr, or WARP, then you need to use --inverty. Also, if you export particles and then import back to CS and link the original CS exposures, the export also needs --inverty. This is because CS will invert the coordinates in Y on import, so --inverty is needed when you do cycles of export and import in CS.

Poko18 · November 24, 2023, 11:01pm

Hi Daniel, thank you for your help!

I managed to fix the pixel size error by creating a curate exposure job and incorporating its .cs files. The solution proved effective and resolved the issue.

Regarding the disparticle.py script, it seemed to function well even with converted.star file without the --inverty flag. However, upon testing with the flag, particle picks with disparticle.py looked wrong, yet it finally yielded good 2D classes.

So if I use Full-frame Motion Correction or MotionCor2 in CS, I don’t need to invert? Only for the Patch Motion?
For the second test, I performed motioncorr in relion, patch ctf, picking and particle extraction in CS, and then import particles (particles.star) back to relion (and link them to micrographs). For that I also had to use --inverty flag, to get good 2D classes.

DanielAsarnow · November 24, 2023, 11:41pm

The intention with disparticle.py was to accurately show where relion will extract, so I’ll double check that. Good 2D classes means the coordinates are right though.

I don’t know what unblur does, but I guess it probably works like cryoSPARC (not flipping the movie). MotionCor2, RelionCorr, and WARP all flip the movie (check it out with 3dmod or e2display.py or something).

Poko18 · November 26, 2023, 7:39pm

I might be doing something wrong, but in my case, whenever I run disparticle.py and get good particle picks, and then proceed with re-extraction and 2D classification → I get bad 2D classes. And of course vice versa, when I use --inverty flag.

Example of how disparticle.py looks like, but yields good 2D classes:

Anyway, everything works fine if I use --inverty flag when converting, Thanks for the help, and I hope this information proves useful to others!