Thank you for your reply! I find these csparc2star.py options are useful. Specifically, the --swapxy is always needed to account for the array order difference, the -inverty is useful when the motion correction is done with the cryosparc’s own implementation of motioncorrection (love it BTW), which produces micrographs y-flipped with respect to these from motioncorr2/Relion (image below).
While I agree that the convention for particle poses are similar between cryosparc and relion and that particles transfer between these two packages works well in general, I suspect that the y-flipped micrographs should also give y-flipped particles, which may introduce some minute but non-trivial differences. I will do some testing and get back. Any suggestions and ideas are welcomed.
The difference would not be minute; you would not be able to get a sensible reconstruction at all if the particle origins were inverted on one axis (try it). The programs are summing out aligned movie frames in a different pixel order, and use different conventions for specifying the center of a particle in a micrograph, but they all treat extracted particle images and their aligned shifts in the same way. (Modulo the transformation inversion I mentioned previously; see my code in par2star.py).
To clarify, the above means that if you take a .star from csparc2star.py with the right options, you can use that as input to Relion extract and then relion_reconstruct and get the right structure, without having to run a new refinement.
After looking at the code cryosparc code of partice extraction, I agree that there is no need to flip the axes for the shiftx/shifty. Thank you for your clarification!
However, there still seems to be an issue when carrying out re-extraction in relion using exported cryosparc partices. Since the micrographs are y-flipped with respect to each other (cryosparc motion correction vs. motioncor2), the re-extracted particles would be y-flipped as well even though the centering is accurate. That y-flip or mirroring along the x-axis would probably require modifications of the Euler angles (my guess is (180+φ, 180-θ, -ψ) according to the 3DEM convention).
cryosparc particles (left) vs re-extracted particles(right)
@sunch OK, you convinced me. If you have to use --invertx/--inverty to match different physical micrographs, then the alignment parameters won’t be correct after re-extraction in the other program.
I think if the shift for the flipped axis is also flipped, then reconstruction will work without modifying the angles, but the reconstructed volume will itself be flipped. Applying a 180 rotation to the skew angle to align the reconstructions is no problem (it would actually be done to the cryoSPARC Rodriguez vector before conversion to Eulers, though).
However, I think the only reason to do this is a rare case where Relion global searches fail completely, so you need to do an Refine3D starting from local searches around the cryoSPARC angles. You pretty much need a Refine3D and PostProcess to run Bayesian Polishing, I don’t think there’s any way to avoid that.
I’ll update this post with a transformation matrix you can try with star.py to apply that flip to the particles and see if the reconstruction is working.
Thank you Daniel! Looking forward to trying out the star.py script.
BTW, I have used awk to change the Euler angles, which substantially improved the local Refine3D job with reextracted particles (with the --inverty option).
you don’t strictly need a Relion Refine3D job for bayesian polishing - postprocessing from half maps generated by relion_reconstruct or quite often just postprocessing starting from the cryosparc half maps often works fine. I guess in this case, relion_reconstruct will do the trick (so the reconstructed volume matches the particles), while using the cryosparc half maps probably will not.
Hi Dan - what was the transformation matrix to use in this case? I am encountering the same issue (re inverty)
The “180+φ, 180-θ, -ψ” transformation mentioned by @sunch seemed to work. Not sure how to do with star.py, but with awk:
awk -v s=180 '{print $1, $2, $3, $4, s+$5, s-$6, -$7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17}' particles_in.star > particles_out.star
Where columns 5-7 are the Euler angles, and the star file has 17 columns in total. Need to replace the header afterwards. There is probably a more elegant way of doing this, but this seems to do the trick!
Cheers
Oli
EDIT:
Hmm this still isn’t quite working. Relion reconstruct of the half maps from the transformed star file gives a reconstruction with a resolution of 5.7 Å, whereas the expected value is ~3Å with the same mask. With the non-transformed star file it is much worse - ~16Å. But normally, relion_reconstruct from a star file imported from cryosparc gives comparable values to those originally determined in csparc, in my experience, so still feels like there is something missing here.
EDIT 2:
Ok, so the easiest way to do this I think is to:
- convert particle.cs to star using
csparc2star.py(with--swapxy&/or--invertyas needed) - Extract in relion from relion motioncorrected mics (try a subset first and confirm particles are well centered)
- Import back into cryosparc, and associate with relion motion corrected mics
- Re-run global refinement.
- Export back to relion.
- Do all the usual things - symlink cryosparc job dir for particles, symlink mrc stacks to mrcs, adjust paths and opticsGroupName in star file as needed
- run relion_reconstruct for two halves
- run relion_postprocess using refinement mask
- CTFrefine if needed (and another round of relion_reconstruct if so)
- Polish.
After this, using relion_reconstruct gives the expected resolution and everything seems to proceed smoothly.
Hey everyone,
I know the thread is very old but I have run into an issue while trying to run Bayesian Polishing on the cryosparc to relion particle set. I had converted my particle set using csparc2star.py (without using the --inverty command) and had run few rounds of 3D classification, Refinements and CTFrefinement as well. When I tried to do Bayesian Polishing with the Relion motion corrected micrographs, it worsened my resolution and so I tested re-extraction in Relion and that showed that the particle coordinates were not correct. When I tested re-extraction with a newly converted .star file from .cs with the --inverty option the particle extraction was proper. Hence, is there an equivalent of an --inverty command which I can apply on the .star file that I have processed so for.
Thanks.
Export the coordinates correctly and then use star.py --copy-micrograph-coordinates to copy them to your current particle file.
Hey,
I managed to export the coordinates properly and I also tested them using a small re-extraction job. But when I am trying to run Bayesian polishing on it, the resolution of the map is worsening. I have attached a screenshot of my training parameters and the polishing job. If someone could have a look and let me know what I can do to get rid of this error.
Thanks
@subhrob15 Unfortunately you will have to make that a complete re-extraction job, re-refine those particles, and then use the results of that refinement to run the polishing, in order to get a good result. Still you are almost there!
Hey I tried what you had recommended but somehow at the end I am getting the following error in the screenshot. Its a bit odd since before the log file was atleast written before.
Please let me know what could be done about this?
Looks like something is wrong with your ghostscript install?
That error doesn’t mean ghostscript has a problem specifically (although it can do) - I’ve seen it myself when gs was writing a logfile to a drive which was 100% full due to another users’ process dumping terabytes of checkpoint files.
It can also mean that some of the files gs expects in the .lst file are missing.
It’s also possible to get this error if the drive is mounted read only because the filesystem freaked out (saw this happen with a kernel regression).
Hey,
I do not have much knowledge about the backend operations so I will ask the IT person in my lab to have a look at this. But from what you said I could gather that I would need to change permissions of the folder. Though it is a bit odd since previously when I ran this in the same folder, I did not get this issue.
I don’t think it’s permissions related. From my experience it’s more free space related (or, worst case, emergency remounts due to hardware error).
Okay, do you think this would somehow still affect the polishing job since even after this, I ran a refinement job and the resolution worsened.
My comment was only regarding the ghostscript error.
Just an update for everyone looking at this thread, For me Bayesian polishing worked and the map quality also improved when I ran it on relion 3.1. I made a different directory and changed the paths to the star files of the motioncorrected and the refinement file to indicate absolute paths and it worked.
Also thanks to @rbs_sci and @DanielAsarnow for guiding me.