hi all, looking for any input on how to get local motion to work better. Every time I try to use it my map ends up looking worse, which makes no sense to me.
Movies aligned in cryosaprc, CTF estimation done, extract, do refinement and end up with a ~4A structure which looks pretty good. I then set up local motion (with parameters below) using as input the particles from the 4A reconstruction and the output from the full motion frame alignment. Then I clone the job that gave the 4A reconstruction, except now I use the particles output from the local motion job and I end up with a 7A reconstruction that is noticeably uglier. I’m clearly doing something wrong because this makes no sense.
notes on the job setup:
this is a filament protein. The box is ~550A diameter. I initially extract 1024px while binning to 512px.
local motion box size is set to 1024px and then I downsample to 512px before doing the recon again.
local motion multi w/ 2 gpus, all settings are default.
tried with smaller box size (640 downsample to 320) to get a shorter filament segment but it also failed.
this has happened with multiple datasets and proteins, so I must really be doing something wrong because I can’t ever get local motion to improve the map.
based on what I’m reading in other threads (particularly this one) it may not be worth doing local motion at all but rather patch motion. If I can’t sort out the local motion (kinda unfortunate really) then I will just do patch motion, re-extract, and re-refine to see if there’s any improvement.
Thanks for reporting. Going from 4A to 7A after local motion definitely doesn’t seem right.
Can you confirm that this is what you did:
full frame motion, CTF, extraction at 1024 box downsampling to 512, refinement to 4A
particles from refinement -> local motion (with extraction box size 1024px)
a separate downsample job to downsample particels from 1024 -> 512 px
refinement with downsampled particles to 7A
There is a known bug in the downsample particles job that may be causing this issue. That bug does not affect downsampling that is done during extraction (i.e. in step 1) but may be causing the issue in step 3. We are going to provide a patch for this. In the meantime, can you try patch motion as you said? Generally until you’re at 2.XÅ with a rigid particle, patch motion and local motion will give very similar results.
yes you are correct in those were the steps. It was quite confusing why/how things could get so drastically worse. If that was indeed the bug then perhaps if I tried to do the refinement on the non-downsampled particles (step 3) it would have worked. Unfortunately I’m trying to save space on our server so I had to delete those jobs, but I could try to run it again and skip step 3 to see if that’s the issue.
anyway thank for your response, I have indeed done the patch CTF and so far things are looking good. Appreciated!
just to update here… the patch motion is working very well. Thank you.
When I compare results from full-frame motion vs patch motion the difference is unbelievable. The latter makes such a drastic improvement! Is full-frame motion not deprecated now and should be removed? Or alternatively you may consider putting patch motion at the top of the list in the GUI.
I am wondering how it is so much better than full-frame. Does full-frame not do ANY patch correction at all, so it is simply one trajectory for all particles in the micrograph?
My understanding is that MotionCorr2 also does patches: 512px by default, which on super-resolution data is 15 x 15 patches (compare to 10 x 10 patches for patch motion in cryosparc). Yet for data motion-corrected with MotionCorr2 seems to consistently be helped by particle polishing. Given your findings that maps processed with patch motion here are rarely helped by per-particle, does this suggest your patch alignment algorithm is simply better than motioncorr2 default alignment?
sorry for all the questions, just trying to figure out whether I should be processing my old data with cryosparc patch motion alignment…
Does full-frame not do ANY patch correction at all, so it is simply one trajectory for all particles in the micrograph?
That’s right, I think it’s using unblur.
My understanding is that MotionCorr2 also does patches: 512px by default
In MotionCor2, you specify -Patch X Y O where X is the X patch count, Y is Y patch count, and O is the overlap in percent. For the K3, -Patch 7 5 40 is optimal, but the overlap can be reduced significantly for speed. The patches are fit to a polynomial model of the ice deformation that can account for any sample doming, when the corrected micrograph is generated.
Given your findings that maps processed with patch motion here are rarely helped by per-particle, does this suggest your patch alignment algorithm is simply better than motioncorr2 default alignment?
Another alternative is Bayesian polishing being better than align_parts_lmbfgs…but the Bayesian polishing paper showed that for many datasets there’s not much advantage to polishing. Just high-res samples in very thin ice. So it’s hard to say what the reason is, MotionCor2 vs. csparc patch vs. lmbfgs vs. BPP.
I am able to process my filament as single particle, so no helical needed. Others in our lab do use cryosparc though in the filament processing pipeline, but mainly at the 2D stage then export to relion for explicit helical processing. The 2D averages tend to be better in cryosparc or cisTEM so we use that for figuring out helical symmetry with power spectra.
So I think typically the initial picks would be done in relion or crYOLO helical picker. Then imported into cryosparc, which I believe will maintain the helical metadata in the passthrough file. You can then use csparc2star.py to export afterwards and continue processing in relion.
Not sure if cryosparc has intentions to incorporate helical processing, would be cool!
Thanks. I only go from Relion to Cryosparc not the other way back. Need to play with the conversion here since I need classification without alignment from relion. Also, in my case, bayesian polishing definitely improves the resolution more than straight patch motion from cryosparc.
@DanielAsarnow thanks for the response, I really appreciate the detail.
I would like to test this out by running Bayesian polishing on my cryosparc map. However as usual I am having difficulty exporting. Not sure if this feature is deliberately omitted from cryosparc so users are forced to stay within. I really feel it’s in the best interest of science and progress that users be given easy flexibility to use the different features available in different software packages. Anyways, I digress.
I imported all of my data into cryosparc as different optics groups; the micrographs are from 2 datasets. I process and get to a refinement. Now I try to export final particles.cs file and get the error below. I grabbed the latest csparc2star.py off your github.
**strong text**python csparc.py cryosparc_P79_J485_007_particles.cs outputparticles.star**strong text**
Traceback (most recent call last):
File "csparc.py", line 114, in <module>
sys.exit(main(parser.parse_args()))
File "csparc.py", line 42, in main
df = metadata.parse_cryosparc_2_cs(cs, passthroughs=args.input[1:], minphic=args.minphic, boxsize=args.boxsize, swapxy=args.swapxy)
TypeError: parse_cryosparc_2_cs() got an unexpected keyword argument 'passthroughs'
I tried including the passthrough file as the auxilliary argument as described on your github but got the same error.
last, I followed your final recommendation to export the refinement from within cryosparc then try to run csparc2star on that exported particle file. got the error: A passthrough file may be required (check inside the cryoSPARC 2+ job directory)
It looks like you have a pretty old, possibly modified version of the program. Please update to the latest master branch version (release is behind on Relion3.1 features right now).
csparc2star.py particles.cs passthrough.cs output.star should be all you need in most cases. The most recent version yields Relion3.1 star files unless --relion2 is passed, so the optics groups you have should be preserved.
FWIW Relion also makes this difficult with their changes to the format, some of which are long overdue, some I disagree with. But regardless, my code can now deal with arbitrary, multi-table star files so keeping things going should be easier if they make any other drastic changes.
I went over and grabbed the latest version from link below. I am running on our cluster, loaded python/2.7.13 and copied the code from the link below into a new python file. https://github.com/asarnow/pyem/blob/master/csparc2star.py
I think last time I had issues like this you responded saying the issue was due to the data coming from multiple datasets. I’m guessing this has been addressed in the latest csparc2star update though?
I suspect that issue was caused by folks re-importing particles so they weren’t actually the “same” particles in cryoSPARC. You need the latest version of all the library modules as well, which is why my installation instructions have you install pyem using egg-links, so all you need to do is update your clone of the git repository.
Oh, I also should point out that Python 2 was end-of-lifed in December and I’m only writing for Python 3 now. I don’t think it will run in Python 2 anymore.
Hi Daniel.
I want to re-import the cryosparc particles file after refinement back to relion for polishing & classification without alignment. So, my main need is the coordinates & also alignment/CTF/beam tilt info from Cryosparc. However, when I exported using your csparc2star.py, the coordinate is obviously wrong.
In my case, when after I imported particles from Relion to cryosparc, I have to tick Flip y for extraction in cryosparc to get the right particles. I saw there is an option for --swapxy but not flip y in this case. Is this something that you can implement?
update: I was able to update the libraries and it works. I have a nice star file now. thanks.
However I wanted to report that our IT guy couldn’t update it on our server. He suggested I report it to you:
" updated to 0.5 HOWEVER setup.py file did not update version=“0.4”->0.5 So I guess it will still show up as 0.4 Maybe you want to report this?"
We use 7x5 & 0-40% overlap for the K3 (seems like 40% may be optimal, but it’s definitely overkill). For K2, we used 5x5 or 7x7. The K3 is more elongated, so the extra patches seem important. The K2 and Falcon are nearly or completely square.
The size of the patches can be related to the size of the particles - for a large, easy to see particle you can use a smaller patch. If you have a smaller or low contrast particle, you may get a better result with bigger patches. Any residual motion blur will act as an effective b-factor that can be overcome by averaging. I think 5-10 patches are all reasonable numbers that usually let something other than motion become resolution-limiting.
Just for anyone getting an unexpectedly-worse resolution when you use the Downsample Particles job somewhere in your processing, this issue is now fixed in the latest cryoSPARC v2.15.