Unusual empirical dose weights at high dose

[olibclarke]

So basically @RickBaker you have to split it up into multiple jobs:

Job 1: normal run, but end at dose weight estimation

Job 2: manually enter the hyperparameters from job 1, but reduce the spatial prior by 1/5 (or 1/10, or whatever - may be worth screening). End at dose weight estimation. This gives your modified dose weights (which will hopefully have less anomalous upweighting in the higher frequency range).

Job 3: Use the unaltered hyperparameters from job 1, but supply the dose weights from job 2 (drag to the hyperparameter slot). It will then use the manually entered priors, but also use the modified dose weights provided in the hyperparameter slot of the inputs. End this one at motion correct - this is where your polished particles are generated.

Hope that all makes sense!

[olibclarke]

Good idea but I’m not exactly sure how to do that - the reverse is easy, but I don’t know how to provide the Patch Motion dose weights to RBMC…

[RickBaker]

Thanks for the detail! I’ll give it a go!

[Indrajit]

I was wondering if apply trajectories would do the trick. I was going through the folders posts on this topic and Harris suggested this as a workaround in his November 23 post.

[rbs_sci]

Interesting. Here’s some quick (brute force, order of magnitude) results from our in-house apoferritin tutorial dataset (which I keep meaning to upload to EMPIAR…) used because it reliably achieves <2 Ang from 30 micrographs. RBMC calculation is fitting to 2.06 Ang (could be better but doing this quickly)…

Dataset is 41 frames, 50 e-/A^2 total dose. Originally EER but converted to TIFF for tutorials. Can achieve 1.8 Ang with NU refine and better optimisation of CTF parameters.

ORIGINAL FSC: (pre-RBMC)

Calculated hyperparameters (extensive search from full particle stack; 3,115 particles):
Spatial prior strength: 1.1985e-02
Spatial correlation distance: 3000
Acceleration prior strength: 4.2394e-03

Dose weights with passthrough of these parameters:

Note the penultimate frame overweighting.

Dose weights with manual input of these parameters: (sanity/consistency check)

Visually identical.

SPATIAL PRIOR STRENGTH
Spatial prior strength 10x increase:

Significant overweighting of penultimate and ultimate frame, appreciable overweighting from approximately frame 32.

Spatial prior strength 10x decrease:

Overweighting gone.

SPATIAL CORRELATION
Spatial correlation distance 10x increase:

Penultimate frame overweight gone.

Spatial correlation distance 10x decrease:

Significant increase in overweighting of penultimate and ultimate frame compared to increasing spatial correlation distance.

ACCELERATION PRIOR
Acceleration prior strength 10x increase:

Effectively unchanged from calculated. Slight overweighting of penultimate and ultimate frames.

Acceleration prior strength 10x decrease:

Significant increase in overweighting of penultimate and ultimate frame compared to increasing acceleration prior.

Particle motion correction currently running, then will run homogeneous refinement to see what final resolution results are. Will update accordingly. Currently appears that weakening spatial prior and strengthening spatial correlation might have the most positive impact on overweighting of late frames. Strengthening acceleration prior might also have a positive impact (or, at least, no negative impact). Let’s see the resolution results…

…

edit:

MOTION CORRECTION
Here’s the motion tracks for the first micrograph in each parameter set.

Dose weights with manual input of these parameters: (sanity/consistency check)

Spatial prior strength 10x increase:

Spatial prior strength 10x decrease:

Spatial correlation distance 10x increase:

Spatial correlation distance 10x decrease:

Acceleration prior strength 10x increase:

Acceleration prior strength 10x decrease:

Final resolutions:
Dose weights with manual input of these parameters: (sanity/consistency check)

Spatial prior strength 10x increase:

Spatial prior strength 10x decrease:

Spatial correlation distance 10x increase:

Spatial correlation distance 10x decrease:

Acceleration prior strength 10x increase:

Acceleration prior strength 10x decrease:

At least for this small dataset, there is not so big a final effect on resolution, except in decreasing spatial prior strength, if 0.06 Ang is significant.

Might try with a larger dataset as I have time as this appears inconclusive, at least compared to @olibclarke’s 0.2 Ang improvement at ~3 Ang… (or apoferritin is a bad thing to test this with, as it’s generally so robust…!)

[olibclarke]

Interesting! I would note that the later frame overweighting was not quite as obvious here, so maybe that is the source of the discrepancy?

[olibclarke]

Yep you’re right - I had forgotten about apply trajectories!

[rbs_sci]

Good point. Yes, that’s a possibility.

With my own data, I’ve never managed to get such severe overweighting in the late frames as others have demonstrated*. Point me at an EMPIAR dataset (not too large, please!) or if you’ve got a data subset you can share privately which will certainly show it?

* which makes me wonder if there is a detector aspect to this…?

[olibclarke]

The one I posted was from the HA trimer untilted dataset in EMPIAR - the one used in the recent CS case study on preferred orientation - EMPIAR-10096

I see it in almost every dataset I test, but almost all of these are K3/K2, so maybe that could be part of it?

[rbs_sci]

OK, cheers; I’ll use EMPIAR 10096 for another test.

Yes, 95-99% of the data I process is Falcon 4(i), and what K2/K3 data I have is either exotic, old or has other limitations.

447 movies is not so many, OK, I’ll download it tomorrow and give it a go!

[RickBaker]

Awesome! That worked.

For anyone else who didn’t realize this, the tokens you drag into jobs often have multiple components and can be expanded. I did not know this! I’m assuming there are other cases where you can do a more fine-grained selection of inputs from various jobs.

Fingers crossed that it leads to an increase in resolution

[olibclarke]

Yes they do! Very handy for a lot of things - e.g. reconstructing from original particles after doing subtraction, or refining starting from global alignments after subtracting aligned on one subregion, for example.

[rbs_sci]

Got extensive hyperparameter calculation running now. I guess it’ll take a few hours. Queued up a dose estimation and motion correction run as well, followed by NU-refine. 10096 has been much easier to process than I expected, basically queued up all the jobs at lunchtime and left it to it. Came back to a 2.95 Ang, cFAR 0.54, pretty reasonable map considering the preferred orientation. Wasn’t expecting that. Bit noisy but turning sharpening down a touch will probably help with that. It’s close enough to Nyquist that I’d actually think about PASRing it…

[olibclarke]

Yes - you can get it even better by splitting picks by power score and just taking the side views (cFAR 0.75-8)

Had the same thought re PASR! It is definitely hitting Nyquist in some directions

[rbs_sci]

OK, I was just plugging in the basics, nothing fancy. Will try some more precise tweaking!

Haha, the forum just suggested I send you a PM as I reply to your posts multiple times…

edit:

This is the worst late-frame weighting I’ve ever seen. Yikes.

Now to start all the tweaking runs.

[olibclarke]

Another example (EMPIAR 10737, cyt bo3 oxidase in nanodisc):

Original FCC & dose weights:

1/5th spatial prior:

1/10th spatial prior:

1/100th spatial prior:

Notably, in addition to the changes in the later frame weighting, adjusting the spatial prior also significantly affects the earlier frames - the frame with the highest weight goes from 3 (original weights estimated using refined spatial prior) to 4 (1/5) to 5 (1/10), then back to 3 (1/100).

This is an example which originally improved quite a lot with polishing (two rounds of RELION polishing took resolution from 2.5 to 2.2), so may be a good example dataset for testing RBMC parameters.

[rbs_sci]

10096 experimenting results:
Baseline: 75,403 particles, 2.93 Ang

Hyperparameters:
Spatial prior strength: 2.9595e-03
Spatial correlation distance: 500
Acceleration prior strength: 5.1466e-02

Dose weights with above priors:

Particle motion:

I forgot to queue the NU refine for this particle stack. Running now!

10x spatial prior

0.1x spatial prior

10x spatial correlation distance

0.1x spatial correlation distance

10x acceleration prior strength

0.1x acceleration prior strength

Waiting on final NU refine output - sorry!

I have a 0.1x spatial prior and 0.1x spatial correlation combo run also processing RBMC/NU-refine queued now since they both gave resolution improvements.

[olibclarke]

Interesting! Would just note - in these cases you changed the priors for both the dose weight estimation & motion correct (looking at the trajectories) - so far I have seen the biggest improvements by just adjusting the priors for dose weight estimation, then reverting to the originally estimated parameters for motion correct - did you try that here too?

[rbs_sci]

Not yet. I’ll add that to the queue!

edit: Finished RBMC’ing a much larger dataset (I quadrupled the number of particles for motion and dose estimation) off a Falcon 4i and we’re back to the “penultimate frame minor overweighting”…

When a few other things have finished, I’m going to grab some EER data off EMPIAR and see if it behaves the same way. I’ll also try some other K3 data I’ve got if time.

[hsnyder]

Excellent discoveries, @rbs_sci and @olibclarke ! Just wanted to weigh in on one point - turning the spatial prior strength parameter down actually causes stronger regularization due to the way we’ve formulated the optimization penalty. So the way you’re seeing the trajectories change with parameter values makes sense.
–Harris