Applying micrograph denoiser to motion corrected data - query

Hi,

Two queries re micrograph denoiser:

  1. Does the version of Patch Motion implemented in Live create the background subtraction training data needed for Micrograph Denoiser? It seems to perform background subtraction, so it should do I guess?

  2. Does the background subtraction needed for Micrograph Denoiser training data strictly require movies? Or can one run pre-aligned data through Patch Motion in Live, just for background subtraction, and use the results for Micrograph Denoiser?

This would be useful for when one downloads data from EMPIAR, which may already be motioncorrected, to allow for denoising of this data in the absence of movies. Currently Live allows the user to pretend that micrographs are movies for the purposes of Patch Motion, but “regular” Patch Motion does not.

Cheers
Oli

Hi @olibclarke!

Yes, Live’s patch motion correction will produce the necessary information to use the pretrained denoising model. (You won’t be able to train your own, since that strictly requires movies). I suppose you could then pick particles on those denoised micrographs and use Reassign Particles to get back to the original micrographs.

We have a note to consider how the Micrograph Denoiser can be extended to handle micrograph sources other than v4.5+ Patch motion Correction.

I hope that helps!

Thanks Rich - why does training the denoiser strictly require movies? I wasn’t entirely clear on this from the description in the guide

Also, how much does the quality of the training data impact the performance of the model? E.g. if micrographs with gold edges or high contrast contaminants are included in the training set, will this negatively impact the quality of the denoising results? We have definitely found that training a model performs much better than using the pretrained model.

Would love to see this incorporated into CS Live, using say the first 200 good mics for training a denoising model and then denoising on the fly for visualization - would help a lot for “at a glance” interpretation of particle behavior, particularly for small particles, which can be important when checking different regions of a grid. Could also help on the fly blob picking…

Why does the denoiser require movies?

Formally, the denoiser requires half micrographs, but these can only be generated by producing separate micrographs from even and odd frames of movies.

The half micrographs are required because the denoiser model is trained using two noisy estimates of the same data. As explained in the original Noise2Noise paper (by which the denoiser is heavily inspired), having two estimates of the same data with independent noise makes it possible to model the underlying noiseless images.

For an intuitive example, consider these two “half-images” which I’m telling you look identical, except for noise:

Since you know that, in the noiseless case, these images should be identical, you know anything that differs between the two half-images is noise (there’s an analogy to be made here with the GSFSC as well). Knowing this, you can produce a denoised image (ideally, just the letter A on a white field). This is the underlying principle of Noise2Noise and the Micrograph Denoiser.

Let me know if it’s still not clear why training a new model requires movies/half-mics! I’m still figuring out the best way to explain it :slight_smile:

How much does the quality of the training data impact performance?

I haven’t done enough extensive testing to give you a quantitative answer here, unfortunately. In my experience, it’s rather robust with regard to the number of training micrographs and the amount of those micrographs covered by ethane contaminants, ice crystals, hole edges, etc. If the grayscale estimate is bad, that does hurt performance.

I quickly pulled 16 movies from EMPIAR 11632 (chosen just because I had it to hand) which had a lot of carbon or ice and trained a denoiser using just those 16 movies with otherwise all default settings. I think it did pretty well, considering:

Indeed, if you have or find any examples where training fails to produce a good denoiser, I’d be interested in seeing them!

Feature request

I’ve recorded your request for adding the Denoiser to CryoSPARC Live!

4 Likes

Ah… got it… that is what was meant by half micrographs, now I understand, makes perfect sense! Thanks!

2 Likes

I’ll add more clarification to the guide on what half-mics are, thank you @olibclarke!

2 Likes

Presumably the even/odd sets are taken with the alignments from patch motion pre-applied then?

Right — we use the rigid motion from the whole movie to align all the frames before splitting them (so they’re not independent, but we don’t want them to be anyway).

1 Like

One more question sorry - is the gain reference used during this process (presumably yes during rigid motion correction)?

I presume that if there is residual fixed pattern noise (which will be the same in even/odd sets) this will not be well corrected by the denoiser?

(by the way, if you want an example with incomplete gain correction for testing, try EMPIAR-10828 - making a gain from averaging the movies substantially improves data quality in this case)

Yes, the gain reference is applied during half-mic generation and motion correction.

Good question about fixed pattern noise. My guess is that it might be less important than you’d expect, since the micrographs are pretty significantly downsampled during training, so typical gain ref problems (like a line of bright/dim pixels) might not show up as brightly. Let me test and get back to you.

1 Like

Hi @olibclarke! Ran a quick test with just 50 movies from 10828.

My intuition was not quite right — the bad (or, really, absent) gain reference does impact the results. However, I still find them subjectively more interpretable than a 14 Å lowpass filter.

To get this result, I had to adjust the grayscale normalization factor a bit (down to 0.5) and increase the edge crop to 0.1 to remove the dark areas at the edge of the frame.