Nominally ~3 Å, SCF* marginal but ok (0.8), map is interpretable, but cFAR is abysmal (0.02).
I’ve seen this before for other datasets, too, early-ish in processing. Is cFAR being thrown off by a couple of directions with a lot of junk, or what? In any case, it seems like in these cases, cFAR is not the best metric for judging anisotropy. Has anyone else noticed similar results?
My impression is that a low cFAR is possibly a reliable indicator of problematic anisotropy once the dataset is fully cleaned up - but when there is a lot of junk remaining in the dataset, not so much.
EDIT:
Perhaps a clue - the “relative signal” FSCs, which are calculated in a torus rather than a cone, show no sign of such dramatic directional resolution variation:
I think the difference is pretty clearly due to just having one or two outlying directions in the first instance, whereas here the plus/minus one sigma distribution spans a wide resolution range.
I’ve seen it a couple of times. It’s why, while I use cFAR as a guide, I ultimately decide on whether a map is worth proceeding with by eyes on - and always the unsharpened map. If that is good, I’ll checked the sharpened map. I’ve also had one case where cFAR was “OK-ish” but the map was a hot mess of streaks from severe preferred orientation.
I’m currently scratching my head over why a dataset is severely anisotropic in RELION but gives a new PDRA a really nice, isotropic map in CryoSPARC with roughly the same number of particles from both, the mics I’ve checked have had the same picks after cleanup… it’s really weird and incredibly frustrating.
Yes agreed - I would always trust my eyes over anything else… would be nice to have a single stat to report on anisotropy for the purposes of validation/table 1 though. Maybe SCF* is more reliable?
Or just the plus/minus one sigma directional resolution range (which unfortunately isn’t given numerically in orientation diagnostics)? I think in this case this more accurately tallies with the visual appearance of the map - approximately 2.7 in one direction, 3.2 in the other feels right
I think this is probably the reason - the cFAR is literally the min/max FSC area ratio, which means a single outlier will throw it off dramatically.
I wonder if either using the plus/minus one-sigma curves for cFAR calculation, or weighting somehow by the population of each resolution bin, would give more useful/reliable results.
Also, updated the original post - the toroidal FSCs do not show this behavior, exhibiting only moderate anisotropy, so I wonder if a wAUC ratio of the min/max or plus/minus one sigma of those might be worth considering.
Hi @olibclarke thanks for posting and for the detailed report!
@olibclarke you are correct that currently cFAR is defined as the ratio of the min/max cFSC area-under-curve, and therefore if there is a single direction with very poor cFSC, that can cause the cFAR to be low even if other directions are okay. One of the reasons we designed cFAR this way is that each cFSC curve that goes into cFAR is the FSC for a cone with a half-angle of 20 degrees - this means that each cFSC is already averaging over a relatively wide angular extent, and so having a single low cFSC really does mean that a big chunk of angular space has poor correlation. That said, we have also seen a similar phenomenon as what you (and @rbs_sci) describe, i.e. that a map otherwise looks good and there isn’t evidence of orientation bias in the density, but the cFAR is still very low. We have only seen this rarely, in particular in one case for a membrane protein where the micelle is a large fraction of the total density.
We’ve been investigating this phenomenon internally - and in fact we have also considered:
using a “tFAR” score (using tFSCs), though it’s unclear if they will behave in a useful way in all cases as we haven’t tested it as widely as cFAR
using the +/- 1-sigma curves (or percentiles, etc) of cFSC to define cFAR - but we are wary of changing the definition of cFAR in a way that will produce very different values for most cases (since cFAR has already become widely used by CryoSPARC users)
It would be great if either/both of you can provide more details about the cases you’ve seen - @kyestachowski will DM you both (probably on Monday) for more info!
Happy to provide more details as needed! This is EMPIAR-11918 - so it is not a membrane protein, and does not have large disordered regions, but is perhaps anomalous in that it is so small. Can send particle stacks, exported jobs etc if useful.
Incidentally I tried progressively increasing the conical half angle used for cFSC calculation - I had to increase it to 60 deg before I got a cFAR score >0.5
I have definitely noticed this phenomenon often enough though (also with membrane proteins as you mentioned, but it does not manifest with all membrane proteins, even ones with large micelles) that it makes me a bit wary of relying on cFAR as a general-purpose metric for diagnosing/quantifying real-space anisotropy, which would be great to have for e.g. a “table 1” type purpose…
Yes @olibclarke almost every time with NU-refine cFAR=0.01 to 0.1. But the same particles + meta data will homogeneous reconstruct with much higher cFAR (0.4-0.8). Playing with dynamic masking can change cFAR a little. Rebalance Orientations also slightly improves our cFARs, but it remains low. In many cases we get decent maps and just uses our eyes to judge.
Interesting - in this case, both homogeneous refinement and NU give a very low cFAR, but the map is clearly interpretable and has details consistent with the global resolution
