Semi-unrelated, but pertaining to one of the earlier posts in the thread - in cases like this:
It would be useful to have an option in local CTF refinement to leave the defocus parameters of particles that “fail” the search (hit the search boundaries without finding a clear minimum) unchanged, rather than setting them to the maximum value.
Perhaps this should even be the default, if no clear minimum as found? In the case above, the local CTF refinement still improves the resolution of the reconstruction, but I suspect this is due to the adjustments in the bimodal, central parts of the distribution - I would like to avoid the outliers changing defocus by physically unreasonable values…
Thanks again for the script @rposert, it will be immensely helpful!
Cheers
Oli
1 Like
Hi @rposert, just some more testing on this script - it works if I directly specify the center in volume alignment tools, but if I use the option in VAT to recenter on the mask center of mass, it fails with this error:
Traceback (most recent call last):
File "adjust_defocus_subparticles.py", line 28, in <module>
recenter_string = job.doc["params_spec"]["recenter_shift"]["value"]
KeyError: 'recenter_shift'
Yes, I directly read the recenter string there, so if you use a mask it won’t know the new center. Replacing that whole bit with the below if/else it should work. Let me know!
import re
if "recenter_shift" in job.doc["params_spec"]:
recenter_pattern = r"([\.\d]+), ?([\.\d]+), ?([\.\d]+) ?(px|A)?"
recenter_string = job.doc["params_spec"]["recenter_shift"]["value"]
recenter_regex = re.match(recenter_pattern, recenter_string)
new_center_A = np.array([
float(x) for x in [
recenter_regex.group(1),
recenter_regex.group(2),
recenter_regex.group(3)
]
])
if len(recenter_regex.groups()) == 3 or recenter_regex.group(4) == "px":
new_center_A *= apix
else:
# rather than re-calculate the center of mass, just get it from the job log
# NOTE: cs.cli.get_job_streamlog() is hidden and experimental --
# it may change without warning
vol_align_log = cs.cli.get_job_streamlog(project_number, vol_align_uid)
centering_line = [x["text"] for x in vol_align_log if "New center will be" in x["text"]]
new_center_vox = re.search(r"([\.\d]+), ([\.\d]+), ([\.\d]+)", centering_line[0])
new_center_vox = np.array([
float(x) for x in [
new_center_vox.group(1),
new_center_vox.group(2),
new_center_vox.group(3)
]
])
new_center_A = new_center_vox * apix
1 Like
Thanks Rich this does the trick for recentering with a mask!
However, I’ve now tried the script on two samples, both high resolution cases where I know the hand is correct. In the first case, adjusting the defocus worked, improving resolution considerably.
In the second case, the resolution actually got worse than without adjustment, which was surprising. I wonder if there is a bug somewhere? In the case that worked, the subparticle was displaced along the z-axis of the volume, while in the case that didn’t, it was displaced off axis (with little displacement in z).
What we want to do (I think) is to subtract the Z component of the recentering vector from the defocus, where Z is the axis corresponding to the projection direction of the particle (not neccessarily the z axis of the volume). Is that what is happening in this part?
for row in particles.rows():
pose = R.from_rotvec(row["alignments3D/pose"])
pose_recenter_vec_A = pose.apply(recenter_vector_A)
row["ctf/df1_A"] -= pose_recenter_vec_A[2]
row["ctf/df2_A"] -= pose_recenter_vec_A[2]
Is the third ([2]) element of pose_recenter_vec_A the component along the projection direction? I guess this will be the case after pose.apply? Apologies for the likely silly questions!
EDIT:
Or maybe it is an issue of the particle source - these particles were polished & extracted in Relion - while the volume is the correct (biological) hand, perhaps if micrographs were flipped/mirrored for extraction that would cause inversion of the “true” hand to be considered when adjusting defocus…? Will try with the inverted hand…
Yes, what we’re doing in that chunk is rotating the recentering operation from the reference frame to the particle’s frame, then subtracting the Z component of the rotated recentering vector from the defocus. I think this should do what we want – if the opposite hand doesn’t work right, please let me know and I’ll think more about it!
1 Like
Opposite hand doesn’t work either - strange. I will try in a couple more test cases and see if I can figure it out, thanks!
My test case didn’t improve in either hand either. I assumed I’d done something wrong, but it’s very possible there’s a flaw in my reasoning. I’ll think more about it!
1 Like
I tried on a second case and same thing - both the “correct” and inverse adjustments made resolution worse (where I would expect one to make it better and one to make it worse, compared to no adjustment).
In the case that worked, the subparticle transformation corresponded to a direct translation along the z-axis - could that somehow make the difference?
So I’m not sure if this is the issue but it is an issue:
Here, if in VAT one doesn’t specify a unit (defaults to px), this loop is never run - because the length of recenter_regex.groups() is 4 regardless whether the unit is explicitly specified. If a unit is not specified, it has the format ('x', 'y', 'z', None). The consequence of this is that if the initial center is entered without units, it remains in pixels - it is never converted to Å.
Whoops! Sorry about that, forgot it’s a conditional group at the end there (“…so I used regex. Now I have two problems”). Replacing that line with
if recenter_regex.group(4) != "A":
ought to fix that issue at least. Still thinking about why it’s not working…
1 Like
Yes that fixes that issue, but main issue (resolution getting worse after adjusting defocus) still remains…
The weird thing is it works really well in one case (where the recentering vector is directly along z), but not in other cases - I wonder if that somehow has something to do with it…
1 Like
I think it might do; I had a case a while ago (tangentally defocus related) where modifications along Z were OK, offset was all messed up - I screwed up the ZYZ handling in my rotation calculations. 
1 Like
I guess it depends on the format of alignments3D/pose … docs for the scipy transformation are here: from_rotvec — SciPy v1.14.0 Manual
This is also where in the docs for CS tools, it would be useful to have a detailed description of all the data structures in CS, in addition to a description of the modules - a description of e.g. how pose is specified would be helpful (maybe it is there already, I just couldn’t find it at a glance)
From here: It's time I admit I don't know what Rodrigues vectors are - #3 by DanielAsarnow
Pose is specified in axis-angle representation, which is what from_rotvec is expecting by default, and printing alignments3d/pose during the script looks consistent with that. What I don’t quite understand is whether something is going wrong when applying that transformation to the recentering vector - that is all I can think of as the issue, but can’t figure out why that would be the case…
1 Like
After a bit of time away from the problem, this:
# assume volume was well-centered
recenter_vector_A = map_size_A / 2 - new_center_A
# recenter_vector_A is the vector pointing from the old center to the
# new center in the reference pose. Now we rotate this vector for each
# recentered particle by applying the pose
is looking incorrect to me. This vector points from the new center to the old box center. @olibclarke could you try swapping the order of those operands, i.e.,
recenter_vector_A = new_center_A - map_size_A / 2
and testing that? Asking you to do it to avoid the confounding variable of my slapdash EMPIAR virus map being in the wrong hand or otherwise bad 
.
If this vector is pointing the wrong way, Z would be the correct length but flipped in the Z-only case (which worked), so you may need to try adding pose_center_vec_A to the defocus rather than subtracting it in the row iterations.
2 Likes
Hi Rich,
I tried that, but unfortunately it still makes things slightly worse than without adjustment… regardless of whether I add or subtract to adjust the defocus…
Oli
1 Like
Thanks for taking a swing at it for me. I’ll keep thinking!
1 Like
@olibclarke I had to think a little bit but I believe I have corrected for different micrograph and particle pixel sizes from cryosparc. I think the defocus transformation should be correct if you don’t use --inverty and do the re-extraction in Relion. In the cryosparc case the Y axis might not be right. Also it’s not strictly necessary to recenter if the subparticles don’t need a smaller box - if the same box is kept then all recentering does is make the particle origins normally distributed around the origin.
1 Like
@olibclarke Also the function was changing the defocus angle based on a very old misunderstanding I had at the beginning of grad school! Now it will only change the defocus values.
1 Like
Thanks Dan I will give this a try!! One thing - why is re-extraction in relion necessary in this case?
It shouldn’t be, I’m just sure it will behave as expected that way. The coordinate flip either on export or when importing back to CS might make the origin shift the wrong direction, I have to check/think about it.
1 Like
