Copy higher order CTF parameters?

CTF Refinement
1

Hi,

Is there any easy way to copy higher order CTF parameters from one refinement to another?

This would be useful when one class refines to high resolution and allows precise estimation of beam tilt, trefoil etc, but another class refines to more moderate resolution and therefore CTF refinement is not as accurate.

I thought maybe this could be accomplished by playing around with the lower level results, but it seems all the CTF params, including both per particle ones like defocus and per dataset (or per micrograph) ones like beam tilt are in particles.ctf.

Cheers
Oli

1 Like
2

Dear @olibclarke ,

We currently don’t explicitly distinguish in the cs file itself which ctf parameters are per-group and per-particle, which means it’s a bit complicated to do this generally.

For the odd terms (residual shift ctf/shift_A, ctf/tilt_A, and ctf/trefoil_A), these are always constant per group; they can just be copied over to the other dataset.

Some of the even terms are constant per dataset (ctf/cs_mm and tetrafoil ctf/tetra_A), but not all of them (phase shift, defoci) unfortunately, and we don’t store the per-group modifications that were fit during the CTF refinement job aside from printing them to the streamlog. Further, since some of the even terms are coupled to each other, it might not be correct to just copy the spherical aberration and tetrafoil values without also changing the defoci…

Unfortunately I think the only safe way to do this is just to copy over the odd aberrations, i.e. only modify the beam tilt, trefoil, and residual shift. This is an interesting use case though, and definitely speaks to the benefits of separating the group values!

Edit: Actually even for the odd terms, the only way to “copy” them and then use them later in cryoSPARC would involve loading the dataset into a python environment using, modifying the dataset values, and then re-writing the cs file. So it would involve the kind of manipulation like in this tutorial.

Best,
Michael

1 Like
5

Actually I think it is copying the higher order CTF parameters - because when I use them as inputs to a global CTF refinement (supplied using the low level results interface), they show as the previous parameters:

image
image946Ă—408 78.7 KB

6

Hi @olibclarke,

Ah, I think I misunderstood your use case. I think this workflow of overriding the low-level CTF group from one CTF refinement, to a particle stack with a different box size, should work, since they are all in physical units thus the box size doesn’t matter. I think you should also be able to do this by fitting CTF at the low box size, and then using extract from micrographs to re-extract, and then you wouldn’t have to override the low-level result anymore since it would just passthrough. This is separate from the original case about copying CTF values across two different classes though, right?

Best, Michael

7

Hi Michael - yes that’s right, and yes it is a different use case from the original post, apologies for the confusion!

Cheers
Oli

8

@mmclean one related thing that would be useful would be to allow the specification of higher order CTF parameters at particle import time. I frequently go back and forth between cryoSPARC and relion - e.g. to perform multiple rounds of Bayesian polishing. In this case, higher order CTF parameters are lost, but I know what they are from previous refinements - it would be useful to be able to specify beam tilt x/y etc at import, in the same way one can specify other parameters such as spherical aberration. Currently, I have to run a reconstruction job, followed by a global CTF job, to re-estimate these parameters prior to further refinement.

9

We’ve added this to our issue tracker. Ideally we would have a way to match particles’ uids with a previously existing particle dataset in cryoSPARC (so that previous ctf result groups can be used to override those of the imported particles), or to just directly convert RELION optics groups to cryoSPARC’s conventions, but either of those would be more complex than just inputting the tilt/trefoil/tetrafoil/etc values.

Best,
Michael

2 Likes
10

I faced a scenario where the same dataset (merge of two data collections) yielded one high resolution structure (1M particles, 2.4 GSFSC) and one low resolution structure coming from a minority conformation (0.1M particles, 3.8 GSFSC). Because the particles are on the same micrographs, I reasoned that the global CTF must be shared.

So, I determined the global CTF from the high resolution structure, and then copied that over to the low res structure. As per this post, I copied the odd terms. I also copied the shared even terms, accepting that defocus and phase shift would be wrong / off.

Then, I performed local ctf refinement on the low res structure twice in series, and on the second iteration the graph of per-particle shift showed no shift at all.

Am I correct in assuming that the local CTF correction should fix the issue with the defocus and phase shift being wrong from copying the global CTF?

Btw, this is the code I used (each block is a jupyter cell - note manual intervention is required to link the target particles to the job)
Code licence in case you want to reuse: CC BY 4.0 (Licence link)
Attribute: Andrea Murachelli a.murachelli@nki.nl

#
import numpy as np
from cryosparc.tools import CryoSPARC

# Connect to the CryoSPARC instance; change as needed.
cs = CryoSPARC(
    license="mylicence",
    host="myhost",
    base_port=39000,
    email="em@example.com",
    password="password",
)

assert cs.test_connection()

myproject = "P287"  
project = cs.find_project(myproject)
# change these parameters as needed
target_workspace = "W14"
source_global_CTF_job = "J724" #the global CTF job you want to copy from

# create a new external job with the global CTF job as input
# and a slot to connect our target particles
particle_slots = [
    "blob",
    "alignments3D",
    "location",
    "pick_stats",
    "alignments2D",
    "ctf",
]
job = project.create_external_job(target_workspace, title="Global CTF settings")
job.add_input(
    type="particle",
    name="CTF_corrected_particles",
    min=1,
    slots=particle_slots,
    title="CTF corrected particles",
)
job.add_input(
    type="particle",
    name="target_particles",
    min=1,
    slots=particle_slots,
    title="Particles to CTF correct",
)
job.connect("CTF_corrected_particles", source_global_CTF_job, "particles")
job.add_output(
    type="particle",
    name="CTF_updated_particles",
    slots=particle_slots,

    title="Particles with updated CTF",
)
## connect the target particles to the job, then execute the next cell to submit the job

ctf = job.load_input("CTF_corrected_particles", slots=["ctf"])
particles = job.load_input("target_particles", slots=particle_slots)
# determine the CTF values for each exposure group
# these are the changing values for each exposure group
# defocus value and defocus angle are changed by global CTF, but we ignore that for now
# local CTF will take care of that
field_names = [
    "ctf/cs_mm",
    "ctf/amp_contrast",
    "ctf/phase_shift_rad",
    "ctf/shift_A",
    "ctf/tilt_A",
    "ctf/trefoil_A",
    "ctf/tetra_A",
    "ctf/anisomag",

]
exposure_groups = list(np.unique(ctf["ctf/exp_group_id"]))
ctf_values = {group: {} for group in exposure_groups}
for group in exposure_groups:
    for field in field_names:
        ctf_values[group][field] = ctf[field][ctf["ctf/exp_group_id"] == group][0]


# Make sure that all exposure groups in the particles exist in the CTF table
target_exposure_groups = list(np.unique(particles["ctf/exp_group_id"]))
assert set(target_exposure_groups) <= set(exposure_groups)

# copy over the CTF values to the target particles
ptcls_copy = particles.copy()
for row in ptcls_copy[:]:
    group = row["ctf/exp_group_id"]
    for key in field_names:
        row[key] = ctf_values[group][key]

# visual sanity check and show changes.
changes = {group: {} for group in exposure_groups}
for i in range(len(particles)):
    current_group = particles["ctf/exp_group_id"][i]
    if not changes[current_group]:  # default value {} is falsy
        for f in field_names:
            old = particles[f][i]
            new = ptcls_copy[f][i]
            changes[current_group][f] = (old, new)
    else:
        continue
    # no point in checking all particles, just one per group is enough
    all_done = all(list(changes.values()))
    if all_done:
        break

for group in exposure_groups:
    print(f"Group {group}")
    for f in field_names:
        print(f"{f}: Original: {changes[group][f][0]}\t New: {changes[group][f][1]}")
    print("\n")

# commit changes and close job
particles = ptcls_copy
job.print_input_spec()
job.save_output("CTF_updated_particles", particles)
job.stop()
11

Hi @Andrea!

Local CTF Refinement simply changes the defocus of each particle — no other parameters are modified (including astigmatism — both the major and minor axis are changed by the same amount).

1 Like