CTF Refinement error in Non-uniform refinement

Izana · October 15, 2024, 1:43pm

Greetings,

I was trying to run a NU-refinement with

refine_scale_min
refine_defocus_refine
refine_ctf_global_refine

and got the following error message:

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 115, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/refine/newrun.py”, line 533, in cryosparc_master.cryosparc_compute.jobs.refine.newrun.run_homo_refine
File “cryosparc_master/cryosparc_compute/jobs/refine/newrun.py”, line 534, in cryosparc_master.cryosparc_compute.jobs.refine.newrun.run_homo_refine
File “cryosparc_master/cryosparc_compute/jobs/ctf_refinement/run.py”, line 185, in cryosparc_master.cryosparc_compute.jobs.ctf_refinement.run.full_ctf_refine
AssertionError: The maximum CTF Refinement fit resolution of 15.89 Angstroms is worse than the minimum Global CTF Refinement fit resolution parameter of 10.00 Angstroms.
This indicates resconstruction quality is not high enough to confidently perform CTF Refinement.
If this is a Homogeneous or Non-Uniform refinement job: please mark the job as complete, or run the job without Global CTF Refinement enabled.
CTF Refinement can be attempted subsequently by running the standalone Global CTF Refinement job.
If this is a standalone Global CTF Refinement job, consider setting a lower (i.e. worse) Minimum fit resolution and/or a higher (i.e. better) Maximum fit resolution.

Since the resolution is still trapped at 10 A and I’m not expecting it to be better than 5 A anyway, I thought Global CTF won’t do much and instead ran a standalone Local CTF Refinement instead. It turned out to be awful, confusing and also with an error:

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 115, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/ctf_refinement/run_local.py”, line 153, in cryosparc_master.cryosparc_compute.jobs.ctf_refinement.run_local.run
File “cryosparc_master/cryosparc_compute/jobs/ctf_refinement/run_local.py”, line 236, in cryosparc_master.cryosparc_compute.jobs.ctf_refinement.run_local.full_defocus_refine
File “cryosparc_master/cryosparc_compute/jobs/ctf_refinement/run_local.py”, line 344, in cryosparc_master.cryosparc_compute.jobs.ctf_refinement.run_local.do_defocus_refine
IndexError: list index out of range

This is really shocking so I double checked my micrographs with a Manually Curate Exposures job. But things seems OK to me?

Most of the exposures do have a CTF fit resolution greater than 5 A though, is this likely the cause of issue? Feeding the Local CTF refined particles back into refinement jobs doesn’t improve the issues either. Is there any thing I can do at this stage to try to utilize this data set?

Many thanks in advance.

Iz

ops an error occurred: Sorry, new users can only put one embedded media item in a post.

Izana · October 15, 2024, 1:45pm

And there are how the exposures look like:

Izana · October 15, 2024, 1:46pm

Izana · October 15, 2024, 1:46pm

rposert · October 21, 2024, 7:59pm

Hi @Izana! Welcome to the forum! Let’s see if we can clear a few things up about CTF refinement.

The standalone Local CTF job

Starting with the most straightforward problem, it looks like your Local CTF job has an empty half-set. This can occur when the particles come straight from an Ab initio reconstruction job. If this is true, first refining the particles (using, e.g., Homogeneous Refinement) or re-splitting them (by, e.g., using Homogeneous Reconstruction Only with Force re-do GS split turned on) should solve this particular problem.

When to do CTF refinement

When you perform CTF refinement, you compare the particle images to projections of your volume and try to adjust any mismatch between the two by adjusting CTF parameters. As we mention in our guide page on CTF Refinement, this generally works best when the particle is rigid and the volume is already of pretty high quality (better than 4 Å for local CTF refinement and even higher resolution for global CTF refinement).

If your volume is still at around 10 Å, CTF refinements will probably not help (and may in fact make things worse)! Generally, we’d recommend trying to improve your map via cleanup (perhaps with Heterogeneous Refinement) and refinement of particle poses (through Homogeneous or Non-Uniform Refinement) before attempting CTF refinement.

I hope that helps! Please let me know if you have any more questions!

Izana · October 23, 2024, 1:01pm

Dear rposert,

Thank you for your reply.

The error with CTF refinement went away after the half-sets were correctly split as you were suggesting. The input came from a failed NU-refinement nevertheless. I guess the failure in the previous job might have leads to the issue in the local CFT job.

I’m aware that local CTF refinement is more about pushing the resolution further for higher resolution volume reconstruction. However, in this particular case, my Homogenous or NU-refinement jobs seems to be struggling. I have run several rounds of 2D classification and ab-initio jobs to clean up the particles beforehand. And according to the volume output from ab-intio, I’m already able see to the expected features from my membrane protein (looks like at 6-10 A already). But both homogenous and NU-refinement are only making the volumes worse, outputting the volumes at >20 A or fails with the CTF refinement error due to low resolution. According to the log it looks like particle defocus is really all over the places.

As local CTF refinement which replies on half-maps in CS is only making things worse, is there any ways or tools that can help me with this issue?

rposert · October 23, 2024, 4:09pm

Hi @Izana!

As you’ve observed, when the map is this poor, CTF refinement will almost always make results much worse. I’d therefore recommend you leave it off during refinements, and do not use standalone CTF refinement jobs either, until your map is much better (maybe better than 4 Å for local refinement).

It sounds like a tough target! Have you read our guide page on tips for processing membrane proteins? There is some advice there that you may find helpful!

Izana · November 15, 2024, 2:04pm

After careful comparison between another csparc project on a slightly different paralog (10 AA apart among a 350 kDa complex) it turned out that my issue, which was essentially the homo refinement right after ab-intio resulted in a much worse volume than expected, was due to the EER Upsampling Factor during movie import. After changing the upsampling factor from the default value of 2 back to 1 as I always used to do (but forgot this time), everything worked just fine.

On the other hand, the spread of per particle defocus remains huge for me, but this might be due to the fact that the raw data (grid squares) was acquired automatedly without manual curation (which has led to lower CTF resolution).