Error in patch CTF estimation

gefch830 · May 8, 2024, 11:36pm

Hi, I was running the patch CTF estimation of my dataset after patch motion correction, and ended up with the following errors:
[CPU: 452.0 MB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/ctf_estimation/run.py”, line 385, in cryosparc_master.cryosparc_compute.jobs.ctf_estimation.run.run
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/pyplot.py”, line 1831, in ylim
ret = ax.set_ylim(*args, **kwargs)
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/_api/deprecation.py”, line 454, in wrapper
return func(*args, **kwargs)
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/_base.py”, line 3882, in set_ylim
return self.yaxis._set_lim(bottom, top, emit=emit, auto=auto)
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axis.py”, line 1184, in _set_lim
v0 = self.axes._validate_converted_limits(v0, self.convert_units)
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/matplotlib/axes/_base.py”, line 3570, in _validate_converted_limits
raise ValueError(“Axis limits cannot be NaN or Inf”)
ValueError: Axis limits cannot be NaN or Inf

this error only happened for the current dataset, but not for my other one, does anyone know why this error happens and how to proceed? thanks a lot.

rbs_sci · May 9, 2024, 12:28am

It used to be that Patch CTF estimation would error at very high magnifications. I’ve not seen it for a while so not sure if it’s fixed. Can you provide some more information about your data? e.g.: magnification, format (maybe EER?), if EER, what the upsampling factor was, were micrographs saved in 16-bit MRC? What version of CryoSPARC are you using?

If micrographs were saved in 32-bit MRC, you can try CTFFIND rather than Patch CTF, because CTFFIND takes a global CTF it can be more forgiving in borderline cases.

Otherwise, check in the Patch Motion log that there were no errors for reading raw movies or writing corrected output.

gefch830 · May 9, 2024, 1:36pm

hi,
many thanks.
I have checked the patch motion log, and I did not see any errors. the data format is tiff, not EER, and the paste the details settings here:

‘’ Param Name=“PixelSizeX” Value=“0.5050” />
Param Name=“PixelSizeY” Value=“0.5050” />
Param Name=“PixelSizeAngle” Value=“0” />
Param Name=“ProcessCTF” Value=“True” />
Param Name=“ProcessMovement” Value=“True” />
Param Name=“ProcessPicking” Value=“True” />
Import>
Param Name=“Extension” Value=“.tiff" />
Param Name=“HeaderlessWidth” Value=“7676” />
Param Name=“HeaderlessHeight” Value=“7420” />
Param Name=“HeaderlessType” Value=“int8” />
Param Name=“HeaderlessOffset” Value=“0” />
Param Name=“BinTimes” Value=“1.00” />
Param Name=“DosePerAngstromFrame” Value=“0.90” />
Param Name=“CorrectGain” Value=“False” />
Param Name=“CorrectDefects” Value=“False” />
Param Name=“GainFlipX” Value=“False” />
Param Name=“GainFlipY” Value=“False” />
Param Name=“GainTranspose” Value=“False” />
Param Name=“Folder” Value=“X:\DoseFractions\customerservice_20211207_182143\movies211207a" />
Param Name=“GainPath” Value=”" />
Param Name=“DefectsPath” Value=“” />
/Import>
CTF>
Param Name=“Window” Value=“512” />
Param Name=“RangeMin” Value=“0.0673333333333333333333333333” />
Param Name=“RangeMax” Value=“0.5050” />
Param Name=“UseMovieSum” Value=“False” />
Param Name=“Voltage” Value=“300” />
Param Name=“Cs” Value=“2.7” />
Param Name=“DoPhase” Value=“False” />
Param Name=“Amplitude” Value=“0.10” />
Param Name=“ZMin” Value=“0.2” />
Param Name=“ZMax” Value=“4.0” />
Param Name=“MinQuality” Value=“0.8” />
Param Name=“Cc” Value=“2.7” />
Param Name=“IllAperture” Value=“30” />
Param Name=“DeltaE” Value=“0.7” />
Param Name=“Thickness” Value=“0” />
/CTF>
Movement>
Param Name=“RangeMin” Value=“0.0673333333333333333333333333” />
Param Name=“RangeMax” Value=“0.5771428571428571428571428571” />
Param Name=“Bfactor” Value=“-100” />
/Movement>
Grids>
Param Name=“CTFX” Value=“6” />
Param Name=“CTFY” Value=“6” />
Param Name=“CTFZ” Value=“1” />
Param Name=“MovementX” Value=“12” />
Param Name=“MovementY” Value=“12” />
Param Name=“MovementZ” Value=“60” />
/Grids>
Tomo>
Param Name=“DimensionsX” Value=“3712” />
Param Name=“DimensionsY” Value=“3712” />
Param Name=“DimensionsZ” Value=“1400” />
/Tomo>
Picking>
Param Name=“Diameter” Value=“100” />
Param Name=“DataStyle” Value=“cryo” />
Param Name=“MinimumScore” Value=“0.11” />
Param Name=“MinimumMaskDistance” Value=“20” />
Param Name=“DoExport” Value=“True” />
Param Name=“BoxSize” Value=“220” />
Param Name=“Invert” Value=“True” />
Param Name=“Normalize” Value=“True” />
Param Name=“DoRunningWindow” Value=“False” />
Param Name=“RunningWindowLength” Value=“10000” />
Param Name=“ModelPath” Value=“BoxNet2Mask_20180918” />
/Picking>
Export>
Param Name=“SkipFirstN” Value=“0” />
Param Name=“SkipLastN” Value=“0” />
Param Name=“DoStack” Value=“False” />
Param Name=“StackGroupSize” Value=“1” />
Param Name=“DoAverage” Value=“True” />
Param Name=“DoDeconvolve” Value=“False” />
Param Name=“DeconvolutionStrength” Value=“1” />
Param Name=“DeconvolutionFalloff” Value=“1” />
/Export>
Tasks>
Param Name=“UseRelativePaths” Value=“True” />
Param Name=“TomoFullReconstructInvert” Value=“True” />
Param Name=“TomoFullReconstructNormalize” Value=“True” />
Param Name=“TomoFullReconstructPrepareDenoising” Value=“False” />
Param Name=“TomoFullReconstructOnlyFullVoxels” Value=“False” />
Param Name=“TomoSubReconstructNormalizedCoords” Value=“False” />
Param Name=“TomoSubReconstructPixel” Value=“1.01000” />
Param Name=“TomoSubReconstructBox” Value=“128” />
Param Name=“TomoSubReconstructDiameter” Value=“100” />
Param Name=“TomoSubReconstructVolume” Value=“True” />
Param Name=“TomoSubReconstructSeries” Value=“False” />
Param Name=“TomoSubReconstructShiftX” Value=“0” />
Param Name=“TomoSubReconstructShiftY” Value=“0” />
Param Name=“TomoSubReconstructShiftZ” Value=“0” />
Param Name=“TomoSubReconstructPrerotated” Value=“False” />
Param Name=“TomoSubReconstructDoLimitDose” Value=“False” />
Param Name=“TomoSubReconstructNTilts” Value=“1” />
Param Name=“TomoSubReconstructMakeSparse” Value=“True” />
Param Name=“TomoMatchTemplatePixel” Value=“1” />
Param Name=“TomoMatchTemplateDiameter” Value=“100” />
Param Name=“TomoMatchTemplateFraction” Value=“100” />
Param Name=“TomoMatchWhitenSpectrum” Value=“True” />
Param Name=“TomoMatchHealpixOrder” Value=“1” />
Param Name=“TomoMatchSymmetry” Value=“C1” />
Param Name=“TomoFullReconstructDeconvHighpass” Value=“300” />
Param Name=“TomoFullReconstructDeconvFalloff” Value=“1” />
Param Name=“TomoFullReconstructDeconvStrength” Value=“1” />
Param Name=“TomoFullReconstructDoDeconv” Value=“False” />
Param Name=“IncludeFilteredOut” Value=“False” />
Param Name=“IncludeUnselected” Value=“False” />
Param Name=“InputOnePerItem” Value=“False” />
Param Name=“InputPixelSize” Value=“1.01000” />
Param Name=“InputShiftPixelSize” Value=“1” />
Param Name=“OutputPixelSize” Value=“1” />
Param Name=“OutputSuffix” Value=“_goodparticles_BoxNet2Mask” />
Param Name=“InputInvert” Value=“True” />
Param Name=“InputNormalize” Value=“True” />
Param Name=“InputFlipX” Value=“False” />
Param Name=“InputFlipY” Value=“False” />
Param Name=“TomoMatchNResults” Value=“1000” />
Param Name=“OutputNormalize” Value=“True” />
Param Name=“AdjustDefocusSkipExcluded” Value=“True” />
Param Name=“AdjustDefocusDeleteExcluded” Value=“False” />
Param Name=“Export2DPixel” Value=“0.5050” />
Param Name=“Export2DBoxSize” Value=“600” />
Param Name=“Export2DParticleDiameter” Value=“100” />
Param Name=“Export2DDoAverages” Value=“False” />
Param Name=“Export2DDoMovies” Value=“True” />
Param Name=“Export2DDoOnlyTable” Value=“False” />
Param Name=“Export2DDoDenoisingPairs” Value=“False” />
Param Name=“Export2DPreflip” Value=“False” />
Param Name=“TomoFullReconstructPixel” Value=“1.01000” />
Param Name=“MicListMakePolishing” Value=“True” />
Param Name=“ReuseCorrVolumes” Value=“False” />
/Tasks>
Filter>
Param Name=“AstigmatismMax” Value=“3” />
Param Name=“DefocusMin” Value=“0.10” />
Param Name=“DefocusMax” Value=“3.50” />
Param Name=“PhaseMin” Value=“0” />
Param Name=“PhaseMax” Value=“1” />
Param Name=“ResolutionMax” Value=“5” />
Param Name=“MotionMax” Value=“1.0” />
Param Name=“ParticlesSuffix” Value=“_BoxNet2Mask_20180918” />
Param Name=“ParticlesMin” Value=“1” />
Param Name=“MaskPercentage” Value=“60” />
/Filter>
Advanced>
Param Name=“ProjectionOversample” Value=“2” />
Settings>
Param Name=“PixelSizeX” Value=“0.5050” />
Param Name=“PixelSizeY” Value=“0.5050” />
Param Name=“PixelSizeAngle” Value=“0” />
Param Name=“ProcessCTF” Value=“True” />
Param Name=“ProcessMovement” Value=“True” />
Param Name=“ProcessPicking” Value=“True” />
Import>
Param Name=“Extension” Value=".tiff” />
Param Name=“HeaderlessWidth” Value=“7676” />
Param Name=“HeaderlessHeight” Value=“7420” />
Param Name=“HeaderlessType” Value=“int8” />
Param Name=“HeaderlessOffset” Value=“0” />
Param Name=“BinTimes” Value=“1.00” />
Param Name=“DosePerAngstromFrame” Value=“0.90” />
Param Name=“CorrectGain” Value=“False” />
Param Name=“CorrectDefects” Value=“False” />
Param Name=“GainFlipX” Value=“False” />
Param Name=“GainFlipY” Value=“False” />
Param Name=“GainTranspose” Value=“False” />
Param Name=“Folder” Value=“X:\DoseFractions\customerservice_20211207_182143\movies211207a" />
Param Name=“GainPath” Value=”" />
Param Name=“DefectsPath” Value=“” />
/Import>
CTF>
Param Name=“Window” Value=“512” />
Param Name=“RangeMin” Value=“0.0673333333333333333333333333” />
Param Name=“RangeMax” Value=“0.5050” />
Param Name=“UseMovieSum” Value=“False” />
Param Name=“Voltage” Value=“300” />
Param Name=“Cs” Value=“2.7” />
Param Name=“DoPhase” Value=“False” />
Param Name=“Amplitude” Value=“0.10” />
Param Name=“ZMin” Value=“0.2” />
Param Name=“ZMax” Value=“4.0” />
Param Name=“MinQuality” Value=“0.8” />
Param Name=“Cc” Value=“2.7” />
Param Name=“IllAperture” Value=“30” />
Param Name=“DeltaE” Value=“0.7” />
Param Name=“Thickness” Value=“0” />
/CTF>
Movement>
Param Name=“RangeMin” Value=“0.0673333333333333333333333333” />
Param Name=“RangeMax” Value=“0.5771428571428571428571428571” />
Param Name=“Bfactor” Value=“-100” />
/Movement>
Grids>
Param Name=“CTFX” Value=“6” />
Param Name=“CTFY” Value=“6” />
Param Name=“CTFZ” Value=“1” />
Param Name=“MovementX” Value=“12” />
Param Name=“MovementY” Value=“12” />
Param Name=“MovementZ” Value=“60” />
/Grids>
Tomo>
Param Name=“DimensionsX” Value=“3712” />
Param Name=“DimensionsY” Value=“3712” />
Param Name=“DimensionsZ” Value=“1400” />
/Tomo>
Picking>
Param Name=“Diameter” Value=“100” />
Param Name=“DataStyle” Value=“cryo” />
Param Name=“MinimumScore” Value=“0.11” />
Param Name=“MinimumMaskDistance” Value=“20” />
Param Name=“DoExport” Value=“True” />
Param Name=“BoxSize” Value=“220” />
Param Name=“Invert” Value=“True” />
Param Name=“Normalize” Value=“True” />
Param Name=“DoRunningWindow” Value=“False” />
Param Name=“RunningWindowLength” Value=“10000” />
Param Name=“ModelPath” Value=“BoxNet2Mask_20180918” />
/Picking>
Export>
Param Name=“SkipFirstN” Value=“0” />
Param Name=“SkipLastN” Value=“0” />
Param Name=“DoStack” Value=“False” />
Param Name=“StackGroupSize” Value=“1” />
Param Name=“DoAverage” Value=“True” />
Param Name=“DoDeconvolve” Value=“False” />
Param Name=“DeconvolutionStrength” Value=“1” />
Param Name=“DeconvolutionFalloff” Value=“1” />
/Export>
Tasks>
Param Name=“UseRelativePaths” Value=“True” />
Param Name=“TomoFullReconstructInvert” Value=“True” />
Param Name=“TomoFullReconstructNormalize” Value=“True” />
Param Name=“TomoFullReconstructPrepareDenoising” Value=“False” />
Param Name=“TomoFullReconstructOnlyFullVoxels” Value=“False” />
Param Name=“TomoSubReconstructNormalizedCoords” Value=“False” />
Param Name=“TomoSubReconstructPixel” Value=“1.01000” />
Param Name=“TomoSubReconstructBox” Value=“128” />
Param Name=“TomoSubReconstructDiameter” Value=“100” />
Param Name=“TomoSubReconstructVolume” Value=“True” />
Param Name=“TomoSubReconstructSeries” Value=“False” />
Param Name=“TomoSubReconstructShiftX” Value=“0” />
Param Name=“TomoSubReconstructShiftY” Value=“0” />
Param Name=“TomoSubReconstructShiftZ” Value=“0” />
Param Name=“TomoSubReconstructPrerotated” Value=“False” />
Param Name=“TomoSubReconstructDoLimitDose” Value=“False” />
Param Name=“TomoSubReconstructNTilts” Value=“1” />
Param Name=“TomoSubReconstructMakeSparse” Value=“True” />
Param Name=“TomoMatchTemplatePixel” Value=“1” />
Param Name=“TomoMatchTemplateDiameter” Value=“100” />
Param Name=“TomoMatchTemplateFraction” Value=“100” />
Param Name=“TomoMatchWhitenSpectrum” Value=“True” />
Param Name=“TomoMatchHealpixOrder” Value=“1” />
Param Name=“TomoMatchSymmetry” Value=“C1” />
Param Name=“TomoFullReconstructDeconvHighpass” Value=“300” />
Param Name=“TomoFullReconstructDeconvFalloff” Value=“1” />
Param Name=“TomoFullReconstructDeconvStrength” Value=“1” />
Param Name=“TomoFullReconstructDoDeconv” Value=“False” />
Param Name=“IncludeFilteredOut” Value=“False” />
Param Name=“IncludeUnselected” Value=“False” />
Param Name=“InputOnePerItem” Value=“False” />
Param Name=“InputPixelSize” Value=“1.01000” />
Param Name=“InputShiftPixelSize” Value=“1” />
Param Name=“OutputPixelSize” Value=“1” />
Param Name=“OutputSuffix” Value=“_goodparticles_BoxNet2Mask” />
Param Name=“InputInvert” Value=“True” />
Param Name=“InputNormalize” Value=“True” />
Param Name=“InputFlipX” Value=“False” />
Param Name=“InputFlipY” Value=“False” />
Param Name=“TomoMatchNResults” Value=“1000” />
Param Name=“OutputNormalize” Value=“True” />
Param Name=“AdjustDefocusSkipExcluded” Value=“True” />
Param Name=“AdjustDefocusDeleteExcluded” Value=“False” />
Param Name=“Export2DPixel” Value=“0.5050” />
Param Name=“Export2DBoxSize” Value=“600” />
Param Name=“Export2DParticleDiameter” Value=“100” />
Param Name=“Export2DDoAverages” Value=“False” />
Param Name=“Export2DDoMovies” Value=“True” />
Param Name=“Export2DDoOnlyTable” Value=“False” />
Param Name=“Export2DDoDenoisingPairs” Value=“False” />
Param Name=“Export2DPreflip” Value=“False” />
Param Name=“TomoFullReconstructPixel” Value=“1.01000” />
Param Name=“MicListMakePolishing” Value=“True” />
Param Name=“ReuseCorrVolumes” Value=“False” />
/Tasks>
Filter>
Param Name=“AstigmatismMax” Value=“3” />
Param Name=“DefocusMin” Value=“0.10” />
Param Name=“DefocusMax” Value=“3.50” />
Param Name=“PhaseMin” Value=“0” />
Param Name=“PhaseMax” Value=“1” />
Param Name=“ResolutionMax” Value=“5” />
Param Name=“MotionMax” Value=“1.0” />
Param Name=“ParticlesSuffix” Value=“_BoxNet2Mask_20180918” />
Param Name=“ParticlesMin” Value=“1” />
Param Name=“MaskPercentage” Value=“60” />
/Filter>
Advanced>
Param Name=“ProjectionOversample” Value=“2” />. ‘’

wtempel · May 9, 2024, 1:37pm

Welcome to the forum @gefch830.
Did the job fail on the first exposure?
Does the situation described in After updating to V3.0, Patch CTF estimation error: Axis limits cannot be NaN or Inf - #5 by YuJun apply to your dataset?

gefch830 · May 10, 2024, 11:39am

Hi, thanks for the helps.
the problem seems from settings of the patch motion correction. I changed “B-factor during alignment” from 100 to 500, and now the patch CTF estimation is going well.

kstachowski · May 13, 2024, 5:21pm

Hi @gefch830,

We, and others, have found that this is typically caused from a corrupt movie where there is nothing present in the movie (all black) which causes these movies to fail in Patch CTF.

In a recent EMPIAR dataset I was processing, I found that the first movie had 62 frames and was all black while all of the others had 70 frames and something present in the images. After removing the exposure from the stack, I was able to continue processing without issue. Here are a few possible things you can do to find and remove a corrupt exposure:

Use manual exposure curation to look for movies that have an excessive amount of motion (ie. 1e32 pixels of motion). Patch motion does not handle these all black images well.

Screenshot 2024-05-13 at 1.05.59 PM2121×1003 251 KB
Import movies with Skip header check disabled and remove movies that lack the correct number of frames using CS-tools. More information in this post.

All the best,
Kye

gefch830 · May 14, 2024, 8:09am

Hi Kye,
thanks a lot for sharing the very helpful experience, and it is correct that I have seen several black movies in my dataset.
regards,
Gefei