XML file format for image shift groups

Features and Functionality
1

Hi,

We use Leginon and we have been using the tilt group_wrangler.py program to divide micrographs into image shift groups. The latest update in 4.4 to allow this within CryoSPARC is great! The Leginon/Appion website already has functionality to export CTF information, including expected beam tilt, in star format. Can someone show me the format of the xml file which EPU outputs, so I could add that option to Appion/Leginon? I think this would be the easiest option for us to get that data into CryoSPARC. A small example file would be very helpful.

Thanks,

Bill

3 Likes
2

Hi Bill,

I have also been interested in generating per-micrograph xmls for the same purpose. Here’s an example EPU-generated xml I could find from a google search:

<MicroscopeImage xmlns="http://schemas.datacontract.org/2004/07/Fei.SharedObjects" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><name>Empty</name><uniqueID>2aaf776b-9208-4f0e-a012-0301b22a204a</uniqueID><CustomData xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays"><a:KeyValueOfstringanyType><a:Key>DoseOnCamera</a:Key><a:Value i:type="b:double" xmlns:b="http://www.w3.org/2001/XMLSchema">51.847079929456996</a:Value></a:KeyValueOfstringanyType><a:KeyValueOfstringanyType><a:Key>Dose</a:Key><a:Value i:type="b:double" xmlns:b="http://www.w3.org/2001/XMLSchema">6.1173923383053008E+21</a:Value></a:KeyValueOfstringanyType><a:KeyValueOfstringanyType><a:Key>PhasePlateUsed</a:Key><a:Value i:type="b:boolean" xmlns:b="http://www.w3.org/2001/XMLSchema">false</a:Value></a:KeyValueOfstringanyType><a:KeyValueOfstringanyType><a:Key>BinaryResult.Detector</a:Key><a:Value i:type="b:string" xmlns:b="http://www.w3.org/2001/XMLSchema">EF-CCD</a:Value></a:KeyValueOfstringanyType><a:KeyValueOfstringanyType><a:Key>AppliedDefocus</a:Key><a:Value i:type="b:double" xmlns:b="http://www.w3.org/2001/XMLSchema">-2.6E-06</a:Value></a:KeyValueOfstringanyType></CustomData><IntensityScale i:nil="true"/><ReferenceTransformation><matrix xmlns:a="http://schemas.datacontract.org/2004/07/System.Windows.Media"><a:_m11>-9.1887911996279176E-11</a:_m11><a:_m12>-5.6540640526482451E-12</a:_m12><a:_m21>-5.6540640526482508E-12</a:_m21><a:_m22>9.1887911996279176E-11</a:_m22><a:_offsetX>0</a:_offsetX><a:_offsetY>0</a:_offsetY><a:_padding>0</a:_padding><a:_type>TRANSFORM_IS_UNKNOWN</a:_type></matrix><unit xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Common.Types"><a:_x003C_PrefixExponent_x003E_k__BackingField>1</a:_x003C_PrefixExponent_x003E_k__BackingField><a:_x003C_Symbol_x003E_k__BackingField>m</a:_x003C_Symbol_x003E_k__BackingField></unit></ReferenceTransformation><SpatialScale><offset><x><numericValue>0</numericValue><unit xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Common.Types"><a:_x003C_PrefixExponent_x003E_k__BackingField>1</a:_x003C_PrefixExponent_x003E_k__BackingField><a:_x003C_Symbol_x003E_k__BackingField>m</a:_x003C_Symbol_x003E_k__BackingField></unit></x><y><numericValue>0</numericValue><unit xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Common.Types"><a:_x003C_PrefixExponent_x003E_k__BackingField>1</a:_x003C_PrefixExponent_x003E_k__BackingField><a:_x003C_Symbol_x003E_k__BackingField>m</a:_x003C_Symbol_x003E_k__BackingField></unit></y></offset><pixelSize><x><numericValue>9.206169854225621E-11</numericValue><unit xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Common.Types"><a:_x003C_PrefixExponent_x003E_k__BackingField>1</a:_x003C_PrefixExponent_x003E_k__BackingField><a:_x003C_Symbol_x003E_k__BackingField>m</a:_x003C_Symbol_x003E_k__BackingField></unit></x><y><numericValue>9.206169854225621E-11</numericValue><unit xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Common.Types"><a:_x003C_PrefixExponent_x003E_k__BackingField>1</a:_x003C_PrefixExponent_x003E_k__BackingField><a:_x003C_Symbol_x003E_k__BackingField>m</a:_x003C_Symbol_x003E_k__BackingField></unit></y></pixelSize></SpatialScale><microscopeData><acquisition><acquisitionDateTime>2019-12-12T18:33:46.9648052+01:00</acquisitionDateTime><analyticalDetectors/><camera><Binning xmlns:a="http://schemas.datacontract.org/2004/07/System.Drawing"><a:x>1</a:x><a:y>1</a:y></Binning><CameraLocation>EnergyFilter</CameraLocation><CameraSpecificInput xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays"><a:KeyValueOfstringanyType><a:Key>ElectronCountingEnabled</a:Key><a:Value i:type="b:boolean" xmlns:b="http://www.w3.org/2001/XMLSchema">true</a:Value></a:KeyValueOfstringanyType><a:KeyValueOfstringanyType><a:Key>AlignIntegratedImageEnabled</a:Key><a:Value i:type="b:boolean" xmlns:b="http://www.w3.org/2001/XMLSchema">false</a:Value></a:KeyValueOfstringanyType><a:KeyValueOfstringanyType><a:Key>SuperResolutionFactor</a:Key><a:Value i:type="b:int" xmlns:b="http://www.w3.org/2001/XMLSchema">2</a:Value></a:KeyValueOfstringanyType><a:KeyValueOfstringanyType><a:Key>FractionationSettings</a:Key><a:Value i:type="b:EquidistantFractionation" xmlns:b="http://schemas.datacontract.org/2004/07/Fei.Applications.Common.Omp.Interface"><b:NumberOffractions>60</b:NumberOffractions></a:Value></a:KeyValueOfstringanyType></CameraSpecificInput><DarkGainCorrection>None</DarkGainCorrection><ExposureTime>2.9960053262316912</ExposureTime><Inserted>true</Inserted><Name>EF-CCD</Name><PreExposurePauseTime>0</PreExposurePauseTime><PreExposureSupported>false</PreExposureSupported><PreExposureTime>0</PreExposureTime><ReadoutArea xmlns:a="http://schemas.datacontract.org/2004/07/System.Drawing"><a:height>3838</a:height><a:width>3710</a:width><a:x>0</a:x><a:y>0</a:y></ReadoutArea><Shutter>PreSpecimen</Shutter></camera><plateCamera><ExposureTime>0</ExposureTime><Use>false</Use></plateCamera><scanSettings><DwellTime>0</DwellTime><ReducedArea i:nil="true" xmlns:a="http://schemas.datacontract.org/2004/07/System.Drawing"/><Resolution xmlns:a="http://schemas.datacontract.org/2004/07/System.Drawing"><a:height>0</a:height><a:width>0</a:width></Resolution><ScanArea xmlns:a="http://schemas.datacontract.org/2004/07/System.Drawing"><a:height>0</a:height><a:width>0</a:width><a:x>0</a:x><a:y>0</a:y></ScanArea><ScanRotation>0</ScanRotation></scanSettings><scanningDetector><DetectorType>SecondaryElectrons</DetectorType><Gain i:nil="true"/><Inserted>false</Inserted><Name i:nil="true"/><Offset i:nil="true"/></scanningDetector></acquisition><core><ApplicationSoftware>EPU</ApplicationSoftware><ApplicationSoftwareVersion>2.4.0.54</ApplicationSoftwareVersion><Guid>3d3830d5-2823-4f0b-9307-a564f04a4d73</Guid></core><gun><AccelerationVoltage>300000</AccelerationVoltage><ExtractorVoltage>4100</ExtractorVoltage><Filament i:nil="true"/><GunLens>3</GunLens><Sourcetype>FieldEmission</Sourcetype><WehneltBias i:nil="true"/></gun><instrument><AcquisitionSoftware i:nil="true"/><AcquisitionSoftwareVersion i:nil="true"/><ComputerName>TITAN52337040</ComputerName><InstrumentID>3704</InstrumentID><InstrumentModel>TITAN52337040</InstrumentModel></instrument><optics><BeamConvergence i:nil="true"/><BeamDiameter>1.1972449312495386E-06</BeamDiameter><BeamShift xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Types"><a:_x>-0.1381143182516098</a:_x><a:_y>-0.044494681060314178</a:_y></BeamShift><BeamTilt xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Types"><a:_x>-0.045281898230314255</a:_x><a:_y>0.02400130033493042</a:_y></BeamTilt><Cameralength>0</Cameralength><ColumnOperatingMode>TEM</ColumnOperatingMode><ColumnOperatingTemSubMode>BrightField</ColumnOperatingTemSubMode><CondenserStigmator xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Types"><a:_x>0</a:_x><a:_y>0</a:_y></CondenserStigmator><Defocus>-2.4E-06</Defocus><DiffractionFocus>0</DiffractionFocus><DiffractionShift xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Types"><a:_x>0</a:_x><a:_y>0</a:_y></DiffractionShift><DiffractionStigmator xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Types"><a:_x>0</a:_x><a:_y>0</a:_y></DiffractionStigmator><EFTEMOn>true</EFTEMOn><EnergyFilter><AccelerationVoltageOffset>0</AccelerationVoltageOffset><DriftTubeVoltage>0</DriftTubeVoltage><EnergySelectionSlitInserted>true</EnergySelectionSlitInserted><EnergySelectionSlitWidth>20</EnergySelectionSlitWidth><EnergyShift>-0</EnergyShift><EntranceApertureDiameter i:nil="true"/><EntranceApertureType i:nil="true"/></EnergyFilter><Focus>-0.00085227272727272723</Focus><GunStigmator i:nil="true" xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Types"/><IlluminationMode>Parallel</IlluminationMode><IlluminationProbeSubMode i:nil="true"/><ImageShift xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Types"><a:_x>0</a:_x><a:_y>0</a:_y></ImageShift><Intensity>0</Intensity><ObjectiveLensMode>HM</ObjectiveLensMode><ObjectiveStigmator xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Types"><a:_x>0</a:_x><a:_y>0</a:_y></ObjectiveStigmator><ProbeMode>NanoProbe</ProbeMode><ProjectorMode>Imaging</ProjectorMode><SpotIndex>5</SpotIndex><StemDefocus>0</StemDefocus><StemFieldOfView i:nil="true"/><StemMagnification i:nil="true"/><TemMagnification><NominalMagnification>105000</NominalMagnification></TemMagnification></optics><sample><Description i:nil="true"/><ID i:nil="true"/></sample><stage><Holder>Unspecified</Holder><Position><A>0.00012695501341998398</A><B>0.0175425725</B><X>5.5945481799999948E-05</X><Y>-4.55478159999998E-05</Y><Z>-9.5842808022399974E-05</Z></Position><SampleLoader>None</SampleLoader></stage><vacuum><ProjectionChamberPressure>0</ProjectionChamberPressure><SamplePressure>0</SamplePressure><VacuumMode>Ready</VacuumMode></vacuum></microscopeData></MicroscopeImage>

From my limited investigation, it seems like cryoSPARC expects <microscopeData>, <optics>, and <BeamShift> (case-sensitive) from .xml. I was able to import arbitrary beam tilt x & y for filename-matched micrographs via Import Beam Shift with the xml above. Would also be nice to be able to obtain per-micrograph xmls directly from Appion/Leginon! :slight_smile: Thanks!

Best,
Kook

2 Likes
3

https://x.com/DustinMorado/status/1721816758519013511?s=20

4

Thanks! That is helpful,

Bill

5

Dear @wrice,

Just chiming in to show an example of the XML file format that CryoSPARC Import Movies/Micrographs and Import Beam Shift now expects. The only field from the XML file that is read by CryoSPARC is the BeamShift (the _x and _y attributes). This field is arranged as follows within the tree:

<MicroscopeImage xmlns="http://schemas.datacontract.org/2004/07/Fei.SharedObjects" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
    <microscopeData>
        <optics>
            <BeamShift xmlns:a="http://schemas.datacontract.org/2004/07/Fei.Types">
                <a:_x>-0.18816471099853516</a:_x>
                <a:_y>-0.11280126124620438</a:_y>
            </BeamShift>
        </optics>
    </microscopeData>
</MicroscopeImage>

Likely the above could be used as a skeleton to write out one XML file per movie, with the beam shift x and y information stored in the above fields. If you add writing out XML files as an option to Leginon/Appion, please let us know if the above form works when later using these XML files to import beam shift values in CryoSPARC via Import Movies/Micrographs or Import Beam Shift. Please also see this comment for more info and for an alternative option using CryoSPARC Tools.

Best regards,
Michael

4 Likes
6

Thanks a lot Michael! That looks even simpler.

Bill