Import particle.star from Relion-3.1 to cryosparc

shabih · July 19, 2020, 11:23pm

Hi,
I imported particles.star from Extract directory of Relion-3.1 to cryosparc (latest version) successfully using only the “Particle meta path”. However, running 2D classification throws this following error:

[CPU: 90.7 MB] Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 52, in cryosparc2_compute.run.main
File “cryosparc2_compute/jobs/runcommon.py”, line 823, in check_default_inputs
assert False, 'Non-optional inputs from the following input groups and their slots are not connected: ’ + missing_inputs + ‘. Please connect all required inputs.’
AssertionError: Non-optional inputs from the following input groups and their slots are not connected: particles.ctf. Please connect all required inputs.

Can anyone suggest how to fix this.

Thanks.

Shabih

olibclarke · July 20, 2020, 2:22pm

If you look at the log of your particle import job, what data groups does it say are being imported? does it say it is ignoring some of the columns in the star file?

Oli

shabih · July 20, 2020, 2:40pm

Hi Oli,

Thanks a lot. I got your point. It is indeed missing several parameters. Do you or others have a solution for this. Here is the log from cryosparc and I have also added header of my original particles.star file:

License is valid.

Running job on master node

[CPU: 98.3 MB] Project P3 Job J4 Started

[CPU: 98.3 MB] Master running v2.15.0, worker running v2.15.0

[CPU: 98.5 MB] Running on lane lmb-cpu-nodes

[CPU: 98.5 MB] Resources allocated:

[CPU: 98.5 MB] Worker: hal.lmb.internal

[CPU: 98.5 MB] --------------------------------------------------------------

[CPU: 98.5 MB] Importing job module for job type import_particles…

[CPU: 194.5 MB] Job ready to run

[CPU: 194.5 MB] ***************************************************************

[CPU: 194.6 MB] Importing particles from /cephfs/sshakeel/FA20_LMB_Krios3_K3nonCDS_105kX_130320/Extract/test_for_cryolo/particles.star

[CPU: 194.7 MB] File extension is star

[CPU: 194.8 MB] Importing star file.

[CPU: 261.3 MB] Loaded star with 209732 items

[CPU: 261.3 MB] star file fields:

[CPU: 261.3 MB] Index([u’rlnCoordinateX’, u’rlnCoordinateY’, u’rlnImageName’, u’rlnMicrographName’, u’rlnOpticsGroup’, u’rlnAngleRot’, u’rlnAngleTilt’, u’rlnAnglePsi’, u’rlnOriginXAngst’, u’rlnOriginYAngst’, u’rlnClassNumber’, u’rlnNormCorrection’, u’rlnLogLikeliContribution’, u’rlnMaxValueProbDistribution’, u’rlnNrOfSignificantSamples’, u’rlnGroupName’, u’rlnGroupNumber’],
dtype=‘object’)

[CPU: 261.3 MB] Reading particle data locations…

[CPU: 261.3 MB] Reading rlnImageName to get indices and paths…

[CPU: 338.7 MB] Searching for linked data files…

[CPU: 338.7 MB] Warning: The input star file does not contain rlnDefocusU and rlnDefocusV. Output will not contain CTF parameters for particles.

[CPU: 338.7 MB] Warning: The input star file does not contain rlnDefocusAngle. Output will not contain CTF parameters for particles.

[CPU: 338.7 MB] Warning: No micrographs were connected as inputs, so correspondences cannot be found, so output will not contain pick locations for particles.

[CPU: 338.7 MB] Warning: The input star file does not contain rlnAngleRot, rlnAngleTilt, rlnAnglePsi, rlnOriginX, rlnOriginY, rlnRandomSubset. Output will not contain particle orientations.

[CPU: 338.7 MB] --------------------------------------------------------------

[CPU: 338.7 MB] Particle information has now been imported for 209732 particles, creating outputs…

[CPU: 338.7 MB] Found references to 3735 unique data files

[CPU: 338.7 MB] Import paths were unique at level -1

[CPU: 338.7 MB] Example imported relative path:
J4/imported/FoilHole_11641691_Data_11634123_11634125_20200312_202256_fractions.mrcs

[CPU: 338.7 MB] Reading MRC file headers to check shape…

And here is the header of the particle.star file from Extract directory of Relion:

version 30001

data_optics

loop_
_rlnOpticsGroupName #1
_rlnOpticsGroup #2
_rlnMtfFileName #3
_rlnMicrographOriginalPixelSize #4
_rlnVoltage #5
_rlnSphericalAberration #6
_rlnAmplitudeContrast #7
_rlnImagePixelSize #8
_rlnImageSize #9
_rlnImageDimensionality #10
opticsGroup1 1 mtf_k3_sts_300kV.star 0.860000 300.000000 2.700000 0.100000 0.860000 380 2

version 30001

data_particles

loop_
_rlnCoordinateX #1
_rlnCoordinateY #2
_rlnImageName #3
_rlnMicrographName #4
_rlnOpticsGroup #5
_rlnAngleRot #6
_rlnAngleTilt #7
_rlnAnglePsi #8
_rlnOriginXAngst #9
_rlnOriginYAngst #10
_rlnClassNumber #11
_rlnNormCorrection #12
_rlnLogLikeliContribution #13
_rlnMaxValueProbDistribution #14
_rlnNrOfSignificantSamples #15
_rlnGroupName #16
_rlnGroupNumber #17
1750.000000 1886.000000 000001@Extract/job129/Micrographs/Images-Disc1/GridSquare_11628949/Data/FoilHole_11641691_Data_11634123_11634125_20200312_202256_fractions.mrcs MotionCorr/job012/Micrographs/Images-Disc1/GridSquare_11628949/Data/FoilHole_11641691_Data_11634123_11634125_20200312_202256_fractions.mrc 1 0.000000 0.000000 -36.18227 -18.49281 -1.47198 23 0.825706 12507.731622 0.333322 7 group_0009

Thanks.
Shabih

olibclarke · July 20, 2020, 3:23pm

Hi Shabih, your problem is that your original particle star file does not contain defocus estimates, so they cannot be imported. Did you extract your particles from micrographs without ctf estimates?

Cheers
Oli

shabih · July 20, 2020, 7:43pm

Excellent observation, Oli. Yes, by mistake I extracted particles using motion_corrected mics instead of motion_corrected-ctf_estimated mics. It works now.

Thanks.

Shabih

rashed · August 2, 2020, 8:05pm

Hi, I’ve got a similar issue as above.

Using relion_stack_create I’ve created a stack and a star file from Relion Class3D/ Select job output : job039_particles.mrcs, job039_particles.star
Renamed job039_particles.mrcs to job039_particles.mrc as CS doesn’t recognize .mrcs.

My job039_particles.star contains the following columns :

version 30001

data_optics

loop_
_rlnOpticsGroupName #1
_rlnOpticsGroup #2
_rlnMicrographOriginalPixelSize #3
_rlnVoltage #4
_rlnSphericalAberration #5
_rlnAmplitudeContrast #6
_rlnImagePixelSize #7
_rlnImageSize #8
_rlnImageDimensionality #9
20200706 1 0.680000 300.000000 2.700000 0.100000 1.360000 256 2

version 30001

data_particles

loop_
_rlnImageName #1
_rlnMicrographName #2
_rlnCoordinateX #3
_rlnCoordinateY #4
_rlnAnglePsi #5
_rlnOriginX #6
_rlnOriginY #7
_rlnMagnification #8
_rlnDetectorPixelSize #9
_rlnDefocusU #10
_rlnDefocusV #11
_rlnDefocusAngle #12
_rlnPhaseShift #13
_rlnCtfBfactor #14
_rlnOpticsGroup #15
_rlnClassNumber #16
_rlnOriginXAngst #17
_rlnOriginYAngst #18
_rlnGroupNumber #19
_rlnAngleRot #20
_rlnAngleTilt #21
_rlnNormCorrection #22
_rlnLogLikeliContribution #23
_rlnMaxValueProbDistribution #24
_rlnNrOfSignificantSamples #25
000001@>imports/job039_particles.mrc >J1/imported/FoilHole_6759195_Data_6760735_6760737_20200706_172723_aligned_DW.mrc 1207.000000 2705.000000 -151.74943 5.525000 6.175000 10000.000000 1.360000 11316.405273 11214.852539 72.167595 0.000000 0.000000 1 4 -21.71377 4.491798 1 -30.81021 51.142400 0.722931 34685.681850 0.485405 1044

Then I tried to import the star file from cryoSPARC, which seems to be reading the columns from the .star file okay but it struggles with the particle stack :

Project P16 Job J144 Started

[CPU: 90.1 MB] Master running v2.15.0, worker running v2.15.0

[CPU: 90.2 MB] Running on lane default

[CPU: 90.2 MB] Resources allocated:

[CPU: 90.2 MB] Worker: localhost

[CPU: 90.2 MB] --------------------------------------------------------------

[CPU: 90.2 MB] Importing job module for job type import_particles…

[CPU: 164.8 MB] Job ready to run

[CPU: 164.8 MB] ***************************************************************

[CPU: 164.8 MB] Importing particles from /path/cryosparc/P16/imports/job039_particles.star

[CPU: 164.9 MB] File extension is star

[CPU: 164.9 MB] Importing star file.

[CPU: 250.5 MB] Loaded star with 312776 items

[CPU: 250.5 MB] star file fields:

[CPU: 250.5 MB] Index([u’rlnImageName’, u’rlnMicrographName’, u’rlnCoordinateX’, u’rlnCoordinateY’, u’rlnAnglePsi’, u’rlnOriginX’, u’rlnOriginY’, u’rlnMagnification’, u’rlnDetectorPixelSize’, u’rlnDefocusU’, u’rlnDefocusV’, u’rlnDefocusAngle’, u’rlnPhaseShift’, u’rlnCtfBfactor’, u’rlnOpticsGroup’, u’rlnClassNumber’, u’rlnOriginXAngst’, u’rlnOriginYAngst’, u’rlnGroupNumber’, u’rlnAngleRot’, u’rlnAngleTilt’, u’rlnNormCorrection’, u’rlnLogLikeliContribution’, u’rlnMaxValueProbDistribution’, u’rlnNrOfSignificantSamples’],
dtype=‘object’)

[CPU: 250.5 MB] Reading particle data locations…

[CPU: 250.5 MB] Reading rlnImageName to get indices and paths…

[CPU: 342.0 MB] Searching for linked data files…

[CPU: 342.3 MB] Warning: The input star file does not contain rlnVoltage, and parameter accel_kv was not set. Output will not contain CTF parameters for particles.

[CPU: 342.3 MB] Warning: The input star file does not contain rlnSphericalAberration, and parameter cs_mm was not set. Output will not contain CTF parameters for particles.

[CPU: 342.3 MB] Warning: No micrographs were connected as inputs, so correspondences cannot be found, so output will not contain pick locations for particles.

[CPU: 342.3 MB] Warning: The input star file does not contain rlnAngleRot, rlnAngleTilt, rlnAnglePsi, rlnOriginX, rlnOriginY, rlnRandomSubset. Output will not contain particle orientations.

[CPU: 342.4 MB] Particle information has now been imported for 312776 particles, creating outputs…

[CPU: 342.4 MB] Found references to 1 unique data files

[CPU: 342.4 MB] Import paths were unique at level -1

[CPU: 342.4 MB] Example imported relative path: J144/imported/job039_particles.mrc

[CPU: 342.4 MB] Reading MRC file headers to check shape…

[CPU: 342.4 MB] Done importing.

[CPU: 342.3 MB] --------------------------------------------------------------

I tried to run a homogeneous refinement with the imported particles but failed :
[CPU: 90.6 MB] Traceback (most recent call last):
File “cryosparc2_worker/cryosparc2_compute/run.py”, line 52, in cryosparc2_compute.run.main
File “cryosparc2_compute/jobs/runcommon.py”, line 823, in check_default_inputs
assert False, 'Non-optional inputs from the following input groups and their slots are not connected: ’ + missing_inputs + ‘. Please connect all required inputs.’
AssertionError: Non-optional inputs from the following input groups and their slots are not connected: particles.ctf. Please connect all required inputs.

Any help will be highly appreciated.
Rashed

olibclarke · August 3, 2020, 2:21pm

Hi Rashed, the key is this bit:

For some reason cryosparc is not reading the data_optics table. Try setting the voltage, Cs and AC under “Microscope Parameter Overrides” when you import the star file.

Cheers
Oli

rashed · August 3, 2020, 6:08pm

Hi Oli,
thanks very much for your reply. I did notice that CS didn’t read my data_optics table, which is odd. After including those manually, it did import despite two warnings :

[CPU: 250.5 MB] star file fields:

[CPU: 250.5 MB] Index([u’rlnImageName’, u’rlnMicrographName’, u’rlnCoordinateX’, u’rlnCoordinateY’, u’rlnAnglePsi’, u’rlnOriginX’, u’rlnOriginY’, u’rlnMagnification’, u’rlnDetectorPixelSize’, u’rlnDefocusU’, u’rlnDefocusV’, u’rlnDefocusAngle’, u’rlnPhaseShift’, u’rlnCtfBfactor’, u’rlnOpticsGroup’, u’rlnClassNumber’, u’rlnOriginXAngst’, u’rlnOriginYAngst’, u’rlnGroupNumber’, u’rlnAngleRot’, u’rlnAngleTilt’, u’rlnNormCorrection’, u’rlnLogLikeliContribution’, u’rlnMaxValueProbDistribution’, u’rlnNrOfSignificantSamples’],
dtype=‘object’)

[CPU: 250.5 MB] Reading particle data locations…

[CPU: 250.5 MB] Reading rlnImageName to get indices and paths…

[CPU: 342.1 MB] Searching for linked data files…

[CPU: 342.3 MB] Compiling CTF information…

[CPU: 342.4 MB] Warning: No micrographs were connected as inputs, so correspondences cannot be found, so output will not contain pick locations for particles.

[CPU: 342.4 MB] Warning: The input star file does not contain rlnAngleRot, rlnAngleTilt, rlnAnglePsi, rlnOriginX, rlnOriginY, rlnRandomSubset. Output will not contain particle orientations.

[CPU: 342.4 MB] --------------------------------------------------------------

I could still run the refinement because X,Y coordinates were recognized but I’d like CS to be able to read my Relion refined euler angles.

Best,
Rashed

olibclarke · August 3, 2020, 7:01pm

See here: Importing symmetry expanded particles from Relion 3.1 doesn't work [bug]

rashed · August 3, 2020, 7:30pm

I see. So, cryoSPARC still doesn’t recognize coordinate offsets ( _rlnOriginXAngst , _rlnOriginYAngst ) or the angles rlnAngleRot, rlnAngleTilt, rlnAnglePsi.

olibclarke · August 3, 2020, 7:51pm

No - it just doesn’t recognize _rlnOriginXAngst or _rlnOriginYAngst, and as a consequence ignores the angles. It recognizes _rlnOriginX & _rlnOriginY (which are in pixels and were used until Relion 3).

If your pixel size is ~1Å you can probably safely just change the name of these columns without ill effect, otherwise you may need to adjust the values of each parameter (e.g. using awk).

Cheers
Oli

rashed · August 3, 2020, 8:30pm

Hi Oli,
I don’t think offsets were my issue. Just the euler angles. Anyway it was simply a formatting issue. I just reorganized the columns and it worked like a charm.
_rlnImageName #1
_rlnMicrographName #2
_rlnCoordinateX #3
_rlnCoordinateY #4
_rlnAngleRot #5
_rlnAngleTilt #6
_rlnAnglePsi #7
_rlnOriginX #8
_rlnOriginY #9
_rlnMagnification #10
_rlnDetectorPixelSize #11
_rlnDefocusU #12
_rlnDefocusV #13
_rlnDefocusAngle #14
_rlnPhaseShift #15
_rlnCtfBfactor #16
_rlnOpticsGroup #17
_rlnRandomSubset #18
_rlnClassNumber #19

Thanks for getting back.

Best,
Rashed

olibclarke · August 3, 2020, 8:35pm

Hi Rashed - no problem! The offsets were your issue, guaranteed - it is because it could not recognize _rlnOriginYAngst etc that it then ignored the angles. Now when you change to rlnOriginX/rlnOriginY it works