Troubles to convert cs to star with pyem from local refinement from v2.12.4

Troubleshooting
1

Dear all,
I followed the installation guide from the pyem wiki and I updated pyem today. I try to convert a local refinement cs file (CryoSparc 2.12.4) to star with the angles and shifts. I got this error message:

csparc2star.py --copy-micrograph-coordinates …/…/Extract/job014/particles-cS.star cryosparc_P46_J60_007_particles.cs passthrough_particles.cs cryosparc_P46_J60_007_particles.star

Traceback (most recent call last):
File “/home/papai/bin/csparc2star.py”, line 105, in <module>
sys.exit(main(parser.parse_args()))
File “/home/papai/bin/csparc2star.py”, line 68, in main
df = star.smart_merge(df, coord_star, fields=fields, key=key)
File “/home/papai/pyem/pyem/star.py”, line 101, in smart_merge
s2 = s2.set_index(key, drop=False)
File “/home/papai/.conda/envs/pyem-new/lib/python3.7/site-packages/pandas/core/frame.py”, line 4411, in set_index
raise KeyError(“None of {} are in the columns”.format(missing))
KeyError: 'None of [None] are in the columns’

The particles-cS.star is coming from relion 3.1 where I removed the the optic_groups section. The first lines look like this:

data_

loop_

_rlnCoordinateX #1
_rlnCoordinateY #2
_rlnClassNumber #3
_rlnAnglePsi #4
_rlnAutopickFigureOfMerit #5
_rlnImageName #6
_rlnMicrographName #7
_rlnOpticsGroup #8
_rlnCtfMaxResolution #9
_rlnDefocusU #10
_rlnDefocusV #11
_rlnDefocusAngle #12
_rlnCtfBfactor #13
_rlnCtfScalefactor #14
_rlnPhaseShift #15
4175.000000 3796.500000 -999 -999.00000 -999.00000 000001@Extract/job014/average/Complex_00000_Nov28_18.31.41.mrcs average/Complex_00000_Nov28_18.31.41.mrc 1 5.200000 35426.700000 32715.900000 7.000000 0.000000 1.000000 0.000000

If I leave out the micrographs star file then I have an error like that:

csparc2star.py cryosparc_P46_J60_007_particles.cs passthrough_particles.cs cryosparc_P46_J60_007_particles.star

Traceback (most recent call last):
File “/home/papai/.conda/envs/pyem-new/lib/python3.7/site-packages/pandas/core/indexes/base.py”, line 2897, in get_loc
return self._engine.get_loc(key)
File “pandas/_libs/index.pyx”, line 107, in pandas._libs.index.IndexEngine.get_loc
File “pandas/_libs/index.pyx”, line 131, in pandas._libs.index.IndexEngine.get_loc
File “pandas/_libs/hashtable_class_helper.pxi”, line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
File “pandas/_libs/hashtable_class_helper.pxi”, line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: ‘rlnMicrographName’
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/home/papai/bin/csparc2star.py”, line 105, in <module>
sys.exit(main(parser.parse_args()))
File “/home/papai/bin/csparc2star.py”, line 79, in main
star.write_star(args.output, df, resort_records=True)
File “/home/papai/pyem/pyem/star.py”, line 288, in write_star
df = sort_records(df, inplace=True)
File “/home/papai/pyem/pyem/star.py”, line 420, in sort_records
df = natsort_values(df, Relion.MICROGRAPH_NAME, inplace=True)
File “/home/papai/pyem/pyem/util/util.py”, line 132, in natsort_values
idx = np.array(natsort.index_natsorted(df[col]))
File “/home/papai/.conda/envs/pyem-new/lib/python3.7/site-packages/pandas/core/frame.py”, line 2995, in getitem
indexer = self.columns.get_loc(key)
File “/home/papai/.conda/envs/pyem-new/lib/python3.7/site-packages/pandas/core/indexes/base.py”, line 2899, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File “pandas/_libs/index.pyx”, line 107, in pandas._libs.index.IndexEngine.get_loc
File “pandas/_libs/index.pyx”, line 131, in pandas._libs.index.IndexEngine.get_loc
File “pandas/_libs/hashtable_class_helper.pxi”, line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
File “pandas/_libs/hashtable_class_helper.pxi”, line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: ‘rlnMicrographName’

I don’t know what I miss here. Any help would be gratly appreciated.

Best regards,
Gabor

2

It seems like your .cs files have no micrograph names. I at least pushed a fix that doesn’t try sorting if no stack or micrograph names are found.

You can run with --loglevel debug and it might be more clear what the problem is - either the fields aren’t what you expect (find more or different .cs files to convert) or a bug is making a field get dropped (and I can fix it). Thanks.

1 Like
3

Thank you Daniel!

Now it works when I don’t want to copy the micrograph coordinates.
I realized the problem is that I tried to convert an alignment file where I used subtracted particles. It can’t find the corresponding fields there since the image name is changed.
Converting cs file from alignment with original stack worked well if I didn’t ask for micrograph details, if I included it I had an error:

Detected CryoSPARC 2+ .cs file
Reading primary file
Assigning parameters 2D classes or single 3D class
Reading auxiliary file passthrough_particles.cs
Assigning parameters 2D classes or single 3D class
Assigning skew angle from 2D classification
Trying to merge: alignments2D/split, alignments2D/shift, alignments2D/pose, alignments2D/psize_A, alignments2D/error, alignments2D/error_min, alignments2D/resid_pow, alignments2D/slice_pow, alignments2D/image_pow, alignments2D/cross_cor, alignments2D/alpha, alignments2D/alpha_min, alignments2D/weight, alignments2D/pose_ess, alignments2D/shift_ess, alignments2D/class_posterior, alignments2D/class, alignments2D/class_ess
Merging:
Directly copied fields: ucsfImagePath, rlnDefocusU, rlnOpticsGroup, rlnCtfBfactor, ucsfImageIndex, rlnAmplitudeContrast, rlnDefocusV, rlnDefocusAngle, rlnPhaseShift, rlnSphericalAberration, rlnDetectorPixelSize, ucsfUid, rlnVoltage, rlnRandomSubset, rlnOriginX, rlnOriginY, rlnAngleRot, rlnAngleTilt, rlnAnglePsi, rlnClassNumber, rlnMagnification
Converting DEFOCUSANGLE from degrees to radians
Converting PHASESHIFT from degrees to radians
Changing RANDOMSUBSET to 1-based index
Changing CLASS to 1-based index
Converting Rodrigues coordinates to Euler angles
Converted Rodrigues coordinates to Euler angles
Coordinates merge key: None
Traceback (most recent call last):
File “/home/papai/bin/csparc2star.py”, line 105, in
sys.exit(main(parser.parse_args()))
File “/home/papai/bin/csparc2star.py”, line 68, in main
df = star.smart_merge(df, coord_star, fields=fields, key=key)
File “/home/papai/pyem/pyem/star.py”, line 108, in smart_merge
s2 = s2.set_index(key, drop=False)
File “/home/papai/.conda/envs/pyem-new/lib/python3.7/site-packages/pandas/core/frame.py”, line 4411, in set_index
raise KeyError(“None of {} are in the columns”.format(missing))
KeyError: ‘None of [None] are in the columns’

I guess the reason is the missing micrograph names field. I don’t know why I don’t have micrograph names in the cs files.

Best regards,
Gabor

4

You should be able to use an earlier .cs file in the workspace, say from the extraction.

5

I fixed the underlying issue. Please see this post for all the details. I used that one for the detailed answer because I thought the thread title better matched the actual problem. Thanks for helping to debug!