Csparc2star empty starfile

Import and Export
1

Dear all,

I am trying to convert the output of Cryosparc (from a Hetero-refinement job) to a star file. The particles were originally imported from Warp.

If I run:

csparc2star.py Cryosparc/P1/J72/cryosparc_P1_J72_class_01_00062_particles.cs cryosparc_class1.star

I have the following error message:

/mnt/auto-storage/software/pyem/0.5/pyem/pyem/mrc.py:57: SyntaxWarning: “is” with a literal. Did you mean “==”?
elif origin is “center”:
Defocus values not found
Traceback (most recent call last):
File “/mnt/storage/software/pyem/0.5/lib/python3.9/site-packages/pandas/core/indexes/base.py”, line 3080, in get_loc
return self._engine.get_loc(casted_key)
File “pandas/_libs/index.pyx”, line 70, in pandas._libs.index.IndexEngine.get_loc
File “pandas/_libs/index.pyx”, line 101, in pandas._libs.index.IndexEngine.get_loc
File “pandas/_libs/hashtable_class_helper.pxi”, line 4554, in pandas._libs.hashtable.PyObjectHashTable.get_item
File “pandas/_libs/hashtable_class_helper.pxi”, line 4562, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: ‘rlnMicrographName’

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File “/mnt/storage/software/pyem/0.5/pyem/csparc2star.py”, line 105, in
sys.exit(main(parser.parse_args()))
File “/mnt/storage/software/pyem/0.5/pyem/csparc2star.py”, line 79, in main
star.write_star(args.output, df, resort_records=True)
File “/mnt/auto-storage/software/pyem/0.5/pyem/pyem/star.py”, line 295, in write_star
df = sort_records(df, inplace=True)
File “/mnt/auto-storage/software/pyem/0.5/pyem/pyem/star.py”, line 427, in sort_records
df = natsort_values(df, Relion.MICROGRAPH_NAME, inplace=True)
File “/mnt/auto-storage/software/pyem/0.5/pyem/pyem/util/util.py”, line 131, in natsort_values
idx = np.array(natsort.index_natsorted(df[col]))
File “/mnt/storage/software/pyem/0.5/lib/python3.9/site-packages/pandas/core/frame.py”, line 3024, in getitem
indexer = self.columns.get_loc(key)
File “/mnt/storage/software/pyem/0.5/lib/python3.9/site-packages/pandas/core/indexes/base.py”, line 3082, in get_loc
raise KeyError(key) from err
KeyError: ‘rlnMicrographName’

I tried to connect the original images from the Warp star file using:

csparc2star.py Cryosparc/P1/J72/cryosparc_P1_J72_class_01_00062_particles.cs cryosparc_class1.star --copy-micrograph-coordinates goodparticles_BoxNet2Mask_20180918.star

Which ended with a new error:

/mnt/auto-storage/software/pyem/0.5/pyem/pyem/mrc.py:57: SyntaxWarning: “is” with a literal. Did you mean “==”?
elif origin is “center”:
Defocus values not found
/mnt/auto-storage/software/pyem/0.5/pyem/pyem/star.py:376: FutureWarning: Columnar iteration over characters will be deprecated in future releases.
df[UCSF.IMAGE_INDEX], df[UCSF.IMAGE_PATH] =
/mnt/auto-storage/software/pyem/0.5/pyem/pyem/star.py:385: FutureWarning: Columnar iteration over characters will be deprecated in future releases.
df[UCSF.IMAGE_ORIGINAL_INDEX], df[UCSF.IMAGE_ORIGINAL_PATH] =
Traceback (most recent call last):
File “/mnt/storage/software/pyem/0.5/pyem/csparc2star.py”, line 105, in
sys.exit(main(parser.parse_args()))
File “/mnt/storage/software/pyem/0.5/pyem/csparc2star.py”, line 68, in main
df = star.smart_merge(df, coord_star, fields=fields, key=key)
File “/mnt/auto-storage/software/pyem/0.5/pyem/pyem/star.py”, line 108, in smart_merge
s2 = s2.set_index(key, drop=False)
File “/mnt/storage/software/pyem/0.5/lib/python3.9/site-packages/pandas/core/frame.py”, line 4727, in set_index
raise KeyError(f"None of {missing} are in the columns")
KeyError: ‘None of [None] are in the columns’

I don’t really understand what this last message point out to.

I tried also to connect as .cs file from an Import micrographs job:
csparc2star.py Cryosparc/P1/J72/cryosparc_P1_J72_class_01_00062_particles.cs Cryosparc/P1/J79/imported_micrographs.cs cryosparc_class1.star

I got the following message:

Defocus values not found

It does generate the starfile, which is empty (apart from the header, but there is no particle listed).
Any idea how to solve this?

Cheers,
Alexandre

2

Hi Alexandre,

Did you try exporting the particles in cryosparc and supplying the passthrough file? It should come from an Exports directory that’s generated by cryosparc.

Also, when you imported the particles from warp did you supply the particle data path?

Best,

Ryan

3

Hi Ryan,

Thanks for the answer.

Which passthrough are you speaking about? If I export the Hetero-refinement job, I have one cs file for the class 1. If I tried to use convert it:

csparc2star.py P1_J72_particles_class_1_exported.cs P1_J72_particles_class_1_exported.star

I have the following error message:

/mnt/auto-storage/software/pyem/0.5/pyem/pyem/mrc.py:57: SyntaxWarning: “is” with a literal. Did you mean “==”?
elif origin is “center”:
/mnt/auto-storage/software/pyem/0.5/pyem/pyem/metadata.py:332: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify ‘dtype=object’ when creating the ndarray.
df[model[k]] = pd.DataFrame(np.array(
Columns must be same length as key
Traceback (most recent call last):
File “/mnt/storage/software/pyem/0.5/pyem/csparc2star.py”, line 42, in main
df = metadata.parse_cryosparc_2_cs(cs, passthroughs=args.input[1:], minphic=args.minphic, boxsize=args.boxsize, swapxy=args.swapxy)
File “/mnt/auto-storage/software/pyem/0.5/pyem/pyem/metadata.py”, line 390, in parse_cryosparc_2_cs
df = cryosparc_2_cs_model_parameters(cs, df, minphic=minphic)
File “/mnt/auto-storage/software/pyem/0.5/pyem/pyem/metadata.py”, line 332, in cryosparc_2_cs_model_parameters
df[model[k]] = pd.DataFrame(np.array(
File “/mnt/storage/software/pyem/0.5/lib/python3.9/site-packages/pandas/core/frame.py”, line 3160, in setitem
self._setitem_array(key, value)
File “/mnt/storage/software/pyem/0.5/lib/python3.9/site-packages/pandas/core/frame.py”, line 3189, in _setitem_array
raise ValueError(“Columns must be same length as key”)
ValueError: Columns must be same length as key
Required fields could not be mapped. Are you using the right input file(s)?

I ran an import job pointing to the Warp goodparticles star file. My mistake back then was to not link it to the micrographs at that step. Is there any way to reconnect both?

Cheers,
Alexandre

4

Hi Alexandre,

The pyem command should look more like this:
csparc2star.py --swapxy jXXX_particles.cs jXXX_passthrough_particles.cs from_cryosparc.star

The particles.cs and passthrough_particles.cs should be in a directory called Exports that was generated when you ran “export job” in cryosparc.

but I suspect even if you get it to work you will run into this issue which is why copy micrograph coordinates isn’t working. Retain particle data when importing star file

Best,

Ryan