Hi,
I was wondering what is the best way to export particles back to Relion with the new version 2 software?
Thanks,
Best,
Nicolas
1 Like
Use csparc2star.py from Dan Asarnow’s pyem package, same as for v0
Cheers
Oli
Thanks -
Are you using the exact same command as with the old cvs from v0.6 but with one of the “.cs” files instead? It looks to me like the format is different from what it used to be though.
The new version of csparc2star.py on github has been updated to handle v2 meta data.
The file format is different, but Dan has updated csparc2star.py to deal with it - using the same command it works fine in my hands (you may need to edit the star file afterwards though so it points to the original files rather than the cryosparc cache)
Cheers
Oli
Great, thanks for the info!
Cheers,
N.
Hi Oli,
I am trying to export the local motion corrected particles (from V2) in RELION. Which file is the one to be exported? local_motion_corrected_particles.cs or the passthrough_particles.cs (what are the passthrough particles?)
When I use the old pyem script (which was working well with V0.6) and do:
csparc2star.py local_motion_corrected_particles.cs local_motion_corrected_particles.star
It gives me the following error:
/opt/python/Virtualenv/2.7.12/pyem/lib/python2.7/site-packages/scipy/spatial/init.py:95: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .ckdtree import *
/opt/python/Virtualenv/2.7.12/pyem/lib/python2.7/site-packages/scipy/spatial/init.py:96: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .qhull import *
Traceback (most recent call last):
File “/opt/pyem/0.2/bin/csparc2star.py”, line 105, in
sys.exit(main(parser.parse_args()))
File “/opt/pyem/0.2/bin/csparc2star.py”, line 67, in main
df = metadata.parse_cryosparc_2_cs(cs, passthrough=args.passthrough, minphic=args.minphic)
File “/opt/pyem/0.2/pyem/metadata.py”, line 240, in parse_cryosparc_2_cs
df[“rlnDefocusAngle”] = np.rad2deg(df[“rlnDefocusAngle”])
File “/opt/python/Virtualenv/2.7.12/pyem/lib/python2.7/site-packages/pandas/core/frame.py”, line 2688, in getitem
return self._getitem_column(key)
File “/opt/python/Virtualenv/2.7.12/pyem/lib/python2.7/site-packages/pandas/core/frame.py”, line 2695, in _getitem_column
return self._get_item_cache(key)
File “/opt/python/Virtualenv/2.7.12/pyem/lib/python2.7/site-packages/pandas/core/generic.py”, line 2489, in _get_item_cache
values = self._data.get(item)
File “/opt/python/Virtualenv/2.7.12/pyem/lib/python2.7/site-packages/pandas/core/internals.py”, line 4115, in get
loc = self.items.get_loc(item)
File “/opt/python/Virtualenv/2.7.12/pyem/lib/python2.7/site-packages/pandas/core/indexes/base.py”, line 3080, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File “pandas/_libs/index.pyx”, line 140, in pandas._libs.index.IndexEngine.get_loc
File “pandas/_libs/index.pyx”, line 162, in pandas._libs.index.IndexEngine.get_loc
File “pandas/_libs/hashtable_class_helper.pxi”, line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
File “pandas/_libs/hashtable_class_helper.pxi”, line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: ‘rlnDefocusAngle’
—Are some changes to be done?
Best,
Zuhaib
Hi Zuhaib
You want to use the particles.cs file
If you haven’t updated pyem in a while, then I would do that first - Dan updated it relatively recently to accomodate the new format
Cheers
Oli
OK! Will update pyem.
Thanks.
Zuhaib
I just updated the pyem. Still the same error.
The input files I have are local_motion_corrected_particles.cs and passthrough_particles.cs
there is no particles.cs file (for the local motion correction job type)
-Zuhaib
OK I would check with Dan asarnow (who wrote pyem - @DanielAsarnow )
And I would also double check that pyem is still working with 0.6 files (in case something about your python environment has changed)
Cheers
Oli
I may be doing something wrong. Is the generic command cparc2star.py file.cs file.star vaild or do I need other usage flags? like --passthrough --class etc
no that should be fine
@Zuhaib I don’t think you have the latest version given the rlnDefocusAngle error. There is currently an explicit check for that field (which is recalculated when missing). The earlier import errors also indicate some problem with the version of scipy (and perhaps numpy as well).
I see you have used a virtualenv - this is fine, but how have you installed pyem and the dependencies? Did you follow something similar to the installation tutorial on my github wiki?
After we’re sure these versions are right and you can convert, say, a normal refinement job file, we can look into potential issues with align_parts_lmbfgs output. (But I doubt there are any).
Best,
-da
@DanielAsarnow I tried it again on a different job (Homogeneous refinement). Used the command
csparc2star.py cryosparc_P1_J21_004_particles.cs cryosparc_P1_J21_004_particles.star
It was able to make a star file! The following were the warnings though
/opt/python/Virtualenv/2.7.12/pyem/lib/python2.7/site-packages/scipy/spatial/init.py:95: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .ckdtree import *
/opt/python/Virtualenv/2.7.12/pyem/lib/python2.7/site-packages/scipy/spatial/init.py:96: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .qhull import *
The star file header looks like:
data_images
loop_
_rlnAnglePsi #1
_rlnAngleRot #2
_rlnAngleTilt #3
_rlnClassNumber #4
_rlnCtfFigureOfMerit #5
_rlnCtfMaxResolution #6
_rlnDefocusAngle #7
_rlnDefocusU #8
_rlnDefocusV #9
_rlnDetectorPixelSize #10
_rlnImageName #11
_rlnMagnification #12
_rlnOriginX #13
_rlnOriginY #14
_rlnPhaseShift #15
_rlnRandomSubset #16
_rlnSphericalAberration #17
_rlnVoltage #18
123.550316 -134.1915 127.61527 1 0.243878 3.317407 64.88143 18452.254 18227.82 1.06 000001@J16/localmotioncorrected/20180730_1020_A014_G000_H070_D002_particles_local_aligned.mrc 10000.0 -5.45625 -18.50625 0.0 1 2.7 300.0
Is this working alright now? If so, then there is an issue with local motion job type.
Best,
Zuhaib
@Zuhaib Can you post the output of these commands?
python -c 'import scipy; print scipy.__version__'
python -c 'import numpy; print numpy.__version__'
Also, if you are willing to send me your local motion .cs file (or perhaps one with a small number of particles), then I can inspect the .cs directly and see what’s going on. Thanks.
@DanielAsarnow I have the same error message as @Zuhaib’s
I installed pyem using conda follow your github instruction. Here is a link to my local motion.cs file:
Here is my output:
(pyem) spuser@spgpu:/data/zhou/test_cryosparcV2/J9$ python -c ‘import scipy; print scipy.version’
1.1.0
(pyem) spuser@spgpu:/data/zhou/test_cryosparcV2/J9$ python -c ‘import numpy; print numpy.version’
1.14.5
(pyem) spuser@spgpu:/data/zhou/test_cryosparcV2/J9$ csparc2star.py local_motion_corrected_particles.cs 1.star
Traceback (most recent call last):
File “/home/spuser/apps/pyem/csparc2star.py”, line 115, in
sys.exit(main(parser.parse_args()))
File “/home/spuser/apps/pyem/csparc2star.py”, line 67, in main
df = metadata.parse_cryosparc_2_cs(cs, passthrough=args.passthrough, minphic=args.minphic)
File “/home/spuser/apps/pyem/pyem/metadata.py”, line 249, in parse_cryosparc_2_cs
df[star.Relion.DEFOCUSANGLE] = np.rad2deg(np.arctan2(df[star.Relion.DEFOCUSV], df[star.Relion.DEFOCUSU]))
File “/home/spuser/.conda/envs/pyem/lib/python2.7/site-packages/pandas/core/frame.py”, line 2688, in getitem
return self._getitem_column(key)
File “/home/spuser/.conda/envs/pyem/lib/python2.7/site-packages/pandas/core/frame.py”, line 2695, in _getitem_column
return self._get_item_cache(key)
File “/home/spuser/.conda/envs/pyem/lib/python2.7/site-packages/pandas/core/generic.py”, line 2489, in _get_item_cache
values = self._data.get(item)
File “/home/spuser/.conda/envs/pyem/lib/python2.7/site-packages/pandas/core/internals.py”, line 4115, in get
loc = self.items.get_loc(item)
File “/home/spuser/.conda/envs/pyem/lib/python2.7/site-packages/pandas/core/indexes/base.py”, line 3080, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File “pandas/_libs/index.pyx”, line 140, in pandas._libs.index.IndexEngine.get_loc
File “pandas/_libs/index.pyx”, line 162, in pandas._libs.index.IndexEngine.get_loc
File “pandas/_libs/hashtable_class_helper.pxi”, line 1492, in pandas._libs.hashtable.PyObjectHashTable.get_item
File “pandas/_libs/hashtable_class_helper.pxi”, line 1500, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: ‘rlnDefocusV’
(pyem) spuser@spgpu:/data/zhou/test_cryosparcV2/J9$
These are the outputs:
python -c ‘import scipy; print scipy.version’ : 1.1.0
python -c ‘import numpy; print numpy.version’: 1.15.0
Sure, Daniel, I shared the local motion .cs files with you.
I tried exporting the particles from a “select 2D” job, and it also gives a similar error.
Cheers,
Zuhaib
Was the problem fixed already? I get the same error message.
Best,
David
Hi David,
Nope. The issue is still there. The script seems to work using output from a refinement job, not not from a local motion or 2d selection job.