Using pyem to relion re-extract

Import and Export
1

Hi everyone. I wanted to use pyem to re-extract my local-refine particles in relion.
Firstly, I used the command

csparc2star.py cryosparc_P359_J540_006_particles.cs P359_J540_passthrough_particles.cs P359_J540_particles.star

Then, I used match *_gctf.star by the micrographs’ name. I combined the rlnCtfImage, rlnVoltage, rln SphericalAberration, rlnAmplitudeContrast, rlnMagnification, rlnDetectorPixelSize, rlnCtfFigureOfMerit, rlnFinalResolution in gctf.star with the P359_J540_particles.star.

And I got the star file like this:
data_

loop_
_rlnMicrographName
_rlnCoordinateX
_rlnCoordinateY
_rlnAngleRot
_rlnAngleTilt
_rlnAnglePsi
_rlnDefocusU
_rlnDefocusV
_rlnDefocusAngle
_rlnPhaseShift
_rlnCtfBfactor
_rlnRandomSubset
_rlnClassNumber
_rlnCtfImage
_rlnVoltage
_rlnSphericalAberration
_rlnAmplitudeContrast
_rlnMagnification
_rlnDetectorPixelSize
_rlnCtfFigureOfMerit
_rlnFinalResolution
stack_0007_cor_DW.mrc 1502 753 -1.850596 57.413094 7.797913 23543.001953 23484.187500 -77.690010 -2.661191 0.000000 1 1 stack_0007_cor_DW.ctf:mrc 300.000000 2.700000 0.100000 45998.160156 5.000000 0.143954 2.948065

Finally, I re-extract particles in relion. An error happened in relion. I didn’t know why the coordinates were outside micrographs. Actually, I tried deleting these particles. However, there were too many particles having the problem. I think maybe something went wrong.

ERROR:
Preprocessing::extractParticlesFromOneFrame ERROR: particle1 lies completely outside micrograph stack_2221_cor_DW.mrc
micrograph x,y,z,n-size= 5760 , 4092 , 1 , 1
particle position= 299 , 4597
in: /ShareNV/app/relion/3.0.6_cuda10.1/src/preprocessing.cpp, line 881
micrograph x,y,z,n-size= 5760 , 4092 , 1 , 1
particle position= 2066 , 5429
in: /ShareNV/app/relion/3.0.6_cuda10.1/src/preprocessing.cpp, line 881

2

Hi @Yuqi,

Did you motion correct and pick in cryoSPARC? If so, you may need to use the --swapxy and possibly also --inverty flags for pyem in order that the coordinates in the star file match relion conventions.

Cheers
Oli

3

OK. Thank you very much! Can you give me a intact command? I am afraid that I will take mistakes again because I am new in pyem.

4

I have attempted your advice. And there was no parameter called “–inverty”. I didn’t know whether my pyem version was too old.

5

Hi Yuqi,
Be aware you probably cannot copy and paste this command as the double-hyphen may be lost.

I never used the --inverty command, but was able to get around this by using awk to manually invert the y coordinate in the star file as described here (if you don’t want to take on updating yourself).

I would do a small test after generating new star file - relion subset select, 1000 particles per subset, extract with binning and ensure you hit real particles.

1 Like
6

Example command:

csparc2star.py cryosparc_P1_J1467_012_particles.cs P1_J1467_passthrough_particles.cs J1467_swapXY_invertY.star --swapxy --inverty

Try with swapxy, or swapxy & inverty. Re-extract a small subset in relion, then import into cryosparc and check if particles look centered. Good luck!

7

OK. Thank you very much. However, my pyem didn’t have --inverty.

8

@olibclarke I am so sorry that how can I get the inverty? Should I install new version of pyem?

9

I find that there were --invertx & --inverty in the newest version.

1 Like
10

Hello!
I used the command
csparc2star.py cryosparc_P359_J540_006_particles.cs P359_J540_passthrough_particles.cs P359_J540_particles.star --swapxy --inverty (or plus --invertx)

And I could do the no-recenter re-extract in relion. That was very good without any error imformation!!

After comparing the star files, I found out the OriginX & OriginY are not convert.
I didn’t know how to revise the OriginXAngst & OriginYAngst in star file generated by pyem. I just did ${OriginXAngst} / ${MyPixelSize (unit: angstrom)} to get the OriginX. I guessed the rlnOriginX & rlnOriginY should also be --swapped? maybe? and maybe --inverty? maybe --invertx? And maybe I didn’t know how to do those.

The head of star file after csparc2star.py:

data_optics

loop_
_rlnVoltage #1
_rlnSphericalAberration #2
_rlnAmplitudeContrast #3
_rlnOpticsGroup #4
_rlnImageSize #5
_rlnImagePixelSize #6
_rlnImageDimensionality #7
300.000000 2.906047 0.100000 32 384 1.087000 2
300.000000 2.871463 0.100000 33 384 1.087000 2
300.000000 2.856842 0.100000 34 384 1.087000 2

data_particles

loop_

_rlnImageName #1
_rlnMicrographName #2
_rlnCoordinateX #3
_rlnCoordinateY #4
_rlnAngleRot #5
_rlnAngleTilt #6
_rlnAnglePsi #7
_rlnOriginXAngst #8
_rlnOriginYAngst #9
_rlnDefocusU #10
_rlnDefocusV #11
_rlnDefocusAngle #12
_rlnPhaseShift #13
_rlnCtfBfactor #14
_rlnOpticsGroup #15
_rlnRandomSubset #16
_rlnClassNumber #17
000002@J529/extract/20210719_stack_0007_cor_DW_particles.mrc J469/imported/20210719_stack_0007_cor_DW.mrc 2114 3557 -1.850596 57.413094 7.797913 13.618072 -1.656782 23543.001953 23484.187500 -77.690010 -2.661191 0.000000 32 1 1

And the head of star file I deleted, added, parsed some columns:

data_

loop_
_rlnImageName #1
_rlnMicrographName #2
_rlnCoordinateX #3
_rlnCoordinateY #4
_rlnAngleRot #5
_rlnAngleTilt #6
_rlnAnglePsi #7
_rlnOriginX #8
_rlnOriginY #9
_rlnDefocusU #10
_rlnDefocusV #11
_rlnDefocusAngle #12
_rlnPhaseShift #13
_rlnCtfBfactor #14
_rlnRandomSubset #15
_rlnClassNumber #16
_rlnVoltage #17
_rlnSphericalAberration #18
_rlnAmplitudeContrast #19
_rlnMagnification #20
_rlnDetectorPixelSize #21

000002@/Share/home/cryosparc/P359/J529/extract/20210719_stack_0007_cor_DW_particles.mrcs stack_0007_cor_DW.mrc 3646 3557 -1.850596 57.413094 7.797913 12.64 11.73 23543.001953 23484.187500 -77.690010 -2.661191 0.000000 1 1 300.000000 2.700000 0.100000 45998.160156 5.000000

relion extract :
Command:
which relion_preprocess_mpi --i micrographs_all_gctf.star --reextract_data_star micrographs_particles_NEW.star --recenter --recenter_x 0 --recenter_y 0 --recenter_z 0 --part_star Extract/job044/particles.star --part_dir Extract/job044/ --extract --extract_size 384 --scale 128 --norm --bg_radius 48 --white_dust -1 --black_dust -1 --invert_contrast --only_extract_unfinished

There was also an error:

Warning: There are only 9 particles in micrograph stack_1198_cor_DW.mrc. Consider joining multiple micrographs into one group.
Warning: There are only 8 particles in micrograph stack_2314_cor_DW.mrc. Consider joining multiple micrographs into one group.
Warning: There are only 8 particles in micrograph stack_0007_cor_DW.mrc. Consider joining multiple micrographs into one group.
Warning: There are only 9 particles in micrograph stack_2674_cor_DW.mrc. Consider joining multiple micrographs into one group.
micrograph x,y,z,n-size= 5760 , 4092 , 1 , 1
particle position= 2147488825 , 2147485082
in: /ShareNV/app/relion/3.0.6_cuda10.1/src/preprocessing.cpp, line 881
=== Backtrace ===
/ShareNV/app/relion/3.0.6_cuda10.1/build/bin/relion_preprocess_mpi(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x41d691]
/ShareNV/app/relion/3.0.6_cuda10.1/build/bin/relion_preprocess_mpi(ZN13Preprocessing28extractParticlesFromOneFrameER13MetaDataTable8FileNameiiiS2_S2_RllRdS4_S4_S4+0x265f) [0x44371f]
/ShareNV/app/relion/3.0.6_cuda10.1/build/bin/relion_preprocess_mpi(_ZN13Preprocessing31extractParticlesFromFieldOfViewE8FileNamel+0x657) [0x444a37]
/ShareNV/app/relion/3.0.6_cuda10.1/build/bin/relion_preprocess_mpi(_ZN16PreprocessingMpi19runExtractParticlesEv+0x34c) [0x45482c]
/ShareNV/app/relion/3.0.6_cuda10.1/build/bin/relion_preprocess_mpi(_ZN16PreprocessingMpi3runEv+0x25) [0x454c95]
/ShareNV/app/relion/3.0.6_cuda10.1/build/bin/relion_preprocess_mpi(main+0x3b) [0x4177cb]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x2b8547bbd3d5]
/ShareNV/app/relion/3.0.6_cuda10.1/build/bin/relion_preprocess_mpi() [0x417eaf]

ERROR:
Preprocessing::extractParticlesFromOneFrame ERROR: particle1 lies completely outside micrograph stack_1198_cor_DW.mrc

MPI_ABORT was invoked on rank 3 in communicator MPI_COMM_WORLD
with errorcode 1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

[nv04:409956] 27 more processes have sent help message help-mpi-btl-openib.txt / ib port not selected
[nv04:409956] Set MCA parameter “orte_base_help_aggregate” to 0 to see all help / error messages
[nv04:409956] 26 more processes have sent help message help-mpi-btl-openib.txt / error in device init

Can someone help me?

11

My problem was how to do the recenter re-extract in relion. Thank you very much!