Csparc2star output Relion compatibility

nkumar · August 11, 2020, 11:06pm

I need help. Thanks in advance.

Hi Daniel:
My colleague Elmar has been using CryoSPRAC at OHSU and ran into an issue (using cs files in Relion after converting it to star). I could not resolve it myself. His notes on the issue:

I run CryoSPARC v2.15.0+200710.
I try to convert cs files to star files that I would be able to read, either with relion 3.1.0-commit-72f783 or relion 3.0.7

I generated star files in 3 ways:

executing an export job in cryosparc and then:
csparc2star.py P2_J12_particles_exported.cs P2_J12_particles_exported_relion31.star
csparc2star.py -r2 P2_J12_particles_exported.cs P2_J12_particles_exported_relion30.star
csparc2star.py cryosparc_P2_J12_020_particles.cs P2_J12_passthrough_particles.cs cryosparc_P2_J12_020_particles_passthrough_relion31.star
csparc2star.py -r2 cryosparc_P2_J12_020_particles.cs P2_J12_passthrough_particles.cs cryosparc_P2_J12_020_particles_passthrough_relion30.star
csparc2star.py cryosparc_P2_J12_020_particles.cs cryosparc_P2_J12_020_relion31.star
csparc2star.py -r2 cryosparc_P2_J12_020_particles.cs cryosparc_P2_J12_020_relion30.star

It does not matter, which of the 3 version I take.

If I try to load a relion 31 compatible star file into relion 31 I get the error:
=== Backtrace ===
/sw/relion/3.1/bin/relion_refine(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x529ce1]
/sw/relion/3.1/bin/relion_refine(_ZNK16ObservationModel10getBoxSizeEi+0x6b) [0x5769eb]
/sw/relion/3.1/bin/relion_refine(_ZN7MlModel20initialiseFromImagesE8FileNamebR10ExperimentRbS3_S3_dbbb+0x605) [0x50d115]
/sw/relion/3.1/bin/relion_refine(_ZN11MlOptimiser17initialiseGeneralEi+0x4f1) [0x47ca91]
/sw/relion/3.1/bin/relion_refine(_ZN11MlOptimiser10initialiseEv+0x484) [0x47f8c4]
/sw/relion/3.1/bin/relion_refine(main+0x33) [0x430f83]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7fd4360421a3]
/sw/relion/3.1/bin/relion_refine() [0x4343ff]

ERROR:
ObservationModel::getBoxSize: box sizes not available. Make sure particle images are available before converting/importing STAR files from earlier versions of RELION.

If I try ti load a relion 30 compatible star file into relion 30 I get the error:

in: /sw/relion/relion-3.0-src/src/image.h, line 175
=== Backtrace ===
/sw/relion/3.0/bin/relion_refine(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x438941]
/sw/relion/3.0/bin/relion_refine() [0x43a3a6]
/sw/relion/3.0/bin/relion_refine(_ZN5ImageIdE4readERK8FileNameblbb+0xce) [0x498ace]
/sw/relion/3.0/bin/relion_refine(_ZN10Experiment4readE8FileNamebbbb+0x8f5) [0x4e6ae5]
/sw/relion/3.0/bin/relion_refine(_ZN11MlOptimiser17initialiseGeneralEi+0x43a) [0x4632fa]
/sw/relion/3.0/bin/relion_refine(_ZN11MlOptimiser10initialiseEv+0x484) [0x46f4a4]
/sw/relion/3.0/bin/relion_refine(main+0xb03) [0x42b083]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7f3b8df071a3]
/sw/relion/3.0/bin/relion_refine() [0x42e3cf]

ERROR:
Cannot read file J10/extract/1782638280813807455_14sep05c_00024sq_00003hl_00005es.frames_patch_aligned_doseweighted_particles.mrc It does not exist

Is there any obvious thing I do wrong?
How could I get this running?
Any help is appreciated.

Thanks
–Kumar, Ryan, Elmar
kumar@healthtechnologyinnovations,.com
ryan@healthtechnologyinnovations.com
buchere@ohsu.edu

olibclarke · August 12, 2020, 1:53pm

Hi Kumar, see here: Converting from Cryosparc to Relion using csparc2star.py: Box size error

Basically you need to add the _rlnImageSize parameter to your optics table

olibclarke · August 12, 2020, 1:54pm

And in the second case (relion 3), have you symlinked the J10 directory into your relion directory?

nkumar · August 12, 2020, 2:45pm

Thank you very much. I’ll respond once we are thru with this. Regards.

bue · August 12, 2020, 5:09pm

Dear @olibclarke,

I opened the star file.

Right on top is the data_optics entry.
I think this is the table you referred too.

Studying the table, I conclude that the _* #n entries are the column names of the following data string

(not sure what the loop_ does, but I think this is just syntax important for the parser. Also _rln, what stands this for? right line?).

So, of course I can add a new column label and an entry to the data string.
For _rlnImageSize, is this data entry just the image size in pixels?

These might be obvious questions. In this case I am sorry for that, I am quite new to cryoEM, cryosparc and relion.

Thank you, Elmar

bue · August 12, 2020, 5:24pm

Having a look at the cryospark “import movie” job, I think I could derive the required imagesize information from there:

[CPU: 198.0 MB] ===========================================================

[CPU: 198.0 MB] Loaded 21 movies.

[CPU: 198.0 MB] Common fields:

[CPU: 198.0 MB] mscope_params/accel_kv : set([300.0])

[CPU: 198.0 MB] mscope_params/cs_mm : set([2.7])

[CPU: 198.0 MB] mscope_params/total_dose_e_per_A2 : set([53.0])

[CPU: 198.0 MB] mscope_params/exp_group_id : set([0])

[CPU: 198.0 MB] mscope_params/phase_plate : set([0])

[CPU: 198.0 MB] mscope_params/neg_stain : set([0])

[CPU: 198.0 MB] movie_blob/psize_A : set([0.65750003])

[CPU: 198.0 MB] movie_blob/shape : [ 38 7676 7420]

[CPU: 198.1 MB] movie_blob/is_gain_corrected : set([0])

[CPU: 198.1 MB] ===========================================================

[CPU: 198.1 MB] Making example plots…
[CPU: 198.1 MB] Reading file…

From the movie_blob/shape entire?

olibclarke · August 12, 2020, 5:39pm

Hi Elmar, yes that is correct - add the _rlnImageSize label and the corresponding entry in the data string. Yes it is just the box size in pixels (of the particles, not the movies). I believe rln is shorthand for relion.

Good luck!

Oli

bue · August 12, 2020, 6:05pm

Hi Olibcarke,

Thank you for your replay!

This makes sense now, and I found the correct value in the T20S tutorial (that is the data I am working at the moment with) https://cryosparc.com/docs/tutorials/t20s ,
In Step 7.5 Box size is written: “In this case, enter a box size of 384”.

If I enter this value I run into the same error like I run with the relion30 version.
This is a good sign. So I have now to work on the folder structure.

One step further, Elmar

olibclarke · August 12, 2020, 6:21pm

yes just symlink the particle and micrograph folders to your relion directory

and then in the cryosparc particle directory you will need to make symlinked copies of each particle stack with an mrcs extension rather than mrc

Something like:

for i in *mrc; do ln -s ${i} "${i}s" ; done

executed from within the directory where the particle stacks reside ought to do the trick

Oli

bue · August 12, 2020, 10:35pm

Hi Oli,

I still can not resolve this issue, and I don’t see where the problem is.
This is what I did:

I went to cryosparc ‘extract from micrographs’ job and exported the job.
I copied the exported job into a workingdirectory relionprj to mess around with it: cp -r ../cspark/P2/exports/jobs/P2_J10_extract_micrographs_m_rlnImageSizeulti .
I followed the ‘Relion project setup’ instruction in the “official documentation” to set up the correct relion project folder structure:
https://github.com/asarnow/pyem/wiki/Export-from-cryoSPARC-v2
The produced star file is now in the main relionprj folder
The symbolic lincs to the mrc files are at relionprj/J10/extract/*.mrcs
Inside the star file I add the _rlnImageSize into the data_optics table, as discussed
I fire up the relion form the relionprj folder
I go to 2D_classification/IO/Input_image_STAR _file, and selece the star file form its real location (not form the .Node/3/ folder).
I hit the Run! button.
I get the error
ERROR: Cannot read file >J10/extract/10910654411162275952_14sep05c_00024sq_00004hl_00002es.frames_patch_aligned_doseweighted_particles.mrcs It does not exist
At the location I started relion (~/relionprj/) it do:
ls -l J10/extract/10910654411162275952_14sep05c_00024sq_00004hl_00002es.frames_patch_aligned_doseweighted_particles.mrcs
The file is there!
xxd J10/extract/10910654411162275952_14sep05c_00024sq_00004hl_00002es.frames_patch_aligned_doseweighted_particles.mrcs
I can open the file with no problem!
I thought it migh be a unix right issue, so I followed the symbolic lincs.
readlink -f J10/extract/10910654411162275952_14sep05c_00024sq_00004hl_00002es.frames_patch_aligned_doseweighted_particles.mrcs
I copied the real mrc file right there, renamed the extension to mrcs.
Made a chmod 777 on the file and later on, recursive on the whole J10 folder.
Nothing helped.
It is like region is not looking at the right place.

Elmar

olibclarke · August 13, 2020, 1:20pm

Hi Elmar,

Hmmm. Can you post the first few lines of your star file? optics table and a few lines of data_particles?

Oli

bue · August 13, 2020, 4:44pm

Hi Oli,

This is the file head

data_optics

loop_
_rlnVoltage #1
_rlnSphericalAberration #2
_rlnAmplitudeContrast #3
_rlnOpticsGroup #4
_rlnImagePixelSize #5
_rlnImageDimensionality #6
_rlnImageSize #7
300.000000 2.700000 0.100000 0 0.657500 2 384

data_particles

loop_
_rlnImageName #1
_rlnMicrographName #2
_rlnCoordinateX #3
_rlnCoordinateY #4
_rlnDefocusU #5
_rlnDefocusV #6
_rlnDefocusAngle #7
_rlnPhaseShift #8
_rlnCtfBfactor #9
_rlnOpticsGroup #10
000001@>J10/extract/1782638280813807455_14sep05c_00024sq_00003hl_00005es.frames_patch_aligned_doseweighted_particles.mrcs >J2/motioncorrected/1782638280813807455_14sep05c_00024sq_00003hl_00005es.frames_patch_aligned_doseweighted.mrc 5228 4687 12989.263672 12879.832031 264.003967 0.000000 0.000000 0
000002@>J10/extract/1782638280813807455_14sep05c_00024sq_00003hl_00005es.frames_patch_aligned_doseweighted_particles.mrcs >J2/motioncorrected/1782638280813807455_14sep05c_00024sq_00003hl_00005es.frames_patch_aligned_doseweighted.mrc 1085 6542 13111.923828 13002.492188 264.003967 0.000000 0.000000 0
000003@>J10/extract/1782638280813807455_14sep05c_00024sq_00003hl_00005es.frames_patch_aligned_doseweighted_particles.mrcs >J2/motioncorrected/1782638280813807455_14sep05c_00024sq_00003hl_0

Elmar

bue · August 13, 2020, 4:47pm

Looking at this file now, maybe I have to make a relion folder structure for J2 as well?
Could this cause the error?
Also, these doseweighted.mrc files are still mrc files, not mrcs files, because the sed command I used only replaced s/ particles.mrc/particle.mrcs/g.
Although the error message is clearly about the _particles.mrcs file.

This is the whole error message I get:

   - Warning: the optics groups in P2_J10_particles_exported_relion31.star are not in the right order - renaming them now
in: /sw/relion/relion-3.1-src/src/image.h, line 272
ERROR: 
Cannot read file >J10/extract/10910654411162275952_14sep05c_00024sq_00004hl_00002es.frames_patch_aligned_doseweighted_particles.mrcs It does not exist
=== Backtrace  ===
/sw/relion/3.1/bin/relion_refine(_ZN11RelionErrorC1ERKSsS1_l+0x41) [0x529ce1]
/sw/relion/3.1/bin/relion_refine() [0x447056]
/sw/relion/3.1/bin/relion_refine(_ZN11MlOptimiser41calculateSumOfPowerSpectraAndAverageImageER13MultidimArrayIdEb+0x3c8) [0x478c08]
/sw/relion/3.1/bin/relion_refine(_ZN11MlOptimiser10initialiseEv+0x6be) [0x47fafe]
/sw/relion/3.1/bin/relion_refine(main+0x33) [0x430f83]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7fae7a3ba1a3]
/sw/relion/3.1/bin/relion_refine() [0x4343ff]
==================
ERROR: 
Cannot read file >J10/extract/10910654411162275952_14sep05c_00024sq_00004hl_00002es.frames_patch_aligned_doseweighted_particles.mrcs It does not exist

bue · August 13, 2020, 4:52pm

I found a fix:
in the star file I replaced all @> with @.
Then it relion finds the files.

So, this sed command should to the trick:

sed -i 's/@>/@/g' from_csparc.star

olibclarke · August 13, 2020, 7:35pm

yep just saw your star file and was about to suggest this, glad you found a fix!

bue · August 14, 2020, 12:49am

Yes, nice that it works now. Thank you Oli for all your help!
Do you think this s/@>/@/g issue is caused by a bug in the latest csparc2star.py source code?
Would it make sense to file a bug report?

Elmar

olibclarke · August 14, 2020, 2:05pm

I’ve never seen that issue before - not sure whether it is from csparc2star or something weird about your input cs files, does this happen with every file you try to convert?

bue · August 14, 2020, 10:13pm

Hi Oli,

Actually, the issue is already known, and seem to root form the current cs version 2 formats implementation.

For my self, I got everything running that I needed to, and I learned a lot along the way.
Thank you once again for your help!
Best wishes, Elmar

DanielAsarnow · September 4, 2020, 9:04pm

It has some specific cause, maybe caused by some pattern of symbolic links. In normal operation, the > isn’t prepended. I can lstrip(">") but I’d rather know what situation causes the > is prepended, because then there might be something more useful to do (like dereferencing).

matzovd · January 20, 2021, 10:15am

Hi all,

I’ve also been trying to follow the ‘Relion project setup’ instruction in the “official documentation” to set up the correct relion project folder structure:

but when I try to create the symbolic link I get the following error message:

“/usr/bin/ln: Argument list too long”

As far as I understand the issue is that there are to many files to be linked (I tried to follow the same instruction with a smaller extract job and everything worked fine).

Any suggestions on how to work around that?

Thanks!