Hi, when the 3D classification finishes, I have a file of the form, for example cryosparc_P1_J100_00010_particles.cs and others of the form cryosparc_P1_J100_classXXXXX_00010_particles.cs for each of the generated classes. As in other jobs (heterogeneous refinement), in the cryosparc_P1_J100_00010_particles.cs file are all the particles where the class to which each one belongs is specified. In my case, I generate the classes from this file and now I even have my doubts if I am doing it correctly.
In 3D classification, when converting to .star, all particles belong to the same class. My question is: Is it a CS bug or is it a pyem library bug? It would be convenient if this file had that information so as not to have to go through each .cs of each class.
Hi @cfonseca,
After some investigation, this is indeed a bug in the 3D classification outputs. Apologies! We’ll get a fix out in the next patch. For now, you can fix this with the following cryosparcm cli script (you can launch an icli session in a terminal window by executing cryosparcm icli):
from cryosparc_compute import dataset
#Import the CS file
particle_ds = dataset.Dataset()
particle_ds.from_file('/path/to/cryosparc_P1_J100_00010_particles.cs')
#Find all keys that look like alignments_class3D_X/class
alignment_keys = [key for key in particle_ds.fields() if (key.startswith('alignments_class3D_') and key.endswith('class'))]
#Update class keys to appropriate integers
for c_i, key in enumerate(alignment_keys):
particle_ds.data[key] = c_i
#Save
particle_ds.to_file('/path/to/cryosparc_P1_J100_00010_particles_fixedclasses.cs')
You should be able to now use this new fixed .cs file with csparc2star.py as expected.
Let me know if you run into any issues!
Valentin
Hi again!!!
One of the fixes in the new CS version is: Fixed bug in 3D Classification that caused csparc2star.py to produce erroneous star files if class reordering was turned on.
I have updated to v4.2 and the error persists: when converting to .star, all the particles belong to the same class.
Hi @vperetroukhin, with/without class reorderin, all particles are classified in the same class.
Ok, thank you for checking! We will investigate.
@cfonseca Patch 220302 should fix this! FYI: this was tested with the latest version of csparc2star.py using the main particle cs file and the all_classes passthrough. For example, if 3D Classification is J100 and converged at iteration 10, the command
./csparc2star.py /pathto/J100/J100_00010_particles.cs /pathto/J100/J100_passthrough_particles_all_classes.cs J100.star
should give the particles assigned to the appropriate (1-indexed) classes. Let us know if this works!
Hi @vperetroukhin, I tried what you told me and in the end I got the same result: all particles in the same class.
@cfonseca Hmm, that is strange – we just double checked this on our servers and we are seeing correct class outputs in 4.2. We used a fresh pull of pyem – i.e., we ran,
git clone https://github.com/asarnow/pyem.git
cd pyem
pip install -e .
followed by the command I mentioned above. Are you using any wrappers on top of csparc2star.py?
Hi @vperetroukhin, I have the master branch of pyem and precisely from Scipion I call csparc2star.py to generate the .star file
By command line I use the same command that you told me to use and still, in the .star file, the _rlnClassNumber field is assigned a 1.
I have downgraded CS to version 4.1.1 again and I have tried both from Scipion and from the command line and everything works fine.
Update for others who may be reading this: after some DMing with @cfonseca, we figured it out! The conversion is working with 4.2 w/ patch 230302 applied but you will need to make sure that the job itself was run after the patch was applied.