For background, at the end of processing I realized I needed to expand my box size to include density for my entire molecule. I attempted to simply re-extract the whole particle set, but got an error about needing to re-extract separately if I had different alignment pixel sizes (they all should have been the same as they came from the same micrographs). The error:
“AssertionError: Particles must all have the same alignment pixel size. If multiple particle stacks were input into the job, please instead run separate extract jobs for each particle stack.”
I had done a lot of processing on this dataset and joining of particles from different jobs, so I went back to previous jobs trying to figure out at which point I could successfully re-extract my particles. I found a point at which I could re-extract all of my particles successfully by excluding one group of particles and re-extracting it separately, but now when I attempt to do a new ab initio job with all of the particles, cryoSPARC will not read in the 297k particles from this trouble-group.
I’m wondering if the particle header information is somehow corrupt, but don’t know what I should be looking for. Everything was re-extracted using exactly the same parameters, and there wasn’t an issue with including these particles when I did the initial job and refinement.
I may have solved my own problem. I think the issue was that the particles came from 2D classification jobs with slightly different parameters. I re-did 2D classification on all of my particle sets and now they all seem to be able to feed into a new 3D ab initio job without any particles being discarded.
It would be nice to get some kind of warning message if all of the particles feeding into a job were not included in the particle stack. If it had been 30k particles instead of 300k particles I may not have even noticed, but 30k particles really make a difference for this dataset.
@jenchem A single-class ab initio reconstruction job might indeed not output all input particles. We are planning to implement a notification to that effect.
After single-class ab initio reconstruction, one can still include all particles in a subsequent job by connecting as inputs
@wtempel - is this because ab initio implements a class probability threshold or similar internally? so anything less than say 0.9 probability of class X is discarded?
I’m not so sure that was the problem in my case though. It appeared that even though I gave the job the full particle set, it only recognized and read in a portion of them. For example, I input over 700,000 particles and the ParticleStack only loaded 590,805 of them to start the job:
@olibclarke It’s not so much a rejection of particles, but an “omission from consideration” for single class ab initio reconstruction, where the total number of iterations to run is determined differently from the multi-class case.
@jenchem The loading of only 590,805 particles indeed seems to have a different cause. Is there some heterogeneity in the connected input particle set, such as a combination of particles with different processing histories?
Yes, this was the case. I had done processing separately for a few groups of particles, combined them, and then did a bit more downstream processing. In my case it seems that I couldn’t re-extract all of them at once due to some kind of difference in upstream processing at the level of 2D classification. I re-extracted them separately, but this still didn’t solve the problem.
Then I ran a job that 2D-classified them all together (didn’t even have to bother picking classes as I knew I wanted all of the particles) and the output of this job was enough to allow me to successfully get the 3D ab-initio job to recognize all of the particles.
Hello, I want to follow up with your last thread you posted back in Feb 20. I encountered the same problem as you did that not all particle stack was loaded. I wonder if you solved the problem by re-doing 2D classification using the same params in separate jobs or in a single job where you loaded the entire pctl stack. I did the latter but found the ptcl stack was only partially loaded. I would appreciate your help.
Looking back, it appears I loaded all of the particles from all stacks (ie all particles that I wanted for further processing) into one 2D-classification job, then selected all of the output from that one 2D-classification job to start a 3D ab initio. That seems to have worked for my case.
I followed what you described in your last posting to include all particle stacks in one big 2D classification. But it did not solve my problem. I got 584K particles out of a total of 747K particles. I don’t know why I lost part of the stacks. Any suggestions to further trouble shoot would be very helpful. Thank you for your response.
