Very interesting. So if I understand correctly, you have
topaz extract (job "A") → extract from mics (1/4 downsample) → 2D class → select 2D → ab initio (success)
Then you also have a chain that uses the particles from select 2D but ultimately fails.
Perhaps try this:
Clone the extract from mics (1/4 downsample) job, change the downsampling (to no downsample), clear the particles input, and connect your final particles from select 2D. Run the result through ab-initio and see what happens.
Hi @hsnyder,
So this is slightly complicated by the fact that I ran three topaz extract jobs and three corresponding extract from micrographs jobs, but then combined the particles after 2 rounds of 2D classification. But since all of those worked cloning any one of them and adding the micrograph inputs from the other topaz extract jobs should be fine? I’ll try this and let you know how it goes.
Thanks for clarifying. Yes I think trying what you proposed is a good next step. It might also be worth trying extracting the three upstream topaz jobs seperately, and combining downstream. I have a strong suspicion that this problem has something to do with combining multiple inputs, though I’m not sure yet exactly what the cause is. Hopefully with some permutations of the experiment, we can nail down the exact step that causes this to happen…
Hi @hsnyder,
I tried re-extracting using the cloned job, adding in the micrograph outputs from the other topaz extract jobs and and the particles from select 2D, and the ab initio job was giving the exact same error. Since it kept mentioning the same micrograph in the error, I decided to just do a curate exposures job linking both the particles and the micrographs, and remove that particular micrograph from the dataset. I also checked for duplicate filenames but I didn’t catch anything. This workaround worked, but I have no idea why this issue happened in the first place.
Well, I’m glad you found a solution! If you wouldn’t mind, could you run the following command on the offending particle stack (the one that gets printed out in the error message) and post the output?
xxd -e -g 4 /path/to/offending-particles.mrc | head
That helps, thank you. There are 88 particles in the particle stack according to the mrc file header (88 is ‘58’ in hexadecimal, on the first line), but cryoSPARC was asking for the 89th frame (notice the ‘89, 90’ bit near the end of the RuntimeError message, meaning “fetch frames starting from the 89th, and stopping before the 90th”). Somehow, the dataset files (.cs files) have gotten out-of-sync with the particle stacks that are actually on disk. The associated .cs file must “think” there are more particles in that file than there actually are, which is why it’s asking for the 89th. The only time we had previously seen this is when there were micrographs with the same names, but it looks like something about your present workflow has produced the same kind of situation.
Oh I see! When I did the curate exposures job it did say it excluded 90 particles assigned to this micrograph. I’ve done the same workflow on other datasets and never had issues, but at least I know what to do if this happens again. Thanks for all your help!