KeyError: 'background_blob/path' after Topaz Extract workflow

Particle Picking
1

Hi everyone, I’m on CryoSPARC 4.3.0 and trying to extract particles after Topaz Train and Topaz Extract and getting this error about background blob missing - here’s the first occurrence:

Starting multithreaded pipeline ... 
Started pipeline
GPU 0 using a batch size of 1024

-- 0.0: (1 of 140) processing J113/motioncorrected/003627282102744355943_n23jan23b_DER8Gr3_00007gr_00002sq_v02_00005hl_00056er2.frames_patch_aligned_doseweighted.mrc
        14 particles extracted (1 rejected near edges)

-- 0.1: (2 of 140) processing J113/motioncorrected/011693116031145724613_n23jan23b_DER8Gr3_00008gr_00014sq_v03_00006hl_00087er2.frames_patch_aligned_doseweighted.mrc
        25 particles extracted (1 rejected near edges)

GPU 1 using a batch size of 1024

-- 1.0: (3 of 140) processing J113/motioncorrected/010112720111706024746_n23jan23b_DER8Gr3_00008gr_00014sq_v03_00008hl_00049er2.frames_patch_aligned_doseweighted.mrc
        18 particles extracted (1 rejected near edges)

-- 1.1: (4 of 140) processing J113/motioncorrected/015191867797214508776_n23jan23b_DER8Gr3_00012gr_00001sq_v02_00006hl_00012er2.frames_patch_aligned_doseweighted.mrc

-- 0.0: (5 of 140) processing J113/motioncorrected/002846102498849015578_n23jan23b_DER8Gr3_00012gr_00010sq_v02_00002hl_00089er2.frames_patch_aligned_doseweighted.mrc
        14 particles extracted (0 rejected near edges)

Error occurred while processing micrograph J113/motioncorrected/003627282102744355943_n23jan23b_DER8Gr3_00007gr_00002sq_v02_00005hl_00056er2.frames_patch_aligned_doseweighted.mrc
Traceback (most recent call last):
  File "/home/drichman/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/pipeline.py", line 60, in exec
    return self.process(item)
  File "/home/drichman/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/extract/run.py", line 485, in process
    path_abs = os.path.join(proj_dir_abs, micrograph['background_blob/path'])
KeyError: 'background_blob/path'

Marking J113/motioncorrected/003627282102744355943_n23jan23b_DER8Gr3_00007gr_00002sq_v02_00005hl_00056er2.frames_patch_aligned_doseweighted.mrc as incomplete and continuing...

I have the Topaz Extract output micrographs and particles connected as input for Extract from Micrographs. Output from Extract from Micrographs should be around 140 exposures and 3600 particles, based on what’s coming from Topaz Extract, minus edge particles. But Extract from Micrographs is giving 21 micrographs and 438 particles, and many instances of this error that it continues through.

I’ve tried connecting the background blob from upstream and even tried just using the motion correction job output micrographs as inputs. In the latter case, there’s no error but the extraction output is still 21 micrographs and 438 particles.

2

Is there any divergence in this exposure set’s upstream workflow? Were any of the exposures pre-processed differently from the others, or were the exposures recorded during different microscope sessions?

3

Yes, this 140 exposure set is a randomized subset of several datasets from different scope sessions but with the same hardware and settings. 5 of 6 of them were pre-processed exactly the same way in this workspace. The remaining one was motion corrected in a different project (a rash decision about storage managment) and then I imported the motion corrected micrographs in this workspace.

That must have lost the background blob, but the error is showing up for the other 5 of 6 dataset paths, not that one. Also, I have not seen this error in other extraction jobs, only this one that follows Topaz picking.

4

@drichman Please can you try pre-processing the dataset whose motion-corrected micrographs you imported earlier inside the current project and check whether

KeyError: 'background_blob/path

is resolved when all six datasets were pre-processed the same way?

5

Hi @wtempel sorry for my delay. I now have enough storage to work on these data all in one project and workspace. The datasets have all been pre-processed in that one project and the error is resolved.

2 Likes