KeyError background_blob/path in ExtractMicrographs job

Troubleshooting
1

Hi, Developers,

We occasionally have the the following error failing the jobs after doing motioncor2 using other source.
We are on version: Current cryoSPARC version: v2.14.3-live_privatebeta, standalone installation.

[CPU: 4.85 GB]   (4376 of 11255) Finished processing micrograph 4375.

[CPU: 1.17 GB]   -- 0.0: processing J1/imported/20mar17b_a_00035gr_00050sq_v02_00023hl_00003en-a-DW.mrc

[CPU: 4.85 GB]   (4377 of 11255) Finished processing micrograph 4376.

[CPU: 1.11 GB]   -- 0.1: processing S1/motioncorrected/20mar30a_00022gr_00004sq_v02_00002hl_00005en-a-DW_patch_aligned_doseweighted.mrc

[CPU: 4.85 GB]   (4378 of 11255) Finished processing micrograph 4377.

[CPU: 1.41 GB]   Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/data/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 53, in stage_target
    work = processor.process(item)
  File "cryosparc2_compute/jobs/extract/run.py", line 245, in process
    path_abs = os.path.join(proj_dir_abs, micrograph['background_blob/path'])
KeyError: 'background_blob/path'

The job.log has

Traceback (most recent call last):

  File "/data/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed

    send(obj)

IOError: [Errno 32] Broken pipe

Traceback (most recent call last):

  File "/data/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed

    send(obj)

IOError: [Errno 32] Broken pipe

Cloning and rerunning the job failed at exactly the same file, image 4378. I checked the 20mar30a_00022gr_00004sq_v02_00002hl_00005en-a-DW_patch_aligned_doseweighted.mrc, it seems to be intact. The images are on a NFS export, but I don’t see there is any problem with the NFS export.

The command_core.log does not state any info except for marking this job status failed.
Upon initial failing of this job, it tried to remove the output, but since it is on NFS, and some file in the extract dir of the job are not closed, so there is a .nfs lock file, restarting the cryosparc instance got rid of it. Still rerunning failed. Please help.

Thanks
Wei

2

Hi @WeiGuo,
Can you please see if this post is helpful re: CUDA version and setup? IOError: [Errno 32] Broken pipe

3

HI, @spunjani,

Thanks for looking into this.

In the log, I don’t have the cuda error. The cuda path is in our library path. The instance is running other GPU jobs correctly.
Also we do the patch ctf within cryosparc, but imported motioncor’ed images to cryosparc.

4

Hi @WeiGuo,

Did you connect multiple micrograph inputs to this job?
Also, can you try upgrading to the latest version of cryoSPARC Live and then re-try?

5

Upgraded to 2.15.1-live-privatebeta. Still fails at the same point same error.