IOError: [Errno 32] Broken pipe

Troubleshooting
1

Hi -

Trying to run a job in v2.14. The job starts, but abends. This happens in more than one job.
I tried updating the system, and even doing a clean install.
The system is Ubuntu 18.04, Cuda 10.2, with 4 GPUs.
It’s a single node - the master is also the worker.

The Broken pipe error repeats itself many times over.
Can anyone give some insight as to what the problem can be, and a solution?

The important part of the job log below.

Thanks-
-Gershon

0.345238924026
**custom thread exception hook caught something
**** handle exception rc
set status to failed
Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/yarden/@Cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 153, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/ctf_estimation/run.py", line 210, in cryosparc2_compute.jobs.ctf_estimation.run.run.ctfworker.pr
ocess
AssertionError: Job is not in running state - worker thread with PID 26505 terminating self.
311514854431
0.190490007401
0.493827104568
0.375034093857
**custom thread exception hook caught something
**** handle exception rc
set status to failed
Traceback (most recent call last):
  File "cryosparc2_compute/jobs/runcommon.py", line 1685, in run_with_except_hook
    run_old(*args, **kw)
  File "/home/yarden/@Cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "cryosparc2_compute/jobs/pipeline.py", line 153, in thread_work
    work = processor.process(item)
  File "cryosparc2_worker/cryosparc2_compute/jobs/ctf_estimation/run.py", line 210, in cryosparc2_compute.jobs.ctf_estimation.run.run.ctfworker.pr
ocess
AssertionError: Job is not in running state - worker thread with PID 26502 terminating self.
Traceback (most recent call last):
  File "/home/yarden/@Cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/yarden/@Cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
2

Hey @Gershon, could you post the full job log, including all information from before the exception? Thanks!

3

Hi -

I found the problem, it was a corrupt dataset.
Thanks!

4

I met the same problem. We have mount a storage system on the machine, and here is the full job log:

Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat

Here is the log in the job window:

[CPU: 419.1 MB]  Starting multithreaded pipeline ... 

[CPU: 419.7 MB]  Started pipeline

[CPU: 508.1 MB]  GPU 0 using a batch size of 1024

[CPU: 508.3 MB]  -- 0.0: processing J7/imported/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00002esn-a-DW.mrc
        Writing to /data/P3/J35/extract/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00002esn-a-DW_particles.mrc

[CPU: 508.3 MB]  -- 0.1: processing J7/imported/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00003esn-a-DW.mrc
        Writing to /data/P3/J35/extract/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00003esn-a-DW_particles.mrc

[CPU: 1.41 GB]   -- 0.0: processing J7/imported/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00004esn-a-DW.mrc
        Writing to /data/P3/J35/extract/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00004esn-a-DW_particles.mrc

[CPU: 1.73 GB]   -- 0.1: processing J7/imported/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00005esn-a-DW.mrc
        Writing to /data/P3/J35/extract/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00005esn-a-DW_particles.mrc

[CPU: 1.11 GB]   (1 of 362) Finished processing micrograph 0.

[CPU: 1.11 GB]   (2 of 362) Finished processing micrograph 1.

[CPU: 1.63 GB]   -- 0.0: processing J7/imported/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00006esn-a-DW.mrc
        Writing to /data/P3/J35/extract/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00006esn-a-DW_particles.mrc

[CPU: 1.11 GB]   (3 of 362) Finished processing micrograph 2.

[CPU: 1.74 GB]   -- 0.1: processing J7/imported/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00007esn-a-DW.mrc
        Writing to /data/P3/J35/extract/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00007esn-a-DW_particles.mrc

[CPU: 1.11 GB]   (4 of 362) Finished processing micrograph 3.

[CPU: 1.84 GB]   -- 0.0: processing J7/imported/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00008esn-a-DW.mrc
        Writing to /data/P3/J35/extract/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00008esn-a-DW_particles.mrc

[CPU: 1.11 GB]   (5 of 362) Finished processing micrograph 4.

[CPU: 1.84 GB]   -- 0.1: processing J7/imported/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00009esn-a-DW.mrc
        Writing to /data/P3/J35/extract/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00009esn-a-DW_particles.mrc

[CPU: 1.11 GB]   (6 of 362) Finished processing micrograph 5.

[CPU: 1.11 GB]   (7 of 362) Finished processing micrograph 6.

[CPU: 1.69 GB]   -- 0.0: processing J7/imported/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00010esn-a-DW.mrc

[CPU: 1.82 GB]   -- 0.1: processing J7/imported/20jan24a_R1-23-Au1_1_00009gr_00082sq_v02_00002hl_00011esn-a-DW.mrc
[CPU: 1.11 GB]   (8 of 362) Finished processing micrograph 7.

(And it stocked without error, job status is running)

However, we ran the imported data with Gctf and it succeed. I think if the job was corrupted, Gctf job shouldn’t success.

5

Hi @sunny1226, please check to see whether Cuda is set up correctly as noted here: Raise OSError('CUDA driver library not found')

If that doesn’t work run the cryoSPARC command line utility and inspect the output to ensure your environment is set up as expected:

cryosparcm env
cryosparcm status

Feel free to private message the output to me to help interpret it.

Nick