Python errors when running standalone version of 2.15.0

Troubleshooting
1

We seem to be having a problem identical to the one in this ticket:

The issue there appears to be the use of an incompatible CUDA version, but that can’t be the case here: we’re running a master/worker version of cryoSPARC 2.15.0 using exactly the same CUDA libraries. When I say exact, I mean the same installation: both cryoSPARC instances are using CUDA from the same NFS-mounted file system. The version of CUDA we’re using is 10.1.

The error we get seems identical to the one in the ticket mentioned above:

cryosparc_user@rattlesnake:~$ cryosparcm joblog P1 J2
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/     anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/     anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/     anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/     anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/     anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/     anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe

<these lines repeated many times>

Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/     anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/deps/     anaconda/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
IOError: [Errno 32] Broken pipe
========= sending heartbeat
========= sending heartbeat
========= sending heartbeat
**** handle exception rc
set status to failed
Traceback (most recent call last):
  File "cryosparc2_worker/cryosparc2_compute/run.py", line 85, in           cryosparc2_compute.run.main
  File "cryosparc2_worker/cryosparc2_compute/jobs/ctf_estimation/run.py",   line 258, in cryosparc2_compute.jobs.ctf_estimation.run.run
AssertionError: Child process with PID 28891 has terminated unexpectedly!
========= main process now complete.
========= monitor process now complete.

Given that it seems clear that it can’t be a problem with the CUDA libraries, is this a bug in the standalone implementation?

2

Any thoughts on this? Could this be a bug with the standalone version?

3

Hi @pgoetz,

Can you navigate to the job folder and open up the job.log file and copy the contents here? I believe the joblog command is cutting off the actual error message since there are so many “Broken pipe” messages.

4

Hi Stephan –

Whereabouts would I find this job.log file?

5

Hi @pgoetz,

You can run the command:
cryosparcm cli "get_job_dir_abs('<project_uid>', '<job_uid>')"

For example:

cryosparcm cli "get_job_dir_abs('P1', 'J2')"
> /bulk6/data/cryosparc_projects/P1/J2

The job.log file will be found inside this folder