Deep picker train error: `Job is not in running state`

Troubleshooting
1

Hi,

I’m trying to run the Deep Picker Train job but I get this error:

Traceback (most recent call last):
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1790, in run_with_except_hook
    run_old(*args, **kw)
  File "/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 186, in thread_work
    work = processor.exec(item)
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 43, in exec
    return self.process(item)
  File "cryosparc_worker/cryosparc_compute/jobs/deep_picker/run_deep_picker.py", line 186, in cryosparc_compute.jobs.deep_picker.run_deep_picker.run_deep_picker_train.train_multiprocess.process
AssertionError: Job is not in running state - worker thread with PID 197421 terminating self.

[CPU: 35.60 GB]  Traceback (most recent call last):
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1790, in run_with_except_hook
    run_old(*args, **kw)
  File "/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 186, in thread_work
    work = processor.exec(item)
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 43, in exec
    return self.process(item)
  File "cryosparc_worker/cryosparc_compute/jobs/deep_picker/run_deep_picker.py", line 186, in cryosparc_compute.jobs.deep_picker.run_deep_picker.run_deep_picker_train.train_multiprocess.process
AssertionError: Job is not in running state - worker thread with PID 197410 terminating self.

[CPU: 257.5 MB]  Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main
  File "cryosparc_worker/cryosparc_compute/jobs/deep_picker/run_deep_picker.py", line 199, in cryosparc_compute.jobs.deep_picker.run_deep_picker.run_deep_picker_train
AssertionError: Child process with PID 197421 has terminated unexpectedly!

[CPU: 35.60 GB]  Traceback (most recent call last):
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1790, in run_with_except_hook
    run_old(*args, **kw)
  File "/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 186, in thread_work
    work = processor.exec(item)
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 43, in exec
    return self.process(item)
  File "cryosparc_worker/cryosparc_compute/jobs/deep_picker/run_deep_picker.py", line 186, in cryosparc_compute.jobs.deep_picker.run_deep_picker.run_deep_picker_train.train_multiprocess.process
AssertionError: Job is not in running state - worker thread with PID 197414 terminating self.

[CPU: 35.60 GB]  Traceback (most recent call last):
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 1790, in run_with_except_hook
    run_old(*args, **kw)
  File "/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 186, in thread_work
    work = processor.exec(item)
  File "/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 43, in exec
    return self.process(item)
  File "cryosparc_worker/cryosparc_compute/jobs/deep_picker/run_deep_picker.py", line 186, in cryosparc_compute.jobs.deep_picker.run_deep_picker.run_deep_picker_train.train_multiprocess.process
AssertionError: Job is not in running state - worker thread with PID 197407 terminating self.
2

Hi,

What version of CryoSPARC are you running? Were there any other stack traces that were printed, or potentially any warnings? These errors mean that another error killed these threads. Can you try running this job with only a single thread? You can do that by setting the Number of parallel threads parameter to 1 in the job builder.

Best,
Louis