Patch ctf fails on K3 mics w/ Index Error and Assertion Error

Troubleshooting
1

Job has failed three times now on ~3k mics from krios w/ K3 in CDS mode (MotionCorrected thru relion GUI). I’ve never gotten these errors following the same processing protocol. The only difference is CDS mode (not that it’s the cause). After completing 1/3 of the 3k mics and giving the below error, I marked job as complete and run the same process on the incomplete set and it gives the same error. Any help appreciated.

Errors:
[CPU: 947.9 MB] Traceback (most recent call last):

File “/home/lab/software/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py”, line 1726, in run_with_except_hook

**run_old(*args, kw)

File “/home/lab/software/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py”, line 870, in run

**self._target(*self._args, self._kwargs)

File “/home/lab/software/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/pipeline.py”, line 186, in thread_work

work = processor.exec(item)

File “/home/lab/software/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/pipeline.py”, line 43, in exec

return self.process(item)

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 112, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 118, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 119, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/patchctf.py”, line 71, in cryosparc_compute.jobs.ctf_estimation.patchctf.patchctf_v217

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/patchctf.py”, line 756, in cryosparc_compute.jobs.ctf_estimation.patchctf.patchctf_v217

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/patchctf.py”, line 621, in cryosparc_compute.jobs.ctf_estimation.patchctf.patchctf_v217.EPA_suppress_past_k2k4crossover_point

IndexError: index -1 is out of bounds for axis 0 with size 0

[CPU: 841.6 MB] Traceback (most recent call last):

File “/home/lab/software/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py”, line 1726, in run_with_except_hook

**run_old(*args, kw)

File “/home/lab/software/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py”, line 870, in run

**self._target(*self._args, self._kwargs)

File “/home/lab/software/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/pipeline.py”, line 186, in thread_work

work = processor.exec(item)

File “/home/lab/software/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/pipeline.py”, line 43, in exec

return self.process(item)

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 209, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process

AssertionError: Job is not in running state - worker thread with PID 45092 terminating self.

[CPU: 437.2 MB] Outputting partial results now…

[CPU: 434.1 MB] Traceback (most recent call last):

File “cryosparc_worker/cryosparc_compute/run.py”, line 84, in cryosparc_compute.run.main

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 256, in cryosparc_compute.jobs.ctf_estimation.run.run

AssertionError: Child process with PID 45092 has terminated unexpectedly!

2

I should note no other processes are running on our lab’s GPU and it has ample resources available.

The first time the process failed, it gave a different error about 350 mics in. Then the second time I ran the same process, it failed at 1k mics with the error given above. Running on the incomplete mics gives the above error each time, after completing 1 mic.

Below is the error from the first failure.

[CPU: 839.4 MB] Traceback (most recent call last):

File “/home/lab/software/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py”, line 1726, in run_with_except_hook

**run_old(*args, kw)

File “/home/lab/software/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py”, line 870, in run

**self._target(*self._args, self._kwargs)

File “/home/lab/software/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/pipeline.py”, line 186, in thread_work

work = processor.exec(item)

File “/home/lab/software/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/pipeline.py”, line 43, in exec

return self.process(item)

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 112, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 118, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 119, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/patchctf.py”, line 71, in cryosparc_compute.jobs.ctf_estimation.patchctf.patchctf_v217

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/patchctf.py”, line 756, in cryosparc_compute.jobs.ctf_estimation.patchctf.patchctf_v217

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/patchctf.py”, line 626, in cryosparc_compute.jobs.ctf_estimation.patchctf.patchctf_v217.EPA_suppress_past_k2k4crossover_point

ValueError: operands could not be broadcast together with shapes (0,) (30,)

[CPU: 435.7 MB] Outputting partial results now…

[CPU: 432.2 MB] Traceback (most recent call last):

File “cryosparc_worker/cryosparc_compute/run.py”, line 84, in cryosparc_compute.run.main

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 256, in cryosparc_compute.jobs.ctf_estimation.run.run

AssertionError: Child process with PID 39256 has terminated unexpectedly!

[CPU: 320.1 MB] Cleared -645984458 bytes by deleting 0 items in 0.01s

3

OK, fixed with cryosparcm patch.

1 Like