BrokenPipeError when running patch motion correction on non-Ada GPUs

When running patch motion correction on non-Ada GPUs, our users are receiving the following error:

Traceback (most recent call last):
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/cryosparc-worker/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

These errors continue until the job crashes completely. Strangely, we don’t have this issue on our RTX 6000 Ada GPUs.

@UCBKurt Please can you provide additional details:

  1. CryoSPARC version (and patch, if applicable).
  2. The models of non-Ada GPUs with which the errors occur.
  3. Any other differences, aside from Ada versus non-Ada, that you think might be relevant.
  1. v4.4.0+231114
  2. RTX 2070, RTX 2080 ti, RTX A4500, RTX A5000, RTX A4000, RTX 3090, Titan RTX
  3. Actually, it looks like it’s working on GTX 1080 ti machines, so maybe it’s an issue limited to ampere and turing? Any other job runs fine on the machines though.

@UCBKurt Please can you email us the job report for the job that gave

.