Error Extensive Validation after upgrade to V5.0.2

Running the Extensive Validation job after upgrading to version V5.0.2 yields an error after having started the Curate Exposures job. The Curate Exposures job itself starts properly and waits for manual completion, but an time out error is reported in the Extensive Validation job. Any suggestions how to fix this?

Traceback (most recent call last):
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpx/_transports/default.py”, line 101, in map_httpcore_exceptions yield
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpx/_transports/default.py”, line 250, in handle_request 
        resp = self._pool.handle_request(req)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py”, line 256, in handle_request 
        raise exc from None
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py”, line 236, in handle_request 
        response = connection.handle_request(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpcore/_sync/connection.py”, line 101, in handle_request raise exc
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpcore/_sync/connection.py”, line 78, in handle_request 
        stream = self._connect(request)
                 ^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpcore/_sync/connection.py”, line 124, in _connect 
        stream = self._network_backend.connect_tcp(**kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpcore/_backends/sync.py”, line 207, in connect_tcp 
        with map_exceptions(exc_map):
             ^^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/contextlib.py”, line 158, in _exit_ 
        self.gen.throw(value)
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpcore/_exceptions.py”, line 14, in map_exceptions 
        raise to_exc(exc) from exc 
httpcore.ConnectTimeout: timed out 
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
    File “cli/run.py”, line 236, in cli.
        run.register_thread_excepthook.init.run_with_except_hook
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/threading.py”, line 1012, in run 
        self._target(*self._args, **self._kwargs)
    File “/home/lnd/Cryosparc/cryosparc_worker/compute/jobs/workflows/run_extensive_validation.py”, line 297, in interactive_exposure_curation 
        fieldsres = jobs.interact(subjob, “get_fields_and_thresholds”, {})
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/core/jobs.py”, line 768, in interact 
        r = httpx.post(
            ^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpx/_api.py”, line 304, in post 
        return request(
               ^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpx/_api.py”, line 109, in request 
        return client.request(
               ^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpx/_client.py”, line 825, in request 
        return self.send(request, auth=auth, follow_redirects=follow_redirects)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpx/_client.py”, line 914, in send 
        response = self._send_handling_auth(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpx/_client.py”, line 942, in _send_handling_auth 
        response = self._send_handling_redirects(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpx/_client.py”, line 979, in _send_handling_redirects 
        response = self._send_single_request(request)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpx/_client.py”, line 1014, in _send_single_request
        response = transport.handle_request(request)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpx/_transports/default.py”, line 249, in handle_request 
        with map_httpcore_exceptions():
             ^^^^^^^^^^^^^^^^^^^^^^^^^
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/contextlib.py”, line 158, in _exit_ 
        self.gen.throw(value)
    File “/home/lnd/Cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/httpx/_transports/default.py”, line 118, in map_httpcore_exceptions 
        raise mapped_exc(message) from exc 
httpx.ConnectTimeout: timed out

Thanks @Pielhaas for your post. The timeout may sporadically occur during an extensive validation run. The issue is frequently resolved by rerunning extensive validation.

We have run the Extensive Validation job multiple times and on various workers, all resulting in the same error. Up to now we have not seen issues with other jobs.

Thanks @Pielhaas for the additional details. Please can you share with us:

  1. the tgz file created with the command
    cryosparcm snaplogs
  2. the job report of the Extensive Validation job
  3. the job report of the Curate Exposures job.

I’ll send a personal message regarding sharing arrangements.