Jobs sporadically hanging at launch after update to 5.0.6

DXLee · June 24, 2026, 10:39pm

Since updating to version 5.0.6 on our cluster we have started to have some jobs hang at launch.

The jobs fail, but are stuck in a launch state with a slurm_load_jobs error and ServerSelectionTimeoutError.

Our IT group hasn’t been able to identify a problem.

Thank you for your help.

Event Log Error:

[2026-06-18 14:52:15.15] [CPU: 637.5 MB]Cluster job status update for Job P94-J323 (Heterogeneous Refinement) failed (41 status update request retries): 'squeue -j

10006470': Command failed (code 1)

Output: slurm_load_jobs error: Invalid job id specified

Error:

Job Log Error:

================= CRYOSPARC =================
Project P94 Job J323
Master reichow-cs2.ohsu.edu Port 61000
===========================================================================
MAIN PROCESS PID 312104
========= updating job startup information at 2026-06-18 14:52:16.091007
================= CRYOSPARC =================
Project P94 Job J323
Master reichow-cs2.ohsu.edu Port 61000
===========================================================================
MAIN PROCESS PID 312104
Process Process-1:
Traceback (most recent call last):
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "cli/run.py", line 25, in cli.run.start_and_update_job_runtime_info
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/core/core.py", line 61, in startup
    self.mongo = get_pymongo_client(conf.mongo_db_name, conf)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/core/database_management.py", line 227, in get_pymongo_client
    assert client[database_name].list_collection_names() is not None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/database.py", line 1226, in list_collection_names
    return self._list_collection_names(session, filter, comment, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/database.py", line 1191, in _list_collection_names
    result["name"] for result in self._list_collections_helper(session=session, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/database.py", line 1138, in _list_collections_helper
    return self._client._retryable_read(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1863, in _retryable_read
    return self._retry_internal(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/_csot.py", line 119, in csot_wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1830, in _retry_internal
    ).run()
      ^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 2554, in run
    return self._read() if self._is_read else self._write()
           ^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 2689, in _read
    self._server = self._get_server()
                   ^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 2645, in _get_server
    return self._client._select_server(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/mongo_client.py", line 1649, in _select_server
    server = topology.select_server(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 398, in select_server
    server = self._select_server(
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 376, in _select_server
    servers = self.select_servers(
              ^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 283, in select_servers
    server_descriptions = self._select_servers_loop(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/exacloud/gscratch/reichowlab/local/cryosparc/reichow-cs2.ohsu.edu/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/pymongo/synchronous/topology.py", line 333, in _select_servers_loop
    raise ServerSelectionTimeoutError(
pymongo.errors.ServerSelectionTimeoutError: reichow-cs2.ohsu.edu:61001: timed out (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms), Timeout: 60.0s, Topology Description: <TopologyDescription id: 6a346890e6e9394c6c892a09, topology_type: Single, servers: [<ServerDescription ('reichow-cs2.ohsu.edu', 61001) server_type: Unknown, rtt: None, error=NetworkTimeout('reichow-cs2.ohsu.edu:61001: timed out (configured timeouts: socketTimeoutMS: 20000.0ms, connectTimeoutMS: 20000.0ms)')>]>

wtempel · June 25, 2026, 2:31pm

Thanks @DXLee for this report.
The ServerSelectionTimeoutError may be the cause of slurm_load_jobs: Invalid job id specified. For analysis of the ServerSelectionTimeoutError
Please can you
post the outputs of the command (on the reichow-cs2 computer)

ps -eo user,pid,ppid,start,cmd | grep -e cryosparc_ -e mongo

and the commands on the cluster node where job P94.J323 failed

uname -a
curl reichow-cs2.ohsu.edu:61001

The relevant cluster node can be displayed by running the following command on reichow-cs2:

cryosparcm cli "api.jobs.find_one('P94', 'J323').instance_information.platform_node"

and send us:

the job report for job P94.J323
the tgz file created with the command
cryosparcm snaplogs

I will send you a forum PM about the email address.

DXLee · June 25, 2026, 8:24pm

Thank you @wtempel

It looks like one of the gpu nodes is failing to connect to the reichow-cs2 computer. See the output below. I sent you the job report for P94.J323, but the snaplogs were not permitted by your email system.

Outputs from reichow-cs2:

[ledavid@reichow-cs2 ~]$ ps -eo user,pid,ppid,start,cmd | grep -e cryosparc_ -e mongo
reichow+  194976       1   Jun 23 python /usr/local/cryosparc/cryosparc_master/.pixi/envs/master/bin/supervisord -c /usr/local/cryosparc/cryosparc_master/config/supervisord.conf
reichow+  195051  194976   Jun 23 mongod --auth --dbpath /var/local/cryosparc --port 61001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all --networkMessageCompressors snappy
reichow+  195124  194976   Jun 23 python3.12 /usr/local/cryosparc/cryosparc_master/.pixi/envs/master/bin/uvicorn api.main:app --fd 0 --log-config /usr/local/cryosparc/cryosparc_master/config/log.conf
reichow+  195125  194976   Jun 23 python3.12 /usr/local/cryosparc/cryosparc_master/.pixi/envs/master/bin/uvicorn api.main:app --fd 0 --log-config /usr/local/cryosparc/cryosparc_master/config/log.conf
reichow+  195126  194976   Jun 23 python3.12 /usr/local/cryosparc/cryosparc_master/.pixi/envs/master/bin/uvicorn api.main:app --fd 0 --log-config /usr/local/cryosparc/cryosparc_master/config/log.conf
reichow+  195155  194976   Jun 23 python3.12 /usr/local/cryosparc/cryosparc_master/.pixi/envs/master/bin/flask --app command.command_vis:start() run -h 0.0.0.0 -p 61003 --with-threads
root      212862  212803 14:22:56 sudo -u reichow-cryosparc /usr/local/cryosparc/cryosparc_master/bin/cryosparcm log app
reichow+  212863  212862 14:22:56 less -R +F /usr/local/cryosparc/cryosparc_master/run/app.log
ledavid   226494  226453 09:51:10 grep --color=auto -e cryosparc_ -e mongo

The cryosparcm cli command gave no output, so I ran the uname and curl commands on every gpu node in the cluster. One node failed to connect.

Output from gpu nodes:

# Typical output

[ledavid@arc-infra-1 ~]$ srun -p gpu --gres=gpu:0 -w cnode-10-0 bash -c "uname -a && curl reichow-cs2.ohsu.edu:61001"
srun: job 10057235 queued and waiting for resources
srun: job 10057235 has been allocated resources
Linux cnode-10-0 5.14.0-570.12.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Apr 4 10:41:31 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    85  100    85    0     0  14166      0 --:--:-- --:--:-- --:--:-- 14166
It looks like you are trying to access MongoDB over HTTP on the native driver port.

# Failed node

[ledavid@arc-infra-1 ~]$ srun -p gpu --gres=gpu:0 -w cnode-11-16 bash -c "uname -a && curl reichow-cs2.ohsu.edu:61001"
srun: job 10057311 queued and waiting for resources
srun: job 10057311 has been allocated resources
Linux cnode-11-16 5.14.0-570.12.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Apr 4 10:41:31 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:09 --:--:--     0
curl: (28) Failed to connect to reichow-cs2.ohsu.edu port 61001: Connection timed out
srun: error: cnode-11-16: task 0: Exited with exit code 28

DXLee · June 25, 2026, 8:27pm

Snaplogs have been uploaded to the provided link.

wtempel · June 25, 2026, 8:32pm

@DXLee Do you have access to the sacct command on the cluster? What is the output of the command

sacct -j 10006470 -onodelist,start,end

DXLee · June 25, 2026, 8:34pm

[ledavid@arc-infra-1 ledavid]$ sacct -j 10006470 -onodelist,start,end
       NodeList               Start                 End
--------------- ------------------- -------------------
    cnode-11-16 2026-06-18T14:52:13 2026-06-18T14:54:17
    cnode-11-16 2026-06-18T14:52:13 2026-06-18T14:54:17
    cnode-11-16 2026-06-18T14:52:13 2026-06-18T14:54:17

wtempel · June 25, 2026, 8:38pm

Do you know what distinguishes

from the others?

DXLee · June 25, 2026, 8:41pm

As far as I know, there are no differences. I have forwarded this question to our IT group.

DXLee · June 25, 2026, 9:46pm

The node in question had an invalid default route, which kept it from being able to contact the reichow-cs2 computer. Hopefully fixing it will correct our issue.

Thank you for helping solve the problem.