Hi All,
I’ve got cryoSPARC running on a small cluster setup with two (mostly) identical nodes. After updating to 4.7.0 one of the two nodes seems to have issues starting up cryosparc live workers. They will start up and then fail with an error, get killed and restart endlessly. I managed to snag the error below (lightly redacted) between cycles of crash and restart. I have tried forcing the update again on this node to see if it was a dependency issue, but it doesn’t seem to have helped. Everything I have tried other than cs live seems to work just fine on this node.
Best,
Nathanael
Blockquote
Traceback (most recent call last):
File “~/software/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py”, line 105, in func
with make_json_request(self, “/api”, data=data, _stacklevel=4) as request:
File “~/software/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/contextlib.py”, line 135, in enter
return next(self.gen)
File “~/software/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py”, line 226, in make_request
raise CommandError(error_reason, url=url, code=code, data=resdata)
cryosparc_tools.cryosparc.errors.CommandError: *** (http://REDACTED:61005/api, code 500) URL Error [Errno 113] No route to host
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 129, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/rtp_workers/run.py”, line 323, in cryosparc_master.cryosparc_compute.jobs.rtp_workers.run.rtp_worker
File “~/software/cryosparc/cryosparc_worker/cryosparc_compute/jobs/rtp_workers/rtp_common.py”, line 24, in get_rtp_cli
rtp = client.CommandClient(sysinfo[‘master_hostname’], int(sysinfo[‘port_command_rtp’]), service=“command_rtp”)
File “~/software/cryosparc/cryosparc_worker/cryosparc_compute/client.py”, line 38, in init
super().init(service, host, port, url, timeout, headers, cls=NumpyEncoder)
File “~/software/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py”, line 97, in init
self._reload() # attempt connection immediately to gather methods
File “~/software/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py”, line 135, in _reload
system = self._get_callable(“system.describe”)()
File “~/software/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py”, line 108, in func
raise CommandError(
cryosparc_tools.cryosparc.errors.CommandError: *** (http://REDACTED:61005, code 500) Encounted error from JSONRPC function “system.describe” with params ()