Dear CS-team,
A few days after our update to CS v4.5.3, all was fine. However, now experience an error with the service command_vis. It fairly regularly seems to crash in the morning. I am not entirely convinced it is related to the upgrade. These crashes lead to 2 problems:
CS volume viewer
The viewer seems to error out after 60s and the command_vis log file stops.
I can only see the last entry, when it worked, here at 08:39:57.
2024-07-01 08:39:57,189 recreate_mesh INFO | Loading mesh for P888 J16.volume.map_sharp
2024-07-01 08:39:57,201 recreate_mesh INFO | Volume path: /data/cryo-em/manual_processing/cryosparc/xxx/xxxx/xx/CS-xxxx/J16/J16_011_volume_map_sharp.mrc
2024-07-01 09:33:42 info INFO | Handling signal: quit
2024-07-01 09:33:43 info INFO | Worker exiting (pid: 2623539)
2024-07-01 09:33:54 info INFO | Starting gunicorn 20.1.0
2024-07-01 09:33:54 info INFO | Listening at: http://0.0.0.0:39003 (496696)
2024-07-01 09:33:54 info INFO | Using worker: threads
2024-07-01 09:33:54 info INFO | Booting worker with pid: 496698
I can restarted the service at 09:33:42 using cryosparcm restart command_vis without problems. Afterwards, all works fine for the next about 20-24 hours. But at around the same time in the morning it crashes again, same error in front end (see below) and last entry in log file.
CS tools API connection
While the command_vis service is down, I can’t connect to the CS instance using the CS tools API. The command is simple running until the timeout is reached.
from cryosparc.tools import CryoSPARC
cs = CryoSPARC()
Here the full error
/xxx/lib/python3.10/site-packages/cryosparc/command.py:135: UserWarning: *** CommandClient: (http:///xxx.com:39003/api) URL Error [Errno 111] Connection refused, attempt 1 of 3. Retrying in 30 seconds
system = self._get_callable("system.describe")()
/xxx/lib/python3.10/site-packages/cryosparc/command.py:135: UserWarning: *** CommandClient: (http:///xxx.com:39003/api) URL Error [Errno 111] Connection refused, attempt 2 of 3. Retrying in 30 seconds
system = self._get_callable("system.describe")()
Traceback (most recent call last):
File "/xxx/lib/python3.10/site-packages/cryosparc/command.py", line 105, in func
with make_json_request(self, "/api", data=data, _stacklevel=4) as request:
File "/xxx/lib/python3.10/contextlib.py", line 135, in __enter__
return next(self.gen)
File "/xxx/lib/python3.10/site-packages/cryosparc/command.py", line 226, in make_request
raise CommandError(error_reason, url=url, code=code, data=resdata)
cryosparc.errors.CommandError: *** (http://xxx:39003/api, code 500) URL Error [Errno 111] Connection refused
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "xxx/lib/python3.10/site-packages/cryosparc/tools.py", line 160, in __init__
self.vis = CommandClient(
File "xxx/lib/python3.10/site-packages/cryosparc/command.py", line 97, in __init__
self._reload() # attempt connection immediately to gather methods
File "xxx/lib/python3.10/site-packages/cryosparc/command.py", line 135, in _reload
system = self._get_callable("system.describe")()
File "/xxx/lib/python3.10/site-packages/cryosparc/command.py", line 108, in func
raise CommandError(
cryosparc.errors.CommandError: *** (http://xxx:39003, code 500) Encounted error from JSONRPC function "system.describe" with params ()
During handling of the above exception, another exception occurred:
Would you be able to help us finding a more permant solution? For now, I will simple restart the command_vis every few hours.
Thanks
Ferdinand