Volume viewer error

Dear CS-team,
A few days after our update to CS v4.5.3, all was fine. However, now experience an error with the service command_vis. It fairly regularly seems to crash in the morning. I am not entirely convinced it is related to the upgrade. These crashes lead to 2 problems:

CS volume viewer

The viewer seems to error out after 60s and the command_vis log file stops.
I can only see the last entry, when it worked, here at 08:39:57.

2024-07-01 08:39:57,189 recreate_mesh        INFO     | Loading mesh for P888 J16.volume.map_sharp
2024-07-01 08:39:57,201 recreate_mesh        INFO     | Volume path: /data/cryo-em/manual_processing/cryosparc/xxx/xxxx/xx/CS-xxxx/J16/J16_011_volume_map_sharp.mrc
2024-07-01 09:33:42 info                 INFO     | Handling signal: quit
2024-07-01 09:33:43 info                 INFO     | Worker exiting (pid: 2623539)
2024-07-01 09:33:54 info                 INFO     | Starting gunicorn 20.1.0
2024-07-01 09:33:54 info                 INFO     | Listening at: http://0.0.0.0:39003 (496696)
2024-07-01 09:33:54 info                 INFO     | Using worker: threads
2024-07-01 09:33:54 info                 INFO     | Booting worker with pid: 496698

I can restarted the service at 09:33:42 using cryosparcm restart command_vis without problems. Afterwards, all works fine for the next about 20-24 hours. But at around the same time in the morning it crashes again, same error in front end (see below) and last entry in log file.

CS tools API connection

While the command_vis service is down, I can’t connect to the CS instance using the CS tools API. The command is simple running until the timeout is reached.

from cryosparc.tools import CryoSPARC
cs = CryoSPARC()
Here the full error
/xxx/lib/python3.10/site-packages/cryosparc/command.py:135: UserWarning: *** CommandClient: (http:///xxx.com:39003/api) URL Error [Errno 111] Connection refused, attempt 1 of 3. Retrying in 30 seconds
  system = self._get_callable("system.describe")()
/xxx/lib/python3.10/site-packages/cryosparc/command.py:135: UserWarning: *** CommandClient: (http:///xxx.com:39003/api) URL Error [Errno 111] Connection refused, attempt 2 of 3. Retrying in 30 seconds
  system = self._get_callable("system.describe")()
Traceback (most recent call last):
  File "/xxx/lib/python3.10/site-packages/cryosparc/command.py", line 105, in func
    with make_json_request(self, "/api", data=data, _stacklevel=4) as request:
  File "/xxx/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/xxx/lib/python3.10/site-packages/cryosparc/command.py", line 226, in make_request
    raise CommandError(error_reason, url=url, code=code, data=resdata)
cryosparc.errors.CommandError: *** (http://xxx:39003/api, code 500) URL Error [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "xxx/lib/python3.10/site-packages/cryosparc/tools.py", line 160, in __init__
    self.vis = CommandClient(
  File "xxx/lib/python3.10/site-packages/cryosparc/command.py", line 97, in __init__
    self._reload()  # attempt connection immediately to gather methods
  File "xxx/lib/python3.10/site-packages/cryosparc/command.py", line 135, in _reload
    system = self._get_callable("system.describe")()
  File "/xxx/lib/python3.10/site-packages/cryosparc/command.py", line 108, in func
    raise CommandError(
cryosparc.errors.CommandError: *** (http://xxx:39003, code 500) Encounted error from JSONRPC function "system.describe" with params ()

During handling of the above exception, another exception occurred:

Would you be able to help us finding a more permant solution? For now, I will simple restart the command_vis every few hours.

Thanks
Ferdinand

Hi @fkrupp, thank you for the detailed report! We are actively investigating this and hoping to have a fix in the next CryoSPARC release.

The code in this post is outdated and may disrupt CryoSPARC function. Do not use. @fkrupp We have been investigating the issue and though we have not been able to reproduce it internally yet, we have identified a potential configuration change that may help.

To apply the change, please

  1. stop CryoSPARC
    cryosparcm stop
  2. append the following line to the file cryopsparc_master/config.sh
    # Outdated advise. Do not use this code.
    export GUNICORN_CMD_ARGS="--worker-class gthread --threads 8 --max-requests 1000"
    
  3. start CryoSPARC
    cryosparcm start

Please let us know if this configuration change helps.

[edited 2024-09-19]

Patch 240807 for CryoSPARC v4.5.3 includes a fix for this issue.
Before applying the patch, please remove from or comment out in

/path/to/cryosparc_master/config.sh

any

export GUNICORN_CMD_ARGS=

line.
After patching, please restart CryoSPARC.

1 Like

@fkrupp @builab Users who experienced problems with the web app or cryosparc-tools in CryoSPARC version 4.5.x may have, as a workaround, included a line beginning with

export GUNICORN_CMD_ARGS=

in the cryosparc_master/config.sh file.

We have learned that these custom settings may disrupt an Attach Project action and other long-running processes.

If the cryosparc_master/config.sh file of your CryoSPARC instance includes a definition of GUNICORN_CMD_ARGS, we recommend you

  1. remove from cryosparc_master/config.sh any line that includes the GUNICORN_CMD_ARGS definition
  2. then, if the instance version is
    1. 4.5.3 with the 240807 patch applied or a newer version of CryoSPARC, restart CryoSPARC
    2. 4.5.3 without the 240807 patch, apply patch 240807 for v4.5.3 or upgrade CryoSPARC to the latest version (v4.6.0 at time of this writing)
    3. 4.5.0, v4.5.1 or v4.5.2, upgrade CryoSPARC to the latest version
  3. examine CryoSPARC projects for which the Attach Project action has been performed while custom GUNICORN_CMD_ARGS were effective. It is possible that not all jobs present in the relevant project directory at that time were imported. The affected job directories may be overwritten by jobs newly created in the incompletely-attached project.