Cryosparc-tools hanging

Troubleshooting
1

When we try to use cryosparc-tools (current 4.6.1) instantiating the CryoSPARC object hangs indefinitely with no output. Everything else about our instance (4.6.2) is operating normally.

Has any one seen this before / is it because of the minor version number difference? Thanks.

To clarify - the HTTP request times out:

UserWarning: *** CommandClient: command (http://site.edu:39003/api) did not reply within timeout of 10 seconds, attempt 1 of 3
  system = self._get_callable("system.describe")()

The status for the command core and other components are all “RUNNING” (except dev api).

2

@DanielAsarnow Please can you post the outputs of these commands (on the CryoSPARC master):

ls -l /path/to/cryosparc_master/run/
ps -eo user:16,pid,ppid,start,command | grep cryosparc_
cryosparcm log subervisord | tail -n 40

and test whether volumes can be displayed inside the app at http://site.edu:39000.
After running these tests, can you try if functionality can be recovered by running

cryosparcm restart command_vis
3

Nice, that seems to have fixed everything. Outputs of the test commands below, JIC. From before restarting the command_vis. Volume display was also indeed broken.

[cryosparc@orca ~]$ ls -1 cryosparc2_master/run/
app_api.log
app_api.log.1
app_api.log.2
app_api.log.3
app_api.log.4
app.log
app.log.1
app.log.10
app.log.2
app.log.3
app.log.4
app.log.5
app.log.6
app.log.7
app.log.8
app.log.9
command_core.log
command_core.log.1
command_core.log.2
command_core.log.3
command_core.log.4
command_core.log.5
command_core.log.6
command_rtp.log
command_rtp.log.1
command_vis.log
command_vis.log.1
database.log
database.log.1
database.log.2
database.log.3
database.log.4
supervisord.log
supervisord.pid
update.log
vis
webapp.log

[cryosparc@orca ~]$ ps -eo user:16,pid,ppid,start,command | grep cryosparc_
cryosparc        15693     1   Nov 25 python /home_local/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home_local/cryosparc/cryosparc2_master/supervisord.conf
cryosparc        15815 15693   Nov 25 mongod --auth --dbpath /home_local/cryosparc/cryosparc_db --port 39001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
cryosparc        15934 15693   Nov 25 python /home_local/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home_local/cryosparc/cryosparc2_master/gunicorn.conf.py
cryosparc        15935 15934   Nov 25 python /home_local/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home_local/cryosparc/cryosparc2_master/gunicorn.conf.py
cryosparc        15987 15693   Nov 25 python /home_local/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:39003 -c /home_local/cryosparc/cryosparc2_master/gunicorn.conf.py
cryosparc        15988 15987   Nov 25 python /home_local/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:39003 -c /home_local/cryosparc/cryosparc2_master/gunicorn.conf.py
cryosparc        16005 15693   Nov 25 python /home_local/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:39005 -c /home_local/cryosparc/cryosparc2_master/gunicorn.conf.py
cryosparc        16006 16005   Nov 25 python /home_local/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:39005 -c /home_local/cryosparc/cryosparc2_master/gunicorn.conf.py
cryosparc        16054 15693   Nov 25 /home_local/cryosparc/cryosparc2_master/cryosparc_app/nodejs/bin/node ./bundle/main.js
cryosparc        26861 25007 14:21:42 grep --color=auto cryosparc_
cryosparc        30550 15935   Dec 29 python -c import cryosparc_compute.run as run; run.run() --project P857 --job J25 --master_hostname orca.cryoem.local --master_command_core_port 39002
cryosparc        30578 30550   Dec 29 python -c import cryosparc_compute.run as run; run.run() --project P857 --job J25 --master_hostname orca.cryoem.local --master_command_core_port 39002

[cryosparc@orca ~]$ cryosparcm log supervisord | tail -n 40
2024-11-25 08:38:19,446 INFO success: app entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-25 08:38:19,697 INFO spawned: 'app_api' with pid 15585
2024-11-25 08:38:20,702 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-25 08:38:21,606 INFO waiting for app to stop
2024-11-25 08:38:21,606 INFO waiting for app_api to stop
2024-11-25 08:38:21,606 INFO waiting for command_core to stop
2024-11-25 08:38:21,606 INFO waiting for command_rtp to stop
2024-11-25 08:38:21,606 INFO waiting for command_vis to stop
2024-11-25 08:38:21,606 INFO waiting for database to stop
2024-11-25 08:38:21,616 WARN stopped: app_api (terminated by SIGTERM)
2024-11-25 08:38:21,617 WARN stopped: app (terminated by SIGTERM)
2024-11-25 08:38:22,073 INFO stopped: command_core (exit status 0)
2024-11-25 08:38:22,231 INFO stopped: database (exit status 0)
2024-11-25 08:38:22,249 INFO stopped: command_rtp (exit status 0)
2024-11-25 08:38:22,356 INFO stopped: command_vis (exit status 0)
2024-11-25 08:38:23,854 INFO RPC interface 'supervisor' initialized
2024-11-25 08:38:23,854 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2024-11-25 08:38:23,855 INFO daemonizing the supervisord process
2024-11-25 08:38:23,855 INFO supervisord started with pid 15693
2024-11-25 08:38:28,366 INFO spawned: 'database' with pid 15815
2024-11-25 08:38:30,005 INFO success: database entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-25 08:38:31,966 INFO spawned: 'command_core' with pid 15934
2024-11-25 08:38:37,898 INFO success: command_core entered RUNNING state, process has stayed up for > than 5 seconds (startsecs)
2024-11-25 08:38:45,114 INFO spawned: 'command_vis' with pid 15987
2024-11-25 08:38:46,337 INFO success: command_vis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-25 08:38:46,507 INFO spawned: 'command_rtp' with pid 16005
2024-11-25 08:38:47,728 INFO success: command_rtp entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-25 08:38:50,613 INFO spawned: 'app' with pid 16035
2024-11-25 08:38:51,615 INFO success: app entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-11-25 08:38:51,854 INFO spawned: 'app_api' with pid 16054
2024-11-25 08:38:52,854 INFO success: app_api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-01-08 14:22:24,136 INFO waiting for command_vis to stop
2025-01-08 14:22:26,132 INFO waiting for command_vis to stop
2025-01-08 14:22:28,132 INFO waiting for command_vis to stop
2025-01-08 14:22:30,132 INFO waiting for command_vis to stop
2025-01-08 14:22:32,132 INFO waiting for command_vis to stop
2025-01-08 14:22:34,075 WARN killing 'command_vis' (15987) with SIGKILL
2025-01-08 14:22:34,148 WARN stopped: command_vis (terminated by SIGKILL)
2025-01-08 14:22:35,710 INFO spawned: 'command_vis' with pid 27120
2025-01-08 14:22:37,560 INFO success: command_vis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
1 Like