Cannot connect to CryoSPARC Command Core API

Hi All,

This is a fresh install of Version 4.4.1 and I’m getting this issue. When I go to the GUI (localhost:39000/tcp) I can’t get past the software license agreement. Clicking Accept does nothing, and when inspecting element any click on the Accept button generates the following error:

Action

POST http://**[IP removed]**:39000/api/cmd/core/set_user_state_var 500 (Internal Server Error)

[command:core:set_user_state_var] Cannot connect to CryoSPARC Command Core API

We’ve tried restarting and reinstalling Cryosparc with no results. Here is some relevant information about the system and Cryosparc:

Type: Single Workstation

cryosparcm status:

----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/cryosparc_user/software/cryosparc/cryosparc_master
Current cryoSPARC version: v4.4.1
----------------------------------------------------------------------------

CryoSPARC process status:

app                              RUNNING   pid 6088, uptime 0:33:28
app_api                          RUNNING   pid 6107, uptime 0:33:27
app_api_dev                      STOPPED   Not started
command_core                     RUNNING   pid 5968, uptime 0:33:39
command_rtp                      RUNNING   pid 6041, uptime 0:33:32
command_vis                      RUNNING   pid 5999, uptime 0:33:33
database                         RUNNING   pid 5851, uptime 0:33:42

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID="****"
export CRYOSPARC_MASTER_HOSTNAME="20231109000"
export CRYOSPARC_DB_PATH="/home/cryosparc_user/software/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000
export CRYOSPARC_INSECURE=false
export CRYOSPARC_DB_ENABLE_AUTH=true
export CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10
export CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000
export CRYOSPARC_PROJECT_DIR_PREFIX='CS-'
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_CLICK_WRAP=true
export CRYOSPARC_HEARTBEAT_SECONDS=180

cryosparc log command core:

[cryosparc_user@20231109000 cryosparc_master]$ cryosparcm log command_core
2023-12-11 18:43:20,979 log_scheduler_targets INFO     |              GPU 3 Name     NVIDIA RTX A5000
2023-12-11 18:43:20,979 log_scheduler_targets INFO     |              GPU 3 Mem      23.68 GiB
2023-12-11 18:43:20,979 log_scheduler_targets INFO     |
2023-12-11 18:43:20,980 startup              INFO     | CryoSPARC instance ID: ****
2023-12-11 18:43:20,980 get_gpu_info         INFO     | UPDATING WORKER GPU INFO
2023-12-11 18:43:20,980 update_all_job_sizes INFO     | UPDATING ALL JOB SIZES IN 10s
2023-12-11 18:43:20,980 export_all_projects  INFO     | EXPORTING ALL PROJECTS IN 60s...
2023-12-11 18:43:30,993 update_all_job_sizes_run INFO     | Finished updating all job sizes (0 jobs updated, 0 projects updated)
2023-12-11 18:44:21,019 export_all_projects_run INFO     | Finished exporting all projects
2023-12-11 18:49:07,192 start                INFO     |  === STARTED ===
2023-12-11 18:49:07,193 background_worker    INFO     |  === STARTED ===
2023-12-11 18:49:07,193 run                  INFO     | === STARTED TASKS WORKER ===
 * Serving Flask app "command_core" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
2023-12-11 18:49:10,384 startup              INFO     | Starting CryoSPARC v4.4.1
2023-12-11 18:49:10,384 startup              INFO     |   platform_node : 20231109000
2023-12-11 18:49:10,384 startup              INFO     |   platform_release : 4.18.0-513.9.1.el8_9.x86_64
2023-12-11 18:49:10,384 startup              INFO     |   platform_version : #1 SMP Wed Nov 29 18:55:19 UTC 2023
2023-12-11 18:49:10,384 startup              INFO     |   platform_architecture : x86_64
2023-12-11 18:49:10,384 startup              INFO     |   cpu_model : AMD Ryzen Threadripper PRO 5965WX 24-Cores
2023-12-11 18:49:10,384 startup              INFO     |   physical_cores : 24
2023-12-11 18:49:10,384 startup              INFO     |   total_memory : 503.19GB
2023-12-11 18:49:10,384 startup              INFO     |   available_memory : 498.21GB
2023-12-11 18:49:10,384 startup              INFO     |   used_memory : 1.53GB
2023-12-11 18:49:10,384 startup              INFO     |   ofd_soft_limit : 1024
2023-12-11 18:49:10,384 startup              INFO     |   ofd_hard_limit : 262144
2023-12-11 18:49:10,384 startup              INFO     |   version : v4.4.1
2023-12-11 18:49:10,450 log_scheduler_targets INFO     | Target       Property       Value
2023-12-11 18:49:10,450 log_scheduler_targets INFO     | -----------  -------------  -----------------------------------------------------------------------
2023-12-11 18:49:10,450 log_scheduler_targets INFO     | 20231109000  Type           node
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              Host           20231109000
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              Lane           default
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              Worker         /home/cryosparc_user/software/cryosparc/cryosparc_worker/bin/cryosparcw
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              SSH Target     cryosparc_user@20231109000
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              CPUs           48
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              RAM            512 GiB
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              Cache Path     /scr/cryosparc_cache
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              Cache Reserve  10.00 GB
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              GPUs           4
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              GPU 0 Name     NVIDIA RTX A5000
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              GPU 0 Mem      23.68 GiB
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              GPU 1 Name     NVIDIA RTX A5000
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              GPU 1 Mem      23.68 GiB
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              GPU 2 Name     NVIDIA RTX A5000
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              GPU 2 Mem      23.68 GiB
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              GPU 3 Name     NVIDIA RTX A5000
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |              GPU 3 Mem      23.68 GiB
2023-12-11 18:49:10,450 log_scheduler_targets INFO     |
2023-12-11 18:49:10,451 startup              INFO     | CryoSPARC instance ID: ****
2023-12-11 18:49:10,451 get_gpu_info         INFO     | UPDATING WORKER GPU INFO
2023-12-11 18:49:10,451 update_all_job_sizes INFO     | UPDATING ALL JOB SIZES IN 10s
2023-12-11 18:49:10,452 export_all_projects  INFO     | EXPORTING ALL PROJECTS IN 60s...
2023-12-11 18:49:20,462 update_all_job_sizes_run INFO     | Finished updating all job sizes (0 jobs updated, 0 projects updated)
2023-12-11 18:50:10,516 export_all_projects_run INFO     | Finished exporting all projects

uname -a && free -g output:

Linux 20231109000 4.18.0-513.9.1.el8_9.x86_64 #1 SMP Wed Nov 29 18:55:19 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
total        used        free      shared  buff/cache   available
Mem:            503           2         499           0           1         497
Swap:             9           0           9

Welcome to the forum @chris.chau .

Does the problem also occur when you drop the /tcp suffix from the URL, instead access the GUI via
http://localhost:39000
?

Hi wtempel. I tried that and I’m still having the same issue. I also tried reinstalling cryosparc as well.

@chris.chau Please can you email us the tgz file created by the command

cryosparcm snaplogs

Sent the email thanks for the help

Thanks. Please can you check whether the CRYOSPARC_MASTER_HOSTNAME defined inside cryosparc_master/config.sh is causing trouble by either

  1. (potentially) not being allowed (see discussion) or
  2. not being resolved correctly

?

Hi wtempel,

The host name I had it set to started it with a number and swapping it to a letter fixed the issue. Thanks for the help I appreciate it.

1 Like