Error connecting to cryoSPARC license server, v4.2.1

Install, Configure and Update
1

Hi,

I have cryoSPARC 4.2.1 server running on an HPC system. I have run many jobs successfully. However, suddenly, I got an error yesterday afternoon when submitting jobs. Here is a screenshot.

Screen Shot 2023-05-02 at 1.25.16 PM

My license is valid.

ā–¶ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/vf/users/zhuj6/apps/cryosparc4/cryosparc_master
Current cryoSPARC version: v4.2.1+230427
----------------------------------------------------------------------------

CryoSPARC process status:

app                              RUNNING   pid 6539, uptime 0:36:17
app_api                          RUNNING   pid 8849, uptime 0:36:15
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     RUNNING   pid 6014, uptime 0:36:37
command_rtp                      RUNNING   pid 6269, uptime 0:36:28
command_vis                      RUNNING   pid 6221, uptime 0:36:30
database                         RUNNING   pid 5842, uptime 0:36:42

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID="****************************"
export CRYOSPARC_MASTER_HOSTNAME="cn2317"
export CRYOSPARC_DB_PATH="/data/zhuj6/apps/cryosparc4/cryosparc_database"
export CRYOSPARC_BASE_PORT=39022
export CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000
export CRYOSPARC_INSECURE=false
export CRYOSPARC_DB_ENABLE_AUTH=true
export CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10
export CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000
export CRYOSPARC_PROJECT_DIR_PREFIX='CS-'
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_CLICK_WRAP=true
export CRYOSPARC_HEARTBEAT_SECONDS=180

I replaced my license with ā€œ*ā€. Here is the last bit of the cryosparcm log command_core -n COMMAND.CORE -f get_ssl_context -f request_run output.

2023-05-02 13:15:15,459 COMMAND.SCHEDULER    scheduler_run_core   INFO     | Licenses currently active : 0
2023-05-02 13:15:15,459 COMMAND.SCHEDULER    scheduler_run_core   INFO     | Now trying to schedule J100
2023-05-02 13:15:15,459 COMMAND.SCHEDULER    scheduler_run_core   INFO     |    Scheduling directly onto master node cn2317
2023-05-02 13:15:15,459 COMMAND.SCHEDULER    scheduler_run_job_master_direct INFO     |    Scheduling directly onto master node cn2317
2023-05-02 13:15:15,562 COMMAND.CORE         scheduler_run_job_master_direct WARNING  | Error connecting to cryoSPARC license server. Checking local license file. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>
2023-05-02 13:15:15,563 COMMAND.SCHEDULER    scheduler_run_core   INFO     |     Queue status waiting_licenses
2023-05-02 13:15:15,563 COMMAND.SCHEDULER    scheduler_run_core   INFO     |     Queue message Error connecting to cryoSPARC license server. Checking local license file. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)>Local license file is expired. Please re-establish your connection to the license servers.
2023-05-02 13:15:15,564 COMMAND.SCHEDULER    scheduler_run_core   INFO     | Finished

Based on the questions on this forum, I know this issue has come up quite a few times before. But I still have not figured out how to fix it. Please let me know if you have any suggestions. Thanks!

2

Is it possible that outgoing https requests are intercepted by a proxy?

3

I tried the following on the node running my server.

ā–¶ curl https://get.cryosparc.com/checklicenseexists/$LICENSE_ID | python3 -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    17  100    17    0     0     73      0 --:--:-- --:--:-- --:--:--    73
{
    "success": true
}

ā–¶ curl https://get.cryosparc.com
{"message":"Missing Authentication Token"}%

What else can I check?

4

The cryosparc server and all jobs are behind a squid proxy. We can use http, https, and ftp queries on the proxy.

5

Please can you try to
append
export REQUESTS_CA_BUNDLE=/path/to/custom_ca_bundle.crt
to the
/vf/users/zhuj6/apps/cryosparc4/cryosparc_master/config.sh file
and restart CryoSPARC to activate that setting.

6

The certificate bundle is not the entire story. What started this problem is that the installation of the server was moved from one location (/gs9/users) to another (/vf/users). The conda code was edited with the new filepath, and the server was able to start, but when jobs are created the license error pops up. So, the question is, how to deal with a complete change in the absolute path of the cryosparc server?

7

It is related to this discussion: Problems with license when installation moved to a new disk

8

You could try (all under the Linux account that runs CryoSPARC):

  1. cryosparcm stop.
  2. confirm all CryoSPARC processes have exited, for example:
    ps -u $USER -opid,ppid,cmd | grep -e cryosparc -e mongo
  3. ensure that CRYOSPARC_DB_PATH, defined in /new/path/to/cryosparc_master/config.sh, corresponds to the current path of the database directory
  4. cryosparcm forcedeps
  5. cryosparcm start
  6. if /path/to/cryosparc_worker also has changed:
    /new/path/to/cryosparc_worker/bin/cryosparcw forcedeps

Please let us know if these steps work for you.

9

Sorry, I cannot test it anymore. I needed to move the projects forward. I had to reinstall the server, and things are fine now.

1 Like
10