Cryosparc v3.3.1 - Error connecting to cryoSPARC license server

Troubleshooting
1

Hi @stephan,
we have recently upgraded to the v3.3.1. Our setup is cluster type with slurm. Today every job we are trying to run returns:

Error connecting to cryoSPARC license server. Checking local license file. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)>Local license file is expired. Please re-establish your connection to the license servers.

I have gone through similar posts and I can confirm that everything suggested returns expected values. Except we cannot submit any job

What else I can check?

curl https://get.cryosparc.com/checklicenseexists/$LICENSE_ID | python3 -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    17  100    17    0     0     11      0  0:00:01  0:00:01 --:--:--    11
{
    "success": true
}

Ping:

$ ping -c 3 activate.cryosparc.com
PING activate.cryosparc.com (13.35.89.58) 56(84) bytes of data.
64 bytes from server-13-35-89-58.lax3.r.cloudfront.net (13.35.89.58): icmp_seq=1 ttl=243 time=1.21 ms
64 bytes from server-13-35-89-58.lax3.r.cloudfront.net (13.35.89.58): icmp_seq=2 ttl=243 time=1.24 ms
64 bytes from server-13-35-89-58.lax3.r.cloudfront.net (13.35.89.58): icmp_seq=3 ttl=243 time=1.30 ms

--- activate.cryosparc.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.217/1.256/1.303/0.045 ms

curl:

$ curl https://get.cryosparc.com | python3 -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    42  100    42    0     0    165      0 --:--:-- --:--:-- --:--:--   165
{
    "message": "Missing Authentication Token"
}

cryosparcm status:

----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/cryoem/software/cryosparc_master
Current cryoSPARC version: v3.3.1
----------------------------------------------------------------------------

CryoSPARC process status:

app                              RUNNING   pid 3138585, uptime 0:13:04
app_dev                          STOPPED   Not started
command_core                     RUNNING   pid 3138499, uptime 0:13:17
command_rtp                      RUNNING   pid 3138533, uptime 0:13:12
command_vis                      RUNNING   pid 3138518, uptime 0:13:14
database                         RUNNING   pid 3138374, uptime 0:13:20
liveapp                          STOPPED   Not started
liveapp_dev                      STOPPED   Not started
webapp                           RUNNING   pid 3138569, uptime 0:13:06
webapp_dev                       STOPPED   Not started

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID="xxx"
export CRYOSPARC_MASTER_HOSTNAME="xxx.xxx.edu"
export CRYOSPARC_DB_PATH="/var/cryosparcdb"
export CRYOSPARC_BASE_PORT=50500
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=true
export CRYOSPARC_CLICK_WRAP=true
export CRYOSPARC_FORCE_HOSTNAME=true
export CRYOSPARC_HEARTBEAT_SECONDS=180

cryosparcm log command_core

2021-12-09 14:47:28,050 COMMAND.CORE         scheduler_run_core   WARNING  | Error connecting to cryoSPARC license server. Checking local license file. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)>
2

@osinskit What’s the output of
cryosparcm log command_core -n COMMAND.CORE -f get_ssl_context -f request_run?

3

Nothing really:

~
~
~
~
~
~
~
~
~
~
...skipping...
Waiting for data... (interrupt to abort)
4

cryeosparcm status output looks the same as previously
Can our setup be an issue?
The server where the master is hosted is not accessible directly from the internet. The firewall is forwarding a few ports, so there is no issues with accessing the internet, but the server itself is not accessible.
Obviously the compute nodes on the cluster do not have access to the internet at all.
What port is being used by the cryosparc to contact the license server?

5

@osinskit Thank you for reporting the problem. We have identified a likely cause and are working on a fix.

6

Great! Please keep me posted

7

I can confirm this also happened to me once with v3.3.0, but after restarting cryoSPARC I haven’t seen the issue again.

8

I have restarted cryosparc several times, but it did not fix the issue unfortunately

9

@osinskit This patch should address the issue you are experiencing.

10

Patch 211214 solved the issue. Amazing work, thank you!