Error connecting to cryoSPARC license server (again)

doug-oneal-nih · January 25, 2022, 7:18pm

It seems like this topic comes up periodically but not with a definitive answer. Launching jobs from the server returns the error

Error connecting to cryoSPARC license server. Checking local license file. <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)>Local license file is expired. Please re-establish your connection to the license servers.

We have gotten this periodically and have cleared it in the past by rebooting the server. Not an optimal solution. The master submits to a slurm cluster and generally performs well. Diagnostics are

$ cryosparcm status

CryoSPARC System master node installed at
/home/[redacted]/cryosparc_master
Current cryoSPARC version: v3.3.1+211214

CryoSPARC process status:

app RUNNING pid 2573, uptime 7 days, 1:01:24
app_dev STOPPED Not started
command_core RUNNING pid 2303, uptime 7 days, 1:01:58
command_rtp RUNNING pid 2497, uptime 7 days, 1:01:41
command_vis RUNNING pid 2478, uptime 7 days, 1:01:43
database RUNNING pid 2092, uptime 7 days, 1:02:04
liveapp STOPPED Not started
liveapp_dev STOPPED Not started
webapp RUNNING pid 2564, uptime 7 days, 1:01:25
webapp_dev STOPPED Not started

License is valid

global config variables:

export CRYOSPARC_LICENSE_ID="[redacted]"
export CRYOSPARC_MASTER_HOSTNAME="[redacted]"
export CRYOSPARC_DB_PATH="/var/lib/mdb"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true
export CRYOSPARC_HEARTBEAT_SECONDS=180

$ curl https://get.cryosparc.com/checklicenseexists/$LICENSE_ID
{“success”: true}

The general suggestion is to set CRYOSPARC_INSECURE to true. Are there any drawbacks to implementing this? We have several other cryoSPARC instances in the organization and this one is the only one experiencing the problem.

wtempel · January 26, 2022, 7:13pm

Welcome to the forum @doug-oneal-nih.
There are drawbacks to enabling CRYOSPARC_INSECURE.
You may want to rule out the possibility that SSL validation fails due to an aberrant system time on the computer in question. Does the date command show an unexpected system time/date?
If not, please can you share outputs of

cryosparcm log command_core
eval $(cryosparcm env) && env

Please remove confidential information before posting.

doug-oneal-nih · January 26, 2022, 8:07pm

The system does have a reasonable date configured. I have also tested SSL connections to other servers and the certificate handshakes come back ok. System time isn’t an issue.

Easier one first.

$ eval $(cryosparcm env) && env
CRYOSPARC_MASTER_HOSTNAME=[redacted]
CRYOSPARC_HTTP_PORT=39000
MANPATH=:/opt/puppetlabs/puppet/share/man
CRYOSPARC_CLICK_WRAP=true
XDG_SESSION_ID=340
CRYOSPARC_COMMAND_VIS_PORT=39003
LICENSE_ID=[redacted]
HOSTNAME=[redacted]
SHELL=/bin/bash
TERM=xterm
PYTHONNOUSERSITE=true
HISTSIZE=1000
CONDA_SHLVL=1
CONDA_PROMPT_MODIFIER=(cryosparc_master_env)
LD_PRELOAD=[redacted]
CRYOSPARC_CONDA_ENV=cryosparc_master_env
CRYOSPARC_FORCE_USER=false
CRYOSPARC_INSECURE=true
CRYOSPARC_DEVELOP=false
USER=[redacted]
LD_LIBRARY_PATH=[redacted]
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.axv=01;35:.anx=01;35:.ogv=01;35:.ogx=01;35:.aac=01;36:.au=01;36:.flac=01;36:.mid=01;36:.midi=01;36:.mka=01;36:.mp3=01;36:.mpc=01;36:.ogg=01;36:.ra=01;36:.wav=01;36:.axa=01;36:.oga=01;36:.spx=01;36:*.xspf=01;36:
CRYOSPARC_DB_PATH=/var/lib/mdb
CONDA_EXE=[redacted]
CRYOSPARC_HTTP_RTP_PORT=39006
CRYOSPARC_LICENSE_ID=[redacted]
CRYOSPARC_MONGO_PORT=39001
CRYOSPARC_HOSTNAME_CHECK=[redacted]
MAIL=[redacted]
PATH=[redacted]
CONDA_PREFIX=[redacted]
CRYOSPARC_MONGO_CACHE_GB=4
CRYOSPARC_HEARTBEAT_SECONDS=180
CRYOSPARC_ROOT_DIR=[redacted]
PWD=[redacted]
CRYOSPARC_HTTP_RTP_LEGACY_PORT=39007
LANG=en_US.UTF-8
CRYOSPARC_BASE_PORT=39000
CRYOSPARC_COMMAND_CORE_PORT=39002
HISTCONTROL=ignoredups
SHLVL=1
HOME=[redacted]
CRYOSPARC_PATH=[redacted]
CRYOSPARC_LIVE_ENABLED=true
CONDA_PYTHON_EXE=[redacted]
PYTHONPATH=[redacted]
CRYOSPARC_SUPERVISOR_SOCK_FILE=/tmp/cryosparc-supervisor-b4f5f42e4c02908dcaa7cb91ee6ed245.sock
CRYOSPARC_COMMAND_RTP_PORT=39005
LOGNAME=[redacted]
CONDA_DEFAULT_ENV=cryosparc_master_env
LESSOPEN=||/usr/bin/lesspipe.sh %s
CRYOSPARC_FORCE_HOSTNAME=false
_=/bin/env

$ cryosparcm log command_core
[redacted]…
2022-01-26 14:51:42,367 COMMAND.DATA 2022-01-26 14:51:42,368 COMMAND.DATA 2022-01-26 14:51:42,368 COMMAND.DATA 2022-01-26 14:51:42,423 COMMAND.DATA 2022-01-26 14:51:42,423 COMMAND.DATA 2022-01-26 14:51:42,425 COMMAND.DATA 2022-01-26 14:51:42,433 COMMAND.DATA 2022-01-26 14:51:42,440 COMMAND.DATA 2022-01-26 14:51:42,453 COMMAND.DATA 2022-01-26 14:51:42,454 COMMAND.DATA 2022-01-26 14:51:42,461 COMMAND.DATA 2022-01-26 14:51:42,461 COMMAND.DATA 2022-01-26 14:51:42,478 COMMAND.JOBS 2022-01-26 14:51:42,482 COMMAND.CORE 2022-01-26 14:53:47,320 COMMAND.DATA 2022-01-26 14:53:47,325 COMMAND.DATA 2022-01-26 14:53:47,327 COMMAND.DATA 2022-01-26 14:53:47,473 COMMAND.DATA 2022-01-26 14:53:47,473 COMMAND.DATA 2022-01-26 14:53:47,473 COMMAND.DATA 2022-01-26 14:53:47,844 COMMAND.DATA 2022-01-26 14:53:47,844 COMMAND.DATA 2022-01-26 14:53:47,846 COMMAND.DATA 2022-01-26 14:53:47,861 COMMAND.DATA 2022-01-26 14:53:47,861 COMMAND.DATA 2022-01-26 14:53:47,868 COMMAND.DATA 2022-01-26 14:53:47,868 COMMAND.DATA 2022-01-26 14:53:47,898 COMMAND.JOBS 2022-01-26 14:53:47,901 COMMAND.CORE Waiting for data… (interrupt to abort) dump_job_database INFO | Writing 120 database images to [redacted]
dump_job_database INFO | Done. Exported 120 images in 0.16s
dump_job_database INFO | Exporting all job’s streamlog events…
dump_job_database INFO | Done. Exported 1 files in 0.06s
dump_job_database INFO | Exporting job metafile…
dump_job_database INFO | Creating .csg file for particles_all_classes
dump_job_database INFO | Creating .csg file for particles_class_0
dump_job_database INFO | Creating .csg file for volume_class_0
dump_job_database INFO | Done. Exported in 0.03s
dump_job_database INFO | Updating job manifest…
dump_job_database INFO | Done. Updated in 0.01s
dump_job_database INFO | Exported P24 J339 in 0.26s
set_job_status INFO | Status changed for P24.J339 from running to completed
run WARNING | Insecure mode - no SSL for external requests
dump_job_database INFO | Request to export P43 J8
dump_job_database INFO | Exporting job to /mnt/beegfs/lea/ZorAB_susan/P43/J8
dump_job_database INFO | Exporting all of job’s images in the database to [redacted]…
dump_job_database INFO | Writing 102 database images to /mnt/beegfs/lea/ZorAB_susan/P43/J8/gridfs_data/gridfsdata_0
dump_job_database INFO | Done. Exported 102 images in 0.15s
dump_job_database INFO | Exporting all job’s streamlog events…
dump_job_database INFO | Done. Exported 1 files in 0.37s
dump_job_database INFO | Exporting job metafile…
dump_job_database INFO | Creating .csg file for exposures
dump_job_database INFO | Done. Exported in 0.02s
dump_job_database INFO | Updating job manifest…
dump_job_database INFO | Done. Updated in 0.01s
dump_job_database INFO | Exported P43 J8 in 0.55s
set_job_status INFO | Status changed for P43.J8 from running to completed
run WARNING | Insecure mode - no SSL for external requests

wtempel · January 26, 2022, 10:15pm

@doug-oneal-nih The output indicates that CRYOSPARC_INSECURE is currently set to true. Setting CRYOSPARC_INSECURE=false is needed for troubleshooting and also preferable for cryoSPARC operation. What’s the output of cryosparcm log command_core (for completeness redirected to a file) when cryoSPARC runs with CRYOSPARC_INSECURE=false? If you prefer, you may send me the output in a direct message.

doug-oneal-nih · January 28, 2022, 1:44pm

The output of cryosparcm log command_core is 15k lines. Is there a way to upload this file to you?

wtempel · January 28, 2022, 10:30pm

@doug-oneal-nih Please can you try if updating certificates with the commands

eval $(cryosparcm env)
conda upgrade -c conda-forge ca-certificates certifi

corrects the problem?

doug-oneal-nih · January 31, 2022, 4:02pm

I have updated the two packages and restarted the cryoSPARC daemons. The problem takes several days to manifest so I’ll post an update at the end of the week.

Error connecting to cryoSPARC license server (again)

$ cryosparcm status

CryoSPARC System master node installed at /home/[redacted]/cryosparc_master Current cryoSPARC version: v3.3.1+211214

License is valid

CryoSPARC System master node installed at
/home/[redacted]/cryosparc_master
Current cryoSPARC version: v3.3.1+211214