I have been attempting to update a cryoSPARC master instance on a host running AlmaLinux 8.6. Upon doing the update from cryoSPARC 3.3.2 to 4.0.0, the master updated successfully, but attempts to update the workers all failed with the below message:
-------------------------------------------------
Updating worker cryosparc-worker1.example.com
Remote update
scp ./cryosparc_worker.tar.gz cryosparc_user@cryosparc-worker1.example.com:/opt/cryosparc/cryosparc_worker
/usr/bin/ssh: symbol lookup error: /usr/bin/ssh: undefined symbol: EVP_KDF_ctrl, version OPENSSL_1_1_1b
lost connection
Failed to update cryosparc-worker1.example.com! Skipping...
-------------------------------------------------
After manually updating the workers, attempts to use the new cryosparcm test workers command resulted in all jobs failing with a similar error:
$ cryosparcm test workers P1
Using project P1
Running worker tests...
2022-10-03 17:10:41,313 WORKER_TEST log CRITICAL | Worker test results
2022-10-03 17:10:41,313 WORKER_TEST log CRITICAL | cryosparc-worker1.example.com
2022-10-03 17:10:41,313 WORKER_TEST log CRITICAL | ✕ LAUNCH
2022-10-03 17:10:41,313 WORKER_TEST log CRITICAL | Error:
2022-10-03 17:10:41,313 WORKER_TEST log CRITICAL | See P1 J1 for more information
2022-10-03 17:10:41,313 WORKER_TEST log CRITICAL | ⚠ SSD
2022-10-03 17:10:41,313 WORKER_TEST log CRITICAL | Did not run: Launch test failed
2022-10-03 17:10:41,313 WORKER_TEST log CRITICAL | ⚠ GPU
2022-10-03 17:10:41,314 WORKER_TEST log CRITICAL | Did not run: Launch test failed
This is likely being caused due to the inclusion of the conda environment lib folders in LD_LIBRARY PATH:
$ cryosparcm env
<snip>
export "LD_LIBRARY_PATH=/opt/cryosparc/cryosparc_master/cryosparc_compute/blobio:/opt/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib"
<snip>
In order to fix it, I had to install the openssh package from conda-forge into the conda environment, in order to use its copy of ssh that can function with the openssl package that cryoSPARC installed:
$ source /opt/cryosparc/cryosparc_master/deps/anaconda/etc/profile.d/conda.sh
$ conda activate cryosparc_master_env
$ conda install -c conda-forge --freeze-installed openssh
After doing so, cryoSPARC now uses the below ssh binary, due to it coming first in PATH, and no longer errors out:
/opt/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/ssh
I have not yet verified whether this also occurs on other distros, but we also have some 3.3.2 instances that we manage running on CentOS 7 and Ubuntu hosts, so hopefully I’ll be able to find out soon once I’ve been given the go-ahead to update them as well.