Cryosparc_worker can not connect to cryosparc_master

Install, Configure and Update
1

I installed the cryosparc master before. Recently we installed a SSD in our server, so we tried to use the SSD in cryosparc. After a lot of tries to connect the ssd to cryospar, we failed. I am thinking maybe because our cryosparc worker hasn’t been connected to cryosparc master properly, so I tried the command discussed here :Add cache/SSD after standalone install.
It shows “ConnectionError” :requests.exceptions.ConnectionError: HTTPConnectionPool(host=‘xxx.xxx.xx.x’, port=xxxxx): Max retries exceeded with url: /api (Caused by NewConnectionError(‘<urllib3.connection.HTTPConnection object at 0x7f025b363410>: Failed to establish a new connection: [Errno 111] Connection refused’)).

Can anyone help me to fix this problem?

2

Welcome to the forum @mhu.
Just to confirm:

  1. Was the cryosparcw connect [...] run on the worker host?
  2. Were CryoSPARC master processes running at the time?

Please can you post the output of the following commands:

cryosparcm cli "get_scheduler_targets()"
cryosparcm status | grep CRYOSPARC_MASTER_HOSTNAME
3

Thank you for your reply.

  1. Was the cryosparcw connect [...] run on the worker host? Yes.
  2. Were CryoSPARC master processes running at the time? We tried both when the CryoSPARC is running or not running.
4

CryoSPARC master processes need to be running during cryosparcw connect.
Additional suggestions depend on the output of

5

Thank you for your input. Finally, we decided to clean everything and re-installed the software from the beginning and everything works well now.

6

I am wrong: it shows the SSD works, but the newly installed version(3.3.1) lost a lot of features, such as I don’t have choice for 2D classification parameter optimization, I don’t have choice for exposures curate.

7

With which CryoSPARC version are you comparing v3.3.1? Did you consider updating to the latest release, currently v4.2.0 with the 230302 patch, which has additional features and bug fixes?
In v3.3.1, have you enabled access to advanced parameters using the (unlabeled) toggle?
Screenshot 2023-03-08 at 12.03.04

8

Thank you for your quick response. Yes, I updated to the newest version and trying now.

9

After the command “cryosparcm --update” finished. I tried to run a run a job. The Job can only laugh but not run. It shows : "================= CRYOSPARCW ======= 2023-03-08 10:46:51.244486 =========
Project P2 Job J12
Master TBGPU Port 61002

========= monitor process now starting main process
MAINPROCESS PID 2172144
Traceback (most recent call last):
File “”, line 1, in
File “cryosparc_worker/cryosparc_compute/run.py”, line 162, in cryosparc_compute.run.run
File “/Data/local/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 97, in connect
assert cli.test_connection(), “Job could not connect to master instance at %s:%s” % (master_hostname, str(master_command_core_port))
File “/Data/local/cryosparc_worker/cryosparc_compute/client.py”, line 64, in func
assert False, res[‘error’]
AssertionError: {‘code’: 403, ‘data’: None, ‘message’: ‘ServerError: Authentication failed - License-ID request header missing.\n This may indicate that cryosparc_worker did not update,\n cryosparc_worker/config.sh is missing a CRYOSPARC_LICENSE_ID entry,\n or CRYOSPARC_LICENSE_ID is not present in your environment.\n See CryoSPARC Architecture and System Requirements - CryoSPARC Guide for more details.\n’, ‘name’: ‘ServerError’}
Process Process-1:
Traceback (most recent call last):
File “/Data/local/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/multiprocessing/process.py”, line 297, in _bootstrap
self.run()
File “/Data/local/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/multiprocessing/process.py”, line 99, in run
self._target(*self._args, **self._kwargs)
File “cryosparc_worker/cryosparc_compute/run.py”, line 31, in cryosparc_compute.run.main
File “/Data/local/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 97, in connect
assert cli.test_connection(), “Job could not connect to master instance at %s:%s” % (master_hostname, str(master_command_core_port))
File “/Data/local/cryosparc_worker/cryosparc_compute/client.py”, line 64, in func
assert False, res[‘error’]
AssertionError: {‘code’: 403, ‘data’: None, ‘message’: ‘ServerError: Authentication failed - License-ID request header missing.\n This may indicate that cryosparc_worker did not update,\n cryosparc_worker/config.sh is missing a CRYOSPARC_LICENSE_ID entry,\n or CRYOSPARC_LICENSE_ID is not present in your environment.\n See CryoSPARC Architecture and System Requirements - CryoSPARC Guide for more details.\n’, ‘name’: ‘ServerError’}"

I ran the command “cryosparcm test install”, it shows:

image
image996×356 92.7 KB

10

Then I ran the command “cryosparcm patch” and “cryosparcm patch --force”, both show “Patch not fully applied”

image
image1103×462 96.9 KB

Is the patch necessary for the jobs run succesfully?

11

The patch is recommended. Release notes can help you decide whether to apply the patch.
To ensure that releases and patch levels match instance-wide, the update and patch becomes necessary for all cryosparc_worker installations of a given CryoSPARC instance once the CryoSPARC master has been updated and patched. It seems update and/or patching of worker nodes failed after successful master update and patching. Please follow the instructions for manual worker update and patching under the Cluster tab of that guide section.

12

I cleaned all the previous installation again and downloaded the latest version of cryosparc_master and cryosparc_worker. Installed again. When I run job, it shows the master and workers are not matchAssertionError: Version mismatch! Worker version (v4.2.0) and Master version (v4.2.0+230302) are not the same. Please update.
But it shows my Master version is v4.2.0 when I specified to version.
image

13

You may want to try applying patch 230302 to the cryosparc_worker installation(s) following the Cluster worker patch instructions