[Fixed] Jobs stuck in launched-mode after update to v4.0.1

Install, Configure and Update
1

Hi,

I updated cryosparc to version v4.0.1 and since then I cannot start a job since it stays in “launched”-mode forever. The only job I could get to run was an “import micrographs”-job.
I tried running the command cryosparcm test worker which told me that there is indeed a problem while launching the jobs.
The log of the test job looks as follows:

MAINPROCESS PID 33040
Process Process-1:
Traceback (most recent call last):
File “/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/multiprocessing/process.py”, line 297, in _bootstrap
self.run()
File “/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/multiprocessing/process.py”, line 99, in run
self._target(*self._args, **self._kwargs)
File “cryosparc_worker/cryosparc_compute/run.py”, line 31, in cryosparc_compute.run.main
File “/home/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 97, in connect
assert cli.test_connection(), “Job could not connect to master instance at %s:%s” % (master_hostname, str(master_command_core_port))
File “/home/cryosparc/cryosparc_worker/cryosparc_compute/client.py”, line 64, in func
assert False, res[‘error’]
AssertionError: {‘code’: 403, ‘data’: None, ‘message’: ‘ServerError: Authentication failed - License-ID request header missing.\n This may indicate that cryosparc_worker did not update,\n cryosparc_worker/config.sh is missing a CRYOSPARC_LICENSE_ID entry,\n or CRYOSPARC_LICENSE_ID is not present in your environment.\n See CryoSPARC Architecture and System Requirements - CryoSPARC Guide for more details.\n’, ‘name’: ‘ServerError’}
Traceback (most recent call last):
File “”, line 1, in
File “cryosparc_worker/cryosparc_compute/run.py”, line 162, in cryosparc_compute.run.run
File “/home/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 97, in connect
assert cli.test_connection(), “Job could not connect to master instance at %s:%s” % (master_hostname, str(master_command_core_port))
File “/home/cryosparc/cryosparc_worker/cryosparc_compute/client.py”, line 64, in func
assert False, res[‘error’]
AssertionError: {‘code’: 403, ‘data’: None, ‘message’: ‘ServerError: Authentication failed - License-ID request header missing.\n This may indicate that cryosparc_worker did not update,\n cryosparc_worker/config.sh is missing a CRYOSPARC_LICENSE_ID entry,\n or CRYOSPARC_LICENSE_ID is not present in your environment.\n See CryoSPARC Architecture and System Requirements - CryoSPARC Guide for more details.\n’, ‘name’: ‘ServerError’}

Here, it says that the license-ID is somehow not read correctly. But when I run the command cryosparcm test install it says that my license is valid. It also confirms that my license is valid when I queue a job.
I also tried to force-install the dependencies again with the command cryosparcm forcedeps but that did not help either.

Can someone please tell me what is wrong in this case?

Thanks a lot and kind regards,
Michael

2

Hi Michael,

I’m not expert but I would recommend checking the version of your worker.

I had a similar issue when updating to v4.0.0. It turned out that the worker was still running v3.3.2 and instead of giving an error that the version of the master and worker didn’t match, any jobs (except for import) just hanged in the launching phase. Updating to v4.0.1 fixed the problem with an update of the master and a successful worker update.

Good luck,
Andrew

1 Like
3

I had a similar issue. The worker was running an older version. To fix the issue, I deleted the worker directory, re-downloaded it, installed/configured it, and connected it to the master node using the standard cryosparc installation instructions. Good luck!

1 Like
4

Thanks a lot for your input. The worker was not updated properly and was still running the old version.

To fix this issue I redownloaded the worker archive file into the cryosparc_worker directory using the command curl -L https://get.cryosparc.com/download/worker-latest/$LICENSE_ID -o cryosparc_worker.tar.gz and forced it to update again using this command: bin/cryosparcw update --override.
In addition, I also had to make sure that the master and worker have the same LICENSE_ID in their respective config.sh file.

2 Likes
5

Hi Oliver, Thank you so much for your suggestions. Both master and worker have the same version. I deleted and unzip the cryosparc_worker.tar.gz and need one more step. will try. L.