CTF Patch estimation job gets stuck at launching status

Install, Configure and Update
1

Hi,

I imported micrographs in cryosparc v4.2.1 after upgrading from v3, then run CTF Patch estimation job.
But the job gets stuck at lauching status (see picture below).

Thanks!
Jianhao
20230609

2

Please can you provide additional information:

  • did you import motion-corrected, single-frame mrc files?
  • what re the outputs of these commands:
    cryosparcm status | grep MASTER_HOSTNAME
cryosparcm cli "get_scheduler_targets()"
  • a file listing (ls -al) of the job directory
3

  • did you import motion-corrected, single-frame mrc files?
    Yes, all micrographs are imported successfully.

  • what re the outputs of these commands:
    cryosparcm status | grep MASTER_HOSTNAME
    cryosparcm cli “get_scheduler_targets()”

$ cryosparcm status | grep MASTER_HOSTNAME
export CRYOSPARC_MASTER_HOSTNAME=“zhanglab”

$ cryosparcm cli “get_scheduler_targets()”
[{‘cache_path’: ‘/ssd0/cryosparc2_cache/’, ‘cache_quota_mb’: None, ‘cache_reserve_mb’: 10000, ‘desc’: None, ‘gpus’: [{‘id’: 0, ‘mem’: 11554717696, ‘name’: ‘GeForce RTX 2080 Ti’}, {‘id’: 1, ‘mem’: 11554717696, ‘name’: ‘GeForce RTX 2080 Ti’}, {‘id’: 2, ‘mem’: 11554717696, ‘name’: ‘GeForce RTX 2080 Ti’}, {‘id’: 3, ‘mem’: 11554717696, ‘name’: ‘GeForce RTX 2080 Ti’}], ‘hostname’: ‘zhanglab’, ‘lane’: ‘default’, ‘monitor_port’: None, ‘name’: ‘zhanglab’, ‘resource_fixed’: {‘SSD’: True}, ‘resource_slots’: {‘CPU’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], ‘GPU’: [0, 1, 2, 3], ‘RAM’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, ‘ssh_str’: ‘spuser@zhanglab’, ‘title’: ‘Worker node zhanglab’, ‘type’: ‘node’, ‘worker_bin_path’: ‘/opt/cryosparc2/cryosparc2_worker/bin/cryosparcw’}]

  • a file listing (ls -al) of the job directory
    $ ls -al J2/
    total 67
    drwxrwxr-x. 3 spuser spuser 6 Jun 9 09:02 .
    drwxrwxr-x. 4 spuser spuser 8 Jun 9 09:01 …
    -rw-rw-r–. 1 spuser spuser 18 Jun 9 09:01 events.bson
    drwxrwxr-x. 2 spuser spuser 2 Jun 9 09:01 gridfs_data
    -rw-rw-r–. 1 spuser spuser 26282 Jun 9 09:01 job.json
    -rw-rw-r–. 1 spuser spuser 2900 Jun 9 09:02 job.log
4

Thanks. Please can you post the contents of the job log: either

  • run
    cat J2/job.log
  • or, in UI, Metadata|Log
5 
================= CRYOSPARCW =======  2023-06-09 09:02:01.216626  =========
Project P47 Job J2
Master zhanglab Port 39002
===========================================================================
========= monitor process now starting main process
MAINPROCESS PID 141747
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "cryosparc_worker/cryosparc_compute/run.py", line 162, in cryosparc_compute.run.run
  File "/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py", line 97, in connect
    assert cli.test_connection(), "Job could not connect to master instance at %s:%s" % (master_hostname, str(master_command_core_port))
  File "/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/client.py", line 65, in func
    assert 'error' not in res, f"Encountered error for method \"{key}\" with params {params}:\n{res['error']['message'] if 'message' in res['error'] else res['error']}"
AssertionError: Encountered error for method "test_connection" with params ():
ServerError: Authentication failed - License-ID request header missing.
   This may indicate that cryosparc_worker did not update,
   cryosparc_worker/config.sh is missing a CRYOSPARC_LICENSE_ID entry,
   or CRYOSPARC_LICENSE_ID is not present in your environment.
   See https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#command-api-security for more details.

Process Process-1:
Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "cryosparc_worker/cryosparc_compute/run.py", line 31, in cryosparc_compute.run.main
  File "/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py", line 97, in connect
    assert cli.test_connection(), "Job could not connect to master instance at %s:%s" % (master_hostname, str(master_command_core_port))
  File "/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/client.py", line 65, in func
    assert 'error' not in res, f"Encountered error for method \"{key}\" with params {params}:\n{res['error']['message'] if 'message' in res['error'] else res['error']}"
AssertionError: Encountered error for method "test_connection" with params ():
ServerError: Authentication failed - License-ID request header missing.
   This may indicate that cryosparc_worker did not update,
   cryosparc_worker/config.sh is missing a CRYOSPARC_LICENSE_ID entry,
   or CRYOSPARC_LICENSE_ID is not present in your environment.
   See https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#command-api-security for more details.
6

Please can you confirm:

  1. the cryosparc_worker version:
    cat /opt/cryosparc2/cryosparc2_worker/version
  2. that /opt/cryosparc2/cryosparc2_worker/config.sh includes a definition
    export CRYOSPARC_LICENSE_ID= whose value matches the definition inside
    /path/to/cryosparc_master/config.sh
7

  1. It’s strange, the version is v3.3.2.
    $ cat /opt/cryosparc2/cryosparc2_worker/version
    v3.3.2

  2. Yes, both CRYOSPARC_LICENSE_ID are the same

8

You may want to

  1. check the file /path/to/cryosparc_master/run/update.log for potential hints about why the cryosparc_worker installation was not updated automatically.
  2. manually update cryosparc2_worker following the “cluster” instructions.
9

Thank you! manually update figure it out!

10