V4.0.1_can not launch refinement jobs

[Lan]

After I updated to v 4.0.1, I am not able to start a new job or rerun a job (local refinement).
The license is valid.
Launching job on lane default target xxxxx…
Running job on master node hostname xxx

It stayed “Launched” forever.

Your suggestions will be appreciated.
Thanks,
Lan

[olibclarke]

Hi Lan - likely that the master was updated, but the update failed for the worker, so there is a mismatch. See below link for discussion and fix.

[Fixed] Jobs stuck in launched-mode after update to v4.0.1

Hi, I updated cryosparc to version v4.0.1 and since then I cannot start a job since it stays in “launched”-mode forever. The only job I could get to run was an “import micrographs”-job. I tried running the command cryosparcm test worker which told me that there is indeed a problem while launching the jobs. The log of the test job looks as follows: MAINPROCESS PID 33040 Process Process-1: Traceback (most recent call last): File “/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryospar…

[Lan]

HI Oliver, I deleted the previsions “cryosparc_worker” folder and moved out the Cryosparc_worker.tat.gz from v4.0.1 cryosparc_master folder and extracted it. Both are v4.01. I still need help for the next steps.

Thank, L.

[wtempel]

I suspect you will have to

1. cd cryosparc_worker
2. ./install.sh <relevant-options> (guide)
3. Inspect the output of
   cryosparcm cli "get_scheduler_targets()"
   If the intended worker is listed, take note of configuration details for the following command and include --update in that following comand
4. ./bin/cryosparcw connect <relevant-options> [--update] (guide)
   (Edit: This step may not be needed in this specific case.)

[Lan]

Hi wtempel: Thanks for the instruction. I completed the your step 1-3, and have problems with step 4 ./bin/cryosparcw connect –

./bin/cryosparcw connect --worker <worker.cryosparc.xxxxxx.com>–master –port <port_39000>–update–ssquota 500000
bash: worker.cryosparc.xxxxx.com: No such file or directory

what is the worker_hostname? The " Install The cryosparc_worker Package" said:
–worker <worker_hostname> = worker.cryosparc.hostname.com
Looking forward to your further instructions.
Thanks, Lan

[wtempel]

@Lan

what is the worker_hostname?

Please can you post the output of
cryosparcm cli "get_scheduler_targets()"

[Lan]

[cryosparc_user@r16763 cryosparc_worker]$ cryosparcm cli “get_scheduler_targets()”
[{‘cache_path’: ‘/scr/cryosparc_cache’, ‘cache_quota_mb’: None, ‘cache_reserve_mb’: 10000, ‘desc’: None, ‘gpus’: [{‘id’: 0, ‘mem’: 8369799168, ‘name’: ‘NVIDIA GeForce RTX 3070’}, {‘id’: 1, ‘mem’: 8366784512, ‘name’: ‘NVIDIA GeForce RTX 3070’}, {‘id’: 2, ‘mem’: 8369799168, ‘name’: ‘NVIDIA GeForce RTX 3070’}, {‘id’: 3, ‘mem’: 8369799168, ‘name’: ‘NVIDIA GeForce RTX 3070’}], ‘hostname’: ‘r16763’, ‘lane’: ‘default’, ‘monitor_port’: None, ‘name’: ‘r16763’, ‘resource_fixed’: {‘SSD’: True}, ‘resource_slots’: {‘CPU’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], ‘GPU’: [0, 1, 2, 3], ‘RAM’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]}, ‘ssh_str’: ‘cryosparc_user@r16763’, ‘title’: ‘Worker node r16763’, ‘type’: ‘node’, ‘worker_bin_path’: ‘/home/cryosparc_user/software/cryosparc/cryosparc_worker/bin/cryosparcw’}]

[wtempel]

@Lan

I deleted the previsions “cryosparc_worker” folder and moved out the Cryosparc_worker.tat.gz from v4.0.1 cryosparc_master folder and extracted it.

If you unpacked cryosparc_worker.tar.gz inside /home/cryosparc_user/software/cryosparc
CryoSPARC may already work, given

'worker_bin_path': '/home/cryosparc_user/software/cryosparc/cryosparc_worker/bin/cryosparcw'

shown by get_scheduler_targets().
Can you test the local refinement job?

[Lan]

No. I copied a refinement job or start a new job. none of them work. Stay in Launched.

At the Queue Job panel, “queue to lane”. also, i need to select specific GPU before "Queue button can be functional. Is this new?
Lan

[wtempel]

Please can you post screenshots of these observations

@Lan

At the Queue Job panel, “queue to lane”. also, i need to select specific GPU before "Queue button can be functional.

as well as the outputs of these commands

• cryosparcm status | grep -v LICENSE
• cat /home/cryosparc_user/software/cryosparc/cryosparc_worker/version
• cat /home/cryosparc_user/software/cryosparc/cryosparc_worker/patch

[Lan]

[cryosparc_user@r16763 cryosparc_worker]$ cryosparcm status | grep -v LICENSE

CryoSPARC System master node installed at
/home/cryosparc_user/software/cryosparc/cryosparc_master
Current cryoSPARC version: v4.0.1+221017

CryoSPARC process status:

app RUNNING pid 37568, uptime 0:42:52
app_api RUNNING pid 37589, uptime 0:42:50
app_api_dev STOPPED Not started
app_legacy STOPPED Not started
app_legacy_dev STOPPED Not started
command_core RUNNING pid 37482, uptime 0:43:05
command_rtp RUNNING pid 37525, uptime 0:42:57
command_vis RUNNING pid 37511, uptime 0:42:58
database RUNNING pid 37371, uptime 0:43:07

---

License is valid

global config variables:
export CRYOSPARC_DB_PATH="/home/cryosparc_user/software/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true
export CRYOSPARC_MASTER_HOSTNAME=“r16763”
export CRYOSPARC_FORCE_HOSTNAME=“r16763”

cat /home/cryosparc_user/software/cryosparc/cryosparc_worker/version
the path is different but i can find this version file. v4.0.1
(home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparc_worker/version
there is another one in
home/cryosparc_user/software/cryosparc/cryosparc_worker/cryosparcs_worker/version)

cat /home/cryosparc_user/software/cryosparc/cryosparc_worker/patch
the path is different. I can find out Patch file (221017)

/home/cryosparc_user/software/cryosparc/cryosparc_worker_patch/cryosparc_worker/Patch (there are two foldes, bin and cryosparc_compute under same directory)

I have path problem.

L.

[wtempel]

@Lan

I have path problem.

I think you are right. Based on your worker registration in the CryoSPARC database (V4.0.1_can not launch refinement jobs - #7 by Lan), CryoSPARC will not work unless the worker software is installed in a directory cryosparc_worker directly under
/home/cryosparc_user/software/cryosparc
Moreover, the version and patch level of that worker software must match version and patch level of the master software.
In broad brushstrokes, you could

1. move your existing /home/cryosparc_user/software/cryosparc/cryosparc_worker directory aside (rename it).
2. download the specific version of the cryosparc_worker package that matches the version of your master. Not specifying the version would download the latest version, which might not match your master’s version. If your download command includes $LICENSE_ID, ensure the LICENSE_ID environment variable has been correctly defined.
3. unpack the downloaded archive inside /home/cryosparc_user/software/cryosparc
4. inside the new cryosparc_worker directory, run ./install.sh with the correct --license and --cudapath specs (guide)
5. patch the worker (follow instructions for Cluster workers) to match master’s patch level

Please let us know if you have any questions regarding the general procedure, or regarding specific commands described in the linked documentation.

[Lan]

thanks for instruction.
instead of the down new, does the file cryosparc_worker.tar.gz I moved out from the Cryosparc_master directory works?

[Lan]

Hi WTEMPEL,
Thank you very much for the detailed instructions. The new version works now!

Lan

[wtempel]

@Lan

The new version works now!

Were you able to run any GPU jobs?

[Lan]

homogenous refinement and motion correction work.