Job cannot run on worker in another workstation

Install, Configure and Update
1

Hi,
My cryosparc (single-workstation) was installed in Platanus workstation. Since we recently have built another workstation, Salix, I was hoping to run my jobs also in Salix to reduce the crowds in Platanus.
I was then installed a worker version on Salix and connected to Platanus. With this, when I launched a job in Platanus, I could see the GPUs from Salix. And I tried to run a job using a GPU from Salix. It didn’t run.
There were no errors.

License is valid.
Launching job on lane default target salix.. …
Running job on remote worker node hostname salix..
(It stopped here. The status was “Launched”, but never turn “Running”)

Best,
Ming

2

One possible explanation is that Salix cannot access the project directory at the expected path.
What are the project and job uids of the launched, but not running, job?

3

It’s a local refinement job, P11 J98.

4

@Ming99
Please can you post the outputs of these commands, run on Platanus:

cryosparcm cli "get_scheduler_targets()"
cs_project_dir=$(cryosparcm cli "get_project_dir_abs('P11')")
ssh Salix "hostname && stat -f $cs_project_dir"
5 
[{'cache_path': '/ssd/cryosparc2_cache', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 1, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 2, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 3, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 4, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 5, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 6, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 7, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}], 'hostname': 'platanus', 'lane': 'default', 'monitor_port': None, 'name': 'platanus', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], 'GPU': [0, 1, 2, 3, 4, 5, 6, 7], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, 'ssh_str': '*@platanus', 'title': 'Worker node platanus', 'type': 'node', 'worker_bin_path': '/data/*/*/software/cryosparc/cryosparc2_worker/bin/cryosparcw'}, {'cache_path': '/ssd/cryosparc2_cache', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 1, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 2, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 3, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 4, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 5, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 6, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 7, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}], 'hostname': 'platanus.*.*.*', 'lane': 'default', 'monitor_port': None, 'name': 'platanus.*.*.*', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], 'GPU': [0, 1, 2, 3, 4, 5, 6, 7], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, 'ssh_str': '*@platanus.*.*.*', 'title': 'Worker node platanus.*.*.*', 'type': 'node', 'worker_bin_path': '/data/*/*/software/cryosparc/cryosparc2_worker/bin/cryosparcw'}, {'cache_path': '/ssd/*', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 1, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 2, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 3, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 4, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 5, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 6, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 7, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}], 'hostname': 'salix.*.*.*', 'lane': 'default', 'monitor_port': None, 'name': 'salix.*.*.*', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95], 'GPU': [0, 1, 2, 3, 4, 5, 6, 7], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]}, 'ssh_str': '*@salix.*.*.*', 'title': 'Worker node salix.*.*.*', 'type': 'node', 'worker_bin_path': '/data/*/*/cryosparc_worker/bin/cryosparcw'}]

ssh: Could not resolve hostname salix: Name or service not known
6

Please can you re-run the command on platanus

ssh Salix "hostname && stat -f $cs_project_dir"

replacing Salix with whatever the "hostname": value in the get_scheduler_targets() output.

[Edited for correction and clarity]

7

ssh: Could not resolve hostname [{cache_path:: Name or service not known

8

I meant replace Salix in the second command of this sequence

with the actual hostname that is concealed in your earlier post:

The purpose of the command is to confirm that the project directory is correctly shared between the platanus and salix hosts.

9 
salix
stat: cannot read file system information for '/home/*/CS-*': No such file or directory
10

… indicates that your project directory is not correctly shared with host salix. Correct sharing of project directories is a CryoSPARC prerequisite.
I also noticed in the get_scheduler_targets() output duplicate entries for platanus, one under "hostname": "platanus", another under "platanus.*.*.*". If this apparent duplication causes problems, you can remove an unwanted target with the remove_scheduler_target_node() cli function (details).

11

Hi, I meet similar problem!
When connect a new worker to cryosparc master, the job just “launched” without moving.

License is valid.
Launching job on lane JYLABEM2 target 192.168.202.14 …
Running job on remote worker node hostname 192.168.202.14

I have tried the commands below:

cryosparcm cli "get_scheduler_targets()"
cs_project_dir=$(cryosparcm cli "get_project_dir_abs('P6')")
ssh 192.168.202.14 "hostname && stat -f $cs_project_dir"

The output

$ cryosparcm cli "get_scheduler_targets()"
[{'cache_path': '/data2/scratch/cryosparc_cache', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 1, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 2, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 3, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 4, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 5, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 6, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 7, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}], 'hostname': 'jylab', 'lane': 'default', 'monitor_port': None, 'name': 'jylab', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127], 'GPU': [0, 1, 2, 3, 4, 5, 6, 7], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128]}, 'ssh_str': 'cryosparc@jylab', 'title': 'Worker node jylab', 'type': 'node', 'worker_bin_path': '/data3/cryosp_projs/cryosparc/cryosparc_worker/bin/cryosparcw'}, {'cache_path': '/data2', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 1, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 2, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 3, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 4, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 5, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 6, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 7, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}], 'hostname': '192.168.202.14', 'lane': 'JYLABEM2', 'monitor_port': None, 'name': '192.168.202.14', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127], 'GPU': [0, 1, 2, 3, 4, 5, 6, 7], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128]}, 'ssh_str': 'cryosparc@192.168.202.14', 'title': 'Worker node 192.168.202.14', 'type': 'node', 'worker_bin_path': '/data3/cryosp_projs/cryosparc/cryosparc_worker/bin/cryosparcw'}]
$ ssh 192.168.202.14 "hostname && stat -f $cs_project_dir"
user-R5350-G6
  File: "/data3/cryosp_projs/cryosparc/lixinzhu/CS-lxz240815-f2-vf"
    ID: 0        Namelen: 255     Type: nfs
Block size: 1048576    Fundamental block size: 1048576
Blocks: Total: 106387690  Free: 27247135   Available: 21906421
Inodes: Total: 1709023232 Free: 1706358042
12

Maybe the project directory is still not correctly shared? Could you give some suggestions how to fix it?

13

Please can you also post the outputs of these commands:

ssh cryosparc@192.168.202.14 /data3/cryosp_projs/cryosparc/cryosparc_worker/bin/cryosparcw gpulist
ssh cryosparc@192.168.202.14 ls -l /data3/cryosp_projs/cryosparc/lixinzhu/CS-lxz240815-f2-vf/
ssh cryosparc@192.168.202.14 grep "$(df /data3/cryosp_projs/cryosparc/lixinzhu/CS-lxz240815-f2-vf/ | tail -n 1 | awk '{print $NF}')\ " /proc/mounts
14

Thanks for your response!

$ ssh cryosparc@192.168.202.14 /data3/cryosp_projs/cryosparc/cryosparc_worker/bin/cryosparcw gpulist
  Detected 8 CUDA devices.
                                                                                                              id           pci-bus  name
   ---------------------------------------------------------------
       0                33  NVIDIA GeForce RTX 4090

       1                34  NVIDIA GeForce RTX 4090

       2                39  NVIDIA GeForce RTX 4090

       3                42  NVIDIA GeForce RTX 4090

       4                47  NVIDIA GeForce RTX 4090

       5                53  NVIDIA GeForce RTX 4090

       6                56  NVIDIA GeForce RTX 4090

       7                57  NVIDIA GeForce RTX 4090

   ---------------------------------------------------------------
$ ssh cryosparc@192.168.202.14 ls -l /data3/cryosp_projs/cryosparc/lixinzhu/CS-lxz240815-f2-vf/
total 1212
-rwxrwxrwx 1 cryosparc cryosparc    76  8月 17 13:12 cs.lock
drwxrwsrwx 4 cryosparc cryosparc  4096  8月 17 13:32 J1
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:20 J10
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 09:33 J100
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 09:37 J101
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 09:47 J102
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 10:23 J103
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 10:23 J104
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 16:53 J105
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 21:17 J106
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 21:25 J107
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 21:15 J108
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 21:16 J109
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:26 J11
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 22:36 J110
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 08:22 J111
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 08:23 J112
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 09:20 J113
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 09:04 J114
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 09:54 J115
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 10:02 J116
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 11:13 J117
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 11:16 J118
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 10:40 J119
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:30 J12
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 11:00 J120
drwxrwsrwx 4 cryosparc cryosparc  4096  8月 28 12:04 J121
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 12:23 J122
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 12:25 J123
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 12:26 J124
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 12:26 J125
drwxrwsrwx 3 cryosparc cryosparc 12288  8月 28 13:48 J126
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 14:25 J127
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 14:52 J128
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 15:09 J129
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:28 J13
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 18:57 J130
drwxrwsrwx 5 cryosparc cryosparc  4096  8月 28 16:45 J131
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 17:08 J132
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 28 19:14 J133
drwxrwsrwx 4 cryosparc cryosparc  4096  8月 29 01:03 J134
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 29 02:04 J135
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 29 09:00 J136
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 29 09:01 J137
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 29 17:03 J138
drwxrwsrwx 4 cryosparc cryosparc  4096  9月  1 23:17 J139
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:36 J14
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 29 16:58 J140
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 29 16:51 J141
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 29 15:13 J142
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 29 17:57 J143
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 29 20:20 J144
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 29 20:19 J145
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 10:04 J146
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 10:01 J147
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 10:02 J148
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 09:53 J149
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:35 J15
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 10:07 J150
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 10:13 J151
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 17:34 J152
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 15:28 J153
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 15:05 J154
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 15:48 J155
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 15:49 J156
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 19:50 J157
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 30 20:57 J158
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 31 10:30 J159
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:44 J16
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 31 11:48 J160
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 31 15:49 J161
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 31 16:58 J162
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 31 17:21 J163
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 31 18:24 J164
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 31 18:31 J165
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  1 12:54 J166
drwxrwsrwx 4 cryosparc cryosparc  4096  9月  1 23:17 J167
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  2 00:44 J168
drwxrwsrwx 4 cryosparc cryosparc  4096  9月  2 00:43 J169
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:43 J17
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  2 12:54 J170
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  2 13:48 J171
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  2 16:10 J172
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  2 16:12 J173
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  2 17:03 J174
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  3 09:07 J175
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  3 09:05 J176
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  3 12:00 J177
drwxrwsrwx 4 cryosparc cryosparc  4096  9月  3 10:23 J178
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  4 02:24 J179
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:54 J18
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  3 13:53 J180
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  3 13:57 J181
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  3 14:00 J182
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  3 14:03 J183
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  3 16:52 J184
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  3 16:52 J185
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  3 16:55 J186
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  3 16:57 J187
drwxrwsrwx 3 cryosparc cryosparc 12288  9月  3 21:12 J188
drwxrwsrwx 3 cryosparc cryosparc 12288  9月  4 15:26 J189
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 18 13:48 J19
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  4 22:38 J190
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  4 09:58 J191
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  4 09:59 J192
drwxrwsrwx 3 cryosparc cryosparc 12288  9月  4 10:01 J193
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  4 11:12 J194
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  4 11:12 J195
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  4 11:20 J196
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  5 10:06 J197
drwxrwsrwx 3 cryosparc cryosparc 12288  9月  4 12:02 J198
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  5 13:53 J199
drwxrwsrwx 5 cryosparc cryosparc  4096  8月 17 16:12 J2
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 18 15:03 J20
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  5 09:50 J200
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  5 09:53 J201
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  5 13:05 J202
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  5 10:05 J203
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  5 16:27 J204
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  5 16:57 J205
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  5 20:06 J206
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  5 23:42 J207
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  5 23:06 J208
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  6 00:20 J209
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 18 15:11 J21
drwxrwsrwx 3 cryosparc cryosparc 12288  9月  6 11:15 J210
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  6 10:30 J211
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  6 11:37 J212
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  6 14:19 J213
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  6 12:58 J214
drwxrwsrwx 3 cryosparc cryosparc 12288  9月  6 14:50 J215
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  6 16:59 J216
drwxrwsrwx 3 cryosparc cryosparc 12288  9月  6 18:35 J217
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  6 19:51 J218
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  6 23:37 J219
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 18 15:13 J22
drwxrwsrwx 4 cryosparc cryosparc  4096  9月  7 00:45 J220
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  7 01:40 J221
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  7 01:56 J222
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  7 15:54 J223
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  7 16:01 J224
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  7 16:03 J225
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  7 16:03 J226
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  7 16:11 J227
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  7 16:12 J228
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  7 16:19 J229
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 18 15:15 J23
drwxrwsrwx 3 cryosparc cryosparc 12288  9月  7 16:54 J230
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 00:15 J231
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 09:43 J232
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 09:45 J233
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 10:03 J234
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 10:53 J235
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 11:11 J236
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 10:53 J237
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 12:29 J238
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 18:52 J239
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 18 16:02 J24
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 22:34 J240
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 23:09 J241
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 23:16 J242
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 23:41 J243
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  8 23:42 J244
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  9 00:30 J245
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  9 09:03 J246
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  9 18:17 J247
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  9 18:20 J248
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  9 18:21 J249
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 18 16:03 J25
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  9 18:22 J250
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  9 18:22 J251
drwxrwsrwx 3 cryosparc cryosparc 12288  9月  9 18:56 J252
drwxrwsrwx 3 cryosparc cryosparc  4096  9月  9 20:45 J253
drwxrwsrwx 3 cryosparc cryosparc  4096  9月 11 18:21 J254
drwxrwsrwx 3 cryosparc cryosparc  4096  9月 16 12:42 J255
drwxrwsrwx 3 cryosparc cryosparc  4096  9月 10 11:20 J256
drwxrwsrwx 3 cryosparc cryosparc  4096  9月 10 11:20 J257
drwxrwsrwx 3 cryosparc cryosparc  4096  9月 13 20:46 J258
drwxrwsrwx 3 cryosparc cryosparc  4096  9月 16 12:23 J259
drwxrwsrwx 4 cryosparc cryosparc  4096  8月 18 17:39 J26
drwxrwsrwx 3 cryosparc cryosparc  4096  9月 16 12:24 J260
drwxrwsrwx 3 cryosparc cryosparc  4096  9月 16 12:35 J261
drwxrwsrwx 3 cryosparc cryosparc  4096  9月 16 12:36 J262
drwxrwsrwx 3 cryosparc cryosparc  4096  9月 16 12:48 J263
drwxrwsrwx 3 cryosparc cryosparc  4096 10月 28 00:09 J264
drwxrwsrwx 3 cryosparc cryosparc  4096 10月 28 09:13 J265
drwxrwsrwx 3 cryosparc cryosparc  4096 10月 28 09:44 J266
drwxrwsrwx 3 cryosparc cryosparc  4096 11月  4 16:05 J267
drwxrwsrwx 3 cryosparc cryosparc  4096 11月  4 20:10 J268
drwxrwsrwx 3 cryosparc cryosparc  4096 11月  5 19:35 J269
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 18 19:13 J27
drwxrwsrwx 3 cryosparc cryosparc  4096 11月  5 19:35 J270
drwxrwsrwx 3 cryosparc cryosparc  4096 11月  5 14:15 J271
drwxrwsr-x 3 cryosparc cryosparc  4096 11月  5 19:35 J272
drwxrwsr-x 3 cryosparc cryosparc  4096 11月  5 19:06 J273
drwxrwsr-x 3 cryosparc cryosparc  4096 11月  5 19:02 J274
drwxrwsr-x 3 cryosparc cryosparc  4096 11月  5 19:02 J275
drwxrwsr-x 3 cryosparc cryosparc  4096 11月  5 19:04 J276
drwxrwsr-x 3 cryosparc cryosparc  4096 11月  5 19:04 J277
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 08:38 J28
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 08:46 J29
drwxrwsrwx 4 cryosparc cryosparc  4096  8月 17 16:34 J3
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 15:25 J30
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 15:25 J31
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 15:30 J32
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 15:43 J33
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 15:47 J34
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 15:45 J35
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 15:48 J36
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 15:47 J37
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 15:51 J38
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 16:58 J39
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 16:48 J4
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 17:02 J40
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 17:01 J41
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 17:04 J42
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 17:04 J43
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 17:04 J44
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 17:06 J45
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 18:02 J46
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 19:06 J47
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 19 19:21 J48
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 20 10:18 J49
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 21 09:44 J5
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 20 10:49 J50
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 20 15:35 J51
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 20 16:23 J52
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 20 16:31 J53
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 20 16:30 J54
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 20 16:32 J55
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 20 16:34 J56
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 21 09:21 J57
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 21 10:46 J58
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 21 11:03 J59
drwxrwsrwx 4 cryosparc cryosparc  4096  8月 17 17:01 J6
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 21 16:35 J60
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 21 17:02 J61
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 16:35 J62
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 16:35 J63
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 23 11:05 J64
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 23 14:36 J65
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 23 15:36 J66
drwxrwsrwx 4 cryosparc cryosparc  4096  8月 25 12:37 J67
drwxrwsrwx 5 cryosparc cryosparc  4096  8月 25 23:35 J68
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 25 23:50 J69
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:12 J7
drwxrwsrwx 4 cryosparc cryosparc  4096  8月 26 01:10 J70
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 08:37 J71
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 29 19:06 J72
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 14:08 J73
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 16:12 J74
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 11:32 J75
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 14:34 J76
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 12:29 J77
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 12:33 J78
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 15:21 J79
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:20 J8
drwxrwsrwx 4 cryosparc cryosparc  4096  8月 26 18:00 J80
drwxrwsrwx 4 cryosparc cryosparc  4096  8月 26 18:00 J81
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 16:07 J82
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 16:32 J83
drwxrwsrwx 4 cryosparc cryosparc  4096  8月 26 19:03 J84
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 23:56 J85
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 20:57 J86
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 26 22:42 J87
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 08:28 J88
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 08:48 J89
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 17 17:15 J9
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 08:41 J90
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 08:46 J91
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 09:43 J92
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 09:06 J93
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 08:54 J94
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 08:54 J95
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 09:15 J96
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 09:09 J97
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 09:18 J98
drwxrwsrwx 3 cryosparc cryosparc  4096  8月 27 09:35 J99
-rwxrwxrwx 1 cryosparc cryosparc  4346 11月  5 19:35 job_manifest.json
-rwxrwxrwx 1 cryosparc cryosparc  3111 11月  4 10:00 project.json
-rwxrwxrwx 1 cryosparc cryosparc  5180 11月  5 19:02 workspaces.json
$ ssh cryosparc@192.168.202.14 grep "$(df /data3/cryosp_projs/cryosparc/lixinzhu/CS-lxz240815-f2-vf/ | tail -n 1 | awk '{print $NF}')\ " /proc/mounts
/dev/sda1 /data3 xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=128,swidth=896,noquota 0 0
192.168.202.13:/data3 /data3 nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.14,local_lock=none,addr=192.168.202.13 0 0
15

I also tried the “Test job launch” in web browser
image

and command

$ cryosparcm test workers P6
Using project P6
Running worker tests...
2024-11-05 19:04:31,681 log                  CRITICAL | Worker test results
2024-11-05 19:04:31,681 log                  CRITICAL | jylab
2024-11-05 19:04:31,681 log                  CRITICAL |   ✓ LAUNCH
2024-11-05 19:04:31,681 log                  CRITICAL |   ✓ SSD
2024-11-05 19:04:31,681 log                  CRITICAL |   ✓ GPU
2024-11-05 19:04:31,684 log                  CRITICAL |  ⚠ NVIDIA GeForce RTX 4090 @ 00000000:21:00.0: Persistence Mode is Disabled. Enable Persistence mode by running `nvidia-smi -pm 1` as root to persist the NVIDIA driver, reducing GPU load times.
...
2024-11-05 19:04:31,685 log                  CRITICAL | 192.168.202.14
2024-11-05 19:04:31,685 log                  CRITICAL |   ✕ LAUNCH
2024-11-05 19:04:31,685 log                  CRITICAL |     Error:
2024-11-05 19:04:31,685 log                  CRITICAL |     See P6 J275 for more information
2024-11-05 19:04:31,685 log                  CRITICAL |   ⚠ SSD
2024-11-05 19:04:31,685 log                  CRITICAL |     Did not run: Launch test failed
2024-11-05 19:04:31,685 log                  CRITICAL |   ⚠ GPU
2024-11-05 19:04:31,685 log                  CRITICAL |     Did not run: Launch test failed
16

I did not expect two matches for the /data3 mountpoint. Are there really a local and a network filesystem mounted under /data3 on 192.168.202.14, or did I improperly apply quotes in the ssh command?
Please can you post the output of the df -h command on 192.168.202.14?

17

You are right, a local and a network filesystem mounted under /data3 on 192.168.202.14.
I remount the file and connect again, but the job is still “launched”. cryosparc_worker was reinstalled in /data

cryosparc@user-R5350-G6:~$ df -h
Filesystem             Size  Used Avail Use% Mounted on
tmpfs                  101G   37M  101G   1% /run
/dev/sdb2              1.8T   32G  1.6T   2% /
tmpfs                  504G     0  504G   0% /dev/shm
tmpfs                  5.0M  4.0K  5.0M   1% /run/lock
/dev/sdb1              511M  6.1M  505M   2% /boot/efi
tmpfs                  101G   84K  101G   1% /run/user/128
tmpfs                  101G  116K  101G   1% /run/user/1000
tmpfs                  101G   80K  101G   1% /run/user/1001
/dev/nvme1n1           3.5T   28K  3.3T   1% /data1
/dev/nvme0n1           7.0T   28K  6.6T   1% /data2
/dev/sda1              102T  744G  102T   1% /data
192.168.202.13:/data3  102T   76T   21T  79% /data3

I also think about that previous cryosparc_master was installed as “single workstation” mode not “master node only”, does this matter?

18

Did you update the worker connection to reflect the new worker_bin_path?

19

I had updated the worker connection, but it still not work.
I found some clue using “cryosparcm joblog P6 J297” when the stuck job was queueing.

================= CRYOSPARCW =======  2024-11-11 19:10:12.763815  =========
Project P6 Job J297
Master jylab Port 39002
===========================================================================
MAIN PROCESS PID 79097
========= now starting main process at 2024-11-11 19:10:12.764350
/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py:135: UserWarning: *** CommandClient: (http://jylab:39002/api) URL Error [Errno -3] Temporary failure in name resolution, attempt 1 of 3. Retrying in 30 seconds
  system = self._get_callable("system.describe")()
/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py:135: UserWarning: *** CommandClient: (http://jylab:39002/api) URL Error [Errno -3] Temporary failure in name resolution, attempt 1 of 3. Retrying in 30 seconds
  /data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py:135: UserWarning: *** CommandClient: (http://jylab:39002/api) URL Error [Errno -3] Temporary failure in name resolution, attempt 2 of 3. Retrying in 30 seconds
  system = self._get_callable("system.describe")()
/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py:135: UserWarning: *** CommandClient: (http://jylab:39002/api) URL Error [Errno -3] Temporary failure in name resolution, attempt 2 of 3. Retrying in 30 seconds
  system = self._get_callable("system.describe")()


 cli = client.CommandClient(master_hostname, int(master_command_core_port), service="command_core")
  File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_compute/client.py", line 38, in __init__
    cli = client.CommandClient(master_hostname, int(master_command_core_port), service="command_core")
  File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_compute/client.py", line 38, in __init__
    super().__init__(service, host, port, url, timeout, headers, cls=NumpyEncoder)
  File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 97, in __init__
    super().__init__(service, host, port, url, timeout, headers, cls=NumpyEncoder)
  File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 97, in __init__
    self._reload()  # attempt connection immediately to gather methods
  File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 135, in _reload
    self._reload()  # attempt connection immediately to gather methods
  File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 135, in _reload
    system = self._get_callable("system.describe")()
  File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 108, in func
    system = self._get_callable("system.describe")()
  File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 108, in func
    raise CommandError(
cryosparc_tools.cryosparc.errors.CommandError: *** (http://jylab:39002, code 500) Encounted error from JSONRPC function "system.describe" with params ()
    raise CommandError(
cryosparc_tools.cryosparc.errors.CommandError: *** (http://jylab:39002, code 500) Encounted error from JSONRPC function "system.describe" with params ()
20

What are the outputs of these commands on jylab:

uname -a
curl jylab:39002
curl 127.0.0.1:39002
ssh cryosparc@192.168.202.14 "curl jylab:39002 && uname -a"