@Ming99
Please can you post the outputs of these commands, run on Platanus:
cryosparcm cli "get_scheduler_targets()"
cs_project_dir=$(cryosparcm cli "get_project_dir_abs('P11')")
ssh Salix "hostname && stat -f $cs_project_dir"
[{'cache_path': '/ssd/cryosparc2_cache', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 1, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 2, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 3, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 4, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 5, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 6, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 7, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}], 'hostname': 'platanus', 'lane': 'default', 'monitor_port': None, 'name': 'platanus', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], 'GPU': [0, 1, 2, 3, 4, 5, 6, 7], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, 'ssh_str': '*@platanus', 'title': 'Worker node platanus', 'type': 'node', 'worker_bin_path': '/data/*/*/software/cryosparc/cryosparc2_worker/bin/cryosparcw'}, {'cache_path': '/ssd/cryosparc2_cache', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 1, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 2, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 3, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 4, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 5, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 6, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}, {'id': 7, 'mem': 11546394624, 'name': 'NVIDIA GeForce RTX 2080 Ti'}], 'hostname': 'platanus.*.*.*', 'lane': 'default', 'monitor_port': None, 'name': 'platanus.*.*.*', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], 'GPU': [0, 1, 2, 3, 4, 5, 6, 7], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, 'ssh_str': '*@platanus.*.*.*', 'title': 'Worker node platanus.*.*.*', 'type': 'node', 'worker_bin_path': '/data/*/*/software/cryosparc/cryosparc2_worker/bin/cryosparcw'}, {'cache_path': '/ssd/*', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 1, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 2, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 3, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 4, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 5, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 6, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}, {'id': 7, 'mem': 25438126080, 'name': 'NVIDIA GeForce RTX 3090'}], 'hostname': 'salix.*.*.*', 'lane': 'default', 'monitor_port': None, 'name': 'salix.*.*.*', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95], 'GPU': [0, 1, 2, 3, 4, 5, 6, 7], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]}, 'ssh_str': '*@salix.*.*.*', 'title': 'Worker node salix.*.*.*', 'type': 'node', 'worker_bin_path': '/data/*/*/cryosparc_worker/bin/cryosparcw'}]
ssh: Could not resolve hostname salix: Name or service not known
Please can you re-run the command on platanus
ssh Salix "hostname && stat -f $cs_project_dir"
replacing Salix with whatever the "hostname": value in the get_scheduler_targets() output.
[Edited for correction and clarity]
ssh: Could not resolve hostname [{cache_path:: Name or service not known
I meant replace Salix in the second command of this sequence
with the actual hostname that is concealed in your earlier post:
The purpose of the command is to confirm that the project directory is correctly shared between the platanus and salix hosts.
salix
stat: cannot read file system information for '/home/*/CS-*': No such file or directory
… indicates that your project directory is not correctly shared with host salix. Correct sharing of project directories is a CryoSPARC prerequisite.
I also noticed in the get_scheduler_targets() output duplicate entries for platanus, one under "hostname": "platanus", another under "platanus.*.*.*". If this apparent duplication causes problems, you can remove an unwanted target with the remove_scheduler_target_node() cli function (details).
Hi, I meet similar problem!
When connect a new worker to cryosparc master, the job just “launched” without moving.
License is valid.
Launching job on lane JYLABEM2 target 192.168.202.14 …
Running job on remote worker node hostname 192.168.202.14
I have tried the commands below:
cryosparcm cli "get_scheduler_targets()"
cs_project_dir=$(cryosparcm cli "get_project_dir_abs('P6')")
ssh 192.168.202.14 "hostname && stat -f $cs_project_dir"
The output
$ cryosparcm cli "get_scheduler_targets()"
[{'cache_path': '/data2/scratch/cryosparc_cache', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 1, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 2, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 3, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 4, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 5, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 6, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 7, 'mem': 25393692672, 'name': 'NVIDIA GeForce RTX 4090'}], 'hostname': 'jylab', 'lane': 'default', 'monitor_port': None, 'name': 'jylab', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127], 'GPU': [0, 1, 2, 3, 4, 5, 6, 7], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128]}, 'ssh_str': 'cryosparc@jylab', 'title': 'Worker node jylab', 'type': 'node', 'worker_bin_path': '/data3/cryosp_projs/cryosparc/cryosparc_worker/bin/cryosparcw'}, {'cache_path': '/data2', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 1, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 2, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 3, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 4, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 5, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 6, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}, {'id': 7, 'mem': 25386352640, 'name': 'NVIDIA GeForce RTX 4090'}], 'hostname': '192.168.202.14', 'lane': 'JYLABEM2', 'monitor_port': None, 'name': '192.168.202.14', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127], 'GPU': [0, 1, 2, 3, 4, 5, 6, 7], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128]}, 'ssh_str': 'cryosparc@192.168.202.14', 'title': 'Worker node 192.168.202.14', 'type': 'node', 'worker_bin_path': '/data3/cryosp_projs/cryosparc/cryosparc_worker/bin/cryosparcw'}]
$ ssh 192.168.202.14 "hostname && stat -f $cs_project_dir"
user-R5350-G6
File: "/data3/cryosp_projs/cryosparc/lixinzhu/CS-lxz240815-f2-vf"
ID: 0 Namelen: 255 Type: nfs
Block size: 1048576 Fundamental block size: 1048576
Blocks: Total: 106387690 Free: 27247135 Available: 21906421
Inodes: Total: 1709023232 Free: 1706358042
Maybe the project directory is still not correctly shared? Could you give some suggestions how to fix it?
Please can you also post the outputs of these commands:
ssh cryosparc@192.168.202.14 /data3/cryosp_projs/cryosparc/cryosparc_worker/bin/cryosparcw gpulist
ssh cryosparc@192.168.202.14 ls -l /data3/cryosp_projs/cryosparc/lixinzhu/CS-lxz240815-f2-vf/
ssh cryosparc@192.168.202.14 grep "$(df /data3/cryosp_projs/cryosparc/lixinzhu/CS-lxz240815-f2-vf/ | tail -n 1 | awk '{print $NF}')\ " /proc/mounts
Thanks for your response!
$ ssh cryosparc@192.168.202.14 /data3/cryosp_projs/cryosparc/cryosparc_worker/bin/cryosparcw gpulist
Detected 8 CUDA devices.
id pci-bus name
---------------------------------------------------------------
0 33 NVIDIA GeForce RTX 4090
1 34 NVIDIA GeForce RTX 4090
2 39 NVIDIA GeForce RTX 4090
3 42 NVIDIA GeForce RTX 4090
4 47 NVIDIA GeForce RTX 4090
5 53 NVIDIA GeForce RTX 4090
6 56 NVIDIA GeForce RTX 4090
7 57 NVIDIA GeForce RTX 4090
---------------------------------------------------------------
$ ssh cryosparc@192.168.202.14 ls -l /data3/cryosp_projs/cryosparc/lixinzhu/CS-lxz240815-f2-vf/
total 1212
-rwxrwxrwx 1 cryosparc cryosparc 76 8月 17 13:12 cs.lock
drwxrwsrwx 4 cryosparc cryosparc 4096 8月 17 13:32 J1
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:20 J10
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 09:33 J100
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 09:37 J101
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 09:47 J102
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 10:23 J103
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 10:23 J104
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 16:53 J105
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 21:17 J106
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 21:25 J107
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 21:15 J108
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 21:16 J109
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:26 J11
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 22:36 J110
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 08:22 J111
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 08:23 J112
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 09:20 J113
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 09:04 J114
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 09:54 J115
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 10:02 J116
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 11:13 J117
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 11:16 J118
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 10:40 J119
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:30 J12
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 11:00 J120
drwxrwsrwx 4 cryosparc cryosparc 4096 8月 28 12:04 J121
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 12:23 J122
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 12:25 J123
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 12:26 J124
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 12:26 J125
drwxrwsrwx 3 cryosparc cryosparc 12288 8月 28 13:48 J126
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 14:25 J127
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 14:52 J128
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 15:09 J129
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:28 J13
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 18:57 J130
drwxrwsrwx 5 cryosparc cryosparc 4096 8月 28 16:45 J131
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 17:08 J132
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 28 19:14 J133
drwxrwsrwx 4 cryosparc cryosparc 4096 8月 29 01:03 J134
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 29 02:04 J135
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 29 09:00 J136
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 29 09:01 J137
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 29 17:03 J138
drwxrwsrwx 4 cryosparc cryosparc 4096 9月 1 23:17 J139
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:36 J14
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 29 16:58 J140
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 29 16:51 J141
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 29 15:13 J142
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 29 17:57 J143
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 29 20:20 J144
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 29 20:19 J145
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 10:04 J146
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 10:01 J147
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 10:02 J148
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 09:53 J149
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:35 J15
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 10:07 J150
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 10:13 J151
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 17:34 J152
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 15:28 J153
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 15:05 J154
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 15:48 J155
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 15:49 J156
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 19:50 J157
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 30 20:57 J158
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 31 10:30 J159
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:44 J16
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 31 11:48 J160
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 31 15:49 J161
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 31 16:58 J162
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 31 17:21 J163
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 31 18:24 J164
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 31 18:31 J165
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 1 12:54 J166
drwxrwsrwx 4 cryosparc cryosparc 4096 9月 1 23:17 J167
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 2 00:44 J168
drwxrwsrwx 4 cryosparc cryosparc 4096 9月 2 00:43 J169
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:43 J17
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 2 12:54 J170
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 2 13:48 J171
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 2 16:10 J172
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 2 16:12 J173
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 2 17:03 J174
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 3 09:07 J175
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 3 09:05 J176
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 3 12:00 J177
drwxrwsrwx 4 cryosparc cryosparc 4096 9月 3 10:23 J178
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 4 02:24 J179
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:54 J18
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 3 13:53 J180
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 3 13:57 J181
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 3 14:00 J182
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 3 14:03 J183
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 3 16:52 J184
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 3 16:52 J185
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 3 16:55 J186
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 3 16:57 J187
drwxrwsrwx 3 cryosparc cryosparc 12288 9月 3 21:12 J188
drwxrwsrwx 3 cryosparc cryosparc 12288 9月 4 15:26 J189
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 18 13:48 J19
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 4 22:38 J190
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 4 09:58 J191
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 4 09:59 J192
drwxrwsrwx 3 cryosparc cryosparc 12288 9月 4 10:01 J193
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 4 11:12 J194
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 4 11:12 J195
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 4 11:20 J196
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 5 10:06 J197
drwxrwsrwx 3 cryosparc cryosparc 12288 9月 4 12:02 J198
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 5 13:53 J199
drwxrwsrwx 5 cryosparc cryosparc 4096 8月 17 16:12 J2
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 18 15:03 J20
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 5 09:50 J200
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 5 09:53 J201
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 5 13:05 J202
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 5 10:05 J203
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 5 16:27 J204
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 5 16:57 J205
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 5 20:06 J206
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 5 23:42 J207
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 5 23:06 J208
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 6 00:20 J209
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 18 15:11 J21
drwxrwsrwx 3 cryosparc cryosparc 12288 9月 6 11:15 J210
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 6 10:30 J211
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 6 11:37 J212
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 6 14:19 J213
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 6 12:58 J214
drwxrwsrwx 3 cryosparc cryosparc 12288 9月 6 14:50 J215
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 6 16:59 J216
drwxrwsrwx 3 cryosparc cryosparc 12288 9月 6 18:35 J217
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 6 19:51 J218
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 6 23:37 J219
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 18 15:13 J22
drwxrwsrwx 4 cryosparc cryosparc 4096 9月 7 00:45 J220
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 7 01:40 J221
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 7 01:56 J222
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 7 15:54 J223
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 7 16:01 J224
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 7 16:03 J225
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 7 16:03 J226
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 7 16:11 J227
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 7 16:12 J228
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 7 16:19 J229
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 18 15:15 J23
drwxrwsrwx 3 cryosparc cryosparc 12288 9月 7 16:54 J230
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 00:15 J231
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 09:43 J232
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 09:45 J233
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 10:03 J234
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 10:53 J235
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 11:11 J236
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 10:53 J237
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 12:29 J238
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 18:52 J239
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 18 16:02 J24
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 22:34 J240
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 23:09 J241
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 23:16 J242
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 23:41 J243
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 8 23:42 J244
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 9 00:30 J245
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 9 09:03 J246
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 9 18:17 J247
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 9 18:20 J248
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 9 18:21 J249
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 18 16:03 J25
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 9 18:22 J250
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 9 18:22 J251
drwxrwsrwx 3 cryosparc cryosparc 12288 9月 9 18:56 J252
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 9 20:45 J253
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 11 18:21 J254
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 16 12:42 J255
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 10 11:20 J256
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 10 11:20 J257
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 13 20:46 J258
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 16 12:23 J259
drwxrwsrwx 4 cryosparc cryosparc 4096 8月 18 17:39 J26
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 16 12:24 J260
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 16 12:35 J261
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 16 12:36 J262
drwxrwsrwx 3 cryosparc cryosparc 4096 9月 16 12:48 J263
drwxrwsrwx 3 cryosparc cryosparc 4096 10月 28 00:09 J264
drwxrwsrwx 3 cryosparc cryosparc 4096 10月 28 09:13 J265
drwxrwsrwx 3 cryosparc cryosparc 4096 10月 28 09:44 J266
drwxrwsrwx 3 cryosparc cryosparc 4096 11月 4 16:05 J267
drwxrwsrwx 3 cryosparc cryosparc 4096 11月 4 20:10 J268
drwxrwsrwx 3 cryosparc cryosparc 4096 11月 5 19:35 J269
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 18 19:13 J27
drwxrwsrwx 3 cryosparc cryosparc 4096 11月 5 19:35 J270
drwxrwsrwx 3 cryosparc cryosparc 4096 11月 5 14:15 J271
drwxrwsr-x 3 cryosparc cryosparc 4096 11月 5 19:35 J272
drwxrwsr-x 3 cryosparc cryosparc 4096 11月 5 19:06 J273
drwxrwsr-x 3 cryosparc cryosparc 4096 11月 5 19:02 J274
drwxrwsr-x 3 cryosparc cryosparc 4096 11月 5 19:02 J275
drwxrwsr-x 3 cryosparc cryosparc 4096 11月 5 19:04 J276
drwxrwsr-x 3 cryosparc cryosparc 4096 11月 5 19:04 J277
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 08:38 J28
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 08:46 J29
drwxrwsrwx 4 cryosparc cryosparc 4096 8月 17 16:34 J3
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 15:25 J30
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 15:25 J31
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 15:30 J32
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 15:43 J33
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 15:47 J34
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 15:45 J35
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 15:48 J36
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 15:47 J37
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 15:51 J38
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 16:58 J39
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 16:48 J4
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 17:02 J40
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 17:01 J41
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 17:04 J42
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 17:04 J43
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 17:04 J44
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 17:06 J45
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 18:02 J46
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 19:06 J47
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 19 19:21 J48
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 20 10:18 J49
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 21 09:44 J5
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 20 10:49 J50
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 20 15:35 J51
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 20 16:23 J52
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 20 16:31 J53
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 20 16:30 J54
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 20 16:32 J55
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 20 16:34 J56
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 21 09:21 J57
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 21 10:46 J58
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 21 11:03 J59
drwxrwsrwx 4 cryosparc cryosparc 4096 8月 17 17:01 J6
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 21 16:35 J60
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 21 17:02 J61
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 16:35 J62
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 16:35 J63
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 23 11:05 J64
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 23 14:36 J65
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 23 15:36 J66
drwxrwsrwx 4 cryosparc cryosparc 4096 8月 25 12:37 J67
drwxrwsrwx 5 cryosparc cryosparc 4096 8月 25 23:35 J68
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 25 23:50 J69
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:12 J7
drwxrwsrwx 4 cryosparc cryosparc 4096 8月 26 01:10 J70
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 08:37 J71
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 29 19:06 J72
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 14:08 J73
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 16:12 J74
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 11:32 J75
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 14:34 J76
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 12:29 J77
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 12:33 J78
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 15:21 J79
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:20 J8
drwxrwsrwx 4 cryosparc cryosparc 4096 8月 26 18:00 J80
drwxrwsrwx 4 cryosparc cryosparc 4096 8月 26 18:00 J81
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 16:07 J82
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 16:32 J83
drwxrwsrwx 4 cryosparc cryosparc 4096 8月 26 19:03 J84
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 23:56 J85
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 20:57 J86
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 26 22:42 J87
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 08:28 J88
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 08:48 J89
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 17 17:15 J9
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 08:41 J90
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 08:46 J91
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 09:43 J92
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 09:06 J93
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 08:54 J94
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 08:54 J95
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 09:15 J96
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 09:09 J97
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 09:18 J98
drwxrwsrwx 3 cryosparc cryosparc 4096 8月 27 09:35 J99
-rwxrwxrwx 1 cryosparc cryosparc 4346 11月 5 19:35 job_manifest.json
-rwxrwxrwx 1 cryosparc cryosparc 3111 11月 4 10:00 project.json
-rwxrwxrwx 1 cryosparc cryosparc 5180 11月 5 19:02 workspaces.json
$ ssh cryosparc@192.168.202.14 grep "$(df /data3/cryosp_projs/cryosparc/lixinzhu/CS-lxz240815-f2-vf/ | tail -n 1 | awk '{print $NF}')\ " /proc/mounts
/dev/sda1 /data3 xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,sunit=128,swidth=896,noquota 0 0
192.168.202.13:/data3 /data3 nfs4 rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.202.14,local_lock=none,addr=192.168.202.13 0 0
I also tried the “Test job launch” in web browser
and command
$ cryosparcm test workers P6
Using project P6
Running worker tests...
2024-11-05 19:04:31,681 log CRITICAL | Worker test results
2024-11-05 19:04:31,681 log CRITICAL | jylab
2024-11-05 19:04:31,681 log CRITICAL | ✓ LAUNCH
2024-11-05 19:04:31,681 log CRITICAL | ✓ SSD
2024-11-05 19:04:31,681 log CRITICAL | ✓ GPU
2024-11-05 19:04:31,684 log CRITICAL | ⚠ NVIDIA GeForce RTX 4090 @ 00000000:21:00.0: Persistence Mode is Disabled. Enable Persistence mode by running `nvidia-smi -pm 1` as root to persist the NVIDIA driver, reducing GPU load times.
...
2024-11-05 19:04:31,685 log CRITICAL | 192.168.202.14
2024-11-05 19:04:31,685 log CRITICAL | ✕ LAUNCH
2024-11-05 19:04:31,685 log CRITICAL | Error:
2024-11-05 19:04:31,685 log CRITICAL | See P6 J275 for more information
2024-11-05 19:04:31,685 log CRITICAL | ⚠ SSD
2024-11-05 19:04:31,685 log CRITICAL | Did not run: Launch test failed
2024-11-05 19:04:31,685 log CRITICAL | ⚠ GPU
2024-11-05 19:04:31,685 log CRITICAL | Did not run: Launch test failed
I did not expect two matches for the /data3 mountpoint. Are there really a local and a network filesystem mounted under /data3 on 192.168.202.14, or did I improperly apply quotes in the ssh command?
Please can you post the output of the df -h command on 192.168.202.14?
You are right, a local and a network filesystem mounted under /data3 on 192.168.202.14.
I remount the file and connect again, but the job is still “launched”. cryosparc_worker was reinstalled in /data
cryosparc@user-R5350-G6:~$ df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 101G 37M 101G 1% /run
/dev/sdb2 1.8T 32G 1.6T 2% /
tmpfs 504G 0 504G 0% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
/dev/sdb1 511M 6.1M 505M 2% /boot/efi
tmpfs 101G 84K 101G 1% /run/user/128
tmpfs 101G 116K 101G 1% /run/user/1000
tmpfs 101G 80K 101G 1% /run/user/1001
/dev/nvme1n1 3.5T 28K 3.3T 1% /data1
/dev/nvme0n1 7.0T 28K 6.6T 1% /data2
/dev/sda1 102T 744G 102T 1% /data
192.168.202.13:/data3 102T 76T 21T 79% /data3
I also think about that previous cryosparc_master was installed as “single workstation” mode not “master node only”, does this matter?
I had updated the worker connection, but it still not work.
I found some clue using “cryosparcm joblog P6 J297” when the stuck job was queueing.
================= CRYOSPARCW ======= 2024-11-11 19:10:12.763815 =========
Project P6 Job J297
Master jylab Port 39002
===========================================================================
MAIN PROCESS PID 79097
========= now starting main process at 2024-11-11 19:10:12.764350
/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py:135: UserWarning: *** CommandClient: (http://jylab:39002/api) URL Error [Errno -3] Temporary failure in name resolution, attempt 1 of 3. Retrying in 30 seconds
system = self._get_callable("system.describe")()
/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py:135: UserWarning: *** CommandClient: (http://jylab:39002/api) URL Error [Errno -3] Temporary failure in name resolution, attempt 1 of 3. Retrying in 30 seconds
/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py:135: UserWarning: *** CommandClient: (http://jylab:39002/api) URL Error [Errno -3] Temporary failure in name resolution, attempt 2 of 3. Retrying in 30 seconds
system = self._get_callable("system.describe")()
/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py:135: UserWarning: *** CommandClient: (http://jylab:39002/api) URL Error [Errno -3] Temporary failure in name resolution, attempt 2 of 3. Retrying in 30 seconds
system = self._get_callable("system.describe")()
cli = client.CommandClient(master_hostname, int(master_command_core_port), service="command_core")
File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_compute/client.py", line 38, in __init__
cli = client.CommandClient(master_hostname, int(master_command_core_port), service="command_core")
File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_compute/client.py", line 38, in __init__
super().__init__(service, host, port, url, timeout, headers, cls=NumpyEncoder)
File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 97, in __init__
super().__init__(service, host, port, url, timeout, headers, cls=NumpyEncoder)
File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 97, in __init__
self._reload() # attempt connection immediately to gather methods
File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 135, in _reload
self._reload() # attempt connection immediately to gather methods
File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 135, in _reload
system = self._get_callable("system.describe")()
File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 108, in func
system = self._get_callable("system.describe")()
File "/data/cryosp_projs/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/command.py", line 108, in func
raise CommandError(
cryosparc_tools.cryosparc.errors.CommandError: *** (http://jylab:39002, code 500) Encounted error from JSONRPC function "system.describe" with params ()
raise CommandError(
cryosparc_tools.cryosparc.errors.CommandError: *** (http://jylab:39002, code 500) Encounted error from JSONRPC function "system.describe" with params ()
What are the outputs of these commands on jylab:
uname -a
curl jylab:39002
curl 127.0.0.1:39002
ssh cryosparc@192.168.202.14 "curl jylab:39002 && uname -a"
It seems that worker does not know “jylab”, but it can work with “192.168.202.13”. Are there some places we can replace the name?
$ uname -a
Linux jylab 6.8.0-45-generic #45~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Sep 11 15:25:05 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
$ curl jylab:39002
Hello World from cryosparc command core.
$ ssh cryosparc@192.168.202.14 "curl jylab:39002 && uname -a"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: jylab
$ ssh cryosparc@192.168.202.14 "curl 192.168.202.13:39002 && uname -a"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Hello World from cryosparc command core.
100 41 100 41 0 0 23203 0 --:--:-- --:--:-- --:--:-- 41000
Linux user-R5350-G6 6.2.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 16:27:29 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
I think maybe this situation.
https://discuss.cryosparc.com/t/irresponsive-worker-node-after-installation/10842/22?u=panbx
OK, the problem was finally resolved.
hostname “jylab” of master was added into /etc/hosts in worker.
It is running now.