FAILED TO LAUNCH ON WORKER NODE return code 255

Install, Configure and Update
1

Hi Kye,

The “Reassign Particles” job fails after starting on default CPU node. ERROR | FAILED TO LAUNCH ON WORKER NODE return code 255. It cannot be run in GPU mode (it gets stuck in launched state). Only once it worked on gpu, after 12 hours waiting. Other jobs are running fine.

This what I got from chat gpt - “because Reassign Particles is a utility job running in “CPU mode,” CryoSPARC completely bypasses your standard SLURM submission path. Instead, the CryoSPARC master server (c1925) is attempting a raw, direct SSH background connection over your internal network straight to node c2021 to fire up a standalone worker thread (c2021). Because your master server c1925 does not have the target hostname c2021 saved in its secure list of known computers, the SSH connection instantly panics, throws return code 255, and aborts.”

What are your suggestions? Why the master node is not available in this job?

Thank you!

Reassign Particles - Reprocessed Micrographs
2

@andro What version of CryoSPARC are you running?
Please can you post the scheduler configuration for your CryoSPARC instance.
The command for CryoSPARC v5:
cryosparcm resources
for v4:
cryosparcm cli "get_scheduler_targets()"

3

Hi wtempel,

it is ver 4.7.1, I ran the command for v4 in cluster terminal but the command was not found.

thank you!

4

Ensure you run the cryosparcm command

If the cryosparcm command ins not in the $PATH, specify the full “absolute” path, like (modify according to your actual CryoSPARC installation path):

/path/to/cryosparc_master/bin/cryosparcm cli "get_scheduler_targets()"
5

here is the information

[@c1505 ~]$ cryosparcm cli "get_scheduler_targets()"
[{'cache_path': '/scratch', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'hostname': 'c2021', 'lane': 'default', 'monitor_port': None, 'name': 'c2021', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5], 'GPU': [], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, 'ssh_str': '@c2021', 'title': 'Worker node c2021', 'type': 'node', 'worker_bin_path': '/opt/cryosparc/cryosparc_worker/bin/cryosparcw'}, {'cache_path': '/scratch', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'custom_var_names': [], 'custom_vars': {}, 'desc': None, 'hostname': 'swan-highmem', 'lane': 'swan-highmem', 'name': 'swan-highmem', 'qdel_cmd_tpl': 'scancel {{ cluster_job_id }}', 'qinfo_cmd_tpl': 'sinfo', 'qstat_cmd_tpl': 'squeue -j {{ cluster_job_id }}', 'qstat_code_cmd_tpl': None, 'qsub_cmd_tpl': 'cd /work/ /  && sbatch {{ script_path_abs }}', 'script_tpl': '#!/bin/bash\n#SBATCH --job-name=cryosparc_{{ project_uid }}_{{ job_uid }}\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node={{ num_cpu }}\n{%- if num_gpu == 0 %}\n#SBATCH --partition=batch\n{%- else %}\n#SBATCH --partition=gpu,gpu\n#SBATCH --gres=gpu:{{ num_gpu }}\n{%- endif %}\n#SBATCH --mem={{ (4*ram_gb*1000)|int }}MB\n#SBATCH --time=48:00:00\n#SBATCH --output={{ job_dir_abs }}/job_%J.out\n#SBATCH --error={{ job_dir_abs }}/job_%J.err\n#SBATCH --export=NONE\n\nmodule purge\nmodule load apptainer\n\necho "CUDA_VISIBLE_DEVICES="\n\n#export RUN_ARGS="{{ run_args }}"\n\napptainer exec -B /mnt/nrdstor///cryosparc2/config.sh:/opt/cryosparc/cryosparc_worker/config.sh -B /work/HCC/BCRF/app_specific/deepemhancer/2023.08.23:/usr/local/share/deepemhancer_models /work/HCC/APPTAINER/IMAGES/cryosparc-4.7.1.sif {{ worker_bin_path }} run {{ run_args }}\n', 'send_cmd_tpl': '{{ command }}', 'title': 'swan-highmem', 'tpl_vars': ['run_args', 'command', 'ram_gb', 'project_uid', 'job_dir_abs', 'worker_bin_path', 'cluster_job_id', 'num_gpu', 'num_cpu', 'job_uid'], 'type': 'cluster', 'worker_bin_path': '/opt/cryosparc/cryosparc_worker/bin/cryosparcw'}, {'cache_path': '/scratch', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'hostname': 'c1905', 'lane': 'default', 'monitor_port': None, 'name': 'c1905', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5], 'GPU': [], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, 'ssh_str': ' @c1905', 'title': 'Worker node c1905', 'type': 'node', 'worker_bin_path': '/opt/cryosparc/cryosparc_worker/bin/cryosparcw'}, {'cache_path': '/scratch', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'hostname': 'c2016', 'lane': 'default', 'monitor_port': None, 'name': 'c2016', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7], 'GPU': [], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, 'ssh_str': ' @c2016', 'title': 'Worker node c2016', 'type': 'node', 'worker_bin_path': '/opt/cryosparc/cryosparc_worker/bin/cryosparcw'}, {'cache_path': '/scratch', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'hostname': 'c2126', 'lane': 'default', 'monitor_port': None, 'name': 'c2126', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7], 'GPU': [], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, 'ssh_str': ' @c2126', 'title': 'Worker node c2126', 'type': 'node', 'worker_bin_path': '/opt/cryosparc/cryosparc_worker/bin/cryosparcw'}, {'cache_path': '/scratch', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'hostname': 'c2017', 'lane': 'default', 'monitor_port': None, 'name': 'c2017', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7], 'GPU': [], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, 'ssh_str': ' @c2017', 'title': 'Worker node c2017', 'type': 'node', 'worker_bin_path': '/opt/cryosparc/cryosparc_worker/bin/cryosparcw'}, {'cache_path': '/scratch', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'hostname': 'c1505', 'lane': 'default', 'monitor_port': None, 'name': 'c1505', 'resource_fixed': {'SSD': True}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7], 'GPU': [], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]}, 'ssh_str': ' @c1505', 'title': 'Worker node c1505', 'type': 'node', 'worker_bin_path': '/opt/cryosparc/cryosparc_worker/bin/cryosparcw'}, {'cache_path': '/scratch', 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'custom_var_names': [], 'custom_vars': {}, 'desc': None, 'hostname': 'swan', 'lane': 'swan', 'name': 'swan', 'qdel_cmd_tpl': 'scancel {{ cluster_job_id }}', 'qinfo_cmd_tpl': 'sinfo', 'qstat_cmd_tpl': 'squeue -j {{ cluster_job_id }}', 'qstat_code_cmd_tpl': None, 'qsub_cmd_tpl': 'cd /work/ /  && sbatch {{ script_path_abs }}', 'script_tpl': '#!/bin/bash\n#SBATCH --job-name=cryosparc_{{ project_uid }}_{{ job_uid }}\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node={{ num_cpu }}\n{%- if num_gpu == 0 %}\n#SBATCH --partition=batch\n{%- else %}\n#SBATCH --partition=gpu,gpu\n#SBATCH --gres=gpu:{{ num_gpu }}\n{%- endif %}\n#SBATCH --mem={{ (ram_gb*1000)|int }}MB\n#SBATCH --time=48:00:00\n#SBATCH --output={{ job_dir_abs }}/job_%J.out\n#SBATCH --error={{ job_dir_abs }}/job_%J.err\n#SBATCH --export=NONE\n\nmodule purge\nmodule load apptainer\n\necho "CUDA_VISIBLE_DEVICES="\n\n#export RUN_ARGS="{{ run_args }}"\n\napptainer exec -B /mnt/nrdstor/ / /cryosparc2/config.sh:/opt/cryosparc/cryosparc_worker/config.sh -B /work/HCC/BCRF/app_specific/deepemhancer/2023.08.23:/usr/local/share/deepemhancer_models /work/HCC/APPTAINER/IMAGES/cryosparc-4.7.1.sif {{ worker_bin_path }} run {{ run_args }}\n', 'send_cmd_tpl': '{{ command }}', 'title': 'swan', 'tpl_vars': ['run_args', 'command', 'ram_gb', 'project_uid', 'job_dir_abs', 'worker_bin_path', 'cluster_job_id', 'num_gpu', 'num_cpu', 'job_uid'], 'type': 'cluster', 'worker_bin_path': '/opt/cryosparc/cryosparc_worker/bin/cryosparcw'}]
[ @c1505 ~]$
6

Do I understand correctly that you removed the username prefix from @c2021 for privacy?
In this case, please confirm

  1. On the CryoSPARC master computer (presumably c1505?), what is the output of the the command (replace username)
    ssh username@c2021 'ls -l /opt/cryosparc/cryosparc_worker/bin/cryosparcw && hostname -s'
    ?
  2. Does the command in the previous step run without any additional prompt for user input (not for password, not to confirm the c2021 ssh host key, etc.)?
  3. Is the project directory mounted on c2021 at the expected path?
7

Our IT team resolved the issue. Below is response from them -

The default lane should only contain one node, which is the current “master” node.

After the CryoSPARC OOD job terminates successfully, this lane is reset.

However, if some of the CryoSPARC OOD jobs do not terminate successfully (which is the case with some of your runs), the new “master” node is appended to the default lane, which is causing the issue you encountered.

This can be fixed by removing all non-master nodes from the default lane by running cryosparcm cli “remove_scheduler_target_node(‘c2021’)” from the Terminal from the active CryoSPARC OOD Session.

Thank you!