Multi-lane cluster configuration problem

Install, Configure and Update
1

Hello,

I have cryosparc working just fine with a single lane/cluster setup however, if I attempt to add a second cluster into cluster_info.json I receive parsing errors even though I have valid json. Would be great if someone would let me know what cryo is expecting as far as multi-lane configurations as i don’t see any examples in the docs.

Thanks

[{
“name”: “centralhpc-cluster”,
“worker_bin_path”: “xxx/bin/cryosparcw”,
“cache_path”: “/central/scratchio/xxx”,
“send_cmd_tpl”: “{{ command }}”,
“qsub_cmd_tpl”: “sbatch {{ script_path_abs }}”,
“qstat_cmd_tpl”: “squeue -j {{ cluster_job_id }}”,
“qdel_cmd_tpl”: “scancel {{ cluster_job_id }}”,
“qinfo_cmd_tpl”: “sinfo”,
“transfer_cmd_tpl”: “scp {{ src_path }} loginnode:{{ dest_path }}”
}, {
“name”: “centralhpc-cluster2”,
“worker_bin_path”: “xxx/bin/cryosparcw”,
“cache_path”: “/central/scratchio/xxx”,
“send_cmd_tpl”: “{{ command }}”,
“qsub_cmd_tpl”: “sbatch {{ script_path_abs }}”,
“qstat_cmd_tpl”: “squeue -j {{ cluster_job_id }}”,
“qdel_cmd_tpl”: “scancel {{ cluster_job_id }}”,
“qinfo_cmd_tpl”: “sinfo”,
“transfer_cmd_tpl”: “scp {{ src_path }} loginnode:{{ dest_path }}”
}]

Traceback (most recent call last):
File “<stdin>”, line 7, in <module>
TypeError: list indices must be integers, not str

2

This is an old topic but has an easy solution if anyone is still looking for it.

You have to add multiple lanes sequentially with their own cluster_info.json/cluster_script.sh pair.

I keep a directory structure like

lane1/cluster_info.json
     /cluster_script.sh
lane2/cluster_info.json
     /cluster_script.sh

Then I run the following overengineered script to update all of them:

#!/bin/bash
# Connect all lanes to cryosparc

: "${CRYOSPARCM:=$(which cryosparcm)}"

if [[ ! -x "$CRYOSPARCM" ]]; then
    echo "ERROR: Unable to find cryosparcm at $CRYOSPARCM" >&2
    exit 1
fi

LANES=("lane1" "lane2")

success=1
for lane in "${LANES[@]}"; do
    cd "$lane"
    WORKER=$(sed -nE 's/.*"worker_bin_path" *: *"([^"]*)".*/\1/p' cluster_info.json)
    if [[ ! -x "$WORKER" ]]; then
        echo "ERROR: Worker $WORKER from lane $lane not found." >&2
    else
        echo "$CRYOSPARCM" cluster connect >&2
        "$CRYOSPARCM" cluster connect

        if [[ $? != 0 ]]; then
            echo "ERROR connecting $lane" >&2
            success=0
        fi
        echo >&2

    fi
    cd ..
done

if [[ $success == 0 ]]; then
    echo "Errors occured. See above." >&2
fi

The checks are probably unneccessary for simple use. It was designed for an environment where multiple users have cryosparc installed in different places.

2 Likes