'Cluster submission script variables' ignored

Troubleshooting
1

I recently added a ‘cluster’ lane to my v 4.1.2 instance of Cryosparc. The lane submits jobs to SLURM scheduler and generally works fine, except that the variables ram_gb, num_gpu, and num_cpu (used to construct the SLURM submission script) are always set to values pre-defined for a given job class rather than the user-defined ones. Although at job submission time a dialog ‘Cluster submission script variables’ appears where it should be possible to override these pre-defined parameters using the corresponding ‘This job’ textboxes, values entered there are always ignored. I would appreciate any advice on how to fix this.

2

Hi @robukowski,

The variables ram_gb, num_gpu, and num_cpu are actually meant to be unchangeable values set by CryoSPARC when submitting jobs to the cluster representing the job’s estimated required resources to run. Any custom variables using these names will have their values overwritten.

However, it is still possible to modify the requested resources in a cluster submission script using custom variables.
Below is a SLURM cluster configuration where the value for the amount of RAM required is modified by a ram_gb_multiplier custom variable.

#!/usr/bin/env bash
#SBATCH --job-name=cryosparc_{{ project_uid }}_{{ job_uid }}
#SBATCH -n {{ num_cpu }}
#SBATCH --gres=gpu:{{ num_gpu }}
#SBATCH --partition=slurmpar

## Example of a modifier variable
#SBATCH --mem={{ (ram_gb|float * ram_gb_multiplier|float)|int }}G

#SBATCH --output={{ job_log_path_abs }}
#SBATCH --error={{ job_log_path_abs }}

available_devs=""
for devidx in $(seq 0 15);
do
    if [[ -z $(nvidia-smi -i $devidx --query-compute-apps=pid --format=csv,noheader) ]] ; then
        if [[ -z "$available_devs" ]] ; then
            available_devs=$devidx
        else
            available_devs=$available_devs,$devidx
        fi
    fi
done
echo "AVAILABLE DEVICES: $available_devs"
export CUDA_VISIBLE_DEVICES=$available_devs

echo {{ instance_test_var }}

{{ run_cmd }}

The important line,
#SBATCH --mem={{ (ram_gb|float * ram_gb_multiplier|float)|int }}G
multiplies a user-controlled custom variable with the CryoSPARC job’s original estimation for required RAM, and submits the result to the cluster. This can be done similarly for num_cpu and num_gpu by using additive modifiers.

The documentation page Guide: Configuring Custom Variables for Cluster Job Submission Scripts - CryoSPARC Guide is incorrect at the time of writing this in that it uses ram_gb, num_gpu, and num_cpu as examples of modifiable custom variables, and I apologize for the confusion caused. It will be updated soon to clarify the internal CryoSPARC variables during cluster script submission.

4 Likes
3

Thanks so much for this explanation, using modifiers like ram_gb_multiplier would certainly solve the problem. However, it is not clear to me where during a job submission process these modifiers can actually be specified. Right now there is no textbox anywhere asking for the value of ram_gb_multiplier, for example. How to ‘tell’ crosparc to generate such a textbox?

4

Hi @robukowski,

Custom variables for cluster submission scripts are parsed from the script itself, so to get it to appear as a custom variables you need to modify the script file and reconnect the cluster configuration with cryosparcm cluster connect . (more info here Downloading and Installing CryoSPARC - CryoSPARC Guide ) CryoSPARC will parse the script variables and make the modifiable from the “Cluster configuration” tab of the admin panel as well as for each job queued to a cluster target.

From Guide: Configuring Custom Variables for Cluster Job Submission Scripts - CryoSPARC Guide :

At the target level, template variable names are parsed from the cluster submission script. Custom variables that only exist for that target can also be added. Any values for custom variables applied at the target level will override those that are set at the instance level.

If you have multiple cluster targets set up, you can also configure the variable at the instance level, but beware if the target script doesn’t use the variable it will be ignored:

At the instance level, custom variables and their values will be applied to all targets which include the respective custom variables in their templates. If a target’s submission script does not contain an instance level custom variable, that custom variable will not apply to the target.

1 Like
5

So why are those 3 variables always available to set in queueing dialog and in the submission variables settings, if the defaults cannot be overwritten and the value one set in the interface does not mean anything anywhere? I was very excited about this feature, exactly because I needed it for ram_gb and because default internal estimates are useless for many of our users, sometimes they are too small even multiplied by 2.