Limiting jobs to specific lane(s)

Install, Configure and Update
1

We have CPU and GPU SLURM partitions for jobs from a Cryosparc instance (and other programs), and a couple of different lanes to target them depending on resources needed. However, users sometimes choose lanes that are not fit for the job, and eg. end up booking a GPU node for a CPU-only job such as Extract Particles.

Is it possible to restrict specific jobs to specific lane(s), or would you consider implementing that function?

Along the same line, is it possible to hide individual jobs? We often have free CPU nodes and a queue on the GPU nodes, but still some users will submit Extract Particles GPU jobs instead of using the CPU version. Hiding the GPU Extract job could be useful.

2

@boggild Would Yang’s suggestion apply to your use case? Block CPU-only jobs on GPU servers - #2 by leetleyang

3

It’s a good start, but more control to limit job types to/from certain lanes would still be appreciated. Eg. to make sure Patch Motion jobs are not submitted to lanes pointing to nodes with large SSDs and lots of RAM + VRAM, when other jobs would be more in need of those resources.

A somewhat related question, regarding lane restrictions: is it possible to have certain lanes restricted by default, in cases where we just want a few users testing them (ie. an allow list instead of a block list)?

4

One could include in the script template something like

{%- if job_type in ["patch_motion_correction_multi", "patch_ctf_estimation_multi"] %}
#SBATCH --partition=smallgpu
{%- elif num_gpu > 0 %}
#SBATCH --partition=largegpu
{%- else %}
#SBATCH --partition=cpu
{%- endif %}

One might try cryosparc_username-dependent routing to the restricted SLURM partitions, using jinja contstructs similar to the one above. cryosparc_username (list of available variables) would be the email address associated with the CryoSPARC login.

2 Likes