Limiting simultaneous submitted/running/queued jobs per user

Features and Functionality
,
#1

Wondering if it is possible to limit the number of simultaneous jobs users can submit in the cryosparc interface?

We currently run both a worker and a cluster configuration. Since all cryosparc jobs run under the same user, we cannot limit further on the Slurm side.

#2

@vatese The {{ cryosparc_username }} template variable can be used in the cluster submission script as a comment or similar job submission parameter. Can slurm be configured to parse that information and include it in the resource allocation formula?

#3

Hi @wtempel,

Indeed we extract the username from the submit script and use it for our accounting and metrics. Nevertheless, we require the simultaneous job limitation to also extend for the Cryosparc workers that are not in Slurm. Cryosparc seems to allow jobs to be submitted on a firs come first served basis unless the priorities get changed for each user individually, which does not prevent a single user from hogging the whole cryosparc instance. Would by awesome if cryosparc could offer a per user job quota to prevent this.

Hope this make sense.

#4

@vatese
Am I correctly assuming that placing all workers under slurm control, possibly with multiple queues, is not an option?
In that case, the owner of the cryoSPARC instance can

  1. assign and modify user default priorities via GUI or cli: set_user_default_priority(email_address, priority)
  2. (dis-)allow users’ access to priority settings
  3. (*) modify jobs’ priorities via a low-level database update

Perhaps a combination of periodic usage analysis and updates of these settings according to a custom formula can ensure fairer access to the workers?

#5

Unfortunately that will not work for us. We have 20+ users and more are starting to use cryosparc. Slurm already has a QoS in place for cryosparc and we extract the user from the job for accounting and fairshare. Nevertheless, that only affects the Slurm side, user’s can still submit to the dedicated workers with no limitation. It would be good to have a setting to limit the maximum number of queued/running jobs straight from Cryosparc.

Cheers and thanks.