SLURM accounting

Hi all,
The SLURM cluster I am administrating needed to activate accounting, so each users activities would be measured and fairshare in SLURM could applied.
Now this introduced a problem with our cryosparc master setup, because all jobs from cryosparc master needs to run as a particular cryosparc single user.

How could we register usage of the individual users of cryosparc, given the obvious challenges?

The solution was to add the cryosparc user to all the SLURM accounts that wanted to run cryosparc and had licenses for it with “sacctmgr add user …”

Then replace the SLURM lanes in the cryosparc master with lanes where “sbatch” was replaced with “bash” in cluster_info.json.
The cluster_script.sh should look something like this:

SLURM_ACCOUNT=$(sacctmgr show user | grep  " {{ job_creator }} " | awk '{print $2}')
cat <<EOF > /tmp/${CUSER}_{{ project_uid }}_{{ job_uid }}.sbatch
#!/bin/bash
#SBATCH --job-name ${CUSER}_{{ project_uid }}_{{ job_uid }}
#SBATCH -n {{ num_cpu }}
#SBATCH -p gpu
#SBATCH --gres=gpu:{{ num_gpu }}
#SBATCH --mem={{ (ram_gb)|int }}G
#SBATCH -o {{ job_dir_abs }}/slurm.log
#SBATCH -e {{ job_dir_abs }}/error.log
#SBATCH -A $SLURM_ACCOUNT
#
# Run job
{{ run_cmd }}
############
EOF
sbatch /tmp/${CUSER}_{{ project_uid }}_{{ job_uid }}.sbatch

As you can see this makes a sbatch script in /tmp on the crysparc master server. Which is then run with the bash command from cluster_info.json.
Each cryosparc user must be registered with a “username” a.k.a. {{ job_creator }} that matches their linux username for this to work. Otherwise one might be able to retrieve username from your LDAP/AD/NIS server, by using the crysoarc login email.

This works, but it would have been nicer if cryosparc master could run the job as the individual linux users, through e.g. “sudo -H -u user …”. But unfortunately one cannot do this, because the master database is locked by the cryosparc master user. Hopefully the cryoSPARC team might reconsider @adesgeorges nice suggestion (https://discuss.cryosparc.com/t/multi-user-installation/52/11) to have separate databases per user. This could maybe be setup with ACLs, so the user would have write access to own database, while cryosparc master user would only read, to feed back metadata into web-interface.

Best regards,
Jesper

1 Like

Hi @jelka,

Thanks for pointing this out! Creating a multi-user cryoSPARC is something we’re actively designing and developing.

To get around this problem, we just created multiple lanes for each User, then ask them to only use that lane. the lane config has the appropriate project in the sbatch script. you do need to add the user that is running the server to each slurm account.

1 Like

Hi @stephan,

That sounds very interesting. If you need alpha/beta testers just ask.

Thanks for the suggestion @karcaw.
This will also work, but I my case it would result i a lot of lanes.
There is also the question of a system based on trust entirely, versus the more automated/controlled solution I present above.
I guess it is a matter of taste or number of users?