Hi all,
The SLURM cluster I am administrating needed to activate accounting, so each users activities would be measured and fairshare in SLURM could applied.
Now this introduced a problem with our cryosparc master setup, because all jobs from cryosparc master needs to run as a particular cryosparc single user.
How could we register usage of the individual users of cryosparc, given the obvious challenges?
The solution was to add the cryosparc user to all the SLURM accounts that wanted to run cryosparc and had licenses for it with “sacctmgr add user …”
Then replace the SLURM lanes in the cryosparc master with lanes where “sbatch” was replaced with “bash” in cluster_info.json.
The cluster_script.sh should look something like this:
SLURM_ACCOUNT=$(sacctmgr show user | grep " {{ job_creator }} " | awk '{print $2}')
cat <<EOF > /tmp/${CUSER}_{{ project_uid }}_{{ job_uid }}.sbatch
#!/bin/bash
#SBATCH --job-name ${CUSER}_{{ project_uid }}_{{ job_uid }}
#SBATCH -n {{ num_cpu }}
#SBATCH -p gpu
#SBATCH --gres=gpu:{{ num_gpu }}
#SBATCH --mem={{ (ram_gb)|int }}G
#SBATCH -o {{ job_dir_abs }}/slurm.log
#SBATCH -e {{ job_dir_abs }}/error.log
#SBATCH -A $SLURM_ACCOUNT
#
# Run job
{{ run_cmd }}
############
EOF
sbatch /tmp/${CUSER}_{{ project_uid }}_{{ job_uid }}.sbatch
As you can see this makes a sbatch script in /tmp on the crysparc master server. Which is then run with the bash command from cluster_info.json.
Each cryosparc user must be registered with a “username” a.k.a. {{ job_creator }} that matches their linux username for this to work. Otherwise one might be able to retrieve username from your LDAP/AD/NIS server, by using the crysoarc login email.
This works, but it would have been nicer if cryosparc master could run the job as the individual linux users, through e.g. “sudo -H -u user …”. But unfortunately one cannot do this, because the master database is locked by the cryosparc master user. Hopefully the cryoSPARC team might reconsider @adesgeorges nice suggestion (https://discuss.cryosparc.com/t/multi-user-installation/52/11) to have separate databases per user. This could maybe be setup with ACLs, so the user would have write access to own database, while cryosparc master user would only read, to feed back metadata into web-interface.
Best regards,
Jesper