CryoSPARC submits all jobs to SLURM as the cryosparc user. This is running into problems on our cluster, because we have per-user limits set up in SLURM. As a result, person B’s jobs may not start, even though there are compute nodes available, because person A has running jobs.
A less-serious problem arises from this – occasionally a job gets into state where it never finishes, and currently only a sysadmin can cancel that job, since it’s running as the cryosparc user.
There is also a security aspect to this – people’s files are all owned by the cryosparc user.
In the future, might people be able to log into cryoSPARC with their Linux credentials, and could jobs be submitted to SLURM under that user account, and files owned by that user?
Thanks,
Matthew Cahn
Linux Sysadmin/Programmer
Dept. of Molecular Biology
Princeton University
mcahn@princeton.edu
Thanks for posting! We are currently in the middle of designing a multi-user “Hub” for exactly this need (Single sys-admin with full configuration capabilities, multiple semi-trusted users).
Would your cluster users be interested in installing their own cryoSPARC instance for each of their user accounts? This would allow them to:
Submit jobs using their own user account
Own their own files
Take advantage of the inherent isolation of unix user accounts
I do realize that this might not be possible, as allowing users to download, install and maintain their own cryoSPARC instances may become a big support burden for your team. However, we have created cryoSPARC such that the master instance is very light weight- having multiple instances on the same machine does not take up much resources (we maintain a similar setup for our developer instances). Please let me know what you think.
That’s good news about the multi-user hub. I’d like to avoid installing cryoSPARC separately for each. As the sysadmin I would be installing and maintaining each of those installations. Fortunately we have a special partition and QOS on the cluster for cryoEM work, so I was able to have the per-user job limit increased for that QOS to get around the immediate problem of jobs not starting.