We’re having a problem with our installation, whereby two CryoSPARC jobs compete for the same cache space. We have a small Slurm cluster, where each node has multiple GPUs and a dedicated CryoSPARC SSD cache. We have started seeing cases where multiple separate single-GPU jobs are getting allocated on the same node, and the cache requirements of job 1 effectively prevent the job 2 from from starting as there is not enough free space left to cache the job 2 particles.
There is no overall resource limitation as other nodes are free, but apart from allowing users to directly specify the node allocated as described here:
It looks like the best solution is to change the scheduler settings to use the least loaded node (CR_LLN) as described in the slurm documentation. Although it will not eliminate the chance of this occurring, it should at least reduce the probability by spreading jobs across nodes.
FS-Cache/CacheFS is system-level (as opposed to an application level caching mechanism), and provides a more sophisticated mechanism than CryoSPARC for caching datasets that are subject to read-heavy I/O patterns (e.g., Topaz training). If multiple jobs compete for the cache, cached pages from one or the other dataset can be evicted, and both workloads can continue running uninterrupted (though there may be a performance penalty).