Hello Structura Friends,
We’ve been using CryoSPARC Live with our Slurm cluster for a while, and overall it works rather well for our purposes (barring the occasional bug).
We have noticed, however, that the CryoSPARC Live worker has a tendency to busy wait on top of GPUs if our microscope operators aren’t conscientious about pausing their live sessions. Phrased differently, we occasionally find a few long-running live sessions that are allocated one or more GPUs that aren’t being used, and these live sessions block other jobs on our cluster from using otherwise idle GPUs.
Would it be possible to make it so that if no new exposures are discovered within a configurable time interval (e.g., 1 hour), the CryoSPARC master automatically kills the worker jobs? It could then restart them when new images are detected.
This would allow us to achieve much better resource utilization on our cluster, which is increasingly becoming a higher priority for us as server components increase in price due to the AI boom / macroeconomic conditions.
Let me know what you think!
–John
1 Like
Thanks @jpellman for your post. Have you already considered related functionality in CryoSPARC v5, which is currently in beta testing?
Hi @wtempel ,
Sorry for the noise- I was unaware that CryoSPARC v5 already had this functionality implemented / I missed this in the release notes. This looks to be exactly what we want.
–John
2 Likes
Following on, since we find ourselves in a similar situation…
Is it possible to adjust default values of this (and other) CSLive session parameters? For example, to have every session create with this auto-pause feature turned on with some timeout duration? Would also be nice to be able to set default values for the exposure group directory to watch.
Omar
I think that you should be able to create a profile with some sensible settings for auto-pause using the instructions here, although I’m not sure if there’s a way to assign a default profile for all future sessions.
Being able to set up a default profile would be nice. I fear that if people aren’t willing to pause their sessions, they also won’t be willing to apply a profile before starting their sessions. So it would be nice to have new sessions effectively auto-apply a profile to themselves