Pause job? (feature request)


It would be nice to have the option to pause a currently running job - to allow for example one to restart cryosparc with different configuration, update cryosparc, or run a quick relion job on a GPU that CryoSPARC is otherwise occupying, and then resume the original job.


Pausing is unfortunately quite complex software-wise since the entire state of the optimizer as well as GPU memory needs to be dumped out, and then rebuilt to be able to resume. We’ve avoiding having to worry about this so far by just trying to make things faster :slight_smile: We’re considering building in pause functionality but it’s not on the roadmap yet.

Fair enough! I won’t complain about faster! :smile:


I’d like to give my support for / extend this idea. Some things can be made faster, but e.g. motion correction on 20 000 movies or 500 000 particles is always going to take a while. And more often than one would want, a job can fail - e.g. due to connectivity issues. It would be very preferable if jobs could have checkpoints from which they are able to continue, not to end up losing 2 days of calc due to a single error. Especially in the case of motion corr - the files (averaged micrographs, shift information, …) have in fact already been generated, but the user simply can’t reconstruct the cryosparc2 object that can be then passed around / use these already generated micrographs.

Of course in case of motion corr techincally it’s possible to split the data into chunks and make it happen this way, but I think it makes much more sense to implement this into how the program behaves than having users hack around it this way.

One other very prominent situation is Manual picking - there is nothing complex in pausing an manual pick job at any given moment - whether you save it as star or box files like e.g. relion would, or in your prefered database format. And it’s a tedious and time-consuming process that can take some time, and not only computation time. I can of course start a new manual pick job every time I go back to picking to ensure I don’t lose anything, but that’s in stark contrast to the elegance and robustness which cryosparc provides or aspires to provide. :slight_smile: