is there a way to prevent individual worker nodes from starting new jobs? Similar to the maintenance mode, but not blocking the whole instance.
I am managing a Cryosparc instance with a few worker nodes and need to do some maintenance work that includes reboots from time to time on the individual workers. It would be nice to do this without shutting down the whole Cryosparc instance.
cryosparcm cli "get_scheduler_targets()" | tee cryosparc_targets_$(date +%s).out
Identify the the 'name': of the worker node to be maintained and keep the output for reference during worker re-connection (after maintenance).
remove the target from the CryoSPARC instance cryosparcm cli remove_scheduler_target_node('worker_name')" (guide)
Allow jobs running on the “removed” worker to complete. In my testing, a 3-minute 3D classification job completed even though I had “removed” the worker shortly after job start.
After maintenance is complete, re-connect the worker with the appropriate arguments. On the worker, run cryosparcw connect with the appropriate parameters