Edit available GPUs on master node

I wonder if it’s possible to change numbers of GPUs available on master node – I couldn’t find the respective setting in config.sh or anywhere.

The issue is, one of the GPUs stopped functioning, and machine currently has 1 less GPU than displayed in e.g. “Instance information” tab in CryoSPARC GUI: it says 4, whereas physically it’s 3 (and nvidia-smi also shows 3). Hence I want to set the number of GPUs on master node to 3, because currently when more than 3 jobs run simultaneously on the node, one of them throws IndexError on GPU allocation.

@marinegor Do I understand correctly that the CryoSPARC master host also acts as a GPU worker node?
In this case, you may try:

  1. saving the the output of the command
    cryosparcm cli "get_scheduler_targets()"
    for future reference
  2. removing the target that corresponds to the master host with the faulty GPU with the command
    cryosparcm cli "remove_scheduler_target_node('single_quoted_master_hostname_here')"
  3. reconnecting the master host as a worker
    cryosparcw connect ...
    using parameters derived from the saved get_scheduler_targets() output (guide).

Does that help?

yes, it actually does, thanks!

Was able to remove and reconnect the node. One thing required tweaking was copying ssh keys from localhots to localhots, which took some time to figure out ( cryosparcuser@server $ ssh-copy-id cryosparcuser@localhots), but that’s ok. Thanks!