Edit available GPUs on master node

Install, Configure and Update
1

I wonder if it’s possible to change numbers of GPUs available on master node – I couldn’t find the respective setting in config.sh or anywhere.

The issue is, one of the GPUs stopped functioning, and machine currently has 1 less GPU than displayed in e.g. “Instance information” tab in CryoSPARC GUI: it says 4, whereas physically it’s 3 (and nvidia-smi also shows 3). Hence I want to set the number of GPUs on master node to 3, because currently when more than 3 jobs run simultaneously on the node, one of them throws IndexError on GPU allocation.

2

@marinegor Do I understand correctly that the CryoSPARC master host also acts as a GPU worker node?
In this case, you may try:

  1. saving the the output of the command
    cryosparcm cli "get_scheduler_targets()"
    for future reference
  2. removing the target that corresponds to the master host with the faulty GPU with the command
    cryosparcm cli "remove_scheduler_target_node('single_quoted_master_hostname_here')"
    (guide)
  3. reconnecting the master host as a worker
    cryosparcw connect ...
    using parameters derived from the saved get_scheduler_targets() output (guide).

Does that help?

3

yes, it actually does, thanks!

Was able to remove and reconnect the node. One thing required tweaking was copying ssh keys from localhots to localhots, which took some time to figure out ( cryosparcuser@server $ ssh-copy-id cryosparcuser@localhots), but that’s ok. Thanks!

4