Change order of GPUs

Install, Configure and Update
1

Hi @team,
I have lately swapped some gpus in my test node to have 2x card A and 1x card B.
How can I set the order within cryosparc?
Always cards A are listed on top, no matter how I physically plug them or what I define during the connect job.

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.197.02   Driver Version: 418.197.02   CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    On   | 00000000:19:00.0 Off |                  N/A |
| 24%   28C    P8     7W / 180W |      1MiB /  8119MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  On   | 00000000:1A:00.0 Off |                  N/A |
| 27%   33C    P8    16W / 250W |      0MiB / 10989MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  On   | 00000000:67:00.0 Off |                  N/A |
| 27%   38C    P8    21W / 250W |      0MiB / 10989MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+

 ---------------------------------------------------------------
  Autodetecting available GPUs...
  Detected 3 CUDA devices.

And here the cryosparc output:

   id           pci-bus  name
   ---------------------------------------------------------------
   0      0000:1A:00.0  GeForce RTX 2080 Ti
   1      0000:67:00.0  GeForce RTX 2080 Ti
   2      0000:19:00.0  GeForce GTX 1080
   ---------------------------------------------------------------
   Devices specified: 2, 0, 1
   Devices 2, 0, 1 will be enabled now.
   This can be changed later using --update

Best,
Tarek

2

What exactly are you trying to change? You can define which GPU is used within cryosparc.

3

yes, but not in CS Live.
By default cryosparc is loading the gpus from top to bottom.

4

Hi @tarek,

I just tried this myself, let me know if it works for you.
On the worker node, open up a shell.
Then, set the sort order by setting the following environment variable:
export CUDA_DEVICE_ORDER=PCI_BUS_ID
Finally, run the cryosparcw connect command with the --update flag. It should populate the GPU list using the BUS ID order rather than the “fastest card” order

5

perfect, that worked!
Thanks @stephan