Here is the nvidia-smi output of a typical worker node.
±----------------------------------------------------------------------------+
| NVIDIA-SMI 440.82 Driver Version: 440.82 CUDA Version: 10.2 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Quadro P5000 Off | 00000000:3B:00.0 Off | Off |
| 22% 37C P0 42W / 180W | 0MiB / 16278MiB | 0% Default |
±------------------------------±---------------------±---------------------+
| 1 Quadro P5000 Off | 00000000:D8:00.0 Off | Off |
| 22% 35C P0 42W / 180W | 0MiB / 16278MiB | 0% Default |
±------------------------------±---------------------±---------------------+±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+
Here is our larger GPU server while running a live session that generates the error.
±----------------------------------------------------------------------------+
| NVIDIA-SMI 450.66 Driver Version: 450.66 CUDA Version: 11.0 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Quadro RTX 6000 Off | 00000000:1A:00.0 Off | 0 |
| N/A 31C P0 56W / 250W | 908MiB / 22698MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 Quadro RTX 6000 Off | 00000000:1B:00.0 Off | 0 |
| N/A 24C P8 13W / 250W | 8MiB / 22698MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 2 Quadro RTX 6000 Off | 00000000:3D:00.0 Off | 0 |
| N/A 24C P8 13W / 250W | 8MiB / 22698MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 3 Quadro RTX 6000 Off | 00000000:3E:00.0 Off | 0 |
| N/A 25C P8 13W / 250W | 8MiB / 22698MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 4 Quadro RTX 6000 Off | 00000000:8B:00.0 Off | 0 |
| N/A 24C P8 12W / 250W | 8MiB / 22698MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 5 Quadro RTX 6000 Off | 00000000:8C:00.0 Off | 0 |
| N/A 26C P8 13W / 250W | 8MiB / 22698MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 6 Quadro RTX 6000 Off | 00000000:B5:00.0 Off | 0 |
| N/A 25C P8 14W / 250W | 8MiB / 22698MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 7 Quadro RTX 6000 Off | 00000000:B6:00.0 Off | 0 |
| N/A 24C P8 13W / 250W | 8MiB / 22698MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1755 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 9952 C python 225MiB |
| 0 N/A N/A 10064 C python 225MiB |
| 0 N/A N/A 10150 C python 225MiB |
| 0 N/A N/A 10238 C python 225MiB |
| 1 N/A N/A 1755 G /usr/lib/xorg/Xorg 4MiB |
| 2 N/A N/A 1755 G /usr/lib/xorg/Xorg 4MiB |
| 3 N/A N/A 1755 G /usr/lib/xorg/Xorg 4MiB |
| 4 N/A N/A 1755 G /usr/lib/xorg/Xorg 4MiB |
| 5 N/A N/A 1755 G /usr/lib/xorg/Xorg 4MiB |
| 6 N/A N/A 1755 G /usr/lib/xorg/Xorg 4MiB |
| 7 N/A N/A 1755 G /usr/lib/xorg/Xorg 4MiB |