Dear all,
After upgrading to the version 3.0.1 I noticed that all launched jobs started to use all available GPUs. Before each job used its designated GPU and the queuing worked well. Now jobs are getting really slow and I wonder whether this behavior is the reason why. Is this the expected method of job distribution?
As an example, I launched one job on the GPU 0 and I see this in nvidia-smi:
-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01 Driver Version: 455.45.01 CUDA Version: 11.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:41:00.0 Off | 0 |
| N/A 53C P0 29W / 70W | 166MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 Tesla T4 Off | 00000000:61:00.0 Off | 0 |
| N/A 51C P0 33W / 70W | 640MiB / 15109MiB | 16% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 Tesla T4 Off | 00000000:81:00.0 Off | 0 |
| N/A 47C P8 10W / 70W | 3MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 Tesla T4 Off | 00000000:C1:00.0 Off | 0 |
| N/A 53C P8 10W / 70W | 3MiB / 15109MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 61783 C python 163MiB |
| 1 N/A N/A 61783 C python 647MiB |
| 2 N/A N/A 61783 C python 0MiB |
| 3 N/A N/A 61783 C python 0MiB |
+-----------------------------------------------------------------------------+
I would greatly appreciate your help!
Best regards,
Gabor