Hi
The new lane values are quite off from real usage:
Best wishes
@biocit Please can you confirm that you are comparing the numerators in this display
with the outputs of some (which?) resource monitors?
If not, please
- highlight the details you found to be incorrect
- post command outputs that show the correct values
- post the output of the command
cryosparcm resources
my understanding of the top values are that it shows what is currently in use within the lane. So it claims to use 7 CPUs and 4 GPUs in total. But alone in that screenshot it uses 15 CPUs and 7 GPUs.
The total value of the resources are correct. But the calculation is not working.
@biocit Were any of the jobs shown started via Run on specific GPU, over the CryoSPARC scheduler?
Don’t know. Officially I tell the users not to do it. Is there a indication how the job was started?
I just saw that a “Streaming 2D Classification” job from a live session was started and the GPU used count was not updated.
@biocit Please can you post the output of the command
cryosparcm resources so we may check if the counts you see may be affected by the structure of your scheduler lanes.
Sure.
Target Property Value
------------------------ ------------- ----------------------------------------------------
b-gpu01 Type node
Host b-gpu01
Lane RTX4090
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 256
RAM 512 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 4090
GPU 0 Mem 23.52 GiB
GPU 1 Name NVIDIA GeForce RTX 4090
GPU 1 Mem 23.52 GiB
GPU 2 Name NVIDIA GeForce RTX 4090
GPU 2 Mem 23.52 GiB
GPU 3 Name NVIDIA GeForce RTX 4090
GPU 3 Mem 23.52 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 1.02 GB
------------------------ ------------- ----------------------------------------------------
b-gpu02 Type node
Host b-gpu02
Lane RTX4090
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 256
RAM 512 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 4090
GPU 0 Mem 23.52 GiB
GPU 1 Name NVIDIA GeForce RTX 4090
GPU 1 Mem 23.52 GiB
GPU 2 Name NVIDIA GeForce RTX 4090
GPU 2 Mem 23.52 GiB
GPU 3 Name NVIDIA GeForce RTX 4090
GPU 3 Mem 23.52 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 1.02 GB
------------------------ ------------- ----------------------------------------------------
b-gpu03 Type node
Host b-gpu03
Lane RTX4090
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 256
RAM 512 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 4090
GPU 0 Mem 23.52 GiB
GPU 1 Name NVIDIA GeForce RTX 4090
GPU 1 Mem 23.52 GiB
GPU 2 Name NVIDIA GeForce RTX 4090
GPU 2 Mem 23.52 GiB
GPU 3 Name NVIDIA GeForce RTX 4090
GPU 3 Mem 23.52 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 1.02 GB
------------------------ ------------- ----------------------------------------------------
b-gpu04 Type node
Host b-gpu04
Lane RTX4090
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 256
RAM 512 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 4090
GPU 0 Mem 23.52 GiB
GPU 1 Name NVIDIA GeForce RTX 4090
GPU 1 Mem 23.52 GiB
GPU 2 Name NVIDIA GeForce RTX 4090
GPU 2 Mem 23.52 GiB
GPU 3 Name NVIDIA GeForce RTX 4090
GPU 3 Mem 23.52 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 1.02 GB
------------------------ ------------- ----------------------------------------------------
m-gpu01 Type node
Host m-gpu01
Lane RTX2080
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 64
RAM 128 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 2080 Ti
GPU 0 Mem 10.57 GiB
GPU 1 Name NVIDIA GeForce RTX 2080 Ti
GPU 1 Mem 10.57 GiB
GPU 2 Name NVIDIA GeForce RTX 2080 Ti
GPU 2 Mem 10.57 GiB
GPU 3 Name NVIDIA GeForce RTX 2080 Ti
GPU 3 Mem 10.57 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 1.02 GB
------------------------ ------------- ----------------------------------------------------
m-gpu02 Type node
Host m-gpu02
Lane Maintenance
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 128
RAM 128 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 2080 Ti
GPU 0 Mem 10.57 GiB
GPU 1 Name NVIDIA GeForce RTX 2080 Ti
GPU 1 Mem 10.57 GiB
GPU 2 Name NVIDIA GeForce RTX 2080 Ti
GPU 2 Mem 10.57 GiB
GPU 3 Name NVIDIA GeForce RTX 2080 Ti
GPU 3 Mem 10.57 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 1.02 GB
------------------------ ------------- ----------------------------------------------------
m-gpu03 Type node
Host m-gpu03
Lane RTXA5000
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 64
RAM 256 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA RTX A5000
GPU 0 Mem 23.55 GiB
GPU 1 Name NVIDIA RTX A5000
GPU 1 Mem 23.55 GiB
GPU 2 Name NVIDIA RTX A5000
GPU 2 Mem 23.55 GiB
GPU 3 Name NVIDIA RTX A5000
GPU 3 Mem 23.55 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 10.00 GB
------------------------ ------------- ----------------------------------------------------
m-gpu04 Type node
Host m-gpu04
Lane RTX2080
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 40
RAM 128 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 2080 Ti
GPU 0 Mem 10.57 GiB
GPU 1 Name NVIDIA GeForce RTX 2080 Ti
GPU 1 Mem 10.57 GiB
GPU 2 Name NVIDIA GeForce RTX 2080 Ti
GPU 2 Mem 10.57 GiB
GPU 3 Name NVIDIA GeForce RTX 2080 Ti
GPU 3 Mem 10.57 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 1.02 GB
------------------------ ------------- ----------------------------------------------------
o-gpu01 Type node
Host o-gpu01
Lane RTX2080
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 40
RAM 128 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 2080 Ti
GPU 0 Mem 10.57 GiB
GPU 1 Name NVIDIA GeForce RTX 2080 Ti
GPU 1 Mem 10.57 GiB
GPU 2 Name NVIDIA GeForce RTX 2080 Ti
GPU 2 Mem 10.57 GiB
GPU 3 Name NVIDIA GeForce RTX 2080 Ti
GPU 3 Mem 10.57 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 1.02 GB
------------------------ ------------- ----------------------------------------------------
r-gpu01 Type node
Host r-gpu01
Lane RTX2080
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 40
RAM 128 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 2080 Ti
GPU 0 Mem 10.57 GiB
GPU 1 Name NVIDIA GeForce RTX 2080 Ti
GPU 1 Mem 10.57 GiB
GPU 2 Name NVIDIA GeForce RTX 2080 Ti
GPU 2 Mem 10.57 GiB
GPU 3 Name NVIDIA GeForce RTX 2080 Ti
GPU 3 Mem 10.57 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 1.02 GB
------------------------ ------------- ----------------------------------------------------
r-gpu02 Type node
Host r-gpu02
Lane RTX2080
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 40
RAM 128 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 2080 Ti
GPU 0 Mem 10.57 GiB
GPU 1 Name NVIDIA GeForce RTX 2080 Ti
GPU 1 Mem 10.57 GiB
GPU 2 Name NVIDIA GeForce RTX 2080 Ti
GPU 2 Mem 10.57 GiB
GPU 3 Name NVIDIA GeForce RTX 2080 Ti
GPU 3 Mem 10.57 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 1.02 GB
------------------------ ------------- ----------------------------------------------------
r-gpu03 Type node
Host r-gpu03
Lane RTX2080
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 40
RAM 128 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 2080 Ti
GPU 0 Mem 10.57 GiB
GPU 1 Name NVIDIA GeForce RTX 2080 Ti
GPU 1 Mem 10.57 GiB
GPU 2 Name NVIDIA GeForce RTX 2080 Ti
GPU 2 Mem 10.57 GiB
GPU 3 Name NVIDIA GeForce RTX 2080 Ti
GPU 3 Mem 10.57 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 10.00 GB
------------------------ ------------- ----------------------------------------------------
r-gpu04 Type node
Host r-gpu04
Lane RTX2080
Worker /opt/cryosparc/cryosparc_worker/bin/cryosparcw
SSH Target cryosparcuser
CPUs 40
RAM 128 GiB
GPUs 0,1,2,3
GPU 0 Name NVIDIA GeForce RTX 2080 Ti
GPU 0 Mem 10.57 GiB
GPU 1 Name NVIDIA GeForce RTX 2080 Ti
GPU 1 Mem 10.57 GiB
GPU 2 Name NVIDIA GeForce RTX 2080 Ti
GPU 2 Mem 10.57 GiB
GPU 3 Name NVIDIA GeForce RTX 2080 Ti
GPU 3 Mem 10.57 GiB
Cache Path /disks/cryosparc-cache/cache
Cache Reserve 10.00 GB
------------------------ ------------- ----------------------------------------------------