Slow performance on multiple GPU

Hardware and Performance
1

HI all,
i have 4 GPU GTX1080Ti. they are all recognized by cryoSPARC, nvidia-smi show that they are mobilised if i asked a job to run on multiple GPU. However if i choose to run multiple GPu the job is way slower, up to 10 times slower than on a single GPU.
thoughts?
eric

2

Hi Eric,
Could you be running out of RAM?
//Jesper

3

I don’t think so. there is no indication in terms of usage of RAM, GPU memory or CPU that i am running out of anything.

4

BS"D

Is your case adequately ventilated? Are they “blower” GPUs (centripetal fans) that expel heat out the back of the cards? Heat build-up from 4 gpu’s could be throttling your cpu, or the graphics cards themselves, or other components like M.2 SSD’s on the motherboard.

Harry

1 Like
5

Hi @ehanssen,

Adding to the other questions, is the file system that your files are stored on able to saturate the disk read requirements of four GPUs accessing the files at the same time? What are your system specs?

6

Heat is a possibiity since it get slower as a job progress, even with only one GPU. the specs are

  • 1.00 BBN.SDP.NT8.4R08 NITRO T8 Quad GPU Workstation Barebone - Intel C621/16xDDR4-2666/AST2500 VGA/6xPCIe3
  • x16/1xPCIe3 x4/10xSATA3/2x 10GBase-T LAN/8xHot-Swap 3.5" HDD/2200W RPSU//IPMI w/KVM
  • 2.00 CPU.IX.GLD-5120 Intel Xeon Gold 5120/2.20GHz 10.4GT/s QPI/14Cores/19.25MB/LGA3467/TPD=105W
  • 8.00 RAM.D4-26.RE.032M.C1 32GB DDR4-2666 Registered ECC 1.2V Crucial Module 2Rank
  • 1.00 SSD.2SA850P.1000G Samsung 850 PRO 1TB, 3D V-NAND, SATA III 6Gb/s, R/W(Max) 540MB/s/520MB/s, 100K/90K IOPS,
  • 1.00 HDD.3SG.ST1000NM0008 Seagate Enterprise Capacity 3.5" 1TB SATA 6Gb/S 7200RPM 128MB Cache HDD
  • 5.00 HDD.3SG.ST10000NM001 Seagate Enterprise Capacity Helium 3.5" 10TB SATA 6Gb/S 7200RPM 256MB Cache HDD
  • 4.00 VGA.GTX1080TI.TRB-AS Asus GTX1080 Ti Turbo 1480HZ(BOOST 1582MHZ) 11G DDR5X PCIE3.0,2xHDMI,2xDP
  • 1.00 ODD.LIO.IHAS124 Lite-On DVD±24X8X8/4, DVDRAM12X, CD48X32X48, SATA, OEM PACK
  • 1.00 CAS.SRV.A034 XENON 2.5" HDD Tray in 6th Generation 3.5" Hot Swap Tray (SC747, 936, 938 and Blade)
  • 1.00 NET.XC10G.SFP.2P.LP XENON 10-Gigabit 2x SFP+ (Intel 82598ES) PCI-E x8 2.0 LP
7

Hi @ehanssen,

Your workstation definitely seems capable, so heat really might be the issue here. Also, if you can, I’d suggest upgrading from a SATA SSD to a PCIe based SSD to store your particles.

8

thanks i will try all of this and see how that goes.

9

Did you figure this out @ehanssen? One of my GPU workstations has started to get very slow all of a sudden (taking overnight for a single NU-refine iteration at 512px box size) and I am trying to figure out how to troubleshoot… it seems to be taking forever to compute FSCs - @stephan is the FSC computation code GPU accelerated? I think not, right?

10

Hi @olibclarke,

At the moment, no. A new version of our refinement jobs (which will be released in the near future) will include a GPU-accelerated version of the FSC computation, as well as a few other tasks that are currently CPU-bound.

2 Likes
11

not yet no. we have ordered some new hardware (SSD) to see if suggestions will be working. i will update as soon as i have everything running

1 Like