Hi there,
I am currently in the processing of speccing a GPU node (4 to 8 GPUS), I was wondering if the Cryosparc folks have any recommendations on GPUs.
Currently, L4 Ada and RTX 4500 Ada look like a good choice. I am leaning towards 4500 in an 8x format. Appreciate any pointers!
Best,
Abhiram
L4’s have a TDP of 72W. RTX 4500 Ada has a TDP of 210W. Even if all else were equal (4500 Ada has a few more shader units), you should see better performance from the 4500 Ada.
The L4 is about equivalent to a 3070Ti or 5060Ti 16GB (in terms of performance) while the 4500 Ada is roughly a 3090.
For me, it would all depend on price, power budget and where the system will be living (having a loud box on your desk is not fun)…
That makes sense. Appreciate the help!
I just checked with a vendor and it looks like the supply chain decided it for us. I am able to source A5000 in a 4x/8x format. Since this system will be housed in a data center, I should be okay with noise and power.
More GPUs are always better, as well as more GPU memory.
But local scratch disk performance is the limiting factor, not the GPUs.
As rbs_sci mentioned, RTX4500 Ada and L4 Ada are not the latest generation. CryoSparc is currently unfortunately still not ready for the Blackwell generation (RTX5000 series) of nVidia GPUs, but I would expect so soon.
Your system builder will likely not support “blower type” gamnig cards like the RTX5090, even in “2-slot” form factor: NVIDIA GeForce RTX 5090 Graphics Cards This is because servers are designed for linear airflow and the position of the power connector makes it harder to connect those gamnig cards. So the Ada cards may be your only choice (I would not get Ampere generation at this time - A5000 is at the performance level of a RTX2080Ti and 2 generations behind). Ada has twice the performance of Ampere, so L4 or RTX4500Ada is much better. RTX PRO 6000 Blackwell is the latest server GPU, if the vendor can source them (which will be difficult) - but it doesn’t work with cryosparc yet. You could consider a “mixed” configuration of Ada and Blackwell GPUs.
You need a system with enough PCIe lanes to feed those GPUs with data, which favors AMD processors (likely EPYC for your server). For 8 slots, they will probably only support PCIe 4.0, but that’s okay. Get at least 512GB RAM. And enough NVMe drives for scratch (e.g. minimum four 8GB drives with linux md_raid), or a PCIe-NVMe RAID0 card (e.g. Highpoint https://www.highpoint-tech.com/gen5) for fast data access from the scratch. Depending on the task, your GPUs will likely outperform even the bandwidth of PCIe5 NVMe RAID.
If your mass storage is in another box, make sure the machine has plenty networking bandwidth (multiple aggregated 10Gb/s fiber ports, or faster).
Your datacenter won’t like it, but for a quarter of the cost of your server you can build something like this https://x.com/HiCryoEM/status/1763712323104235892.
@mwolf May I ask what the “particle mining rig” costed you? Do you keep it in an airconditioned server room?
Hi @daniel.s.d.larsson , the linked one has 6x RTX4090. About US$10k with 6x NVMe4 4TB, 6x 20TB HDD (~90TB RAID5), 512GB, 24-core TRpro from Amazon, but without the GPUs. Final price will depend on your source. I buy the cheapest RTX GPUs with most cores - they all follow the nVidia reference design, rest is cosmetic. The Blackwell generation stays cooler. Pretty quiet in operation. Regular small room (3x4m) with building A/C. There are now 3 such servers in a rack in that room, plus a 4U 1PB storage server and a workstation. It’s warm, but can still work in there.
For base hardware, I recommend: Asus ASUS Pro WS WRX90E-SAGE SE (it has 7 PCIe5-16x slots, more stable at PCIe4 than WRX80E used in the above). AMD Ryzen Threadripper 9960X 24-core is sufficient (typically <=3 threads/GPU). For relion may want more cores. 256GB mostly enough. I did buy ECC memory. The WRX90E also has 2 PSU connectors on the motherboard, and 4 M.2 slots. Need a RAID card with enough ports for 6xHDD, which takes up one PCIe slot. Two Thermaltake ToughPower GF3 1650W come with all the power and cables you need. PCIe5 NVMe drives are significantly more expensive than gen4 and hard to find >4TB. I use NVMeGen4 software raid0 for scratch for simplicity, and one 1TB NVMeGen5 drive for system and database.Alma Linux or Rocky.
Mixed GPU configuration is better for price/performance. The last build has 6x RTX4070 and 2x 5080. The former just as fast for motioncor and patch CTF. Make sure you get quality riser cables and PCIe splitters (if you use) rated for at least PCIe4. Otherwise errors on the PCIe bus.
CryoSparc is now supporting Blackwell chips (RTX 5000 series) - Thanks, Structura Biotech!