GPU - Founders edition or OC (ZOTAC, ROG, TUF etc)?

Dear cryoSPARC community,

We are currently setting up a new workstation. Initially we were going with a single 4080 S, but they are not available with our provider anymore. However, they already have the new 5080s.

My main question is – are the custom OC cards (the one from ZOTAC, ROG, TUF etc) acceptable or one should only go with the founders edition? The way I heard it a way back was that the OC cards tend to overheat themselves and throttle, thus founders edition is a better choice. Is it true?

Also, I want to go with Intel Core i9 14900KF CPU (24 Cores, 32 Threads, 3.2GHz Base, 6.0GHz Turbo) – do you think that will be enough or should one go with a more powerful one?
For context, we rarely would go above 450 pix box and probably would not do cryoSPARC LIVE on this machine.

Below is the full configuration of the workstation I had in mind:
Motherboard: ASUS ROG STRIX Z790-E GAMING WIFI
CPU: Intel Core i9 14900KF (24 Cores, 32 Threads, 3.2GHz Base, 6.0GHz Turbo, 125W)
CPU cooler: Corsair NAUTILUS 360 RS
GPU: 16GB NVIDIA GeForce RTX 5080 (16GB GDDR7 VRAM @ 960.0 GB/s)
RAM: 64GB (2x32GB) Corsair Vengeance 6000MHz
Network Adapter: ASUS XG-C100C, 10GbE RJ45 Port
Power supply: Corsair HX1000i
Case: Fractal Design Define 7
Storage:
4TB Corsair MP700 PRO NVMe PCIe 5.0 (12400MB/s Read, 11800MB/s Write)
2TB Samsung 990 PRO NVMe PCIe 4.0 (7450MB/s Read, 6900MB/s Write)
40TB HDD (configured to RAID5)

I’d greatly appreciate any comments or suggestions!
Thank you,
Mat

Hey Mat,

I have two workstations—one with 2× 3090 Ti (FE) and another, a year old, with 1× 4090 (Zotac). I’d recommend going with GPUs that have at least 24 GB of VRAM to avoid running out of memory. Instead of investing in Intel CPUs, I’d suggest going for two powerful GPUs. If you have tight budget the one 5090 or one 4090 i would recommend.

My recommendation: go for either 2× 5090 (32 GB each—future-proof and powerful) or 2× 4090 (24 GB each). I haven’t noticed any significant difference between FE (Founders Edition) and third-party cards, but FE models are usually more affordable.

For the rest of the build:

  • CPU: Go for the Ryzen 9 9950—Intel is expensive. Even Ryzen 9 7950x should be fine. For RELION or CryoSPARC; GPU power is what matters.
  • Storage:
    • 4TB M.2 NVMe SSD for CryoSPARC scratch (Samsung 980 should be fine).
    • 1TB M.2 NVMe SSD for the OS boot drive.
    • At least 2× 20TB HDDs for data storage.
  • Cooler: Noctua D15 (quiet and reliable) or a liquid cooler for even better silence.
  • RAM: At least 128GB DDR5 (RELION loads particles into RAM, so you may run out with less).
  • PSU:
    • 2000W for 2× GPUs.
    • 1500W for a single high-end GPU setup.
  • Case: Fractal Design is expensive, so consider Corsair 750D for a dual-GPU build or Antsports Mid Tower for a single GPU.

Bottom line

Prioritize GPU and storage. A decent CPU is enough—don’t waste money on Intel CPUs or expensive cases.

2 Likes

I found the 3090, 4090 to be less efficient than the A5000 and A600 ada.

If you can I would take a serious look at the “ada” versions A5000 ada and A6000 ada before going to the new 5090.

You will find disk space being an issue and at some point you cannot have all the raw data can CryoSparc projects in the work station. You basically need network storage when you go over ~70TB of data. You can keep putting 3.5in RAID5 but eventually run out of space.

RAID5 is fine, 3.5in 20TB WD data center HDs are nice. You can put a few NVMe or M.2 drives in a rocket card in RAID0, the SSD scratch space is critical.

Are people still using intel CPUs ? The AMD Thread Rippers seemed to be better value. If going i9 would the Xeon Silver or Gold be more stable, memory cache, and have reliable ECC ?

Agree with 10Gbps ethernet

3 Likes

Apologies in advance for what will probably be quite a long post.

First, while this might sound obvious, decide budget. Budget will inform you regarding options for everything else. If tight, a 9950X will do quite well (you might even be able to squeeze two full systems in, if careful) but if there is lots of leeway, a Threadripper Pro might serve better (or even an Epyc dual-socket rack system).

We have no solid benchmarks (at least that I am aware of) which people have shared yet for the 5090’s with CryoSPARC and/or RELION. They perform well in most tasks, but have odd weaknesses. In benchmarks I’ve seen, they can be up to twice as fast in compute loads over a 4090, but also draw more peak and average power. Sometimes they show barely any improvement. With a quick skim, GPU Passmark has it losing compute tests to the 4090, but is unclear whether the 4090 score is overclocked or not. The increased power draw may be a concern.

My primary issue with using other results as a guide for performance in CryoSPARC/RELION is that it’s a poor estimate: my labs most recently commissioned system exhibits higher temperatures with a RELION run optimised to fill RAM and maximise both CPU and GPU load than it does when running mprime, GPU benchmarks and I/O benchmarks in parallel. Cryo-EM image processing is a good stress test!

I would get Founder’s Edition cards, or Quadros if possible and budget allows, as third party cards frequently have larger coolers (which are not always quieter!) and occasionally corners are cut with respect to VRMs (although I really hope no-one is doing this any more with cards that pull 600W!) @Mark-A-Nakasone is right to suggest looking at Ada Quadros, too, although when I could I still bought Ampere cards as they were 40% cheaper…

I would not recommend any modern consumer Intel CPU. They draw more power, are less performant and are recently coming off the back of a degradation scandal which Intel basically ignored and denied until there was no other choice. That doesn’t mean AMD are spotless (thinking of early AM5 boards cooking CPUs due to bad settings in the UEFI) but at least AMD didn’t deny it for well over a year. I’m also wary of heterogeneous cores for compute loads, but that’s a topic for another day…

I also don’t recommend closed-loop liquid coolers for systems which will be running 24/7 and spend time unattended. Perhaps I’ve just been unlucky, but for CLCs I’m currently batting a 20% failure rate (either pump failure or leaks) so really would recommend air cooling. As @Das says, a Noctua D15 (or equivalent) is going to be a pretty good option. Also look at the Thermalright PA120SE, we now have four workstations running with those and they are excellent for the price (here they are 30% of the cost of a Noctua for almost the same performance).

Get an overspec PSU (at least 1600W if going dual 5090s) which comes with reliable, high quality 12VHIPWR cables. Skimping here is folly, even if you have to cut back somewhere else. Dual PSUs are also an option depending on case.

You don’t need a large boot drive, but 1-2TB NVMe is not so expensive now and will have room for the space-inefficient horror which is Anaconda virtual environments. A pair of 4TB NVMes for RAID scratch would be a good idea. If using RELION and large stacks, RELION currently converts its particle stack to 32-bit MRC before loading to scratch, so that 16-bit stack will double in size - just a little surprise to be aware of. HDDs as you feel. I like Toshiba Enterprise drives, others like WD or Seagate. Buy spares.

RAM quantity will depend completely on what platform you decide on. AM5 (9950X) supports up to 192GB of RAM (I’m running this personally) but the memory controller will downclock it to 3600MT/s for stability, and I’ve been unable to get mine stable anything above that, although I admit I haven’t put as much time into it as I would have done 10-15 years ago. 128GB will run at 5600MT/s. Outside of synthetic benchmarks, I’ve not noticed an appreciable difference - at least in RELION/CryoSPARC the many other overhead factors mitigate some of the bandwidth loss between 96GB @ 5600MT/s and 192GB @ 3600MT/s, and the abiilty to handle larger volumes is more valuable to me, but not valuable enough to multiply the price of the motherboard by three, CPU by six and RAM by at least two.

In the past I’ve wholeheartedly recommended Fractal Design cases, although the Define 7 (XL) made several design decisions which I dislike… if wanting to load up on HDDs, be prepared to faff around moving and removing bits of the internals. If wanting to stuff the case full, I would avoid the 7 series cases. Airflow takes a hit if you exceed 2x HDDs in the PSU bay (for the 7) or 4x HDDs in the PSU bay (for the 7 XL). I eventually gave up as cable routing in the “HDD optimised” layout was driving me up the wall. The fans which come with it are quiet but aren’t great. Get some ML140 Pros if budget allows. For our next workstation systems, I will be investigating the Silverstone Seta H2 and Alta D1, particularly the latter.

10GbE NIC is a good idea; if your IT supports it, perhaps a dual-port card for teaming (especially if you get network storage which can support it) although with just one or two cards, CryoSPARC will not stream data fast enough to saturate 10GbE.

For a “budget” (with how much this will cost, that word makes me wince) workstation, I’d look at something like this:
AMD 9950X (can set 105W eco mode for significant heat reduction with minimal performance loss)
Thermalright PA120SE or Noctua D15
Asus ProArt X870E Creator (has 10GbE)
128GB or 192GB DDR5
1x 2TB NVMe (root)
2x 4TB NVMe (scratch, ZFS stripe)
4x 20TB HDD (RAIDZ1) (+2 spares)
1600W PSU (maybe Seasonic PRIME?)
1x or 2x 5090 Founders Edition
Fractal Design 7 XL (+ ML140 Pro fans) (yes, despite what I said above - I have no experience of the Silverstone cases I mentioned so will not recommend yet)

With a bit more budget, I’d switch out the AM5 parts for a Threadripper Pro 7995WX or 7975WX, 512GB RAM and relevant motherboard.

Definitely look toward large-scale NAS storage as well. Cryo-EM data eats those petabytes faster than you might expect. :wink:

5 Likes

Thank you for all your comments and suggestions! No need to apologize for long answers, I really appreciate all the information.

What I probably should have mentioned from the start is that we do have another workstation in the lab. It’s a few years old but still sufficient for our calculations. It has 2× Quadro RTX 5000 (16 GB VRAM), an AMD Threadripper 2950X, and 128 GB of RAM.

We want to add another “cheap” workstation to distribute the load. That’s also the reason for the 10 Gbit/s network card. While the local network does not fully support such transfer speeds, we were thinking about a direct point-to-point connection.

Regarding all your points concerning the hardware:
CPU: I am aware of Intel’s issues and the big scandal they faced regarding the stability of their CPUs. That’s why I initially looked at AMD, and of course, the 9950X was an obvious choice. But it costs around 20% more while being about 10% weaker compared to the i9-14900KF, according to general benchmarks. However, these are general benchmarks and may not translate directly into CryoSPARC/RELION processing speed, which is why I’m still a bit unsure.
The Threadripper Pro 7975WX costs 860% more than the i9-14900KF, which is far beyond our budget for this machine. At the end of the day, with a single GPU in this machine and very limited CryoSPARC Live usage, I don’t think we need a beast of a CPU, that’s why I didn’t look at Threadripper series.

GPU: Interesting suggestions regarding enterprise cards. In the same price range as the 4080S or 5080 (16 GB VRAM), we could get an RTX A4000 (16 GB) or an RTX 4000 ADA (20 GB). I initially looked at the 4000 ADA, mostly because of the slightly larger VRAM, while being only 20-30% more expensive. However, what really caught my attention was the memory bandwidth on these cards: 360 GB/s for the 4000 ADA vs. 960 GB/s for the 5080. That’s almost a threefold difference. From my understanding, this is quite an important metric?

Currently, we have no issues processing data on a single 16 GB card (Quadro RTX 5000), which is why I thought it would be better to go for a faster card, even if it still has only 16 GB VRAM.
It would be really interesting to see a comparison between consumer and enterprise cards with the same VRAM, such as the 4080S vs. A4000 (16 GB) or the 4090 vs. A5000 (24 GB).

Of course, I would love to go the 5090 route (or A5000 ADA), but that would mean a 2-fold increase in GPU cost…

RAM: The initial configuration includes 64 GB DDR5 RAM (6000 MHz, 2×32 GB), with two additional slots available on the motherboard in case we need to expand to 128 GB. We could go with 128 GB from the start, but with a single GPU, I thought I’d begin with 64 GB and upgrade if needed. Do you think this will be inevitable with a single 5080 GPU?

SSD Storage: A pair of 4 TB SSDs would indeed be great, but this motherboard has PCIe 5.0 NVMe support, with only one slot. That allows us to install the latest SSDs with 12,400 MB/s read + 11,800 MB/s write speeds, compared to the typical 7,450 MB/s read + 6,900 MB/s write speeds of PCIe 4.0 SSDs. I think we never needed more than 4TB of scratch for a given calculation, so I hope one 4TB drive will be enough.

Case and CPU Cooling: Thank you, @rbs_sci, for the great comments regarding the case and CPU coolers. The reason I want to go with a closed-loop liquid cooler is because of the case and HDD placement. The font mounting of HDD (between the front fans and the motherboard), while great for PSU space and cable management (and total number of HDDs) impacts the ventilation and the air flow. That’s why I wanted a 360 mm liquid cooler mounted at the top for the CPU.
I also looked into the Silverstone Seta H2 and Alta D1 cases, but unfortunately, we cannot get them from our usual providers.

Thank you again @Das, @Mark-A-Nakasone and @rbs_sci for your suggestions and comments! I really appreciate it!

2 Likes

good luck @Mat. I understand how hard it is financially$£€ these days. You still may find some deals with older parts, but not sure if it is worth the time. Regardless the minimum system requirements for CryoSparc should allow you to process most SPA data sets. https://www.exxactcorp.com/category/Deep-Learning-AI-Workstations?page=1 you are not too far off commercial options or workstations.

1 Like

Acknowledged. :slight_smile:

Bear in mind the KF has no iGPU, so the nVidia card will be pulling double duty for display and compute. That may change your calculus on that front as nothing worse than having a run crash because it’s fighting with X/Wayland for VRAM. Or having the screen lock up because the GPU is 100% loaded by something else…

16GB VRAM will hit a lot of limits - we got a new system loaded with A4000s just before CryoSPARC updated to 4.4, and a lot of tasks which ran with 4.3 and earlier would crash running out of memory with 4.4. I’d honestly recommend a minimum of 24GB of VRAM, most tasks in CryoSPARC seem to be optimised around that (particularly RBMC, 3D flex…)

I’d go 128GB from the start. RBMC, 3D variability, 3D flex and high-class-count 3D classification will use it.

Fair enough on the cooler; I’d set up a script which polls the pump every, oh, 30 seconds and issues a shutdown command if pump RPM is zero, though. How to do this depends on what brand and model you buy, though. Some report like it was a fan, others abstract and hide it from sensors without their (usually Windows only) software.

1 Like

Thank you for the link @Mark-A-Nakasone!
Good points @rbs_sci, thank you! Especially with lack of iGPU, maybe I should indeed go with AMD 9950X.

Good point for the fan as well. I was thinking about going with Corsair NAUTILUS 360 RS, would need to check if one can read the RPM from sensors.

Regarding 16GB VRAM, I should read more about it. I briefly saw that in 16GB now one should enable low-memory mode that will effectively run the old code. Does 24 GB VRAM allows to “escape” low-memory mode?

1 Like

Hi @Mat please see this extensive testing from @rbs_sci https://discuss.cryosparc.com/t/cryosparc-benefits-from-more-gpu-vram-or-processing-power/15696/12?u=mark-a-nakasone

I have never used less than 24GB of vRAM, but I also look at $nvtop and some logs, but rarely do I even get close to using all of it.

1 Like

Several jobs do not have a “low memory mode” option (local res jobs) and will crash on 16GB cards where NU refine was run successfully with Low memory mode. :frowning: CS devs are aware, I made a thread about it last year.

24GB is fine for most things; only really big boxes will cause issues on 24GB cards (currently - this advice also held true for 16GB cards pre-4.4!) :rofl:

1 Like

There are currently reports (again! De ja vu of the 4090 release…) of 5090s burning their 12VHPWR cables from a few enthusiast/overclocking sources (even without overclocking). It appears to be ATXv3 and ATXv3.1 standard related (3.1 is OK, 3.0 is not), so that is something else to watch out for.

The opinion appears to be swinging in the direction of the 12VHPWR connector standard being woefully underspecced for the demands of modern GPUs, and nVidia cutting corners on their cards by not including shunt resistors appropriately.

I hold my own opinions on the above, but I’m quite concerned about running any cards with a 12VHPWR connector 24/7 unattended and have thus far resisted purchasing any cards which use it.

I blanked on a minor point which might be of interest to you if you can get things cheap - the 2950X/2990WX only officially support 128GB of RAM, but depending on the motherboard they can support 256GB. I’m running 256GB at 3200MHz on an Asrock X399 Taichi with a 2950X on one system. It had 64GB, but I decided to see if it was possible to get 256GB working, as 32GB DIMMs were not available when X399 launched, and most manufacturers are terrible at updating QVL lists. And it worked. It’s not our most heavily loaded system, but it gets a workout in RELION a fair bit.

That said, you probably will want to stick with “known good” options/combinations to minimise potential headaches.