AI server improvements

I wrote a few months ago about my new AI server named "Six" (from Battlestar Galatica). I have seen added two more GPUS (Nvidia RTX 6000 Pro Workstation) and put them on a real server board.

Current Specs

Xeon 658X
128G DDR5 6400
4x Nvidia RTX 6000 Pro (Workstation)
Silverstone 2500W PSU

After a lot of tweaking, and a new inference engine I am getting amazing performance.

Using llm inference bench, I am getting near 300 t/sec decode and 18.5K prefill running Deepseek V4.1 Flash (full precision). Under actual use, I'm seeing speculative decoding hitting as high as 700t/sec for a single stream. With 32 agents I am seeing 1632t/sec before speculative decoding.

This is a massive improvement from when I first ran Deepseek V4.1 Flash (~110 decode and 10K prefill). The main reason for the improvement is the software tweaks to vllm and now a custom inference engine that is more optimized to this hardware.

I was previously running two cards on PCI 4 x8 as I didn't have enough PCI lanes on a standard gaming board. By switching to the Xeon 658X, I now have 128 lanes which is plenty to run 4 cards a PCI 5 x16.

One issue I ran into is the Silverstone PSU, it is rated at 2500W but will only do 1650W on 120v. It also makes a lot of clicky sounds when the gpus are working. I plan to get a power station that I can charge at 120v and send 240v to the server which will eliminate the clicky sounds. I am told the PSU doesn't make noise at 240v and usually doesn't at 120v but the combination of my components do.

You will notice the GPUS are not plugged directly into the motherboard and are using risers typically found in gpu mining systems. These are high quality PCI 5 risers and do not slow it down or introduce pci errors provided you don't go longer than 40cm that I have here. The four gpus are sitting behind 4 120mm Arctic PST 3000 rpm fans. Typical temps are 55c to 71c while under 100% load, well below the official 92c threshold.

I would have loved to do water cooling, but I really dread tearing apart $16,000 gpus and surrounding them with water in tubes that will likely leak at some point in it's lifespan. The temps I am getting are very good so I saw no need to go that route.

I did have to go with a miner rig open air case which I wasn't thrilled about. The rig supports two PSU but I had to drill holes to support a EBB motherboard and take up space where a second PSU would go. I also had to remount the fans on the opposite side as the recommended layout caused the riser cables to be twisted. This rig supports two rows of GPUS but the cpu cooler eats into the first row and I was only able to use the top row. Considering I only plan 4-6 GPUS, this isn't a problem.

I also ran into some issues going Intel (aka Shitel). I am a big AMD fan for motherboard/cpu, but I went with Intel as friend had good success with it. It wasn't until I got things up and running I ran into issues and realized the limitation and had a lot or regrets not going threadripper as I originally planned. Intel is limited to around 45gb/s throughput for gpu to gpu communication and has about 1.51us latency. AMD however we are seeing 56.4gb/s peaks and 0.91us latency. Significantly lower, but after a lot of testing, we have been getting good results on Intel. I would never recommend going the Intel route building a high end AI box though.

---