Yeah, DeepSeek 4 Flash on vLLM has been an adventure indeed. It finally stabilized for me on 2 x H200 using a commit a few days before 0.28, so this release should be good for you.
Yes, I mentioned the setup, but on vllm you can only use TP with speculative decoding or pipeline parallelism without, so there's tradeoff to both.
I gave general numbers of what I'm getting above, the performance ratios seemed similar regardless of setup (eg. getting a AWQ-in4 quant on a single GPU vs PP without speculative decoding vs TP with speculative decoding).
Overall single GPU is fastest, and TP+speculative decoding is still faster than PP, but for fp8 models you need dual GPUs whether you want it or not.
I also like that the article uses whatever system emoji you have, so everything is just showing apple emoji in text for me. All I see are a few 3D video renders of theirs.
I think if you had something like a theoretical used/refurb 2U rackmount server with two older multi core CPUs, 768GB of RAM, you would see faster performance loading a Q6 or Q8 GGUF of GLM5.2 into a freshly-compiled latest copy of llama-server, with the "no-mmap" option turned on to intentionally load the whole thing into RAM at the time the llama-server daemon launches.
If you want a CPU-only machine with 512GB to 1024GB of RAM, despite extreme cost rises, there are still some great options out there from companies selling ex-lease stuff that's 3, 4, 5 years old. It'll be loud as hell under full CPU load when running inference, so if you plan to use it at home, put it in your garage or basement or laundry room or somewhere similar on the far end of a network cable.
The software that OP has published appears to be specifically designed to hold only the active parameters in RAM (<100GB) and read content off local NVME SSD as needed on the fly. All that NVME SSD read wouldn't be necessary if you can hold the model in RAM, even in the absence of any GPUs.
Actually not that crazy of a spread. E.g. I have 48 GB + 32 GB in my gaming PC because if you go beyond 48 GB you start having to trade off more and more performance to keep the memory controller from falling over, so you really have to have a good reason to want to load more. Server platforms, like Epyc, it tends not to matter as much because you have so many channels for bandwidth and a beefier memory controller to handle them. Then on the VRAM side it's more about what makes sense for the GPU and how you plan on using it there (games or AI or modeling or whatever), and for a lot of cases the 5090 is just a good card to get for one reason or another (it just has a ton of compute + bandwidth for a consumer part).
DRAM chips aren't always manufactured in power of two sizes. It's been common for years to have non power of two capacities for LPDDR used in phones, and has started to show up in other DRAM types with the current generation standards: DDR5 for desktops/servers and GDDR7 for GPUs. That's how there have been 24GB single-rank DIMMs and 48GB dual-rank DIMMs for desktops and 96GB RDIMMs for servers for a few years, and how a mobile RTX 5090 has 24GB VRAM vs mobile RTX 5080 having only 16GB VRAM despite both GPUs being different bins of the same silicon and both configurations using a 256-bit memory bus.
Not that simple. 4 dimms were getting higher clocks on 2 CCD Ryzen models (12 & 16 cores) compared to those with one CCD. Motherboard topology is a factor too.
But there is no single configuration where having 4 DIMMs populated gives higher speeds than when 4 DIMMs are populated on the same configuration. This is because while the higher end parts tend to have the higher binned components they still inly have 1 shared memory die between the CCD and the motherboard topology is either it has 4 slots or it doesn't, but no matter how they are ran it's still better to only use 1 rank of each channel.
More capacity is also harder to drive, even on the same number of channels, but needing to go from 2 to 4 channels is also a (bigger) drag.
You can go up to 64 GB per DIMM on the current consumer offerings (max of 256 GB total across 4 DIMMs). So you could could 128 GB over 2 DIMMs, but it's still going to perform worse than 2x24 GB or 2x16 GB.
It's fine for dense models where you need them in VRAM, less so for MoE where you're offloading layers to ram. But 32/32 is pretty good for both in the popular ~30b range right now.
My money's on whatever models qwen does release edging ahead. Probably not by much, but I reckon they'll be better coders just because that's where qwen's edge over gemma has always been. Plus after having seen this land they'll probably tack on a couple of epochs just to be sure.
I just made a new installer for Discourse on CharmRuby, now I gotta check this out and see if porting is feasible. Hopefully this reduces the app size, that is quite large with CharmRuby
