> I don't know of any other mainstream platform with >1TB/s memory bandwidth.
I mean, RTX 5080 has nearly 1TB/s, 5090 has nearly 2TB/s. Maybe you are talking about CPUs / unified memory platforms? I agree nobody else does it better. But for LLMs, GPUs can still be significantly faster than even the most advanced Apple silicon on the planet. TTFT in particular is super inferior with Apple, for now.
That's probably also the reason Apple had to reluctantly give into Nvidia servers for the initial rollout of Siri AI, though they claim to use trusted computing extensions to reach an acceptable level of privacy. (I do not trust that nearly as much as the Apple Silicon nodes)
It'll be amazing five years or whatever down the line to see Apple reaching those figures. They seem to be heading in that direction lately.
I suppose it depends what models, but even heavily quantized the mid tier local models need >100GB of memory so unified memory platforms are the only option I would consider a mainstream option (doesn't require special order etc). If you want to do it in two slots that's two Blackwell RTX 6000s which is around $50k list just for the cards, and if you want to do it in four+ slots that's outside the realm of normal desktop PCs. Right now 2x DGX Spark, 2x Ryzen 395, or 1x Mac Studio are the three configurations that get to 256GB of reasonably fast memory that you can order online and plug in, for around $8k to $12k depending on the details.
I mean, RTX 5080 has nearly 1TB/s, 5090 has nearly 2TB/s. Maybe you are talking about CPUs / unified memory platforms? I agree nobody else does it better. But for LLMs, GPUs can still be significantly faster than even the most advanced Apple silicon on the planet. TTFT in particular is super inferior with Apple, for now.
That's probably also the reason Apple had to reluctantly give into Nvidia servers for the initial rollout of Siri AI, though they claim to use trusted computing extensions to reach an acceptable level of privacy. (I do not trust that nearly as much as the Apple Silicon nodes)
It'll be amazing five years or whatever down the line to see Apple reaching those figures. They seem to be heading in that direction lately.