4. H100 is 21 month old architecture. MI300X is 7 months. Nvidia moved into new architecture every year pace. AMD is generation behind and must step up the pace. B100 comes out this year.
AMD is getting closer, but don't except them to catch Nvidia in not time.
It has fp8 support. Not sure whether fp8 on MI300x is supported by vLLM yet.
Also, many of these comparisons use vLLM for both setups, but for Nvidia you can and should use TensorRT-LLM which tends to do quite a bit better than vLLM at high loads.
I haven't ever seen a server with 8x H100 NVL 188GB. The H100 NVL has 94GB of VRAM but they sell them in pairs connected with NVLink, so I guess they sometimes market them as 188GB, but in fact it's two cards and a server usually has 4 pairs.
Less than that, we paid for ours in January and received it in March. The first batch had problems and we had to send them back, which took another 3 weeks. So, let's consider the start date closer to April.
Nvidia has multiple advantages.
1. Same software written in PyTorch works across all relatively modern Nvidia chips. With AMD and ROCm that's not the case.
2. M̶I̶3̶0̶0̶x̶ h̶a̶s̶ n̶o̶ F̶P̶8̶ s̶u̶p̶p̶o̶r̶t̶.
3. Comparisons against H100 are always like this:
The fair comparison would be against And that they never do.4. H100 is 21 month old architecture. MI300X is 7 months. Nvidia moved into new architecture every year pace. AMD is generation behind and must step up the pace. B100 comes out this year.
AMD is getting closer, but don't except them to catch Nvidia in not time.