^^ this is huge. I was looking at CPU's for a ML recently and Intel is out of the question. Chips with enough lanes to hit full speed on 4 GPUs + SSD cost twice as much as AMD.
This may even apply to high end gaming machines. Soon as you have 2 SSD's or video cards you will exceed link budget on most Intel CPU's and everything slows down.
It also happens to routers. 10 gig NICs plus attached SSD storage, over link budget again.
Intel's stupid market segmentation is biting them in the rear
What? Why do you think the Intel 100GbE NIC is good?
We've been quite happy with Mellanox and Chelsio 100GbE NICs. The latest from each can do in-line HW TLS offload, which is a killer feature for us. No Intel NIC can do that.
IMHO the last good Intel NIC was the 10GbE "ixgbe" NIC. The design of the NIC was so tight as to be almost beautiful.
Recent 40GbE (and 10GbE based on the 40GbE chipset), and the new 100GbE NIC have the feel of being designed by a committee with endless features of questionable value stuffed in and consuming power and chip area.
Primarily the state of its Linux driver, and rock solid support in VyOS (derived from Debian stable). For a router, TLS offload isn't so much of a consideration when the system isn't doing anything at layers 4-7 in the OSI model.
If I had to make a perhaps overly broad generalization, I see more Chelsio and Mellanox NICs used in end point servers, and more Intel used in DIY whitebox network equipment.
Intel's E810 based NICs require only a PCIe3.1x16 slot. 16 lanes will accommodate the 100GbE port just fine. Theoretical PCIe throughput for 16 PCI lanes is around 252Gbps. The 800 NIC chipset is just four 25Gb Ethernet lanes stitched together. PCIe4 won't help this NIC much.
Your throughput calculation is based on a single port 100GbE NIC working okay at full duplex bidirectional line rate in a PCI-E 3.0 x16 slot, which is true. Let's say that we budget 220-230Gbps of throughput per optical transceiver, such as if you were to max out a 100GbE link both directions with iperf3 for testing. But the intel X820 card has two ports on it, so your bandwidth needs are going to be in the range of 500 Gbps.
also, cumulative number of pci-e 3.0 or 4.0 lanes in a system is a big consideration if you want to have, for instance, four dual-port 100GbE NICs all talking to one CPU. Or some mixture like three dual-port 100GbE NICs + one or two 4-port 10GbE NICs.
Where the lower end of the Intel server CPU offerings really falls flat is not having anything close to 64 or 128 PCIE lanes at a reasonable price.
The maximum throughput of Intel's Columbiaville NICs is 100G max. When the 2nd port is used the total bandwidth is split in half, so there will be two 50G links. The 2nd port is usually in stand-by mode when the 1st port is used as a 100G link. So, two active ports on a Intel 100G NIC does not result in 200G throughput.
I don't understand why you are combining tx and RX. PCI, fiber, and the switch/nic are all full duplex, so you don't need to add the two directions together. A dual port nic, even with pcie4, is unlikely to be able to hit 200G ( single direction) unless it's using very large packet sizes.
Since you're limited by the 100G negotiated speed, you nic will never send more than 100G with iperf on a single port.
This may even apply to high end gaming machines. Soon as you have 2 SSD's or video cards you will exceed link budget on most Intel CPU's and everything slows down.
It also happens to routers. 10 gig NICs plus attached SSD storage, over link budget again.
Intel's stupid market segmentation is biting them in the rear