I’m incredibly skeptical that OpenAI is spinning up custom ASICs for improved inference performance, but they never thought of optimizing KV cache until a tiny Chinese lab did it? Give me a break.
I think the reason they advertise like that is purely money. Verizon can make $1000 convincing 1 in 1000 people to purchase a new iPhone. A tech startup can make millions by getting a new customer.