Everyone says "those autoregressive transformer LLMs are obviously flawed", and then fails to come up with anything that outperforms them.

I'm not too bullish on architectural gains. There are efficiencies to be had, but far closer to "+5% a year" than "+5000% in a single breakthrough".

You can try to build a novel AI architecture, at a small scale. Just be ready. This field will kick your teeth in. ML doesn't like grand ideas and isn't kind to high aspirations.

Physics is obviously incomplete and yet nobody can solve quantum gravity. Being obviously flawed doesn't mean the solution is obvious. That's the whole problem.

I think in this case, people tend to underrate just how capable and flexible the basic LLM architecture is. And, also, underrate how many gains are there in better training vs better architecture.

Not obvious but the brain manages to think in ways LLMs don't really and the design is presumably of fairly finite complexity to be encoded in DNA.

Most people are not ML researchers. Most of the AI industry is not AI researchers. Most of the AI spending is not going to AI researchers.

AI researchers came up with an architectural improvement that made a lot of previously impossible stuff barely possible. Then, industry ran with it. Scaling that particular trick to the limits by throwing as much raw compute and data at it as humanly possible.

You don't need to be an AI expert to know that there are probably more advances to be had and that funding foundational research is the way to get them.

The results from "funding foundational research" are, too, middling at best.

It's not certain if something like JEPA would ever reach production grade AI models.