I've read the transformers paper, it is still pretty simple math. And the architecture isn't particularly interesting from a theoretical sense, the concepts are basic - we're not talking Gauss, Newton or Euler here, we're talking 2-10% of software engineers [0]. The limit is most software engineers don't have enough hardware to test whether their ideas work or not.
And it is actually the same matrix multiplication people might learn at high school. The standard definition of matrix multiplication is generally accepted.
[0] We'd expect the figuring-things-out process to be faster for more cluey people, so the people who actually pick all this stuff up first probably are geniuses. But a more normal engineer with the hardware access would still figure it out independently sooner or later. It is just too neatly on the beaten path where we'd expect to be finding optimisations.
My point is that a discovery always seems simple and inevitable once it has happened. Newton's laws are pretty simple, all things considered, but it still took most of human history for those to be discovered.
An average high school student doing physics today could derive Newton's laws. That doesn't mean that that student put in the same situation as Newton, would have been capable of discovering newton's laws.
Newton had to invent calculus before his laws made sense. There isn't anything equivalent to that in modern machine learning. It uses concepts that are centuries old except for maybe a few stats techniques that might be around WWII era since there was a big stats push around then. Plus back-propagation which is a pretty big deal, but it is hardly new to the boom we're seeing right now. It is an idea that was sitting around for a while before hardware changes made it effective.
And it is actually the same matrix multiplication people might learn at high school. The standard definition of matrix multiplication is generally accepted.
[0] We'd expect the figuring-things-out process to be faster for more cluey people, so the people who actually pick all this stuff up first probably are geniuses. But a more normal engineer with the hardware access would still figure it out independently sooner or later. It is just too neatly on the beaten path where we'd expect to be finding optimisations.