Am I missing something here. Not finding a major bug/vulnerability just means that maybe the code is really good, not that the model is not what is claimed?
Akamai CTO Robert Blumofe offers four tips for business leaders striving to foster AI fluency by empowering employees with the right tools and best use cases.
Fair points, but the reason everyone is amazed is that five years ago this was entirely impossible for computers irrespective of the competition format or rules.
It’s as-if we had learned whale song, and then within two years a whale had won a Nobel prize for their research in high pressure aquatic environments. You’d similarly get naysayers debating the finer points of what special advantage whales may have in that particular field, neglecting the stunned shock of the general population — “Whales are publishing research papers now!? Award winning papers at that!?”
And it’s very impressive that whales can write papers.
A computer system that can perform these tasks that were unthinkably complex a few years ago is quite impressive. That is a big win, and it can be celebrated. They don’t need to be celebrated as a “gold medalist” if they didn’t perform according to the same criteria as a gold-medalist.
The same argument works in the reverse: the computer was not given 18 years of calendar time to study. It doesn’t have the benefit of a meat brain with a billion years of evolution optimising it for efficient thought. Etc…
That the architecture of the machine mind is different to ours is the point.
If it was identical then nobody would be excited! That’s a high school student equipped with a biological brain.
That the computer used silicon, that it used parallel agents, that it used whatever it has in its programming is irrelevant except in the sense that these very differences make the achievement more amazing — not less.
It's a good point - IMO is about performance under some specific resource constraints, and those constraints don't make sense for AIs. But I wonder how far we are from an AI solving a well-studied unsolved math problem. That would be more of a decisive "quantum supremacy" type milestone.
> there will be a proposal at some point to actually have an AI math Olympiad where at the same time as the human contestants get the actual Olympiad problems, AI’s will also be given the same problems, the same time period and the outputs will have to be graded by the same judges, which means that it’ll have be written in natural language rather than formal language.[1]
Last month, Tao himself said that we can compare humans and AIs at IMO. He even said such AI didn't exist yet and AIs won't beat IMO in 2025. And now that AIs can compete with humans at IMO under the same conditions that Tao mentioned, suddenly it becomes an apples-to-oranges comparison?
As I understand, the LLM uses the techniques of searchformer - https://arxiv.org/abs/2402.14083. To do "slow thinking" doing a A* search using a transofrmer.
There was no answer to my last question which I think is the most important thing when considering if we are going to have another GFC this year or next year.
Does "GFC" stand for "global financial crisis" here? It seems implausible that the collapse of the LLM bubble will cause one; it might have a pretty dramatic impact on the markets, but it's unclear how it would cause the sort of systemic failure that we saw in the noughties.
TLDR: Don’t buy H100s. The market has flipped from shortage ($8/hr) to oversupplied ($2/hr), because of reserved compute resales, open model finetuning, and decline in new foundation model co’s. Rent instead.
At $2/hr, that's 2.8 years to RoI. And that's just for the GPU and not the other hardware you'll need to plug it into, and doesn't include the power, and also assumes you're using it 100% of the time. Really, you're probably looking at 3.5+ years to RoI.
