Only if you were planning to scrape by. If you actually know the material (the thing the exam is supposed to determine), you should do fine, and any good or bad luck will tend to average out by the time you graduate. Besides, there really aren't any alternatives left. Completed homework had some weak-ish signal for a while, but now it doesn't.
It's been suggested elsewhere that the AA-Omniscience benchmark [1] is more representative than the rest. It puts K3.1 about 6 months behind the frontier and the new version of DS 4.1-flash about 12 months. That roughly lines up with my gut feeling as well, at least when it comes to the trickier problems. Still rather impressive, but not quite frontier.
I skimmed through the publicly available questions. The first programming-related one I saw has the answer in the question, not a great start :D [1]
Q: In C23, attributes are allowed in an unbraced (forward) union declaration only in the form “union [[...]] Tag;”. What single character must immediately follow the identifier in that declaration for the attributes to be permitted?
A: ;
Huh, this is an interesting hypothesis! It would make sense, to a degree. Part of "big model smell" to me is the model being able to pull a huge range of interesting algorithms, architectures, mathematics, and everything else that would come with greater world knowledge and having a much more sort of innate, like, full, well-rounded understanding of it instead of surface-level ICL pattern matching on web results.
Although even this one needs to be taken with a grain of salt. Fable/Opus still act very differently to Astra and Argon imo. But maybe it's just because I am used to these. I wouldn't clearly put them above each other though, although I still prefer anthropics models, but that could be simply out of comfort.
That definitely matches my experience, I have found Astra to be more capable than many other benchmarks show, have been very impressed with Argon for text, and generally like Sol. I find Haiku and Sonnet to be not worth my time as compared to the speed and cheapness of something like DeepSeek.
The only exception on the chart is DeepSeek - it scores exceptionally low there, which doesn't exactly track my experience. It usually does what you tell it to, and it's not exceptionally intelligent, but I do find it quite good at what it does (the bulk of code, scripting, etc). I know a lot of folks like it too ... curious if I'm just not exploring the simpler models enough?
The inspiration from the F35 seems to be quite literal. Spent a lot of time and money making vertical takeoff happen, but forgot to give the vehicle any useful capability once it's actually in the air.
This is false. It's being fielded in large quantities by militaries all over the world as a workhouse multirole fighter. Additionally, it's the best STOVL and SEAD aircraft ever produced.
I've had the same experience, and likewise I have no idea what to do.
Or rather, I do have ideas, but they do require substantial restructuring. E.g.:
1) Strict, in-person (no exceptions), electronics-free exams with written and oral portion.
2) Transition to guided independent learning: students are given written materials and recommended exercises to study. They are shown how to use AI assistants productively. Instead of lectures, there are scheduled discussions, where students can ask questions and listen to additional explanations from the lecturer. They can also seek feedback for completed exercises. However, none of this contributes to grades in any way.
With such an incentive realignment, I think it's possible that higher education could be salvaged. But... apart from general institutional inertia, there is also the political angle standing in the way. Everybody needs a degree in the "developed economy", and the above changes are basically the opposite of the nobody left behind policy that enables half the population to attain one...
I have stopped reading Ed Zitron quite a while ago because of his long-winded and polemic style. But counting his failed predictions is maybe not that useful, in the sense that his core thesis is really just the AI Bubble. While it hasn't burst, all of his predictions remain wrong; when (if) it does, he'll have been right. Trying to guess the exact time or failure mode (whether it's open models or corporate sticker shock or a bond crisis) is a fool's errand when there are so many powerful actors all-in on keeping up appearances. Exposing the financial and corporate shenanigans of the AI world is nonetheless useful. I just wish someone would do it in a more level-headed way.
Note to self and others who have read Richard Preston's "Hot Zone": The 1989 Reston (US) incident was NOT caused by a single-base mutation of Ebola, but by a distinct monkey virus. Preston seems to have made that up to dramatize events.
I read that book when I was probably way too young for it, and that incident made me spend a lot of time staring at the ceiling in the middle of the night. I'll have to dig into it more!
Cheating and overfitting, as discussed in the article, are the most obvious problems with benchmarking LLMs. But there is also the aspect that, at least for closed models, the tokens still have to be sent to the provider's servers for inference. This makes the holdout set not as held out as it may appear. OpenAI and Antropic probably don't care about your private set of regex benchmarks, but for the headline "closed" benchmarks, I'd be surprised if they haven't collected a nice representative set of "holdout" problems to be examined at leisure.
... which is entirely unsurprising given that exhaled air is about 50.000 ppm CO2 and can vary by several 10.000s depending on depth and rate of breathing. I actually consider the recent wave of findings that CO2 levels as low as 500-1000 ppm measurably affect cognitive performance and well-being to be a great example of how you can prove literally anything with statistics and a sufficiently small sample size.
reply