yeah I'm also getting this for standard dev work, anything with kubernetes etc
completely nerfs the model because you can't let it do stuff over a few hours unattended because 90% it's going to switch to opus in first 10 minutes anyway
so seems best thing now is to have it write plans and then default to using opus for work anyway?
It is nerfed even just for plans. It switches to Opus in the first few minutes of me trying to build a plan to extract a component out of my larger codebase.
Trying to minimize privileged access codebase and was careful not to mention security explicitly.
A lot of the soft and hard sciences use hacky matplotlib code to produce results and visualisation, without being necessarily data science
From the bits I've seen, I'd take claude-generated code any time over that written by maths, physics, biology, linguistics people. Even though I've seen Claude make some super-big mistakes while doing data analysis I'd guess it's already more reliable than most academics trying to code.
Matplotlib? Ha! There are loads of academic fields where you still write data analyses by hand, one at a time, in Matlab, without proper version-control or libraries.
Conveniently, you can use published results as tests of equivalence, provide the ugly code as context, and regenerate it to your liking. I think the odds of such a regeneration introducing a bug that's within the usage domain but that dodges the golden tests are quite low... so long as you resist the urge to add features along the way.
I think presentation via software just isn't a lot of their strong suits. A lot of researchers' personal or research lab sites too are usually way out of date or just really badly presented from what I've seen. They could all do with some thinking about aesthetics and understandability more.
sure that's why we look at a mix of formal benchmarks, one longer analysis of a side-by-side, and various other people who we trust to form an opinion, all covered in the article - not intended to be a formal benchmark, there are enough of those.
yeah I don't think that's going to work - it would be kind of like "we're releasing model answers to all assignments but please only use them as a teaching aid and don't copy from them"
best to
a) adapt assignments so that agents are bad at producing solutions
b) have more scenarios where students have to do things in controlled environments. Universities managed to adapt to 'any solution you need is readily available online' so I don't think it will be that different to have several times a month/year where students have to go into a room with nothing but pencil and paper to prove what knowledge they have vs what they have the skills to access
Assignments the agent is bad at seems like a losing battle.
Just need to base the mark off the in person test, maybe keep 20-30% to encourage people to still do the assignments. Some will cheat but it will just be hurting them for the test.
20 years ago this was not unheard of. One exam we had to translate C code to assembly for one of the exercises, convert to numbers to IEEE754 representations and similar, both tasks where access to a laptop would make it possible to cheat. Also had to modify some small computer architecture diagrams if I recall correctly.
For the linear algebra written exam it didn’t work as if you learned to solve the 4 previous years exams, you could be sure most of it was familiar, so you could just prepare for a few standard exercises without really understanding the content.
Our advanced algorithm course used a bit of a combination, with a project take home exam (knapsack like optimization problem - competing for the fastest implementation) combined with a two hour written exam with multiple choice answers, but again only with books, pencil and paper to get to the right answer. This I think could work today, having both the opened ended project + some multiple choice with pencil/paper.
I did most of my CS class tests this way within the last year. It’s not that bad because prof doesn’t care about syntax so much (unless that’s what we’re testing on of course) and details, but wanting instead to make sure we understand broader concepts.
I agree it's not a complete solution. But as those don't exist as a society we are looking for a step function in the right direction. and IMO this is one such step. You may disagree that it's not a very large step, but I would argue it's still in the right direction therefore it is neccesary, especially in education space, and I'm happy to see someone publishing at attempt.
It's annoying when people post slop with no substance, but it's somehow worse when there's actual substance _hidden_ by the slop.
I'd probably much rather have read the prompt for this article than whatever this is.
Anyway, I use my `buzzoff.wtf` slop site for this kind of thing. Originally built so I could figure out what startups actually did behind all their buzzword landing pages, but now also to get a summary of articles like this to decide if it's worth reading or not
> “Only two things are important right now in life: fertility and deep learning,” the University of Pennsylvania economist Jesús Fernández-Villaverde said at the conclusion of a recent lecture. “Everything else is noise. Once you start thinking about these, it’s hard to start thinking about anything else.”
Not sure if I agree but that's quite a memorable quote
It's also a useless quote. Fertility requires a whole supply chain pyramid: food, housing, infrastructure, medical industry, alcohol... the same goes for the needs of deep learning: energy, rare earths, nuclear science,... together they cover almost all of the economy anyway, with the exception of luxury goods, but then again you need those to make yourself attractive. So yes, we need hoverboards to procreate and coffee and cape-shit to keep scientists entertained.
Maybe humanity will have more babies when AI takes over, capitalism implodes and we’re all farming and crafting things from scratch, since we’re forced out of the “economy”.
reply