Especially because good questions are more than simply asking about factoid X in Book Y. Good questions test if you really understood something and can transfer knowledge.
Coming up with such questions is not easy, and just because you did well on a test or "kept up with the field a bit" does not mean you can ASK such questions very well.
I agree 100%. If you open a PR, you need to add value somehow - having thought about the problem, understanding the code base deeply or just your time by thinking about a solution you can't otherwise get easily.
The problem is that the last part changed drastically with LLMs. You can get a lot of solutions easily by prompting an LLM. But that also means you don't need outside PRs for that anymore, and just prompt the LLM yourself.
That means two things in my opinion:
1) People haven't internalized that yet. For a lot of people, a functioning solution to a problem with often quite a bit of source code is equivalent to "adding value" - because in the past it was. It meant you thought about the problem, understood it and found a solution. But it was that thinking that added the value.
2) The minimum threshold for adding this "quality" rises quite a bit. A lot of problems can now be solved very fast with an LLM by someone who understands the code base well enough. You don't need a PR for that.
I suspect, that means in the future you could both get and need less contributions to OSS projects in general.
> But that also means you don't need outside PRs for that anymore, and just prompt the LLM yourself.
No, I don't think that is true. Anyone who used LLMs in greater degree (e.g. at work) knows for a fact that the output and therefore the end result work heavily depends on the human involved, and their thought process. Of course for anything that isn't completely trivial.
If this means that the signal of valued contribution swings from "PRs merged" to "Issues raised with PRs merged" then I'm all for it.
In LLM-friendly repositories it saves everyone's time if the maintainers are the ones steering the machines, and in LLM-unfriendly ones it saves everyone's time by not having generated PRs thrown at the maintainers.
A maintainer literally can't just prompt everything themselves since there are only 24 hours in a day. Open source thrives on all the different use cases the original author never personally ran into.
> Open source thrives on all the different use cases the original author never personally ran into.
That's what issue trackers are meant to capture.
I could see maintainers wanting to take sole control over the code, and rather than swatting down PRs they can just guide AI to fix reported issues themselves, and merge what they approve of. Cut out the middle men.
The maintainer doesn't have to prompt the feature. The maintainer needs your prompt and the discussion around the prompt. He doesn't need your code. In fact, I would say that for open source projects, the prompt should go through the project's own software factory.
Is that a bad thing? MANY of these projects ban PRs made with LLMs, without explicitely mentioning being against AI in general. Since AI drastically increases how much code you can write, they might not need LLM contributions from outside - they can just prompt themselves.
In that case, only getting a few, hand written PRs could mean getting fewer, but higher quality contributers.
Other projects might not care about getting many contributers, many stars or being widely used. It might simply be a small project someone wants to code by hand. That's absolutely valid, many people still have fun coding by hand.
The truth is just that there are a lot of slop PRs currently, where ppl just sicced an LLM on a problem, did not validate any findings, did not check the code, did not read the output and just acted as a meat proxy. In many cases, these bans on LLM PRs are simply a reaction to this flood of PRs. After all, as I said above - everyone can do those prompts themselves, they don't need a PR for that.
I feel like there are a few dishonest points made in your comment.
First, the way you phrased your argument makes it sound like it benchmaxxing is mostl a problem from the chinese models. As recent articles seem to have shown, it's smaller models, closed and open weights, which are benchmaxxed. That has nothing to do with the country they're coming from.
Second, I feel like your argument about chinese models being far behind is not quite relevant to the argument and was mostly added as a jab against these models.
Third, arguing against publishing the benchmarks is not the same as arguing FOR using private benchmarks. You don't have to use benchmarks at all, especially when models are so hard to quantify anyway. Simply shrugging and saying "well, no way around that" is not a solution. In addition, there are other, better ways to prevent benchmaxxing than simply "not publishing your solution". The recent maxxbench benchmark has shown one example.
Fourth, you don't need to provide a better alternative to criticize something. Criticism is valid even if you don't have a better alternative yourself
And lastly, I actually feel like there IS a better alternative than AA - not using aggregate score at all. If you need a model for coding, look at the individual benchmarks themselves for a / very / rough overview of how a models is doing. Trying to boil down so many different areas and capabilities into a single score, which is then weighted by little more than subjective vibes from the people behind AA is bound to be too little information and too much information loss. This, I personally feel, seems to mostly exist as a meaningless leaderboard for marketing purposes.
A lot of companies ARE forced to use API billing though, as Anthropic only allows up to 150 users to use a sub. And companies are Anthropics biggest revenue stream - criticizing the cost per task of their models is a valid criticism.
Wouldn't benchmark exactly show that the open models are only a few months behind the Anthropic ones? GLM 5.3 sits a few points behind Opus 5, and these models came out within about a month. Opus 5 was July 24th and GLM 5.3 was August 14th.
Obviously these models are benchmaxxed, and it's frustrating to see not only so much attention on these benchmarks, from both users and the labs, but there is a difference between "The models are benchmaxxed" and "that means the open models are much worse than the closed ones"
A lot of users here seem to draw a correlation between these two, but what the article shows mostly is that the smaller models are more benchmaxxed than the bigger ones. That includes both open weights and closed weight models, as Luna seems to be among the worst benchmaxxed models. It's just that the open weights models are smaller than the closed weights ones.
Obviously though, every benchmark has to be taken with a grain of salt, including this one. But we have to be careful to not cherry pick the conclusions that prove our pre-existing notions
Academically a - very - interesting article, which confirms my ongoing suspicion of LLM Benchmarks in general ( probably why I like it so much). Another thing that caught my eye is DeepSWE having a bunch of bugs in them. The more I learn about these benchmarks the more sceptical I become. I would expect such a prolific benchmark to have these bugs ironed out by now.
The practical takeaway, however, seems to be an interesting "don't use max mode". The passage about deepseeks weird API behavior dropping to pretty much zero the moment you used high says a lot.
Also, personally, even though Opus is clearly so much ahead of the Open Source models after the changes, I think the adjusted score is still very much competitive. I see other commenters using this as proof to justify why their opinions of the open source models - but a drop from 74% to ~50% is, while a big drop, still about a ~50% pass rate afterwards. That's far from useless for coding, especially since DeepSWE seems to be such a difficult benchmark. It simply seems to more accurately reflect the capabilities of these models.
All in all, I'd love to see more of such articles, and will probably keep an eye on this one - if they continue "benchmarking" these models this way.
AI Labs actually. They would love the perfect benchmar representing real usecase. I'd even dare and guess that they don't share much publically because it is a huge competitive advantage.
It is "a" problem, but honestly not a huge one. Chasing benchmarks is an integral part of AI research, and it is how we have gotten this far, which is pretty damn far. Discovering the ways in which benchmarks fail is in itself an interesting process, and the amount of interesting new benchmarks has grown with the capabilities of the models as a result. The frontier labs might be a bit selective about which benchmarks they publish as marketing, but overall they have so far been reasonably diligent about avoiding outright cheating. Most of the truly benchmaxed models are not on the frontier (eg. see muse spark performance dropping off a cliff in the article).
This is a scenario where the interest of the users and the labs actually align perfectly, since labs that have people actually paying for their product want benchmarks to align as close as possible to real life performance. Otherwise they can't actually easily tell whether the new model they are developing is any good or not, and so risk wasting literal billions of dollars on training runs for a product that people won't end up paying for. A failed major training run is an absolute disaster at the frontier, GPT-4.5 for instance was an almost existential failure for OpenAI at the time.
Case in point, the most popular coding model, Opus 5.5, also just happens to be the only one that avoided the benchmaxing impulse in this article. This is the result of years of Anthropic working very hard to avoid it.
That’s missing the point. Frontier models get shittier in certain ways with each release, due to benchmaxxing, and there is nothing we can do about that.
Very much disagree, to me it clearly shows the gap between Open Source and the frontier. You can not do real code work with open source models right now.
Especially since this benchmark is still mostly DeepSWE, and the only reason the open source models are doing decent is because much of the information hasn't changed. The open source models have been benchmaxxed to death.
> Very much disagree, to me it clearly shows the gap between Open Source and the frontier.
Yes, this is true.
> You can not do real code work with open source models right now.
This is false, or you're using "real" as a loophole. You can absolutely do real coding work with open source models, right now, right this second.
I'm doing so today using Qwen 3.8 Flash Next running locally, and GLM 5.3 Flash hosted.
We could do real work with Opus 4.5, 4.6. These models (for coding) are more than good enough to do work that Opus 4.6 could do, and in some ways are superior.
This DEFINITELY squares with my anecdata that Gemini 3.8 Flash is not even remotely close to Opus 5.5 on the types of problems DeepSWE CLAIMS to be benchmarking.
Astra does seem close.
I'd be curious to see Argon on there, since my experience is that Argon is also significantly behind Opus 5.5 and Astra despite the benchmarks (obviously much closer than Flash, though).
I've been meaning to take a closer look at what's behind the curtain of DeepSWE - since I built what I thought should be the easiest possible coding benchmark - and the models I tested it on scored roughly the same as they do on DeepSWE... Which is supposed to be hard...
We recently benchmarked our AI code review agent (just pi with some custom prompts based on info from our context layer). Previously we used Gemini flash 3.8 as it was capable and affordable. However, haiku 5.5 just blows it out of the water. It's more than 10x cheaper, 2x faster and finds 3x as many real issues.
If you mean the `alias ssh="kitty +kitten ssh"` then no, it's not fixed at all.
For example pikvm exists with read-only mounted filesystem. Having to dive into the details to figure out that the issue is kitty trying to write a file to make some obscure terminal features work doesn't strike me as good design.
`alias ssh="TERM=xterm-256color ssh"` fixes it but isn't mentioned in the docs.
Yep, i switched to kitty a few months ago and Claude dug around and found the solution external to the docs. The developer is very opinionated about how he thinks you should use Kitty.
Doesn't sound like it would work if connecting to a locked down server. However, the problem OP described is easily solved, for ghostty and kitty, by a single line in your local ssh config.
I believe I got rid of most of the issues with some .ssh setting which simply uses the normal xterm I believe. I can't check right at this moment, but apparently, simply using xterm and not the xterm-xkitty or what it's called does not actually make much problems when ssh-ing into a machine - not in my case at least.
It was a setting in some ~/.ssh config file though I believe, not a kitty setting.
Irregardless of wether or not Mistral Large 4 is competitive or not, I do not trust the AA numbers anymore, for various reasons. Them changing their numbers around so much recently brings the entire validity of this in question, not to mention if such a combined number is even important at all. The combined "intelligence" score of AA includes areas I do not care about or need at all.
Agree. They had just fiddled with the formula a few weeks prior to this, too. There seems to be an effort to get a certain result from the model rankings.
Exactly, these general scores are pointless. Let's say the model is amazing on A but crap on B and C. It gets low score, but you still would benefit greatly if you need it for A.
It's gonna be interesting to see the final version, since this is a preview model still. I'm not completely sure as to how this works, but from what I gathered it will be further trained during this period, so it might improve a bit.
Since cost is so much higher than deepseek though it really seems like the biggest reason to use it would be "in the mistral subscription" for personal use or "because you need to abide by european data privacy laws" for companies.
Mostly I'm happy they're still in competition, as I mentioned already elsewhere.
Are they still in competition? Or are they a government sanctioned monopoly? I’m not sure that will be competitive. There are no market forces to do so but pride.
Just the lack of capital, lack of electricity, lack of datacenters, lack of gpus. Of course to navigate all that you have to fight endless regulations in the EU.
No one prevented other people from running their own AT&T phone company in the US. Except the lack of capital, and endless regulations with running telephone polls across the US.
Google even tried more recently to run their own fiber optics and gave up in the end because of entrenched regulations making it impossible to challenge the modern monopolies internet providers have.
Lack of capital, datacentres and GPUs being the barrier to competition in European AI giants is the exact opposite of a "government sanctioned monopoly"...
That is like saying AT&T didn't have a government sanctioned monopoly, even though the primary thing stopping datacenters in the EU is regulatory hurdles.
Mistral isn't even close to the most used AI services in the EU, there are very few industries where EU sourced/hosted models are required (mostly ones where the US also either mandates US models or is seeking to...) and the potential behaviour of the US government (rational and irrational) is at least as big a contributor to European companies being cautious about dependency on US cloud models as EU regulation. Regulatory hurdles exist to building datacentres in the US and basically everywhere else, and don't favour Mistral over any other company, and European states and the EU are busy funding [notional] competitors
It is not, even really stretching the definition, a state sanctioned monopoly.
France is the last country to lack electricity, the nuclear parc is underexploited despite being World's first exporter with around 8 EPR by year last time I checked.
A lot of national defense stuff is a de facto "government sanctioned monopoly" and this is certainly the same flavor. Market forces are different if the industry is perceived to be vital to continued sovereignty.
Actually, with Trump in the Office, "AI/Data Sovereignty" is quite a strong market forces. From Europe point-of-view, any US company is a potential risk (hello Starlink)
I mean, the model architecture is basically that of GLM 5.3 with slightly more parameters and slightly more active parameters, so it should probably be able to beat GLM 5.3 enough post-training.
Coming up with such questions is not easy, and just because you did well on a test or "kept up with the field a bit" does not mean you can ASK such questions very well.
reply