This is based on one of the smaller Qwen models, just like Cloudflare's Clef, Strands decider, and a plethora of others released in the last couple of weeks.
Kind of funny how much hype they can all get out of this, but Qwen really is the little engine that could. Great to see open weights (if not open source) driving the whole ecosystem like this though.
My biggest learning after some experiments - a BF16 (unquantized) Qwen beats a Q8 of double its size for decisions. I guess that’s the reason Kev switched to 4B BF16, from the original 8B version. Isn’t it interesting that quantization seems to mess with decision accuracy?
That’s fascinating, but not that surprising to me. We act like quantisation is free “Q8 is basically lossless” is often said in the local LLM community, but it really isn’t. The trade offs are worth it, personally, and the damage to coding ability seems low: decision model approaches are stricter though
FWIW, my tests used only ranking. I used Kev to drive E2E test runs using agent-device, where Kev had to decide the next action - like pressing a button, or scrolling down etc, to reach a certain goal (UI state). It went surprisingly well, but 4B BF16 was significantly better than 8B Q8. The latter made some wrong decisions in every run while the former was almost at Luna precision.
Plus, a lot of IP from a previous startup (that was acquired last year), Pi Labs, was used to help push the model to SOTA performance quickly. The startup was building scoring models long before Jev released
Surprised they aren't doing these sorts of one-off models with Microsoft Phi, which is intentionally smaller, but there's no reason Microsoft couldn't try to make a slightly larger Phi model with more capabilities...
I don't know if Fabio (who's leading work on this model line) would agree, but if I was going to start over I'd probably pick Gemma4 as the base rather than Qwen3.5. But it's not surprising to see a lot of Qwen competition, and I totally agree it's nice to see the work open.
I been using gemma 26b moe model as the system one variant for my own uses and its great and far smarter then any alternatives at around 200ms, bigger is better for these things as long as you dont need lower latencies, only caviat is calibration, if you need calibration better use JEV.
The insistence of naming it open weights as opposed to open source is getting ridiculous, and it's both irrelevant (i.e. no one cares in practice) and factually incorrect.
Weights are source in language models. Apache defines source as ""Source" form shall mean the preferred form for making modifications". That is precisely what's happening here. Everyone is using the preferred form for making modifications to these models (including the model creators themselves). A model is "created" at init time, and then "trained" by modifying the weights.
All these models are open source. What's not open sourced (with qwen et all) is the training code. So open source model, no training code. And that's ok. There are labs that release those as well. Apertus and Olmo series come with open source models, open source training and open datasets. Nemotron comes with open source models, open source training and some open datasets, while others are not published. And that's ok too.
The fact that you see all these models being modified (from AR completion models to "decision models") and re-released should be all the proof you need. That's what a license offers you. The right to inspect, run, modify and re-release a model. A license cannot (and never did) give you any other rights. OpEnWeIgHtS is silly.
Weights are source in the same way as any x86 binary is source.
You easily modify a x86 binary and change behaviour or examine the machine code instructions. You probably are not aware how easy it is to change the behaviour of a binary executable.
I mean I don't have a strong opinion but if I used the phrase open source models there'd be 5 comments going in the other direction.
I'm happy to have and be able to serve these models and see the ecosystem thrive. And lots of open innovation is outside of weights anyway as DeepSeek repeatedly shows.
You used the right term. I don't know what the parent commenter is on about.
Open source has historically meant you could download the source and build your own binary. The appropriate analogue here, IMHO, to the build->binary process is training->weights.
As a Siemens employee this is so sad to watch. Some of us have been pushing hard for more open source culture at the company and it's disheartening hitting setbacks like this.
I am very happy to see this, a continuation under the stewardship of rocky linux. I really hope it takes off and continues with force and velocity… I intend to help if I can.
My team runs open models for devs at our company, mostly on H200s, and I'd also say yes, if you want to always stay on the bleeding edge (and not use their model-specific images they publish before it lands in a release), then some amount of patching might be needed especially for new models.
The good thing is the models themselves are good enough to usually find the root cause if you give them read access to your deployment, logs and upstream issues/PRs to analyze.
If you do A/B deploys and E2E test them with popular harnesses (we do opencode/codex/claude), you'll catch most things. It'd be interesting to hear what the more nimble inference/neo-cloud providers do when they deploy models within days of them being released, as I know it definitely needs some patching.
But I think things have improved since the days when even chat templates/tool parsers were problematic, and their new flat model approach might help as well. I suspect some of the issues came from models inheriting config and parsers.
> if you want to always stay on the bleeding edge (and not use their model-specific images they publish before it lands in a release), then some amount of patching might be needed especially for new models.
Well, except this isn't exactly a bleeding edge architecture. DeepSeek-V4-Flash was released four months ago. I think it's not unreasonable to expect it to work well and be somewhat stable on mainstream hardware and not have critical bugs like e.g. garbage output?
Yes, agreed. I meant in general it's usually new models I'm seeing the biggest issues with. I hope the refactors on model arch/config will help with this so model changes have a smaller blast radius.
It's a lot of moving parts between reasoning parsers, tool parses, all kinds of MTP algorithms with different levels of support among popular etc. Even Kimi K3 saw more improvements in the latest release and it's essentially old news at this point.
On our side I've seen a lot of this garbled output in reasoning output but not in the output itself, though we did have to revert initially when we saw that a few versions ago.
Can I ask your setup? I’ve been tasked with taking a beefy machine (vague on details on purpose, the specs are legit) and trying to figure out how to run on-prem inference using open models. Gated in that I cannot use non-US models. I don’t think I have a handle on how to set it up for multiuser use without nuking TPS or context length. The current power users set and forget tasks overnight and expect results in the morning. Which… I have mixed feelings about, but that is the bar set by paying all the money for GitHub copilot enterprise.
We used to run GLM-5 class models but have now changed to smaller ones as we're able to serve more concurrent users with our limited hardware (DeepSeek-v4-Flash-0731, Qwen-3.8-27b). We run on the order of hundreds of parallel requests right now, Qwen with data parallelism and DSv4F with P/D disaggregation, but will probably continue to tweak this.
Qwen 3.8 27b in my experience is more than capable of churning out features overnight with the right tools (don't rely on its world knowledge, give it tools like playwright and github MCP for upstream context and search, and give it goals to work on). Bonus with another model like DSv4 as an adversarial reviewer.
This is incredibly silly. First I would try to convince whoever has decreed this to reconsider. The only SOTA open-weight models are non-US, and rather than being afraid of potential Chinese backdoors secretly planted in open weight models you should be afraid of LLMs simply just doing something dumb. To give you an analogy: no one has ever demonstrated a backdoor in a popular model which maliciously runs 'rm -Rf *'; plenty of people have encountered a model running 'rm -Rf *' by mistake. You need to protect yourself against the model doing something stupid anyway, and, paradoxically, a US open-weight model makes this more likely due to them being weaker.
If you need a precedent then you can cite Mistral[1] hosting GLM-5.2 on their platform, or plenty of US-based providers on Open Router hosting Chinese models.
Right now, the best US-based open-weight model (Inkling) has 42 points and is 975B parameters. The best Chinese open-weight model has 60 points. Also, note, Qwen3.8-27B on low thinking beats Inkling here and gets 43 points.
Of course, AA scores are definitely not bulletproof and there is a certain amount of benchmaxxing going on, but the point stands -- it is *incredibly* silly to limit yourself to US-only models, just because of an imagined threat that doesn't exist. Let me state this in stronger terms: whoever decided this for you is clueless. They might have the authority to override you, but it's your job to tell them that the only thing this achieves is essentially pointing a gun at your own foot and pulling the trigger.
I think I’m just going to find the best model I’m authorized to use (of a few of them) and get on with it. You’re much more passionate than I am about Chinese models, I don’t really care about the topic much one way or the other. I show up to work to get paid and go home. I use qwen at my house, it’s pretty great.
Not gonna make problems for myself at work that I don’t need to make, and I sure don’t have any interest in sticking my neck out for models that come out of a country that doesn’t even hide the fact that they are a technological adversary.
A part of me wishes the open source community would focus making research and industry-backed initiatives like the vLLM Semantic Router rock solid. Then I'd spend less time every month checking if this or that new model router has differentiating over vllm-sr :)
At least for open source inference, it seems like there's healthy competition centered around vllm/sglang, but 2026 seems to be for model routers what 2025 was for agent harnesses.
Yeah, I think also efforts like Docling and similar are showing that smaller, specialized models with the right tooling might be more effective (and efficient) at this than throwing everything at Opus.
They don't seem to release as much open-weights at Mistral as they used to though :)
Kind of funny how much hype they can all get out of this, but Qwen really is the little engine that could. Great to see open weights (if not open source) driving the whole ecosystem like this though.