> Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks.
A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext - maybe because of tradition, to make numbers more comparable? It's not measuring how well the model follows bf16 on agentic tasks.
I've been trying to check KLD recently for some quants of Qwen 3.8 27B, and the numbers are dramatically different, depending on which dataset you use. KLD computed on agentic traces is much higher, and top-1 % is way lower than if you compute it on chats or wiki text.
You look at a published number, and see "oh, nice, top1 is 99% - quant is different just in 1 token out of 100", but chances are it's computed on wiki, and on agentic / coding it can be 10 tokens out of 100.
Common intuition is that on agentic tasks errors compound, and that's why it degrades more than metrics show - but maybe the metrics themselves are also wrong, too optimistic.
> A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext
Thank you for calling this out. Using Wikipedia snippets for these is a terrible choice. I did a bunch of KL and other stats with the five Gemma 4 models, and the results were non-obvious. Anthropomorphizing:
Gemma 4 31B: "I guess we'll pretend I said this, but it's not me." (Baseline for stats)
Gemma 4 26B: "Dude, I'm certain I wouldn't have said this." (Bad KL)
Gemma 4 12B: "Umm, Me either!" (Similarly Bad KL)
Gemma 4 E4B: "I might say almost anything, this is fine." (Much better KL!!!)
Gemma 4 E2B: "I'm basically a toy. Let's play a game!" (Same KL as E4B)
Anyway, for comparing quantizations, it seems like the largest precision version should be given a one-shot prompt, and the result from that should be used as the corpus for the quantized versions.
> Anyway, for comparing quantizations, it seems like the largest precision version should be given a one-shot prompt, and the result from that should be used as the corpus for the quantized versions.
Way better than wikitext-- but tells you nothing about errors tending to compound or cancel out.
Like say a test shows that only one token in a 10,000 token test would be different. Sounds very close, ship it!-- but what if trajectories with that single different token guarantees failure because it sets in motion a cascade of differences that ultimately result in a final distribution that doesn't include the solution?
Pass. When articles keep mentioning models like DeepSeek R1, or Llama 3.1, or Qwen3 32B, it is a pretty robust indicator of AI slop. LLMs love to suggest DeepSeek R1, etc. - training data cut-off?
No person with real practical experience and real use cases will be using these ancient models as examples, when talking about local LLMs.
DeepSeek needs more RAM for weights, Qwen requires more compute.
Also, DeepSeek's KV cache requires less RAM than Qwen's. In concurrent situations (on servers) you load model weights once, but you have different context in each parallel session. So, it can also need less RAM than Qwen to serve, even if it's a larger model.
I have an internal automated benchmark, which roughly follows my workflow, and I've been testing various models on it, local and cloud. Qwen 3.8 27B did awesome. Its understanding is correct, research is better than e.g. glm's (and I like glm), and implementation is good and careful.
Qwen 3.8 27B doesn't look benchmaxxed. These "52 AA score" numbers feel real, which is surprising. I've been using it locally for a few days for other tasks as well. If not the speed, I'd be totally happy to use it as a daily driver instead of cloud models, it is that good.
---
(benchmark, to get an idea):
1. First, initial prompt which is not super precise - similar to how I'd write a task when talking e.g. to Opus. I'm describing an idea, and asking model to come up with some plan, and also to criticize the approach. Task is about implementing a particular pi extension. I'm checking if a model actually understands what I'm asking.
2. Then, as a follow-up, I ask to research alternative implementations, research UX of similar extensions, etc. It needs to do web searches, inspect open source codebases, read articles and papers, etc. I don't prompt to do this exactly, but I expect good models to figure out they need to do it.
3. Then, implementation.
Also, one finding: Q4 and Q8 seem to have very different behavior in this benchmark. Q4 produces 2-3x thinking in the end, and makes more turns - it seems it makes more mistakes, and needs effort to recover from them, while Q8 gets more things right in a first try. In the end, quality is roughly similar, but Q8 gets there much faster, especially the implementation (tried it several times). Could be a difference between concrete artifacts, or between runtimes, I don't know, but be careful - it seems the real-world experience with qwen 3.8 27B can be vastly different, depending on how it's set up.
Regarding DeepSeek 0731 vs Qwen 3.8 27B. On this benchmark, Qwen understand my intent better, it's better at research, and I also liked its implementation more. But: if you're more precise in what you ask, 0731 is also very good, and it's quite a lot faster on mac; raw speed is better, and it needs less thinking to get there. So, I'd say it's a tie in practice, both are awesome :)
I do have this experience. I've used Claude Code (with Opus mostly), and then switched to opencode (mostly with Kimi 2.6) for my personal projects; it's based on a couple months of use.
Claude Code is better. But Opencode + kimi 2.6 is workable, which is big. For bare code writing, if you know what exactly you want, most popular models are fine (deepseek, kimi, etc), it feels more or less the same as anthropic models.
At the same time, Opus seems to understand my intent way better than e.g. deepseek. I need to be much more precise with my prompts when using deepseek - it often goes in a wrong direction if I'm lazy. This results in a workflow which feels quite a lot different from Claude Code.
Kimi is in between - for me it brings back "lazy prompting" workflow, and I can trust its plans more than deepseek. It enables a workflow similar to Claude Code, it's workable, but it is a bit worse everywhere. Smaller context, a bit more errors, decisions are a bit worse, recommendations are a bit worse, debugging capabilities are a bit worse, etc.
On the usage side, $100 Claude plan is a great value actually. On paper, per-token kimi is way cheaper, but Claude subscriptions are heavily subsidized - you get much more tokens than $100 can buy you. So, in the end, opencode + kimi vs claude code could be of a similar cost, for similar usage patterns. Deepseek can be cheaper, and it has insanely cheap cached tokens, but experience may vary - depending on your habits, you may need to adjust how you work, coming from claude code.
I'd say for side projects something like $10 Opencode Go plan + $10 of extra DeepSeek v4 credits (e.g. on OpenRouter) can be very workable.
I wonder if they’re truly subsidised or if the API pricing is just massively inflated. Genuine doubt.
My CC stats show me using almost 300$ of Sonnet tokens on the 20$ plan. Is Anthropic willing to forgo 93% of the profit? A bit less than that but API is priced, say, 3x what it should be?
CC is great, but Sonnet (my main model) isn’t worth the API pricing. The cheap-but-good models arrive at similar results for much less (for context I’m using Aivo with CC).
Anthropic is making money from people who under-utilize their subscriptions, and presumably by sneaky throttling or not-sneaky throttling power users. Currently they are in an adoption race. Whether being first will actually let them "win" the market (and the market is a bit ill-defined) is unclear.
To my feeling, I'm getting usage of Opus (and Fable before the cut) that's greater than what I got from Sonnet last year. I reached $100 of usage when weekly was at 50%. This means, I could squeeze $800 worth of tokens for $20.
Anthropic has sent out a newsletter explaining they were more or less adding 50% (even 100%?) of quota to everyone, due to some great deal they made. That might be it. I do get lots more usage lately.
This is generally been my experience as well, but i think the main reason for claude code being better at understanding intent is their massive system prompt.
>At the same time, Opus seems to understand my intent way better than e.g. deepseek. I need to be much more precise with my prompts when using deepseek - it often goes in a wrong direction if I'm lazy. This results in a workflow which feels quite a lot different from Claude Code.
how much of that is Opus injecting prior conversations from memory?
Almost none of it, if you're using Claude Code. Until recently Claude only had the option of retaining memory across conversations for the desktop app.
I almost never use the desktop app, I have maybe 2-3 conversations over the last year that have nothing to do with my job. Opus (and now Fable) genuinely do seem to "understand" what you intend based off what you're explaining a lot better than other models I've tried.
Gemini gets close in some cases, but it falls over in the actual implementation sometimes. I haven't tried Kimi yet but MiMo isn't too shabby either.
I'm using Claude code + (a patched) litellm proxy + openrouter + Qwen 3.7 max/kimi k2.6/deepseek v4 pro. The only feature that doesn't work is webfetch and web search, which I've replaced with the ddg MCP. Memory, caching, and everything else works fine.
Qwen comes close to opus for planning but fable is clearly superior. Kimi and deepseek are pretty much indistinguishable from opus for coding if opus writes the plan.
I'm now testing out fable for research and planning and deepseek v4 flash for coding. I'm guessing results will be pretty similar to opus + deepseek v4 pro and costs should be lower overall.
reply