Not sure if I qualify for the "day to day work" part, but I have an inference PC with an RTX 3060 12GB and 32GB DDR4. I use it to run Qwen3.6-35B-A3B and Gemma4-26B-A4B with llama.cpp. Just in chat mode.
Got the same thing a few days ago when registering a .com domain. A .online was offered for free. I’ve, unfortunately, been trained to be wary of gifts, so I removed the "free" offer, especially because I knew it can’t be free forever. Plus had no interest in an .online domain.
Just the other day I was thinking why does the American Osteopathic College of Radiology website link to a listitcle about Halloween constumes on a website with 10 pageviews/week? Google Analytics was showing two visits even after I used several filters to weed out most of the uninteresting traffic. Checking the logs showed that it was clearly referrer spam.
A few days earlier, I got an alert that the same aocr.org was possibly hotlinking some image. Again, checking the logs on my image server showed that it was just some bot crawling images and using the aforementioned website as referrer.
I was wondering the same thing. Maybe it’s a licensing issue. Or the models are too big and there isn’t enough demand. But size is not a solid argument because they also don’t support Qwen3.6 27B and 35B-A3B, especially since they just added Gemma 31B and 26B-A4B.
I just hit that error a few minutes ago. I build my llama.cpp from source because I use CUDA on Linux. So I made the mistake of trying to run Gemma4 on an older version I had and I got the same error. It’s possible brew installs an older version which doens’t support Gemma4 yet.
And that's exactly why llama.cpp is not usable by casual users. They follow the "move fast and break things" model. With ollama, you just have to make sure you're getting/building the latest version.
Its not possible to run the latest model architectures without 'moving fast'. The only thing broken here is that they are trying to use an old version with a new model.
I'm a bit unsure what that has to do with someone running an outdated version of the program while trying to use a model that is supported in the latest release.
The 14B Q4_K_M needs 9GB, but Q3_K_M is 7.3GB. But you also need some room for context. Still, maybe using `--override-tensor` in llama.cpp would get you a 50% improvement over "naively" offloading layers to the GPU. Or possibly GPT-OSS-20B. It's 12.1GB in MXFP4, but it’s a MOE model so only a part of it would need to be on the GPU. On my dedicated 12GB 3060 it runs at 85 t/s, with a smallish context. I've also read on Reddit some claims that Qwen3 4B 2507 might be better than 8B, because Qwen never released a "2507" update for 8B.
Haven't tried GPT-OSS-20B yet — the MOE approach is interesting for keeping VRAM usage down while getting better reasoning. 85 t/s on a 3060 is impressive. I'll look into that.
I've been on Qwen3 8B mostly because it was "good enough" for the mechanical stages (scanning, scoring, dedup) and I didn't want to optimize the local model before validating the orchestration pattern itself. Now that the pipeline is proven, experimenting with the local model is the obvious next lever to pull.
The Qwen3 4B 2507 claim is interesting — if the quality holds for structured extraction tasks, halving the VRAM footprint would open up running two models concurrently or leaving more room for larger contexts. Worth testing.
Thanks for the pointers — this is exactly the kind of optimization I haven't had time to dig into yet.
I wasn't sure where I'd seen that "retiring" spiel before, but then I remembered someone was (still is) selling a handmade jewelry website claiming $4.3M revenue and $1.3M profit.
reply