I highly encourage anyone e.g. on MacOS to get a cheap laptop and give Omarchy a try. My feeling of using this system compared to MacOS is a whole another level. And now one of the promises of Omarchy Quattro is that AI agents will help you tweak it to your liking very easily, avoiding having to spend dozens of hours ricing your system.
That is actually insane. In my opinion, the future is local AI: for most daily tasks, you absolutely don't need Fable level intelligence - you need Fable level agentic capabilities. And this model has (almost) just that. If we get a similarly capable MoE model in a few months (yes we will), it's going to be an utterly wild ride.
The multimodal abilities are great, but if you deal with text only, what is the benefit of using this over DS V4 Flash/Pro? 13-26x cheaper with comparable intelligence, and available across many different inference providers.
I fail to see the usecase where DS V4 Pro is not enough, but Flash 3.7 is - except multimodal.
Luna is similar, and also 8x cheaper. Source: artificialanalysis
The only benefit I can see is the speed, that looks to be outstanding, probably thanks to their TPUs.
> 13-26x cheaper with comparable intelligence, and available across many different inference providers.
Well, compared to 2 months ago, it's no longer 100x more expensive for similar levels of quality...
If they continue monthly-ish releases by 3.9 - by Halloween - they should be close to the best in terms of what you get for what you pay for.
In 2 months, they've gone from basically the bottom of the pack to at least being somewhat usable and competitive.
OpenAI and Anthropic release in a month, and change things. OpenAI is claiming to be close to an Astra release - but that seems like a Fable type release - where they're just releasing a better more expensive model, not more cost effective models.
for non-coding applications, i think speed is a real differentiator. Im building an app that uses LLMs for some functionality that the user would not have any reason to expect is using AI and therefore having then wait seconds or minutes is just not feasible. latency is a huge upside for me
Anything interacting with the real world seems like latency would be hugely important. Something more asynchronous friendly (like coding) is for obvious reasons over represented here
did you try to ingest 1M documents per hour with any provider except GCP with Flash? None work at scale. Deepseek, Luna, Mistral all fail. 1 in 3 requests is a fail. I stopped trying.
The only thing that works at scale is gemini flash.
I guess the question then becomes "are you sure you'll do text only?"
I could probably do text only for my workflow (feature development/debugging for web microservices) but sometimes it is easier to just toss a screenshot into the Claude prompt, so that gives it an edge.
If your workflow is 100%, certifiably never ever going to involve an image, then yeah, this isn't going to be huge.
> If you buy your own GPU, rack it, and run an open model, there is no unit cost.
No. There is economic opportunity cost (borrowing), energy cost, infra cost, depreciation / risk of failure with each unit of work, bandwidth, maintenance, and lots more. Small, but not zero, and often overlooked - especially the opportunity cost.
These are basically all fixed costs, not unit costs. You buy the GPU once and use it for as many calls as you have traffic for, and depreciate it over a fixed lifetime. You have to power it regardless of whether it's fully utilized or not. You have to maintain it by virtue of owning it, not really based on how many queries it has served. Bandwidth is the only one that really scales as a unit cost.
Open question whether this model is actually more economical than using the cloud AI service. The whole reason the industry moved to cloud computing in the first place was because computing had very high fixed costs, and the more these could be amortized over a fully-loaded query stream, the lower the unit costs.
You have to power it all the time, but the amount of power it uses while it’s on will change by up to a few orders of magnitude depending on the gpu. It’s not uncommon for a gpu to be pulling just a couple of watts at idles and several hundred at full tilt.
So the only way it’s a fixed cost is if you don’t pay for power. If you only consider the cost of the power, it might still be cheaper paying for an api.
Good points. Speaking from experience, it’s really hard to make it more economical than using a cloud AI service - even if you utilize the GPU to its fullest. There are, of course, other benefits, such as privacy/control/compliance/security, which should be the real reasons to do this, not cost.
Spoken like a guy who has never had to maintain bare metal infrastructure ops at scale. These things break, need re-imaged, have parts that break, have to be configured (now you need provisioning pipelines, monitoring, alarms, etc), have to be maintained when something goes wrong (swapping hardware and software, debugging the alarms into actually figuring out which bits are broken and/or misconfigured), have to be catalogued, have to be planned for, deprecated, and the finances accounted for through complicated accounting to show investors the Capex at quarterly meetings.
Then you have to make fleet decisions on how much of each type of server you will want to buy, expanded storage, how long you will support each generation of server, when you will order new hardware, how to order new hardware, lag for real world installs, hiring actual humans to fly around to all of your datacenters and do the actual installations / maintainance / etc. Then you will have to do contracts with individual datacenter operators for margins, electricity rates, hosting contracts, white glove ops hourly rates, etc.
Businesses that own their own hardware tend to have a lot of employees whose jobs are maintaining it and running the business side of that.
How is this different from hosting infra or just SaaS in general?
TFA contends that there is some fundamental shift in the economics of software, but it doesn't look to be very different from either a new SaaS dependency or racking new hardware.
Exactly my point in another comment. Just to illustrate this further: a rough ballpark of how the cost of intelligence fell since 2022 could be about 1000x, and continues to fall. Unfortunately, it’s really hard to measure.
It’s so cheap that companies choose to spend more on AI inference (more reasoning, more capabilities, longer context), not less - see Jevons paradox.
> Meeting that expectation means making LLM calls, and LLM calls cost money.
Of course, and so does everything in the software world. The point is getting the cost so low that it’s basically free. The new DS V4 Flash or the smaller Qwen3.6 models are still really expensive compared to what we were used to in the economics of software, but it’s not unreasonable to expect these costs to continue falling down.
Rough chatgpt estimate says 3-5 orders of magnitude of difference compared to a typical user interaction with a SPA (db/cache lookup, CDN…)
Yeah, my point is that LLMs can reach that point too. Especially if you do it clientside. Copying was also way more expensive back in the 60s (accounting just for machine time and electricity, not storage cost, about 100 million times more expensive than today). Everything has a cost.
Definitely, of course. But I think the logic of the author’s article is based on there being a huge difference between the two, or rather a high cost of inference in absolute terms. And that can change and we have seen it change. Which breaks the entire premise of the article going forward, no?
Inference will be always more expensive than db operations or copying, sure. But how much more expensive is the question.
Reminds me of Slavoj Zizek’s classic example of synthetic sex (look it up), or his grievances about today’s academia: paper written by ChatGPT, peer reviewed by ChatGPT, and consumed by users as a synthesis from ChatGPT.
