Similar to Finland. Quite lot of time in the year like right now outside temperature is above zero, but below say 15C. So heatpumps are in very good efficiency range.
I’m slightly confuse as to how all this works. Do the GPUs just sit there with the models on them when the models are not in use?
I guess I’d assumed this sort of thing would be allocated dynamically. Of course, there’s a benefit to minimizing the number of times you load a model. But surely if a GPU+model is idle for more than a couple minutes it could be freed?
(I’m not an AI guy, though—actually I’m used to asking SLURM for new nodes with every run I do!)
Let's say, then, that it's not so much "dumb and wasteful" as "energy inefficient". In fact, this can be quite wise in a modern world full of surveillance-as-a-business and "us-east-1 disasters"
Can you elaborate the last statement? Don't quite understand why loading local LLM to GPU RAM, using it for the job and then "ejecting" is "dumb and wasteful" idea?
Because as a function of hardware and electricity costs, a “cloud” GPU will be many times more efficient per output token. You aren’t loading/offloading models and don’t have any parts of the GPU waiting for input. Everything is fully saturated always.
I believe GP means it still to be connected to 'if this kind of latency is unacceptable to you' - i.e. you can't load/use/unload, you have to keep it in RAM all the time.
In that case it's massively increasing your memory requirement not just to the peak the model needs, but to + whatever the other biggest use might be that'll be inherently concurrent with it.
> This (along with batching) is why large local models are a dumb and wasteful idea if you're not serving them at enterprise scale.
Local models are never a dumb idea. The only time it's dumb to use them in an enterprise is if the infra is Mac Studio with M3 Ultra because pp time is terrible.
Models take a lot of VRAM which is tightly coupled to the GPU so yeah, it's basically sitting there with the model waiting for use. I'm sure they probably do idle out but a few minutes of idle time is a lot of waste--possibly the full 82% mentioned. In this case they optimized by letting the GPUs load multiple models and sharing the load out by token.
They definitely won't idle out- if they idle out, it'll take on the order of up to 60 seconds to load the model back into VRAM, depending on the model.
That's an eternity for a request. I highly doubt they will timeout any model they serve.
> That's an eternity for a request. I highly doubt they will timeout any model they serve.
That's what easing functions are for.
Let's say 10 GPUs are in use. You keep another 3 with the model loaded. If demand increases slowly you slowly increase your headroom. If demand increases rapidly, you also increase rapidly.
The correct way to do this is more complicated and you should model based on your usage history, but if you have sufficient headroom then very few should be left idle. Remember that these models do requests in batches.
If they don't timeout models, they're throwing money down the drain. Though that wouldn't be uncommon.
That's only if you're expecting 10 GPUs in use. They're dealing with ~1 GPU in use for a model, just sitting there. Alibaba has a very long tail of old models that barely anyone uses anymore, and yet they still serve.
Here's a quote from the paper above:
> Given a list of M models to be served, our goal is to minimize the number of GPU instances N required to meet the SLOs for all models through auto-scaling, thus maximizing resource usage. The strawman strategy, i.e., no auto-scaling at all, reserves at least one dedicated instance for each model, leading to N = O(M)
For example, Qwen2 72b is rarely used these days. And yet it will take up 2 of their H20 gpus (with 96GB VRAM) to serve, at the bare minimum, assuming that they don't quantize the BF16 down to FP8 (and I don't think they would, although other providers probably would). And then there's other older models, like the Qwen 2.5, Qwen 2, Qwen 1.5, and Qwen 1 series models. They all take up VRAM if the endpoint is active!
Alibaba cannot easily just timeout these models from VRAM, even if they only get 1 request per hour.
That's the issue. Their backlog of models take up a large amount of VRAM, and yet get ZERO compute most of the time! You can easily use an easing function to scale up from 2 gpus to 200 gpus, but you cannot ever timeout the last 2 gpus that's serving the model.
If you read the paper linked above, it's actually quite interesting how Alibaba goes and solves this problem.
Meanwhile on the other hand, Deepseek solves the issue by just saying "fuck you, we're serving only our latest model and you can deal with it". They're pretty pragmatic about it at least.
If I had to handle this problem, I'd do some kind of "split on existing loaded GPUs" for new sessions, and then when some cap is hit, spool an additional GPU in the background and the transfer the new session to that GPU as soon as the model is loaded.
I'd have to play with the configuration and load calcs, but I'm sure there's a low param, neat solution to the request/service problem.
> I guess I’d assumed this sort of thing would be allocated dynamically
At the scale of a hyperscaler I think Alibaba is the one that would be doing that. AWS, Azure and I assume Alibaba do lease/rent data centers, but someone has to own the servers / GPU racks. I know there are specialized companies like nscale (and more further down the chain) in the mix, but I always assumed they only lease out fixed capacity.
The paper is about techniques to do that dynamic allocation to maximize utilization without incurring unacceptable latencies. If you let a GPU sit idle for several minutes after serving a single request, you're setting money on fire. So they reuse it for a different model as soon as possible, starting even before the first request is finished, because: If you don't have a dedicated GPU for a model, are you going to wait for a multi-gigabyte transfer before each request? So they have a dedicated GPU (or two, one for prefill, one for decode) for a group of models that are processed in an interleaved fashion, scheduled such that they stay within the latency budget.
>Do the GPUs just sit there with the models on them when the models are not in use
I've assumed that as well. It makes sense to me since loading up a model locally takes a while. I wonder if there is some sort of better way I'm not in the know about. That or too GPU poor to know about.
> Reserving full GPU instances for these models leads to allocating 17.7% of our GPUs to serve only 1.35% of requests
> Deployment results show that Aegaeon reduces the number of GPUs required for serving these models from 1,192 to 213, highlighting an 82% GPU resource saving.
82% of their CPUs were serving 98.6% of all traffic. If they reduced the cluster size, they got it to 96.2% of their CPUs serving 98.6% of their traffic. If they reallocated those, which is more likely, then 96.8% of their CPUs are serving 98.6% of all requests, or around 17% more capacity for popular requests on the same hardware.
Not really, Figure 1(a) of the paper says that the 17.7% are relative to a total of 30k GPUs (i.e. 5310 GPUs for handling those 1.35% of requests) and the reduction is measured in a smaller beta deployment with only 47 different models (vs. the 733 "cold" models overall.) Naïve extrapolation by model count suggests they would need 3321 GPUs to serve all cold models, a 37.5% reduction to before. (Or 6.6% reduction of the full 30k-GPU cluster.)
From the SCMP article you might get the impression that the various figures all refer to the same GPU cluster, but in the paper itself it's very clear that this is not the case, i.e. the 213 GPUs in the smaller cluster are not serving 1.35% of the requests in the larger cluster. Then if you want to scale it, you have a choice of different numbers you could scale, and each would get different results. Since they're constrained by the limited number of different models a single GPU can serve, I think scaling by the number of models is the most realistic option.
In the past, software and computer engineers would tackle problems head-on, designing algorithms and finding creative solutions.
thanks to the US restrictions on semiconductor industry (Chinese), Chinese engineers are being forced to innovate and find their own ways to overcome challenges like the old school engineers (What Silicon Valley used to be)
If you're one who sees progress as an end goal unto itself, what you describe is a good thing. When one party is attempting novel solutions to outcompete the competition we will be faster to whatever the next change is.
That said, I'm not sure what the US policies specifically have to do with this. Countries are always in competition with one another, and if one industry or technology is considered a national security threat they will guard it.
Team X is responsible for feature Foo; feature Foo is slow; team X introduces Foo-preload, metrics go up, person responsible gets a bonus.
Multiply that by tens (or even hundreds) of teams and your app startup (either on desktop or mobile) is now a bloated mess. Happened to Office, Facebook iOS and countless others.
One solution is to treat startup cycles as a resource similar to e.g. size or backend servers.
> One solution is to treat startup cycles as a resource similar to e.g. size or backend servers.
The only way to achieve performance metrics in a large org IMO.
Google Search is still fast because if you degrade p99 latency an SRE will roll back your change. Macbooks still have good battery life because Apple have an army of QA engineers and if they see a bump on their Ammeters that MacOS release doesn't go ahead.
Everything else (especially talking about "engineers these days just don't know how to write efficient code") is noise. In big tech projects you get the requirements your org encodes in its metrics and processes. You don't get the others. It's as simple as that.
Never worked at MS but it's obvious to me that the reason Windows is shit is that the things that would make it good simply aren't objectives for MS.
As an ex-Microsoft SDET who worked on Windows, we used to test for those things as well. In 2014.
Then Microsoft made the brave decision that testers were simply unnecessary. So they laid off all SDETs, then decided that SDE’s should be in charge of the tests themselves.
Which effectively made it so there was no test coverage of windows at all, as the majority of SDE’s had not interacted with the test system prior to that point. Many/most of them did not know how to run even a single test, let alone interpret its results.
This is what Microsoft management wanted, so this is what they got. I would not expect improvement, only slow degradation as Windows becomes Bing Desktop, featuring Office and Copilot (Powered By Azure™).
Makes perfect sense. It recently became clear to me (e.g. [2]) that it's not a cohesive concept but to me personally this is the meaning of POSIWID [1].
Basically making Windows a good desktop OS is not in any meaningful way the "purpose" of that part of MS. The "purpose" of any team of 20+ SWEs _is_ the set of objectives they measure and act upon. That's the only way you can actually predict the outcomes of its work.
And the corrolary is that you can usually quite clearly look at the output of such an org and infer what its "purpose" is, when defined as such.
Then the OS team will fight back with options to disable all of these startup things. Like the Startup tab in Windows Task Manager with an "impact" column and easy button to disable annoying startup programs. It's interesting to even see it play out within the same company.
That could already be the case. The initial release is from 1990, so the codebase is at least 35 years old.
I don't have a good guess for the average age of software developers at Microsoft, but claude.ai guesses the average "around 33-38 years" and the median "around 35-36 years old".
I'm told from MS friends that there are still files with the intact 1987 changelog in Word; as well as workarounds for dot matrix printers that were released 40+ years ago.
Also, the Office codebase is significantly larger than Windows (and has been for a while), that was surprising to me.
Microsoft need to update the spec for all new personal computers to include mandatory pre-load hardware. This would have a secondary CPU, RAM and storage used for pre-loading licensed Office products before your laptop boots. AI would analyse your usage patterns and fire-up Office for you before you even get to work in the morning.
Perhaps, this could even allow you to have Office on-hand, ready-to-use on its own hardware module, while you develop Linux application on your main CPU.
Further down the line. Someone see an opportunity to provide access to compatible modules in the cloud, allowing re-use of older incompatible hardware. But there would be the danger that service (without the support of MS), may go bust, leaving those users without their mandatory instant access to licensed Office products, forcing upgrades to even newer hardware.
I believe a big crux is in definition of "war ended".
You (and Donald Trump) seem to be using "Ukraine and Russia stop shooting at each other right now", while Ukraine operates more under "Russia stops shooting at us for the foreseeable future, 20 years at least." Russia has previously broken a number of ceasefires and written agreements (including the infamous Budapest memorandum) and so Ukraine is not super trusting to agreements not backed by anything.
What Ukraine will accept is entirely dependent on how much funding they will get from foreign powers to continue their war effort.
I've had a lot of responses to my comment, yet I've seen no alternative ideas presented that will result in a different outcome. What is your plan for getting Russia to lose this war?
> or you can include them in the broad pool, and the people with a full-cinderblock home in a non-flammable state pay $20 more a year so the entire endeavour can work
And you immediately start loosing customers to insurers that either did the former or left LA alltogether. This changes $20 surcharge into $25 surcharge, causing more customers to leave, causing surcharge to increase and so on.
> IRI measures how much a car moves vertically as it travels over a given distance, and is typically given in units like “inches per mile” or “millimeters per meter.
How accurate are phone accelerometers these days? Could Uber/Lyft/etc. start collecting that data from drivers phones.
There's a reasonably simple change that would allow application of Harberger's taxes.
1. Restrictions have to have a "controlling party": a dedicated party that controls the restrictions and can agree to lift it. Classical example would be HOA, but it can also be a seller if they want to sell property with additional restrictions.
2. The controlling party sets the price of a restriction
3. Restricted party can remove the restriction paying price set in 2 to controlling part.
4. The controlling party pays tax as percentage of price set in 2.
