I just dont go there at all because they added the weird copied of stackoverflow feature when copying code from there. Like even if I want to use its actively hostile, I need to paste it somewhere remove all of the extra things added and then copy properly

ffffffff.... damnit I love python but why is it so difficult like this. JIT would have been so perfect so the language, increasingly overtime with AI code coming through I am not sure not focusing on making the performance better is going to be better for longer term. Even though a lot of the other operations done in python are done faster in libraries I would still like faster startup times, or faster loops.

Not really, the time I set aside for shows, podcast, movies and games is all different. I am not going to give up watching slow horses to watch a podcast,

Sure, but for people without abundant time to spend alone then watching a show this evening kind of precludes also ingesting a podcast before tomorrow. It's not like the labour movements around the world have negotiated significantly more free time for the big mass of working humans since Netflix said that sleep is their biggest competitor back in 2017.

Yeah but podcast is low quality time; shows,movies are high quality, games are high quality high focus, music and podcast are lower on the rung. Its something you can have playing in the background and not pay attention.

Like the time I have for high quality stuff and low stuff are different and therefore different audiences. For Low quality stuff podcasts are fighting against tiktok

But why is the question, I run about 400k models every 30 minutes at work. I have tried all other approaches, but xgboost model per timeseries has never been beaten before. Even if I wasn't targetting accuracy, Xgboost loading a pretrained model and predicting with it and still faster.

Have you tried the latest TSFMs? Research (even in 2025) shows they outperform XGBoost models in certain scenarios. https://www.sciencedirect.com/science/article/pii/S266682702...

Yep, tried them multiple times but too many tradeoffs and not matching rmse kills it for us. MAE isn't that impressive for us.

Are they very stationary time series with not very changing dynamics over time?

They are very dynamic, like too dynamic but we had really good predictors for it based on our existing data and past historical data.

I forecast 400k timeseries every 30minutes at work.

What kind of infrastructure do you use for that scale? Are these computationally light ARIMA models?

We're running 150k models per day and while I'm sure I could scale this to running them every 30 minutes, it would mostly be me throwing money at the problem to parallelize the compute.

Some HPC Clusters, xgboost, read pretrained model from NFSRDMA Server, read data, parallelise over MPI with one process per model. And the latest Zen5 Epycs. I could do it all on just one of the 192 cores EPYCs but we like to parallelise it over multiple machines to get the extra speed and reliablity.

It wasn't that expensive, the main problem was the storage bottleneck which got fixed with just some LVM cache and gen5 ssds.

I dont see why people arent mentioning edge in the comments? I think with e5 license and purview, its the same as what I am seeing mentioned in Firefox Enterprise?

Not unsurprising, excluding grid efficiency and industrial demand, its also just households. More and more places are getting batteries wherever possible, plug in solar is also avaialble. The amount of demand I don't forsee increasing at all unless people start getting air source heatpumps.

National Grid can mean NGED, NGET or NESO, god I love the UK. 2/3 of those are completely cuckoo regarding long term planning so not surprising to hear those comments.

> 2/3 of those are completely cuckoo regarding long term planning

The network operators or the system operator? Hard to do any form of sane long term planning when the regulator changes their mind every 5 years.

system operator and the transmission operator, system operator keep throwing a wrench into pre-made plans, transmission operator are a complete monopoly who bully their way around

I think their strategy right now is just spend as much raw cash as possible into companies

A bit surprised about the datafusion results from the post, I have tried it time and time again, but datafusion has always been the leading/trading blowers with polars for our workfloads with duckdb being vastly slower.

There are some benchmarks for the previous version https://benchmark.clickhouse.com/#system=+ti%20rud|Dusa,s|PD...

Doing the benchmarks for 2.0 on the large AWS metal machines at small data sizes (SF=10) really opened my eyes that we have some low-hanging fruit in Polars when it comes to optimizing our constant overhead for smaller queries.

For example our join currently does a full partition into T partitions, for each of the T threads. Overall we create T^2 partitions, which on a 192-core machine is non-trivial. Great if you have a ton of data to feed that with, but if you 'only' have a few dozen million rows it becomes rather small. This is the primary reason we saw in the benchmarks that Polars pinned to 32 threads beats 192 thread Polars at SF=10.

I'll be working on improving that soon. I expect that to have a big impact on SF=10, and a decent impact on ClickBench, which sits between SF=10 and SF=100 in terms of rows.