Max from Nixtla here. We are surprised that this has gained so much attention and are excited about both the positive and critical responses.
Some important clarifications:
The primary goal of this first version of the paper is to present TimeGPT-1 and showcase our preliminary findings from a large-scale experiment, demonstrating that transfer learning at this scale is indeed possible in time series. As mentioned in the paper, we deeply believe that pre-trained models can represent a very cost-effective solution (in terms of computational resources) for many applications. Please also consider that this is a pre-print version. We are working on releasing a reproducible set of experiments on a subset of the data, so stay tuned!
All previous work of Nixtla has been open source and we believe TimeGPT could be a viable commercial product, offering forecasting and anomaly detection out of the box for practitioners. Some interesting details were omitted because they represent a competitive advantage that we hope to leverage in order to grow the company and keep providing better solutions and continuing to build our ecosystem.
As some others have mentioned in the thread, we are working to onboard as many people as possible into a free trial so that more independent practitioners can validate the accuracy for their particular use cases. You can read some initial impressions of the creators Prophet [1] and GluonTS [2] or listen to an early test by the people from H20 [3]. We hope to see some more independent benchmarcks soon.
We love ARIMAs. That is why we put so much effort into creating fast and scalable Arimas and AutoArima in Python [1].
Regarding your valid concern. There are several reasons for the high computational costs. First, ARIMA and other "statistical" methods are local, so they must train one different model for each time series. (ML and DL models are global, so you have 'one' model for all the series.) Second, the ARIMA model usually performs poorly for a diverse set of time series, like the one considered in our experiments. The AutoARIMA is a better option, but its training time is considerably longer, given the number and length of the series. Also, AutoARIMA tends to be very slow for long series.
In short: for the 500k series we used for benchmarcking, ARIMA would have taken literally weeks and would have been very expensive.
That is why we included many well-performing local "statistical" models, such as the Theta and CES. We used the implementations on our open-source ecosystem for all the baselines, including StatsForecast, MLForecast, and Neuralforecast. We will release a reproducible set of experiments on smaller subsets soon!
Using 30,490 series of daily sales at Walmart (M5) we show that, Amazon Forecast is 60% less accurate and 669 times more expensive than running an open-source alternative in a simple cloud server.
A lot of M3 datasets we use are high-frequency, with large seasonal inputs. Considering Gaussian Processes (GP) complexity is O(N^3), a careful study of their performance would be challenging.
Also... I'm not aware of any efficient GP Python implementations.
GPs over time series can leverage low-dimensional index sets for O(N lg N) fitting and inference. This can be done by interpolating the inputs onto a regular grid which admits Toeplitz kernels. See https://arxiv.org/abs/1503.01057.
Bonferroni's correction on hold-out data is an excellent suggestion. To adapt it into time series forecasting, one could perform temporal cross-validation with rolling windows and follow the performance's variance through time.
Unfortunately, the computational time would explode if the ML method's optimization is performed naively. Precise measurements of the statistical significance would crowd out researchers except for Big Tech.
In Machine Learning literature, the variance of accuracy measurements originates from different network parameters initialization.
Since the deep learning ensembles already use aggregate computation in the hundreds of days, computing the variance would elevate the computational time into thousands of days.
In contrast, statistical methods that we report optimize convex objectives; their optimal parameters are deterministic.
That being said, we like the idea of including cross-validation with different splits for future experiments.
Imagine you want to forecast the next day of electricity load. We ran the test with PJM data.
Results:
SeasonalNaive is 20% more accurate than NeuralProphet and 366 times faster. StatsForecast's MSTL is 24% more accurate than NeuralProphet and 11 times faster.
SeasonalNaive is 23% more accurate than Prophet and 75 times faster. StatsForecast's MSTL is 26% more accurate than Prophet and 2 times faster.
