> The future is going to be general agentic models doing search over specific forecasting problems, and then fitting something closer to a structural time-series model.
Over what time period? In the limit, an agent should be able to predict the time series itself right?
it’s an interesting question. My prediction is they use time-series parameters almost like a calculator. In the limit they could free-hand it. But parameterizing it reduces high-dimensional info to lower dimensional parameters in an observable way that makes it easy to update over time and introspect changes. But in the limit it’s hard to say. This is more of my “five year prediction”
On the one hand, anything worth forecasting is probably worth forecasting well (which means, at least to some extent, by hand).
On the other hand, an organization specialized to consume one type of forecast (say, 1d price return forecasts) might want the best-in-class one-size-fits-all approach to forecasting things that are inputs into that.
Yeah, if someone has a stack of well behaved series and just wants to maximum-juice one-step-ahead forecast, it's probably hard to beat some slightly modified attention model plus covariates plus GPUs. It's the long multi-step-ahead forecasts with esoteric pre-training on other series that I am taking umbrage with here.
A parsimonious model with justifiable parameter space and explainable output should trump a black box with billions of weights for most applications, and will for as long as there’s someone accountable for decisions based on the forecast. Can’t defend your forecast? Can’t use it.
An aside to your aside — in my experience working in regulated industries, the “insanity” of manually adjusting model outputs can well be part of an established, business as usual process. :)
Do you think there is a similar argument to be made in the context of foundation models for tabular data? In recent years tabular foundation models have performed well against benchmarks, but you made the point quite well that benchmarks across diverse datasets may not really be representative of practical use cases.
In particular, I would be curious to hear if you think the argument on latent patterns extends to tabular data. Do you think that the kinds of structural shapes and bounds learned by foundation models would similarly fail to generalize for many tabular datasets?
1. If you could elaborate a bit more on the section about "What the agentic system actually looks like". Will it use symbolic and arithmetic data for reasoning? What type of information do the agents exchange?
2. You have not discussed (I think) the possibility to fine-tune FTS models on proprietary data. I would like your opinion on this.
One thing I'm pondering is whether the bet could be not about "our model can encode good informational priors" but "our model can learn a faster approximate optimizer for a broad class of models". A lot of models are pretty clear how to specify in broad sense but are PITA to get to actually converge, and converge in reasonable time; what neural network can learn is basically an estimator for the parameters that runs in predictable time. (And then maybe is used to init these params for a proper fitter)
honestly this was beautiful. i loved reading it. i did my MS in stats (BA/BS in econ and stats) but i ended up in MLE instead and this gave me a beautiful insight into the time series production environment world that i needed to know about. love to see it. keep up the great work.
Cool stuff, can I ask what you studied for your bachelor's?
basically econ
Great post—enjoyed it especially in how specific you got. Definitely resonates with my experience as well.
thanks :)
> The future is going to be general agentic models doing search over specific forecasting problems, and then fitting something closer to a structural time-series model.
Over what time period? In the limit, an agent should be able to predict the time series itself right?
it’s an interesting question. My prediction is they use time-series parameters almost like a calculator. In the limit they could free-hand it. But parameterizing it reduces high-dimensional info to lower dimensional parameters in an observable way that makes it easy to update over time and introspect changes. But in the limit it’s hard to say. This is more of my “five year prediction”
Good post.
On the one hand, anything worth forecasting is probably worth forecasting well (which means, at least to some extent, by hand).
On the other hand, an organization specialized to consume one type of forecast (say, 1d price return forecasts) might want the best-in-class one-size-fits-all approach to forecasting things that are inputs into that.
This post of mine from last year is relevant to your last point, about making forecasts useful to decision-makers, whoever they may be: https://jordanmrubin.substack.com/p/dimensionalizing-forecast-value
Yeah, if someone has a stack of well behaved series and just wants to maximum-juice one-step-ahead forecast, it's probably hard to beat some slightly modified attention model plus covariates plus GPUs. It's the long multi-step-ahead forecasts with esoteric pre-training on other series that I am taking umbrage with here.
A parsimonious model with justifiable parameter space and explainable output should trump a black box with billions of weights for most applications, and will for as long as there’s someone accountable for decisions based on the forecast. Can’t defend your forecast? Can’t use it.
An aside to your aside — in my experience working in regulated industries, the “insanity” of manually adjusting model outputs can well be part of an established, business as usual process. :)
Do you think there is a similar argument to be made in the context of foundation models for tabular data? In recent years tabular foundation models have performed well against benchmarks, but you made the point quite well that benchmarks across diverse datasets may not really be representative of practical use cases.
In particular, I would be curious to hear if you think the argument on latent patterns extends to tabular data. Do you think that the kinds of structural shapes and bounds learned by foundation models would similarly fail to generalize for many tabular datasets?
Great post. Two questions.
1. If you could elaborate a bit more on the section about "What the agentic system actually looks like". Will it use symbolic and arithmetic data for reasoning? What type of information do the agents exchange?
2. You have not discussed (I think) the possibility to fine-tune FTS models on proprietary data. I would like your opinion on this.
Thank you.
One thing I'm pondering is whether the bet could be not about "our model can encode good informational priors" but "our model can learn a faster approximate optimizer for a broad class of models". A lot of models are pretty clear how to specify in broad sense but are PITA to get to actually converge, and converge in reasonable time; what neural network can learn is basically an estimator for the parameters that runs in predictable time. (And then maybe is used to init these params for a proper fitter)
honestly this was beautiful. i loved reading it. i did my MS in stats (BA/BS in econ and stats) but i ended up in MLE instead and this gave me a beautiful insight into the time series production environment world that i needed to know about. love to see it. keep up the great work.