18 Ā· Backtesting
A single train/test split gives you one lucky-or-unlucky number. Rolling-origin (walk-forward) backtesting evaluates across many cutoffs for an honest mean Ā± std of your error.
Rolling-origin backtesting repeatedly moves the train/test cutoff forward through the series, forecasting the next window each time using only the past. The distribution of per-origin errors (mean Ā± std) is a far more honest accuracy estimate than one split.
Without this:
A single split can land on an easy (or impossible) stretch and report a misleading error; you have no sense of variance and can over-trust a fluke.
In ordinary ML you'd reach for k-fold cross-validation, but you can't shuffle a time series ā folds would put the future in training. The time-series equivalent is rolling-origin (a.k.a. walk-forward) backtesting:
- Pick an initial training window (the first chunk of history).
- Forecast the next
Hsteps (your horizon). - Score the forecast against what actually happened.
- Roll the origin forward ā extend (or slide) the training window to include those steps ā and repeat.
You end up with one error per origin. Their mean is your expected accuracy and their standard deviation tells you how stable it is. A single split gives you point (1) only, on whichever slice of the series you happened to pick.
The cardinal rule: at every origin the model may only see data strictly before the cutoff. No peeking. Two flavors of the training window:
- Expanding ā the window grows each fold (uses all history so far). Realistic for production retraining.
- Sliding ā a fixed-length window slides forward (drops old data). Better when the process drifts and old data is stale.
Below we run an expanding-window rolling-origin backtest of a seasonal-naive forecaster across several cutoffs, collect the per-origin MAEs, and print the mean Ā± std.
Python (in browser)
An expanding-window rolling-origin backtest: each fold forecasts the next 7 days using only its past, yielding a mean Ā± std error instead of one fragile number.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
What is the defining rule that makes rolling-origin backtesting valid for time series (and that ordinary k-fold violates)?
- Rolling-origin (walk-forward) backtesting moves the cutoff forward, forecasting the next window each time using only the past.
- Report the mean Ā± std of per-origin errors ā a single split hides the variance and can over-trust a lucky slice.
- Expanding windows grow with all history (production retraining); sliding windows drop stale data (drifting processes).
Model selection, hyperparameter tuning, and production monitoring all rely on walk-forward backtests (sklearn TimeSeriesSplit, sktime, Darts).
If you remove it: Rely on a single split and you'll tune your model to one slice of history and be blindsided when the next stretch behaves differently.