Rolling-Origin Evaluation

Description

Rolling-origin evaluation, also called time-series cross-validation or backtesting, estimates how accurately a forecasting model will predict the future by repeatedly recreating the position it will be in when used: trained only on data up to a point in time, it forecasts the period that follows. The evaluator chooses a first forecast origin (the last time point the model may see) late enough to leave adequate training data, fits the model on the data up to that origin, forecasts every step of a fixed horizon ahead, for example one to seven days, and records the error at each lead time against the values later observed. The origin then moves forward by one step or a block of steps and the process repeats, with the model either refitted at each origin on all the data so far or on a window of fixed length (recalibration, in Tashman's terms), or kept unchanged and simply given the new observations as inputs (updating), whichever matches how it will be maintained in service. Because every forecast uses only data from before its origin, the method avoids the leakage of future information that randomly shuffled cross-validation can cause on time-ordered data, and because it uses many test periods rather than the single one of a fixed train-test split, its result depends less on which period happened to be held out. The output is the forecast error, under whatever error measure suits the forecast, broken down by lead time and by origin date for each candidate model, which supports claims about expected accuracy on unseen future data over a stated horizon and shows how that accuracy varies across seasons and conditions.

Example Use Cases

Reliability

A regional electricity distributor deciding whether to replace its regression model for day-ahead substation demand with a gradient-boosted model evaluates both over the last two years, with an origin each day at the time forecasts are issued, a horizon of the next 48 half-hours and weekly refitting, as planned for operation. It compares error at each lead time and in each season, and switches only after finding the new model better in winter peaks as well as on average.

A national weather service's post-processing team testing a machine-learning correction for station temperature forecasts places origins every day over three years, with lead times of one to ten days, and refits the correction monthly as it would in service. Error by lead time and by month shows whether the correction still helps at the longest lead times and whether its gain in winter holds in all three winters rather than one.

A hospital's operations analysts choosing a model to forecast emergency admissions one to fourteen days ahead for staff rotas compare a seasonal statistical model with a neural network, using weekly origins over two years that include two winter peaks. They report error at the seven-day lead time the rota planners use, and keep the simpler model because the neural network's advantage disappears at that horizon.

Limitations

Refitting at every origin multiplies training cost by the number of origins: daily origins over two years means more than 700 fits. Machine-learning forecasters are often refitted only now and then and otherwise just fed new data, which Hewamalage et al. describe as common practice; that is cheaper but measures something different, so the refit schedule in the evaluation must match the one planned for deployment or the errors do not describe the deployed system.
The time-ordered split prevents leakage only through the split itself. Scaling, smoothing, decomposition, feature selection or tuning done on the whole series before splitting, or features summarising a period not yet over at the forecast time, still let future information in. Hewamalage et al. show a model beating the best possible forecast of a random walk only because a decomposition was fitted to the whole series; every such step must be redone within each origin.
Errors from neighbouring origins share most of their training data and, for multi-step horizons, overlapping test periods, so they are correlated: an average over hundreds of origins is less precise than the count suggests, and the spread across origins is not a confidence interval. Deciding whether one model beats another needs a test designed for serially correlated forecast errors, such as the Diebold-Mariano test, not a plain comparison of average errors.
The first origins have little training data, so their errors can overstate those of a model trained on the full history, and a short series leaves few test periods and a noisy average. For purely autoregressive models (past values as the only inputs) on stationary series, whose errors show no remaining autocorrelation, Bergmeir et al. show that standard k-fold cross-validation remains valid, and Hewamalage et al. recommend it for short series as it uses the data more efficiently.
It measures how the model would have performed over the history available, not how it will perform after a change that history does not contain, such as a new tariff, a replaced sensor or an abrupt structural break; Hewamalage et al. note that no evaluation design can anticipate a break that lies only in the future. Results should be stated for the periods and conditions they cover and repeated as new data arrive.

Resources

Research Papers

Out-of-sample tests of forecasting accuracy: an analysis and review

Leonard J. Tashman•Oct 1, 2000

Forecast evaluation for data scientists: common pitfalls and best practices

Hansika Hewamalage, Klaus Ackermann, and Christoph Bergmeir•Dec 2, 2022

Software Packages

sktime: a unified framework for machine learning with time series

Oct 12, 2026

A unified framework for machine learning with time series

Tutorials

Forecasting: Principles and Practice (3rd ed.), section 5.10: Time series cross-validation

Documentations

evaluate: evaluate forecaster using timeseries cross-validation (sktime documentation)

Evaluate forecaster using timeseries cross-validation.

Related Techniques

Name	Description	Assurance Goals
Cross-validation	Cross-validation estimates how well a model will perform on data it has not seen, and how much that estimate depends on the particular data used. The dataset is divided into k parts called folds, the model is trained on all but one fold and scored on the held-out fold, and this repeats until every fold has served as the held-out set once. The scores are averaged to give a performance estimate, and their spread gives a rough indication of how much the result depends on which data the model was trained on. Common variants are stratified cross-validation, where each fold keeps the overall class proportions, and leave-one-out, where each fold is a single example. A large gap between training scores and held-out scores points to overfitting, where a model has learned quirks of its training data that do not carry over to new data.	Reliability
Continuous Ranked Probability Score	The continuous ranked probability score (CRPS) measures the quality of a probabilistic forecast of a numerical quantity, such as tomorrow's temperature or next week's demand, by scoring the whole predicted distribution, not just a single best guess, against the value later observed. For each case it compares the forecast's cumulative distribution (the predicted probability that the outcome will fall below each possible value) with a step that jumps from 0 to 1 at the observed value, squares the gap at every value and adds these up, which is the same as adding up the Brier score of the yes-or-no forecasts 'the outcome will be below this value' over every possible threshold. For a forecast given as an ensemble or a sample it can be computed as the average distance between the members and the observation minus half the average distance between pairs of members, closed forms exist for common distributions such as the Gaussian, and for a single-value forecast it reduces to the absolute error, so probabilistic and deterministic forecasts can be compared directly. The score is proper: a forecaster expects the best score by reporting the distribution they actually believe, so it cannot be improved on average by making forecasts narrower or wider than warranted, and it is lowest when the forecast distribution matches how outcomes actually vary. Scores are averaged over many cases, are in the units of the forecast quantity with lower values better, and are often reported as a skill score relative to a reference forecast such as climatology (the long-run historical distribution), giving evidence for claims about the accuracy of a forecaster's probabilities by variable, location and lead time.	Reliability
Bootstrap Confidence Intervals for Evaluation Metrics	Bootstrap confidence intervals put a range around a model's measured performance, showing how much the score could differ had another test set of the same size been drawn from the same population. The labelled held-out evaluation set is resampled with replacement thousands of times, each resample holding as many cases as the original so that some cases appear more than once and others not at all, and the metric (an error rate, the area under a curve, a translation score) is recomputed on each resample from the model's stored predictions, with no retraining. The spread of these recomputed values stands in for the sampling distribution of the metric: the percentile interval takes its 2.5th and 97.5th percentiles for a 95 per cent interval, and the bias-corrected and accelerated (BCa) interval shifts those percentiles to correct for bias and skew in that distribution. The statistic can also be the difference between two models scored on the same cases, recomputed on each shared resample, which shows whether an observed improvement is larger than the noise from test-set sampling. The result is the metric with lower and upper bounds at a stated confidence level, so a requirement such as a maximum false-positive rate can be checked against the bound rather than the point estimate.	Reliability