Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 40 ยท Evaluation and Cross-validation

Time Series Cross-validation

Evaluate forecasting models honestly: why random k-fold leaks the future, TimeSeriesSplit, expanding and rolling windows, gaps, a backtest loop for any model and how to compare models fairly.

Ordinary cross-validation shuffles rows into folds. For time series this is a trap. A random fold trains on observations from after the validation rows, so the model gets to peek at the future, and neighbouring points are so correlated that the score looks wonderful. The result: a model that cheats in testing and disappoints in production.

Time series cross-validation (also called backtesting or rolling-origin evaluation) keeps the arrow of time: every model is trained only on data before the period it is tested on, exactly as it will be used in real life. You repeat this at several forecast origins and average the errors, which gives a far more reliable estimate than one single train/test split.

flowchart TB
    subgraph F1["Fold 1"]
      a1["train"] --> b1["test"]
    end
    subgraph F2["Fold 2"]
      a2["train (longer)"] --> b2["test"]
    end
    subgraph F3["Fold 3"]
      a3["train (longer)"] --> b3["test"]
    end

Expanding window: the training set grows, and the test block always lies strictly after it.

Why random folds fail

Python Random k-fold versus time-ordered folds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The shuffled score is much better, but only because validation rows have training rows on both sides. In production the future is never available.

TimeSeriesSplit: what the folds look like

TimeSeriesSplit produces expanding training windows. Print the indices to see the structure.

Python The folds of TimeSeriesSplit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Useful parameters:

  • test_size: how many steps each validation block covers; set it to your forecast horizon.
  • gap: rows skipped between train and test. Use it when features look back over a window that overlaps the test period, or when real-world delays mean you could not train on the most recent rows.
  • max_train_size: caps the window length, which turns expanding into rolling.

Expanding versus rolling window

  • Expanding window: all history up to the origin. More data, but old regimes stay in forever.
  • Rolling (sliding) window: only the most recent W points. Adapts to change faster; useful when behaviour drifts (prices, user habits).

Compare both by backtesting; do not assume.

A backtest for any forecaster

cross_val_score fits a table model. Statistical models like ARIMA or ETS need an explicit loop: for each origin, fit on the history, forecast h steps, compare with what actually happened.

Python Rolling-origin evaluation of a seasonal naive and an ETS-style model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table shows each origin, so you can see whether a model wins consistently or only on one lucky period. Report the mean and the spread across origins.

Choosing the splits

  1. Set the horizon first. test_size = the horizon you will forecast in production.
  2. Make the first training window long enough to contain at least two full seasonal cycles.
  3. Use at least 5 to 10 origins if the data allows, spaced so that tests do not overlap much.
  4. Keep a final holdout. Tune on the backtest folds, then report one last number on a block you never touched.
  5. Mind the features. Any feature built from the whole series (a global mean, a scaler, a target encoding) must be computed inside each fold from training data only.

Cross-validation for many series

With thousands of series you can validate across both time and series: hold out the last horizon of every series and, if the model is global, also keep some series entirely unseen. Aggregate with a scale-free metric such as MASE (see forecast accuracy metrics).

Common mistakes

  • Shuffling rows or using cross_val_score(cv=5) on a time series.
  • Scaling or imputing on the full dataset before splitting.
  • Tuning hyperparameters on the same folds you report as final accuracy (use nested validation or a final holdout).
  • A test block shorter than the real horizon, which flatters long-range accuracy.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is shuffled k-fold cross-validation wrong for forecasting?

    A model that trains on the future of its own validation points reports unrealistically good errors.
  2. 2. What does the gap parameter of TimeSeriesSplit do?

    A gap prevents overlap or leakage when features or delays span several rows.
  3. 3. Which setting makes the training window rolling instead of expanding?

    Capping the training size keeps only the most recent W observations.
  4. 4. A StandardScaler is fitted on the whole dataset before backtesting. What is the problem?

    Any statistic computed on all data includes future information, so it must be fitted inside each training fold.

Summary

  • Time series validation must respect order: train on the past, test on the future.
  • TimeSeriesSplit gives expanding windows; max_train_size makes them rolling; gap prevents leakage.
  • Write an explicit rolling-origin loop for statistical models.
  • Set the test size to your real horizon, keep a final untouched holdout and fit all preprocessing inside the folds.

Next: Prediction Intervals and Uncertainty