Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 40 ยท Machine Learning Forecasting

Tree Models for Forecasting

Forecast with random forests and histogram gradient boosting in scikit-learn: nonlinear lag features, the extrapolation limit of trees, detrending, feature importance and tuning.

Once a series is a table of lags and calendar features, any regression algorithm can forecast it. Tree ensembles (random forests and gradient boosting) are the popular choice. They capture nonlinear effects and interactions ("sales jump on Fridays when it is also payday week") without feature scaling and with little tuning. In forecasting competitions they have long been among the strongest practical tools, often as part of the winning solutions for retail and demand data.

But trees have a famous blind spot for time series: they cannot extrapolate. This lesson shows what that means and how to work around it.

A first tree-based forecaster

We use a series with weekly seasonality and a nonlinear weekday effect. The features are two lags, the seasonal lag, a rolling mean and the day of week.

Python Random forest versus the naive baseline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The forest comfortably beats "yesterday", and it matches or beats the seasonal naive baseline (last week's value) because it also averages out noise using the rolling mean and weekday feature.

What features matter?

Tree ensembles report feature importance. It tells you which inputs the model leaned on, a useful debugging tool: if the top feature is suspiciously close to the target, check for leakage.

Python Feature importance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

dow and lag7 dominate, which is what the data generation implies: the weekday decides the level.

The extrapolation problem

A tree predicts by averaging the training targets that fall in its leaves. So it can never predict a value outside the range of the targets it was trained on. For a series with an upward trend, the test period contains values above anything in the training set, and the forecast flattens at the old maximum.

Python Trees cannot follow a trend

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The forest predicts roughly the largest training value (84.5) however far ahead we look; the linear model follows the line. The usual cure is to remove the trend before the tree sees the data.

Strategy 1: model the change, not the level

Predict the difference y[t] - y[t-1] (or the ratio) and add the forecast back to the last known level. Differenced data has no trend, so the target stays within the training range.

Python Forecast differences, then integrate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Strategy 2: detrend with a simple model, forecast the residual

Fit a linear trend, let the tree model the remainder, and add the trend back. It keeps the target in a stable range and lets a simple model do what it is good at.

Gradient boosting

Boosting builds trees one after another, each fixing the previous errors. scikit-learn's HistGradientBoostingRegressor is fast, handles missing values natively, and needs no extra install; libraries such as LightGBM and XGBoost work the same way, with the same table.

Python Histogram gradient boosting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Tuning without cheating

  • Tune with time-ordered validation (see time series cross-validation), never random folds.
  • Start with modest settings: a few hundred trees, depth 3 to 6, a learning rate of 0.03 to 0.1, min_samples_leaf of 3 to 20.
  • Re-train regularly: a tree model has no built-in way to adapt to drift, so schedule refits as new data arrives.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why can a random forest trained on raw levels fail on a strongly trending series?

    Each leaf predicts an average of training targets, so predictions are capped by the highest value seen in training.
  2. 2. Which remedy keeps trees useful on trending data?

    Differencing removes the trend so the target stays inside the training range; the forecast is then integrated back to the level.
  3. 3. A feature importance chart shows one feature carrying almost all importance and it is computed from the target at the same row. What do you suspect?

    A feature that contains the target's own value is a leak; the model memorizes it and will fail in production.
  4. 4. Why should hyperparameters of a time series model be tuned with time-ordered validation?

    Shuffled folds mix future observations into training, which overstates accuracy compared with real forecasting.

Summary

  • Random forests and gradient boosting forecast from lag, rolling and calendar features and capture nonlinear effects.
  • Trees cannot extrapolate beyond the training range: difference or detrend trending data first.
  • Use feature importance as a diagnostic, and tune with time-ordered validation.
  • Always compare with naive and seasonal-naive baselines.

Next: Multi-step Forecasting Strategies