Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 40 ยท Evaluation and Cross-validation

Prediction Intervals and Uncertainty

Report forecast uncertainty: ARIMA and ETS intervals, empirical residual intervals, quantile regression with gradient boosting, coverage checks and the pinball loss.

A single number ("sales will be 1,204") hides how sure you are. A prediction interval gives a range that should contain the future value with a stated probability, for example 80% or 95%. Intervals turn a forecast into a decision tool: how much stock to order, how many servers to provision, how large a safety margin to keep.

Two properties matter:

  • Calibration (coverage). A 90% interval should contain the actual value about 90% of the time. Too low means overconfident, too high means uselessly wide.
  • Sharpness. Among calibrated intervals, narrower is better.

Intervals widen with the forecast horizon, because uncertainty accumulates. Intervals describe the future value; a confidence interval describes uncertainty about a parameter or the mean, and is narrower. Do not mix them up.

Intervals from a statistical model

ARIMA and exponential smoothing derive intervals from the fitted error variance and the model structure. In statsmodels, get_forecast returns the mean and the interval.

Python ARIMA forecast with a 95% interval

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The column names say mean_ci, but for forecasts these bounds are prediction intervals. Notice the width grows with the horizon and flattens for a stationary series (it approaches the series' own spread).

Empirical intervals from residuals

When a model has no built-in intervals (any machine learning forecaster), use the errors it made on validation data. Collect the backtest residuals per horizon, take their quantiles, and add them to the point forecast.

Python Residual-quantile interval for any model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Always check coverage on data you did not use to build the interval. Coverage near the nominal level means the interval is calibrated; a gap means the errors changed or the horizon is different.

Quantile models

Instead of one mean forecast, train models to predict quantiles directly. Gradient boosting supports the quantile (pinball) loss, so a model for the 10th percentile and another for the 90th give an 80% interval that can vary in width with the inputs.

Python Quantile gradient boosting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The interval is much wider where the data are noisier (about 1.7 at small x against 10.5 at large x): quantile models adapt the width to the situation, while a single residual quantile cannot. Notice also that the measured coverage (about 0.70) is below the nominal 0.80. Boosted quantile models are often slightly overconfident, which is exactly why you check coverage on held-out data and, if needed, widen the interval or calibrate it with residual quantiles.

The pinball loss

To score a quantile forecast at level q, use the pinball loss: it penalizes under-prediction with weight q and over-prediction with weight 1 - q. Its minimum is exactly the true q-quantile.

Python Pinball loss by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss is smallest when the prediction sits at the true 90th percentile (about 113).

Practical advice

  • Report intervals with a nominal level and verify the coverage in backtests (time series cross-validation).
  • Intervals from statistical models assume correct model structure and roughly normal residuals; check the residual diagnostics.
  • Intervals that are too narrow are the common failure: add parameter and model uncertainty, or widen empirically.
  • For many horizons, estimate residual quantiles separately for each horizon.
  • Conformal prediction is a general way to wrap any model with intervals of guaranteed coverage under mild assumptions; the empirical-residual idea above is its simplest form.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A 90% prediction interval should:

    Coverage is the share of actual values that fall inside the interval; it should match the nominal level.
  2. 2. Why do prediction intervals normally widen with the horizon?

    Each additional step adds more unpredictable shocks, so the future value is less certain.
  3. 3. How can you get an interval from a model with no built-in uncertainty, such as a random forest?

    Empirical residual quantiles or quantile (pinball loss) models provide calibrated intervals for any learner.
  4. 4. What does the pinball loss measure?

    It scores a forecast for a chosen quantile and is minimized by the true quantile.

Summary

  • Intervals express uncertainty; a good one is both calibrated and sharp.
  • ARIMA and ETS give model-based intervals; other models use residual quantiles, quantile loss or conformal methods.
  • Always verify coverage on held-out data and by horizon.
  • Prediction intervals are about the future value and are wider than confidence intervals for the mean.

Next: Anomaly Detection in Time Series