Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 40 ยท Evaluation and Cross-validation

Forecast Accuracy Metrics

Measure forecast errors correctly: MAE, RMSE, MAPE, sMAPE, MASE and bias, when each one misleads, why MASE compares fairly across series, and how to report errors by horizon.

A forecast is only useful if you know how good it is. Choosing the error metric is not a detail: different metrics reward different behaviour, and a bad choice can crown the wrong model. This lesson covers the metrics every forecaster should know, how they behave on awkward data such as zeros and outliers, and how to compare models fairly across series of different scale.

Throughout, the error at time t is e[t] = actual[t] - forecast[t]. Metrics summarize the errors on a test set the model never saw (see time series cross-validation).

Scale-dependent metrics: MAE and RMSE

  • MAE (mean absolute error): the average of |e|. Easy to explain: "on average we are off by 12 units". Robust to outliers.
  • RMSE (root mean squared error): the square root of the average of e^2. Penalizes large errors more heavily. Optimizing RMSE targets the mean; optimizing MAE targets the median.

Both are in the units of the data, so you cannot compare them across series with different scales.

Python MAE, RMSE and bias

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

RMSE is larger than MAE because one error (10) is bigger than the rest. When RMSE is far above MAE, a few large misses dominate: investigate them. Bias (mean error) is separate: a model can have a small MAE but a systematic tilt that matters for stock or capacity decisions.

Percentage metrics: MAPE and sMAPE

  • MAPE: the average of |e| / |actual| in percent. Unit-free and popular with managers.
  • sMAPE: symmetric version dividing by the average of actual and forecast magnitudes.

MAPE has known problems: it is undefined when the actual is zero, it explodes for tiny actuals, and it penalizes over-forecasts less than under-forecasts (an under-forecast is capped at 100%, an over-forecast is not).

Python When MAPE misbehaves

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Equal absolute misses give equal MAPE, but the scale is lopsided: an under-forecast can never exceed 100%, while an over-forecast can be 200% or more, so a model tuned for MAPE learns to forecast low. One tiny actual (the value 1 above) is enough to wreck the average. Avoid MAPE for intermittent demand and series that touch zero.

Scale-free and robust: MASE

MASE (mean absolute scaled error) divides the MAE by the MAE of the naive forecast on the training data:

  • MASE < 1: better than the (in-sample) naive method.
  • MASE = 1: no better than naive.
  • MASE > 1: worse than naive.

It is defined for zeros, symmetric in sign, and comparable across series, which makes it the recommended default when you evaluate many series. For seasonal data scale by the seasonal naive errors (lag m).

Python MASE with a non-seasonal and a seasonal scale

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Relative metrics and skill scores

You can also divide a model's error by a benchmark's error on the same test set. A relative RMSE of 0.8 means 20% lower error than the benchmark. Reporting skill versus seasonal naive is the clearest statement for a business audience: "we cut error by 23% against the best simple rule".

Metrics by horizon and aggregation

Error normally grows with the forecast horizon, so report it per horizon, not just one overall number.

Python Error by horizon

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When you average over many series, say how: the mean of per-series MASE (a fair, scale-free comparison) is different from the sum of errors over all series (which lets large series dominate). Choose the aggregation that matches the business cost.

Choosing a metric

Situation Suggested metric
One series, plain-language reporting MAE (and bias)
Large errors are especially costly RMSE
Many series with different scales MASE or a relative error
Strictly positive series far from zero MAPE / sMAPE are acceptable
Intermittent demand with zeros MAE, MASE, or a scaled error; never MAPE
Probabilistic forecasts Pinball loss / interval coverage (see prediction intervals)

Also match the metric to the loss you train on when you can, and report at least one scale-free number next to a scale-dependent one.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which metric is undefined when an actual value is zero?

    MAPE divides by the actual value, so zeros break it; MAE, RMSE and MASE do not divide by the actual.
  2. 2. A MASE of 0.7 means:

    MASE below 1 means a smaller average error than the naive forecast.
  3. 3. RMSE is much larger than MAE on the same errors. What does this tell you?

    Squaring emphasizes large errors, so a big RMSE/MAE gap indicates outlying misses.
  4. 4. Why report errors per horizon?

    A model can be excellent at h = 1 and poor at h = 12; the per-horizon view shows where it can be trusted.

Summary

  • MAE is the clear, robust default; RMSE punishes big misses; bias reveals systematic tilt.
  • MAPE and sMAPE are intuitive but break near zero and are asymmetric; avoid them for intermittent series.
  • MASE is scale-free, defined for zeros and compares fairly across series.
  • Report error per horizon and as skill versus a naive baseline.
  • Evaluate on held-out data only, with time-ordered splits.

Next: Time Series Cross-validation