Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 111 ยท Regression

Regression Metrics

Measure a regression model honestly with MAE, MSE, RMSE, R squared, adjusted R squared and MAPE, read residual plots, and choose the metric that fits your problem.

A model that outputs numbers needs a way to say how wrong those numbers are. There is no single answer, because "wrong" depends on what hurts you. Being off by 10 twice and being off by 20 once are the same total error, but a delivery company, a bank and a weather service might each disagree about which is worse. Regression metrics are the different ways to turn a pile of errors into one score.

In this lesson you will compute the common metrics by hand and with scikit-learn, see how outliers and zeros trick some of them, learn why training R squared flatters you, and read residual plots, the visual check that no single number can replace. We use the fitted models from Multiple Linear Regression; the same metrics work for any regressor in the track.

Four ways to add up errors

MAE Average size of the miss, in the target's own units.
MSE Average squared miss. Big errors dominate.
RMSE Square root of MSE, back in the target's units.
R squared Share of variation explained, compared with the mean.
1 / 2

Errors, one at a time

Everything starts with the residual for each example: actual - predicted. Positive means the model guessed too low, negative too high. The metrics differ in how they combine residuals e1, e2, ..., en:

Metric Formula Unit Notes
MAE mean of abs(e) same as target Easy to explain; every unit of error counts equally
MSE mean of e ** 2 target squared Smooth, the quantity least squares minimizes
RMSE square root of MSE same as target Penalizes big errors more than MAE
R squared 1 - SSE / SST none Compared with always predicting the mean

where SSE is the sum of squared residuals and SST is the sum of squared distances from the mean of y. R squared of 1 is perfect, 0 is no better than the mean, and it can even go negative on held-out data when a model is worse than predicting the mean.

Python Four metrics by hand and with scikit-learn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that RMSE is larger than MAE here. That is always true (or equal when all errors have the same size), and the gap tells you something: the more RMSE exceeds MAE, the more the error is concentrated in a few large misses. In scikit-learn the argument order is always (y_true, y_pred); swapping them leaves MAE and MSE unchanged but changes R squared, a classic silent bug.

MAE versus RMSE: what do big errors cost?

Because RMSE squares errors, one huge miss moves it far more than MAE. Neither is "better". If an error of 100 is truly ten times as bad as an error of 10, MAE reflects your costs. If an error of 100 is a disaster much worse than ten errors of 10 (a bridge sensor, a medicine dose), RMSE reflects them. Whichever you choose, say so, and report it in the target's units so that a stakeholder can read it.

Python One outlier moves RMSE far more than MAE

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One bad prediction out of 100 roughly triples RMSE but raises MAE much less. A ratio of RMSE to MAE close to 1.25 is what normal-looking noise gives; a ratio well above that is a hint to go and look at your largest residuals.

R squared and its traps

R squared answers "how much better is this model than the laziest model, which always predicts the mean?" It is useful because it has no units. But it has pitfalls:

  • It never decreases when you add features on the training data, even random noise columns. A model with 50 junk features can show a higher training R squared than one with 3 real ones.
  • It depends on the spread of the target. The same absolute error gives a high R squared on a wide target range and a low one on a narrow range, so you cannot compare R squared across different datasets.
  • It says nothing about bias or curvature. A line fitted to a clearly curved pattern can still have a decent R squared.

Adjusted R squared corrects the first problem by charging a penalty for each extra feature: 1 - (1 - R2) * (n - 1) / (n - p - 1), with n examples and p features. It rises only when a new feature helps more than chance would. The most reliable cure, however, is simply to compute R squared on held-out data.

Python Junk features: training R squared rises, test R squared does not

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As junk columns are added, training R squared climbs steadily (from about 0.56 to 0.84), while test R squared drifts down and drops sharply with 40 junk columns. The adjusted version barely moves, which makes it a better quick check, but nothing replaces the held-out score. This is a preview of overfitting, treated fully in Overfitting, Underfitting and Bias-Variance.

Percentage errors and their traps

Business people like percentages. MAPE (mean absolute percentage error) is the mean of abs(e / actual), expressed in percent. Two traps make it risky:

  • Zeros and tiny actuals explode it. If the true value is 0, the percentage is undefined; if it is 0.01 the percentage is huge and dominates the average.
  • It is asymmetric. Under-predicting can cost at most 100 percent, while over-predicting has no cap, so a model tuned to minimize MAPE tends to under-predict.

If you need a relative measure, consider the median of the percentage errors, or work on the log scale (RMSLE, available as root_mean_squared_log_error), or just report MAE together with the typical size of the target.

Python MAPE when the target gets close to zero

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first four predictions are within a few percent, yet the single tiny actual drags the MAPE to a value that looks catastrophic. scikit-learn returns MAPE as a fraction (0.2 means 20 percent), so multiply by 100 to print percent. The last lines show the asymmetry: guessing half the truth costs 50 percent, guessing double costs 100.

Residual plots: look before you trust a number

Metrics compress thousands of errors into one value and hide the pattern. A residual plot (residuals on the vertical axis, predictions on the horizontal) shows what the number hides:

  • A shapeless cloud around zero is what a well-specified model looks like.
  • A curve (U-shape or arch) means the relationship is nonlinear and the model misses it.
  • A funnel (spread growing with the prediction) means the error is proportional to size, a hint to model the log of the target.
Python Two residual plots: a good fit and a bad one

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three models are fitted with the same straight line, and the first two even have an R squared above 0.9, yet only the first plot is healthy. The curved case shows a clear arch and the funnel case a widening spread: two different problems that the single score cannot tell apart.

Choosing a metric

Ask what an error costs in your setting.

  • Report one primary metric in the target's units (MAE or RMSE) and one unitless companion (R squared) if needed.
  • Optimize the metric you report. A model chosen by RMSE may rank differently under MAE.
  • Compute on held-out data only, from a split like the one in Train, Validation and Test Sets.
  • Compare with a baseline such as predicting the mean (Baselines and Dummy Models); a metric only means something relative to a reference.
  • In scikit-learn model selection, metrics are passed as strings such as "neg_mean_absolute_error" or "neg_root_mean_squared_error"; the neg_ prefix exists because scorers are always "higher is better".

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which metric punishes one very large error most heavily?

    RMSE squares every residual before averaging, so a large miss contributes far more than it does to MAE or the median absolute error.
  2. 2. A model's R squared on the test set is negative. What does that mean?

    R squared compares the model with the mean predictor. On held-out data a poor or overfitted model can do worse than that, giving a negative value.
  3. 3. Why is training R squared a poor way to compare a 3-feature model with a 50-feature model?

    Adding any column gives least squares more freedom, so training R squared never goes down. Held-out R squared or adjusted R squared shows whether the extra features really help.
  4. 4. When is MAPE a risky choice?

    Dividing by a tiny or zero actual value makes the percentage error huge or undefined, and those few points dominate the average.
  5. 5. A residual plot shows a clear U-shaped arch. What is the likely issue?

    Residuals should look like random noise. A systematic arch means the model's shape is wrong, which polynomial terms or a nonlinear model can fix.

Summary

  • Residuals are actual - predicted; MAE, MSE, RMSE and R squared are different ways to summarize them.
  • MAE is the typical miss in target units; RMSE punishes large errors; the RMSE to MAE ratio hints at outliers.
  • R squared compares with the mean predictor, rises with any added feature on training data, and can be negative on test data; prefer held-out scores and adjusted R squared.
  • MAPE breaks near zero and is asymmetric; use it carefully or choose another relative measure.
  • Always plot residuals: curves and funnels reveal problems that scores hide.

Next: see how a model finds its own best weights in Gradient Descent from Scratch.