Skip to content
Tutorials Center
My learning

Topics

Lesson 51 of 76 ยท Modelling

Evaluating Models with the Right Metrics

Pick the right metric: MAE, RMSE and R2 for regression, confusion matrices, precision, recall, F1, ROC and PR curves, calibration and cost-based choices.

A model is only as good as the question you ask about it. "How accurate is it?" sounds like one question but hides several: accurate on average or in the worst cases? Accurate at ranking, or at giving trustworthy probabilities? Accurate for all customers, or only the common ones? The metric you choose decides which model looks best, so it should come from the decision the model will support, not from habit.

This lesson covers the main regression and classification metrics, how to read a confusion matrix, why ROC and precision-recall curves tell different stories, what calibration means, and how to turn error costs into a threshold. The data splits come from train/test splits and cross-validation; all scores in this lesson are computed on held-out rows.

Regression metrics

For numeric targets, the metrics start from the error of each prediction, actual - predicted:

  • MAE (mean absolute error): the average size of the miss, in the target's own units. Easy to explain: "we are off by 7 units on average".
  • RMSE (root mean squared error): square the errors, average, take the square root. Large errors count much more, so one terrible miss can dominate it. Choose it when big errors are disproportionately costly.
  • R-squared: the share of variance explained relative to predicting the mean. It is 0 for a model no better than the mean, 1 for a perfect one, and it can be negative on test data for a model worse than the mean.

There is no universal winner. The example below compares model A (tiny misses and one large one) with model B (moderate misses everywhere):

Python MAE, RMSE and R-squared can disagree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

By MAE, model A (6.75) beats B (10.00). By RMSE and R-squared, B wins, because A's single large miss is punished heavily. Which is right depends on the use. If a forecast that is off by 40 once causes a stock-out and a lost customer, RMSE reflects your pain. If every unit of error costs the same, MAE does. Always compare against a baseline too: here, predicting the mean gives an MAE of 15.6, so both models beat it.

Percentage metrics such as MAPE look natural but explode when actual values are near zero; use them only when values stay well away from zero.

The confusion matrix and its metrics

For classification, begin with the confusion matrix, which counts predictions against reality. With two classes there are four cells:

flowchart TB
    subgraph M["Confusion matrix"]
        direction LR
        A["True negative<br/>(TN)<br/>said no, was no"]
        B["False positive<br/>(FP)<br/>said yes, was no"]
        C["False negative<br/>(FN)<br/>said no, was yes"]
        D["True positive<br/>(TP)<br/>said yes, was yes"]
    end

Each prediction lands in one of four cells; every classification metric is arithmetic on these counts.

From the four counts:

  • Accuracy = (TP + TN) / all. Misleading when classes are imbalanced.
  • Precision = TP / (TP + FP). When we say yes, how often are we right?
  • Recall = TP / (TP + FN). Of all real positives, how many did we find?
  • F1 = the harmonic mean of precision and recall; it is low if either is low. It ignores true negatives.

Precision matters when false alarms are expensive (spam filters that hide real mail). Recall matters when misses are expensive (screening for disease). The two pull against each other as the threshold moves, as you saw in logistic regression.

ROC and precision-recall curves

A classifier produces scores, and every threshold gives a different confusion matrix. Two curves summarize all thresholds at once:

  • The ROC curve plots the true positive rate (recall) against the false positive rate (the share of real negatives wrongly flagged). ROC AUC is the area under it: the probability that a random positive gets a higher score than a random negative. 0.5 is a coin flip, 1.0 is perfect ranking.
  • The precision-recall (PR) curve plots precision against recall. Its summary, average precision (AP), is much more sensitive to how rare the positive class is: a random model's AP equals the positive rate.

When positives are rare, ROC AUC can look comfortable while precision is poor, because the huge number of negatives keeps the false positive rate small even when false positives outnumber true positives. Look at the PR curve then.

Python ROC and PR curves on a 11%-positive problem

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model ranks well: ROC AUC is 0.88. But at the 0.5 cutoff it finds only 38 of the 136 real positives (recall 0.28). The PR curve shows the same weakness more honestly: average precision is 0.54, far above the 0.11 baseline but far from perfect, and the curve falls fast as recall increases. A high AUC does not mean the model suits your operating point.

Calibration

A calibrated model's probabilities mean what they say: among cases scored 0.3, about 30% should be positive. This matters whenever you use the number itself, for example to compute expected revenue, to rank by risk, or to show a probability to a user. Ranking metrics (AUC) cannot see calibration.

To check, group predictions into bins and compare the average predicted probability with the observed positive rate in each bin (calibration_curve). The Brier score, the mean squared difference between probability and outcome, summarizes both calibration and sharpness (lower is better). The next example compares a logistic model, a random forest and a logistic model with class_weight="balanced", which we saw distorts probabilities.

Python Calibration table: do predicted probabilities match reality?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For the plain logistic model, predicted and observed lines nearly coincide (0.54 predicted versus 0.57 observed in the fourth bin). The balanced model inflates every bin (0.19 predicted versus 0.14 observed in the second bin) and its Brier score is worse (0.153 against 0.144). If you need trustworthy probabilities from a model that lacks them, CalibratedClassifierCV can recalibrate it on separate data.

From metrics to costs

The best threshold is not a statistical question; it depends on what mistakes cost. Suppose a missed case (a false negative) costs 100 and a false alarm (a false positive) costs 5. Compute the total cost at each threshold and pick the cheapest. Theory says the best cutoff is close to cost_FP / (cost_FP + cost_FN), here 5/105, about 0.048.

Python Choosing a threshold by total cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The cheapest threshold is 0.05, almost exactly the theoretical value, and its total cost (2,775) is about 72% lower than at the default 0.5 (9,860). The classic "accuracy" ranking would have picked the opposite end. The costs are invented here; in a real project, getting them from the business, even roughly, is one of the most valuable things an analyst can do.

Choosing metrics in practice

  • Start with the decision: what will someone do with the output, and what does each kind of error cost?
  • For regression, report MAE (typical miss) and RMSE (sensitivity to big misses), plus a baseline.
  • For imbalanced classification, report precision, recall and the confusion matrix at a chosen threshold, plus PR AUC; use ROC AUC for a quick ranking summary.
  • Report calibration when probabilities are used directly.
  • Report scores per segment (region, customer type, device). An overall average can hide a group the model serves badly; see explaining model behaviour.
  • Use cross-validated means and spreads, not one number.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which metric punishes a single very large error the most?

    RMSE squares the errors before averaging, so large errors have outsized influence. MAE treats all errors in proportion to their size.
  2. 2. A cancer screening test must miss as few real cases as possible, even at the price of extra follow-up tests. Which measure should you emphasize?

    Recall is the share of real positives that are found. Missing a case is the costly error here, so recall comes first, with precision watched as the price.
  3. 3. Positives are 1% of the cases. Which plot gives the more honest view of performance on the positive class?

    ROC hides how many false positives there are among the flagged cases when negatives are plentiful. The PR curve shows precision directly and is anchored to the positive rate.
  4. 4. A model has an excellent ROC AUC, but among cases it scores at 0.30 only 10% turn out positive. What is the issue?

    AUC measures ranking only. Calibration asks whether a score of 0.30 means about 30% positives. Recalibrate if you use the probability value.
  5. 5. A missed fraud costs 200 and a false alarm costs 2. Roughly where does theory put the best probability threshold?

    cost_FP / (cost_FP + cost_FN) = 2 / 202, about 0.01. When misses are very expensive, flag even low-probability cases.

Summary

  • Choose the metric from the decision and the cost of each error, then compare against a simple baseline.
  • MAE is the typical miss; RMSE and R-squared weigh large misses more; they can rank models differently.
  • The confusion matrix is the base of accuracy, precision, recall and F1; accuracy misleads when classes are imbalanced.
  • ROC AUC measures ranking; PR curves and average precision are more informative when positives are rare.
  • Calibration checks that predicted probabilities match observed frequencies; reweighting classes breaks it.
  • With known costs, choose the threshold that minimizes total cost instead of defaulting to 0.5.

Next, bundle preprocessing and models into one safe object in pipelines and hyperparameter tuning.