Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 73 ยท Training in Practice

Evaluating Models

Measure a model honestly: confusion matrix, precision, recall, F1, ROC and AUC, calibration, imbalanced data, regression metrics and validation strategy.

A trained network gives you predictions. Whether those predictions are any good depends entirely on how you measure them. The loss you minimize during training is rarely the number you care about in the real world, and a single headline figure such as "97% accuracy" can hide a model that is useless.

This lesson covers the metrics you will use again and again, written from scratch so you see what they count: the confusion matrix, precision, recall and F1, the ROC curve and AUC, calibration, and the common regression metrics. It ends with how to choose a validation strategy so the numbers you report are trustworthy. Splitting data into train, validation and test sets is introduced in Overfitting, Underfitting and Regularization; here we focus on what to compute on the held-out part.

Four outcomes for every prediction

True positive Said yes, the answer was yes.
False positive Said yes, the answer was no.
False negative Said no, the answer was yes.
True negative Said no, the answer was no.

Almost every classification metric is a ratio of these four counts.

1 / 2

The confusion matrix

For a binary problem, compare every prediction with the true label and count the four outcomes: true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN). Arranged in a 2 by 2 table, that is the confusion matrix. The metrics follow directly:

  • Accuracy = (TP + TN) / total: the fraction of all predictions that were right.
  • Precision = TP / (TP + FP): of the cases flagged positive, the fraction that truly were.
  • Recall (sensitivity) = TP / (TP + FN): of the truly positive cases, the fraction found.
  • F1 = 2 * precision * recall / (precision + recall): the harmonic mean, which is low whenever either input is low.

Which one matters depends on the cost of each mistake. A spam filter wants high precision (a real email in the spam folder is costly). A cancer screening test wants high recall (a missed case is costly), accepting extra false alarms that a second test can clear.

Python Metrics from scratch, checked against scikit-learn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note the layout scikit-learn uses: rows are the true classes and columns the predicted ones, so the top-left cell is the true negatives. Other tools transpose it, so always check the axis labels.

Why accuracy lies on imbalanced data

Suppose 1 transaction in 100 is fraud. A "model" that always answers "not fraud" is 99% accurate and catches zero fraud. When classes are imbalanced, accuracy mostly measures how common the majority class is. Look at precision, recall and F1 for the rare class instead, and compare against a trivial baseline.

Python The always-negative model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The useless model scores the higher accuracy and the working detector the lower one, which shows why accuracy alone is a trap here. Remedies for imbalance include reporting per-class metrics, weighting the loss so rare examples count more, resampling, and choosing a decision threshold deliberately.

Thresholds, the ROC curve and AUC

Networks usually output a probability or score, and a threshold (often 0.5) turns it into a yes or no. Moving the threshold trades precision against recall: a lower threshold flags more cases, catching more positives but raising false alarms.

The ROC curve shows this trade-off at every threshold at once. For each threshold it plots the true positive rate (the same as recall) against the false positive rate (FP / (FP + TN)). A random scorer follows the diagonal; a perfect one hugs the top-left corner. The AUC is the area under the curve: 0.5 means chance, 1.0 means perfect ranking. It has a neat meaning: the probability that a randomly chosen positive example gets a higher score than a randomly chosen negative one. AUC judges the ranking quality independent of any threshold, but on very imbalanced data a precision-recall curve is often more informative.

Python Building a ROC curve by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three AUC numbers agree, because they are three views of the same quantity.

Calibration: do the probabilities mean anything?

If a model says "80% likely" about 100 cases, about 80 of them should be positive. A model whose probabilities behave like this is calibrated. Many networks, especially large ones trained with cross-entropy for a long time, become overconfident: they say 99% and are right 85% of the time. The check is simple: bucket predictions by predicted probability and compare each bucket's average prediction with the fraction of real positives. Plotted, that is a reliability diagram. Fixes such as temperature scaling rescale the logits after training.

Python A reliability table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the calibrated rows, the two columns match. In the overconfident rows, predictions in the extreme bins are more extreme than the observed frequency.

Regression metrics

For predicting numbers, the standard metrics describe the size of the error:

  • MAE (mean absolute error): the average size of the miss, in the target's units. Easy to explain.
  • MSE and RMSE: squaring punishes large errors more; the square root of MSE returns to the target's units.
  • R squared (Rยฒ): the fraction of the target's variance the model explains. 1 is perfect, 0 is no better than always predicting the mean, and it can be negative.

Always compare with a baseline such as predicting the training mean. MAE reacts less to outliers than RMSE, so a big gap between the two signals a few very large errors.

Python Regression metrics and the outlier effect

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two big misses lift RMSE far more than MAE, and drag Rยฒ down.

Choosing a validation strategy

Metrics are only as honest as the data they are computed on. Pick the strategy that matches how the model will be used:

  • Hold-out split: one train, validation and test split. Fine when you have plenty of data. Tune on validation; touch the test set once, at the end.
  • Stratified split: keeps the class proportions the same in every part. Essential when a class is rare.
  • K-fold cross-validation: train K times, each time validating on a different fold, and average. Gives a steadier estimate on small data, but costs K trainings, so it is used less for large networks.
  • Time-ordered split: for time series, always train on the past and validate on the future. Random shuffling leaks future information.
  • Group split: when several rows belong to the same patient, user or document, keep each group entirely on one side, or the model is tested on people it has already seen.

Never tune hyperparameters, thresholds or architecture on the test set. Every decision you make by looking at test results quietly turns it into a second validation set, and the final score becomes optimistic.

Python Stratified splitting keeps rare classes represented

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a plain split the number of rare positives in the test set swings from run to run (sometimes there are none, which makes recall undefined), while the stratified split always gives the same count, and the folds each get an equal share.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A fraud detector must catch as many real frauds as possible, even at the cost of some false alarms. Which metric should you prioritize?

    Recall is TP / (TP + FN), the fraction of real frauds found. Accepting extra false alarms to reduce false negatives means favoring recall.
  2. 2. A dataset is 99% negative. A model that always predicts negative has what accuracy and recall?

    It is right on every negative (99%) and finds none of the positives (recall 0). This is why accuracy misleads on imbalanced data.
  3. 3. What does an AUC of 0.5 mean?

    AUC is the probability that a random positive outscores a random negative. At 0.5 the scores carry no ranking information.
  4. 4. You are forecasting next week's sales from past weeks. Which validation split is appropriate?

    A time-ordered split mimics real use, where the model never sees the future. Random shuffling leaks information from after the prediction date.
  5. 5. A model's calibration is poor if...

    Calibration is about whether predicted probabilities match observed frequencies, independent of how accurate the hard labels are.

Summary

  • Count true and false positives and negatives in a confusion matrix; accuracy, precision, recall and F1 are ratios of those counts.
  • Pick the metric that matches the cost of each mistake, and on imbalanced data never rely on accuracy alone.
  • A threshold converts scores to labels; the ROC curve and AUC summarize performance across all thresholds.
  • Calibration asks whether predicted probabilities match real frequencies; overconfidence is common in networks.
  • For regression use MAE, RMSE and Rยฒ, always next to a simple baseline.
  • Validate in a way that mirrors deployment: stratified, grouped or time-ordered splits, and keep the test set untouched until the end.

Next: Debugging Training Runs.