Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 40 ยท Training and Evaluation

Metrics and Validation

Measure Keras models honestly: accuracy traps, confusion matrix, precision, recall, F1, ROC AUC, regression metrics, validation splits and k-fold, in NumPy.

The loss guides training, but it is rarely the number you care about. Metrics are the measurements humans use to judge a model: how often is it right, how many fraud cases does it catch, how far off are its price estimates? And validation is how you get those numbers without fooling yourself. This lesson covers the metrics to know, the traps (especially with imbalanced classes), and validation schemes beyond a single split.

Loss versus metric

  • The loss must be differentiable and is minimized by the optimizer. Cross-entropy is a typical loss.
  • A metric can be anything, including non-differentiable ones such as accuracy or F1. Metrics are computed and reported, never trained on.

You can have a model with a falling loss and a flat accuracy, or the reverse; always check the metric you actually need.

Classification metrics

Everything starts from the confusion matrix for a yes/no problem:

predicted positive predicted negative
actually positive true positive (TP) false negative (FN)
actually negative false positive (FP) true negative (TN)

From these four counts:

  • Accuracy = (TP + TN) / total. Share of correct predictions.
  • Precision = TP / (TP + FP). When it says positive, how often is it right?
  • Recall (sensitivity) = TP / (TP + FN). Of the real positives, how many did it find?
  • F1 = harmonic mean of precision and recall: 2PR / (P + R).

Which matters depends on the cost of mistakes. A spam filter wants high precision (do not bin real mail); a cancer screening test wants high recall (miss nobody).

The accuracy trap

Suppose 1 in 100 transactions is fraud. A model that always says "not fraud" is 99% accurate and completely useless. Run the numbers:

Python Stand-in: accuracy versus precision and recall (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "always no" model has about 99% accuracy and zero recall; the detector scores lower accuracy yet is the only one doing any work. For imbalanced data, report precision, recall, F1 or AUC, not accuracy alone.

Thresholds, ROC and AUC

A classifier outputs a probability; the decision threshold (default 0.5) turns it into a label. Moving the threshold trades precision for recall: a lower threshold catches more positives (higher recall) and raises more false alarms (lower precision).

The ROC curve plots recall (true-positive rate) against false-positive rate across all thresholds, and AUC is the area under it: 0.5 is a coin flip, 1.0 is perfect. It equals the probability that a randomly chosen positive example is scored higher than a randomly chosen negative one, which gives a simple way to compute it.

Python Stand-in: thresholds and AUC by ranking (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Multi-class metrics

With more than two classes, accuracy is still defined (SparseCategoricalAccuracy), and the confusion matrix becomes k x k: row = true class, column = predicted class. Precision and recall are computed per class and then averaged: macro (each class counts equally) or micro (each example counts equally).

Python Stand-in: a 3-class confusion matrix and per-class recall (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Regression metrics

For numeric predictions:

  • MAE: average absolute error, in the target's units. Easy to explain.
  • RMSE: square root of MSE, also in the target's units but weighting big errors more.
  • R-squared: fraction of the target's variance the model explains: 1 - SS_res / SS_tot. 1 is perfect, 0 equals predicting the mean, and it can go negative.
Python Stand-in: MAE, RMSE and R-squared (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Metrics in Keras

Python
model.compile(
    optimizer="adam",
    loss="binary_crossentropy",
    metrics=[
        "accuracy",
        keras.metrics.Precision(name="precision"),
        keras.metrics.Recall(name="recall"),
        keras.metrics.AUC(name="auc"),
    ],
)

For regression use keras.metrics.MeanAbsoluteError(), RootMeanSquaredError() and R2Score(). Keras metrics accumulate over the batches of an epoch, so the reported number covers the whole epoch (not just the last batch). For quantities Keras lacks (macro F1, confusion matrix), compute them after training from model.predict with NumPy or scikit-learn:

Python
from sklearn.metrics import classification_report, confusion_matrix

pred = model.predict(x_test).argmax(axis=1)
print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred))

Validation strategies

Hold-out split. The standard: train, validation, test (see Data Basics for Keras). Pass validation_data= or validation_split=. Fast and enough for large datasets.

Stratified split. For imbalanced classes, keep the class proportions the same in each split so the validation set has enough rare examples.

k-fold cross-validation. For small datasets a single validation split is noisy. Split the data into k folds (often 5), train k times, each time validating on a different fold, and average the scores. Costly (k trainings) but more reliable, and the spread across folds shows how stable the result is.

Time-ordered split. For time series, validate on data later than the training data. Random shuffling would leak the future into the past (Chapter 6).

Python Stand-in: building k-fold index sets (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In Keras, loop over the folds, build a fresh model for each, fit and record evaluate. Never reuse a trained model across folds; its weights have already seen the other folds.

Reading learning curves

Plot training and validation loss per epoch:

  • Both high and flat: underfitting. Use a bigger model, train longer, or lower the regularization.
  • Training loss falls but validation loss rises: overfitting (Overfitting and Callbacks).
  • Both fall and level off close together: healthy.
  • Validation noisier than training: expected when the validation set is small.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. 1 percent of cases are positive. A model predicts "negative" for everything. What is its accuracy and recall?

    It is right on the 99 percent negatives but finds no positives, so accuracy looks great while recall is 0.
  2. 2. When is high recall the priority?

    Recall measures how many real positives you find, so it matters when misses are expensive.
  3. 3. What does an AUC of 0.5 mean?

    AUC is the probability a random positive outranks a random negative; 0.5 is chance level.
  4. 4. Why use k-fold cross-validation?

    Averaging over k different validation folds reduces the noise of a single split.
  5. 5. Training loss keeps falling while validation loss rises. What is happening?

    The model is memorizing training data and generalizing worse.

Summary

  • Loss guides training; metrics are what humans read and are not trained on.
  • Use precision, recall, F1 and AUC when classes are imbalanced; accuracy alone can mislead.
  • The threshold trades precision against recall; AUC summarizes all thresholds.
  • Regression: MAE, RMSE, R-squared.
  • Validate with hold-out, stratified, k-fold or time-ordered splits, and read learning curves.

Next: Overfitting and Callbacks