Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 111 ยท Classification

Classification Metrics

Why accuracy misleads, how to read a confusion matrix, and how precision, recall, specificity and F1 describe a classifier, with runnable scikit-learn examples.

You trained a classifier and score says 97%. Is that good? It depends entirely on what the model gets wrong and how much each kind of mistake costs. A model that flags every transaction as "not fraud" can score 99.9% accuracy and be completely useless. To judge a classifier honestly you need metrics that look at the kinds of errors it makes.

This lesson starts with the accuracy trap, then builds up the confusion matrix and the metrics derived from it: precision, recall, specificity and F1. We use the model from Logistic Regression, and the next lesson, Thresholds, ROC and Precision-Recall Curves, shows how these numbers change when you move the decision threshold.

Two kinds of mistakes

False positive You said yes, the truth was no.
False negative You said no, the truth was yes.

Which one hurts more depends on the problem.

1 / 3

The accuracy trap

Accuracy is the fraction of predictions that are correct. It is easy to explain and perfectly fine when the classes are balanced and every error costs the same. It falls apart on imbalanced data, where one class is rare. Imagine 1 in 100 patients has a condition. Predicting "healthy" for everyone gives 99% accuracy while catching exactly zero sick patients.

Python A useless model with 98 percent accuracy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The recall of 0.0 exposes the problem: not a single real positive was found. This DummyClassifier is exactly the kind of baseline described in Baselines and Dummy Models, and any real model must beat it on a metric that matters.

The confusion matrix

A confusion matrix counts every combination of true and predicted class. For two classes, with 1 as the "positive" class (the thing you are looking for):

Predicted 0 Predicted 1
Actual 0 true negatives (TN) false positives (FP)
Actual 1 false negatives (FN) true positives (TP)

scikit-learn's confusion_matrix uses exactly this layout: rows are actual classes, columns are predicted classes, in the order of the sorted labels. ravel() flattens it to TN, FP, FN, TP.

Python Reading a confusion matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every metric in this lesson is just arithmetic on these four numbers.

Precision, recall, specificity and F1

  • Precision = TP / (TP + FP). Of everything the model called positive, how much really was? High precision means few false alarms.
  • Recall (also sensitivity or true positive rate) = TP / (TP + FN). Of everything that really was positive, how much did the model find? High recall means few misses.
  • Specificity (true negative rate) = TN / (TN + FP). Of the real negatives, how many were correctly left alone? scikit-learn has no direct function; compute it by hand or use recall_score(..., pos_label=0).
  • F1 = the harmonic mean of precision and recall, 2PR / (P + R). The harmonic mean is dragged toward the smaller of the two, so you cannot get a high F1 by excelling at only one.

Precision and recall pull against each other. A model that calls everything positive has perfect recall and awful precision; a model that only fires when it is nearly certain has high precision and poor recall. Which side to favor is a business question: a cancer screening test wants high recall (missing a case is costly), a spam filter that deletes mail wants high precision (losing a real email is costly).

Python Computing every metric by hand and with scikit-learn

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note: Always say which class is the "positive" one. Precision and recall depend on that choice. In scikit-learn the default is label 1; use pos_label= for anything else.

classification_report: everything at once

classification_report prints precision, recall and F1 for each class, plus the number of true examples (support) and averaged rows. It is the quickest sanity check you can run.

Python classification_report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The report has two averages. The macro average is the plain mean over classes, so a rare class counts as much as a common one. The weighted average weights each class by its support, so it looks closer to accuracy. For a rare-positive problem, read the positive row first. More on averaging in Multiclass and Multilabel Classification.

How class imbalance bends the numbers

When positives are rare, accuracy and specificity look great almost automatically, while precision and recall for the positive class tell the real story. Compare the same model on a balanced and on a heavily imbalanced version of the same task.

Python Same model, different class balance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As the positive share shrinks, accuracy climbs toward 1 while recall collapses: the model learns that predicting "negative" is almost always right. Ways to fight this are reshaping the loss with class weights, moving the threshold, and choosing better metrics. These are the topics of Thresholds, ROC and Precision-Recall Curves and Imbalanced Classes.

Two single-number summaries that survive imbalance: MCC and Cohen's kappa

Accuracy rewards a model for agreeing with the majority class, and F1 ignores the true negatives completely. Two metrics use all four cells of the confusion matrix and correct for chance:

  • The Matthews correlation coefficient (MCC) is the correlation between the predicted and the true labels, from -1 (total disagreement) through 0 (no better than chance) to +1 (perfect). It is high only if the model does well on both classes, so it is a popular single number for imbalanced problems. MCC = (TPยทTN - FPยทFN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN))
  • Cohen's kappa measures agreement between predictions and truth beyond what chance would produce given the class frequencies. It was designed for two human raters, and 0 again means chance level. A common rule of thumb reads 0.4 to 0.6 as moderate, 0.6 to 0.8 as substantial and above 0.8 as almost perfect agreement.

Both live in sklearn.metrics (matthews_corrcoef, cohen_kappa_score). The example compares them with accuracy and F1 on a 99%-negative problem for three "models": one that always says negative, a cautious one that finds a few positives, and a better one. It also draws the confusion matrix as text; with matplotlib you would use ConfusionMatrixDisplay.from_predictions(y_true, y_pred) for a heat map.

Python Accuracy, F1, MCC and kappa on a 99% / 1% problem

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model that never predicts a positive scores 0.99 accuracy and exactly 0 on F1, MCC and kappa, which is the right verdict. The cautious model keeps the highest accuracy (0.991) and has an MCC of 0.42. The better model catches 8 of the 10 positives at the price of 20 false alarms: its accuracy is lower (0.978), because every false alarm costs accuracy on the huge negative class, but MCC (0.47), kappa and F1 all rise, since they reward the extra true positives. Which of the two models you want depends on the relative cost of a miss and a false alarm (thresholds); the point is that accuracy cannot even rank them sensibly. When classes are very unequal, quote MCC (or kappa) next to precision and recall, not accuracy alone.

Choosing a metric

Situation Look at
Balanced classes, equal error costs Accuracy
Missing a positive is costly (disease, fraud) Recall, at an acceptable precision
False alarms are costly (auto-blocking, legal action) Precision
Rare positives and you want one number F1 or average precision
Several classes of different importance Per-class report, macro average

Metrics belong to the problem, not to the model. Decide what matters before you compare models, and compute them on held-out data (see Train, Validation and Test Sets).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A fraud dataset has 0.2% fraud. A model that predicts 'not fraud' for every transaction has what accuracy and recall for fraud?

    It is right on all non-fraud cases (99.8%) but finds none of the frauds, so recall for the positive class is 0.
  2. 2. Which formula is precision?

    Precision asks: of everything predicted positive (TP + FP), how much was truly positive (TP)? TP / (TP + FN) is recall.
  3. 3. In scikit-learn's confusion_matrix, what does the entry in row 1, column 0 (for labels 0 and 1) count?

    Rows are actual classes and columns are predicted classes. Row 1, column 0 holds samples that are actually 1 but were predicted 0: false negatives.
  4. 4. Why is F1 the harmonic mean of precision and recall rather than the plain average?

    A model with precision 1.0 and recall 0.01 has an arithmetic mean near 0.5 but an F1 near 0.02. The harmonic mean punishes lopsided models.

Summary

  • Accuracy hides failures on rare classes; always check what the model does for the class you care about.
  • The confusion matrix (TN, FP, FN, TP) is the source of all the other numbers; rows are actual, columns are predicted.
  • Precision = TP/(TP+FP), recall = TP/(TP+FN), specificity = TN/(TN+FP), F1 = harmonic mean of precision and recall.
  • classification_report gives per-class precision, recall, F1 and support, plus macro and weighted averages.
  • Pick the metric from the cost of each error type before comparing models.

Next up: Thresholds, ROC and Precision-Recall Curves.