Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 38 ยท Evaluation and Metrics

Classification Metrics: Precision, Recall, F1

Evaluate classifiers properly: confusion matrix, accuracy pitfalls, precision, recall, F1, macro versus micro averaging, thresholds and a Trainer compute_metrics function.

Accuracy, the share of correct predictions, is the first number everybody reports and the first one that misleads. Suppose 2% of emails are phishing. A model that predicts "safe" for everything is 98% accurate and completely useless. To see what a classifier really does, you need to break errors down by kind, which is what the confusion matrix, precision, recall and F1 are for.

In this lesson you build all of them from scratch, learn when each one matters, and wire them into a compute_metrics function for the Trainer.

The confusion matrix

For a two-class problem call one class positive (the thing you want to find: phishing, disease, toxic comment). Every prediction falls into one of four cells:

Predicted positive Predicted negative
Actually positive True positive (TP) False negative (FN)
Actually negative False positive (FP) True negative (TN)

From these four numbers come the standard metrics:

  • Accuracy = (TP + TN) / all
  • Precision = TP / (TP + FP): of the items I flagged, how many were truly positive? Cares about false alarms.
  • Recall (sensitivity) = TP / (TP + FN): of the truly positive items, how many did I find? Cares about misses.
  • F1 = harmonic mean of precision and recall = 2PR / (P + R). The harmonic mean punishes imbalance: F1 is low when either P or R is low.
Python From confusion matrix to F1

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The do-nothing model scores 0.90 accuracy and an F1 of zero. The cautious model has perfect precision but misses most positives; the eager model finds almost all of them at the price of false alarms. Which is better depends on the cost: a cancer screen prefers high recall, a spam filter that hides important mail prefers high precision.

More than two classes: averaging

With several classes you compute precision, recall and F1 per class (treating each class as "positive" against the rest), then combine them:

  • Macro: the plain average over classes. Every class counts equally, so rare classes matter as much as common ones. Use it when rare classes are important.
  • Micro: pool all TPs, FPs and FNs across classes and compute once. For single-label classification micro-F1 equals accuracy. Dominated by frequent classes.
  • Weighted: average per-class scores weighted by how often each class occurs. A compromise that still hides rare-class failures.
Python Macro, micro and weighted F1

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The rare class "science" is never predicted correctly, which drags macro-F1 down far more than micro-F1, which equals accuracy here. If you reported only accuracy you would never see the failure.

Thresholds: moving along the precision-recall trade-off

A classifier outputs a probability; "positive" means probability above a threshold, 0.5 by default. Lowering it finds more positives (recall up) but adds false alarms (precision down). Raising it does the opposite. Choose the threshold on a validation set to fit your costs, not by habit. Metrics that summarize all thresholds are ROC-AUC and average precision (area under the precision-recall curve); for rare positives, average precision is the more informative of the two.

Python Sweeping the decision threshold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

compute_metrics for the Trainer

Python
import numpy as np
from sklearn.metrics import precision_recall_fscore_support, accuracy_score

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    preds = np.argmax(logits, axis=-1)
    p, r, f1, _ = precision_recall_fscore_support(
        labels, preds, average="macro", zero_division=0)
    return {"accuracy": accuracy_score(labels, preds),
            "precision": p, "recall": r, "f1": f1}

For multi-label problems, the model gives one logit per label; apply a sigmoid and threshold each label independently (usually 0.5), then compute micro or macro F1 over the label matrix. Use metric_for_best_model="f1" so the best checkpoint is picked by F1 rather than by loss. Remember that metrics are noisy on small evaluation sets: a validation set with 50 positives gives recall with an error bar of several percentage points.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A model predicts 'negative' for every email in a dataset where 2% are positive. What are accuracy and recall for the positive class?

    It is right about 98% of the time but finds none of the positives.
  2. 2. What does precision measure?

    Precision = TP / (TP + FP): how trustworthy a positive prediction is.
  3. 3. Why is F1 a harmonic rather than arithmetic mean?

    The harmonic mean is dominated by the smaller of the two numbers.
  4. 4. When do you prefer macro-F1 to micro-F1?

    Macro averaging gives every class the same weight regardless of frequency.
  5. 5. What happens when you lower the decision threshold?

    More items are flagged positive: more true positives found, but more false alarms.

Summary

  • Accuracy hides imbalance; use the confusion matrix to see TP, FP, FN and TN.
  • Precision punishes false alarms, recall punishes misses, F1 balances them.
  • Macro, micro and weighted averaging answer different questions for multi-class tasks.
  • Choose the decision threshold on validation data to match real costs.
  • Use metric_for_best_model with a class-aware metric, and remember small test sets give noisy scores.

Next: Text generation metrics: BLEU, ROUGE, perplexity