Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 63 ยท Text Classification

Evaluating Classifiers

Learn why accuracy misleads on imbalanced data, then compute precision, recall, F1 and the confusion matrix, compare macro and micro averages, and tune thresholds.

"My model is 90% accurate" sounds great until you learn that 90% of the emails in the dataset are not spam. A classifier that predicts "not spam" for everything would score exactly the same, and it would be useless. Choosing the right metric is as important as choosing the right model, because the metric decides what you will optimize and what you will notice going wrong.

This lesson builds the standard toolkit for classifiers: the confusion matrix, precision, recall and F1, the differences between macro, micro and weighted averages, and the decision threshold that trades one kind of error for another. Every number is computed on a small imbalanced spam problem so you can see the metrics disagree.

Four outcomes

True positive Spam, flagged as spam.
False positive Normal mail, flagged as spam.
False negative Spam that slipped through.
True negative Normal mail, left alone.

Every metric in this lesson is built from these four counts.

1 / 2

The confusion matrix and its metrics

For a yes/no problem we call the class we care about the positive class (here, spam). Comparing predictions with the truth gives four counts: true positives (TP), false positives (FP), false negatives (FN) and true negatives (TN). Laid out as a table, they form the confusion matrix. From them:

TEXT
accuracy  = (TP + TN) / total
precision = TP / (TP + FP)
recall    = TP / (TP + FN)
F1        = 2 * precision * recall / (precision + recall)

Precision punishes false alarms; recall punishes misses. F1 is the harmonic mean of the two, which is dominated by the smaller one: you cannot get a high F1 by excelling at one and failing the other.

The example generates 400 messages, only 40 of them spam, and holds out half for testing. We compare a trivial "always ham" baseline, a default logistic regression, and the same model with class_weight="balanced", which makes mistakes on the rare class cost more.

Python Accuracy versus precision and recall

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two numbers tell the whole story. The always-ham baseline scores 0.9, and so does the default logistic regression: it predicted zero spam messages at all, because with so few examples the cheapest way to minimize its loss was to say "ham" every time. Its accuracy looks identical to the baseline and its recall for spam is zero.

The balanced model's accuracy (0.94) is only a little higher, but it behaves completely differently. The matrix reads: 17 spam messages caught (TP), 3 missed (FN), 9 legitimate messages wrongly flagged (FP), and 171 correctly left alone (TN). Precision is 17 / 26 = 0.654, recall is 17 / 20 = 0.85, and F1 is 0.739. These match the spam row of classification_report, which also prints the same numbers for ham by treating that class as the positive one.

Which number matters depends on the cost of each error. For a spam filter that routes mail to a junk folder, false positives (lost real mail) are painful, so precision matters. For flagging toxic comments for human review, missing one is worse than a false alarm, so recall matters. Decide this before you look at scores.

Macro, micro and weighted averages

With more than two classes you get one precision, recall and F1 per class, and you need a way to combine them. The three standard choices answer different questions:

  • Micro averaging pools all the individual decisions first. For single-label problems, micro precision, micro recall, micro F1 and accuracy are all the same number.
  • Macro averaging computes the metric per class and takes the plain average, so each class counts equally, however rare it is.
  • Weighted averaging takes the average of the per-class values weighted by class size.

The next example uses a made-up three-class problem where the rare class is never predicted correctly, so you can see how each average reacts.

Python Micro, macro and weighted F1

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The micro F1 of 0.800 equals the accuracy, as promised. The macro F1 is only 0.560 because the rare class contributes an F1 of 0 and counts as one third of the average, a loud warning that the model ignores a whole class. The weighted F1 (0.758) sits in between: it mostly reflects the big classes. If you care about every class, report macro; if you care about overall volume, micro or weighted; and always look at the per-class numbers too. The zero_division=0 argument says "if a class is never predicted, count its precision as 0 without a warning".

Moving the decision threshold

A probabilistic classifier produces a score, and predict simply cuts it at 0.5. That cut is a choice, not a law. Lowering the threshold flags more messages as spam: recall goes up, precision usually goes down. Raising it does the opposite. You can pick the point that matches your costs.

Python The precision and recall trade-off

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The default model is so cautious that no message ever reaches 0.5, which is why it predicted no spam. Look at the table: at 0.2 the filter catches half the spam (recall 0.500) and 91% of what it flags is really spam (precision 0.909), giving the best F1 in the table (0.645). At 0.1 it catches 95% of the spam but only 30% of its flags are correct. Same model, very different products. When a model is poorly calibrated, tuning the threshold on validation data is often cheaper than retraining; scikit-learn also offers precision_recall_curve to see all thresholds at once.

Good evaluation habits

  • Always compare with a trivial baseline such as "predict the majority class". A model that cannot beat it has learned nothing.
  • Look at the confusion matrix, not only a summary, to see which classes get mixed up.
  • Do not tune on the test set. Choose thresholds and hyperparameters with cross-validation on training data.
  • Report variance. On 200 test messages with 20 spam examples, one extra miss moves recall by 5 points.
  • Use stratified splits so rare classes appear in every fold.
  • Read the errors. Numbers tell you how much is wrong; reading misclassified texts tells you why (see Evaluation, Bias and Responsible NLP).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A dataset has 95% negative examples. A model that always predicts "negative" gets what accuracy?

    It is right on every negative example, which is 95% of the data. This is why accuracy is a poor metric for imbalanced problems.
  2. 2. A spam filter flags 26 messages and 17 of them are really spam. What is its precision?

    Precision is TP divided by everything flagged (TP + FP). Here that is 17 / 26, about 0.654. The fraction 17 / 20 would be recall if there were 20 spam messages in total.
  3. 3. Why is F1 the harmonic mean of precision and recall rather than the ordinary average?

    The harmonic mean drops sharply when either input is small. A model with precision 1.0 and recall 0.05 gets an F1 of about 0.10, not 0.525.
  4. 4. In a three-class problem the rarest class is never predicted correctly. Which average shows the problem most clearly?

    Macro averaging gives every class equal weight, so a class with F1 of 0 pulls the average down hard. Micro F1 equals accuracy and is dominated by the big classes.
  5. 5. What happens to precision and recall if you lower the decision threshold of a probabilistic classifier?

    A lower threshold labels more items as positive. You catch more true positives (higher recall) but also admit more false alarms (lower precision).

Summary

  • Accuracy hides failure on imbalanced data: our default model equalled the "always ham" baseline at 0.9 while catching no spam.
  • The confusion matrix counts TP, FP, FN and TN; precision = TP / (TP + FP), recall = TP / (TP + FN), F1 is their harmonic mean.
  • Micro averaging pools decisions (equals accuracy for single-label data), macro averages classes equally, weighted averages by class size; report per-class numbers as well.
  • class_weight="balanced" and threshold tuning both shift the balance between precision and recall.
  • Choose the metric from the cost of each error, compare against a baseline, and never tune on the test set.

Next, apply these ideas in Sentiment Analysis.