Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 111 ยท Classification

Thresholds, ROC and Precision-Recall Curves

Move beyond 0.5: use predict_proba, decision thresholds, ROC and AUC, precision-recall curves and average precision, and pick a threshold by cost.

Most classifiers do not output a class directly. They output a score or probability, and a threshold turns it into a yes or no. By default that threshold is 0.5, but nothing makes 0.5 special. Lower it and the model says "yes" more often: more positives are found but more false alarms appear. Raise it and the model gets pickier. The threshold is a dial between precision and recall, and you get to choose where to set it.

This lesson shows how to get scores with predict_proba, how the ROC curve and AUC summarize a model across all thresholds, why the precision-recall curve is often more honest for rare positives, and how to pick a threshold from the actual costs of mistakes. It builds on Classification Metrics and uses Logistic Regression.

One model, many classifiers

Threshold Slide it from 0 to 1.
Low threshold Catch more positives, more false alarms.
High threshold Fewer false alarms, more misses.

A curve plots the result for every threshold at once.

1 / 2

Scores, probabilities and thresholds

predict_proba returns the probability of each class; for a binary problem you almost always want column 1. predict is the same as "probability above 0.5". To use another threshold, compare the probabilities yourself.

Python Moving the threshold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As the threshold climbs, fewer cases are flagged, precision generally rises and recall falls. Some models (like LinearSVC) have no predict_proba but offer decision_function, an unbounded score where the natural threshold is 0. Everything below works with either kind of score.

Note: Never choose a threshold by looking at the test set and then report test-set performance for it; that is tuning on test data. Choose the threshold on validation data or with cross-validation (see Train, Validation and Test Sets).

The ROC curve and AUC

The ROC curve (receiver operating characteristic) sweeps the threshold from high to low and plots two rates:

  • True positive rate (TPR) = recall = TP / (TP + FN) on the y-axis.
  • False positive rate (FPR) = FP / (FP + TN) = 1 - specificity on the x-axis.

A model that guesses randomly follows the diagonal. A perfect model goes straight up to the top-left corner. The AUC (area under the curve) squeezes the curve into one number: 0.5 is chance, 1.0 is perfect. It has a nice meaning: AUC is the probability that a randomly chosen positive gets a higher score than a randomly chosen negative. That makes it independent of the threshold and of how the scores are scaled; only the ranking matters.

Python ROC curve and AUC

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Use roc_auc_score with the probabilities or scores, not the hard 0/1 predictions; with hard predictions the curve has a single point and the AUC is meaningless.

Why ROC can flatter you: precision-recall curves

ROC uses the false positive rate, whose denominator is the (huge) number of negatives. When positives are rare, thousands of false positives barely move the FPR, so an ROC curve can look excellent while the model's alerts are mostly wrong. The precision-recall (PR) curve plots precision against recall and ignores true negatives entirely, so it shows the cost of those false alarms directly.

Average precision (average_precision_score) summarizes the PR curve. Its baseline is not 0.5: a random model has an average precision roughly equal to the positive share of the data (0.1 for a 10% positive rate).

Python ROC AUC versus average precision on rare positives

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ROC AUC stays at a comfortable value in both cases, but the average precision drops sharply for the 2% data. Rankings are almost equally good; what changed is that finding a rare positive among many negatives is genuinely harder, and the PR view is the one that tells you so. Use ROC AUC when classes are balanced or when you care about both classes equally; prefer PR and average precision when the positive class is rare and is what you are hunting for.

Choosing a threshold by cost

Curves show trade-offs. A decision needs a price list. Suppose a false negative costs 50 (a missed fraud) and a false positive costs 2 (a manual review). Then the best threshold is the one that minimizes the total cost, not the one that maximizes F1.

Python Picking the cheapest threshold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When a miss is much more expensive than a false alarm, the best threshold sits far below 0.5. For a quick rule of thumb with well-calibrated probabilities, flag a case when p > cost_FP / (cost_FP + cost_FN). Calibration is covered in Probability Calibration and Uncertainty. Remember to choose the threshold on validation data, then confirm on a test set you have not touched.

Putting it together

  1. Fit the model and get scores with predict_proba[:, 1] (or decision_function).
  2. Compare models threshold-free with ROC AUC, or average precision for rare positives.
  3. Convert costs or requirements ("recall of at least 90%") into a threshold on validation data.
  4. Report precision, recall and the confusion matrix at that threshold on the test set.

Next, a very different kind of model that makes its predictions from neighbors: k-Nearest Neighbors.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You lower the decision threshold from 0.5 to 0.2. What typically happens?

    A lower threshold labels more cases positive, so more real positives are found (higher recall) but more negatives are wrongly flagged (lower precision, usually).
  2. 2. What does an AUC of 0.5 mean?

    AUC is the probability that a random positive scores above a random negative. 0.5 means the scores carry no ranking information.
  3. 3. With 1% positives, which metric usually shows the difficulty of the problem more clearly?

    ROC's false positive rate divides by the huge number of negatives, so many false alarms barely register. The PR curve and average precision track them directly.
  4. 4. Which input should you pass to roc_auc_score?

    ROC analysis needs a continuous score to sweep thresholds over. Hard predictions collapse the curve to a single point.

Summary

  • Classifiers produce scores; a threshold (default 0.5) turns them into classes. Lower thresholds raise recall and cut precision.
  • The ROC curve plots TPR against FPR for all thresholds; AUC is the chance that a random positive outranks a random negative (0.5 = chance).
  • For rare positives use the precision-recall curve and average precision; a random model's AP equals the positive share.
  • Choose the threshold on validation data from real costs or requirements, not from the test set.
  • With calibrated probabilities, flag when p > cost_FP / (cost_FP + cost_FN).

Next up: k-Nearest Neighbors.