Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 40 ยท Saving and Using Models

Probability Calibration and Decision Thresholds

Make predicted probabilities trustworthy with CalibratedClassifierCV and calibration curves, then choose a decision threshold with precision-recall trade-offs and costs.

Many classifiers output a number between 0 and 1 and call it a probability. But a random forest saying "0.8" does not necessarily mean the event happens 80% of the time. If you use that number to price insurance, rank leads or decide how many alerts to review, you need it to be calibrated: among all cases scored 0.8, about 80% should be positive.

Separately, predict turns probabilities into classes with a fixed cutoff of 0.5. That cutoff is rarely the best business choice. This lesson shows how to check and fix calibration, and then how to pick a decision threshold deliberately.

Reading a calibration curve

Group predictions into bins by predicted probability, then compare the average prediction in each bin with the actual fraction of positives. A perfectly calibrated model lies on the diagonal.

Python calibration_curve for three models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The Brier score (mean squared error of the probabilities, lower is better) combines calibration and sharpness. Logistic regression is usually reasonably calibrated by construction. Naive Bayes is often over-confident because it treats features as independent, and random forests are often under-confident at the extremes: look at the forest above, where cases it scores 0.16 are positive only about 5% of the time, because averaging many trees keeps scores away from 0 and 1. Brier score also rewards sharpness, so the forest still wins overall: a better ranking can outweigh imperfect calibration.

Calibration in a nutshell

Ranking Is the order of scores right? (AUC)
Calibration Do scores match real frequencies?
Different goals A model can rank well and be miscalibrated.

Calibration only changes the numbers, not much the ranking.

1 / 2

Calibrating with CalibratedClassifierCV

CalibratedClassifierCV wraps a classifier. With cv=5 it trains the base model on part of the data and fits a small calibration map (sigmoid or isotonic) on the held-out part, then averages across folds. The map is learned on data the model did not train on, which is essential: calibrating on the training predictions would just reflect overfitting.

Python Calibrating a random forest and naive Bayes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Log loss improves noticeably for naive Bayes after isotonic calibration, and the forest improves a little, while ROC AUC barely changes: ranking quality stays, the numbers become more honest.

Choosing a decision threshold

Once probabilities are trustworthy, the question becomes: above what probability do we act? The default 0.5 is only optimal when false positives and false negatives cost the same and classes are balanced. Instead, sweep thresholds and look at the trade-off.

Python Precision and recall across thresholds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Lower thresholds find more positives (higher recall) at the cost of more false alarms (lower precision). Neither extreme is "right"; the context decides.

Picking a threshold with costs

If a missed fraud costs 200 and a false alarm costs 5, you can choose the threshold that minimizes total expected cost on validation data.

Python Cost-based threshold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With these costs the break-even probability is 5 / (5 + 200), about 0.024, so the best cutoff is very low: it is worth reviewing many false alarms to catch each fraud. In a real project choose the threshold on validation data (or cross-validated predictions), then report the final cost on the untouched test set. Newer scikit-learn versions also offer TunedThresholdClassifierCV, which does this search for you and wraps the result as a classifier whose predict already uses the tuned cutoff.

Python TunedThresholdClassifierCV

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other ways to handle imbalance

Moving the threshold is often better than resampling. Other tools: class_weight="balanced" (penalizes errors on the rare class more), choosing average precision as your tuning metric, and collecting more positive examples. Note that class_weight shifts the probabilities, so recalibrate if you need real probabilities afterward.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a calibrated probability of 0.7 mean?

    Calibration means predicted probabilities match observed frequencies.
  2. 2. Why should calibration be fitted on data the base model did not train on?

    A model's training-set probabilities are too optimistic. CalibratedClassifierCV uses cross-fitting to avoid that.
  3. 3. A false negative costs far more than a false positive. How should the threshold change from 0.5?

    A lower threshold flags more cases, trading extra false alarms for fewer misses.
  4. 4. Which metric combines calibration and sharpness of probabilities and is lower when better?

    The Brier score is the mean squared error between predicted probabilities and outcomes.

Summary

  • Calibration asks whether predicted probabilities match real frequencies; check with calibration_curve and the Brier score.
  • CalibratedClassifierCV (sigmoid for small data, isotonic for larger) fixes over- or under-confidence without hurting ranking.
  • Choose the decision threshold from your costs or targets, on validation data; TunedThresholdClassifierCV automates it.
  • Consider threshold moves and class weights before resampling imbalanced data.

Next: Project: Predict House Prices