Skip to content
Tutorials Center
My learning

Topics

Lesson 49 of 111 ยท Model Evaluation and Tuning

Probability Calibration and Uncertainty

Check whether predicted probabilities can be trusted with reliability curves, Brier score and log loss, fix them with CalibratedClassifierCV, and build prediction intervals.

A classifier that says "90% likely" should be right about nine times out of ten when it says that. A model whose probabilities behave that way is calibrated. Many decisions depend on it: a bank that prices risk, a doctor who weighs a screening result, a system that routes only low-confidence cases to a human. Ranking cases correctly (high AUC) is not enough when someone reads the number itself.

Regression has a matching question. A single predicted number hides how uncertain it is, so we often want a prediction interval: a range expected to contain the true value 90% of the time. This lesson shows how to measure calibration, repair it, and build intervals both by hand and with quantile regression. It follows Thresholds, ROC and Precision-Recall Curves and Classification Metrics.

Good ranking is not good probability

Discrimination Do positives score above negatives?
Calibration Does 70% mean 70 in 100?

AUC measures the first. Reliability curves and Brier score measure the second.

1 / 2

Reliability curves

To check calibration, group the test predictions into bins by predicted probability. In each bin, compare the average predicted probability with the observed fraction of positives. Plot one against the other: a perfectly calibrated model lies on the diagonal. calibration_curve computes the points. With strategy="quantile" every bin holds the same number of samples, which avoids empty bins at the extremes.

Different algorithms have characteristic shapes. Logistic regression optimizes log loss directly, so it is usually close to the diagonal. Naive Bayes multiplies many "independent" pieces of evidence even when they are correlated, so it piles up too much confidence and pushes probabilities toward 0 and 1. Random forests average many trees, which pulls probabilities away from 0 and 1 and tends to under-confidence near the ends.

Two numbers for probability quality

  • Brier score: the mean squared difference between the predicted probability and the actual outcome (0 or 1). Lower is better; always predicting 0.5 gives 0.25.
  • Log loss (cross-entropy): the average of -log of the probability given to the true class. Lower is better, and it punishes confident mistakes severely: saying 0.99 for something that does not happen costs far more than saying 0.7.

Both mix discrimination and calibration, so a model with great accuracy can still have a worse score than a plainer, better-calibrated one. Look at them together with the reliability curve.

Python Reliability curves for three models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the plot like this: a curve below the diagonal at the right means the model claims more certainty than it earns (overconfident); a curve that is too flat or sags at the ends means it is underconfident. Notice that the forest has the best accuracy and Brier score here even though its curve is not on the diagonal, while naive Bayes has similar accuracy to logistic regression but a much worse log loss. Accuracy hides exactly this.

Fixing calibration with CalibratedClassifierCV

A calibrator is a small model that maps a classifier's raw scores to better probabilities. scikit-learn offers two:

  • method="sigmoid" (Platt scaling) fits a logistic curve to the scores. It has two parameters, so it works with little data and suits distortions that look S-shaped.
  • method="isotonic" fits a monotonic step function. It is more flexible but needs plenty of data (thousands of samples) and can overfit when there is little.

CalibratedClassifierCV wraps your classifier and uses cross-validation so that the calibrator never learns from the same rows the base model was trained on. It does cv-fold splits, trains a clone on each training part, calibrates on the held-out part, and averages the results. Calibration should be evaluated on data the calibrator has not seen, so keep your test set outside.

Python Before and after calibration

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Log loss drops sharply after calibration and the curves move toward the diagonal, while accuracy barely changes. That is the typical outcome: calibration reshapes the probabilities and leaves the ranking, and therefore most of the accuracy, alone. If you use a threshold other than 0.5 (see Imbalanced Classes), a calibrated probability also makes that threshold easier to reason about, because "act when the risk exceeds 20%" then means what it says.

Prediction intervals for regression

A regression model returns one number, but the truth scatters around it. A prediction interval at 90% is a range that should contain the actual value in about 90% of future cases. It is not the same as a confidence interval for the model's coefficients, which describes uncertainty about the fitted line, not about individual outcomes.

The simplest honest method works with any model:

  1. Fit the model on the training data.
  2. Compute the residuals (actual minus predicted), ideally on data the model did not fit.
  3. Take the 5th and 95th percentiles of those residuals.
  4. For a new prediction p, report p + low to p + high.

This gives an interval of constant width, which is fine when the noise is the same everywhere. When the noise grows with the input (a common situation), the constant band is too narrow where the data are noisy and too wide where they are calm.

Quantile regression

Ordinary least squares predicts the mean. Quantile regression predicts a chosen quantile instead: fit one model for the 5th percentile and one for the 95th, and the pair forms an interval whose width can change with the input. In scikit-learn, QuantileRegressor(quantile=0.95, alpha=0) fits a linear model for that quantile, and gradient boosting accepts loss="quantile" for non-linear shapes.

Python Constant-width interval versus quantile regression

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Overall both intervals cover close to the 90% target, which looks fine until you split the coverage by region. The constant band covers everything at low x and misses clearly too many points at high x. The quantile pair follows the funnel. Always check coverage in slices, not only overall: an interval that is right on average can be wrong exactly where it matters.

Intervals are only as good as their assumptions. Quantile estimates from small samples are noisy, and neither method accounts for a shift between training and future data. Check coverage on held-out data and keep checking it after deployment.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A model predicts 0.8 for 1,000 customers, and 800 of them actually convert. What does that say?

    Calibration means predicted probability matches observed frequency. A group predicted at 0.8 that converts 80% of the time is calibrated there. It says nothing directly about accuracy or AUC.
  2. 2. On a reliability curve, which shape signals an overconfident model?

    An overconfident model pushes probabilities toward the extremes: when it says 0.9 the real rate is lower, and when it says 0.1 the real rate is higher, which gives an S-shaped curve flatter than the diagonal.
  3. 3. Why does CalibratedClassifierCV use cross-validation internally?

    Scores on the base model's own training rows are overly confident. Calibrating on held-out folds gives the calibrator realistic scores to learn from.
  4. 4. You build a 90% prediction interval and its coverage is 90% overall but only 60% for the highest-value cases. What is the most likely cause?

    A fixed-width interval fits the average noise level. If the noise is larger for large inputs, it under-covers there and over-covers elsewhere. An interval whose width varies with the input, such as quantile regression, handles it.

Summary

  • A calibrated model's probabilities match observed frequencies; ranking quality (AUC) and calibration are different properties.
  • Check calibration with calibration_curve, and quantify it with the Brier score and log loss, both lower-is-better.
  • Logistic regression is usually well calibrated; naive Bayes tends to be overconfident; random forests tend to be underconfident near 0 and 1.
  • CalibratedClassifierCV with sigmoid (small data) or isotonic (lots of data) repairs probabilities without changing the ranking much.
  • Prediction intervals for regression can be built from residual percentiles, or with quantile regression when the spread varies; check coverage in slices.

Next: Scaling and Normalization.