Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 40 ยท Imbalanced Data and Custom Losses

Thresholds and Metrics for Imbalanced Data

Pick the decision threshold of a boosted classifier with precision-recall curves, F-scores and costs, using out-of-fold predictions so the choice is honest and stable.

A boosted classifier outputs a score between 0 and 1. Turning that score into a decision ("flag this transaction") needs a threshold, and the default of 0.5 is rarely right for imbalanced problems. When positives are 2% of rows, a well-trained model gives most true positives a probability of 0.2 or 0.3, and a 0.5 cut-off misses them.

This lesson shows how to see the trade-off, pick a threshold on purpose, and avoid the most common mistake: choosing it on the same data you report results on.

ROC-AUC can flatter a weak model

On rare-positive data the ROC curve uses false positive rate, whose denominator is the huge negative class. A model can have a 0.9 AUC and still produce many false alarms per true alert. The precision-recall curve exposes this because precision counts false positives in absolute numbers:

Python ROC-AUC versus PR-AUC on rare positives

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the ROC-AUC was 0.90 but the PR-AUC only 0.35 (a random model scores 0.022). Catching 50% of the positives needed a threshold near 0.08 with precision 0.24; catching 90% pushed precision down to 0.06, meaning 16 false alarms per true hit. The ROC-AUC looks excellent while the precision needed to catch most positives is low. A random model has a PR-AUC equal to the positive rate, so compare the PR-AUC against that baseline, not against 1.0.

Pick a threshold on purpose

There are three common recipes.

  1. Maximize F1 (or F-beta when recall matters more: beta=2).
  2. Meet a business constraint: "the highest recall such that precision is at least 90%", or "flag at most 100 cases a day".
  3. Minimize expected cost using the costs of a missed positive and a false alarm.

The cost approach follows directly from the confusion matrix. Total cost is cost_FN * FN + cost_FP * FP; scan all thresholds and pick the cheapest:

Python Choosing the threshold on out-of-fold predictions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The important structure is in the code, not the numbers. The threshold is chosen on out-of-fold predictions of the training set, then the final model is scored once on a test set that played no part in the choice. A threshold found on the test set would report a score that is slightly too good, because you tuned a parameter on it.

In one run the F1-optimal threshold was 0.23 and the cost-optimal one 0.10, and the test costs were 497 (default 0.5), 404 (F1 threshold) and 363 (cost threshold). So the cost-optimal threshold is well below 0.5 when a missed positive costs ten times a false alarm, the default gives the worst cost of the three, and the F1-optimal threshold sits in between. The cost threshold has the lowest cost but not the best F1: each threshold optimizes what you ask for.

Thresholds are specific to a model and a score distribution. If you retrain, change the weights or calibrate the model, pick the threshold again.

Beyond one threshold

  • Top-k. If a team can review 200 cases a day, rank by score and take the top 200. This needs no threshold and stays stable as the score distribution drifts.
  • Several thresholds. Use a high threshold for automatic action, and a medium one for human review.
  • Precision at k, recall at k, lift. Report these when the business thinks in lists rather than probabilities.
  • Calibration. If you need the probability itself (expected loss, pricing), check a calibration curve (sklearn.calibration.calibration_curve) and calibrate with CalibratedClassifierCV when needed. Models trained with scale_pos_weight need this most (previous lesson).
flowchart LR
  A["Train model"] --> B["Out-of-fold scores<br/>on training data"]
  B --> C["Choose threshold<br/>(F1, constraint or cost)"]
  A --> D["Final model"]
  C --> E["Evaluate once on the test set"]
  D --> E

Keep the data that picks the threshold apart from the data that reports performance.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a threshold of 0.5 often a poor default for rare positives?

    With few positives, even a good model gives many true positives modest probabilities, so 0.5 leaves them unflagged.
  2. 2. A model has ROC-AUC 0.95 but a PR-AUC of 0.25 on data with 2% positives. How should you read this?

    PR-AUC punishes false positives in absolute numbers. With a 2% baseline, 0.25 is a real improvement but precision is still modest.
  3. 3. On which predictions should the threshold be tuned?

    Tuning on predictions the model has not trained on keeps the choice honest, and the test set stays untouched for the final report.
  4. 4. A missed positive costs 10 times a false alarm. What usually happens to the cost-optimal threshold?

    When misses are expensive you accept more false alarms, which means flagging at lower scores.

Summary

  • ROC-AUC can hide poor precision on rare positives; use PR curves and compare against the positive rate.
  • Choose a threshold by F-score, a business constraint, or expected cost, and re-pick it whenever the model changes.
  • Use out-of-fold predictions to choose it, and the test set only once to report.
  • Top-k lists and calibration are alternatives when a single cut-off is not enough.

Next: Custom Objective Functions