Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 40 ยท Imbalanced Data and Custom Losses

Class Imbalance and Sample Weights

Handle rare positives in XGBoost and LightGBM with scale_pos_weight, sample weights and is_unbalance, and learn what weighting does to recall, precision and probability calibration.

Fraud, churn, defects and diseases are rare. When only 1 in 20 rows is positive, a model that always predicts "negative" is 95% accurate and completely useless. Gradient boosting handles imbalance better than many models, but the default settings still pay little attention to the rare class. This lesson shows the tools you have, what each one really changes, and the trade-offs that are easy to miss.

First, choose metrics that see the minority class

Accuracy hides the problem. Use metrics built for imbalance:

  • PR-AUC (average_precision_score): area under the precision-recall curve, the best single number when positives are rare.
  • ROC-AUC: measures ranking quality and is less sensitive to class ratio, but can look good while precision is poor.
  • Precision, recall and F-scores at a chosen threshold, which the next lesson covers.
  • Log loss or Brier score, when the predicted probabilities themselves must be accurate.

The three weighting tools

All three work by making mistakes on positives cost more in the loss, which scales their gradients and hessians:

Tool Library What it does
scale_pos_weight XGBoost, LightGBM multiplies the weight of every positive row by one number
sample_weight in fit both a weight for every row, so you can also weight by cost or recency
is_unbalance=True LightGBM automatically sets the positive weight to negatives divided by positives

The usual starting point is scale_pos_weight = number of negatives / number of positives. A value that high is often too aggressive, so try a few values, including the square root of the ratio.

Experiment: weights on a 5% positive dataset

Python Effect of scale_pos_weight on ranking, recall and calibration

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One run printed this (the positive rate is 5.7%):

TEXT
none  w=  1.0  PR-AUC=0.674  ROC-AUC=0.889  recall=0.42  precision=0.90  logloss=0.122  mean p=0.048
sqrt  w=  4.1  PR-AUC=0.663  ROC-AUC=0.882  recall=0.51  precision=0.79  logloss=0.132  mean p=0.085
full  w= 16.6  PR-AUC=0.661  ROC-AUC=0.877  recall=0.65  precision=0.53  logloss=0.194  mean p=0.155

Read the rows from top to bottom. As the weight grows:

  • Recall at the 0.5 threshold rises and precision falls. The model flags more rows as positive.
  • The ranking metrics (PR-AUC, ROC-AUC) change little. Weighting mostly shifts scores rather than improving the order, so it is no magic fix for ranking.
  • Probabilities are no longer calibrated. The mean predicted probability climbs far above the true positive rate, and log loss gets worse. A weighted model's predict_proba output is not a real probability.

That last point is the most important practical lesson. If you only need a good ranking or you will pick a threshold anyway, weights are fine. If you need real probabilities (pricing, expected-loss calculations), train without weights and move the threshold instead, or recalibrate afterwards with Platt scaling or isotonic regression (see Pitfalls).

flowchart TD
  A["Rare positive class"] --> B{"What do you need?"}
  B -->|"Good ranking or top-k list"| C["Optional weights,<br/>judge by PR-AUC"]
  B -->|"Decisions at a threshold"| D["Train unweighted,<br/>tune the threshold on validation"]
  B -->|"True probabilities"| E["Unweighted training<br/>+ calibration"]

Weighting is one option among several, and which one is right depends on the output you use.

Row-level weights

sample_weight is more flexible than a single class weight. You can give each row its own cost:

Python Cost-sensitive training with sample_weight

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In one run the plain model missed 68 positives and raised 11 false alarms (cost 555), while the weighted one missed 38 and raised 67 (cost 371). The weighted model misses fewer positives and raises more false alarms, which can be the right trade when a miss is eight times as expensive as a false alarm. The same effect can often be reached by lowering the threshold of the unweighted model, so compare both on validation data.

Other options

  • Resampling. Undersampling the majority class makes training faster but throws information away; oversampling the minority (SMOTE and variants) rarely beats weights for tree ensembles and can leak when done before splitting. If you resample, do it only inside the training fold.
  • Focal and custom losses. A custom objective can emphasize hard examples; see Custom Objective Functions.
  • Stratified splits. Always use stratify=y or StratifiedKFold, so each fold holds a stable share of positives.
  • Early stopping on the right metric. Use aucpr (XGBoost) or average_precision (LightGBM) as the evaluation metric when positives are rare, rather than the default log loss.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A dataset has 1,000 positives and 19,000 negatives. What is a common starting value for scale_pos_weight?

    The usual starting point is the ratio of negatives to positives: 19,000 / 1,000 = 19.
  2. 2. What happens to predicted probabilities when you train with a large scale_pos_weight?

    Weighting pretends positives are more common, so predicted probabilities shift upward. Recalibrate, or use unweighted training and a tuned threshold.
  3. 3. Which metric is the most informative single number for a heavily imbalanced problem?

    Accuracy is dominated by the majority class; PR-AUC focuses on how well positives are ranked and retrieved.
  4. 4. Where must oversampling or SMOTE be applied when you cross-validate?

    Resampling before splitting copies information into validation rows, which leaks and inflates scores.

Summary

  • Judge imbalanced models with PR-AUC, recall and precision at a threshold, not accuracy.
  • scale_pos_weight, sample_weight and LightGBM's is_unbalance all raise the loss weight of positives.
  • Weighting shifts scores and recall more than it improves ranking, and it breaks probability calibration.
  • Train unweighted and move the threshold, or recalibrate, when you need honest probabilities.
  • Stratify splits and resample only inside training folds.

Next: Thresholds and Metrics for Imbalanced Data