Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 111 ยท Classification

Imbalanced Classes

Learn what to do when one class is rare: why accuracy lies, class_weight, undersampling and oversampling by hand, stratified splits, PR metrics and costs.

Fraud, equipment failure, a rare disease, a customer who cancels: in many real problems the thing you care about is rare. If only 5 in 100 transactions are fraudulent, a model that never flags anything is right 95 times out of 100. That model is worthless, yet it scores a "great" accuracy. Imbalanced data is the situation where the usual defaults quietly mislead you.

This lesson shows how to see the problem, and then the practical tools for it: class weights, resampling you can write yourself, careful splitting, and metrics and thresholds that respect what the errors cost. It builds on classification metrics and thresholds, ROC and PR curves.

Rare class, misleading score

Accuracy trap Predicting "never" can score 95%.
Look at recall How many rare cases did we catch?
Costs differ Missing fraud costs more than a false alarm.
1 / 2

Seeing the problem

We generate a dataset where about 5 percent of rows are positive. The stratify=y argument keeps that share the same in the training and test sets (more on that below).

Python The accuracy trap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "always 0" rule gets about 94 percent accuracy with a recall of exactly zero. Our logistic regression reaches about 96 percent, which looks only slightly better, but its confusion matrix tells the real story: of 50 positive test rows it finds only 18. Accuracy hid the fact that the model misses most of what we are looking for. With a rare class, always read the confusion matrix, recall and precision, not accuracy alone.

Why does this happen? Most models minimize an average error over all rows. When 95 percent of rows are negative, getting the negatives right dominates the average, and the cheapest way to lower the loss is to be cautious about predicting the rare class.

Lever 1: class weights

Most scikit-learn classifiers accept class_weight. It multiplies each row's contribution to the loss by the weight of its class, so one rare row can count as several common ones.

  • class_weight="balanced" sets weights inversely proportional to class frequency: n_samples / (n_classes * count_of_class).
  • A dictionary such as {0: 1, 1: 5} lets you choose the ratio yourself.
Python class_weight: more recall, less precision

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the trade-off. "balanced" pushes recall from 0.36 up to 0.80 but precision drops to about 0.23: you now raise many false alarms. The modest {0: 1, 1: 5} lands in between and has the best F1 of the three. Two details are worth noticing:

  • The average precision (AP) barely moves, because weighting mainly shifts where the model's cut-off sits; it does not change how well the model ranks positives above negatives much. That is an early hint that the threshold, not the model, is often the real lever.
  • Balanced accuracy is the mean of recall on each class. It is a better single number than accuracy here, because a model that ignores the rare class cannot score above 0.5.

Lever 2: resampling by hand

Instead of reweighting, you can change the training set itself, only the training set. Never resample the test set: it must keep the real-world class balance.

  • Undersampling throws away some majority rows. It is fast and keeps data honest, but wastes information.
  • Random oversampling duplicates minority rows. It keeps all the data but can make the model memorize those few points.
  • SMOTE-style oversampling invents new minority rows by interpolating between a minority point and one of its minority neighbors. The idea is simple enough to write in a few lines.
Python Under-, over- and synthetic sampling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All three resampling variants move the model toward higher recall, with the same cost as class weights: lower precision. Undersampling to a 3:1 ratio happens to give the best F1 here. Do not read that as a law: on this dataset the logistic regression is already close to what the data can support, so resampling mostly shifts the cut-off. Real gains from fancy oversampling are often small, and they can mislead you if you resample before a train/test or cross-validation split, because copies of one row then appear on both sides. Resample inside each training fold only (a pipeline with a resampling step does this safely).

The imbalanced-learn package (imblearn) provides RandomUnderSampler, RandomOverSampler, SMOTE and a pipeline that resamples only during fit. The site's browser runner cannot install it, so the code above is a hand-written stand-in for the same idea. In your own project you would write from imblearn.over_sampling import SMOTE and use imblearn.pipeline.Pipeline.

SMOTE variants: cleaning the boundary and mixed data

Plain SMOTE creates synthetic minority points but never removes anything, so noisy or overlapping points near the class boundary stay. Several variants combine over-sampling with cleaning:

  • Tomek links. A Tomek link is a pair of points from different classes that are each other's nearest neighbor: they sit right on the boundary. Removing the majority member of each pair makes the boundary clearer. SMOTE-Tomek applies SMOTE first and then removes Tomek links.
  • Edited Nearest Neighbors (ENN). Remove every point whose label disagrees with the majority of its k nearest neighbors (usually k = 3). This deletes noisy points and overlapping majority points more aggressively than Tomek links. SMOTE-ENN is SMOTE followed by ENN cleaning.
  • Borderline-SMOTE and ADASYN generate synthetic points only (or mostly) near the boundary, where the minority is hardest to classify.
  • SMOTE-NC (nominal and continuous) handles tables that mix numeric and categorical columns. Plain SMOTE interpolates, which makes no sense for a category such as city. SMOTE-NC interpolates the numeric columns as usual and gives each synthetic row the most common category among the minority neighbors. (SMOTEN is its counterpart for purely categorical data.)

The Python package imbalanced-learn provides all of these (SMOTETomek, SMOTEENN, BorderlineSMOTE, ADASYN, SMOTENC), inside pipelines that resample only the training folds. The package is not part of the in-browser runner, so the example below writes the cleaning steps by hand with scikit-learn's nearest-neighbor search, on a dataset where the classes overlap.

Python SMOTE, SMOTE-Tomek and SMOTE-ENN written by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at what the numbers say. The ranking quality (average precision) does not improve: it is 0.48 without resampling and about 0.47 with every variant, because over-sampling and cleaning move the decision boundary but do not give the model more information. What changes is the operating point at the default threshold: the minority recall jumps from 0.23 to about 0.68, at the cost of many false alarms, so the F1 score actually falls (0.37 to 0.25). Tomek links removed only 17 of the 7,894 rows, so SMOTE-Tomek is almost identical to SMOTE; ENN removed 448 rows, which cleans the overlap more aggressively and pushes the recall a little higher (0.69) while lowering the F1 further. This is the honest summary of resampling: it is mostly a way to shift the threshold while training, which you can also do directly by choosing a better threshold (thresholds). Judge any variant by average precision or a cost-based score on an untouched validation set, and resample only inside the training fold.

Splitting: stratify everything

With 33 positives in 1,000 rows, a plain shuffled split can by chance put very few positives in one fold, which makes scores wildly noisy. Stratified splitting keeps the class proportions in every split.

Python KFold versus StratifiedKFold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain KFold gives one fold with 12 positives and another with only 4. The stratified version keeps every fold at 6 or 7. For classification, use stratify=y in train_test_split and StratifiedKFold for cross-validation (scikit-learn already uses it by default when you pass an integer cv to a classifier).

Lever 3: metrics and thresholds that match the cost

Choose metrics that focus on the rare class:

  • Recall and precision for the positive class, and their combination F1 (or fbeta_score with beta=2 when recall matters more).
  • Average precision (the area under the precision-recall curve). Unlike ROC AUC, it does not look flattering just because negatives are plentiful.
  • Balanced accuracy or the Matthews correlation coefficient when you need one robust number.

Then think about cost. If a missed fraud costs 20 times more than a manual review of a false alarm, the right threshold is not 0.5. You can compute the expected cost at each threshold and pick the cheapest one.

Python Choosing a threshold by cost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Lowering the threshold from 0.5 to 0.1 cuts the missed positives from 32 to 12 while false alarms grow from 7 to 69, and the total cost drops from 647 to 309. Go too low (0.05) and the false alarms start to cost more than the extra catches save. The best threshold depends entirely on the cost ratio you choose, which is a business decision, not a statistical one. Pick the threshold on a validation set, never on the final test set.

A practical order of attack

  1. Stratify your splits and report precision, recall, F1 and average precision.
  2. Get a baseline so you know what "always majority" scores.
  3. Try class_weight and a tuned threshold. They need no extra code.
  4. Only then consider resampling, inside cross-validation folds.
  5. Collect more real minority examples if you can. No sampling trick beats real data.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A dataset is 98% negative. A model that always predicts negative has what accuracy and recall?

    It is right on every negative row (98% accuracy) and finds none of the positives, so recall is zero. That is why accuracy is a poor guide for rare classes.
  2. 2. What does class_weight="balanced" do?

    It multiplies each row's loss contribution by a weight that is larger for the rarer class. The data itself is not changed, and the threshold stays at its default.
  3. 3. Where should you apply oversampling when using cross-validation?

    Resampling before the split puts copies of the same row in both train and test parts, which leaks information and inflates scores. Resample only the training part of each fold.
  4. 4. Missing a positive costs 20 times more than a false alarm. How should that change your model?

    When misses are expensive you accept more false alarms, which means flagging rows with lower predicted probabilities, so the threshold goes down.

Summary

  • With a rare class, accuracy can look excellent while recall is zero. Read the confusion matrix, precision, recall, F1 and average precision.
  • class_weight is the cheapest fix: it raises recall at the price of precision.
  • Undersampling, random oversampling and SMOTE-style synthesis change the training set only; never touch the test set, and resample inside CV folds.
  • Use stratify=y and StratifiedKFold so every split contains the rare class.
  • The right threshold comes from the relative costs of the two kinds of mistakes; choose it on validation data.

Next, trees: Decision Trees.