Skip to content
Tutorials Center
My learning

Topics

Lesson 49 of 76 ยท Modelling

Decision Trees, Random Forests and Boosting

Understand decision trees, overfitting, bagging in random forests and gradient boosting with scikit-learn, and why feature importance can mislead.

Linear and logistic models draw straight boundaries. Real data often has thresholds and interactions: a loan is risky when debt is high and income is low, a discount only works above a minimum basket size, and the effect of age changes after retirement. Tree models capture these patterns without you specifying them in advance, and tree ensembles (random forests and gradient boosting) are among the strongest methods for tabular data.

This lesson builds from one tree to forests to boosting, and shows with live code why a single deep tree memorizes noise, how averaging fixes that, and why the "feature importance" numbers that come out of a forest need careful handling. All examples use synthetic data with a known pattern, so you can see whether each model finds it. For the evaluation ideas used here, see train/test splits and model evaluation.

A decision tree is a flowchart of questions

A decision tree repeatedly splits the data with a yes/no question about one feature ("is income at most 47.7?"). Each split is chosen to make the groups on both sides as pure as possible: for classification, groups that are mostly one class; for regression, groups with similar values. Splitting continues until a stopping rule applies, and each final box (a leaf) predicts the majority class, or the average value, of the training rows that landed there.

flowchart TD
    A{"income <= 47.7?"} -->|"yes"| B{"debt <= 20.3?"}
    A -->|"no"| C{"debt <= 27.1?"}
    B -->|"yes"| D["Low risk"]
    B -->|"no"| E["High risk"]
    C -->|"yes"| F["Low risk"]
    C -->|"no"| G["High risk"]

A depth-2 tree: two questions deep, four leaves, each with its own prediction.

The data below hides a simple rule: a person is risky when their debt exceeds about half of their income, with 8% of labels flipped at random to add noise. We fit trees of different maximum depth and compare accuracy on training data and on held-out data.

Python Tree depth: from underfitting to memorizing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unlimited tree scores 100% on its own training rows, because it keeps splitting until every row has its own leaf, even the rows with flipped labels. On new data it drops to 82%. The best test accuracy (90%) appears at depth 3, a small tree that matches the real rule. This is overfitting: the model has learned the noise of the training sample, not the pattern. Deeper is not better. Common ways to limit a tree are max_depth, min_samples_leaf (a leaf must contain enough rows), and ccp_alpha (cost-complexity pruning).

Trees have a big practical advantage: export_text prints rules a person can read, and they need no feature scaling, handle mixed units and ignore monotone transformations of a feature. Their weakness is instability: a small change in the data can produce a completely different tree.

Bagging and random forests

If one tree is unstable, average many. Bagging (bootstrap aggregating) trains each tree on a different bootstrap sample (rows drawn with replacement) and averages the predictions. Individual trees overfit in different ways, and averaging cancels much of that noise. A random forest adds a second source of variety: at every split, each tree may only consider a random subset of the features, which makes the trees less alike, so the average gains more.

flowchart LR
    D["Training data"] --> S1["Sample 1"] --> T1["Tree 1"]
    D --> S2["Sample 2"] --> T2["Tree 2"]
    D --> S3["Sample 3"] --> T3["Tree 3"]
    T1 --> V["Average (or vote)"]
    T2 --> V
    T3 --> V

Trees trained on different resamples are averaged; each is noisy, the average is steady.

Boosting: learn from mistakes

Gradient boosting takes the opposite route. It builds shallow trees one after another, and each new tree is trained to correct the errors left by all the previous trees. The model's prediction is the sum of many small corrections, scaled by a learning rate. Boosting is more sensitive to settings than a forest, but with good settings it is often the most accurate method on tabular data.

scikit-learn's HistGradientBoostingRegressor and HistGradientBoostingClassifier bin each feature into at most 255 buckets, which makes them fast on large tables. They also handle missing values natively and support categorical columns, so they are a strong default.

Here we compare everything on one problem with a smooth curve (sin), a threshold, and an interaction, plus a useless third feature. The metric is mean absolute error (MAE): the average size of the miss, in the units of the target.

Python Linear model vs tree vs forest vs boosting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the two columns together. The full-depth tree has a training MAE of zero and a test MAE near 0.86, the signature of memorizing. The forest and boosting are the best on the test set (about 0.63 and 0.65), close to the noise floor of roughly 0.56 that no model can beat, because that part of the target is pure randomness. The linear model, at 1.8, cannot bend, and improves only modestly on the mean-only baseline of 2.5. Notice, too, that a depth-4 tree beats a full-depth tree: simplicity helped.

Which should you pick? A practical order is: a simple baseline first, a regularized linear model next, then HistGradientBoosting or a random forest. If the boosted model is only slightly better than the linear one, the extra complexity may not be worth it. See pipelines and tuning for how to choose settings without fooling yourself.

Feature importance and its caveats

Forests and boosting report feature importances, so it is tempting to read them as "what drives the outcome". Two traps make this unsafe.

  1. Impurity-based importance is biased. The default feature_importances_ measures how much each feature reduced impurity on the training data. Features with many distinct values, even meaningless ones such as row IDs, get splits by chance and look important.
  2. Correlated features share credit. If two columns carry the same signal, the importance is split between them, and each looks weaker than the underlying fact.

A better check is permutation importance: shuffle one column of the test set and see how much the score drops. The next example includes a real signal, an almost-identical copy of it, a pure noise column and a random row ID.

Python Impurity importance vs permutation importance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The impurity column gives the meaningless row_id the same importance as the pure-noise column (about 0.04 to 0.05), while permutation importance puts both at zero. The signal's credit is split between signal and signal_copy in both columns, so neither alone tells the whole story. Importance also says nothing about direction or causation; for that you need tools like partial dependence, covered in explaining model behaviour.

Practical guidance

  • Trees need no scaling, but they do need numeric inputs: encode text categories (or use the native categorical support in HistGradientBoosting).
  • Forests are hard to break. The defaults work well, and more trees rarely hurt, only slow things down.
  • Boosting needs a validation set to choose the number of iterations and the learning rate; too many iterations will overfit.
  • Trees cannot extrapolate. A tree predicts an average of training values, so it never goes beyond the range it has seen, which matters for trends and forecasting.
  • Look at the gap between train and test error. A large gap says overfitting; a small gap with poor scores says underfitting.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A decision tree has no depth limit and scores 100% on the training data but much lower on the test data. What is this?

    The tree kept splitting until it memorized the training rows, including their noise. That does not carry over to new data.
  2. 2. Why does a random forest usually beat a single deep tree?

    Each tree overfits in its own way. Training on bootstrap samples and random feature subsets makes the errors differ, and the average of the predictions is more stable.
  3. 3. How does gradient boosting differ from a random forest?

    Boosting is sequential: each new shallow tree fits what the current ensemble still gets wrong. A forest trains its trees independently and averages them.
  4. 4. Why should you be careful with the default feature_importances_ of a forest?

    Impurity importance is computed on training data and rewards features offering many split points, even meaningless ones. Permutation importance on held-out data is a safer check.

Summary

  • A decision tree splits the data with yes/no questions and predicts the average or majority in each leaf; it is readable but overfits when grown deep.
  • Limit trees with max_depth, min_samples_leaf or pruning, and judge them on held-out data.
  • Random forests average many trees trained on bootstrap samples with random feature subsets.
  • Gradient boosting adds shallow trees in sequence, each fixing the remaining error; HistGradientBoosting is fast and a strong default for tables.
  • Tree models find thresholds and interactions automatically but cannot extrapolate beyond the training range.
  • Impurity-based importance can reward junk and split credit between correlated columns; check permutation importance on test data, and never read importance as causation.

Next, learn how to measure all of this honestly in train/test splits and cross-validation.