Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 40 ยท Gradient Boosting Basics

Decision Trees and Ensembles

Understand the building block behind XGBoost and LightGBM: how a decision tree splits data, why one tree over- or underfits, and how bagging and boosting combine many trees.

Gradient boosting is built from decision trees, so you need a clear picture of what one tree does and where it fails. In this lesson you will grow single trees of different sizes, see overfitting happen, and then compare the two main ways of combining trees: bagging (random forests) and boosting.

How a regression tree works

A regression tree splits the rows into groups using yes/no questions about features, such as "is age below 30?". Every row ends in a leaf, and the leaf predicts the average target of the training rows in it. The tree picks each split to make the groups as uniform as possible, usually by minimizing the squared error inside the groups.

flowchart TD
  A["age < 30 ?"] -->|yes| B["income < 40k ?"]
  A -->|no| C["leaf: predict 180"]
  B -->|yes| D["leaf: predict 60"]
  B -->|no| E["leaf: predict 110"]

A depth-2 tree has up to four leaves. Each leaf outputs one constant.

Because leaves output constants, a tree's prediction is a staircase function. Deeper trees create more, smaller steps.

Underfitting and overfitting

Depth is the dial. A depth-1 tree (a stump) makes a single split and is too crude. A very deep tree gives almost every training row its own leaf and memorizes noise. Try it on a noisy curve:

Python Tree depth: underfit to overfit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The train error keeps falling as depth grows, while the test error falls and then rises again. That gap between the two is overfitting. A single tree has high variance: change a few rows and the tree changes a lot.

Ensembles reduce error

Combining many models often beats any single one, provided their mistakes are not all the same. Two classic recipes:

  • Bagging (random forests): train many deep trees in parallel on random resamples of the data and average them. Averaging cancels the random wobble, so it mainly reduces variance.
  • Boosting: train many shallow trees in sequence, each fixing the previous errors, and add them. Sequential correction mainly reduces bias (it keeps improving the fit), and the learning rate and regularization keep variance in check.

Bagging versus boosting

Bagging Deep trees, trained in parallel, averaged.
Boosting Shallow trees, trained in sequence, summed.

Bagging fights variance. Boosting fights bias.

1 / 2

Try all three on one problem

Python One tree, a forest and a boosted model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see the single tree far behind, the forest much better and the boosted model best or close to it. The generated target (the Friedman function) mixes interactions and non-linear terms, the kind of structure that boosting fits well.

Why boosting uses weak learners

If every boosting round used a very deep tree, the first tree would already fit the training data almost perfectly and later trees would have nothing left to fix, so the ensemble would memorize noise. Using weak learners (trees of depth 3 to 6, sometimes just stumps) makes each step small and lets the ensemble build up the pattern gradually. This is why max_depth in boosting is typically 3 to 8, while a random forest may grow trees to depth 30.

You can even boost stumps and still capture interactions, because later stumps split on residuals that already depend on earlier splits. The next lesson builds exactly this, by hand.

Splitting in practice

How does a tree find the best split on a numeric column? In theory it tries every threshold between sorted values. For big data that is slow, so XGBoost and LightGBM use histograms: they bucket each feature into at most 255 bins (the max_bin setting) once, then evaluate only the bin edges. That is a large speedup with little loss in accuracy. It is also why the training reads its data through special structures (DMatrix, Dataset) that store the binned version.

Python Cutting a feature into bins

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Quantile bins keep a similar number of rows per bin, so skewed columns are not wasted. The tree tests only 7 candidate thresholds here instead of nearly 10,000.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a leaf of a regression tree predict?

    A leaf outputs one constant, typically the mean of the targets of its rows.
  2. 2. Which statement about bagging and boosting is correct?

    Bagging reduces variance by averaging; boosting reduces bias by sequential correction.
  3. 3. Why are boosting trees usually shallow?

    Weak learners let the ensemble build up the pattern step by step.
  4. 4. What is the benefit of histogram-based splitting?

    Binning each feature once reduces the number of thresholds to test from the number of rows to at most max_bin.

Summary

  • A tree splits rows by feature thresholds and predicts a constant in each leaf.
  • Deep single trees overfit; shallow ones underfit.
  • Bagging averages deep trees to cut variance; boosting sums shallow trees to cut bias.
  • Histogram binning makes split search fast and is built into both libraries.

Next: Gradient Boosting from Scratch