Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 111 ยท Trees and Ensembles

Gradient Boosting

Understand gradient boosting by building it by hand: fit residuals with small trees, balance learning rate and trees, use subsample and early stopping.

Random forests build many deep trees independently and average them. Gradient boosting does the opposite: it builds many small, weak trees one after another, and each new tree is trained to fix the mistakes the ensemble has made so far. The model starts with a crude guess and improves it step by step, a bit like sculpting: rough shape first, then finer and finer corrections.

Boosted trees are the workhorse of modern tabular machine learning, behind most winning solutions in data competitions on spreadsheet-style data. In this lesson you will build gradient boosting for regression by hand in about ten lines (so the idea is not magic), then use GradientBoostingClassifier, see how learning_rate and the number of trees trade off, and learn to stop at the right moment with early stopping.

Boosting in four steps

Start simple Predict the mean for everyone.
Measure the error Residual = truth minus prediction.
Fit a small tree Train it to predict the residuals.
Add a fraction Prediction += learning rate x tree.

Repeat the last three steps hundreds of times.

1 / 2

Boosting by hand: fitting residuals

For squared-error regression, the idea is simple. Let pred be the current prediction for every row. The residual y - pred is what is still wrong. Fit a small tree to predict the residuals, then nudge the prediction in that direction:

TEXT
pred = mean(y)
repeat:
    residual = y - pred
    tree = small tree fitted to (X, residual)
    pred = pred + learning_rate * tree.predict(X)

The residual is the negative gradient of the squared-error loss, which is where the name comes from: each tree takes a step downhill in function space, the same way gradient descent steps downhill in parameter space. For other losses (log loss for classification, absolute error, quantiles) the residual is replaced by that loss's gradient, but the loop is identical.

Let us run it on a noisy sine wave and compare with scikit-learn's version.

Python Gradient boosting from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The error falls fast at first (0.53 before any tree, 0.18 after two rounds, 0.06 after five) and then more slowly (0.015 after 30). The hand-made version matches GradientBoostingRegressor to six decimals, because for squared error the algorithm really is this simple. Here is what it looks like.

Python The fit improving round by round

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One tree gives a four-step staircase, five trees already trace the wave, and 50 trees follow it closely, starting to bend toward the noise.

Learning rate versus number of trees

learning_rate (also called shrinkage) scales each tree's contribution. A small rate takes small careful steps, so you need more trees (n_estimators) to get there, but the result generalizes better. A large rate moves fast and overshoots, and with many trees it fits noise. The two parameters work as a pair: a good rule is to pick a small learning rate (0.05 to 0.1) and then use enough trees, found by early stopping (below).

The grid below shows test accuracy for different rates and tree counts on noisy half-moons.

Python learning_rate x n_estimators

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With learning_rate=1.0, test accuracy gets worse as trees are added (0.88 to 0.86): the model chases noise. The smaller rates are higher and steadier, and at 0.03 even 200 trees stay at 0.90. Differences of one or two hundredths on 320 test rows are small, so do not read the grid to the last digit, but the direction is the reliable part: lower rate, more trees, better and more stable.

Boosting can overfit: watching the stages

Unlike a random forest, adding more trees to a boosted model can overfit, because every tree is trying to fit what remains, and eventually what remains is noise. staged_predict_proba lets you replay the model after each tree, so you can see training and test loss as the model grows. Log loss is more sensitive than accuracy here.

Python Training versus test loss by number of trees

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Training loss keeps falling all the way to 0.019, and training accuracy reaches 1.000: the model has memorized the training set. Test loss is lowest after about 63 trees (0.253) and then rises to 0.397 at 500 trees. The gap between training and test is the picture of overfitting. The best number of trees is where the validation loss bottoms out.

Early stopping

You do not have to search for that point by hand. Give GradientBoostingClassifier an n_iter_no_change and it holds out validation_fraction of the training data, watches the validation loss, and stops adding trees when the loss has not improved for that many rounds. Set n_estimators to a large ceiling; n_estimators_ tells you how many trees were actually built.

Two other regularizers are worth knowing. subsample below 1.0 trains each tree on a random fraction of the rows (stochastic gradient boosting), which adds randomness like bagging and often helps on bigger, noisier data. max_depth stays small (often 2 to 5): boosting wants weak learners.

Python Early stopping and subsample

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Early stopping ends after only 35 trees and achieves a test log loss of 0.273, against 0.397 for the full 500 trees: it is both faster and better. The subsample runs tie on accuracy here. Do not expect subsample to win on a small, clean toy set; treat it as a knob for cross-validated tuning.

Key parameters at a glance

Parameter Typical range Effect
n_estimators 100 to 1000+ (with early stopping) Number of trees
learning_rate 0.01 to 0.2 Step size; lower needs more trees
max_depth 2 to 5 Complexity of each tree and interactions captured
subsample 0.5 to 1.0 Row fraction per tree; below 1 adds regularization
min_samples_leaf 1 to 50 Smoother leaves, less noise fitting
n_iter_no_change 5 to 20 Early stopping patience

Strengths and limits

  • Often the most accurate model on tabular data when tuned, with no need for scaling.
  • Handles nonlinear effects and interactions automatically, like other tree models.
  • Sequential training cannot be parallelized across trees, so on large data the classic GradientBoostingClassifier is slow. The histogram-based version in the next lesson is much faster.
  • More hyperparameters to tune than a random forest, and more sensitive to noisy labels.
  • Probabilities from boosting can be overconfident; check calibration when you rely on them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In gradient boosting for squared error, what is each new tree trained to predict?

    Each tree fits what the ensemble still gets wrong. Its (scaled) output is added to the prediction, which shrinks the residuals step by step.
  2. 2. You lower learning_rate from 0.3 to 0.03. What should you usually do next?

    Smaller steps need more of them to reach the same fit. A small learning rate with enough trees generalizes better than a large rate with few.
  3. 3. Training loss keeps falling while validation loss rises after 60 trees. What does this show?

    Falling training loss with rising validation loss is the signature of overfitting. Use early stopping or fewer trees.
  4. 4. Compared with a random forest, a boosted ensemble usually uses...

    Boosting stacks many weak (shallow) learners in sequence. Forests average deep, independent trees.

Summary

  • Gradient boosting builds shallow trees in sequence; each fits the residuals (the negative gradient of the loss) and is added with a small step.
  • Squared-error boosting is a ten-line loop you can write yourself; scikit-learn's version matches it.
  • A smaller learning_rate needs more trees but generalizes better; the pair works together.
  • Unlike forests, boosting can overfit as trees are added: watch validation loss and use n_iter_no_change early stopping.
  • max_depth stays small, and subsample < 1 adds extra regularization.

Next: Histogram Gradient Boosting.