Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 40 ยท Gradient Boosting Basics

Gradient Boosting from Scratch in NumPy

Build a gradient boosting regressor yourself with small decision trees and residuals, watch the error fall round by round, and compare it with the real XGBoost library.

The best way to stop boosting from feeling like magic is to write it yourself. The algorithm is short: start with a constant, then repeat "compute residuals, fit a small tree to them, add it with a small step". In this lesson you will build it with NumPy and scikit-learn's DecisionTreeRegressor, then check that XGBoost behaves the same way.

The algorithm for squared error

Let the model's current prediction for every row be F. For squared error (y - F)^2 / 2, the negative gradient with respect to F is exactly the residual y - F. Boosting then does:

  1. F starts as the mean of y.
  2. For each round m = 1..M: compute residuals r = y - F; fit a shallow tree h_m to r; update F = F + learning_rate * h_m(X).
  3. The final model is F (the mean plus all scaled trees).
flowchart TD
  S["F = mean of y"] --> R["r = y - F"]
  R --> T["Fit shallow tree h to r"]
  T --> U["F = F + eta * h(X)"]
  U --> Q{"more rounds?"}
  Q -->|yes| R
  Q -->|no| Z["Final model F"]

The only moving parts are the residuals, the weak learner and the step size eta.

A working implementation

We wrap it in a small class with fit and predict, so it feels like a real estimator.

Python A tiny gradient booster

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At zero rounds the model predicts the mean everywhere, so the error equals the spread of y. Every additional tree lowers the training error, and the test error follows it down for many rounds. Predicting with only the first k trees (n_trees=k) lets you see the whole learning curve from one fitted model, which is a trick the real libraries also offer through iteration_range and num_iteration.

Compare with XGBoost

We now give XGBoost the same shape of problem and similar settings. They will not match to the last digit because XGBoost uses a regularized split criterion with second-order information, histogram bins and a different leaf formula, but the scores should be close.

Python Our booster versus XGBoost

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Setting reg_lambda=0.0 removes XGBoost's default L2 penalty, which brings it closer to our unregularized version. The predictions correlate almost perfectly, confirming that the library is doing the same thing, only far faster and with extra safeguards.

What changes for other losses

Nothing in the loop depends on squared error except one line: how the "residual" is computed. For any differentiable loss L(y, F) we replace the residual by the negative gradient -dL/dF evaluated at the current predictions. For logistic loss that is y - p, where p is the sigmoid of F. The next lesson derives this, and shows why XGBoost also uses the second derivative to take smarter steps.

Stumps and interactions

Try max_depth=1. Each tree is a single threshold on one feature, so the sum is an additive model: a separate step function per feature, with no interactions. With depth 2 or more, trees can split on a second feature below the first and capture interactions. This explains a rule of thumb: max_depth is the highest order of interaction the model can represent.

Python Depth controls interactions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The target is the product x0 * x1, which no sum of one-feature functions can represent. Stumps barely beat the baseline; depth 2 captures the interaction and the error drops sharply.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. For squared-error loss, what does each boosting round fit its tree to?

    The residual is the negative gradient of the squared error.
  2. 2. What does the learning rate do in the update F = F + eta * h(X)?

    Small steps make learning gradual and usually generalize better, at the cost of needing more trees.
  3. 3. With max_depth=1 (stumps), what kind of function can the ensemble represent?

    A stump splits on a single feature, so the sum separates by feature. Depth 2 or more allows interactions.
  4. 4. What is the main change when moving from squared error to another loss?

    The loop is the same; only the quantity the trees are fitted to (the negative gradient) changes.

Summary

  • Gradient boosting for squared error is: start at the mean, fit a tree to the residuals, add it times a small learning rate, repeat.
  • Predicting with the first k trees gives the learning curve from one fitted model.
  • XGBoost does the same with regularization, second-order information and histograms.
  • max_depth bounds the interaction order the model can capture.

Next: Loss, Gradients and Hessians