Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 40 ยท Getting Started

Why Gradient Boosting Wins on Tabular Data

Meet gradient boosting, the idea behind XGBoost, LightGBM and CatBoost, see why it dominates spreadsheet-style data, and train your first two boosted models in the browser.

If your data fits in a spreadsheet, with one row per customer, loan, house or click and one column per measurement, then the most accurate model you can build in an afternoon is very likely a gradient boosted decision tree model. Libraries such as XGBoost and LightGBM (and a third one, CatBoost) implement this idea and have won a large share of data-science competitions on tabular data. Neural networks rule images, audio and text, but for tables the humble tree ensemble is still hard to beat.

This course teaches both libraries from the ground up. You will build a tiny booster by hand so that nothing is magic, then learn the real APIs, tune them, explain their predictions and ship them. In this first lesson you will see what boosting is, why it works so well on tables, and train a first model with each library.

What runs on this site. The XGBoost 2.1 and LightGBM 4.6 examples run for real in your browser (through Pyodide), so every TryIt is live. A few things cannot run here: CatBoost has no browser build, GPU training needs a GPU, and the SHAP package is shown read-only in a later chapter. Those places are always labelled, and where possible we show a working stand-in.

The core idea in one picture

A single decision tree is easy to understand but a weak predictor on its own. Boosting builds many small trees one after another. Each new tree is trained to fix the mistakes the ensemble has made so far, and the final prediction is the sum of all trees.

flowchart LR
  A["Start: predict the average"] --> B["Measure the errors (residuals)"]
  B --> C["Fit a small tree to the errors"]
  C --> D["Add it, scaled by a small learning rate"]
  D --> B
  D --> E["After many rounds: sum of all trees"]

Each round shrinks what is left to explain. The learning rate keeps every step small and safe.

"Gradient" refers to how the errors are measured. For squared error the errors are plain residuals. For other loss functions (log loss, quantile loss, ranking loss) the trees are fitted to the gradient of the loss. You will derive this in Loss, gradients and hessians.

Boosting in three moves

Start simple A constant guess, like the mean.
Find the errors Compare predictions with the truth.
Add a small tree It learns to correct those errors.

Repeat hundreds of times. The sum of the trees is the model.

1 / 2

Why it works so well on tables

Tabular data has traits that favor trees. Columns have different units and scales, some are categories, some have missing values, and the relationship between a column and the target is often a threshold ("income above 50k", "age under 25") rather than a smooth curve. Trees split on thresholds, so they do not care about scale, they ignore irrelevant columns, and they capture interactions between columns by splitting one after another.

Boosting adds two strengths: it turns many weak trees into one strong model, and it is controllable. The learning rate, tree depth, row sampling and regularization let you trade accuracy against overfitting in a direct way.

XGBoost, LightGBM and CatBoost

Library Known for Typical tree growth
XGBoost Robust, widely deployed, regularized objective, many language bindings Level-wise by default (also leaf-wise with grow_policy)
LightGBM Very fast on large data, low memory, histogram binning and leaf-wise growth Leaf-wise (best-first)
CatBoost Excellent default settings, strong categorical handling, ordered boosting Symmetric (oblivious) trees

All three read a table, learn an ensemble of trees and expose a scikit-learn style interface. Skills transfer between them, which is why this course teaches them together.

Your first XGBoost model

The breast cancer dataset ships with scikit-learn: 569 tumors, 30 numeric measurements and a label (malignant or benign). We split it, train an XGBClassifier and measure accuracy on data it has not seen.

Python First XGBoost model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice what you did not do: no scaling, no encoding, no missing-value filling. You chose only three settings. n_estimators is the number of trees, max_depth limits how deep each tree may grow, and learning_rate scales every tree's contribution.

Your first LightGBM model

LightGBM follows the same pattern. The class is LGBMClassifier; verbose=-1 silences its training log.

Python First LightGBM model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both models land in the high nineties on this easy dataset. The point of the exercise is the shape of the code, not the score. On harder, bigger data the differences in speed, memory and tuning behavior start to matter, and later chapters show how.

When not to use boosting

Boosting is not always the answer. Prefer something else when:

  • the data is images, audio, video or free text (use deep learning or embeddings, then perhaps boost on top of them);
  • you have very few rows (a few dozen) and a simple linear model is enough and easier to explain;
  • you need to extrapolate beyond the range of the training data: trees predict constants inside regions, so they cannot continue a trend upward forever;
  • you need strictly smooth predictions (use a spline or a neural network).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In gradient boosting, what is each new tree trained to do?

    Every round fits a small tree to the current errors, and its output is added to the running prediction.
  2. 2. Which preparation step do tree-based boosters usually NOT need?

    Trees split on thresholds, so monotone rescaling of a column does not change the splits.
  3. 3. Which task is a poor fit for a boosted tree ensemble?

    Trees output constants within regions, so they cannot continue a trend outside the data they saw.
  4. 4. What does the learning rate control?

    A small learning rate means small steps, which usually needs more trees but generalizes better.

Summary

  • Gradient boosting builds trees one by one, each correcting the current errors, and sums them.
  • It dominates tabular data because trees ignore scale, handle missing values and capture interactions.
  • XGBoost, LightGBM and CatBoost are the three major libraries; they share one mental model.
  • n_estimators, max_depth (or num_leaves) and learning_rate are the first settings to learn.
  • Boosting is not for raw images, audio, text or far extrapolation.

Next: Installing and Setting Up