Skip to content
Tutorials Center
My learning

Topics

Lesson 114 of 149 ยท Data and the Web

Gradient Boosting with XGBoost and LightGBM

Train, tune and compare XGBoost and LightGBM on a tabular dataset in Python. Learn n_estimators, learning_rate, early stopping, feature importance and a fair comparison with scikit-learn.

On spreadsheets, databases and other tabular data, one family of models wins most real-world contests: gradient-boosted decision trees. They build many small decision trees one after another, and each new tree corrects the mistakes of all the trees before it. The result is accurate, handles odd data well and needs far less tuning than a neural network. Two libraries dominate in practice: XGBoost and LightGBM. Both are fast, both are used in production at large companies, and both follow the same fit and predict pattern you learned in Machine Learning with scikit-learn.

This lesson shows how to train each of them, which settings matter, how early stopping chooses the number of trees for you, and how to compare the libraries fairly. If you want the mathematics of boosting, the Gradient Boosting lesson in the machine learning course explains it step by step, and Histogram Gradient Boosting covers scikit-learn's own version. Here we focus on using the two libraries.

Setup

Both libraries are third-party packages. In the browser runner they are available as ready-made packages and load automatically when you import them. On your own computer you would install them with pip install xgboost lightgbm (see Packages and pip).

One quirk of the browser build: LightGBM checks whether pandas is installed at the moment it is first imported, so the LightGBM examples import pandas first. On a normal installation this never matters.

All the examples use the breast cancer dataset that is bundled with scikit-learn: 569 patients, 30 measurements of cell nuclei, and a label saying whether the tumor was malignant or benign. It needs no download. (A real project would of course never try to replace a doctor; the dataset just makes a tidy teaching example.)

The scikit-learn style API

Both libraries offer classes that behave like any scikit-learn model: XGBClassifier, XGBRegressor, LGBMClassifier and LGBMRegressor. You can drop them into pipelines and cross-validation unchanged.

Python XGBoost with fit, predict and score

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three settings do most of the work:

Setting Meaning Typical range
n_estimators How many trees to build 100 to several thousand
learning_rate How much each tree is allowed to correct. Smaller means more careful and needs more trees 0.01 to 0.3
max_depth How deep each tree may grow. Shallow trees underfit, deep ones overfit 3 to 8

learning_rate and n_estimators pull against each other: halve the learning rate and you usually need about twice the trees. Rather than guess the tree count, let the model find it.

Early stopping: let the data choose the number of trees

If you keep adding trees, the score on the training data keeps improving while the score on new data eventually gets worse. That is overfitting. Early stopping watches a validation set during training, and stops once it has not improved for a given number of rounds. You set n_estimators high as an upper limit.

Python XGBoost early stopping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model used the best round, not all 1000. The three-way split matters: training data fits the trees, the validation set decides when to stop, and the test set gives the final honest score. If you stopped on the test set, the test score would no longer be an unbiased estimate (this is a form of data leakage).

LightGBM: the same idea with different engineering

LightGBM produces the same kind of model, but grows trees differently. XGBoost, by default, grows trees level by level. LightGBM grows them leaf by leaf, always splitting the leaf that reduces the error most. It also sorts feature values into histogram bins first, which makes it very fast on large tables. The consequence is that the main complexity control is num_leaves (the maximum leaves per tree) rather than only max_depth.

Python LightGBM with early stopping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The API differs in small ways: LightGBM uses a callback for early stopping, and its fitted attributes end in an underscore (best_iteration_). The logic is identical. On a table this small, the two libraries give very similar results. Their differences matter on big data and on awkward columns, as the next section shows.

What is special about each library?

  • Missing values. Both accept NaN in features directly, and they learn which side of a split missing values should go to. You do not need to fill gaps first, though you may want to understand why values are missing (see Missing Values).
  • No scaling needed. Trees compare thresholds, so standardizing features does not change anything. Skip StandardScaler.
  • Categorical columns. LightGBM can use pandas category columns directly. XGBoost supports them too when you set enable_categorical=True. Both avoid the explosion of columns that one-hot encoding would create.
  • Speed. LightGBM is usually fastest on large tables with many rows. XGBoost is robust, and also supports GPUs and distributed training.
Python Missing values and categories handled natively

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Neither model needed imputation, scaling or one-hot encoding. They accepted the NaN values and the three-level plan column as they were. The LightGBM half of this example uses its native API: lgb.Dataset wraps the data, lgb.train takes a dictionary of parameters (note the native names, such as min_data_in_leaf) and predict returns probabilities that you threshold yourself. The scikit-learn wrapper classes are a thin layer over this.

Which features mattered?

Boosted trees can tell you how much each feature contributed. Both libraries expose feature_importances_, and scikit-learn's permutation importance is a more trustworthy alternative: it shuffles one feature at a time and measures how much the test score drops.

Python Feature importance and permutation importance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Treat importance as a hint, not a proof. Correlated features share their importance between them, and importance says nothing about cause. More on this in Feature Importance.

A fair comparison

To compare the two libraries (and scikit-learn's own HistGradientBoostingClassifier) fairly, give them the same data, similar settings and cross-validation, not one lucky split.

Python Cross-validated comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Expect the three scores to land within a point or two of each other. Differences that small are inside the noise (look at the plus-or-minus), so don't crown a winner from one run on one small dataset. Pick based on your data size, how awkward your columns are, deployment needs and what your team already uses. To tune settings, combine the models with hyperparameter search and keep the early-stopping validation set separate from the test set.

A starting recipe

For a first attempt on a new tabular problem:

  1. Split into train, validation and test sets.
  2. Start with learning_rate=0.05, max_depth=4 (or num_leaves=15), n_estimators=2000 and early stopping at 50 rounds.
  3. Compare with a simple baseline such as logistic regression. If boosting does not clearly win, prefer the simpler model.
  4. Raise regularization (min_child_samples, reg_lambda, subsample, colsample_bytree) if validation scores are far below training scores.
  5. Check the test score once, at the end.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How does gradient boosting build its ensemble of trees?

    Boosting is sequential: each new tree focuses on what the current ensemble gets wrong. Training many independent trees and averaging them is a random forest.
  2. 2. Why set n_estimators=1000 together with early_stopping_rounds=20?

    Early stopping lets the data choose the tree count. You only need a generous maximum.
  3. 3. Which data should you use as the early-stopping eval_set?

    Stopping on the test set leaks information from it, so your final test score would be too optimistic. Use a separate validation set.
  4. 4. How does LightGBM mainly differ from XGBoost's default tree growth?

    Leaf-wise growth and histogram binning make LightGBM fast on big tables, which is why num_leaves is its main complexity control.
  5. 5. Do boosted-tree models need StandardScaler for their features?

    Splits depend on the order of values, not their scale. That is one reason tree models need less preprocessing than linear models or neural networks.

Summary

  • Gradient boosting builds shallow trees sequentially, each correcting the last. XGBoost and LightGBM are its most popular fast implementations for tabular data.
  • Both offer scikit-learn-style classes (XGBClassifier, LGBMClassifier, and the regressors), so pipelines and cross-validation work unchanged.
  • The key settings are n_estimators, learning_rate and tree size (max_depth for XGBoost, num_leaves for LightGBM). Smaller learning rates need more trees.
  • Use early stopping with a separate validation set to choose the number of trees. Keep the test set for one final check.
  • Both handle NaNs natively and need no feature scaling; both can use categorical columns directly.
  • Compare models with cross-validation and treat tiny differences as noise. Compare to a simple baseline too.

Next: Neural Networks: TensorFlow and PyTorch at a Glance, a different family of models that shines on images, text and sound.