Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 40 ยท Getting Started

First Models with XGBoost and LightGBM

Train XGBoost and LightGBM side by side for classification and regression, compare the key parameters, read predictions and probabilities, and see how fit time and accuracy compare.

You have seen what boosting is, how to prepare a table and how to evaluate fairly. This lesson puts it together: a classification model and a regression model in each library, written side by side so you can see how little changes between them. You will also learn the small vocabulary of parameters that the rest of the course keeps returning to.

The shared vocabulary

Idea XGBoost name LightGBM name
Number of trees n_estimators n_estimators
Step size learning_rate (eta) learning_rate
Tree size max_depth num_leaves (and max_depth)
Row sampling per tree subsample subsample (needs subsample_freq=1)
Column sampling per tree colsample_bytree colsample_bytree
L2 penalty on leaf values reg_lambda reg_lambda
Minimum data per leaf min_child_weight min_child_samples

Good news: the scikit-learn wrappers use almost the same names, so moving between libraries is mostly a rename of the tree-size parameter. Chapter 5 explains what each one does to the bias-variance balance.

Binary classification

Python Classification side by side

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

predict returns class labels, predict_proba returns an (n, classes) array of probabilities. The raw model output is a log-odds score per row, and the probability is its sigmoid. Probabilities from boosting are usually good for ranking but can be a little over- or under-confident; see calibration later in the course.

Regression

For a numeric target use XGBRegressor or LGBMRegressor. The default objective is squared error.

Python Regression side by side

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The small num_leaves and min_child_samples matter: this dataset has only about 300 training rows. LightGBM's default min_child_samples=20 and num_leaves=31 would be too aggressive here. Boosting libraries have defaults tuned for medium-sized data, and tiny datasets need smaller trees.

Multiclass

For more than two classes, both classifiers detect it from y and switch to a softmax objective. predict_proba then has one column per class.

Python Multiclass wine classification

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

XGBoost expects class labels to be integers starting at zero in older versions; in 2.x the scikit-learn wrapper encodes labels for you only for some cases, so a safe habit is to use LabelEncoder when your labels are strings.

What the trained model contains

A trained booster is just a list of small trees. You can count them and peek at their structure. Here we dump the first tree of an XGBoost model as text.

Python Peek inside the first tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each line is a node: f22<105.1 means "feature index 22 below 105.1" with yes, no and missing branches (the default direction for NaN). Leaves carry leaf= values; the model's raw score is the sum of one leaf per tree plus a base score.

Training time

Both libraries are fast, but LightGBM's advantage grows with data size. A quick timing experiment on synthetic data (the numbers depend on your device, so compare the ratio rather than the milliseconds):

Python A rough timing comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Timing in a browser is noisy and single-threaded, so treat it as a demonstration of method, not a benchmark. We run a fairer comparison, with accuracy too, in Choosing a Booster.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which parameter plays the role of tree size in LightGBM?

    LightGBM grows trees leaf-wise, so the number of leaves is the main size control (XGBoost uses max_depth).
  2. 2. What does predict_proba return for a 3-class problem with 10 rows?

    One probability per class for each row, and each row sums to 1.
  3. 3. Why might the default settings be too large for a 300-row dataset?

    Smaller trees and bigger minimum leaf sizes relative to the data size usually generalize better on tiny sets.
  4. 4. A booster's raw prediction is...

    Every tree contributes one leaf value, and the contributions are added up and then passed through the objective's link function.

Summary

  • XGBoost and LightGBM share the same scikit-learn style; the main rename is max_depth versus num_leaves.
  • predict gives labels or numbers; predict_proba gives class probabilities.
  • Multiclass is detected automatically from the labels.
  • A model is a list of small trees you can dump and inspect.
  • Tiny datasets need smaller trees and larger minimum leaf sizes than the defaults.

Next: Trees and Ensembles