Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 40 ยท LightGBM and CatBoost

Choosing a Booster

Compare XGBoost, LightGBM, CatBoost and scikit-learn's HistGradientBoosting on speed, accuracy, categorical handling, GPU support and defaults, and learn a practical way to decide.

You now know three production-grade gradient boosting libraries, plus the histogram booster that ships inside scikit-learn. Which one should you use? The honest answer is "any of them, often", because on typical tabular problems their accuracy is within noise of each other after tuning. The choice is mostly about workflow: data size, categorical columns, hardware, deployment and what your team already knows. This lesson gives you a way to decide and a fair way to compare.

A fair comparison

A fair comparison uses the same data split, comparable complexity, early stopping on the same validation set and a held-out test set. This experiment runs XGBoost, LightGBM and scikit-learn's HistGradientBoostingClassifier side by side. Treat the times as rough (single thread, in a browser) and the scores as one sample.

Python Three boosters on the same split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Expect the three AUCs to land within a couple of thousandths of each other. If one model looks much better, check that the settings match before believing it: tree size, learning rate and the stopping rule all move the score by more than the choice of library does. The practical conclusion is to pick by workflow, then spend your effort on features and validation, which move results far more.

How they differ

Question XGBoost LightGBM CatBoost sklearn HGB
Tree growth level-wise (leaf-wise optional) leaf-wise symmetric leaf-wise
Big data speed fast, great on GPU usually fastest on CPU fast on GPU decent
Categorical input pandas categories (hist) native, subset splits native, ordered statistics native (limited cardinality)
Missing values learned direction learned direction handled (modes) learned direction
Defaults good good, but can overfit small data very strong reasonable
Extras many objectives, distributed, ranking, survival distributed, ranking, monotone constraints text/embedding features, uncertainty no extra dependency
Install footprint wheel, optional GPU wheel larger wheel already in scikit-learn

Other features worth knowing exist in all three major libraries: monotone constraints (force the prediction to rise with a feature), interaction constraints, sample weights, custom objectives and metrics, and GPU training. Chapter 5 and later chapters use these.

A decision guide

Pick by situation

Many categories Try CatBoost first
Huge tables LightGBM or XGBoost on GPU
Need it simple scikit-learn HGB
Unusual loss XGBoost objectives

Then confirm with cross-validation.

1 / 2

Some guidelines that work well in practice:

  1. Start with LightGBM or XGBoost and a sensible baseline; both have huge communities and examples.
  2. Lots of categorical columns, small or medium data, little time to tune? Try CatBoost.
  3. Hundreds of millions of rows or a GPU? XGBoost and LightGBM both scale; test both on a sample.
  4. Need minimal dependencies or an easy teaching setup? HistGradientBoostingClassifier in scikit-learn is a solid default and plugs into the whole scikit-learn ecosystem.
  5. Need an unusual loss, ranking or survival analysis? XGBoost has the widest choice of built-in objectives.
  6. Latency matters at prediction time? Compare real prediction speed; symmetric CatBoost trees and small LightGBM models are quick, and compiled exporters exist for all three.
  7. Ensemble them. Averaging a LightGBM and a CatBoost model often gives a small but reliable gain, because their errors are not identical.
flowchart TD
  S["Start"] --> C{"Many categorical columns?"}
  C -->|yes| CB["Try CatBoost, then LightGBM categorical"]
  C -->|no| N{"Very large data or GPU?"}
  N -->|yes| LX["LightGBM or XGBoost, compare on a sample"]
  N -->|no| H["Any booster; sklearn HGB is a fine start"]
  CB --> V["Validate with CV, tune, consider a blend"]
  LX --> V
  H --> V

The flow ends in validation: the data decides, not reputation.

The shared workflow you will use from now on

The rest of the course works with XGBoost and LightGBM interchangeably. The recipe is the same: split data honestly, build a baseline, train with early stopping on a validation set, tune the few parameters that matter, explain the model, and test on held-out data. The next chapter begins with the tuning workflow and validation.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. After tuning, how different is accuracy between XGBoost, LightGBM and CatBoost on a typical tabular problem?

    Gaps are normally tiny compared with the effect of features, validation and tuning.
  2. 2. Which situation most favors trying CatBoost first?

    Ordered target statistics and strong defaults make it a good first try on category-heavy data.
  3. 3. What makes a comparison of boosters unfair?

    Complexity settings and stopping rules change scores more than the library does.
  4. 4. Which is NOT a reason to blend two different boosters?

    Blending still needs honest validation; it never removes that need.

Summary

  • XGBoost, LightGBM, CatBoost and scikit-learn's HGB reach similar accuracy once tuned; choose by workflow.
  • LightGBM tends to be fastest on CPU, XGBoost has the broadest objectives, CatBoost shines on categories.
  • Compare libraries fairly with the same split, similar complexity and the same stopping rule.
  • Features, validation and tuning matter more than the library.
  • Blending two boosters is a cheap, reliable improvement.

Next: Tuning Workflow and Validation