Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 111 ยท Trees and Ensembles

Histogram Gradient Boosting

Use HistGradientBoostingClassifier and Regressor: binned features for speed, native missing values and categories, early stopping, and how it compares with XGBoost and LightGBM.

The previous lesson built boosted trees one at a time, each fixing the mistakes of the ones before it. That recipe works very well on tables of numbers, but the classic GradientBoostingClassifier has a practical weakness: to grow each tree it must sort and scan every feature value of every row, over and over. On tens of thousands of rows it becomes slow, and on millions it is unusable.

Histogram gradient boosting solves this with one clever step: before training, every numeric feature is sorted into at most 255 bins (think of a histogram with 255 bars). A tree then only has to look at 255 candidate split points per feature instead of thousands, and the work per tree no longer grows with the number of distinct values. scikit-learn ships this algorithm as HistGradientBoostingClassifier and HistGradientBoostingRegressor. It is usually the first boosted model you should try on tabular data, and it is the same idea that powers LightGBM.

Why "histogram"?

Bin once Each feature is cut into up to 255 bins before training.
Cheap splits A tree tests bin edges, not every raw value.
Scales Training time grows gently with the row count.
1 / 2

The same API as every other estimator

The class is used exactly like the models you already know: construct it, call fit, then predict or score. Notice two small naming differences from the older class: the number of trees is called max_iter (one tree per iteration for binary problems), and the model has no n_estimators.

The first example compares the old and new boosters on the same synthetic data. The data set is kept modest so that it runs in your browser, so the difference you will see is smaller than it would be on a large table; the older class scales poorly as rows grow, while the histogram version barely notices.

Python Classic vs histogram boosting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Accuracy is in the same range (the histogram model is even a little ahead here), because both are the same kind of boosted ensemble. What differs is how fast they get there: the histogram version finishes in a fraction of the time. Binning loses a little precision in where a split can fall, but with 255 bins that loss is almost always smaller than the noise in the data.

Hyperparameters that matter

The knobs are the ones you met in Gradient Boosting, under slightly different names:

Parameter Meaning Typical direction
learning_rate How much each tree contributes (shrinkage) Lower (0.05 to 0.1) with more iterations
max_iter Maximum number of boosting rounds Large, together with early stopping
max_leaf_nodes Leaves per tree (default 31) Smaller means a simpler model
max_depth Optional depth limit (default none) Set it if trees overfit
min_samples_leaf Minimum rows in a leaf (default 20) Raise it on noisy data
l2_regularization Penalty on leaf values Raise it to smooth predictions
max_bins Bins per feature (default 255) Rarely changed

Trees here grow leaf-wise: the algorithm always splits the leaf that reduces the loss most, until max_leaf_nodes is reached. That is why the main size control is the number of leaves rather than the depth.

If you tune only three things, tune learning_rate, max_leaf_nodes and min_samples_leaf, and let early stopping choose the number of iterations.

Missing values without an imputer

Most scikit-learn models refuse to see a NaN. You must fill the gaps first (you will learn how in Handling Missing Data). The histogram models are different: during training each split also learns which side the missing values should go to, whichever lowers the loss most. At prediction time a NaN follows that learned direction.

The next example hides 15% of all cells at random, shows a linear model failing, and shows the booster carrying on. It then builds a feature that is pure noise except for where it is missing, to show that the model can even use the missingness as a signal.

Python Native missing value support

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last two lines are a double-edged sword. In this made-up data, people in one class often have the value missing, so the model learns that "missing" means "probably class 1". If missingness reflects something real (a customer who never filled in a field), that is useful. If it reflects a data-collection accident that will not repeat in production, it is a leak. Always ask why a value is missing before trusting that behavior.

Native categorical features

Categorical columns normally need one-hot or ordinal encoding first (Encoding Categorical Features covers this). HistGradientBoosting can split on categories directly: a split sends a subset of the categories to the left and the rest to the right, so it can group "Oslo and Turin" together without ever treating the labels as ordered numbers.

The simplest way to use this is a pandas column with the category dtype and categorical_features="from_dtype". The model then finds the categorical columns by itself.

Python Categories without one-hot encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In this small example the integer-coded version scores almost the same, because five categories can be isolated with a few extra splits. The native version pulls ahead as the number of categories grows, since one split can separate any group of categories at once and the integer codes impose an order that does not exist. Two details are worth remembering. Categorical values must be non-negative integers below max_bins (pandas categories are converted for you), and a category with a few dozen levels is fine, but a column with thousands of levels (like a user id) is not a good fit.

Early stopping

Boosting keeps adding trees, and past some point the extra trees only memorize noise. HistGradientBoosting can watch for that for you. With early_stopping=True it holds out validation_fraction of the training rows, scores them after every iteration, and stops when the score has not improved for n_iter_no_change rounds. You can then set max_iter generously.

The default is early_stopping="auto", which turns it on only when you have more than 10,000 rows. For smaller data sets, switch it on explicitly, as below. The fitted model keeps the curves in train_score_ and validation_score_ and the number of rounds it really used in n_iter_.

Python Early stopping picks the number of trees

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The training score keeps improving while the validation score flattens: the gap between them is overfitting. Because the validation rows are carved out of the training data, they are not the test set, which stays untouched for the final check (Train, Validation and Test Sets explains the three roles).

XGBoost, LightGBM and CatBoost

Three popular libraries implement the same family of algorithms, and you will meet them in job descriptions and competitions. They are not installed in the browser runner, so the snippet below is read-only; the model names and parameters are the same idea you just practiced.

Python
# Read-only: these packages are not available in the browser.
from xgboost import XGBClassifier
from lightgbm import LGBMClassifier
from catboost import CatBoostClassifier

XGBClassifier(n_estimators=500, learning_rate=0.05, max_depth=6)
LGBMClassifier(n_estimators=500, learning_rate=0.05, num_leaves=31)
CatBoostClassifier(iterations=500, learning_rate=0.05, depth=6)
Library Known for
XGBoost The long-time competition favorite; many options, GPU support
LightGBM Leaf-wise growth and histograms; very fast on large data
CatBoost Strong handling of high-cardinality categories out of the box

For a first model on a table that fits in memory, HistGradientBoosting gets you most of the way with no extra dependency, and all three libraries also work with scikit-learn pipelines and cross-validation because they follow the same fit/predict API.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What makes HistGradientBoosting faster than GradientBoostingClassifier on large data?

    Binning feature values up front means each split search looks at bin edges instead of every distinct value, so cost no longer grows with the number of unique values.
  2. 2. You pass a feature matrix containing NaN values to HistGradientBoostingClassifier. What happens?

    Missing values are supported natively. Each split learns a default direction for NaN, so no imputer is required.
  3. 3. Which setting stops training automatically when a held-out score stops improving?

    With early stopping enabled, the model holds out validation_fraction of the training rows and stops after n_iter_no_change rounds without improvement.
  4. 4. A feature is missing far more often for one class, but only because of a one-off data-entry accident. What is the risk of native NaN handling?

    The booster can use missingness as a signal, which is helpful when it is real but harmful when it is an artifact of how the data was collected.

Summary

  • HistGradientBoostingClassifier and HistGradientBoostingRegressor bin each feature into up to 255 bins, which makes boosting fast on large tables.
  • Control complexity with learning_rate, max_leaf_nodes, min_samples_leaf and l2_regularization; trees grow leaf-wise.
  • NaN values are handled natively, with a learned default direction at every split. Check why values are missing before trusting that.
  • Pandas category columns and categorical_features="from_dtype" let the model split on categories without one-hot encoding.
  • early_stopping=True chooses the number of iterations from a validation split; it is on automatically only above 10,000 rows.
  • XGBoost, LightGBM and CatBoost are close relatives, not available in the browser runner here.

Next, learn how to blend several different models into one in Stacking and Blending Models.