Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 40 ยท XGBoost API

XGBoost Categorical and Missing Values

Handle missing values with XGBoost's learned default direction and use enable_categorical for native category splits, compared against one-hot and ordinal codes, with real runnable examples.

Two data problems show up in almost every real table: values that are missing, and columns made of categories such as city, product or device. XGBoost has built-in answers for both. This lesson shows how missing values are routed, how to feed categorical columns directly, and how to compare that with the older encodings.

Missing values are a feature, not a bug

You do not have to impute before XGBoost. When it builds a split such as age < 40, it tries sending all missing rows to the left child and then to the right child, and keeps the direction with the better gain. That direction is stored in the tree as the default direction for that node. At prediction time a missing value simply follows it. Different nodes can send missing values different ways, so the model learns what "missing" means from the data.

Python Missingness that carries a signal

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model never saw an imputed number, yet it is right on the blank rows: the fact that an income is missing is informative here (every blank belongs to class 1), and the trees learned to send NaN down the branch that leads to class 1. Read the dump: each split line ends with missing= pointing to the yes or no branch, which is the learned default direction.

Filling blanks with a constant also works for trees when that constant stands apart from real values, but a fill that lands inside the real range (like a median) mixes blanks with genuine rows, and filling is a bigger problem for linear or distance-based models. Keeping NaN costs nothing and keeps the signal.

Practical rules:

  • If missing values mean something ("not applicable", "never recorded"), leave them as NaN and let the trees learn.
  • If a sentinel such as -999 or 0 marks missing, either convert it to NaN or pass missing=-999 to the model or DMatrix. Otherwise it is treated as a real number.
  • Sparse inputs (SciPy CSR) treat absent entries as missing, not as zero, so be careful converting one-hot data to sparse form.

Categorical features

The classic approach is to encode categories before training:

  • One-hot encoding gives each category its own 0/1 column. It works but explodes the width when there are many categories, and a tree needs several splits to isolate a group of categories.
  • Ordinal codes map categories to 0, 1, 2, ... This is compact, but it invents an order. A split code < 4 groups categories that have nothing in common.

XGBoost (hist and approx methods) can instead split on sets of categories: city in {Lima, Oslo} goes left, the rest right. You enable it with a pandas category dtype plus enable_categorical=True.

Python Native categories vs one-hot vs ordinal codes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a scattered set of good shops and only 30 shallow trees, native categories can express "this group of shops" in a single split, while ordinal codes cannot and one-hot needs many splits. In one run the native model scored about 0.76, ordinal codes about 0.72 and one-hot about 0.65. Exact numbers vary with the seed and sizes, so the lesson is the comparison, not a promise: always test encodings on your own data.

Rules for native categorical support

  • Use enable_categorical=True and the hist, approx or GPU tree methods (hist is the default in XGBoost 2.x).
  • Columns must have the pandas category dtype (or you pass feature_types=["c", "q", ...] to a DMatrix).
  • Categories are stored as integer codes from 0 to n-1. If your train and test frames have different category sets, convert them with the same categories (pd.CategoricalDtype(train_categories)), otherwise the codes will not line up.
  • max_cat_to_onehot switches small categorical columns to one-vs-rest splits, which is usually better for few levels.
  • max_cat_threshold caps how many categories can be sent to one side of a split, as a guard against overfitting.
Python Keeping category codes aligned between train and test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

An unseen level becomes NaN with code -1, which XGBoost then treats like any other missing value. That is a graceful fallback, but it is worth logging how often it happens in production.

High-cardinality columns

A column with thousands of levels (user id, zip code) invites overfitting even with native support, because a set-split can memorize individual categories. Options: group rare levels into "other", use frequency or target statistics computed inside cross-validation folds, or drop identifiers that carry no generalizable meaning.

flowchart TB
  A["Column type"] --> B{"Missing values?"}
  B -->|yes| C["Keep NaN, learned default direction"]
  A --> D{"Categorical?"}
  D -->|"few levels"| E["enable_categorical + max_cat_to_onehot"]
  D -->|"many levels"| F["Group rare levels, then native or target encoding in CV"]
  D -->|"has real order"| G["Ordinal codes are fine"]

Choose per column, then check on a validation set.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does XGBoost route a missing value at a split?

    During training both directions are tried for missing rows, and the better one is stored as the node's default.
  2. 2. What is needed to use native categorical splits in the scikit-learn wrapper?

    Columns must be categorical dtype and the flag must be on, with a histogram-based tree method.
  3. 3. Why can ordinal codes hurt on a nominal column like city?

    A split code < 4 puts categories 0-3 together simply because of their numbering.
  4. 4. A sentinel value such as -999 stands for missing. What should you do?

    Otherwise the sentinel is treated as an ordinary, very small number.

Summary

  • XGBoost learns a default direction for missing values at every split, so imputing is optional and can hide a useful signal.
  • Convert sentinel values to NaN or set the missing parameter.
  • Native categorical splits need pandas category columns and enable_categorical=True.
  • Keep category sets identical between training and prediction; unseen levels act as missing.
  • Compare native, one-hot and ordinal encodings on validation data and handle high-cardinality columns with care.

Next: Objectives and Evaluation Metrics