Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 40 ยท LightGBM and CatBoost

LightGBM Dataset and Categorical Features

Use lgb.Dataset and lgb.train for the native LightGBM API, run lgb.cv, and handle categorical columns and missing values directly, with the parameters that control category splits.

LightGBM's native API centers on lgb.Dataset, the equivalent of XGBoost's DMatrix, and lgb.train. It is also where LightGBM's best-known feature lives: built-in handling of categorical columns, with no one-hot encoding needed. This lesson covers the native workflow and how categories are treated.

Dataset and lgb.train

A Dataset wraps features, labels and optional weights. Validation sets are created with reference=train_set so that they use the same bin boundaries as the training data. Training takes a parameter dictionary and callbacks.

Python lgb.Dataset and lgb.train with callbacks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notes:

  • The parameter names are the LightGBM ones (objective, metric, num_leaves, seed), and the number of rounds is the num_boost_round argument.
  • booster.predict takes raw arrays (not a Dataset). It returns probabilities for binary, a probability matrix for multiclass (which needs num_class), and the plain value for regression. raw_score=True gives log-odds.
  • Binning uses the first Dataset. A validation set built without reference= computes its own bins, which can give mismatched results.
  • Datasets are constructed lazily; free_raw_data=True (default) discards the original arrays after construction to save memory.

Cross-validation with lgb.cv

lgb.cv returns a dictionary of per-round fold means and standard deviations. Its length tells you how many rounds were kept.

Python lgb.cv for the number of rounds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

(The key names have the form valid <metric>-mean in current versions, so print list(res) if you are on another release.)

Categorical features

Tell LightGBM which columns are categorical and it will search for the best subset of categories to send to one side of each split. Internally it sorts the categories by their accumulated gradient statistics and tries split points along that order, which finds the optimal partition efficiently for squared error and a good one for other losses.

There are three ways to declare them:

  1. Use a pandas category dtype column, and with the default categorical_feature="auto" LightGBM picks them up.
  2. Pass categorical_feature=["shop"] (names) or indexes to Dataset or fit.
  3. Encode categories as non-negative integers (0, 1, 2, ...) and name them as categorical. Negative values are treated as missing, and very large codes (beyond about 2 billion) are not allowed.
Python Native categorical splits vs one-hot vs ordinal

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As in the XGBoost lesson on categorical features, the native approach is expected to do best when a scattered group of categories matters. Check on your own data before relying on it.

Parameters that govern category splits

Parameter Default Role
max_cat_to_onehot 4 features with this many categories or fewer use one-vs-rest splits
cat_smooth 10 smoothing of per-category statistics, which stabilizes rare categories
cat_l2 10 extra L2 regularization for categorical splits
min_data_per_group 100 minimum rows per category group
max_cat_threshold 32 maximum categories in a split's left set

Categorical splits overfit easily on high-cardinality columns because a subset search has many options. If a categorical column hurts validation scores, raise cat_smooth, cat_l2 and min_data_per_group, group rare levels into an "other" bucket, or fall back to an ordinal/target-encoded version.

Missing values

LightGBM also learns where missing values go. With use_missing=True (default) NaN rows are tried on each side and the better direction is kept at each split. Two related switches:

  • zero_as_missing=True treats zeros (and absent sparse entries) as missing too.
  • use_missing=False disables the special handling, so NaN is treated as zero.
Python Missing values are routed, not imputed

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model predicts 1 for the NaN row while predicting 0 for a clearly negative value, so it learned that "missing" has its own meaning.

flowchart TB
  C["Categorical column"] --> A{"Number of categories"}
  A -->|"<= max_cat_to_onehot"| O["One-vs-rest splits"]
  A -->|"more"| S["Sort categories by gradient statistics<br/>and split the sorted list"]
  S --> R["Regularize: cat_smooth, cat_l2, min_data_per_group"]

LightGBM picks the split style for each categorical feature, then regularizes it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why pass reference=train_set when creating a validation Dataset?

    Both sets must be binned the same way for the scores to be comparable.
  2. 2. What happens to a negative integer in a categorical column in LightGBM?

    Categorical features must be non-negative integers; negatives are interpreted as missing.
  3. 3. Which parameter makes LightGBM use one-vs-rest splits for small categorical features?

    Features with at most that many categories are split one versus the rest.
  4. 4. A high-cardinality categorical column makes validation worse. What can you try?

    More regularization of categorical splits, or grouping rare categories, reduces overfitting to individual levels.

Summary

  • lgb.Dataset plus lgb.train is the native API; build validation sets with reference= the training set.
  • lgb.cv returns per-round fold statistics for choosing num_boost_round.
  • Declare categorical columns by dtype, name or index; they use subset splits and need non-negative integer codes.
  • cat_smooth, cat_l2, min_data_per_group and max_cat_to_onehot control category behavior.
  • Missing values follow learned directions; zero_as_missing and use_missing change that.

Next: CatBoost and Ordered Boosting