Skip to content
Tutorials Center
My learning

Topics

Lesson 51 of 111 ยท Feature Engineering and Pipelines

Encoding Categorical Features

Turn categories into numbers with OneHotEncoder, OrdinalEncoder and TargetEncoder, handle unknown and rare categories, and cope with high-cardinality columns safely.

Models do arithmetic, and "red", "green" and "blue" are not numbers. Every categorical column (color, city, product type, browser) must be converted into numeric features before most scikit-learn models can use it. How you convert matters: a careless encoding can invent an order that does not exist, explode your feature count, or leak the target into the features.

This lesson covers the three encoders you will use most: OneHotEncoder, OrdinalEncoder and TargetEncoder. It also shows how to deal with categories that appear only at prediction time and with columns that have hundreds of distinct values. It follows Scaling and Normalization, and the ColumnTransformer lesson shows how to apply encoders to just the right columns.

Choosing an encoder

One-hot One 0/1 column per category. Safe default.
Ordinal One integer column. Only for real order.
Target Category replaced by its average outcome.

Few categories, no order: one-hot. Hundreds of categories: target encoding or grouping.

1 / 2

One-hot encoding

One-hot encoding creates one new column per category, holding 1 where the row has that category and 0 elsewhere. It makes no assumption about order, which is correct for nominal data such as colors. OneHotEncoder learns the categories during fit; get_feature_names_out() tells you what each output column means.

Ordinal encoding

OrdinalEncoder maps categories to integers 0, 1, 2, and so on. This is right when the categories have a real order (S < M < L, "low" < "medium" < "high") and you pass that order explicitly through categories=. Without it, the encoder sorts alphabetically, which is rarely meaningful. For unordered categories, integer codes pretend that "red" is bigger than "blue", and a linear model or a k-NN will take that seriously. Tree models cope with ordinal codes of nominal data much better, because they can split the integers apart.

Python One-hot versus ordinal

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a feature with k categories, one-hot produces k columns. Some linear models then become redundant (the columns always sum to 1), so drop="first" can remove one. For regularized models and trees this does not matter, and keeping all columns is fine.

Unknown categories

At prediction time a new value can show up that training never saw, such as a city that opened last month. By default OneHotEncoder raises an error, which is useful in development and fatal in production. Set handle_unknown="ignore" and an unseen category becomes a row of all zeros. For OrdinalEncoder use handle_unknown="use_encoded_value" with an unknown_value such as -1.

Rare categories

Categories with a handful of rows give the model almost nothing to learn from and bloat the feature matrix. Use min_frequency (a count, or a fraction of rows) or max_categories to pool the rare ones into a single "infrequent" column. Combined with handle_unknown="infrequent_if_exist", unseen values land in that same bucket, which is a graceful default.

Python Unknown and rare categories

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Cities E and F each have fewer than ten rows, so they share one infrequent column, and the never-seen city "Z" falls into it as well.

High-cardinality features

What if a column has 500 postal codes? One-hot creates 500 mostly-empty columns, each backed by about six rows. Options, from simplest to most powerful:

  1. Group rare values (as above) or aggregate them yourself into larger regions.
  2. Ordinal codes with a tree model. Often good enough for forests and boosting; histogram gradient boosting also supports categorical columns natively.
  3. Target encoding. Replace each category with the average target value of its rows. One column, however many categories there are.

Target encoding has a trap. If you compute each category's mean using the same rows you then train on, every row's own label is part of its feature. For categories with few rows the feature nearly is the label, the model trusts it, and test performance collapses. The fix is cross-fitting (also called out-of-fold encoding): the encoding for each training row is computed from the other folds only. scikit-learn's TargetEncoder does this inside fit_transform, and also shrinks small-category means toward the global mean (smooth="auto") so tiny groups are not trusted blindly.

Python Target encoding: naive versus cross-fitted

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The naive encoding looks excellent on the training rows (AUC around 0.87) and drops to about 0.69 on the test rows. The cross-fitted encoding scores about 0.67 on training and 0.70 on the test set: the two numbers agree, so the training score can be trusted. Always keep the encoder inside the pipeline so cross-validation refits it on each training fold.

LabelEncoder: for targets, not for features

LabelEncoder turns the values of a single column into integers 0, 1, 2, ... in sorted order, remembers the mapping (classes_), and can reverse it with inverse_transform. It is meant for the target y: many estimators accept string labels directly, but some tools (for example XGBoost's older interfaces, or code that builds a confusion matrix by index) need integer classes, and you want to turn the predictions back into names afterwards.

It is not meant for input features, for two reasons. It handles one column at a time (use OrdinalEncoder for several feature columns), and the integers it produces invent an order and distances: with the cities Berlin, Cairo, Lima coded 0, 1, 2, a linear model believes that Cairo is "between" Berlin and Lima and that Lima is twice as far from Berlin as Cairo is. Trees can split such codes into groups and are much less affected; linear models, k-nearest neighbors and neural networks are not.

Pros of integer codes: one column per feature (no growth of the data), fast, and fine for tree models and for genuinely ordered categories (use OrdinalEncoder with an explicit order). Cons: a false order for unordered categories, no handling of unseen categories unless you configure it, and the mapping depends on the sorted order of the values in the fit data.

Python LabelEncoder on a target, and why integer codes mislead a linear model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On the integer codes the linear model can only draw a straight line through "0, 1, 2", so it cannot make Cairo expensive while Berlin and Lima stay cheap (its Rยฒ is about 0.01, and it predicts roughly the same price for all three cities). With one-hot columns each city gets its own value and the fit is exact. Rule of thumb: LabelEncoder for y, OrdinalEncoder for ordered features and tree models, OneHotEncoder for unordered features in other models (see the sections above).

Practical advice

  • Few unordered categories: OneHotEncoder(handle_unknown="ignore").
  • Truly ordered categories: OrdinalEncoder with explicit categories=.
  • Hundreds of categories: min_frequency, then TargetEncoder or a tree model with ordinal codes.
  • pandas get_dummies is fine for exploration, but it cannot remember the categories seen in training. Use scikit-learn encoders in real pipelines.
  • Never encode with the labels of the rows you evaluate on (see Nested Cross-Validation and Data Leakage).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is OrdinalEncoder a poor choice for a nominal feature like color when feeding a linear model?

    Integer codes tell a linear model that red=2 is twice blue=1 and further from green=0. Nominal categories have no such order, so one-hot encoding is the safer choice.
  2. 2. What does OneHotEncoder(handle_unknown="ignore") output for a category not seen during fit?

    An unseen category activates none of the learned columns, so every one-hot column is 0 for that row. The model still gets a valid input.
  3. 3. Why does scikit-learn's TargetEncoder use cross-fitting in fit_transform?

    If a row's own label contributes to its category mean, the feature leaks the target, especially for small categories. Computing the encoding from the other folds removes that leak.
  4. 4. A column has 800 distinct values, most appearing fewer than five times. Which option helps first?

    Rare values carry little signal and bloat the matrix. Pooling them or replacing the column with a single smoothed target-encoded number keeps the information while controlling size.

Summary

  • Categories must become numbers; the choice of encoding shapes what the model can learn.
  • OneHotEncoder is the safe default for unordered categories; OrdinalEncoder fits only truly ordered ones (state the order).
  • Use handle_unknown so unseen categories do not crash predictions, and min_frequency to pool rare ones.
  • TargetEncoder replaces a category by a smoothed average outcome, with cross-fitting to prevent leakage; it suits high-cardinality columns.
  • Keep encoders inside a pipeline so they are fitted on training data only.

Next: Handling Missing Data.