Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 76 ยท Cleaning and Preparing Data

Encoding and Scaling Features

Prepare features for models: one-hot and ordinal encoding, target encoding pitfalls, standard, min-max and robust scaling, power transforms and ColumnTransformer.

Most modelling code expects a matrix of numbers. Real tables contain words ("Berlin", "premium"), numbers on wildly different scales (age in years next to income in dollars) and a few extreme values. Encoding turns categories into numbers; scaling puts numeric columns on comparable ranges. Both are small steps with large consequences: the wrong encoding can invent an order that does not exist, and a scaler fitted on the wrong rows can leak information from the test set.

This lesson uses scikit-learn's transformers, which follow one pattern: fit learns something from data (the categories, the mean, the range) and transform applies it. Keep that pair in mind, because the central rule of this lesson is fit on training data only. If you have not seen the engineering side of new columns yet, read Feature Engineering first.

flowchart LR
    T["Training rows"] -->|"fit: learn categories, means, ranges"| S["Fitted transformer"]
    S -->|"transform"| X1["Training features"]
    S -->|"transform"| X2["New or test features"]

A transformer learns from the training rows and then applies the same rules to any new rows.

Encoding categories

One-hot encoding

For nominal categories with no natural order (city, color, payment method), one-hot encoding creates one 0/1 column per category. A row with city "Oslo" gets a 1 in city_Oslo and 0 elsewhere. Nothing is implied about order or distance, which is exactly what you want.

Two practical settings matter:

  • handle_unknown="ignore" makes a category that appeared only after training turn into all zeros instead of raising an error, which is what you want in production.
  • Very high-cardinality columns (thousands of ZIP codes) create thousands of columns. Group rare values first (see Cleaning Text and Categories), or use another encoding.

Ordinal encoding

For ordinal categories, where the order carries meaning (size S < M < L, rating low < medium < high), replace each value by its rank. Always pass the order explicitly with categories=[[...]]; otherwise scikit-learn sorts alphabetically, which would place "high" before "low".

Python One-hot and ordinal encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that "Cairo" becomes a row of zeros instead of crashing, and that S, M, L map to 0, 1, 2 in the order you gave.

Target encoding and its pitfall

Target encoding replaces a category by the average of the target within that category (for example, the churn rate of each city). It is compact and works for high-cardinality columns, but it is a classic source of leakage: a rare category with one row gets that row's own target as its feature, so the model "sees" the answer. The remedies are to compute it only inside cross-validation folds, shrink rare categories toward the global mean, and be sceptical of results that look too good. When in doubt, use one-hot encoding with rare-category grouping.

Scaling numeric columns

Distance-based and gradient-based methods (k-nearest neighbors, clustering, regularized regression, neural networks, PCA) are sensitive to the scale of each column: income in dollars would drown out age in years. Tree models are not affected, but scaling is harmless there. Three common scalers:

Scaler Formula Use when
StandardScaler (x - mean) / standard deviation Roughly bell-shaped data, no big outliers; the default choice
MinMaxScaler (x - min) / (max - min), into [0, 1] You need a fixed range, for example inputs to a neural network or image pixels
RobustScaler (x - median) / IQR Outliers are present; median and IQR barely move when one value explodes

The difference shows up as soon as one extreme value appears:

Python Three scalers, one outlier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With min-max scaling, the seven ordinary values are squeezed into a sliver near 0 because the outlier owns the top of the range. The robust scaler keeps them spread out. Scaling does not remove an outlier; it only changes its units. Deal with errors and extremes first (Outliers, Errors and Impossible Values).

Power transforms

For skewed positive data, a power transform reshapes the distribution toward a bell shape. PowerTransformer(method="yeo-johnson") works on any real numbers (Box-Cox needs strictly positive values) and standardizes the result. A plain np.log1p (Feature Engineering) is simpler and easier to explain; use the power transform when you want the best automatic choice.

Label encoding: for targets, not features

LabelEncoder turns the values of one column into integers 0, 1, 2, ..., sorted alphabetically. It was designed for the target y of a classification problem (class names such as "cat", "dog", "bird"), not for input features: it knows nothing about order and does not accept a two-dimensional table. Using it on a feature column is a common mistake, because the integers suggest an order and a distance (is "dog" twice "cat"?) that do not exist. For features use one-hot encoding for nominal columns and OrdinalEncoder with an explicit order for ordinal columns, as shown above.

Python LabelEncoder for the target and why not for features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how the alphabetical codes put L below S. The result is a column that looks numeric and is silently wrong. With pandas you can also use df["col"].astype("category").cat.codes, which has the same limitations.

Efficient preprocessing for large datasets

When a dataset has millions of rows, the same preprocessing steps still apply, but memory and time start to matter. The ideas below are the first things to try before reaching for a cluster.

  1. Read only what you need. usecols and dtype in read_csv skip unwanted columns and avoid 64-bit defaults.
  2. Use compact dtypes. category for repeated strings, int8/int32 and float32 for numbers that fit.
  3. Process in chunks. read_csv(..., chunksize=N) streams the file so that only one chunk is in memory at a time.
  4. Prefer sparse output for one-hot encoding of many categories (sparse_output=True) and group rare categories (min_frequency).
  5. Fit incrementally. StandardScaler.partial_fit and models with partial_fit learn from one batch after another.
  6. Convert to a columnar format such as Parquet once, so later reads are fast and typed.
Python Compact dtypes and chunked processing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Incremental scaling and sparse encoding

partial_fit lets a scaler or a model learn from batches and gives the same result as fitting on all the data at once. For categories with many levels, a sparse one-hot matrix stores only the ones, and min_frequency merges rare levels into one "infrequent" column.

Python partial_fit scaling and sparse one-hot encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When a table no longer fits

If the data are still too big for one machine after these tricks, use tools that split the work. These examples are read-only, because the site cannot run a cluster.

Python
# Dask: a pandas-like API that processes partitions in parallel and out of core
import dask.dataframe as dd
ddf = dd.read_parquet("events/*.parquet", columns=["user", "amount"])
result = ddf.groupby("user")["amount"].sum().compute()

# PySpark: the same idea on a cluster
from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()
sdf = spark.read.parquet("events/")
sdf.groupBy("user").sum("amount").show()

Two rules carry over from small data. Fit encoders and scalers on training data only, even when the data are huge, and save the fitted objects with the model so that production applies the identical transformation. And sample first: develop the pipeline on a 1% sample, then run it on everything.

The golden rule: fit on training data only

A scaler's mean and standard deviation, an encoder's category list, an imputer's median: all are learned from data. If you learn them from the full table before splitting, information about the test rows leaks into training, and your evaluation becomes optimistic. The correct order is:

  1. Split into train and test.
  2. fit the transformer on the training part.
  3. transform both parts with that fitted object.

ColumnTransformer: different treatment per column

Real tables mix numeric and categorical columns, and each needs its own steps. ColumnTransformer applies chosen transformers to chosen columns and glues the results into one matrix. Combined with a Pipeline (Pipelines and Hyperparameter Tuning) it makes the fit-on-train rule automatic.

Python A ColumnTransformer on mixed columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The training means are exactly 0 because the scaler was fitted on them. The test means are close to, but not exactly, 0. That small difference is normal and honest: the test rows were scaled with the training statistics, just as future production data will be.

Never call fit_transform on the test set. That re-learns the scaling from test rows and quietly changes the meaning of every feature.

Which one should I use?

flowchart TD
    A["Column type?"] --> B["Numeric"]
    A --> C["Categorical"]
    B --> B1["Outliers or skew?"]
    B1 -->|"yes"| B2["RobustScaler or log / power transform"]
    B1 -->|"no"| B3["StandardScaler"]
    C --> C1["Natural order?"]
    C1 -->|"yes"| C2["OrdinalEncoder with explicit order"]
    C1 -->|"no"| C3["OneHotEncoder, group rare values"]

A quick decision guide; trees can usually skip scaling, but they still need categories turned into numbers.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which encoding is appropriate for a payment method column with values card, cash and transfer?

    The values have no natural order. Ranks would suggest that transfer is "more" than card, which is meaningless; one-hot columns imply no order.
  2. 2. What does handle_unknown="ignore" do in OneHotEncoder?

    In production new categories will appear. With this setting they produce a row of zeros rather than crashing the pipeline.
  3. 3. A column has a few huge outliers. Which scaler is least distorted by them?

    RobustScaler uses the median and interquartile range, which hardly move when a few values explode. Min-max uses the extremes and would squeeze everything else together.
  4. 4. You scale first and split afterwards. What is the problem?

    Statistics learned from the whole table include the test rows. Split first, fit on training data only, then transform both parts.
  5. 5. Why is target encoding risky for a category that appears once?

    With one row, the average target of that category is that row's own target. Use cross-validation folds and shrinkage, or choose a different encoding.

Summary

  • Models need numbers: one-hot encode nominal categories, ordinal-encode ordered ones with an explicit order.
  • Target encoding is powerful but leaks easily; use folds and shrinkage, or avoid it.
  • StandardScaler is the default, MinMaxScaler gives a fixed range, RobustScaler resists outliers; power transforms reshape skew.
  • Always fit on training data only, then transform train and test with the same fitted object.
  • ColumnTransformer applies the right steps to each column and keeps the whole recipe in one object.

Next: A Systematic EDA Process, where you start exploring a dataset step by step.