Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 39 ยท Data Preprocessing

Building a Preprocessing Pipeline

Combine cleaning, imputation, scaling and encoding into one reusable scikit-learn pipeline that avoids data leakage, with a complete worked example you can run.

You now know the individual preprocessing steps. In a real project you apply a dozen of them, in the right order, to training data and again to every new batch. Doing it by hand invites two bugs: forgetting a step later, and data leakage, where information from the test set sneaks into the preparation. A pipeline solves both by packaging the steps as one object that is fitted once and reused.

What is leakage?

Suppose you standardize the whole table and then split it into training and test sets. The mean and standard deviation then contain information about the test rows. The model's score is slightly too optimistic. The same happens when you fill missing values, select features or encode categories before splitting. The rule is: split first; learn every preparation statistic from the training part only.

flowchart LR
    A["Raw data"] --> B["Split"]
    B --> C["Train"]
    B --> D["Test"]
    C --> E["fit pipeline"]
    E --> F["train model"]
    D --> G["transform only"]
    G --> H["evaluate"]
    F --> H

The pipeline is fitted on the training branch and only applied on the test branch.

The tools

  • Pipeline chains steps; each step but the last must be a transformer.
  • ColumnTransformer applies different steps to different columns (numbers versus categories).
  • SimpleImputer, StandardScaler, OneHotEncoder are the usual building blocks.

A leaky and a clean workflow

Python Leakage in action: scaling before splitting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two means differ because the leaky one has "seen" the extreme values in the test portion.

A complete pipeline

We build a table with numbers, categories, holes and a class label, then preprocess and model it in one object.

Python ColumnTransformer plus a model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Calling model.predict(new_rows) runs imputation, scaling, encoding and the classifier in one go, using the numbers learned on the training data. New categories in the future (a fourth city) are ignored rather than crashing, thanks to handle_unknown="ignore".

Inspect what the pipeline did

Pipelines are not black boxes: you can look at any step.

Python Look inside the fitted pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Custom steps

When no built-in transformer exists, wrap a function with FunctionTransformer. This keeps the step inside the pipeline.

Python A custom log step in a pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Good habits

  • Keep all preparation inside the pipeline; do no hand edits in between.
  • Split before fitting anything; use stratify=y for imbalanced classes.
  • Use cross-validation on the whole pipeline (cross_val_score(model, X, y)) so each fold re-learns its own statistics. See Cross-Validation.
  • Save the fitted pipeline (joblib), not just the model, so production data is prepared identically.
  • Log what the pipeline removes or imputes, so surprises in new data show up early.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is data leakage in preprocessing?

    Fitting scalers, imputers or selectors on all data lets test information influence training.
  2. 2. Which tool applies different steps to numeric and categorical columns?

    ColumnTransformer routes column groups to separate transformers.
  3. 3. On the test set you should call a fitted scaler's:

    transform reuses the training statistics; fitting again would leak test information.
  4. 4. What does handle_unknown='ignore' do in OneHotEncoder?

    A category never seen in training becomes a row of zeros instead of a crash.

Summary

  • Split first, then learn every preparation statistic from training data only.
  • Pipeline and ColumnTransformer bundle imputation, scaling, encoding and the model into one object.
  • The same fitted pipeline prepares training, test and future data identically.
  • Cross-validate the whole pipeline and save it for deployment.

Next: Summary Statistics and Distributions