Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 40 ยท Pipelines and Feature Engineering

Pipelines and ColumnTransformer

Chain preprocessing and models with scikit-learn Pipeline, handle numeric and categorical columns with ColumnTransformer, avoid data leakage and tune the whole workflow.

Real data is messy: some columns are numbers with gaps, some are categories, and each needs different treatment before a model can use it. If you do those steps by hand, it is easy to make two classic mistakes: applying them differently at training and prediction time, and letting test data influence the preprocessing (data leakage).

A Pipeline chains steps so that data flows through them in order, and the whole chain behaves like one estimator with fit, predict and score. A ColumnTransformer applies different preprocessing to different columns. Together they are the standard way to build a reliable scikit-learn workflow.

Pipeline basics

A pipeline is a list of (name, estimator) pairs. All steps but the last must be transformers (have fit and transform); the last can be any estimator.

Python A simple pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When you call pipe.predict(X_new), the data goes through the already fitted scaler (using the training mean and standard deviation) and then the classifier. You cannot forget a step or use the wrong statistics. make_pipeline(StandardScaler(), LogisticRegression()) is a shortcut that names the steps automatically in lowercase (standardscaler, logisticregression).

Why it prevents leakage

Python Leakage: scaling before cross-validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The leaky version "learns" from the test folds when it chooses features, so it reports skill that does not exist. Inside a pipeline, every fold repeats the selection on its own training rows only.

ColumnTransformer

Mixed-type tables need different steps per column: impute and scale numbers, impute and one-hot encode categories. ColumnTransformer takes a list of (name, transformer, columns) and concatenates the outputs.

A typical tabular pipeline

Numeric Impute median, then scale.
Categorical Impute most frequent, one-hot encode.
Model Gets one clean numeric matrix.

One object holds the whole recipe, from raw table to prediction.

1 / 2
Python Mixed-type pipeline on a small DataFrame

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Points worth noticing:

  • handle_unknown="ignore" means a city never seen in training becomes all-zeros instead of crashing at prediction time.
  • You pass the raw DataFrame; the pipeline knows which columns go where.
  • get_feature_names_out() shows the final column names, handy for inspecting coefficients and importances.
  • Columns not listed are dropped by default. Set remainder="passthrough" to keep them untouched.

Choosing columns by type

Hard-coding names is fragile. make_column_selector picks columns by dtype or regular expression.

Python make_column_selector and make_column_transformer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Tuning a pipeline

Step names make every nested parameter addressable with double underscores, even across pipelines in a ColumnTransformer: prep__num__impute__strategy.

Python Grid search over preprocessing and model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Useful extras

  • Pipeline(..., memory="cache_dir") caches fitted transformers, which saves time during searches.
  • pipe.set_output(transform="pandas") makes transformers return DataFrames with column names.
  • FeatureUnion runs several transformers in parallel on the same input and joins their outputs side by side.
  • Print a pipeline in a notebook for an interactive diagram, or call sklearn.set_config(display="text") for text.
  • Slice a pipeline: pipe[:-1] is all steps but the last; pipe[-1] is the final estimator.

Next you will write your own steps: see custom transformers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main reason to put a scaler inside a Pipeline before cross-validating?

    Fitting preprocessing on all data lets test-fold information leak into training.
  2. 2. Which ColumnTransformer setting keeps columns you did not list?

    By default unlisted columns are dropped; passthrough appends them unchanged.
  3. 3. What does OneHotEncoder(handle_unknown='ignore') do for a category not seen during training?

    An unseen category gets a row of zeros, so prediction does not crash.
  4. 4. How do you reach the strategy parameter of an imputer step named impute inside the transformer num of ColumnTransformer step prep?

    Nested parameter names are joined with double underscores.

Summary

  • A Pipeline chains transformers and a final estimator into one object; use make_pipeline for short names.
  • Pipelines prevent leakage because every fold refits every step on its own training data.
  • ColumnTransformer applies different preprocessing to numeric and categorical columns; select columns by name or with make_column_selector.
  • Set handle_unknown="ignore" for encoders, and tune any nested parameter with step__param names.

Next: Custom Transformers