Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 41 ยท ML Lifecycle and Pipelines

scikit-learn Pipelines for Production

Bundle preprocessing and model into one scikit-learn Pipeline and ColumnTransformer so training and serving use identical steps, with leakage checks and examples.

A trained model is useless without the exact preprocessing used to create its inputs. If you scale and encode data in a notebook, then re-implement those steps in a serving API, any small difference (a different mean, a new category order) produces wrong predictions that nobody notices. This is training-serving skew, and it is one of the most common production bugs.

The cure is to treat preprocessing and model as one object. In scikit-learn that object is a Pipeline. You fit it once, save it once, and call it the same way in training, batch scoring and serving.

The idea in one picture

flowchart LR
  R["Raw input row"] --> I["Impute missing"]
  I --> S["Scale and encode"]
  S --> M["Model"]
  M --> P["Prediction"]

Everything between raw input and prediction lives inside one saved object.

A first pipeline

A ColumnTransformer applies different steps to different columns, and a Pipeline chains the result with the model.

Python Preprocessing and model as one object

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The new row has a missing age and a country the model never saw. The pipeline imputes, scales and encodes it with the training statistics, so it does not crash.

Why the pipeline prevents leakage

Data leakage happens when information from the test set influences training. A classic example is scaling the whole dataset before splitting: the mean and standard deviation then contain test information. Inside a pipeline, cross-validation re-fits every step on each training fold only.

Python Leakage with and without a pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On random labels the honest score hovers around 0.5, while the leaky score looks impressive. A model like that would fail in production.

Custom steps

When no built-in transformer fits, write your own class with fit and transform, or use FunctionTransformer for stateless functions.

Python A custom transformer inside a pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The clipping limits are learned in fit (stored with a trailing underscore) and reused in transform. That is the contract that makes a step safe to save and serve.

Inspecting and tuning a pipeline

Steps are addressed by name: pipe.named_steps["model"], or in a parameter grid as model__C. A tuned pipeline is still one object, so the best model keeps its preprocessing.

Python
from sklearn.model_selection import GridSearchCV
grid = GridSearchCV(pipe, {"model__C": [0.1, 1, 10]}, cv=5)

Checklist for production pipelines

  • No manual steps outside the pipeline. If you do it in a notebook before fit, serving will not do it.
  • Handle unseen categories (handle_unknown="ignore") and missing values.
  • Validate the input columns before the pipeline runs (data validation).
  • Save the whole pipeline, not just the final estimator (saving and loading models).
  • Keep column names stable; pass a DataFrame so order mistakes are impossible.

Pipelines work for classic ML. Deep learning stacks have their own preprocessing layers, but the principle is the same: ship preprocessing together with the model.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What problem does bundling preprocessing and the model in one Pipeline mainly prevent?

    The same fitted steps run at training and serving time, so inputs are treated identically.
  2. 2. Why is scaling the full dataset before cross-validation a mistake?

    Statistics computed on all rows include the validation rows, inflating the score.
  3. 3. In a custom transformer, where should learned values be computed?

    fit learns from training data; transform applies what was learned to any data.
  4. 4. What does OneHotEncoder(handle_unknown='ignore') do for an unseen category?

    Unknown categories produce a row of zeros, so serving does not crash.

Summary

  • Put preprocessing and the model in one Pipeline to avoid training-serving skew.
  • Use ColumnTransformer for per-column steps and handle_unknown="ignore" for new categories.
  • Pipelines keep cross-validation honest by refitting every step on each training fold.
  • Custom steps learn in fit and apply in transform.
  • Save and deploy the whole pipeline as one artifact.

Next: Pipeline Orchestration and DAGs