Skip to content
Tutorials Center
My learning

Topics

Lesson 58 of 111 ยท Feature Engineering and Pipelines

Pipelines

Chain preprocessing and a model into one scikit-learn Pipeline: step names, parameter routing, use in cross-validation and grid search, leakage and caching.

A real model is rarely a single estimator. Data is imputed, scaled, encoded, perhaps reduced, and only then handed to a classifier. Doing these steps by hand means keeping track of which statistics came from which rows, and one slip (fitting the scaler on all the data) quietly inflates your score. A Pipeline chains the steps into one object that behaves like a single estimator: fit runs every step in order, and predict replays them on new data.

Pipelines give you three things at once. The code stays short, the same preprocessing is applied in training and in production, and cross-validation and grid search refit every step on each training fold, so there is no leakage. This lesson covers building pipelines, naming steps, routing parameters to them, and using them with model selection.

What a pipeline does

Transformers Every step but the last calls fit_transform.
Final estimator The last step is the model (or another transformer).
Replay predict runs transform through all steps, then the model.

One object, one fit, one predict.

1 / 2

Building a pipeline

Pipeline takes a list of (name, estimator) pairs. make_pipeline builds the same thing and names the steps for you from the lowercase class names. All steps except the last must be transformers (they need transform); the last can be anything.

Python Pipeline versus doing it by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two scores are identical. You reach steps by name (pipe.named_steps["clf"]), by index (pipe[0]) or by slice (pipe[:-1] is a pipeline of all the transformers). Slicing is a handy way to look at what the model really receives: pipe[:-1].transform(X).

flowchart LR
    A["raw X"] --> B["StandardScaler<br/>fit, transform"]
    B --> C["LogisticRegression<br/>fit"]
    D["new X"] --> E["scaler.transform"]
    E --> F["clf.predict"]

In training each step is fitted on the output of the one before. At prediction time the fitted steps only transform.

Step names and parameter routing

Every parameter of every step can be reached with the name stepname__parameter, using a double underscore. This works in get_params, set_params, and in the grid of a search. Nested pipelines extend the path: columns__num__imputer__strategy (see ColumnTransformer and Mixed Data).

Python Reading and setting parameters through step names

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Setting a step to "passthrough" (or "drop") removes its effect without changing the rest of the pipeline. That makes it easy to search over "scaling or no scaling" as if it were a hyperparameter.

Pipelines in cross-validation

Pass the pipeline, not the transformed data, to cross_val_score. Each fold clones the pipeline and fits every step on that fold's training rows only. The experiment below peeks inside each fold's fitted scaler to prove it.

Python Each fold fits its own scaler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The fold means differ from each other and from the overall mean, because each scaler saw only its own training rows. If you scaled the whole dataset first, every fold would share one mean that was computed partly from its validation rows. For the dataset-wide picture see Cross-Validation.

A grid search can tune the preprocessing and the model together. Keys of the parameter grid are step names joined to parameter names with __.

Python Searching over a model and a preprocessing step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

search.best_estimator_ is the whole fitted pipeline, ready to predict. The next lessons build on this: Hyperparameter Tuning covers the search itself and Nested Cross-Validation and Data Leakage covers the bias of reporting the best search score.

Caching expensive steps

During a grid search the same preprocessing step is refitted for every combination of later parameters, even when only the classifier changed. The memory argument of Pipeline stores each fitted transformer on disk (via joblib) and reuses it when its input and parameters are unchanged. It pays off for slow transformers such as large TF-IDF vocabularies or PCA on big data.

Python
from tempfile import mkdtemp
from sklearn.pipeline import Pipeline

cache_dir = mkdtemp()
pipe = Pipeline(steps, memory=cache_dir)   # fitted transformers are cached here

The counting experiment below shows the effect without needing a slow step. A function transformer increments a counter each time it runs on data; the grid changes only the classifier, so a cache should avoid recomputing the first step.

Python Counting how often a step runs, with and without caching

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without a cache the function runs 18 times: three folds times three values of C gives nine fits, and each fit also transforms its validation fold once for scoring. With the cache it runs 12 times: the fitting calls drop from nine to three (one per fold), because the first step depends only on the fold, while the nine scoring calls remain. The cache is not worth it for cheap steps, because reading it from disk costs time too.

Cached transformers are returned as clones. If you want to inspect the fitted step afterwards, read it from the fitted pipeline, not from the object you passed in.

Common pipeline mistakes

  • Fitting a transformer outside the pipeline (for example scaling all data first) and then cross-validating. This leaks.
  • Forgetting that the last step is the model: pipe.predict needs a final step with predict.
  • Wrong parameter names: C instead of clf__C. pipe.get_params().keys() lists the valid names.
  • Transforming the target inside a feature pipeline. Pipelines transform X only. For y, use TransformedTargetRegressor.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does pipe.fit(X, y) do with the transformer steps?

    Every step except the last is fitted and applied (fit_transform), and its output becomes the next step's input. The last step is only fitted.
  2. 2. How do you set the C of a LogisticRegression step named 'clf' in a grid search?

    Parameters of a step are addressed as stepname, two underscores, parameter name.
  3. 3. Why is it safer to pass a pipeline to cross_val_score than to scale all the data first?

    With a pipeline the validation rows never take part in fitting the scaler. Scaling everything first lets their statistics leak into the training.
  4. 4. What does setting a step to 'passthrough' do?

    passthrough turns the step into an identity transformation, which is useful for searching over 'with or without this step'.
  5. 5. When does the memory argument of Pipeline help most?

    The cache avoids refitting a transformer whose input and parameters did not change. For cheap steps the disk access can cost more than it saves.

Summary

  • A Pipeline chains transformers and a final estimator into one object with fit, predict and score.
  • Steps are reached by name, index or slice; make_pipeline names them automatically.
  • Route parameters with step__param, in set_params and in search grids; use "passthrough" to switch a step off.
  • In cross-validation and grid search every fold refits all steps on its own training rows, which prevents preprocessing leakage.
  • memory= caches fitted transformers when an expensive step is repeated with unchanged input.

Next: ColumnTransformer and Mixed Data.