Skip to content
Tutorials Center
My learning

Topics

Lesson 52 of 76 ยท Modelling

Pipelines and Hyperparameter Tuning

Build leak-free scikit-learn pipelines with ColumnTransformer, tune hyperparameters with grid and randomized search, and know when a simple model is enough.

Real modelling code has several steps: fill missing values, scale numbers, encode categories, fit a model. If you run those steps by hand, it is easy to make a mistake that ruins your results, such as computing a mean on the test rows or forgetting to apply the same encoding to new data. A pipeline packages the steps into one object that learns from the training data and applies exactly the same recipe everywhere else.

Once everything is in a pipeline, tuning becomes safe. Hyperparameters are the settings you choose before training (tree depth, learning rate, regularization strength), as opposed to the parameters the model learns itself. This lesson shows how to build a pipeline with ColumnTransformer, how to search over hyperparameters with grid and randomized search without leaking test information, and why the most elaborate model is not always the right one. It follows train/test splits and cross-validation and uses the preprocessing ideas from encoding and scaling.

Why pipelines

A pipeline chains transformers (steps with fit and transform) and ends with an estimator (a model with fit and predict). Calling pipeline.fit(X_train, y_train) fits every step, in order, on the training rows only. Calling pipeline.predict(X_new) applies the learned transformations and then the model. Three benefits follow:

  1. No leakage. In cross-validation, the whole pipeline is re-fitted inside each fold, so the imputer's median and the scaler's mean come only from that fold's training rows.
  2. Reproducibility. One object holds the whole recipe; you can save it and load it later (see from model to decision).
  3. Tidy tuning. You can search over settings of any step, such as the imputation strategy, with the same code.
flowchart LR
    R["Raw table"] --> N["Numeric columns<br/>impute median, then scale"]
    R --> C["Category columns<br/>impute mode, then one-hot"]
    N --> M["Model<br/>(LogisticRegression)"]
    C --> M
    M --> P["Predictions"]

Numeric and category columns take different routes, then all columns reach one model.

ColumnTransformer: different columns, different treatment

Tables mix types. Numbers need imputing and scaling; categories need imputing and encoding. ColumnTransformer takes a list of (name, transformer, columns) triples and applies each transformer to its columns, then joins the results side by side.

The example below simulates a customer table with missing values in both a numeric and a categorical column. Notice that the raw table can be passed to the pipeline as it is, with missing values and text, and that the encoder is told to ignore categories it did not see during training (handle_unknown="ignore"), so unexpected data does not crash a deployed model.

Python A full preprocessing and model pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One call to cross_val_score(model, X, y) repeated the full recipe five times, each time learning medians, scale factors and categories from that fold's training rows only. Doing this by hand is where leaks are born. The step names ("prep", "clf", "num", "cat") also matter: you will use them to address hyperparameters.

make_pipeline and make_column_transformer are shortcuts that name the steps automatically (standardscaler, logisticregression); explicit names are clearer in tuning code. Columns can also be chosen by type with make_column_selector(dtype_include="number").

Every model has settings that change how it fits: max_depth for a tree, C for logistic regression, learning_rate for boosting. The best values depend on the data and are found by trying several and comparing cross-validated scores, not by looking at the test set.

  • Grid search (GridSearchCV) tries every combination in a grid you list. It is thorough but grows fast: 7 depths times 2 leaf sizes times 5 folds is 70 fits.
  • Randomized search (RandomizedSearchCV) samples a fixed number of combinations (n_iter) from lists or distributions. With many settings it usually finds nearly as good a combination as the full grid, for far less work, because only a few settings really matter.

Both do the same bookkeeping: for each candidate, run cross-validation on the training data, record the mean score, then refit the best candidate on all training rows (refit=True). The held-out test set stays untouched until the end.

flowchart TD
    A["Training data"] --> B["For each candidate settings"]
    B --> C["Cross-validate inside training data"]
    C --> D["Pick best mean CV score"]
    D --> E["Refit on all training data"]
    E --> F["Score once on held-out test set"]

The search happens entirely inside the training data; the test set is a final, single check.

Here is a grid search over a decision tree's depth and minimum leaf size. With return_train_score=True, the table also shows how well each candidate fits the rows it trained on, so overfitting becomes visible as a growing gap between the two scores. The metric is ROC AUC, as described in evaluating models.

Python GridSearchCV: the sweet spot of tree depth

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the mean_train_score and mean_test_score columns together. With min_samples_leaf=1, a depth-12 tree has a training AUC of 0.998 but a cross-validated AUC of only 0.615, worse than a depth-3 tree (0.725). The search picks depth 3 with at least 20 rows per leaf (0.734), and the held-out score (0.762) confirms it. In scikit-learn, mean_test_score in cv_results_ means "the held-out folds inside cross-validation", not your final test set; a confusing name worth remembering.

Two tuning habits keep you honest. First, widen the grid if the best value sits on its edge: it means a better value probably lies outside. Second, look at the spread, std_test_score, too: when many settings are within noise of each other, pick the simplest.

For models with many knobs, sample instead of enumerating. For positive scale-type settings such as a learning rate, use a log-uniform distribution, which spreads samples evenly across orders of magnitude (0.01, 0.1) rather than clustering near the top of a linear range.

Simplicity versus accuracy

The final example tunes a gradient-boosting model with RandomizedSearchCV, then compares it against a plain scaled logistic regression, the simple baseline. Both use cross-validation on the training data for the comparison, and the held-out test set is used once at the end.

Python Does the tuned boosting model beat plain logistic regression?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On this dataset, the simple model is at least as good as the tuned ensemble: cross-validated AUC of 0.773 against 0.760, and 0.826 against 0.809 on the held-out rows. The signal is mostly a plain linear one with a small interaction, and there are only 840 training rows, which gives boosting little to gain from its flexibility. This is common on small tabular datasets and a reason to always include a simple baseline.

Weigh complexity honestly. A more complex model costs more to train, explain, monitor and debug. If it is only within noise of a simpler one (compare the difference with the std of the CV scores), prefer the simpler model. If it is clearly better and the gain matters for the decision, take it.

One last caution: tuning itself overfits when you try hundreds of combinations against the same folds, because the best score of many tries is optimistic. Keep the search modest, and rely on the untouched test set (or nested validation) for the final estimate.

Practical notes

  • Fit pipelines on training data only; never call fit or fit_transform on the test set.
  • Name hyperparameters with a double underscore: clf__C addresses the C setting of the step named clf, and prep__num__impute__strategy reaches inside nested steps.
  • Control the budget. Start with a small search, check that results look sensible, then widen it.
  • Fix random_state in the splitter, the search and the model so runs are repeatable.
  • Inspect the pipeline: pipeline.named_steps and get_feature_names_out() show what the model actually receives.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main reason to put imputing, scaling and the model into one Pipeline when using cross-validation?

    Inside cross-validation, the pipeline is fitted anew for every fold, so preprocessing never sees the held-out rows. That prevents leakage and keeps scores honest.
  2. 2. What does ColumnTransformer let you do?

    You give it (name, transformer, columns) triples, for example scaling for numeric columns and one-hot encoding for categories, and it joins the outputs.
  3. 3. A grid search shows training AUC 0.998 and cross-validated AUC 0.615 for the deepest tree. What does that tell you?

    A large gap between training and cross-validated scores is the signature of overfitting. Choose settings by the cross-validated score.
  4. 4. Why is RandomizedSearchCV often preferred to GridSearchCV when a model has many hyperparameters?

    The number of grid combinations multiplies with each setting. Sampling a fixed budget of candidates usually finds a similarly good point, since only a few settings matter.
  5. 5. A tuned complex model scores only slightly better than a simple baseline, within the standard deviation of the CV scores. What is a sensible choice?

    A difference within noise is not evidence of a real gain, while complexity costs maintenance and explanation. Peeking at the test set repeatedly would also leak.

Summary

  • A Pipeline chains preprocessing and a model into one object fitted on training rows only; inside cross-validation it is re-fitted per fold, preventing leakage.
  • ColumnTransformer routes numeric and categorical columns through different steps, and handle_unknown="ignore" makes encoders safe for new categories.
  • Hyperparameters are chosen by cross-validated score on the training data, with GridSearchCV for small grids and RandomizedSearchCV for large ones.
  • Compare training and validation scores to see overfitting, and widen a grid whose best value is on the edge.
  • Always keep a simple baseline; if a complex model is not clearly better, choose the simple one.
  • Touch the held-out test set once, after all choices are made.

Next, explore data without labels in clustering and segmentation.