Skip to content
Tutorials Center
My learning

Topics

Lesson 48 of 111 ยท Model Evaluation and Tuning

Nested Cross-Validation and Data Leakage

See why a tuned model's best score is optimistic, measure the bias with nested cross-validation, and learn to spot target leakage, preprocessing leakage and duplicate rows.

A model's score is only useful if it predicts how the model will do on data it has never seen. Two things quietly break that promise. The first is selection bias: when you try many settings and report the best score, the winner is partly just the luckiest candidate. The second is data leakage: information that would not exist at prediction time sneaks into training, and the score looks wonderful for the wrong reason.

Both problems produce the same painful story: a model with 95% accuracy in the notebook and 70% in production. This lesson shows each one with a small experiment you can run, then gives you a checklist for catching them early. It builds on Hyperparameter Tuning and Cross-Validation.

Two ways scores lie

Selection bias The best of many tries is lucky.
Leakage Training saw something it should not have.

Both inflate scores. Neither shows up as an error message.

1 / 2

The tuned score is optimistic

Suppose you have 25 candidate settings and score each with 5-fold cross-validation. Every one of those scores has noise in it. Pick the maximum, and you pick a candidate that is good and lucky on these particular folds. best_score_ therefore overestimates how the chosen model will perform on fresh data. The more candidates and the smaller the dataset, the larger the bias.

The cleanest way to see it is to search for signal where there is none. In the next example the labels are pure coin flips, so no model can honestly beat 50% accuracy.

Nested cross-validation

Nested cross-validation repeats the entire tuning procedure inside each fold of an outer loop:

  • The outer loop splits the data into training and test folds. Its test folds are used only for scoring.
  • The inner loop runs the hyperparameter search on each outer training fold, never touching the outer test fold.

The outer scores then measure the quality of the whole procedure ("tune with this grid, then fit"), not of one lucky model. In scikit-learn you get it by passing a search object to cross_val_score, since a search object is just an estimator.

flowchart TB
    D["all data"] --> O1["outer fold 1: train | test"]
    D --> O2["outer fold 2: train | test"]
    O1 --> I1["inner CV on the outer training part<br/>pick best settings"]
    I1 --> F1["refit, then score on outer test fold"]
    O2 --> I2["inner CV on the outer training part<br/>pick best settings"]
    I2 --> F2["refit, then score on outer test fold"]
    F1 --> M["average of outer scores"]
    F2 --> M

The outer test folds never take part in choosing hyperparameters, so their average is an honest estimate.

Python Tuning on noise: best_score_ versus nested CV

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On every dataset the nested number is lower than best_score_ or equal to it, and across the six runs the average best_score_ sits visibly above 0.5 while the nested average hovers around the truth. A single nested estimate is noisy too (it is a mean over only 80 coin flips), but unlike best_score_ it is not biased upward.

Nested CV costs a lot: outer folds times candidates times inner folds. You do not need it for every project. Use it when you compare algorithms or tuning procedures and need an honest comparison, when the dataset is small, or when you must report a trustworthy number. When data is plentiful, a held-out test set that you use exactly once does the same job much more cheaply.

Preprocessing leakage

Any step that learns something from the data must be fitted on the training portion only. That includes scalers, imputers, encoders, PCA, and especially feature selection. If you fit these on all the data and then cross-validate, each validation fold has already influenced the transformation applied to it.

Feature selection is the most dramatic case. With 1,000 noise features and 100 rows, some features will correlate with the random labels by chance. If you pick the 20 best-looking features using all the labels, and then cross-validate, the validation folds are being scored on features chosen because they matched those very rows.

Python Selecting features before cross-validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The leaky version reports around 85% accuracy on labels that carry no information at all. The pipeline version lands near coin-flip level, which is the truth. The rule is simple: if a step looks at y or at statistics of the data, it belongs inside the pipeline. The Pipelines lesson makes this routine.

Target leakage

Target leakage means a feature contains information that is only available because the outcome happened. The model learns to read the answer instead of predicting it. Classic examples:

  • A column such as refund_issued in a churn dataset: refunds happen after customers leave.
  • days_in_icu when predicting whether a patient will be seriously ill.
  • A "late fee" column in a loan-default model.
  • A feature computed over a window that includes the future, such as "average sales this month" in a model that predicts a day within that month.

The test is a question you ask about every column: will I know this value at the moment I need the prediction? If not, drop it.

Python A leaky column that looks like a superpower

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A near-perfect score on a messy real-world problem is a red flag, not a victory. The feature importances give it away: one column dominates. When a result looks too good, examine the top features and ask whether each exists at prediction time.

Duplicates and groups

A subtler leak happens when the same entity appears in both training and validation rows: several photos of one patient, repeated measurements of one machine, the same customer on multiple dates, or accidental duplicate rows. A random split puts near-copies on both sides, and the model scores well simply by remembering. The fix is a split that keeps each group together, covered in Cross-Validation for Groups and Time. Always check df.duplicated().sum() early in a project.

A leakage checklist

Before you trust a score

Timing Is every feature known at prediction time?
Pipeline Do all fitted steps live inside it?
Duplicates Same entity in train and test?
Time order Does validation come after training?
1 / 2

Add these habits to your routine:

  1. Split first. Do not compute any statistic on the full dataset before the split, not even a mean for imputation.
  2. Keep the test set sealed. Look at it once, for the final number. Every peek is a tiny round of tuning.
  3. Use a pipeline for every learned step, and give the pipeline to cross_val_score or the search object, never pre-transformed data.
  4. Mind time. For anything ordered in time, train on the past and validate on the future.
  5. Read your feature list with a skeptical eye, column by column, with a domain expert if you can.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a grid search's best_score_ an optimistic estimate of future performance?

    Every candidate's score contains noise. Taking the maximum over many candidates selects for positive noise, so the winning score is biased upward.
  2. 2. In nested cross-validation, which data does the inner loop use?

    The inner loop tunes on the outer training data only. The outer test fold is never involved in choosing hyperparameters, which is what makes the outer score honest.
  3. 3. You select the 20 best features using the whole dataset, then run 5-fold cross-validation on those 20 columns. What is wrong?

    The features were chosen because they matched the labels of all rows, including those later used for validation. Putting the selector in a pipeline refits it inside every training fold.
  4. 4. Which column is most likely target leakage in a model that predicts, at signup time, whether a customer will cancel within 90 days?

    Tickets from the 90 days after signup are not known when the prediction is needed, and they are tied to the outcome. The other columns exist at signup time.

Summary

  • The best score among many tuned candidates is biased upward; nested cross-validation measures the tuning procedure honestly by keeping the outer test folds away from the search.
  • Nested CV is expensive: use it for small data, for comparing algorithms, and for numbers you must report.
  • Preprocessing and feature selection that learn from the data must be fitted inside each training fold, which means inside a pipeline.
  • Target leakage comes from features known only after the outcome; ask of every column whether it exists at prediction time.
  • Duplicates and repeated entities leak through random splits; keep groups together.
  • A suspiciously great score is a reason to investigate, and the checklist above is a good start.

Next: Probability Calibration and Uncertainty.