Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 76 ยท Modelling

Train/Test Splits and Cross-Validation

Learn holdout splits, k-fold and stratified cross-validation, time-series splits and data leakage, with live scikit-learn examples that show how leaks fool you.

A model that is judged on the data it learned from will always look good. A tree can memorize its training rows and score 100%, and still fail the first day it meets new customers. So the central rule of predictive modelling is simple: evaluate on data the model has not seen. Everything in this lesson is a way of following that rule honestly, and of noticing the many quiet ways it gets broken.

You will learn the holdout split, why a single split can mislead, k-fold and stratified cross-validation, time-aware splits for data that has an order, and data leakage, the most common reason a model looks wonderful in the notebook and fails in production. We use scikit-learn throughout; the previous lessons on trees showed why this matters.

The holdout split

The simplest scheme splits the rows once: most for training, the rest for testing. You fit on the training part and score on the test part, once, at the end. Typical test sizes are 20% to 30%. Always pass a random_state, so your split is repeatable.

A single split is easy, but it is also a single roll of the dice. A different random split gives a different score. The next example fits the same simple model on eight different random splits of one dataset:

Python The same model, eight different test scores

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model and the data are identical, yet test error ranges from 1.68 to 2.73 purely because of which rows landed in the test set. If you ran only one split, you might celebrate a lucky 1.68 or panic over an unlucky 2.73. With small data, a single split is too noisy to compare models reliably.

Cross-validation

k-fold cross-validation uses all of the data for both jobs, in turn. It cuts the rows into k equal folds (often 5 or 10). For each fold, it trains on the other k - 1 folds and scores on the held-out one. The k scores are then averaged, and their spread tells you how much to trust the result.

flowchart LR
    subgraph R1["Round 1"]
        A1["Test"] --- B1["Train"] --- C1["Train"] --- D1["Train"] --- E1["Train"]
    end
    subgraph R2["Round 2"]
        A2["Train"] --- B2["Test"] --- C2["Train"] --- D2["Train"] --- E2["Train"]
    end
    subgraph R3["Round 3 to 5"]
        A3["Each fold is the test fold once"]
    end
    R1 --> R2 --> R3

Five-fold cross-validation: every row is tested exactly once, and trained on in the other four rounds.

In scikit-learn, cross_val_score(model, X, y, cv=5) does this in one line. For regression, scoring="neg_mean_absolute_error" returns negated errors, because scikit-learn assumes bigger is better, so flip the sign when you report.

Stratified folds

If a class is rare, plain folds can end up with very different class mixes, or none of the positives at all. Stratified k-fold keeps each class in about the same proportion in every fold. Scikit-learn's classifiers use stratified splitting by default in cross_val_score when you pass an integer cv, but you should know what it does:

Python Positives per test fold: plain vs stratified vs shuffled

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Plain KFold on sorted data puts all 10 positives into the last fold, so four folds have nothing to test on. Shuffling helps but is uneven. Stratification gives every fold exactly two positives. Shuffle rows whenever the file's order carries no meaning, and stratify when classes are imbalanced. For grouped data (many rows per customer, patient or device), use GroupKFold, so rows from one entity never sit on both sides of a split.

Data leakage

Data leakage means information from outside the training data, usually from the test data or from the future, sneaks into the model. The model then looks excellent in validation and fails in the real world. Typical sources:

  • Preprocessing on all the data. Scaling, imputing, encoding or feature selection fitted on the full dataset lets the test rows influence the training step.
  • Target leakage. A feature that is only known after the outcome, such as "refund issued" when predicting cancellation.
  • Duplicates and near-duplicates across the split (the same customer twice).
  • Random splits of time-ordered data, which train on the future to predict the past.

The leak with the biggest bang is feature selection. Below we create 2,000 pure noise features with random 0/1 labels. There is nothing to learn, so an honest accuracy is about 50%. In the leaky version, we pick the 20 features that correlate best with the labels using all rows, then cross-validate. In the honest version, the selection is part of a pipeline, so it is redone inside every training fold.

Python Leakage demo: a model that learns nothing scores 78%

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The leaky procedure reports 78% accuracy on labels that are pure coin flips. The 20 chosen columns look predictive because they were chosen by looking at the labels of the very rows later used for testing. The honest pipeline reports about 49%, the chance level, which is the truth. Nothing about the model was cheated by greed or bad faith; the mistake was only one line out of order. The rule that prevents it: every step that learns something from data (a scaler, an imputer, an encoder, a selector, the model) must be fitted on the training part only. Wrapping the steps in a Pipeline enforces this automatically, as shown in pipelines and tuning.

Time-aware splits

For forecasting, and for any data where the future should not influence the past (sales, sensor readings, user history), the split must respect time: train on earlier rows, test on later rows. TimeSeriesSplit makes growing training windows, each followed by a test window:

flowchart LR
    A["Fold 1<br/>train: t0 to t4<br/>test: t5 to t9"] --> B["Fold 2<br/>train: t0 to t9<br/>test: t10 to t14"]
    B --> C["Fold 3<br/>train: t0 to t14<br/>test: t15 to t19"]

Each test window comes after its training window; nothing from the future is ever used for training.

A shuffled split on trending data looks excellent because each test point has near neighbours in time that the model has already seen, so it can simply interpolate. The forward-only split asks the harder, real question: can you predict what has not happened yet?

Python Shuffled folds flatter a trending series

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The shuffled estimate says the error is about 0.55; the forward-only estimate says 3.1. The forest cannot extrapolate a trend beyond the range it trained on, and only the honest split reveals that. Trust the number that mimics how the model will actually be used.

Validation set, test set and nested validation

When you also choose things (which model, which settings, which threshold), you need a third piece of data. Use the training part for fitting, a validation set (or cross-validation inside the training part) for choosing, and a final test set touched once, at the very end. If you peek at the test set repeatedly while tweaking, it quietly becomes part of training, and its score becomes optimistic.

The idea has a name when the choosing itself uses cross-validation: nested cross-validation. An inner loop picks the best settings; an outer loop estimates how well the whole procedure, including the picking, performs on fresh rows. Here is the shape in code (read-only, because it is slower than a browser demo should be):

Python
from sklearn.model_selection import GridSearchCV, cross_val_score, KFold
from sklearn.svm import SVC

search = GridSearchCV(SVC(), {"C": [0.1, 1, 10]}, cv=KFold(5, shuffle=True, random_state=0))
nested_scores = cross_val_score(search, X, y, cv=KFold(5, shuffle=True, random_state=1))

The best score inside search is biased upward, because it is the maximum over many tries; nested_scores is not. You rarely need nested validation for a quick project, but it is the right tool when data is small and many settings are compared.

A short checklist

  1. Split first, before looking for patterns or fitting any transformer.
  2. Put all learning steps in a pipeline so they are re-fitted inside each fold.
  3. Stratify for rare classes; group for repeated entities; split by time for time-ordered data.
  4. Report the mean and spread across folds, not one lucky number.
  5. Touch the final test set once.
  6. If a score looks too good, suspect leakage before celebrating.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why is a single train/test split often an unreliable estimate on small datasets?

    With few rows, different random splits give noticeably different scores. Cross-validation averages over several splits and shows the spread.
  2. 2. You scale all features using the whole dataset and then split into train and test. What is wrong?

    The scaler learned means and spreads that include the test rows. Fit it on the training part only, then apply it to the test part.
  3. 3. Which cross-validation scheme is the best choice for predicting next month's sales from past sales?

    The model will predict the future from the past, so validation must do the same. Shuffling lets the model train on rows from after the test period.
  4. 4. Cross-validation accuracy is 78% on data where the labels are random coin flips. What is the most likely cause?

    On random labels, honest accuracy is near 50%. Anything much higher means information leaked from the held-out rows into the training procedure.
  5. 5. When is StratifiedKFold especially useful?

    Stratification keeps class proportions the same in every fold. For grouped rows use GroupKFold, and for time series use TimeSeriesSplit.

Summary

  • Always evaluate on data the model did not use for fitting or for choosing settings.
  • One holdout split is noisy; k-fold cross-validation averages several and shows the spread.
  • Stratify folds for imbalanced classes, group by entity for repeated rows, and split by time for ordered data.
  • Leakage (preprocessing on all data, target leakage, duplicates, training on the future) produces scores that are too good to be true.
  • Put every learning step in a Pipeline so it is fitted inside each training fold only.
  • Keep a final test set untouched until the end; use nested validation when selecting among many options on small data.

Next, decide what "good" means for a model in evaluating models with the right metrics.