Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 111 ยท Getting Started

Train, Validation and Test Sets

Learn why we split data into train, validation and test sets, how to use train_test_split with random_state and stratify, and how data leakage ruins results.

A model that scores perfectly on the data it studied has told you nothing. It may have simply memorized the answers. What you actually care about is how it performs on data it has not seen, because that is the situation in real use. To measure that honestly, you hold back part of your data and treat it as a stand-in for the future.

This lesson explains the three-way split into training, validation and test sets, shows how train_test_split does it, and introduces data leakage, the family of mistakes that make results look better than reality.

Why a test set exists

Imagine a student who gets the exam questions in advance. Their score would not reflect their knowledge. A model that is evaluated on its own training data is that student. The training set is what the model learns from. The test set is the exam it has never seen.

Two rules follow, and they matter more than any algorithm choice.

  1. Never fit anything (the model, a scaler, an encoder) using the test set.
  2. Look at the test set as rarely as possible, ideally once, at the very end.

train_test_split

train_test_split shuffles your data and cuts it into two pieces. It takes any number of arrays of the same length and returns them split consistently.

Python A basic split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

test_size=0.3 puts 30 percent of the rows in the test set (you may also pass an integer count). random_state fixes the shuffle so the split is repeatable. Without it, every run gives different pieces, and results change from run to run. A common test share is 10 to 30 percent: more training data helps the model, more test data makes the score more trustworthy.

stratify keeps class shares

If one class is rare, a random split may leave the test set with too few examples of it, or none at all. stratify=y forces each piece to keep roughly the same class proportions as the whole.

Python Random split versus stratified split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With this seed the plain random split leaves the test set with no positives at all, so nothing could be measured about the rare class. The stratified split keeps exactly 10 percent positives (2 of 20) in the test set. Always stratify for classification, especially when classes are imbalanced (see imbalanced classes). It does not apply to regression targets.

The validation set

You rarely get a model right on the first try. You compare algorithms, try settings, add features, and each decision is made by looking at a score. If you make those decisions using the test set, you slowly fit the test set by choice, and its score becomes optimistic. A third piece fixes this.

  • Training set: fit the model parameters.
  • Validation set: compare models and tune settings.
  • Test set: report the final result, once.
flowchart LR
    D["All data"] --> T["Training set<br/>fit models"]
    D --> V["Validation set<br/>choose and tune"]
    D --> S["Test set<br/>final report only"]

Each part of the data has one job. The test set is the only one that stays untouched until the end.

You can create the three parts by splitting twice. First hold out the test set, then split the remainder into train and validation.

Python A three-way split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After choosing the setting with the best validation score, you may refit on training plus validation data and score on the test set a single time. When data is scarce, a validation set wastes precious examples, and cross-validation rotates the validation role instead.

When training and test data come from different places

Sometimes you cannot get much data from the place where the model will be used. You might have a million clean photos from the internet for training, but only a few thousand blurry photos taken with the phone camera your app will use. The rule is: the validation and test sets must come from the distribution you care about (the phone photos), even if the training set is mostly from another.

That creates a diagnostic problem. If the score on the validation set is bad, is it overfitting or data mismatch? The fix is a fourth slice, the train-dev set: a held-out piece of the training distribution that the model does not train on. Compare three scores:

Training error Train-dev error Dev (target) error Diagnosis
high Bias (underfitting): a bigger model, more training
low high Variance (overfitting): more data, regularization
low low high Data mismatch: make training data look more like the target, add some target data
low low low Done, then check the test set once

The example uses two sources of 4-feature data with different feature statistics but the same labelling rule: a big "clean" source for training and a small "target" source for validation and testing.

Python Training, train-dev and dev scores diagnose data mismatch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first line shows the pattern of data mismatch: the training score (0.969) and the train-dev score (0.943) are both high, so the model has learned the clean data and does not overfit it, while the dev score is clearly lower (0.871), because the target data lives partly outside the region the model has seen (a random forest cannot extrapolate). Adding even a small amount of target-domain data to the training set helps: on the same remaining dev rows the accuracy rises from 0.877 to 0.955, with no loss on the clean train-dev data. Practical rules that go with this: shuffle and split within each source, never put the same item (or near-duplicates of it) in both the training and the evaluation slices, keep the test set purely from the target distribution, and collect validation and test data from real users whenever you can.

Data leakage: a first look

Leakage means information from outside the training data sneaks into the model, so the score looks better than real performance will be. It is subtle and extremely common. Here are the first forms to watch for.

  • Preprocessing leakage. Computing a mean or scaling range on the whole dataset before splitting lets test statistics influence training. Fit transformers on the training set only, then apply them to the test set.
  • Target leakage. A feature that is a consequence of the label, or is only known afterwards, such as a "refund issued" column when predicting fraud. The model reads the answer from the feature.
  • Duplicate or related rows. The same customer appearing in both train and test makes the test easy.
Python Preprocessing leakage in action

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two means differ only slightly here, but the principle matters: with real data and heavier steps (feature selection, imputation, target encoding) the leak can make a worthless model look excellent. Pipelines make the correct order automatic, and nested cross-validation and leakage revisits the topic in depth.

Shuffling and time order

train_test_split shuffles by default, which is right when rows are independent. It is wrong when order carries meaning. For a time series, you must train on the past and test on the future; a shuffled split lets the model peek at the future. Use shuffle=False for a simple chronological cut, or a proper time-aware scheme from cross-validation for groups and time.

Python Chronological split with shuffle=False

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same warning applies when rows are grouped, for example several records per patient. If the same patient lands in both parts, the model recognizes the person and not the pattern. Split by group in that case.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the main purpose of the test set?

    The test set stands in for future data, so its score estimates real-world performance when it is used only once.
  2. 2. When should you use stratify=y in train_test_split?

    Stratification preserves the share of each class in the train and test sets, which matters most for rare classes.
  3. 3. You compute the mean of a feature on the full dataset and then split. What is this called?

    Test rows influenced the statistic, so information from the test set leaked into training.
  4. 4. For forecasting next month's sales, which split is appropriate?

    The model must never see the future during training. A chronological split respects that.
  5. 5. Why does random_state matter in train_test_split?

    A fixed seed produces the same shuffle each run, so results can be repeated and compared.

Summary

  • Training data teaches the model, validation data guides decisions, and test data gives one final honest score.
  • train_test_split(X, y, test_size=..., random_state=..., stratify=y) splits arrays consistently and reproducibly.
  • Stratify classification splits so rare classes appear in every part.
  • Leakage (preprocessing, target and duplicate leakage) makes scores look better than reality; fit all transformers on training data only.
  • Use shuffle=False or time-aware splits for ordered data, and split by group when rows belong together.

Next: Baselines and Dummy Models