Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 40 ยท Getting Started

Train, Validation and Test Sets

Split data correctly with train_test_split: choose test sizes, stratify classes, fix the seed, create a validation set and avoid the data leakage that inflates scores.

A model's job is to work on data it has never seen. To estimate that honestly, you split your dataset before modelling. train_test_split is the function you will call at the start of almost every project. This lesson covers its options, the three-way split, and the mistake that ruins more projects than any other: data leakage.

The basic split

Python train_test_split basics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

  • test_size can be a fraction (0.2 = 20%) or a count (test_size=30). Common choices are 0.2 to 0.3.
  • random_state fixes the shuffle so results are reproducible.
  • The function shuffles by default, which is almost always what you want, except for time series (see below).

Stratified splits

With imbalanced classes, a random split can leave the test set with too few examples of a rare class. stratify=y keeps the class proportions the same in both parts.

Python Why stratify matters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The stratified split gives 10% positives in both parts. Use it for every classification problem unless you have a reason not to.

A validation set

You will try many settings and models. If you pick the winner by its test score, the test set quietly becomes part of training, and its score is optimistic. The fix is three sets:

flowchart LR
    D["All data"] --> TR["Train 60%<br/>fit the model"]
    D --> VA["Validation 20%<br/>compare and tune"]
    D --> TE["Test 20%<br/>one final check"]

Tune on validation, report on test, and only look at the test set once.

Python Three-way split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Cross-validation replaces the fixed validation set with several rotating ones and is usually the better tool when data is limited.

Data leakage

Leakage means information from outside the training set sneaks into the model. The score then looks great and the real-world performance does not. The classic case is scaling before splitting.

Python Leaky versus correct scaling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The rule: split first; fit every preprocessing step on the training data only. Pipelines enforce this automatically.

Other leaks to avoid: a feature computed from the target (for example "refund issued" when predicting "will the customer churn"), duplicate rows present in both train and test, and tuning hyperparameters on the test set.

When a random split is wrong

  • Time series: shuffling lets the model peek at the future. Split by date, with the oldest rows in training and the newest in testing. Pass shuffle=False for a simple version.
  • Grouped data: if the same patient or customer appears in many rows, keep each group wholly in one side (use GroupShuffleSplit).
Python Time-ordered split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does stratify=y do?

    Stratification preserves the share of each class in both parts.
  2. 2. Why use a separate validation set?

    Tuning on the test set makes its score optimistic, so you tune on validation data and test once at the end.
  3. 3. Which order is correct?

    Fitting a scaler before splitting leaks test statistics into training.
  4. 4. How should you split daily sales data used to forecast the future?

    A random shuffle lets the model learn from the future, which is impossible in real use.

Summary

  • train_test_split(X, y, test_size=..., random_state=...) creates reproducible splits.
  • Use stratify=y for classification.
  • Add a validation set (or use cross-validation) so the test set stays untouched until the end.
  • Split first; fit preprocessing on training data only to avoid leakage.
  • Time-ordered and grouped data need special splits.

Next: Scaling and Normalization