Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 40 ยท Model Evaluation and Selection

Train/Test Split and Cross-Validation

Estimate how a scikit-learn model will behave on new data with hold-out splits, k-fold, stratified and grouped cross-validation, cross_val_score and cross_validate.

A model that scores 100% on the data it was trained on has told you almost nothing. It may have memorized the examples. What you really want to know is how it will behave on data it has never seen. Every evaluation technique in this chapter is a way of faking that situation honestly: hide some data, train on the rest, and measure on the hidden part.

The simplest version is a single hold-out split. The more reliable version is cross-validation, which repeats the hold-out idea several times and averages the results. In this lesson you will use both, learn which splitter fits which kind of data, and avoid the most common mistake: letting information leak from the test data into training.

The hold-out split

train_test_split shuffles the rows and cuts them into two parts. Fix random_state so the split is the same every time you run the code.

Python A hold-out split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

stratify=y keeps the share of each class the same in both parts, which matters when one class is rare. The weakness of a hold-out split is that the score depends on which rows landed in the test set. Change random_state and the number moves. With small datasets that wobble can be several percentage points.

k-fold cross-validation

In k-fold cross-validation the data is cut into k equal folds. Each fold takes one turn as the test set while the other k-1 folds train the model. You get k scores and can look at their mean and spread.

5-fold cross-validation

Split Cut the data into 5 folds.
Rotate Each fold is the test set once.
Average Mean score plus its spread.

Every row is tested exactly once and trained on k-1 times.

1 / 2

cross_val_score does the whole loop in one call. You pass the unfitted estimator (or pipeline), the data and cv.

Python cross_val_score

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When cv is an integer and the estimator is a classifier, scikit-learn automatically uses stratified folds. For regressors it uses plain KFold.

Put preprocessing inside the pipeline you cross-validate, as above. If you scale the whole dataset first and cross-validate afterwards, the scaler has already seen the test folds' statistics. That is data leakage, and it makes scores look slightly better than reality. See pipelines for the full story.

Choosing a splitter

The integer shortcut is fine most of the time, but you can pass any splitter object for control.

Splitter Use it when
KFold(n_splits, shuffle=True) Regression, or balanced classes
StratifiedKFold Classification, especially with rare classes
RepeatedStratifiedKFold You want a steadier estimate (repeat with new shuffles)
GroupKFold Rows come in groups (patients, users) that must not be split
TimeSeriesSplit Data has a time order and you must never train on the future
ShuffleSplit Random train/test splits of any size, repeated
LeaveOneOut Tiny datasets; one row per fold (slow and high-variance)

GroupKFold is easy to forget and important. If one patient has ten rows and those rows end up in both training and test folds, the model can recognize the patient and score unrealistically well. Passing groups= keeps every group on one side.

Python Splitters side by side

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how stratified folds keep the rare-class share almost constant, plain k-fold lets it drift, GroupKFold never shares a group, and TimeSeriesSplit only tests on rows that come after its training rows.

Several metrics at once: cross_validate

cross_validate returns a dictionary and can compute many metrics, training scores and timings in one pass.

Python cross_validate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare each train_* mean with its test_* mean. A large gap means the model fits the training data much better than new data, which is overfitting (see learning and validation curves).

Common mistakes

  • Tuning on the test set. If you try ten models and keep the one with the best test score, the test score is no longer an honest estimate. Keep a final test set untouched, and use cross-validation on the training part for choices (see hyperparameter tuning).
  • Shuffling time series. Random folds let the model peek at the future.
  • Ignoring the spread. A mean of 0.91 with a standard deviation of 0.05 is a different story from 0.91 with 0.005.
  • Cross-validating a fitted model. Pass an unfitted estimator; cross_val_score clones and refits it for every fold.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a single train/test split sometimes unreliable?

    With a small dataset, a different random split can change the score noticeably. Cross-validation averages over several splits.
  2. 2. What does stratify=y do in train_test_split?

    Stratification preserves the share of each class in the train and test sets, which matters for imbalanced labels.
  3. 3. Which splitter should you use when each patient contributes several rows?

    GroupKFold keeps all rows of a group in the same fold so the model cannot recognize a patient it trained on.
  4. 4. Why should scaling happen inside the pipeline you cross-validate?

    Fitting the scaler on all data lets test-fold statistics influence training. Inside a pipeline it is refitted per fold.

Summary

  • A hold-out split is quick but noisy; cross-validation averages several splits and shows the spread.
  • cross_val_score gives one metric, cross_validate gives several plus train scores and timings.
  • Classifiers use stratified folds by default when cv is an integer.
  • Pick the splitter to match your data: GroupKFold for groups, TimeSeriesSplit for ordered data.
  • Keep preprocessing inside a pipeline and keep one test set untouched for the final check.

Next: Evaluation Metrics