Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 111 ยท Model Evaluation and Tuning

Cross-Validation for Groups and Time

Standard k-fold fails when rows are related. Learn GroupKFold for repeated subjects, TimeSeriesSplit for ordered data, and how duplicates and groups cause leakage.

Ordinary k-fold cross-validation makes a quiet assumption: that every row is an independent draw from the same population, so shuffling rows into folds is harmless. Real data often breaks that assumption. The same patient has twenty hospital visits, the same customer has dozens of orders, one sensor logs a reading every minute, or tomorrow's value depends on today's. When related rows land on both sides of a split, the model can score well by recognizing who or when instead of learning something that generalizes.

The result is a validation score that looks great and a deployed model that does not. This lesson covers the two main cures. Group-aware splitting keeps all rows of one subject on the same side of the split. Time-aware splitting only ever tests on data that comes after the training data. You should already know basic cross-validation.

Rows that are not independent

Groups Many rows per patient, user or device.
Time Today depends on yesterday.
Duplicates Near-identical rows in train and test.

Shuffled k-fold leaks all three.

1 / 2

Leakage through groups

Imagine 30 patients with 10 measurements each. Every patient has a personal baseline, so measurements from the same patient look alike. Suppose the label is a property of the patient (say, a risk class), and the measurements carry no true signal about it beyond identity. A model validated with shuffled k-fold sees nine of a patient's ten rows in training, spots the familiar baseline in the held-out row and predicts correctly. That is memorizing patients, not learning medicine. A new patient would be a coin flip.

We can build exactly that situation, where the truth is known to be "no learnable signal", and watch the two splitters disagree.

Python Shuffled k-fold versus GroupKFold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Shuffled k-fold reports an accuracy of about 0.98 on a task with no real signal. GroupKFold, which tests on patients the model has never seen, drops to about 0.37, at or below the 0.5 of a coin flip. (With only 24 training patients, the forest matches each new patient to look-alike training patients whose random labels say nothing about it, so the score wanders around chance and can land below it.) The honest conclusion is "no signal". The data are the same and so is the model; only the split changed. The rule: make the validation split mimic how the model will be used. If the model will score new patients, validate on new patients.

How GroupKFold splits

GroupKFold(n_splits=k) takes a groups array with one label per row (patient id, user id, session id) and guarantees that no group appears in both the training and test part of any split. Pass groups= to cross_val_score, cross_validate and the grid searches you will meet later. Its relatives are StratifiedGroupKFold (also balances classes as well as it can), LeaveOneGroupOut, and GroupShuffleSplit for single splits.

Python Check that no group straddles a split

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each fold holds out whole users, and the shared groups count is always 0. Group sizes are not always equal, so folds can differ in size; with very few groups, use fewer splits.

Leakage through duplicates

Groups are not always labelled. Sometimes the leak is hidden in near-duplicate rows: the same product listed twice, the same image at two resolutions, a customer imported from two systems, or a row repeated by an earlier oversampling step. If the copies land on both sides of the split, the test row has a twin in the training data. The model gets credit for remembering, and the score is optimistic.

Defenses:

  • Deduplicate before splitting (drop_duplicates, or a fuzzy match on a key).
  • Build a group key from whatever identifies the entity and use GroupKFold.
  • Split first, then oversample only inside the training part; never oversample before cross-validation.
  • When you see a validation score that seems too good, suspect leakage first.

Leakage through time

For time-ordered data there is an extra rule: the model must never train on information from the future. Random folds break it in two ways. A model trained on rows from March can be tested on February, so it has "seen the future". And neighboring time points are strongly correlated, so a test row has near-twins on both sides in the training data.

TimeSeriesSplit respects the order. Each split trains on an initial stretch of time and tests on the block right after it, and the training window grows with every split. This is also called walk-forward or expanding-window validation. The data must be sorted by time first.

Python What TimeSeriesSplit produces

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the first split has the least training data, and the earliest rows are never part of any test fold. Useful options include test_size (fixed-size test blocks), max_train_size (a rolling window instead of an expanding one, useful when old data goes stale) and gap (skip rows between train and test so that features built from recent history, like a 7-day rolling mean, cannot overlap with the test period).

Shuffled folds flatter a forecaster

Let us forecast a series with a trend and a weekly cycle from simple lag features. We compare shuffled k-fold with walk-forward validation. A true forecast only ever predicts forward, so the walk-forward error is the honest one.

Python Shuffled folds versus walk-forward

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The shuffled estimate (a mean absolute error of about 0.68) is the more flattering one, because the forest has seen points from both before and after every test row. The walk-forward errors (about 1.02 on average) are the realistic ones, and they are larger, probably because a forest cannot extrapolate the upward trend into value ranges it has not seen. The naive "same as yesterday" error of about 1.69 is a baseline any forecaster should beat (see Baselines and Dummy Models); it is large here because the weekly cycle makes yesterday a poor guide. Time-series evaluation is covered in more depth in Machine Learning for Time Series.

Even without a model, preprocessing can leak time. Computing a rolling mean or a scaler over the whole series lets future values influence earlier rows. Build features with only past values, and fit scalers inside each training split (use a pipeline).

Choosing a splitter

Your data Use
Independent rows, classification StratifiedKFold
Independent rows, regression KFold(shuffle=True)
Several rows per subject / customer / device GroupKFold or StratifiedGroupKFold
Ordered in time, forecasting TimeSeriesSplit (with gap if features use history)
Both groups and time Split by time first, then keep groups intact inside each split
Near-duplicates Deduplicate or group them, then split

A simple test for any split: "At prediction time, what will the model have seen about this case?" If the answer is "nothing about this person and nothing from after this date", then your validation split must hide exactly that.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does shuffled KFold give an inflated score when each patient contributes many rows?

    Related rows on both sides of the split let the model memorize identity instead of learning a pattern that generalizes to new patients.
  2. 2. What must you pass to cross_val_score when using GroupKFold?

    GroupKFold cannot guess groups. You provide groups= with one label per row, and it keeps each group entirely in one fold.
  3. 3. What is true of every split produced by TimeSeriesSplit?

    Walk-forward splitting trains on the past and tests on the block right after it. With the default settings, the training window grows with each split.
  4. 4. A feature is a 7-day rolling mean of the target. How can you protect against overlap between train and test periods?

    gap skips rows between the end of training and the start of the test block, so the history windows used for features cannot span both parts.
  5. 5. You oversample the minority class and then run cross-validation. What is the danger?

    Duplicated rows end up on both sides of the split. Oversample only inside each training fold, for example with a resampling step in a pipeline.

Summary

  • Standard k-fold assumes independent rows; groups, duplicates and time order violate that assumption and make scores optimistic.
  • GroupKFold (and StratifiedGroupKFold, LeaveOneGroupOut) keeps each group on one side of the split; pass groups=.
  • TimeSeriesSplit trains on the past and tests on the next block; use gap and max_train_size for lagged features and stale data.
  • Deduplicate before splitting, and oversample only inside training folds.
  • Make validation mimic deployment: if the model will meet new subjects or future dates, validate on new subjects or later dates.

Next up: Learning and Validation Curves.