Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 111 ยท Model Evaluation and Tuning

Cross-Validation

Estimate model performance reliably with k-fold, stratified and repeated cross-validation. Use cross_val_score and cross_validate, and see how much a single split can mislead.

A single train/test split gives you one score, and that score depends on which rows happened to land in the test set. Split the data differently and you get a different number, sometimes by several points. With small datasets the swing can be large enough to change which model looks best. Cross-validation fixes this by repeating the split several times and averaging, so every row is used for testing exactly once and for training the rest of the time.

The most common form is k-fold cross-validation. You cut the data into k equal parts (folds). For each fold in turn, you train on the other k - 1 folds and score on the held-out one. The k scores give you both an estimate of performance (their mean) and a feeling for its reliability (their spread). This lesson covers the variants you will actually use. It builds on Train, Validation and Test Sets and the idea of overfitting.

How k-fold works

Split Cut the rows into k folds.
Rotate Hold out each fold once.
Average The mean is the estimate, the spread is the doubt.
1 / 2

How much can one split mislead?

Before using cross-validation, let us feel the problem. We score the same model on 200 different random train/test splits of a small dataset. If one split were reliable, all 200 scores would be about the same.

Python The spread of single-split scores

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The best and worst splits differ by many points, even though the model, the data and the split size never changed. If you had reported only the lucky split, you would have overstated your model. Cross-validation reduces that luck by averaging over several splits.

cross_val_score

cross_val_score(estimator, X, y, cv=5) does the whole loop for you: it clones the estimator, fits it on each training part and returns an array of fold scores. By default it uses accuracy for classifiers and Rยฒ for regressors; pass scoring= for anything else (for example "f1", "roc_auc" or "neg_mean_absolute_error").

Python Five-fold cross-validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Passing an integer for cv is a shortcut. For classifiers, scikit-learn quietly uses stratified folds without shuffling. If your rows might be sorted (by class, by date, by customer), or you simply want control, build the splitter yourself, as with KFold(shuffle=True, random_state=...) above, and pass it as cv. The splitter also lets you reuse identical folds for several models, which makes comparisons fair.

Report cross-validation results as mean plus or minus standard deviation. Remember the fold scores are not independent (they share training data), so the standard deviation understates the true uncertainty a little. Treat it as a rough guide, and be wary of declaring a winner when two models differ by less than the spread.

Stratified folds for classification

With imbalanced classes, plain KFold can produce folds that have almost no positive examples, or none at all. StratifiedKFold makes each fold keep (roughly) the class proportions of the full dataset. Use it by default for classification.

Python Class balance in each fold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because we sorted the rows, unshuffled KFold gives four test folds with no positives at all and one with 50%: the scores for that data would be meaningless. Shuffling helps on average but still leaves uneven folds (5% to 18% here). Stratification pins every fold at 10%, matching the data. For regression, ordinary KFold is fine (there is no class to balance).

cross_validate: several metrics and training scores

cross_validate returns a dictionary instead of one array. It can compute several metrics at once, report fit and score times, and also return training scores with return_train_score=True. Comparing training and validation scores is the quickest way to spot overfitting.

Python Several metrics and train scores at once

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unlimited tree scores 1.000 on its training folds for every metric, but noticeably less on the validation folds: the gap is the signature of overfitting. The forest also fits its training folds nearly perfectly, yet it loses much less on the held-out folds, and its validation scores are higher. Notice that accuracy looks comfortable because of the 80/20 class split, while f1 is lower: always pick a metric that matches the goal (see Classification Metrics).

Repeated and leave-one-out cross-validation

Repeated k-fold runs k-fold several times with different shuffles (RepeatedKFold, RepeatedStratifiedKFold). It costs more, but it averages out the luck of any particular fold assignment, which is valuable on small datasets.

Leave-one-out (LeaveOneOut) is k-fold taken to the extreme: k equals the number of rows. Each row is held out once. It uses almost all the data for training each time, but it needs n fits, and each fold's score is a single 0-or-1 outcome, which makes the overall estimate high in variance. Reserve it for tiny datasets with cheap models.

How should you choose k? Small k (like 2 or 3) trains on less data, so the estimate is pessimistic. Large k trains on almost everything, so the estimate is less biased, but it costs more time. k = 5 or 10 is the usual compromise.

Python Plain, repeated and leave-one-out

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The means are all within a few points of each other (0.77 to 0.81), but look at the last line: even with 5-fold cross-validation, simply changing the shuffle seed moves the estimate between 0.76 and 0.81 on 100 rows. Repeating the procedure (the 5-fold x 10 repeats line, with 50 fits) averages that luck away and gives a value in the middle of that range. Leave-one-out's std is the spread of 0/1 outcomes, so do not compare it directly with the others.

Practical rules

  • Use StratifiedKFold for classification and KFold(shuffle=True) for regression, unless your data have groups or a time order (see Cross-Validation for Groups and Time).
  • Always put preprocessing inside a pipeline so that scaling, imputing and selection are refit on each training fold. Otherwise information from the held-out fold leaks in.
  • Fix random_state on the splitter so the experiment is reproducible, and reuse the same splitter when comparing models.
  • Keep a final untouched test set for the last check. Cross-validation guides your choices; the test set confirms them.
  • Cross-validation costs k times as much as one fit. In the browser, keep the models and data small.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In 5-fold cross-validation, how many times is each row used for testing?

    Each row sits in exactly one fold, so it is held out once and used for training in the other four rounds.
  2. 2. Why should you prefer StratifiedKFold for an imbalanced classification problem?

    With rare positives, plain folds can contain too few or none. Stratification keeps the class ratio similar in each fold, so each score is meaningful.
  3. 3. What does return_train_score=True in cross_validate help you detect?

    A near-perfect training score with a much lower validation score means the model is memorizing the training folds.
  4. 4. You scale the whole dataset with StandardScaler and then run cross_val_score on the scaled data. What is the problem?

    The mean and standard deviation were computed using rows that later act as test data. Put the scaler in a Pipeline so it is fit on training folds only.
  5. 5. Leave-one-out cross-validation on 1,000 rows requires how many model fits?

    Each row is held out once, so there is one fit per row. That is why leave-one-out is only practical for small datasets and cheap models.

Summary

  • One train/test split gives a noisy estimate; cross-validation averages over several splits and uses every row for testing once.
  • cross_val_score returns one score per fold; report mean and standard deviation. cross_validate adds several metrics, timings and training scores.
  • Use StratifiedKFold for classification and KFold(shuffle=True, random_state=...) for regression; reuse the same splitter when comparing models.
  • Repeated k-fold steadies the estimate on small data; leave-one-out is only for tiny datasets.
  • Put preprocessing in a pipeline so each fold refits it, and keep a separate final test set.

Next up: Cross-Validation for Groups and Time.