Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 40 ยท Model Evaluation and Selection

Learning Curves, Validation Curves and Overfitting

Diagnose underfitting and overfitting in scikit-learn with learning_curve and validation_curve, understand the bias-variance trade-off and know what to do next.

A single score tells you where you are, not what to do next. If a model scores 0.80, should you collect more data, make the model more flexible, or make it simpler? Two diagnostic tools answer that question:

  • A validation curve shows score against one hyperparameter (model complexity).
  • A learning curve shows score against the number of training samples.

Both plot a training score and a cross-validated score. The shape of the gap between them tells you whether the model suffers from high bias (underfitting) or high variance (overfitting).

Bias and variance in plain words

  • High bias: the model is too simple to capture the pattern. It does badly on training data and on new data. More data will not help; a richer model will.
  • High variance: the model is so flexible that it memorizes noise. It does great on training data and much worse on new data. More data, regularization or a simpler model will help.

Underfit, good fit, overfit

Underfit Train low, test low.
Good fit Train and test close and high.
Overfit Train high, test much lower.

The gap between the two scores is the clue.

1 / 2

Validation curve

validation_curve fits the model for each value of one hyperparameter, with cross-validation, and returns two arrays shaped (n_values, n_folds).

Python Validation curve for tree depth

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You should see the training score keep climbing toward 1.0 as the tree gets deeper, while the validation score rises, peaks, then falls. The best depth is near the peak. Left of the peak the model underfits; right of it, it overfits.

Plotting it

In a real project you plot the curve. Matplotlib works in the runner and the figure appears below the code.

Python Plot the validation curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Learning curve

learning_curve trains on growing slices of the training data (train_sizes as fractions) and scores each with cross-validation.

Python Learning curves for two models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the two tables:

  • The tree has train accuracy near 1.0 at every size, while test accuracy is far lower. That is a large gap, which is high variance. The gap closes slowly as n grows, so more data would help.
  • Logistic regression has a small gap and both curves are fairly flat. If the plateau is too low for your needs, the model is limited (bias), and adding rows will not fix it; you need better features or a more flexible model.

A learning curve that is still rising at the right edge means more data is likely to help. A flat curve means the model, not the data quantity, is the limit.

Fixing what you find

Python Regularization versus overfitting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A degree-12 polynomial is very flexible. With tiny alpha it overfits; with huge alpha it is crushed into a nearly flat line (underfit); somewhere in the middle the test R-squared peaks. The same recipe works for every model: find the complexity knob (alpha, C, max_depth, n_neighbors, min_samples_leaf) and sweep it.

A checklist

  1. Compute train and cross-validated scores together (return_train_score=True).
  2. Large gap: overfitting. Try more data, stronger regularization, fewer features, a simpler model, or an ensemble (bagging reduces variance).
  3. Both low: underfitting. Try richer features, a more flexible model, or weaker regularization.
  4. Both high and close: you are done tuning; check the result once on a held-out test set.
  5. Compare with a dummy baseline so "high" has a reference.

Continue with hyperparameter tuning to automate the sweep.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Training accuracy is 0.99 and cross-validated accuracy is 0.78. What is the most likely problem?

    A large gap between a very high training score and a lower validation score is the classic overfitting signature.
  2. 2. Which curve plots score against the number of training samples?

    learning_curve varies the training set size. validation_curve varies a hyperparameter.
  3. 3. Both training and validation scores are low and close together. What helps most?

    That pattern is high bias. The model is too limited, so more of the same data will not change much.
  4. 4. What does increasing Ridge alpha do to model flexibility?

    A larger alpha penalizes large coefficients more, so the model becomes simpler.

Summary

  • Validation curves vary a hyperparameter; learning curves vary the amount of training data.
  • A big train/validation gap means overfitting; two low scores mean underfitting.
  • Fix variance with data, regularization or simpler models; fix bias with richer models or features.
  • Plot with ValidationCurveDisplay and LearningCurveDisplay, or print the arrays.

Next: K-Means Clustering