Skip to content
Tutorials Center
My learning

Topics

Lesson 46 of 111 ยท Model Evaluation and Tuning

Learning and Validation Curves

Use learning_curve and validation_curve to see whether you need more data or a different model. Learn to read the curves, spot gaps and plateaus, and plot them.

When a model underperforms, you face a fork. Should you spend weeks collecting more data, or switch to a more flexible model, or constrain the one you have? Guessing is expensive. Two plots answer the question cheaply by running cross-validation repeatedly while changing one thing.

A learning curve varies the amount of training data and shows how training and validation scores respond. It tells you whether more data would help. A validation curve varies one hyperparameter (such as tree depth or gamma) and shows how the scores respond. It tells you where the model goes from underfitting to overfitting. Both build on cross-validation and on the diagnosis of overfitting and underfitting.

Two curves, two questions

Learning curve Score against the number of training rows.
Validation curve Score against one hyperparameter.

Each point is a cross-validated mean, so each curve costs many fits.

1 / 2

Learning curves

learning_curve(estimator, X, y, train_sizes=..., cv=...) trains the model on growing slices of the training folds. For every size it returns the training scores and the validation scores for each fold. You plot the mean of each, usually with a band for the spread.

Here is a typical experiment with two models on the same data: a logistic regression (simple, will underfit curved boundaries) and an unlimited decision tree (flexible, will overfit).

Python Learning curves for a simple and a flexible model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read each panel from left to right.

  • Logistic regression has training and validation curves that sit almost on top of each other and plateau near 0.85 from the first few dozen rows onward. A straight boundary cannot follow the curved moons, and the gap is tiny. This is high bias: adding rows will not lift either curve. You need a richer model or better features.
  • The unlimited tree has a training curve glued to 1.0 (it memorizes whatever it gets) and a validation curve 0.13 to 0.2 below it. That large gap is high variance. The validation curve climbs from about 0.80 to the mid 0.80s as data grows and then bounces between 0.84 and 0.87, so extra rows help only a little here. The noise level of the data (we set noise=0.3) limits what any model can reach, and a constrained tree or an ensemble would shrink the gap more than data would.

The training curve is always at or above the validation curve in this kind of plot, and with a very small training set, a flexible model can reach a perfect training score trivially. Drawing the band across folds (as the orange area does) shows how much the validation score varies.

What to conclude from a learning curve

What you see Meaning Next step
Both curves low, close together, flat High bias More capacity, better features, less regularization
Big gap, validation curve still rising High variance, data-limited Collect more data, regularize, or average models
Big gap, validation curve flat High variance, data will not fix it Simplify the model or reduce features
Both curves high and converged Good fit Stop adding data; tune details or ship
Validation curve noisy, jumping around Not enough rows or folds More folds or repeats before trusting any conclusion

Validation curves

validation_curve(estimator, X, y, param_name=..., param_range=..., cv=...) returns training and validation scores for each value of one hyperparameter. The parameter name works with pipelines too, using the step prefix ("svc__gamma"). The right plot scale matters: use a log axis for parameters like C, gamma and alpha that span orders of magnitude.

We will vary gamma of an RBF support vector machine. Small gamma makes a smooth, simple boundary; large gamma makes a boundary that wraps around individual points.

Python Validation curve for the SVM gamma

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Moving along the curve from left to right, you can see the three regimes. At tiny gamma both scores sit near 0.84 and barely move: the boundary is almost straight, so the model underfits the curved classes. Around gamma = 1 validation accuracy peaks at about 0.92. At large gamma the training score climbs to 0.99 while the validation score falls back to 0.84: the model has memorized the training points. The dashed line marks the best validation value. The best setting is on the validation curve, not at the training peak, and it sits where the gap has not yet blown open.

A validation curve shows one hyperparameter with the others fixed, so the true optimum might shift if you change C too. To tune several together, use a grid or random search (see Hyperparameter Tuning). The curve is best for understanding a parameter's effect and choosing a sensible search range.

More data or more complexity?

Combine the two views to decide how to spend effort.

Python Does a bigger tree or more data help?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The validation curve picks depth 6, where validation accuracy is highest (0.819); deeper trees keep gaining on the training set but lose a little on validation. The learning curve at that depth then answers the second question. Here the gap shrinks from 0.27 at 128 rows to 0.11 at 640 rows and the validation score is still climbing at the right edge (0.72 to 0.82), so more data is likely to pay off for this model. Had the validation score flattened, collecting more rows would be wasted effort, and you would need better features or a different model family.

Practical tips

  • Use enough folds and shuffle. Curves from 3 folds on a few hundred rows are noisy. Prefer 5 to 10 folds or repeated cross-validation.
  • Respect the data structure. For grouped or time-ordered data, pass a group-aware or time-aware splitter as cv, and pass groups=; see Cross-Validation for Groups and Time.
  • Use the right metric. Plot the score that matches your goal (F1, ROC AUC, negative MAE), not the default.
  • Cost grows quickly. A learning curve with 8 sizes and 5 folds is 40 fits; a validation curve with 11 values and 5 folds is 55. In the browser, keep models small and data modest, and use n_jobs=-1 on your own machine.
  • Mind the plateau. Do not extrapolate a curve beyond the data you plotted. A curve that is rising at the right edge may flatten soon.
  • Scikit-learn also offers display classes, LearningCurveDisplay.from_estimator and ValidationCurveDisplay.from_estimator, which draw these plots in a line or two. Plotting by hand, as above, makes the numbers visible and is easy to customize.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A learning curve shows training and validation accuracy both near 0.65 and flat as rows increase. What does this suggest?

    Low, converged curves mean the model cannot capture the pattern even on its own training data. Add capacity or better features rather than rows.
  2. 2. What does validation_curve vary?

    validation_curve fits the model for each value of one hyperparameter and returns the training and validation scores for each.
  3. 3. In a validation curve for SVM gamma, training accuracy keeps rising but validation accuracy falls at large gamma. What is happening?

    A large gamma lets the boundary wrap tightly around training points. Training accuracy approaches 1.0 while performance on unseen folds drops.
  4. 4. A learning curve has a large gap between training and validation scores, and the validation curve is still rising at the largest size. Which action is most promising?

    A big gap means variance, and a rising validation curve means the model is still learning from more rows. More data or regularization narrows the gap.
  5. 5. Why use a logarithmic axis for the gamma values in a validation curve?

    Such parameters act multiplicatively, so evenly spaced powers of ten (a log scale) show their effect clearly. A linear axis would crush most values together.

Summary

  • A learning curve plots training and validation scores against the training set size; a validation curve plots them against one hyperparameter.
  • Low and converged curves mean high bias; a big gap means high variance; a validation curve that is still rising means more data may help.
  • Use learning_curve and validation_curve with a proper cross-validation splitter (stratified, grouped or time-aware) and a metric that matches your goal.
  • Use a log axis for hyperparameters that span orders of magnitude, and pick the setting with the best validation score, not the best training score.
  • Curves cost many fits, so keep data and models small in the browser; use them to decide between more data, more capacity or more constraint.

Next up: Hyperparameter Tuning.