Skip to content
Tutorials Center
My learning

Topics

Lesson 43 of 111 ยท Model Evaluation and Tuning

Overfitting, Underfitting and Bias-Variance

Understand why models fail on new data. Compare training and test error, see the bias-variance tradeoff in a NumPy simulation, and diagnose problems from scores.

A model that scores perfectly on the data it was trained on has proven almost nothing. What you care about is how it performs on data it has never seen. The gap between those two numbers is the central problem of machine learning, and the two ways to get it wrong have names. Underfitting means the model is too simple to capture the real pattern, so it does poorly everywhere. Overfitting means the model is so flexible that it memorizes the noise in the training set, so it looks great on training data and disappoints on new data.

This chapter is about measuring and controlling that gap. In this first lesson we build the intuition with a controlled experiment where we know the true answer, and then look at how the same story shows up in real scores. You should be comfortable with train and test sets and with polynomial regression before starting.

Two ways to fail

Underfitting Too simple. Bad on train and test.
Overfitting Too flexible. Great on train, bad on test.
Goldilocks Complex enough, not more.
1 / 2

Training error versus test error

Training error is how wrong the model is on the rows it learned from. Test error is how wrong it is on held-out rows. As you make a model more flexible, training error almost always keeps falling: with enough freedom, the model can pass through every point. Test error behaves differently: it falls at first, because the model captures real structure, and then rises again as the extra freedom is spent on noise.

Let us see that with polynomial fits of increasing degree. The data come from a sine wave plus noise, so we know the truth. Degree 1 is a straight line (too simple), degree 12 has enough wiggles to chase most of the 25 points.

Python Train and test error by model complexity

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is the classic picture, called a complexity curve. The training error falls steadily. The test error forms a U: a straight line is far too simple, the best degrees sit near the bottom of the U, and higher degrees climb away because the model starts fitting noise. The dashed line is the noise floor: even a perfect model cannot beat the randomness in the data (here the noise standard deviation is 0.25, so its variance is 0.0625). Test error can approach that floor but never go below it, apart from luck in a small sample.

The bias-variance tradeoff

The U shape has a precise explanation. The expected error of a model on new data splits into three parts:

  • Bias squared: how far the average model, over all possible training sets, is from the truth. Simple models have high bias because they cannot express the real pattern.
  • Variance: how much the model changes when you train it on a different sample. Flexible models have high variance because each sample's noise pulls them in a different direction.
  • Irreducible noise: randomness in the target that no model can predict.

Error is roughly biasยฒ + variance + noise. Adding complexity lowers bias and raises variance, and the best model sits where the sum is smallest. We can measure the two parts directly if we can draw many training sets, which a simulation allows. We keep the 20 input positions fixed, redraw the noise 300 times, refit each model every time, and look at the predictions at 50 fixed points.

Python Measuring bias and variance by simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the plot, the thick black curve is the truth and each blue line is the model trained on a different sample. With degree 1, all the lines are almost the same straight line (low variance), but that line misses the sine entirely (high bias). With degree 7, the lines average out on the truth (bias squared is practically zero), yet they scatter more from sample to sample than the degree-3 lines. Degree 3 gives the best compromise. The printed table confirms it: bias squared shrinks as the degree grows (0.195, 0.005, 0.000), variance is lowest for the straight line and about four times higher at degree 7 (0.008 against 0.031), and the sum is smallest at degree 3. With only 20 points and mild noise the variance effect stays modest; add more noise or raise the degree further and it grows quickly.

Real data gives you only one training set, so you cannot measure bias and variance this way. The simulation is a teaching device. In practice you estimate the combined effect from held-out scores, as in the next section.

Diagnosing from scores

With real data you work from two numbers per model: the training score and a validation score (from a held-out set or cross-validation). Their pattern tells you which problem you have:

Training score Validation score Diagnosis What to try
Low Low (close to training) Underfitting, high bias More flexible model, better features, less regularization
High Much lower Overfitting, high variance More data, simpler model, stronger regularization, fewer features
High High (close) Healthy fit Move on or tune the details

Let us apply that to a decision tree, where depth controls complexity. A depth-1 tree is a single split; an unlimited tree grows until every training row is classified correctly.

Python Diagnosing a decision tree by its scores

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The shallow tree scores modestly on both sets: high bias. The unlimited tree reaches 1.000 on the training data, which is the telltale sign of memorization, and its test score is far lower. The depth in between does best on the test set. The thresholds in the code (0.8 and 0.1) are arbitrary rules of thumb for this example, not universal constants: what counts as a "large gap" depends on how much noise the problem has and on the metric.

Fixing each problem

If you are underfitting (high bias):

  • Use a more flexible model (higher degree, deeper trees, more hidden units).
  • Add informative features or interactions, see feature creation.
  • Reduce regularization.

If you are overfitting (high variance):

  • Collect more training data. Variance falls as the sample grows (the next lessons show learning curves for this).
  • Simplify the model or limit it: shallower trees, fewer features, a larger min_samples_leaf.
  • Add regularization, as in Ridge, Lasso and Elastic Net.
  • Average many models: random forests cut variance by averaging.
  • Use early stopping for iterative learners.

Never "fix" overfitting by tuning on the test set. Each time you look at the test score and adjust, you leak a bit of it into your choices and the score becomes optimistic. Use a validation set or cross-validation for decisions and touch the test set once at the end.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A model scores 0.99 on the training set and 0.72 on the test set. What is the most likely problem?

    A big gap with a near-perfect training score means the model memorized the training data, including its noise. That is overfitting (high variance).
  2. 2. Which pair of scores suggests underfitting?

    Both scores are low and close together, so the model is too simple to capture the pattern. A test score far above the training score would be unusual and is worth checking for mistakes.
  3. 3. As model complexity increases, what typically happens to bias and variance?

    Flexible models can express the true pattern (lower bias) but are more sensitive to the particular training sample (higher variance).
  4. 4. Why can no model reach zero test error on noisy data?

    The expected error includes the noise variance of the target itself. Even the true function would miss each noisy observation.
  5. 5. Which action most directly reduces overfitting?

    More data and constraints (regularization, shallower trees, fewer features) lower variance. The other options hide or worsen the problem.

Summary

  • Underfitting is high bias (too simple); overfitting is high variance (memorizing noise). Judge by performance on unseen data, not training data.
  • Training error keeps falling as complexity grows; test error is U-shaped, bottoming out above the noise floor.
  • Expected error is roughly bias squared plus variance plus irreducible noise; simulation with many datasets can measure each part.
  • Diagnose from the gap: low and close scores mean underfitting, high train with much lower validation means overfitting.
  • Fix underfitting with capacity and features; fix overfitting with data, constraints, regularization and averaging.
  • Never tune on the test set; use validation or cross-validation.

Next up: Cross-Validation.