Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 111 ยท Trees and Ensembles

Controlling Tree Complexity

Stop decision trees from memorizing noise: tune max_depth and min_samples_leaf, prune with ccp_alpha, see regression trees as stairs, and why trees vary.

Left alone, a decision tree keeps splitting until every leaf is pure. On noisy data that means it carves out a tiny region around each odd training point, and it scores 100 percent on the data it has seen and much less on new data. That gap is overfitting, and trees are especially prone to it because they have no built-in brake.

This lesson gives you the brakes. You can stop the tree early (max_depth, min_samples_leaf), or grow it fully and then cut branches back (cost-complexity pruning with ccp_alpha). You will also see why a regression tree predicts in stairs and why small data changes can reshape a tree, which is the motivation for random forests. The general idea is covered in Overfitting, Underfitting and Bias-Variance; here we apply it to trees.

Two ways to tame a tree

Stop early Limit depth or leaf size while growing.
Prune later Grow fully, then cut weak branches.

Pick the setting with cross-validation, never on the test set.

1 / 2

A noisy dataset and a depth sweep

We use two noisy interleaved half-moons. No tree can classify them perfectly, because the classes overlap, so any model that scores 100 percent on training is memorizing noise. We measure with 5-fold cross-validation, which gives an honest score for each setting, and also record the training score.

Python max_depth: train score keeps rising, CV score does not

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the two columns. The training score climbs steadily to 1.000 as the tree gets deeper. The cross-validation score rises until depth 2 or 3 (about 0.87), then falls to about 0.81 for the unlimited tree. Extra depth after the sweet spot adds only memorization. This is the classic overfitting curve, and the best depth is where the validation score peaks, not where the training score does.

min_samples_leaf: refuse tiny leaves

max_depth limits the number of questions in a chain, but a tree can still isolate one point on a short path. min_samples_leaf limits the other thing directly: no leaf may hold fewer than this many training rows (an integer, or a fraction of the data if you pass a float). Because every leaf must be backed by several examples, noise in a single row cannot create its own region.

Python min_samples_leaf sweep

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Going from 1 to about 10-25 improves the CV score from 0.81 to about 0.86. At 40 the tree can no longer follow the curved boundary and the score falls: now it is underfitting. The useful range is wide and flat, which is typical: you do not need the exact optimum, you need to be away from the extremes. max_depth, min_samples_leaf, min_samples_split and max_leaf_nodes all work in the same spirit; in practice, one or two of them are enough.

Cost-complexity pruning

Stopping early can be short-sighted: a poor split near the top might unlock great splits below it, and early stopping never gets to see those. Cost-complexity pruning takes the opposite route. Grow the full tree first, then remove branches whose improvement in training impurity is not worth the extra leaves. The knob ccp_alpha is the price of one leaf: the larger it is, the more aggressively the tree is cut back. alpha = 0 keeps everything.

cost_complexity_pruning_path returns every alpha at which the pruned tree changes. You then pick the best alpha by cross-validation.

Python Pruning path and choosing ccp_alpha

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pruned tree has 6 leaves instead of 82, and its cross-validated accuracy is about 0.87 instead of 0.81. One caution: the alphas come from the whole dataset here, which is fine for illustration. For a rigorous estimate, choose the alpha inside the training folds or use GridSearchCV with a list of ccp_alpha values (see Hyperparameter Tuning).

Regression trees draw stairs

A DecisionTreeRegressor splits the same way, but each leaf predicts the mean target value of its rows. The result is a piecewise-constant function, a staircase. More depth means more, narrower stairs, until every training point has a stair of its own.

Python Regression tree stairs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With max_depth=2 the model has four stairs: too coarse. With no limit it has 80 stairs and zigzags through every noisy point. A tree also cannot extrapolate: outside the training range it simply repeats the value of the nearest leaf, unlike a straight line that keeps going.

Why trees have high variance

Overfitting is one symptom of a deeper property: a fully grown tree is unstable. Change a few training rows and the tree can come out noticeably different. We can measure this by training many trees, each on a bootstrap resample (rows drawn with replacement) of the same data, and watching how much their predictions at one point disagree.

Python Variance of trees on bootstrap samples

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unlimited trees disagree with each other twice as much (a standard deviation of about 0.26, against about 0.13 for trees with min_samples_leaf=10), because each one reflects the quirks of its sample. Regularizing a tree trades a little bias for much less variance. Averaging many unstable trees reduces the variance without adding bias, and that is exactly the plan of the next lesson.

A practical recipe

  1. Start with DecisionTreeClassifier(random_state=0) and look at train versus CV score.
  2. If the gap is large, tune max_depth and min_samples_leaf (or ccp_alpha) by cross-validation.
  3. If the tree must be explainable, prefer a small tree (depth 3 to 5) even if it costs a point of accuracy.
  4. If accuracy matters more than explanation, move to an ensemble instead of tuning one tree for ever.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A tree has train accuracy 1.00 and cross-validation accuracy 0.80. What is the first thing to try?

    A large train/validation gap means overfitting. Constraining the tree reduces it. Scaling does nothing for trees, and tuning on the test set would leak information.
  2. 2. What does a larger ccp_alpha do?

    ccp_alpha is the penalty per leaf in cost-complexity pruning. The higher the penalty, the more branches are removed.
  3. 3. What does a regression tree predict inside a leaf?

    Each leaf predicts the average target of its training rows, which makes the prediction function a staircase.
  4. 4. Why do ensembles of trees help so much?

    Individual deep trees are unstable (high variance). Averaging predictions from many different trees smooths out their individual quirks.

Summary

  • Unconstrained trees reach 100 percent training accuracy by memorizing noise.
  • max_depth and min_samples_leaf stop growth; ccp_alpha prunes a fully grown tree. Choose all of them by cross-validation.
  • The best setting is where the validation score peaks; too strict a limit underfits.
  • Regression trees predict piecewise-constant stairs and cannot extrapolate.
  • Single trees have high variance, which is why averaging them (random forests) works.

Next: Random Forests.