Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 111 ยท Trees and Ensembles

Random Forests

Understand random forests: bootstrap sampling, feature subsampling, voting, out-of-bag score, n_estimators, max_features, feature importances, and regression.

A single decision tree is easy to read but unstable: change a few rows and the tree changes. A random forest turns that weakness into a strength. It trains many different trees, each on a slightly different version of the data, and lets them vote. Individual trees make individual mistakes, but the mistakes are different, so the average is much steadier than any one tree.

Random forests are a sensible first choice for tabular data. They need almost no tuning, need no feature scaling, cope with nonlinear patterns and mixed features, and rarely do badly. This lesson builds the idea from the ground up, then shows the scikit-learn classes, the free out-of-bag score, and feature importances. It follows Controlling Tree Complexity, where you saw how much a deep tree varies.

Two sources of randomness

Bootstrap rows Each tree sees a random sample with repeats.
Random features Each split considers only a few features.

Different data and different questions make the trees different.

1 / 2

Bagging by hand

The first ingredient is bagging (bootstrap aggregating). A bootstrap sample draws n rows from the n training rows with replacement, so some rows appear twice or more and others not at all. On average about 63 percent of the distinct rows appear in a sample; the remaining 37 percent are left out. Train one tree per sample and combine their predictions.

Here is that idea in a few lines, on noisy half-moons. We compare the average single tree against the vote of 50 trees.

Python Bagging by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The average tree scores about 0.84, while the vote scores about 0.87, and it beats even the best individual tree here. The last line shows the 63 percent figure: about 634 of the 1,000 rows made it into the sample.

Feature subsampling: the "random" in random forest

Bagging alone leaves a problem. If one feature is very strong, every tree will split on it first, and the trees end up similar, so averaging them helps less. A random forest adds a second trick: at every split, the tree may only consider a random subset of the features (max_features). Strong features are sometimes unavailable, so other features get a chance, the trees become more diverse, and the average improves.

RandomForestClassifier

The scikit-learn estimator does both steps and parallelizes them. The most important parameter is n_estimators, the number of trees. More trees never hurt accuracy in expectation; they only cost time and memory, and returns diminish quickly.

Python One tree versus a forest

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A forest with a single tree is already a bit better than the plain tree, because its tree is trained on a bootstrap sample with limited features. The score climbs as trees are added and then flattens at about 0.87 (compared with 0.81 for a lone tree). Past roughly 100 trees, the changes are noise. The default is 100 trees, which is a good starting point.

Each tree in a default forest is grown deep (no limit), so each one has low bias and high variance, and averaging then removes the variance. You can still set max_depth, min_samples_leaf and the like when the data is very noisy or you need smaller models.

The out-of-bag score

Every tree leaves out about 37 percent of the rows. Those rows are a free test set for that tree. To score the forest, predict each row using only the trees that did not see it, and compare to the truth. That is the out-of-bag (OOB) score. It behaves like cross-validation without the extra training. Switch it on with oob_score=True.

Python OOB score versus cross-validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The OOB score settles at about 0.865 and agrees closely with cross-validation. Use it for quick checks and for watching the effect of n_estimators. Use proper cross-validation when you compare against other model types or tune many settings. With very few trees some rows are never out of bag and scikit-learn warns that the estimate is unreliable, which is why the loop starts at 25 trees.

max_features and importances

max_features sets how many features each split may consider. For classification the default is "sqrt" (the square root of the number of features), for regression it is all features. Smaller values make trees more different and faster to train but individually weaker. It is the main knob worth tuning.

Forests also report feature_importances_: the average impurity decrease each feature contributed across all trees, normalized to sum to 1. Below, only the first three of eight features carry signal (shuffle=False keeps the informative ones first), and the forest finds them.

Python max_features and feature importances

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Features 0 to 2 receive about 0.2 to 0.3 each, and the five noise features about 0.05 each. Importances never reach exactly zero for noise, because a feature always gets used somewhere. For the max_features loop, 1 gives the lowest score and the others are close (here None, which is every feature, is slightly ahead). The right value depends on the dataset, so treat it as a parameter to tune.

Impurity-based importances favor features with many distinct values and can mislead when features are correlated. For a more reliable view, use permutation importance, covered in Feature Importance and Interpretation.

RandomForestRegressor

For regression, each tree predicts a number and the forest returns the mean. The stairs of single trees blend into a smoother curve. Here is the same noisy sine as before, scored with Rยฒ under cross-validation.

Python RandomForestRegressor smooths the stairs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The forest's cross-validated Rยฒ (about 0.79) beats the tree's (about 0.73), and its red curve follows the sine far more smoothly than the jagged blue one. Like a tree, a forest still cannot extrapolate beyond the range of its training targets.

Strengths, limits and tips

  • Strong defaults. RandomForestClassifier() is a hard-to-beat baseline for tabular data. Compare it with a dummy baseline and a linear model.
  • Parallel. Pass n_jobs=-1 on your own machine to use all cores (the browser runner is single-threaded).
  • Less interpretable. You trade the readable rules of one tree for accuracy; use importances and partial dependence to explain it.
  • Big and slowish at prediction time. Hundreds of deep trees take memory; cap max_depth or min_samples_leaf if size matters.
  • Probabilities from predict_proba are the average of the trees' leaf proportions, useful but often not well calibrated (see Probability Calibration).
  • Class imbalance: try class_weight="balanced_subsample" (see Imbalanced Classes).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What are the two sources of randomness in a random forest?

    Each tree trains on a bootstrap sample of the rows, and each split looks at only a random subset of the features. Both make the trees more different from each other.
  2. 2. About what fraction of the rows is left out of a bootstrap sample?

    Sampling n rows with replacement leaves out about 1/e, roughly 37 percent. Those left-out rows form each tree's out-of-bag data.
  3. 3. What does oob_score_ estimate?

    Each row is predicted only by the trees that did not train on it, which gives an honest, cross-validation-like accuracy without extra fitting.
  4. 4. Adding more trees to a forest mostly...

    Extra trees reduce the randomness of the average, with diminishing returns. They do not cause overfitting, but they make training and prediction slower and the model larger.

Summary

  • A random forest averages many deep trees, each trained on a bootstrap sample and restricted to random features at every split.
  • Averaging cuts variance; bias stays about as low as a single deep tree's.
  • n_estimators (100 is a fine default), max_features, max_depth and min_samples_leaf are the main settings.
  • oob_score=True gives a free validation estimate from the rows each tree left out.
  • feature_importances_ is a quick guide but is biased; verify with permutation importance.

Next: Bagging, Extra Trees and Voting.