Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 111 ยท Trees and Ensembles

Bagging, Extra Trees and Voting

Learn BaggingClassifier, ExtraTreesClassifier and VotingClassifier with hard and soft voting, and see the bias-variance effect of averaging in a simulation.

A random forest is one member of a larger family of averaging ensembles: models that train several learners and combine their answers. This lesson zooms out. You will use BaggingClassifier to bag any model, not just trees, meet Extra Trees, a forest with even more randomness, and combine unrelated models with VotingClassifier. At the end a small simulation shows, in numbers, why averaging works: it cuts variance while leaving bias about where it was.

You should know decision trees and random forests first. The ideas here reappear in stacking, which goes one step further than voting.

Three ways to average models

Bagging Same model, different bootstrap samples.
Extra Trees Random thresholds, even more variety.
Voting Different models, one decision.
1 / 2

BaggingClassifier: bag anything

BaggingClassifier takes a base estimator and trains many copies of it on bootstrap samples, then votes (or averages probabilities). With deep trees as the base it is close to a random forest minus the per-split feature subsampling. It also has the free out-of-bag estimate.

Python Bagging trees, and bagging k-NN

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Bagging lifts the tree from 0.81 to 0.86, which is a large gain from an unchanged learner. Bagging k-NN does not help (0.87 either way): k-NN is already a stable method, so bootstrap copies of it agree with each other and there is nothing to average away. This is the rule of thumb: bagging helps high-variance learners such as unpruned trees, and gives little to stable ones.

Useful parameters of BaggingClassifier:

  • n_estimators: how many copies.
  • max_samples and max_features: fraction of rows and columns per copy. Smaller values make the members more diverse (and faster).
  • bootstrap=False gives pasting (sampling without replacement).
  • oob_score=True and n_jobs behave as in forests.

Extra Trees: even more random

Extremely Randomized Trees (ExtraTreesClassifier, ExtraTreesRegressor) differ from a random forest in two ways:

  1. Each tree trains on the whole training set by default (no bootstrap).
  2. At each split, instead of searching for the best threshold for each candidate feature, it draws one random threshold per candidate feature and keeps the best of those random splits.

Random thresholds make each tree worse on its own but much more different from the others, and they make training faster because no threshold search is needed. The extra randomness acts as a stronger regularizer: it adds bias and removes variance.

Python Random forest versus Extra Trees

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On this dataset the two forests are close, with the random forest ahead by about a point (0.87 against 0.86); on other data Extra Trees win, particularly when features are noisy. They are cheap to try because the API is identical. If you want an OOB score from Extra Trees you must switch the bootstrap on, as the last lines show.

VotingClassifier: combine different models

Bagging averages copies of one model. Voting averages different models, such as a linear model, a nearest-neighbor model and a forest, which tend to make different kinds of mistakes.

  • Hard voting takes the majority of the predicted class labels.
  • Soft voting averages the predicted class probabilities and picks the largest. It uses more information (how confident each member is), so it usually works better when all members have predict_proba.

Pass a list of (name, estimator) pairs. Wrap models that need scaling in a pipeline so each gets its own scaler.

Python Hard and soft voting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ensemble (0.865 to 0.867) is much better than the weakest member (logistic regression, 0.83, which cannot bend around moons) and roughly ties the strongest member k-NN (0.877). That is a typical outcome: voting rarely beats the best member by much, but it is more robust, because you did not have to guess which single model would be best on future data. It pays most when the members are about equally good and make different errors. A very weak member can drag the vote down, and weights=[...] lets you favor the stronger ones.

The last lines show soft voting at work: the final probability is exactly the mean of the three members' probabilities for that row.

Why averaging works: a bias-variance simulation

The expected squared error of a model has three parts: bias squared (systematic error), variance (sensitivity to the particular training sample) and noise you cannot remove. Averaging many models trained on different samples leaves the mean prediction about the same (so bias stays) while the spread around it shrinks (so variance falls). We can measure both. Train 40 models, each on a fresh noisy sample of the same sine curve, and compare their predictions with the true curve.

Python Bias and variance of one tree, a bagged tree, and a shallow tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the three rows:

  • The deep tree has tiny bias (0.003) but large variance (0.085): it follows every sample's noise.
  • Bagging the same trees halves the variance (0.043) and leaves the bias about unchanged. That is averaging doing exactly its job.
  • A shallow tree also cuts variance (0.040), but by restricting the model, so bias jumps ten times higher (0.029). Both fixes reduce variance; only averaging does it without paying in bias.

Averaging does not remove variance completely, because the bagged trees are trained on overlapping data and are correlated. The more different the members, the more variance cancels, which is why forests also randomize features and Extra Trees randomize thresholds.

Choosing between them

Situation Reach for
Tabular data, want a strong default Random forest
Noisy features, want speed and extra regularization Extra Trees
Bag a non-tree model that is unstable BaggingClassifier
Several decent but different models already tuned VotingClassifier (soft)
Want the combiner itself to learn from the members Stacking (later lesson)

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which base model benefits most from bagging?

    Deep trees have high variance, so bootstrap copies differ a lot and averaging them helps. Stable models give nearly identical copies, so there is little to average away.
  2. 2. How does Extra Trees choose a split threshold?

    Random thresholds make trees more different and training faster. They add some bias but remove more variance.
  3. 3. What is the difference between hard and soft voting?

    Hard voting is a majority of class labels. Soft voting averages class probabilities and takes the largest, which uses each member's confidence.
  4. 4. In the simulation, bagging deep trees changed the error mainly by...

    Averaging leaves the mean prediction nearly where it was but shrinks the spread across training samples. Irreducible noise in the data stays.

Summary

  • BaggingClassifier bags any estimator; it helps high-variance learners such as deep trees and barely helps stable ones.
  • Extra Trees use random split thresholds: faster, more regularized, often close to a random forest.
  • VotingClassifier combines different models; soft voting averages probabilities and usually beats hard voting.
  • Voting seldom beats the best member by much, but it is more robust and beats the weak members.
  • Averaging reduces variance and keeps bias roughly constant, which a bias-variance simulation can show directly.

Next: Gradient Boosting.