Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 40 ยท Classification

Random Forests and Boosting

Classify with RandomForestClassifier, ExtraTrees, AdaBoost, GradientBoostingClassifier and HistGradientBoostingClassifier; learn bagging, boosting, voting and key settings.

A single decision tree is easy to read but unstable. Combining many trees gives some of the most accurate models for tabular data. This lesson covers the classifier versions of the ensembles from the regression lesson: random forests, boosting and voting.

RandomForestClassifier

A forest trains many trees on bootstrap samples with random feature subsets, and the trees vote. Probabilities are the average of the trees' leaf probabilities.

Python Forest versus single tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Settings worth knowing:

  • n_estimators: number of trees. More is better until it plateaus; 100-500 is typical.
  • max_features: features tried per split (default "sqrt" for classifiers). Smaller values make trees more different.
  • max_depth, min_samples_leaf: limit tree size to reduce overfitting and memory.
  • class_weight="balanced": gives rare classes more weight.
  • oob_score=True: a free validation estimate from the rows each tree did not see.
Python Out-of-bag score and importances

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

How many trees?

Python Accuracy as trees are added

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

ExtraTreesClassifier

Extremely randomised trees go one step further: split thresholds are chosen at random instead of searched. They train faster and sometimes generalise slightly better, a drop-in alternative to a forest.

Python ExtraTrees versus RandomForest

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Boosting

Boosting builds models in sequence, each one concentrating on what the previous ones got wrong.

  • AdaBoostClassifier re-weights misclassified samples so the next small tree (a "stump") pays attention to them.
  • GradientBoostingClassifier fits each new tree to the gradient of the loss (the remaining error) and adds it with a learning rate.
  • HistGradientBoostingClassifier is the fast, histogram-based version with native missing-value support and early stopping; use it for tables of more than a few thousand rows.
Python Boosting classifiers compared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Learning rate versus number of trees

For boosting, a small learning_rate needs more trees but usually generalises better. Early stopping picks the number of trees for you.

Python Early stopping in HistGradientBoostingClassifier

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

VotingClassifier: combine different models

Averaging different kinds of models can beat any one of them. voting="soft" averages predicted probabilities.

Python Soft voting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stacking (StackingClassifier) goes further by training a final model on the base models' outputs.

Bagging versus boosting

Bagging / forest Boosting
Trees trained Independently, in parallel Sequentially
Main effect Reduces variance Reduces bias
Overfitting risk Low Higher: tune carefully
Tuning effort Little More (learning rate, depth, trees)

A strong workflow for tables: start with a logistic regression baseline, try a random forest, then try HistGradientBoostingClassifier and tune it. No scaling is needed for tree ensembles.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does a random forest use a random subset of features at each split?

    Decorrelated trees give a larger variance reduction when averaged.
  2. 2. What is the out-of-bag score?

    Each bootstrap sample leaves out about a third of the rows, which can be used for validation.
  3. 3. Which ensemble method builds trees sequentially, each correcting earlier mistakes?

    Boosting trains learners one after another on the remaining errors.
  4. 4. What does voting='soft' do in VotingClassifier?

    Soft voting averages predict_proba outputs and picks the class with the highest average.

Summary

  • Random forests average many decorrelated trees: strong, robust, little tuning, no scaling needed.
  • Boosting adds trees sequentially; HistGradientBoostingClassifier is the fast, modern option with early stopping.
  • ExtraTreesClassifier is a faster, more random cousin of the forest.
  • VotingClassifier combines different model types.
  • Use oob_score and cross-validation to judge them honestly.

Next: Support Vector Machines