Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 40 ยท Regression

Ensembles and Support Vector Regression

Boost accuracy with RandomForestRegressor, GradientBoostingRegressor and HistGradientBoostingRegressor, fit non-linear data with SVR, and compare all regressors fairly.

A single decision tree is unstable: change a few training rows and the tree changes a lot. Ensembles fix that by combining many models. This lesson covers the two main ensemble ideas for regression, plus Support Vector Regression, then compares every regressor from this chapter on the same data.

Two ways to combine trees

Bagging Many trees, trained alone, averaged.
Boosting Trees in a chain, each fixes the last.

Forests reduce variance. Boosting reduces bias.

1 / 2

RandomForestRegressor

A random forest trains many trees, each on a random sample of the rows and with a random subset of features at every split, then averages their predictions. The randomness makes the trees different, so their individual errors partly cancel.

Python Forest versus single tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Useful settings: n_estimators (more is rarely worse, only slower), max_depth, min_samples_leaf and max_features. Forests need almost no tuning to perform well, and they do not need scaling.

Feature importance

A fitted forest reports how much each feature helped, as feature_importances_ (they sum to 1).

Python feature_importances_

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Treat importances as hints: they can favour numeric features with many distinct values, and share credit among correlated features.

Out-of-bag score for free

Each tree is trained on a bootstrap sample, leaving about a third of rows unseen by that tree. Setting oob_score=True evaluates each row using only the trees that did not see it, giving a test-like estimate without a separate split.

Python Out-of-bag R-squared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Gradient boosting

Boosting builds trees one after another. Each new small tree is trained to predict the errors that the ensemble so far still makes, and its output is added with a small weight (learning_rate).

Python GradientBoostingRegressor

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The key trade-off: a smaller learning rate with more trees usually generalises better but trains longer. Boosting can overfit if you add too many trees, so watch a validation score.

HistGradientBoostingRegressor: the fast one

HistGradientBoostingRegressor bins features into histograms first, which makes it dramatically faster on large tables (similar to LightGBM). It handles missing values natively and has built-in early stopping.

Python HistGradientBoostingRegressor with early stopping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For tables over about 10,000 rows, prefer this class over GradientBoostingRegressor.

Support Vector Regression (SVR)

SVR fits a function that stays within a tolerance tube of width epsilon around the data; points inside the tube cost nothing. With the rbf kernel it can learn smooth curves. It needs scaled features, and C controls how hard it tries to fit.

Python SVR with an RBF kernel

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

SVR is excellent on small to medium datasets with smooth structure, but training time grows steeply with the number of rows. SVM kernels are explained in more depth elsewhere.

A fair comparison

Use the same cross-validation folds for every model so the numbers are comparable.

Python Compare six regressors with 5-fold cross-validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On a small noisy dataset like this one, the simple linear model is competitive. Always compare against a simple model before reaching for a complex one.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does a random forest combine its trees' predictions for regression?

    Regression forests average the outputs of many decorrelated trees.
  2. 2. How does gradient boosting build its trees?

    Each new tree is fitted to the remaining errors and added with a learning rate.
  3. 3. What is a good reason to choose HistGradientBoostingRegressor?

    Histogram binning makes it fast, and missing values are supported natively.
  4. 4. Which models in this lesson need feature scaling?

    SVR is distance/kernel based and needs scaled inputs. Tree ensembles do not.

Summary

  • Random forests average many decorrelated trees: a strong, low-tuning default.
  • Gradient boosting adds trees one at a time; use a small learning_rate and early stopping.
  • HistGradientBoostingRegressor is the fast, modern choice for larger tables.
  • SVR suits small, smooth problems and needs scaling.
  • Compare models with the same cross-validation folds, and keep a simple baseline.

Next: Logistic Regression