Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 39 ยท Classification

Ensembles and Support Vector Machines

Improve classifiers with bagging, random forests and boosting, then learn support vector machines, margins and kernels. Compare models with cross-validation and ROC AUC in runnable Python.

A single classifier has a ceiling. Two ideas push past it. Ensembles combine many models so their mistakes cancel out, and support vector machines (SVM) find the cleanest boundary between classes, even a curved one. Both usually beat a lone decision tree or Naive Bayes and are standard tools in data mining competitions and business projects.

Why ensembles work

If you ask 100 people who each guess right 60% of the time and make independent mistakes, the majority vote is right far more often than 60%. Models are not perfectly independent, but the principle holds: combine models that are individually decent and different from each other.

Python Majority vote of independent models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The three main strategies:

  • Bagging (bootstrap aggregating): train each model on a random sample drawn with replacement, then vote. It reduces variance. A random forest is bagged trees that also pick a random subset of features at every split.
  • Boosting: train models one after another, each focusing on the records the previous ones got wrong (AdaBoost, gradient boosting). It reduces bias.
  • Stacking: feed the predictions of several different models into a final model.
Python One tree versus a forest versus boosting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The ensembles beat the single tree. Random forests also give feature importances and an out-of-bag (OOB) score: each tree is tested on the records its bootstrap sample left out, a free validation estimate.

Python Out-of-bag score and feature importance

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For more on boosting variants see the gradient boosting lesson.

Support vector machines

An SVM looks for the hyperplane (a line in 2D, a plane in 3D) that separates two classes with the widest possible margin, the empty street between them. The records touching the edge of the street are the support vectors; only they define the boundary.

Idea of an SVM

Margin Widest street between classes
Support vectors Points on the street edge
C parameter Strict versus soft margin
Kernel Curved boundaries

Small C = wide, tolerant margin. Large C = fit every training point.

1 / 2

Real data is rarely separable by a straight line. The kernel trick lets an SVM behave as if the data were mapped into a higher-dimensional space where a flat boundary works, without ever computing that mapping. The usual kernels are linear, poly and rbf (radial basis function, the default). C controls the penalty for misclassified records, and for rbf, gamma controls how local the influence of each point is.

Python Linear versus RBF kernel on curved data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The linear kernel can only draw a straight line through the two interleaved moons, so it makes more mistakes; rbf can bend around them. Like k-NN, SVMs rely on distances, so always scale features. Training is slow on very large datasets (roughly quadratic in the number of records), so SVMs shine on small and medium problems with many features.

Comparing models properly

Pick a metric, use cross-validation and compare on the same folds. Here ROC AUC (the probability that a random positive is ranked above a random negative) compares models without choosing a threshold.

Python Model comparison with ROC AUC

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

There is no universally best classifier (the "no free lunch" idea). Start with a simple baseline, then try a forest or boosting, and keep the model whose cross-validated score and readability fit the job.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does bagging mainly reduce?

    Averaging models trained on different bootstrap samples smooths out their individual variability.
  2. 2. How does boosting differ from bagging?

    Boosting is sequential and re-weights or re-targets records that previous models got wrong.
  3. 3. What are support vectors?

    Only the records on or inside the margin determine the SVM boundary.
  4. 4. Why is feature scaling important for an RBF SVM?

    The RBF kernel is based on distance, so large-valued features would dominate unless scaled.

Summary

  • Ensembles combine many models: bagging and random forests cut variance, boosting cuts bias, stacking mixes different models.
  • Random forests give out-of-bag estimates and feature importances for free.
  • SVMs maximize the margin; kernels create curved boundaries, C and gamma tune flexibility; scale the features.
  • Compare models with cross-validation and a metric that matches the problem, such as ROC AUC.

Next: Prediction in Data Mining