Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 40 ยท Classification

Support Vector Machines

Use SVC and LinearSVC for classification: maximum-margin idea, kernels (linear, rbf, poly), tuning C and gamma, scaling, probabilities and when SVMs shine.

A support vector machine (SVM) finds the boundary between classes with the widest possible margin: the biggest empty gap between the closest points of each class. Those closest points are the support vectors, and only they define the boundary. With a kernel, an SVM can draw curved boundaries too. SVMs are excellent on small and medium datasets with many features.

Maximum margin

Python Linear SVC and its support vectors

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only a few points (here, the ones facing each other) are support vectors. Delete any other point and the boundary does not change.

The C parameter: margin versus mistakes

Real data overlaps, so a perfect separation may not exist. C sets the price of a misclassified point (the "soft margin"):

  • Small C: wide margin, tolerates some mistakes (simpler, more bias).
  • Large C: narrow margin, tries to classify every training point correctly (can overfit).
Python C on noisy data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Kernels: curved boundaries

A kernel lets the SVM act as though the data had been mapped into a higher dimension where a straight boundary works, without actually computing that mapping (the "kernel trick").

Kernel Boundary Use
linear straight Many features, text, large data
rbf (default) smooth, flexible The usual first choice
poly polynomial curve Specific polynomial effects
sigmoid S-shaped Rarely best

rbf has a second parameter, gamma: small values give smooth, broad influence (underfit), large values make each point's influence local (overfit).

Python Circles: linear fails, rbf succeeds

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A class inside a ring cannot be separated by any straight line, so the linear kernel is near chance. The rbf kernel solves it.

Tuning C and gamma together

Python Grid of C and gamma

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a pipeline, parameters are addressed as step__parameter. Hyperparameter search covers this tool properly.

Scaling is required

SVMs rely on distances, so unscaled data gives poor results:

Python Scaled versus raw features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Probabilities and scores

SVC has no natural probability. decision_function returns the signed distance to the boundary (a confidence score). If you really need probabilities, SVC(probability=True) fits an extra calibration with cross-validation, which slows fit down.

Python decision_function and probability=True

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

LinearSVC for large data

Kernel SVMs slow down quickly as rows grow (training time is between quadratic and cubic in the number of samples). For tens of thousands of rows or sparse text, use LinearSVC or SGDClassifier(loss="hinge"), which are far faster.

Python LinearSVC

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Multiclass problems work automatically (SVC uses one-vs-one internally; LinearSVC uses one-vs-rest).

Strengths and limits

  • Strong accuracy on small and medium data, effective in high dimensions.
  • Memory-efficient prediction (only support vectors are kept).
  • Needs scaling, careful tuning of C and gamma, and is slow for very large datasets.
  • No probabilities by default.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What are support vectors?

    Only the points on or inside the margin determine the decision boundary.
  2. 2. What happens with a very large C?

    A large C penalises misclassification heavily, narrowing the margin.
  3. 3. Which kernel can separate a class enclosed by a ring of another class?

    The rbf kernel gives flexible, non-linear boundaries.
  4. 4. How do you set the SVC parameter C inside a pipeline step named svc during grid search?

    Pipeline parameters use the step name, two underscores, then the parameter name.
  5. 5. For a dataset with 500,000 rows, which is the better first try?

    Kernel SVMs scale poorly with the number of samples, while linear solvers scale well.

Summary

  • SVMs find the maximum-margin boundary; only support vectors matter.
  • C trades margin width against training mistakes; gamma sets how local the rbf kernel is.
  • Kernels (rbf first) give curved boundaries; always scale features.
  • decision_function gives scores; probability=True adds slow calibrated probabilities.
  • Use LinearSVC for large or sparse data.

Next: Naive Bayes