Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 111 ยท Support Vector Machines and Other Models

Kernels and Nonlinear SVMs

Learn the kernel trick behind nonlinear SVMs: RBF and polynomial kernels, how gamma and C interact, boundary plots, support vector regression and scalability limits.

A straight line cannot separate a small cluster of points surrounded by a ring of the other class. But what if you could lift the points into a higher dimension, where a flat sheet can separate them? Picture the inner points pushed up like the tip of a cone and the outer ones left on the floor: a horizontal plane now slices between them. This lifting idea is the heart of kernel SVMs.

The clever part is that you never have to build the higher-dimensional coordinates. A kernel function computes the similarity two points would have after lifting, using only their original coordinates. This is the kernel trick, and it lets an SVM draw curved boundaries at the cost of a single extra choice. This lesson explains the common kernels, the two knobs (gamma and C) that you must tune together, support vector regression, and the point where SVMs stop being practical. It builds on Support Vector Machines.

The kernel trick

Lift Imagine mapping points to a higher dimension.
Separate A flat boundary there is a curve back home.
Never compute it The kernel gives the similarity directly.
1 / 2

When a line is not enough

The classic test is make_circles: one class forms an inner disc and the other an outer ring. No straight line can separate them. The example compares a linear SVM with an RBF-kernel SVM and draws both decision regions with a small helper that predicts on a grid.

Python Linear vs RBF on circles

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The linear model is barely better than a coin flip, while the RBF model draws a closed curve around the inner class and scores near-perfectly.

The kernel trick by hand

To see that the "lifting" is real, take the polynomial kernel K(x, z) = (x ยท z + 1)ยฒ for two-dimensional points. Expanding the square shows it equals an ordinary dot product in a six-dimensional space, with the feature map ฯ†(x) = (1, โˆš2ยทxโ‚, โˆš2ยทxโ‚‚, xโ‚ยฒ, xโ‚‚ยฒ, โˆš2ยทxโ‚xโ‚‚). The kernel returns that dot product without ever building the six numbers. This matters most for the RBF kernel, whose feature space is infinite-dimensional: you could never write it down, but the kernel is one cheap formula.

Python A kernel equals a dot product in a lifted space

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

An RBF similarity is 1 for identical points and decays toward 0 as the distance grows. The SVM's prediction is then a weighted vote of similarities to the support vectors: points near a support vector of class 1 push the score up, points near one of class 0 push it down.

The kernels in scikit-learn

SVC(kernel=...) accepts four built-in kernels:

Kernel Formula (idea) Parameters Notes
"linear" x ยท z none Fastest; what the previous lesson used
"poly" (gammaยทxยทz + coef0)^degree degree, coef0, gamma Smooth polynomial curves; keep the degree low (2 to 4)
"rbf" `exp(-gamma ยท x โˆ’ z
"sigmoid" tanh(gammaยทxยทz + coef0) gamma, coef0 Rarely better; mostly of historical interest

The RBF kernel is the standard first choice: it has only one extra parameter and can model almost any smooth boundary. Use "poly" when you know interactions of a limited order matter, and "linear" when you have many features and fewer rows (text is the classic case).

gamma and C together

Two parameters control an RBF SVM, and they must be tuned as a pair:

  • gamma is the reach of a single training point. Small gamma means a wide bump, so far-away points still count and the boundary is smooth and nearly linear. Large gamma means a narrow spike, so each point only influences its immediate neighborhood and the boundary wraps tightly around individual points.
  • C is the penalty for violations, as in the linear case: it decides how hard the model tries to classify every training point.

Large gamma with large C produces an island around each training point, which is classic overfitting. Small gamma with small C produces something close to a linear model, which underfits. The best models usually lie along a diagonal of the grid, where a larger gamma is paired with a smaller C and vice versa. The default gamma="scale" uses 1 / (n_features ยท X.var()), which is a sensible starting point.

The grid below fits a model for every combination on noisy moons and prints test accuracies as a table. The cells you want to find are those near the diagonal, not the corners.

Python A gamma and C grid

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The top-left (tiny gamma, small C) underfits. The bottom-right (huge gamma, large C) memorizes: the training accuracy is near perfect while the test accuracy falls, and most training points become support vectors. In real work you will not read a table by hand, you will let GridSearchCV run this search with cross-validation (Hyperparameter Tuning), usually over a logarithmic grid.

To see the effect of gamma, plot the boundary for three values on the same data.

Python Boundaries for small, medium and large gamma

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Small gamma gives a nearly straight, underfit boundary. Medium gamma follows the two moons. Large gamma carves out little islands around single points: it scores very well on the training data and will generalize badly.

Support vector regression

SVMs also do regression. SVR fits a function and ignores errors smaller than a tolerance epsilon (a tube around the prediction); only points outside the tube become support vectors and shape the fit. The kernels, gamma and C work as before, and epsilon is the tube half-width: a larger tube means a simpler model with fewer support vectors.

Python An epsilon tube around a sine wave

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pink band is the epsilon tube. With a narrow tube almost every point lies outside it and becomes a support vector; with a wide tube most points are inside it and have no effect. For regression you tune epsilon along with C and gamma, and you should scale the target if it is on a very large or very small scale.

Scalability limits

Kernel SVMs have a serious drawback: training time grows between quadratically and cubically with the number of rows, and prediction cost grows with the number of support vectors. Up to a few thousand rows they are excellent. On tens of thousands they get slow, and on hundreds of thousands they are impractical. The next demo times the fit as the data grows. Exact seconds depend on your machine; what matters is how fast the time climbs when the rows double.

Python Training time grows faster than the data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Doubling the rows more than doubles the time. That is why large problems use other tools: LinearSVC or SGDClassifier for linear boundaries, gradient boosting for tables, or approximate kernels such as Nystroem combined with a linear model.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the kernel trick avoid?

    A kernel returns the dot product the points would have in the lifted space directly from the original coordinates, so the lifted space never needs to be built.
  2. 2. Which combination is most likely to overfit an RBF SVM?

    A large gamma makes every point influence only its immediate neighborhood, and a large C punishes any training error, so the boundary wraps around individual points.
  3. 3. In SVR, what does a larger epsilon do?

    Points inside the epsilon tube contribute no loss, so a wider tube means fewer points shape the model.
  4. 4. Why are kernel SVMs a poor fit for a million-row data set?

    Training scales between O(n^2) and O(n^3). Linear SVMs, SGD or boosted trees are better choices at that size.
  5. 5. Which kernel is the usual first choice for a nonlinear problem?

    RBF is smooth, flexible and has just one extra parameter (gamma), which is why it is the default.

Summary

  • A kernel computes the similarity of two points as if they were lifted into a higher-dimensional space, which gives curved boundaries without building that space.
  • rbf is the default and the usual first choice; poly and linear fit special cases.
  • gamma is the reach of each point and C the penalty for errors; tune them together, usually on a logarithmic grid with cross-validation.
  • SVR fits a function inside an epsilon tube; only points outside the tube become support vectors.
  • Kernel SVMs shine up to a few thousand rows and do not scale far beyond that.

Next, a different way to train linear models on large data: Linear Models with SGD.