Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 111 · Support Vector Machines and Other Models

Support Vector Machines

Learn how a support vector machine finds the widest margin between classes: support vectors, the C parameter, hinge loss, feature scaling and decision_function in scikit-learn.

Imagine two clouds of points that can be separated by a straight line. Infinitely many lines would do the job, so which one should you pick? Logistic regression picks the line that best explains the probabilities. A support vector machine (SVM) asks a more geometric question: which line stays as far as possible from the points of both classes? That line leaves the widest empty street between the classes, and the width of the street is called the margin.

A wide margin is a sensible bet. A new point that lands near the boundary is more likely to be on the correct side if the boundary was far from all the training points. SVMs were among the strongest general-purpose classifiers for many years, and the margin idea still appears throughout machine learning. This lesson covers the linear case: the margin, the support vectors, the softening parameter C, the hinge loss behind it all, and the one preprocessing rule you must never forget. The next lesson bends the boundary with kernels.

The idea of an SVM

Margin The empty street between the classes. Make it wide.
Support vectors The few points that touch the street decide everything.
Soft margin C decides how many violations to tolerate.
1 / 2

Margins and support vectors

For a linear SVM the decision boundary is the set of points where w · x + b = 0, where w is a weight vector and b an intercept (the same kind of line as in Logistic Regression). Two parallel lines, w · x + b = +1 and w · x + b = -1, mark the edges of the street. The distance between them is 2 / ||w||, so maximizing the margin means minimizing the length of w.

Only the points on or inside the street edges matter. They are the support vectors: the training points that "support" the street. Delete any other point, even a thousand of them, and the fitted line does not move. Move or delete a support vector and it does. That is why SVMs are described as having a sparse solution.

In the example below the two classes are well separated, so we make C huge to demand a perfect, hard margin. Then we read w, b and the support vectors from the fitted SVC and draw the street.

Python The widest street

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how few of the 60 points are circled. The solid line is the decision boundary and the dashed lines are the street edges; the circled support vectors sit exactly on them. Every other point could move around inside its own half-plane without changing the model.

Soft margins and the C parameter

Real data rarely separates cleanly. A single mislabeled point on the wrong side would make a hard-margin SVM impossible, or force a very narrow street that bends around the outlier. The soft margin fixes this by allowing some points to sit inside the street or even on the wrong side, at a price.

The parameter C sets that price:

  • Large C: violations are expensive, so the model squeezes the street narrow to classify more training points correctly. Low bias, higher variance, more risk of overfitting.
  • Small C: violations are cheap, so the model prefers a wide street even if it misclassifies some training points. Higher bias, lower variance, more regularization.

Notice that C works in the opposite direction to the alpha of Ridge and Lasso: small C means strong regularization, like logistic regression's C.

Python Three values of C on overlapping classes

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As C grows the street narrows and fewer points remain inside it, so the support vector count drops, while training accuracy changes only slightly. The tiny-C model has a very wide street and uses most of the data as support vectors. Neither extreme is "right"; you choose C by cross-validation, usually over a logarithmic grid such as 0.01, 0.1, 1, 10, 100 (Hyperparameter Tuning shows how).

The loss behind it: hinge loss

Every classifier optimizes some loss. For an SVM it is the hinge loss. With labels written as -1 and +1, and f(x) = w · x + b the signed score, the loss for one point is:

loss = max(0, 1 - y · f(x))

Read it in three cases:

  • y · f(x) >= 1: the point is correct and outside the street. Loss is exactly zero, so the point has no influence at all (this is why non-support vectors do not matter).
  • 0 < y · f(x) < 1: correct side, but inside the street. A small loss.
  • y · f(x) < 0: wrong side. A loss that grows linearly with how wrong it is.

The full objective adds the margin term: it minimizes ½ ||w||² + C · Σ hinge losses. Compare it with the log loss of logistic regression: logistic loss is never exactly zero (every point always contributes a little), hinge loss is exactly zero for confident points. The price of the hinge is that an SVM does not give you probabilities directly.

decision_function and predictions

predict returns a class, but the model also has a decision_function that returns the raw signed score w · x + b. Its sign gives the class, and its magnitude measures how far the point is from the boundary, in units where the street edge is at 1. The example verifies by hand that the score is a plain dot product and that the hinge loss formula above matches scikit-learn's hinge_loss.

Python decision_function and hinge loss by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two hinge-loss numbers match, and many rows have a margin below 1, which tells you that the soft margin is actually being used on this overlapping data. Large positive scores mean a confident prediction for class 1, large negative scores for class 0, and scores near 0 mean "this point is close to the boundary, so be careful".

If you do need probabilities, SVC(probability=True) fits an extra calibration step with internal cross-validation (slow), and Probability Calibration and Uncertainty explains the alternatives.

SVC or LinearSVC?

scikit-learn offers two linear SVM classes, built on different libraries:

SVC(kernel="linear") LinearSVC
Library libsvm liblinear
Kernels any linear only
Cost as rows grow between quadratic and cubic roughly linear
Loss hinge squared hinge by default
Use it for small to medium data, or when you need support_vectors_ large data with a linear boundary

If you only need a linear boundary on more than a few tens of thousands of rows, prefer LinearSVC (or the SGDClassifier of the upcoming SGD lesson). Both handle several classes by one-vs-rest or one-vs-one internally (Multiclass and Multilabel Classification).

Always scale the features

The margin is a distance, and distances depend on units. If one feature is in the thousands and another between 0 and 1, the large feature dominates, and the street is effectively measured along that one axis. SVMs are therefore among the models that almost always need scaling (Scaling and Normalization). Fit the scaler on the training data only, and the easiest safe way is a pipeline.

The demonstration inflates one noise column to a scale of thousands. The model's accuracy collapses; scaling restores it.

Python Scaling rescues the SVM

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without scaling, the huge noise column swamps the three informative ones. With a StandardScaler in the pipeline, the score is the same in both rows, because scaling removes the unit difference.

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. What are support vectors?

    Points that are correctly classified and outside the street have zero hinge loss and do not influence the fit. Only points on or inside the margin (or misclassified) do.
  2. 2. You lower C from 100 to 0.01 in a linear SVC. What happens?

    Small C makes violations cheap, so the SVM prefers a wide street even if some training points fall inside it or on the wrong side.
  3. 3. For a point with label +1, which score y*f(x) gives zero hinge loss?

    The hinge loss is max(0, 1 - yf(x)), which is zero whenever yf(x) is at least 1, i.e. correct and outside the street.
  4. 4. Why should you scale features before an SVM?

    Distances depend on units. Without standardizing, a column measured in thousands outweighs columns measured in fractions, whatever their usefulness.

Summary

  • A linear SVM finds the boundary with the widest margin; the margin width is 2 / ||w||.
  • Only the support vectors, the points on or inside the margin, influence the fit.
  • C trades margin width for training errors: small C means a wide, forgiving street; large C means a tight fit.
  • The hinge loss max(0, 1 - y * f(x)) is zero for confident points; decision_function returns the signed score f(x).
  • Use LinearSVC for large linear problems and SVC for flexibility; always scale the features inside a pipeline.

Next, let the boundary curve with Kernels and Nonlinear SVMs.