Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 111 ยท Support Vector Machines and Other Models

Linear Models with SGD

Train linear classifiers with stochastic gradient descent: SGDClassifier losses, partial_fit for online learning, learning rate schedules, large-data use and the Perceptron.

Most models you have met so far look at the whole training set at once. That is fine for thousands of rows, but what if your data has ten million rows, does not fit in memory, or keeps arriving as a stream? Stochastic gradient descent (SGD) trains a model by looking at one row (or a small batch) at a time and nudging the weights a little after each look. You saw the idea in Gradient Descent from Scratch; here we use scikit-learn's ready-made version for classification.

SGDClassifier is not a new kind of model. It is a training method for linear models. By choosing the loss, you decide which model you are fitting: hinge loss gives a linear SVM, log loss gives logistic regression, and so on. In exchange you get speed on huge data, a partial_fit method for learning from chunks, and a handful of learning-rate options to understand. This lesson covers all of it, and ends with the historic Perceptron.

One trainer, many models

hinge A linear SVM.
log_loss Logistic regression, with probabilities.
modified_huber Robust to outliers, gives probabilities.
perceptron The classic mistake-driven update.
1 / 2

Picking the loss

The loss argument of SGDClassifier selects the objective that SGD minimizes. The model itself is always a linear function w ยท x + b, so every choice draws a straight boundary; what changes is how mistakes are penalized.

loss Equivalent model predict_proba?
"hinge" (default) Linear SVM No
"log_loss" Logistic regression Yes
"modified_huber" A smoother, outlier-tolerant SVM Yes
"squared_hinge" Like hinge, with a quadratic penalty (like LinearSVC) No
"perceptron" The perceptron update No

Because SGD is sensitive to feature scales, always put a StandardScaler in front of it. The example fits all five losses on the same data and compares them with the exact solvers.

Python Five losses, one trainer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The SGD versions land within a hair of the exact solvers on this data, which is the point: SGD gets you an almost-as-good linear model at a fraction of the cost when the data is large. The perceptron is typically the noisiest of the group, because it only updates on mistakes and has no notion of margin.

Regularization and the stopping rule

Like the other linear models, SGDClassifier penalizes large weights: penalty="l2" (the default), "l1" (sparse weights) or "elasticnet", with strength alpha. A larger alpha is stronger regularization (the opposite direction to C; roughly alpha โ‰ˆ 1 / (C ยท n_samples)).

The model makes several passes (epochs) over the data, shuffling each time. It stops after max_iter epochs, or earlier when the loss stops improving by more than tol for n_iter_no_change epochs. After fitting, model.n_iter_ tells you how many epochs it used. If you see a ConvergenceWarning, raise max_iter or scale your features.

Online learning with partial_fit

fit always restarts from scratch and wants all the data. partial_fit is different: each call performs a single pass over the batch you give it and continues from the current weights. That lets you train on data that arrives in chunks, or that is too big for memory and is read from disk piece by piece.

Two rules apply:

  1. On the first call, pass classes= with all labels that will ever appear, because later batches might not contain every class.
  2. Scale consistently. StandardScaler also has partial_fit, so the scaler can learn from the stream too.

The example simulates a stream of 40 batches of 100 rows. After each batch it measures accuracy on a fixed test set, so you can watch the model improve as it sees more data.

Python Learning from a stream

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model is already usable after a single batch, and from then on the accuracy wanders within a band instead of rising steadily (it was 0.78 after the first batch and 0.75 after the last in one run). That wandering is a normal property of SGD: each small batch pulls the weights in its own direction. A shrinking learning rate (next section) smooths it out.

Because partial_fit only passes over a batch once, results depend on the order of the batches. If your stream has a built-in order (all of one class first), shuffle it before feeding it in, unless you are deliberately modeling drift.

Learning rate schedules

The learning rate is how big a step the weights take after each update. Too large and the weights bounce around; too small and learning crawls. SGDClassifier offers four schedules through learning_rate:

Schedule Rate over time Parameters
"optimal" (default) Decreases, based on alpha none to set
"constant" Always eta0 eta0 required
"invscaling" eta0 / t^power_t, shrinks steadily eta0, power_t
"adaptive" Stays at eta0 while the loss improves, then divides by 5 eta0

A shrinking rate lets early updates move quickly and later ones fine-tune. A constant rate never settles but reacts quickly to change, which suits non-stationary streams. The default "optimal" is a good start; try the others only if the loss curve looks unstable or training stalls.

Python Comparing learning rate schedules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A tiny constant rate (eta0=0.0001) needs many more epochs, while an enormous one (eta0=1.0) stops early at a clearly worse score because the weights keep jumping around. A well-chosen constant rate or a shrinking schedule such as invscaling reaches a good score in fewer epochs. In practice you will tune alpha and perhaps eta0 with a cross-validated search.

Using SGD on large data

A typical recipe for data that does not fit in memory:

  1. Read the file in chunks (for example pd.read_csv(..., chunksize=...)).
  2. For stateless transforms, use a stateless featurizer. HashingVectorizer turns text into fixed-width features without a vocabulary (Text Features), which is why it is the usual partner of partial_fit.
  3. Call partial_fit per chunk, passing classes the first time.
  4. Evaluate on a held-out slice that the model never trains on.

For data that does fit in memory, compare SGD with the exact solvers. For a few hundred thousand rows, LogisticRegression or HistGradientBoostingClassifier is often fast enough and easier to tune. SGD pays off when rows reach the millions, when you need streaming, or when you want a linear model with an unusual loss. The sibling class SGDRegressor does the same for regression, as in the earlier gradient descent lesson.

The Perceptron

The Perceptron is the ancestor of neural networks and one of the oldest learning algorithms. It keeps a weight vector and visits the rows in turn. When a row is classified correctly, nothing happens. When it is wrong, the weights are nudged toward the row's class: w โ† w + y ยท x (with labels โˆ’1 and +1). That is the whole algorithm.

A beautiful result says that if the data is linearly separable, the perceptron is guaranteed to find a separating line in a finite number of updates. If it is not separable, it never settles. sklearn.linear_model.Perceptron is the same as SGDClassifier(loss="perceptron", learning_rate="constant", eta0=1, penalty=None).

Python A perceptron succeeds, then fails on XOR

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On separable data the perceptron reaches perfect accuracy within a few epochs. On the XOR pattern (the label is 1 when exactly one of the two coordinates is near 1) it hovers around chance, no matter how long it runs. This limitation is exactly what motivates nonlinear models: kernels (the previous lesson), trees, and multi-layer networks, which you will meet in Gaussian Processes, Neural Nets and Choosing a Model.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the loss argument of SGDClassifier determine?

    SGD is a training method. loss='hinge' fits a linear SVM and loss='log_loss' fits logistic regression; the model is linear either way.
  2. 2. What must you pass on the first call of partial_fit?

    Later batches might not contain every class, so the model needs the full label set up front.
  3. 3. Why do SGD models need scaled features?

    Gradient updates scale with the feature values. A single large-scale column can dominate or destabilize training.
  4. 4. On data that is not linearly separable (like XOR), what does a Perceptron do?

    The perceptron convergence guarantee needs linear separability. Without it, a straight line can never classify all the points.

Summary

  • SGDClassifier trains linear models by stochastic gradient descent; loss selects the model (hinge, log_loss, modified_huber, perceptron, ...).
  • It scales to very large data and streams, but you must scale the features and may need to tune alpha and the learning rate.
  • partial_fit learns batch by batch from the current weights; pass classes on the first call and shuffle your stream.
  • learning_rate offers optimal, constant, invscaling and adaptive; shrinking schedules settle faster and more steadily.
  • The Perceptron is the simplest mistake-driven linear classifier: it solves separable problems and fails on XOR.

Next, a classical statistical classifier built on class covariances: Discriminant Analysis.