Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 111 ยท Regression

Gradient Descent from Scratch

Build gradient descent in NumPy: loss, gradient, learning rate, batch, stochastic and mini-batch updates, convergence plots, and scikit-learn's SGDRegressor.

Least squares has a tidy formula for a straight line, but most machine learning models have no formula at all. They are trained by gradient descent: start with a guess for the parameters, measure how wrong the model is, nudge every parameter in the direction that reduces the error, and repeat. It is the engine under linear models, neural networks and gradient boosting, so it is worth building once by hand.

Here we use it on linear regression, where we can check the answer against the exact solution from Linear Regression. You will write the loss and its gradient, see what the learning rate does, compare batch, stochastic and mini-batch versions, and finish with scikit-learn's SGDRegressor.

The descent loop

Start anywhere Initial weights are a guess.
Measure loss How wrong are the predictions?
Follow the gradient Step downhill, against the slope.
Repeat Stop when the loss stops improving.
1 / 3

Loss and gradient

For linear regression with prediction y_hat = w * x + b, the loss function is the mean squared error:

TEXT
loss(w, b) = mean((y_hat - y) ** 2)

Picture the loss as a landscape over the plane of all possible (w, b) pairs: a smooth bowl, with the best line at the bottom. The gradient is the vector of slopes of that landscape, one per parameter. It points uphill, so we step the opposite way. For this loss the gradients have a short closed form:

TEXT
d loss / d w = mean(2 * (y_hat - y) * x)
d loss / d b = mean(2 * (y_hat - y))

and each update is w = w - learning_rate * gradient_w (same for b). The learning rate is a hyperparameter you choose; the number of steps is another.

Gradient descent in NumPy

Gradient descent is much happier when features have similar scales, because a stretched-out bowl makes the steps zigzag. So we standardize x first (mean 0, standard deviation 1; see Scaling and Normalization). We then compare our result with the exact LinearRegression solution.

Python Batch gradient descent for a line

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss starts huge (the line is flat at zero) and collapses within a few steps, and after 60 steps w and b agree with the exact answer to three decimals. Here b is the mean of y, because the standardized x has mean zero.

The learning rate

The step size is the most important knob. Too small and training crawls; too large and each step overshoots the bottom of the bowl so far that the loss grows instead of shrinking. This experiment runs the same loop with four learning rates and plots the loss on a log scale.

Python Four learning rates, one problem

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With the tiny rate, the loss barely falls in 40 steps. The moderate rates settle near the noise floor. The largest rate makes the loss grow at every step: the classic sign of divergence. When a loss curve goes up or turns into nan, the first thing to try is a smaller learning rate. Plotting loss against step is the standard debugging view, and a healthy curve falls quickly and then flattens.

Tip: The gradient grows with the scale of the features, so the "safe" learning rate depends on how you scale your data. That is one more reason to standardize before using any gradient-based model.

Batch, stochastic and mini-batch

The loop above computes the gradient from all rows on every step: batch gradient descent. With millions of rows, that is slow, so two variants estimate the gradient from less data:

  • Stochastic gradient descent (SGD) uses a single random row per update. Each step is very cheap but noisy, so the path wiggles around the minimum instead of settling in it.
  • Mini-batch gradient descent uses a small random group (often 32 to 512 rows). It is less noisy than SGD and still cheap, and it makes good use of vectorized hardware, so it is what most deep learning uses.

One pass over the whole dataset is called an epoch. The three variants differ in how many parameter updates happen per epoch: one for batch, n for stochastic, n / batch_size for mini-batch.

Python Batch, stochastic and mini-batch compared

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Per epoch, the stochastic and mini-batch versions make far more updates, so they get close to the solution within one or two passes, while plain batch descent needs many more. The price is noise: the stochastic path never settles exactly. In practice you also shrink the learning rate over time so the wiggle fades near the end. The last line prints the exact least squares answer; after 15 epochs the mini-batch and stochastic runs are close to it, while plain batch descent has not yet caught up.

SGDRegressor in scikit-learn

You will rarely write this loop yourself. SGDRegressor trains a linear model by stochastic gradient descent, with options for the loss, a learning rate schedule, regularization and early stopping. Because it is gradient-based, it needs scaled features, so put a StandardScaler in front of it in a pipeline (see Pipelines).

Python SGDRegressor versus the exact solution

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two models have nearly the same test error: gradient descent finds essentially the same line as the exact solution. n_iter_ shows how many epochs ran before the loss stopped improving by more than tol. For a small problem like this one, the closed-form solver is simpler and faster; SGDRegressor earns its place when the data is too large to fit in memory at once, because it supports partial_fit for streaming (covered in Linear Models with SGD).

What can go wrong

  • Divergence from a learning rate that is too large: loss increases or becomes nan.
  • Slow progress from unscaled features or too small a rate; standardize and increase the rate.
  • Stopping too early: always plot the loss; if it is still falling steeply, run more epochs.
  • Local minima are not a problem for linear regression, whose loss is a single smooth bowl, but they matter for neural networks and other nonconvex models.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In one gradient descent step, which way do parameters move?

    The gradient points toward increasing loss. Subtracting learning_rate * gradient moves the parameters downhill.
  2. 2. The loss increases at every step and eventually becomes nan. What is the most likely cause?

    Steps that are too big overshoot the minimum and land higher on the opposite slope, so the loss grows until the numbers overflow.
  3. 3. How many parameter updates happen per epoch with mini-batches of 32 rows and 640 rows of data?

    An epoch covers all rows once. With 640 rows in groups of 32 there are 640 / 32 = 20 updates per epoch.
  4. 4. Why put a StandardScaler before SGDRegressor?

    Gradient-based training is sensitive to feature scale. Standardizing gives a rounder loss bowl, so steady progress is possible with a normal learning rate.

Summary

  • Gradient descent trains a model by repeatedly stepping against the gradient of the loss.
  • For linear regression with MSE, the gradients are simple averages, and the result matches the exact least squares fit.
  • The learning rate sets the step size: too small is slow, too large diverges; plot the loss curve to see which.
  • Batch uses all rows per step, stochastic one row, mini-batch a small group; an epoch is one pass over the data.
  • SGDRegressor does this for you but needs scaled features; for small data, the exact solver is simpler.

Next: bend the line with Polynomial Regression and Nonlinearity.