Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 73 ยท Neural Networks From Scratch

Gradient Descent

Learn gradient descent from scratch: the update rule, learning rate effects, loss curves, 2D paths, and batch versus stochastic versus mini-batch, in NumPy.

You now have a way to score a network with a loss function. The loss is a landscape over the parameters, and you want to reach a low point. Gradient descent is the method: stand where you are, measure which direction is steepest uphill, and take a small step the opposite way. Repeat until the ground is flat.

Every deep-learning optimizer, from plain SGD to Adam, is a variation on this one idea. This lesson builds it up from a single number to many parameters, shows what the learning rate does to it, and compares the three ways of feeding it data: full batch, single examples, and mini-batches.

Walking downhill in fog

Gradient The direction of steepest increase.
Step Move against the gradient.
Learning rate How long each step is.

You cannot see the whole landscape, only the slope under your feet.

1 / 2

The update rule

The gradient of the loss L with respect to a parameter w is the slope dL/dw. If it is positive, increasing w raises the loss, so you should decrease w; if it is negative, do the opposite. Both cases are covered by one line:

TEXT
w  <-  w - lr * dL/dw

lr is the learning rate, a small positive number that scales the step. With many parameters you apply the rule to each one, using the vector of all partial derivatives, called the gradient and written as a vector of slopes. In NumPy that is simply params -= lr * grads.

Start with a landscape you can check by hand: f(x) = (x - 3)^2 + 1. Its slope is 2 * (x - 3), and its lowest point is at x = 3.

Python Gradient descent on one number

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the steps shrink by themselves. Far from the minimum the slope is steep and the steps are long; near the minimum the slope flattens and the steps become tiny. Each step here multiplies the distance to the minimum by 1 - 2 * lr = 0.6, which is why the error decays smoothly toward zero.

The learning rate

The learning rate is the single most important setting in training. Try four values on f(x) = x^2, whose slope is 2x and whose minimum is at 0.

Python Four learning rates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What you see:

  • Too small: progress is real but painfully slow; you may run out of time before you arrive.
  • Good: the loss falls fast and smoothly.
  • Too large, but stable: with lr = 0.95 every step overshoots to the other side of the valley. The loss still shrinks, but in a zigzag.
  • Diverging: for this function, lr > 1 makes each step overshoot by more than it gained, so x grows and the loss explodes. In real training this shows up as a loss that climbs, or becomes nan.

A practical rule is to try values spread on a log scale (0.1, 0.03, 0.01, 0.003, ...) and keep the largest one that still trains smoothly. Learning Rate Schedules shows how to start large and shrink the rate over time.

Two parameters and a ravine

Real losses have many parameters, and their landscape is rarely a round bowl. Often it is a long narrow valley: very steep in one direction and nearly flat in another. Take f(x, y) = x^2 + 10 * y^2. The gradient is (2x, 20y), so the y direction is ten times steeper.

Python Descent path on a stretched bowl

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With lr = 0.09 the steep y direction forces a zigzag, while progress along x is gentle. With lr = 0.02, the path is smooth but x crawls. This tension, where one learning rate must suit both a steep and a flat direction, is why smarter optimizers exist. You will build them in Momentum, RMSProp and Adam.

Gradient descent on a model

Now apply the rule to a real model: linear regression with a weight and a bias, trained on noisy data from the line y = 3x - 2. The loss is mean squared error, and the gradients are:

TEXT
dL/dw = mean( 2 * (pred - y) * x )
dL/db = mean( 2 * (pred - y) )
Python Fitting a line with gradient descent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A loss curve like the left plot is your main instrument. It should fall quickly at first and then flatten. When it stops changing, training has converged. The final loss is not zero because the data contains noise; the best possible MSE here is the noise variance, about 0.25.

Batch, stochastic and mini-batch

The gradient above averaged over all 100 examples. That is batch gradient descent: exact, but one step needs a full pass over the data. With a million examples that is slow, and datasets often do not fit in memory.

The shortcut is to estimate the gradient from a subset. Stochastic gradient descent (SGD) in its purest form uses one random example per step. The estimate is noisy but cheap, and on average it points the right way. In between sits mini-batch gradient descent, which uses a small random group, typically 32 to 256 examples. This is what people almost always mean by "SGD" in deep learning.

Variant Examples per step Gradient Cost per step
Batch All Exact High
Stochastic 1 Very noisy Tiny
Mini-batch 16 to 256 Slightly noisy Small, and fast on vectorized hardware

The same model with the three variants shows the difference. Each run makes the same number of passes over the data (epochs), so they have the same amount of data processed, not the same number of updates.

Python Batch, stochastic and mini-batch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Per epoch, the mini-batch run makes ten times more updates than full batch and gets further. The single-example run makes 200 updates per epoch, but its curve is jagged: each step follows a noisy gradient. That noise is not always bad. It can help a network escape sharp, narrow valleys, and it is part of why mini-batches generalize well. Mini-Batches, Epochs and Data Loading covers the practical details.

Vocabulary: an iteration (or step) is one parameter update. An epoch is one full pass through the training set. With 200 examples and a batch size of 20, one epoch is 10 iterations.

Choosing the step size automatically: line search and the Armijo condition

Everything so far used a fixed learning rate that you pick by hand. Classical optimization has a more careful answer: after computing the direction -g, search along that line for a step size t that is good enough. A backtracking line search does it cheaply: start with a large t and shrink it (multiply by 0.5, say) until a test passes.

The test most often used is the Armijo condition (also called the sufficient-decrease condition):

TEXT
f(w - tยทg)  <=  f(w) - c ยท t ยท ||g||ยฒ        with a small c such as 1e-4

It says the step must reduce the loss by at least a fixed fraction of what the slope predicted. A step that is too long overshoots and fails the test, so it is shortened; a step that passes is accepted. The Armijo test alone does not rule out absurdly tiny steps, which is why backtracking starts large. Stronger variants add a curvature (Wolfe) condition, which also requires the slope to have flattened enough at the new point, and the Goldstein conditions, which bound the decrease from both sides. Armijo-style acceptance is also what trust-region and quasi-Newton methods (such as L-BFGS) rely on.

Line search has a real cost: every trial step needs another evaluation of the loss. On a full dataset that is fine, but for deep networks trained on mini-batches the loss is noisy, so deep learning mostly uses a fixed rate with a schedule or an adaptive optimizer. Line search is still a good tool for small, deterministic problems and for fitting the last layers with a full-batch method.

Python Armijo backtracking on a steep bowl

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On this bowl the steep direction limits any fixed rate: 0.11 diverges and 0.01 is safe but slow. The line search picks a suitable step at every iteration, and the loss never goes up.

Knowing when to stop

Gradient descent never announces that it has finished. Common stopping rules:

  • A fixed number of epochs, chosen from experience.
  • The loss improves by less than a small tolerance for several steps.
  • The gradient norm falls below a threshold (a flat spot).
  • Validation loss stops improving, called early stopping, which you will meet in Overfitting, Underfitting and Regularization.

Be aware that a flat spot is not always the global minimum. Deep networks have saddle points and plateaus, and the flat region can be a place to cross rather than to stop. In practice, a slowly falling loss at a decent value is usually good enough.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the update rule w = w - lr * gradient do when the gradient is negative?

    A negative slope means the loss decreases as w increases. Subtracting a negative number adds to w, which moves downhill.
  2. 2. On f(x) = x^2, what happens with a learning rate of 1.05?

    Each step multiplies x by 1 - 2 * 1.05 = -1.1. The magnitude grows by 10 percent per step, so the loss explodes.
  3. 3. With 1,000 training examples and a batch size of 50, how many iterations make one epoch?

    One epoch passes through every example once: 1,000 / 50 = 20 mini-batches, so 20 parameter updates.
  4. 4. Why is mini-batch gradient descent preferred over batch and pure stochastic descent?

    Batches are noisy enough to be cheap and helpful for generalization, yet large enough to use matrix operations efficiently.
  5. 5. On a long narrow valley, why does one learning rate work badly?

    The steep direction forces a small rate to avoid zigzagging or diverging, and that same small rate crawls along the flat direction.

Summary

  • Gradient descent repeats w = w - lr * gradient: step against the slope to lower the loss.
  • The learning rate sets the step size: too small is slow, too large oscillates or diverges.
  • Stretched loss landscapes make one learning rate a compromise, which motivates momentum and Adam.
  • Batch descent uses all data per step, stochastic uses one example, and mini-batch uses a small group; mini-batch is the standard.
  • A loss curve that falls and flattens indicates convergence; an iteration is one update and an epoch is one pass over the data.

Next lesson: Backpropagation by Hand.