Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 111 ยท Regression

Optimizers: Momentum, RMSprop and Adam

Compare gradient descent with momentum, Nesterov, Adagrad, RMSprop and Adam in NumPy, and learn gradient clipping, weight decay and how to pick an optimizer.

Plain gradient descent takes a step of fixed size ฮท (the learning rate) opposite to the gradient. That works, but it has three weaknesses. In a long, narrow valley it zigzags across the steep walls instead of rolling along the floor. It uses the same step size for every parameter, although some parameters need big steps and some tiny ones. And with noisy mini-batch gradients it jitters around the target. Optimizers are small modifications of the update rule that deal with these problems. They are what actually trains neural networks, and they are available as options in many libraries (for example solver="adam" in scikit-learn's MLPClassifier).

This lesson implements the classic ones in a few lines of NumPy, so you can see exactly what each one adds.

The update rules

Let g be the gradient at the current weights w and ฮท the learning rate.

Momentum. Keep a running "velocity" that accumulates past gradients, like a ball rolling downhill:

TEXT
v โ† ฮฒ v + g
w โ† w - ฮท v        (ฮฒ is usually 0.9)

Steps along directions where the gradient keeps the same sign add up and speed you along the valley; steps across the valley, where the sign flips, cancel out. Nesterov momentum computes the gradient at the point where momentum is about to carry you (w - ฮท ฮฒ v) instead of the current point. This look-ahead corrects the course earlier.

Adagrad. Give each parameter its own learning rate by dividing by the square root of the sum of its past squared gradients:

TEXT
G โ† G + gยฒ
w โ† w - ฮท g / (โˆšG + ฮต)

Parameters with large gradients get smaller steps, rarely updated ones get larger steps. This suits sparse features (words in text). The weakness is that G only grows, so the effective step size shrinks until learning stalls.

RMSprop. Replace the ever-growing sum by an exponential moving average, which forgets old gradients:

TEXT
E โ† ฯ E + (1 - ฯ) gยฒ        (ฯ about 0.9)
w โ† w - ฮท g / (โˆšE + ฮต)

Adam. Combine momentum (a moving average of g, called m) with RMSprop (a moving average of gยฒ, called v), plus a bias correction for the first steps, when both averages start at zero:

TEXT
m โ† ฮฒ1 m + (1 - ฮฒ1) g          v โ† ฮฒ2 v + (1 - ฮฒ2) gยฒ
mฬ‚ = m / (1 - ฮฒ1แต—)              vฬ‚ = v / (1 - ฮฒ2แต—)
w โ† w - ฮท mฬ‚ / (โˆšvฬ‚ + ฮต)        (ฮฒ1 = 0.9, ฮฒ2 = 0.999, ฮต = 1e-8)

Adam is the usual first choice for neural networks because its defaults work reasonably well on many problems.

A race on a stretched bowl

The loss 0.5 (wโ‚ยฒ + 25 wโ‚‚ยฒ) is a bowl that is 25 times steeper in one direction than in the other. All optimizers start from (-4, 1.5), and we count the steps until the loss drops below 0.001, for several learning rates. A dash means the method never got there (too slow within 400 steps, or it diverged).

Python Six optimizers, six learning rates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table row by row, not just the "best" column. Plain gradient descent works only in a narrow band of learning rates (0.03 here; anything above 0.08 diverges because of the steep direction) and then needs 148 steps. Momentum cuts that to 66 steps and Nesterov to 30, and both tolerate a wider range of learning rates. Adagrad and RMSprop adapt the step for each coordinate: they need a larger base learning rate, and RMSprop is the fastest here (18 steps at 0.3). Adam is less spectacular on this tiny, smooth problem (76 steps at its best) but works at every rate from 0.03 to 1.0. That robustness to the choice of ฮท, not raw speed on a toy bowl, is why it is the default.

The numbers depend on the problem. On this one, both Adagrad and RMSprop benefit from the bowl being aligned with the axes. Never assume that the most sophisticated optimizer wins; tune the learning rate for whichever you pick.

Gradient clipping

Sometimes one example produces a huge gradient: a corrupted row, a very long sequence, an unlucky mini-batch. A single giant step can throw the weights far away from a good region. Gradient clipping limits the damage by capping the gradient before the update, either each value (clip by value) or the length of the whole gradient vector (clip by norm: if โ€–gโ€– > c, replace g by g ยท c / โ€–gโ€–, which keeps the direction).

The next example fits a slope by mini-batch gradient descent on data where one row is corrupted (x = 40, y = -300 instead of roughly +80). The true slope is 2.

Python One corrupted row, with and without gradient clipping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without clipping, every pass over the corrupted batch yanks the slope far from 2 (it even ends at a negative value, -5.2). With a clip value of 5 or 1 the same bad batches still push the slope in the wrong direction, but only by a bounded amount, and the slope stays near the truth (1.68 and 1.86; with more epochs it would get closer). Clipping is routine when training recurrent networks and transformers, where gradients can explode. It is a safety net: it does not repair bad data, so you still need to look for outliers.

Weight decay and learning-rate schedules

  • Weight decay (L2 regularization) shrinks the weights a little at every step, which fights overfitting (see regularization). With adaptive optimizers the penalty and the update interact in a messy way, so most deep learning code uses AdamW, which applies the decay separately from the adaptive step.
  • Learning-rate schedules shrink ฮท over time (step decay, cosine decay, warm-up followed by decay). Large steps early make fast progress; small steps late settle into the minimum. See the learning rate.

Which optimizer should I use?

  • Linear and logistic models, small data: use the solver the library recommends (lbfgs, liblinear, saga), described in multivariable calculus and SGD for linear models.
  • Neural networks, first attempt: Adam or AdamW with ฮท around 1e-3, then tune ฮท first and the other settings later.
  • Very large vision models: SGD with momentum and a good schedule is still common and sometimes generalizes slightly better.
  • Sparse inputs: Adagrad or Adam.
  • Always: watch the training loss. If it explodes, lower the learning rate or clip gradients; if it crawls, raise the learning rate.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does momentum add to plain gradient descent?

    Momentum keeps a running sum of gradients. Components that keep their sign add up, those that flip cancel.
  2. 2. Why does Adagrad slow down and eventually stall on long training runs?

    The accumulator G never shrinks, so the effective step size keeps getting smaller. RMSprop fixes this with a moving average.
  3. 3. Which two ideas does Adam combine?

    Adam keeps both a first-moment estimate (momentum) and a second-moment estimate (per-parameter scaling), with bias correction.
  4. 4. When is gradient clipping most useful?

    Clipping bounds the size of a single update, so one extreme batch cannot ruin training.

Summary

  • Optimizers modify the gradient step to cope with narrow valleys, different parameter scales and noisy gradients.
  • Momentum and Nesterov accumulate velocity; Adagrad and RMSprop scale each parameter's step by its gradient history; Adam combines momentum with RMSprop scaling and bias correction.
  • Compare optimizers across several learning rates, not at a single value: robustness to the learning rate is a real advantage.
  • Gradient clipping caps the update from extreme gradients; weight decay (AdamW) and learning-rate schedules handle overfitting and late-stage settling.

Next: how to judge a model with numbers other than accuracy, in Loss Functions and Cross-Entropy.