Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 73 ยท Training in Practice

Momentum, RMSProp and Adam

Build SGD with momentum, Nesterov, RMSProp, Adam and AdamW from scratch in NumPy, compare them on a ravine surface, and learn sensible default settings.

Plain gradient descent takes a step of learning_rate * gradient and nothing more. It works, but it has two famous weaknesses. In a long narrow valley, where the surface is steep across the valley and almost flat along it, the steps bounce from wall to wall and crawl toward the minimum. And a single learning rate has to suit every parameter, even though some parameters see big gradients and others tiny ones.

Modern optimizers fix these problems with two ideas: remember where you have been (momentum), and scale each parameter's step by how large its gradients have been (adaptive learning rates). In this lesson you build SGD with momentum, Nesterov momentum, RMSProp, Adam and AdamW in a few lines of NumPy each, race them on a ravine, and finish with the defaults people actually use.

Two ideas behind every optimizer

Momentum Keep a running velocity so steady directions speed up.
Adaptive steps Divide by recent gradient size so each weight gets its own rate.

Adam combines both.

1 / 2

The ravine test surface

To compare optimizers we need a surface where plain gradient descent struggles. Take

TEXT
f(x, y) = 0.05 * x^2 + 2 * y^2

It is a bowl that is 40 times steeper in y than in x: a narrow ravine along the x axis with the minimum at the origin. Its gradient is (0.1 * x, 4 * y). Every optimizer below follows the same tiny interface: step(params, grads) returns the new parameters, keeping any internal state (velocity, running averages) inside the object. That is exactly how optimizer classes in PyTorch and Keras work, and we reuse the pattern in Building a Mini Deep-Learning Framework.

SGD with momentum

Imagine a ball rolling downhill. It does not stop and re-decide at every instant; it carries speed. Momentum keeps a velocity v that is a running sum of past gradients, and moves the parameters along it:

TEXT
v = beta * v + g
p = p - lr * v

With beta = 0.9, the velocity is roughly the average of the last ten gradients times ten. Along the valley floor the gradients all point the same way, so they add up and the ball speeds up. Across the valley the gradient flips sign every step, so the contributions cancel and the zigzag shrinks.

Nesterov momentum makes one change: it evaluates the effect of the velocity before applying it, by adding a look-ahead term. In the common formulation:

TEXT
v = beta * v + g
p = p - lr * (g + beta * v)

It reacts slightly sooner when the ball is about to overshoot.

RMSProp: a rate for every weight

Gradients of different parameters can differ by orders of magnitude. RMSProp keeps a running average of each parameter's squared gradient and divides the step by its square root:

TEXT
s = rho * s + (1 - rho) * g^2
p = p - lr * g / (sqrt(s) + eps)

A weight with consistently large gradients gets a smaller effective learning rate; one with tiny gradients gets a larger one. The small eps (like 1e-8) prevents division by zero. A useful side effect: the step size is roughly lr in magnitude, no matter how big the raw gradient is.

Adam: both ideas, with a fix

Adam (adaptive moment estimation) tracks a running mean of the gradient m (momentum) and a running mean of the squared gradient v (RMSProp):

TEXT
m = b1 * m + (1 - b1) * g
v = b2 * v + (1 - b2) * g^2
m_hat = m / (1 - b1^t)
v_hat = v / (1 - b2^t)
p = p - lr * m_hat / (sqrt(v_hat) + eps)

The divisions by 1 - b^t are the bias correction. Both running averages start at zero, so in the first few steps they are tiny underestimates of the real averages. Dividing by 1 - b^t (where t is the step count) removes that startup bias, and the correction fades away as t grows.

The standard defaults are b1 = 0.9, b2 = 0.999, eps = 1e-8. The next example implements all five optimizers, runs them for 60 steps from the point (-8, 1.5), and plots both their paths and their loss curves. The learning rates are hand-picked for this surface; with different values the ranking can change.

Python Five optimizers on a ravine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the left panel. Plain SGD with this learning rate swings from one side of the ravine to the other in y while it creeps along x. Momentum and Nesterov overshoot a little at the start, because they have built up speed, then glide along the valley floor. RMSProp and Adam divide each coordinate by its own gradient scale, so their steps in x and y are of similar size, and they head almost straight for the minimum. Their paths look more like diagonal lines than the curved paths of the momentum methods.

No optimizer wins everywhere. On this particular surface, with these particular learning rates, Nesterov ends with the lowest loss and Adam, which has a large learning rate that makes it overshoot, ends higher than RMSProp. Learning rates matter more than the algorithm name, which is the topic of Learning Rate Schedules.

Why bias correction matters

Bias correction is the least intuitive line of Adam, so let us look at it directly. We feed Adam a constant gradient of three different sizes (a thousandth, one, and a thousand) and print the first three update sizes, m / (sqrt(v) + eps), with and without correction.

Python Bias correction in the first steps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two things stand out. With correction, the first steps are exactly 1.0 times lr, whatever the gradient scale. That is the scale-free behavior Adam is known for. Without it, the first step would be about 3.16 times lr, because v (with b2 = 0.999) starts out much further below its true value than m does. Early training would take oversized steps.

AdamW and weight decay

A common way to keep weights small is L2 regularization: add wd * p to the gradient. In plain SGD that shrinks weights a little each step. Inside Adam, however, the added term gets divided by sqrt(v_hat) like every other part of the gradient, so the decay becomes uneven: weights with big gradients are barely shrunk, weights with tiny gradients are shrunk a lot.

AdamW decouples the two. It applies the Adam step using only the loss gradient, and then shrinks the weights separately:

TEXT
p = p - lr * m_hat / (sqrt(v_hat) + eps)   # Adam step
p = p - lr * wd * p                         # decoupled decay

The example below gives two parameters the same starting value 1.0 but very different gradient noise (standard deviation 0.01 and 10), runs 1000 steps with wd = 0.1, and compares Adam with L2 in the gradient to AdamW.

Python L2 inside Adam versus AdamW

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With L2 inside Adam, the small-gradient weight is crushed to 0.031 while the big-gradient weight does not shrink at all (it ends at 1.156, pushed around by noise). With AdamW both weights are pulled down by a similar amount, 0.50 and 0.62. Decay alone would leave 0.368, and the rest is random drift from the noisy gradients. The point is that AdamW treats the two weights alike, while Adam with L2 treats them wildly differently. That predictable behavior is why AdamW is the default optimizer for most modern transformers.

Adagrad, Adadelta and the rest of the family

RMSProp and Adam were preceded by Adagrad, and understanding it makes the later ones easier to read. Adagrad also keeps a per-parameter scale, but instead of a moving average it keeps the sum of all squared gradients so far:

TEXT
G = G + g^2
p = p - lr * g / (sqrt(G) + eps)

Every parameter gets its own effective learning rate, lr / sqrt(G). A parameter whose gradients have been large (a feature that fires all the time) accumulates a big G and takes small steps. A parameter that rarely receives a gradient (a rare word, a rare category) has a small G and takes a large step when its moment finally comes. That is why Adagrad was popular for sparse problems such as text features and embeddings.

The price is in the formula: G only ever grows, so the learning rate only ever shrinks. In a long run on a deep network it can decay to almost nothing before training has finished. RMSProp fixes this by replacing the sum with an exponential moving average that forgets old gradients, as you saw above.

Adadelta was proposed at about the same time as RMSProp to fix the same flaw. It also uses a moving average of squared gradients, and in addition it keeps a moving average of squared updates, which replaces the learning rate with a ratio of two running scales. In the textbook version there is no learning rate to tune (rho = 0.95 and a tiny eps are the only settings), though in practice deep-learning libraries still let you scale the result. Adamax is Adam with the L2 norm in the denominator replaced by the infinity norm (a running maximum), which makes its step size more predictable. Nadam is Adam with a Nesterov-style look-ahead, and AMSGrad keeps the largest second-moment estimate seen so far so that the effective step never grows.

The example implements Adagrad and Adadelta next to the optimizers you already know. It first prints Adagrad's effective rates on the ravine. Then it trains a logistic regression with 40 features whose firing rates range from 60% down to under 2%, which is the situation per-parameter rates were designed for.

Python Adagrad, Adadelta and friends on rare features

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On the ravine the two effective rates start at 1.25 for x and 0.17 for y, the same ratio as the gradients, and they fall to 0.47 and 0.16 by step 100: Adagrad automatically gives the flat direction the larger rate. On the rare-feature problem, plain SGD ends with a log loss of 0.234, while Adagrad reaches 0.195 and Adam 0.185. Adadelta, with no learning rate to adjust, is the slowest here (0.328): its first updates are tiny because its running update scale starts at zero. That behavior is typical, and it is one reason Adadelta lost ground to Adam. The differences between the optimizers are modest on a small problem, and a different learning rate or seed would reorder them, so read the table as a feel for each method rather than a ranking.

Optimizer Per-parameter scale Momentum Learning rate Main weakness
SGD none optional required one rate for all weights
Adagrad sum of all past g^2 no required rate decays forever
RMSProp moving average of g^2 no (optional) required no bias correction
Adadelta moving averages of g^2 and of updates no not needed (in theory) slow start
Adam moving average of g^2 yes (first moment) required weight decay must be decoupled
AdamW same as Adam yes required one more hyperparameter
Adamax / Nadam / AMSGrad variants of Adam's rule yes required rarely needed

Choosing and configuring an optimizer

Optimizer Typical settings When
SGD + momentum lr 0.01 to 0.1, beta 0.9 Image models (CNNs), when you can afford tuning and a schedule
RMSProp lr 0.001, rho 0.9 Recurrent nets, some reinforcement learning
Adam lr 0.001, betas (0.9, 0.999) A strong first choice almost everywhere
AdamW lr 0.0001 to 0.001, wd 0.01 to 0.1 Transformers and most modern training

Practical advice:

  • Start with Adam or AdamW at lr = 0.001. It is forgiving. Lower the rate (3e-4, 1e-4) for big models.
  • SGD with momentum often generalizes a little better on vision tasks when carefully tuned, but is touchier.
  • The learning rate matters far more than beta or eps. Tune it first, on a logarithmic scale (0.1, 0.03, 0.01, 0.003, ...).
  • Apply weight decay to weights, not to biases or normalization parameters.
  • Every optimizer here updates parameters in place or by returning new ones; whichever you pick, keep one optimizer object per set of parameters, because the state (v, m, s) belongs to them.

In a framework, switching is one line:

Python
# Read-only: PyTorch is not available in the browser runner
optimizer = torch.optim.SGD(model.parameters(), lr=0.05, momentum=0.9, nesterov=True)
optimizer = torch.optim.RMSprop(model.parameters(), lr=0.001, alpha=0.9)
optimizer = torch.optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01)

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What problem does momentum mainly address?

    Velocity accumulates in directions where gradients agree (along the valley) and cancels in directions where they alternate (across it).
  2. 2. What does RMSProp divide the gradient by?

    That gives every parameter its own effective step size, smaller for weights with consistently large gradients.
  3. 3. Why does Adam use bias correction?

    m and v are initialized to zero, so early values are biased toward zero. Dividing by 1 - b^t removes that bias, and the correction fades as t grows.
  4. 4. How does AdamW differ from Adam with L2 regularization?

    Decoupling the decay keeps it from being rescaled by the adaptive denominator, so all weights shrink in proportion to their size.
  5. 5. You have no idea which optimizer to use for a new model. A sensible default is:

    Adam and AdamW are forgiving and work well across tasks. Tune the learning rate first.

Summary

  • Momentum keeps a velocity of past gradients, speeding up along consistent directions and damping zigzags; Nesterov adds a look-ahead.
  • RMSProp divides each step by the root of a running mean of squared gradients, giving every weight its own rate.
  • Adagrad sums all past squared gradients (good for rare features, but its rate only shrinks); Adadelta and RMSProp use moving averages instead.
  • Adam combines momentum and RMSProp and corrects the startup bias of its running averages (defaults 0.9, 0.999, 1e-8).
  • AdamW applies weight decay separately from the gradient, which behaves much more predictably than L2 inside Adam.
  • Start with Adam or AdamW at 0.001; the learning rate matters more than the choice of optimizer.
  • Optimizer objects hold state, so keep one per set of parameters.

Next: Learning Rate Schedules.