Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 45 ยท Training

Optimizers: SGD, Momentum, Adam

Understand torch.optim: how SGD, momentum, RMSprop, Adam and AdamW update weights, what weight decay does, and why zero_grad and step order matter.

Autograd tells you which way each weight should move to reduce the loss. An optimizer decides how far to actually move it. Every training loop has the same three-beat rhythm: compute the gradients, let the optimizer update the parameters, and clear the gradients so the next batch starts fresh. The choice of optimizer and its settings can be the difference between a model that learns in minutes and one that never does.

This lesson explains the optimizers in torch.optim by writing their update rules yourself. As elsewhere in this track, PyTorch code is shown read-only because PyTorch cannot run in the browser; the live examples are NumPy stand-ins that implement the same formulas, so the numbers you see come from real arithmetic. It builds on Autograd in Practice and Modules, Parameters and Layers.

One job, many strategies

SGD Step against the gradient.
Momentum Keep a running velocity.
RMSprop Scale each weight by its recent gradient size.
Adam Momentum plus per-weight scaling.

They all consume the same .grad values.

1 / 2

Using torch.optim

An optimizer is created from the parameters it should update and a handful of hyperparameters. After that, the loop body barely changes whichever optimizer you choose:

Python
import torch
import torch.nn as nn

model = nn.Linear(10, 1)
loss_fn = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

for x, y in batches:
    optimizer.zero_grad()          # 1. clear gradients from the previous step
    loss = loss_fn(model(x), y)    # 2. forward pass
    loss.backward()                # 3. compute gradients (adds into .grad)
    optimizer.step()               # 4. update the parameters

The order matters. backward() adds into each parameter's .grad, so without zero_grad() the gradients of every batch pile up. Where you call zero_grad() is flexible (before the forward pass, or right after step()), as long as it happens once per update and before the next backward(). Since PyTorch 2.0, zero_grad() defaults to set_to_none=True, which frees the gradient tensors instead of filling them with zeros; the effect on the next backward() is the same and it uses less memory.

flowchart LR
    Z["zero_grad()"] --> F["forward pass"]
    F --> L["loss"]
    L --> B["backward()<br/>fills .grad"]
    B --> S["step()<br/>updates parameters"]
    S --> Z

One training step: clear, forward, loss, backward, update.

The order matters, live

This stand-in has a Param with a .grad, a SimpleSGD optimizer and a backward that accumulates like PyTorch's does. It fits y = 3x once with zero_grad each step and once without.

Python The order of zero_grad, backward and step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With zero_grad the weight ends at 2.9992, practically the target of 3. Without it, the leftover gradients keep pushing, and after 40 steps the weight sits at 1.9226, nowhere near 3. The bug is silent: no error, just a model that trains badly.

SGD and momentum

Stochastic gradient descent is the simplest rule. For each parameter p with gradient g:

TEXT
p = p - lr * g

lr is the learning rate, the step size. It is the most important hyperparameter you will tune. "Stochastic" means the gradient comes from a small random batch rather than the whole dataset, so each step is a noisy estimate of the true direction.

Plain SGD struggles on surfaces that are steep in one direction and flat in another, like a long narrow valley. It bounces across the steep walls while creeping along the floor. Momentum fixes this by remembering the previous direction. PyTorch keeps a velocity v per parameter:

TEXT
v = momentum * v + g
p = p - lr * v

Steps that agree in direction build up speed, while steps that flip sign cancel out. A typical value is momentum=0.9:

Python
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

Adaptive optimizers: RMSprop and Adam

SGD uses one learning rate for every weight. But different weights have gradients of very different sizes. Adaptive optimizers keep a running average of the squared gradient for each weight and divide the step by its square root. Weights with large, noisy gradients take smaller steps; weights with tiny gradients take relatively larger ones.

  • RMSprop does exactly that: s = alpha * s + (1 - alpha) * g**2, then p -= lr * g / (sqrt(s) + eps).
  • Adam combines RMSprop's scaling with momentum. It keeps a running mean of the gradient (m, the first moment) and of the squared gradient (v, the second moment). Because both start at zero, early averages are biased toward zero, so Adam divides them by 1 - beta**t (bias correction) before the update p -= lr * m_hat / (sqrt(v_hat) + eps).

Adam's defaults (lr=1e-3, betas=(0.9, 0.999), eps=1e-8) work remarkably well out of the box, which is why it is the usual first choice for new problems.

Python
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
optimizer = torch.optim.RMSprop(model.parameters(), lr=1e-3, alpha=0.99)

Comparing them on a loss surface

This stand-in runs four optimizers, each written in a few lines of NumPy, on a stretched bowl 0.5 * (xยฒ + 20yยฒ): shallow along x, steep along y. The start is (-8, 1), and the minimum is at (0, 0).

Python Four optimizers on one loss surface

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the columns. After 100 steps all four are close to the minimum, but plain SGD has barely moved along the shallow x axis (final x of -0.135), while momentum (-0.027) and Adam (-0.019) got much closer. Momentum is also well ahead after only 10 steps. Notice that these learning rates were picked by hand for this toy surface; change them and the ranking can change, which is a good reminder that every optimizer needs tuning.

Now see the paths themselves.

Python Optimizer paths on the surface

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the plot, momentum overshoots (its x swings out to about 3.5 on the far side of the minimum, and y dips below zero) before settling, while Adam moves both axes at a similar speed from the first steps, about 0.3 per step each, and then turns toward the minimum. Use the figure as intuition rather than as a ranking: results on real networks depend on far more than a two-dimensional bowl.

More optimizers: Adagrad, Adadelta, Nadam and friends

torch.optim offers many more optimizers than the ones above. Most are variations on the same two ideas, momentum and per-weight scaling.

Optimizer Idea PyTorch
Adagrad Divide the step by the square root of the sum of all past squared gradients torch.optim.Adagrad(params, lr=0.01)
Adadelta Like Adagrad but with a decaying average, and the step size is derived from a running average of past updates, so it needs no learning rate in theory torch.optim.Adadelta(params, lr=1.0, rho=0.9)
RMSprop A decaying average of squared gradients (rho/alpha around 0.9 to 0.99) torch.optim.RMSprop(params, lr=1e-3)
Adam / AdamW Momentum plus RMSprop scaling with bias correction torch.optim.Adam, AdamW
NAdam Adam with Nesterov momentum: the update uses a look-ahead of the momentum, often a little faster to converge torch.optim.NAdam(params, lr=2e-3)
Adamax Adam with an infinity-norm instead of the squared average torch.optim.Adamax
LBFGS A second-order method for small problems; needs a closure that re-evaluates the loss torch.optim.LBFGS

Adagrad is the ancestor of the adaptive family. Its accumulator G only grows, so frequently updated weights get smaller and smaller steps while rarely updated weights keep large ones. That is excellent for sparse features (rare words, one-hot inputs) but the learning rate eventually shrinks toward zero and training stalls. RMSprop and Adadelta fix that by forgetting old gradients with an exponential average. Adadelta additionally keeps a running average of past updates, which sets the units of the step.

Nadam (Nesterov-accelerated Adam) evaluates the momentum at the position it is about to move to, which reduces overshoot. In PyTorch use NAdam; the older TensorFlow name FTRL (follow the regularized leader) is an online-learning method built for very sparse, huge linear models and has no common PyTorch counterpart.

Advantages and limits of Adam

Adam is popular because it adapts each weight's step, needs little tuning, handles noisy and sparse gradients and converges quickly on most deep models. Its limits: it keeps two extra buffers per parameter (memory), it can generalize slightly worse than well-tuned SGD with momentum on some vision tasks, the default weight_decay in plain Adam is not true weight decay (use AdamW), and its adaptive steps can occasionally fail to converge on specially constructed problems.

Python Adagrad, RMSprop, Adadelta and Nadam on the same bowl

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Adagrad and Nadam get essentially to the minimum. RMSprop with a constant learning rate ends close but keeps jittering around it, because its step size does not shrink as it approaches. Adadelta has barely left the start after 200 steps: its first steps are tiny by design (they are scaled by the running average of past updates, which begins at zero), so it needs many more iterations. The right choice on a real model comes from experiments: start with AdamW, try SGD with momentum and a schedule (Learning Rate Schedules) if you need the last bit of accuracy.

Python
# Adagrad suits sparse embeddings; NAdam is a drop-in Adam replacement
optimizer = torch.optim.Adagrad(model.parameters(), lr=0.1)
optimizer = torch.optim.NAdam(model.parameters(), lr=2e-3)

Weight decay and AdamW

Weight decay pulls weights toward zero a little on every step. It is a simple regularizer that discourages very large weights. In SGD it is the same as adding an L2 penalty to the loss, and the option is just weight_decay=1e-4.

With Adam the two ideas come apart. If the decay term is added to the gradient (classic L2, what Adam(weight_decay=...) does), Adam then divides it by sqrt(v) along with everything else, and the penalty gets distorted. AdamW applies the decay directly to the weight, separately from the gradient step: p = p - lr * weight_decay * p. This is called decoupled weight decay, and it is the preferred form for Transformers and most modern models.

Python
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)

The stand-in below uses a weight whose data gradient is zero, so the only thing that can move it is decay. It starts once at 10 and once at 0.1.

Python Weight decay: L2 versus AdamW

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With L2 inside Adam, the big weight (10) and the small weight (0.1) both move by about the learning rate per step (10 to 9.5 in five steps, and 0.1 swings through zero and jitters around it): the normalization erased how large each weight is. AdamW shrinks each weight in proportion to its size, a steady 5% per step here (10 to 7.738, and 0.1 to 0.077). That proportional behavior is what "weight decay" is supposed to mean.

Do not apply weight decay blindly to everything. Biases and normalization parameters are usually excluded by passing the optimizer separate parameter groups, a list of dictionaries such as [{"params": decay_params}, {"params": no_decay_params, "weight_decay": 0.0}].

Choosing an optimizer

  • Start with AdamW (lr=1e-3, or 3e-4 to 1e-4 for Transformers). It is forgiving and quick to get working.
  • SGD with momentum (lr around 0.01 to 0.1, momentum=0.9) is a classic for CNNs on image tasks and can generalize slightly better when well tuned and paired with a schedule, which you will see in Learning Rate Schedules and Gradient Clipping.
  • If the loss blows up to nan or oscillates wildly, the learning rate is too high for the optimizer you picked. If the loss barely moves, it may be too low.
  • Optimizers hold state (velocities, running averages) for every parameter, so Adam uses about three times the memory of the parameters themselves. The state is part of a training checkpoint; you will save it in Saving and Loading Models.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens if you call loss.backward() on every batch but never call optimizer.zero_grad()?

    backward() accumulates into .grad. Without clearing, each update uses the sum of all earlier gradients as well, which silently breaks training.
  2. 2. In PyTorch's SGD with momentum=0.9, what does the velocity v hold?

    The velocity adds the new gradient to a decayed copy of the old velocity, and the parameter then moves by lr * v. Steps that point the same way build up speed.
  3. 3. What is the difference between Adam with weight_decay and AdamW?

    Adam's weight_decay is classic L2, added to the gradient and then divided by sqrt(v). AdamW decouples the decay from the gradient step, shrinking each weight in proportion to its size.
  4. 4. Why does Adam divide its running averages by 1 - beta**t (bias correction)?

    m and v are initialized to 0, so in the first steps they underestimate the true averages. Dividing by 1 - beta**t removes that bias.

Summary

  • An optimizer turns .grad values into parameter updates; the loop is zero_grad(), forward, backward(), step().
  • backward() accumulates gradients, so clear them once per update.
  • SGD steps against the gradient; momentum adds a velocity; RMSprop and Adam scale each weight's step by its recent gradient size, and Adam adds momentum and bias correction.
  • Weight decay shrinks weights; use AdamW for decoupled decay with Adam-style optimizers, and keep biases and norm parameters out of it.
  • Start with AdamW; try SGD with momentum when you want to tune carefully. The learning rate is the setting that matters most.

Next, change the learning rate during training with Learning Rate Schedules and Gradient Clipping.