Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 40 ยท Training and Evaluation

Optimizers

Compare SGD, momentum, RMSprop, Adam and AdamW, set learning rates and schedules, clip gradients, and watch each optimizer run on a hard problem in NumPy.

The gradient tells you which way is downhill; the optimizer decides how to step. Plain gradient descent works, but on real loss surfaces (long narrow valleys, flat plateaus, noisy gradients from mini-batches) smarter update rules converge faster and more reliably. Keras ships all the common ones. This lesson explains how they differ, and lets you race them on a function that makes plain gradient descent struggle.

The common update rules

Let g be the gradient and lr the learning rate.

SGD: w = w - lr * g. Simple, and with a good learning-rate schedule it still produces some of the best final results, but it is sensitive to the learning rate and slow in narrow valleys.

SGD with momentum: keep a velocity that accumulates past gradients: v = beta * v + g, then w = w - lr * v, with beta around 0.9. Steps that point the same way build up speed; zig-zag components cancel. In Keras: SGD(learning_rate=0.01, momentum=0.9). Adding nesterov=True peeks ahead for a slightly better update.

RMSprop: divide each step by a running average of recent gradient sizes: s = rho*s + (1-rho)*g**2, w = w - lr * g / (sqrt(s) + eps). Each parameter gets its own effective learning rate: parameters with big gradients take smaller steps and vice versa.

Adam: momentum plus RMSprop-style scaling. It keeps a running mean m of gradients and of their squares v, corrects their start-up bias, and steps w = w - lr * m_hat / (sqrt(v_hat) + eps). Defaults beta_1=0.9, beta_2=0.999, learning_rate=0.001 work remarkably often, which is why Adam is the default recommendation.

AdamW: Adam with decoupled weight decay: the shrinkage of weights is applied directly rather than being mixed into the gradient, which regularizes better. Standard for transformers (keras.optimizers.AdamW(weight_decay=...)).

flowchart LR
    G["gradient g"] --> M["momentum:<br/>smooth direction"]
    G --> S["scaling:<br/>per-parameter step size"]
    M --> A["Adam update"]
    S --> A
    A --> W["new weights"]

Adam combines a smoothed direction with a per-parameter step size.

Using optimizers in Keras

Python
import keras

opt = keras.optimizers.Adam(learning_rate=1e-3)
# or: "adam" -- the string uses all defaults

model.compile(optimizer=opt, loss="mse")

Other names: SGD, RMSprop, AdamW, Adagrad, Adadelta, Adamax, Nadam, Lion, Muon (newer Keras versions). In a custom loop you call optimizer.apply_gradients(zip(grads, model.trainable_variables)), as in Gradient Descent from Scratch.

A race on a hard function

Consider the bowl f(x, y) = 0.5 * (x**2 + 25 * y**2): gentle along x, steep along y. Gradient descent must use a learning rate small enough for the steep direction, which makes it crawl along the gentle one. We start at (8, 2) and run each optimizer for 60 steps; the minimum is 0. Lower final loss is better.

Python Stand-in: SGD, momentum, RMSprop and Adam on a narrow valley (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All of them make progress, but their routes and final accuracy differ. Try changing the learning rates: SGD with lr=0.1 diverges (the steep direction is unstable, since 25 * 0.1 > 2). There is no universally best optimizer, only good defaults.

Learning rate schedules

A constant learning rate is a compromise: large steps are great early, small steps are great at the end. A schedule reduces the rate over time. Keras accepts a schedule object where it takes a learning rate:

Python
schedule = keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate=1e-2,
    decay_steps=1000,
    decay_rate=0.9,
)
opt = keras.optimizers.Adam(learning_rate=schedule)

cosine = keras.optimizers.schedules.CosineDecay(
    initial_learning_rate=1e-3, decay_steps=10_000, alpha=0.01
)

Other options: PiecewiseConstantDecay, PolynomialDecay, warm-up phases for transformers, and the ReduceLROnPlateau callback, which lowers the rate when validation loss stops improving (Overfitting and Callbacks).

Python Stand-in: exponential and cosine schedules (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Gradient clipping

Occasional huge gradients (common in recurrent networks) can wreck training. Clip them in the optimizer:

Python
opt = keras.optimizers.Adam(learning_rate=1e-3, clipnorm=1.0)    # rescale if norm > 1
opt = keras.optimizers.Adam(learning_rate=1e-3, clipvalue=0.5)   # clip each component

clipnorm keeps the direction of the gradient and limits its length; it is usually the better choice.

Python Stand-in: clipping a gradient by norm (NumPy)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical advice

  1. Start with Adam(1e-3). It is a strong baseline for almost everything.
  2. If results plateau or the loss is noisy, try a lower learning rate (3e-4, 1e-4) before changing the optimizer.
  3. For image models trained to the very best accuracy, SGD with momentum and a decaying schedule is still competitive; for transformers use AdamW with warm-up and decay.
  4. Normalize your inputs; all optimizers behave better on well-scaled problems.
  5. Change one thing at a time and compare validation curves.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does momentum add to SGD?

    Momentum averages recent gradients, so consistent directions speed up and oscillations cancel.
  2. 2. Which optimizer is the usual first choice and why?

    Adam combines momentum with per-parameter scaling and needs little tuning.
  3. 3. What does AdamW change relative to Adam?

    Decoupled weight decay regularizes more cleanly and is standard for transformers.
  4. 4. Why use a learning rate schedule?

    Decaying the rate combines fast early progress with fine convergence at the end.

Summary

  • Optimizers turn gradients into updates: SGD, momentum, RMSprop, Adam, AdamW.
  • Adam at 1e-3 is the default baseline; AdamW for transformers; SGD with momentum for some vision work.
  • Schedules (ExponentialDecay, CosineDecay) lower the rate over time.
  • clipnorm guards against exploding gradients without changing direction.
  • Tune the learning rate first, change one thing at a time.

Next: compile, fit, evaluate and predict