Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 45 ยท Training

Learning Rate Schedules and Gradient Clipping

Learn StepLR, cosine, warmup, OneCycle and ReduceLROnPlateau in PyTorch, how clip_grad_norm_ stops exploding gradients, and how to choose a learning rate.

The learning rate is the single most influential number in training. Too small and nothing happens for hours. Too large and the loss explodes. And the best value is rarely constant: early on you want big steps to cover ground, and near the end you want small steps to settle into a good minimum. A learning rate schedule changes the rate as training progresses. Gradient clipping is a companion safety net that limits how violent a single update can be.

This lesson follows Optimizers: SGD, Momentum, Adam. PyTorch itself cannot run in the browser, so the torch.optim.lr_scheduler code is shown read-only; the live examples are NumPy stand-ins that compute the same schedule formulas and apply the same clipping rule.

Rates that change

Step decay Cut the rate by a factor every few epochs.
Cosine Glide smoothly from high to near zero.
Warmup Start small, ramp up, then decay.
Plateau Cut the rate when progress stalls.

Pick one schedule, then tune the peak learning rate.

1 / 2

Using schedulers in PyTorch

A scheduler wraps an optimizer and rewrites its learning rate. You create it right after the optimizer and call scheduler.step() to advance it. Most schedulers count epochs and are stepped once per epoch, after the optimizer updates:

Python
import torch

optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.5)

for epoch in range(40):
    for x, y in train_loader:
        optimizer.zero_grad()
        loss_fn(model(x), y).backward()
        optimizer.step()              # update weights first ...
    scheduler.step()                  # ... then advance the schedule (once per epoch)
    print(epoch, optimizer.param_groups[0]["lr"])

Call optimizer.step() before scheduler.step(); PyTorch warns if you reverse the order. A few schedulers are meant to be stepped once per batch instead (OneCycleLR and warmup schedules), which the documentation for each one states. You can always read the current rate from optimizer.param_groups[0]["lr"].

The common schedules

Scheduler Shape Typical use
StepLR(step_size, gamma) multiply by gamma every step_size epochs classic CNN training
MultiStepLR(milestones, gamma) the same, at chosen epochs known good recipes
ExponentialLR(gamma) multiply by gamma every epoch smooth simple decay
CosineAnnealingLR(T_max) half a cosine wave down to eta_min the default in many modern recipes
OneCycleLR(max_lr, total_steps) ramp up, then anneal far below the start fast training, "super-convergence"
LinearLR, SequentialLR build warmup followed by another schedule Transformers
ReduceLROnPlateau cut the rate when a metric stops improving when you cannot predict the length

The formulas, live

Each of these is just a function from "how far along are we" to "what rate to use". Here are three of them as plain functions, printed side by side so you can see how they differ.

Python Schedule formulas as plain functions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The step schedule drops in clear jumps (0.1 to 0.05 at epoch 10, then 0.025, then 0.0125). Cosine stays near the top for a while and then falls smoothly, ending at 0.0002 by epoch 39. The warmup version starts at only 0.0200 and climbs to 0.1000 in five epochs before following the cosine curve.

Why warmup?

Adaptive optimizers such as Adam estimate gradient statistics from very few steps at the start, and Transformers in particular are fragile when the first updates are large. Starting with a tiny rate lets those estimates settle, and a rate ramp over the first few hundred or thousand steps removes a whole class of early divergence. In PyTorch you compose it from pieces:

Python
warmup = torch.optim.lr_scheduler.LinearLR(optimizer, start_factor=0.01, total_iters=500)
cosine = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=9500)
scheduler = torch.optim.lr_scheduler.SequentialLR(
    optimizer, schedulers=[warmup, cosine], milestones=[500]
)
# step once per batch: 500 warmup steps, then 9500 cosine steps

Seeing the shapes

The plot compares four curves over 100 steps: step decay, cosine, warmup plus cosine, and a OneCycle-style curve that rises for the first 30% of training from a small rate to the peak and then anneals to almost zero.

Python Plot four schedules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

OneCycleLR in PyTorch has the same two-phase shape with defaults of pct_start=0.3, div_factor=25 (the starting rate is max_lr / 25) and final_div_factor=1e4, and it also cycles momentum in the opposite direction by default. The curve above is a hand-written approximation of the learning-rate part, good enough to understand the idea. You give the real scheduler a peak rate chosen with a range test (see below), and the total number of steps.

ReduceLROnPlateau

The schedules above are decided in advance. ReduceLROnPlateau reacts instead: you give it a metric, usually validation loss, and when it has stopped improving for patience epochs it multiplies the rate by factor. It is the one scheduler whose step needs an argument:

Python
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
    optimizer, mode="min", factor=0.5, patience=2
)

for epoch in range(50):
    train_one_epoch()
    val_loss = evaluate()
    scheduler.step(val_loss)          # pass the metric

Its logic is short enough to write in a few lines:

Python ReduceLROnPlateau in a few lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss improves until epoch 4 and then stalls. After more than patience epochs with no real gain (epoch 7), the rate halves to 0.0500. It halves again at epoch 12, after another stretch of small or no improvement. (The real class has a few more options, such as threshold_mode and cooldown.)

Schedulers are stateful. When you save a checkpoint to resume training later, save scheduler.state_dict() together with the model and optimizer, otherwise the rate restarts from the beginning. See Saving and Loading Models.

Gradient clipping

Sometimes one unlucky batch, an unusually long sequence, or a deep recurrent network produces a gradient hundreds of times larger than normal. A single step with that gradient can throw the weights far from anything sensible, and the loss turns into nan from which it never recovers. Clipping limits the damage without changing the direction of the update.

PyTorch's clip_grad_norm_ looks at the global norm, the square root of the sum of squares across every parameter's gradient. If it exceeds max_norm, every gradient is multiplied by the same factor max_norm / total_norm. If it is already below the limit, nothing changes. You call it after backward() and before step():

Python
loss.backward()
total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()

It returns the norm before clipping, which is a useful number to log: a steady stream of values far above max_norm means your learning rate or architecture needs attention. A related function, clip_grad_value_, instead clamps each individual gradient entry into [-v, v], which does change the direction, so norm clipping is the usual choice.

Clipping in NumPy

This stand-in follows the same rule on a list of arrays.

Python Gradient clipping by global norm

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The exploding entry gave a total norm of about 40; after clipping the norm is exactly 1.0 and every gradient points the same way as before. Small gradients pass through unchanged.

Clipping stops a blow-up

Here a linear model y = w * x is trained with plain mini-batch SGD. A few samples have extreme inputs (twelve times larger than usual), which makes the loss surface extremely steep for batches that contain them. With a learning rate that is fine for normal batches, one such batch throws the weight far away.

Python Clipping tames a training spike

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without clipping, the weight is thrown out to 12.5 and keeps swinging in the early updates; it only ends near the right answer (1.949) because the later normal batches pull it back. With clipping, the weight never leaves the neighborhood of the truth (largest value 2.06, final 1.985). In a deep network with no easy way back, the unclipped run would not recover.

Choosing a learning rate

There is no formula, but there is a dependable method.

  1. Start from a known default. AdamW around 1e-3 for small models (3e-4 is a popular safe pick), SGD with momentum around 0.1 for CNNs with batch norm.
  2. Run a range test. Train for a few hundred steps while increasing the rate exponentially from something tiny to something huge, and record the loss. Choose a rate a bit below where the loss starts to rise.
  3. Watch the first epochs. If the loss is nan or climbs, lower the rate by a factor of 3 to 10. If it barely moves, raise it.
  4. Add a schedule once the base rate works. Cosine with a short warmup is a safe default.

The stand-in below shows why. It fits a linear model with plain gradient descent for 60 steps at several rates.

Python Too small, about right, too large

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At 0.0005 the loss is still 5.4 after 60 steps: far too slow. From 0.05 to 0.9 the model reaches the noise floor of about 0.0076, and at 1.1 the loss explodes to around 1012. The window of good rates is wide on the log scale, which is why a rough guess within a factor of three usually works, and why "too high" fails so dramatically.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In a typical training loop, when do you call scheduler.step() for a StepLR scheduler?

    Epoch-based schedulers advance once per epoch, after the optimizer has updated the weights. Calling it before optimizer.step() triggers a warning and skips the first value of the schedule.
  2. 2. What does torch.nn.utils.clip_grad_norm_(params, max_norm=1.0) do when the global gradient norm is 40?

    Norm clipping scales all gradients by the same factor max_norm / total_norm, which shortens the update but keeps its direction. Clamping individual entries is what clip_grad_value_ does.
  3. 3. Where does clip_grad_norm_ go in the training step?

    Clipping works on the gradients, which exist only after backward(), and it must happen before step() uses them to update the weights.
  4. 4. What is the purpose of learning-rate warmup?

    Early steps are based on very little information. A short ramp from a small rate lets the optimizer's statistics settle before the full rate is used.
  5. 5. ReduceLROnPlateau differs from StepLR because it...

    ReduceLROnPlateau.step takes the metric as an argument and cuts the rate by factor after patience epochs without improvement.

Summary

  • A scheduler changes the learning rate over time: StepLR and MultiStepLR drop it in jumps, CosineAnnealingLR glides down, OneCycleLR ramps up then anneals, and LinearLR with SequentialLR builds warmup.
  • Call optimizer.step() first, then scheduler.step() (per epoch for most, per batch for OneCycleLR and warmup); ReduceLROnPlateau.step takes a metric.
  • Warmup protects the first steps of Adam-style training, especially for Transformers.
  • clip_grad_norm_ scales all gradients by one factor when their global norm is too large, keeping direction; call it after backward() and before step(), and log the returned norm.
  • Pick the learning rate with a range test, then add a schedule; save the scheduler state in checkpoints.

Next, feed data to the loop with Datasets and DataLoaders.