Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 73 ยท Training in Practice

Learning Rate Schedules

Plot and compare step decay, cosine, warmup and one-cycle learning rate schedules in NumPy, run a learning rate range test, and see the effect on loss curves.

The learning rate is the single most important number in training, and the best value is not the same at the start and at the end. Early on, big steps cover a lot of ground quickly. Later, the same big steps keep the model bouncing around the minimum instead of settling into it. A learning rate schedule changes the rate while training runs: large at first, smaller later, sometimes with a gentle ramp up at the very beginning.

In this lesson you will write the four most common schedules as tiny Python functions, plot them, train a small network under each one and compare the loss curves, and use a learning rate range test to find a good base rate in a single short run. If optimizers are new to you, read Momentum, RMSProp and Adam first.

Why not a fixed rate?

Early Big steps make fast progress.
Late Small steps settle into the minimum.
Very start Adaptive optimizers need a gentle warm-up.

One number cannot be best at all three moments.

1 / 2

A schedule is just a function of the step

Every schedule maps a step number s (counting from 0) to a learning rate. That is all. In a training loop you call it before each update:

Python
for step in range(total_steps):
    lr = schedule(step)
    ...                       # forward, backward
    params -= lr * grads      # or hand lr to the optimizer

Here are the four schedules, each with a base (peak) rate:

  • Step decay: multiply the rate by gamma (for example 0.3) every every steps. Easy to reason about, with sudden drops.
  • Cosine annealing: lr = 0.5 * base * (1 + cos(pi * s / total)). It starts at base, falls slowly at first, fastest in the middle, and reaches zero at the end with a gentle landing.
  • Warmup then cosine: for the first w steps the rate climbs linearly from 0 to base, then follows the cosine curve down. Warmup protects the first steps, when Adam's running statistics are rough and the random weights produce wild gradients. Almost every transformer is trained this way.
  • One-cycle: climb from a small rate (say base / 25) up to base over the first 30% of training, then decay all the way down to a tiny rate (base / 10000). It often reaches a good result in fewer total steps.

The next example defines them and plots all four on one axis.

Python Four schedules on one plot

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step decay falls in sudden cliffs at steps 200 and 400. The cosine curve is smooth. Warmup + cosine looks like a hill with a short steep start, and one-cycle has a rounded peak at 30% followed by a long glide to nearly zero. Notice the table: at step 0 warmup starts at about 0.017 and one-cycle at 0.04, while the other two start at their full rate of 1.0.

Seeing the effect on training

A schedule only matters if it changes the result, so let us measure that. The next example trains a small network (2 inputs, 16 tanh hidden units, 1 sigmoid output) on a noisy two-moons problem, with mini-batches of 32 and SGD with momentum 0.9. Every run uses the same data, the same initial weights and the same batch sequence. The only difference is the learning rate at each step. After every step we record the loss on the whole dataset.

The runs are: a constant rate of 1.0 (a high rate), a constant rate of 0.01 (a low one), step decay and cosine, both starting from 1.0.

Python The same network under four learning rates

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers tell a clear story:

  • Constant 0.01 is stable (its spread is about 0.0003) but too timid: after 600 steps it is still at a loss of 0.327.
  • Constant 1.0 learns quickly but never settles. Its loss jitters (the spread is 0.023) and ends at 0.257, because each step still moves the weights a long way.
  • Step decay and cosine both start with the fast rate and then reduce it. They end at 0.215 and 0.207, clearly better than either constant rate. Cosine ends with almost no jitter, since its rate has fallen to nearly zero.

The same effect appears in large models: a high rate to explore, then a low rate to refine. The curve with the visibly noisy tail in the plot is the constant high rate.

The mini-batch noise visible here is explained in Mini-Batches, Epochs and Data Loading. The noise is why a shrinking rate helps: smaller steps average the noise out.

Finding the base rate: the range test

Every schedule above needs a base rate. Guessing is risky, so use the learning rate range test. Start with a tiny rate, multiply it by a constant factor after every mini-batch so it grows exponentially, and record the loss. At first the loss barely moves (rate too small), then it falls quickly (the useful range), and finally it rises or becomes erratic (rate too large). Pick a rate somewhat below the point where the loss is lowest, typically within an order of magnitude below.

The test runs for only a couple of hundred steps. Here we sweep from 1e-4 to 10 in 200 steps, smoothing the noisy loss with a short moving average.

Python A learning rate range test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

On this run the smoothed loss is lowest (0.238) at a rate of about 0.93, so the suggested base rate is about 0.09. That also explains the previous experiment: the rate of 1.0 sits right at the aggressive edge, which is why it was so jittery. The curve is flat on the far left, descends through the middle and then rises again on the right, after the rate becomes too big. The green line marks a conservative choice. Since this test uses momentum 0.9 the effective step is about ten times the printed rate, which is one more reason to stay below the minimum.

The range test changes the weights as it runs. Always start the real training from a fresh initialization (or a saved copy), never from the weights left over after the test.

Practical advice

  • Adam and AdamW: warmup for the first 1 to 5% of steps, then cosine (or linear) decay to roughly 10% of the peak or to zero. This is the default recipe for transformers.
  • SGD with momentum on CNNs: step decay (divide by 10 two or three times) or cosine. One-cycle is a strong choice when your training budget is short.
  • Decay per step, not per epoch, when you can. It makes the curve smooth, and the formulas above take s as the optimizer step count.
  • Restarts: cosine annealing is sometimes repeated in cycles ("warm restarts") to escape poor minima. It is a refinement; start without it.
  • Reduce on plateau: a simple reactive alternative. Watch the validation loss and cut the rate when it stops improving for several epochs.
  • Weight decay and schedules interact. In AdamW the decoupled decay is multiplied by the current rate, so it fades along with it.

In PyTorch these schedules are objects you step once per batch, for example torch.optim.lr_scheduler.CosineAnnealingLR and OneCycleLR. The functions you wrote do the same thing.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why do many training recipes lower the learning rate as training progresses?

    Big steps help early progress but keep the weights bouncing around a minimum. Smaller late steps average out mini-batch noise and let the model settle.
  2. 2. What is the purpose of learning rate warmup?

    Starting with a tiny rate and ramping up avoids wild early updates from randomly initialized weights.
  3. 3. In a learning rate range test, which rate should you typically choose as the base rate?

    The lowest-loss point is already near the edge of instability, so a safe base rate is lower, often about an order of magnitude.
  4. 4. A constant high learning rate gives a noisy, jittery loss near the end of training. Which change is most likely to help?

    A decaying schedule shrinks the step size late in training, so the weights settle instead of bouncing.

Summary

  • A learning rate schedule is a function from the step number to a rate, called before every update.
  • Step decay, cosine annealing, warmup plus cosine, and one-cycle are the standard shapes.
  • Starting high and finishing low beat either constant rate in the experiment above (0.207 and 0.215 against 0.257 and 0.327).
  • Warmup is standard with Adam and transformers; cosine or step decay are common with SGD.
  • The range test finds a base rate in one short run: pick a value a bit below the lowest-loss point.
  • Start every real run from fresh weights after a range test.

Next: Mini-Batches, Epochs and Data Loading.