Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 37 ยท Training and Scaling

The Training Loop and Gradient Descent

Train a tiny bigram language model with gradient descent in NumPy, then learn about Adam, warmup and cosine schedules, gradient clipping, batches and mixed precision.

Training a language model means repeating four steps millions of times: run the model on a batch of text, compute the loss, compute gradients of the loss with respect to every parameter, and nudge the parameters in the direction that lowers the loss. Frontier models use enormous infrastructure for this, but the loop itself is identical to the one in a 30-line script. This lesson builds that script, then explains the extra ingredients real training adds.

The loop in a toy model

We train the simplest neural language model: an embedding-like table W of size V ร— V. Row a holds the logits for the next character after character a. It has no attention, but it has parameters, a loss, gradients, and updates, which is all the loop needs.

Python Train a bigram model with gradient descent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The loss starts near ln V and falls steadily. Each step multiplies the gradient by the learning rate and subtracts it from the parameters.

Sampling from what it learned

Python Generate from the trained bigram model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The samples look like the training text broken into plausible fragments: the model has learned character-to-character habits. Real LLMs do the same with a far richer function in place of the table.

Batches and stochastic gradient descent

Computing the loss over the entire dataset for each step is impossible at scale. Instead, each step uses a mini-batch: a random sample of sequences. The gradient of a batch is a noisy but unbiased estimate of the full gradient, which is stochastic gradient descent (SGD). Larger batches reduce the noise and parallelize across many accelerators; LLM batches are often millions of tokens. When memory is short, gradient accumulation sums gradients over several small micro-batches before one update.

Adam and AdamW

Plain SGD uses one learning rate for every parameter. Adam keeps running averages of the gradient (momentum) and of its square (per-parameter scale), and divides one by the other, so every parameter gets its own effective step size. AdamW adds decoupled weight decay, a small pull of weights toward zero that regularizes the model. It is the standard optimizer for LLMs.

Python Adam versus plain gradient descent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

SGD needs a tiny rate to stay stable along the steep direction and then crawls along the flat one; Adam equalizes the directions and gets much closer in the same number of steps.

Learning-rate schedules

LLM training rarely uses a constant learning rate. The usual shape is a short warmup (rising from near zero to the peak over the first few thousand steps, which avoids unstable early updates) followed by a cosine decay (or a linear decay) to about 10 percent of the peak. Some recent recipes keep the rate constant for most of training and decay in a final short phase.

Python Warmup plus cosine schedule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Keeping training stable

  • Gradient clipping rescales the gradient if its global norm exceeds a threshold (often 1.0) so one bad batch cannot throw the model off.
  • Mixed precision keeps most arithmetic in 16-bit (bfloat16) for speed and memory while holding master weights and optimizer state in 32-bit.
  • Checkpoints save model and optimizer state regularly so a crash after weeks of training loses only a little work.
  • Loss spikes are handled by rolling back to an earlier checkpoint, skipping the offending data, or lowering the rate.
  • Parallelism: data parallelism copies the model and splits the batch, tensor and pipeline parallelism split the model itself, and sharded optimizers (ZeRO/FSDP) spread weights, gradients and optimizer state across devices.

One training step

Batch Sample sequences of tokens.
Forward Compute logits and loss.
1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does one gradient descent step do?

    Parameters move against the gradient to lower the loss.
  2. 2. Why does Adam often work better than plain SGD on LLMs?

    Per-parameter scaling and momentum make training robust across very different gradient magnitudes.
  3. 3. Why use a learning-rate warmup?

    Starting with large steps on random weights can diverge; warmup ramps up gently.
  4. 4. What does gradient clipping do?

    Clipping limits the update size so a single bad batch cannot destabilize training.

Summary

  • Training repeats batch, forward, backward, update; a bigram table demonstrates the whole loop.
  • Mini-batches give noisy but unbiased gradients; accumulation simulates large batches.
  • AdamW is the standard optimizer; schedules use warmup then cosine decay.
  • Clipping, mixed precision, checkpoints and parallelism keep large runs stable and feasible.

Next: Data Pipelines for Pretraining