Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 73 ยท Sequences, Attention and Transformers

Backpropagation Through Time

Train an RNN with backpropagation through time in NumPy, see vanishing and exploding gradients with real numbers, and fix them with clipping and truncation.

In the previous lesson the recurrent weights stayed random and only the readout was trained. To train Wxh and Whh as well, you need their gradients, and those gradients are special: the same weights were used at every time step, so each weight affects the loss through many different paths. The algorithm that handles this is backpropagation through time (BPTT).

BPTT is not a new idea. It is ordinary backpropagation applied to the unrolled network, where every layer happens to share the same weights. The gradients from all time steps are simply added together. What is new is the behavior of the long chain: gradients that travel back many steps get multiplied by the same kind of matrix again and again, so they tend to shrink to nothing or blow up. This lesson implements BPTT, measures those effects with real numbers, and shows the two standard remedies.

BPTT in three moves

Forward Run the cell over the whole chunk and store every state.
Backward Walk from the last step to the first, passing a gradient through the state.
Accumulate Add each step's contribution to the shared weights.
1 / 2

The math of one step

Use the cell from before: h_t = tanh(x_t @ Wxh + h_(t-1) @ Whh + b), with logits z_t = h_t @ Why + by and a cross-entropy loss averaged over the T steps of a chunk. Going backward from the last step to the first, three things happen at step t:

  1. The output layer gives dz_t = (p_t - onehot(y_t)) / T (the softmax-cross-entropy gradient from Backpropagation in NumPy).
  2. The state receives gradient from two sources: its own output, dz_t @ Why.T, and the future, dh_next, which was passed back from step t+1. Their sum is dh_t.
  3. Passing through the tanh gives draw_t = dh_t * (1 - h_tยฒ). This vector is the gradient of the pre-activation. It updates the weights (dWxh for the row of the current input, dWhh += outer(h_(t-1), draw_t), db += draw_t) and it also becomes the message to the past: dh_next = draw_t @ Whh.T.

Step 3 is the whole story of BPTT: the message to the past is multiplied by (1 - hยฒ) and by Whh.T once for every step it travels back. Going back k steps means about k such multiplications.

The code below implements exactly that, then proves it right with a gradient check and trains on the repeating text abacad... in chunks of 24 steps, carrying the hidden state from one chunk to the next.

Python BPTT with a gradient check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All five relative errors are around 1e-10, so the analytic gradient matches the numerical one, including Whh, where one weight influences every step. The loss starts near 1.5 (a bit worse than guessing among the four letters, which would be about 1.39) and falls to almost zero, and the trained network predicts the next character perfectly. Unlike the reservoir in the previous lesson, this network learned its own memory.

Why gradients vanish or explode

Look again at the message to the past. After k steps back, the gradient at step t-k is the gradient at step t multiplied by a product of k terms of the form diag(1 - hยฒ) and Whh.T. That is a product of k Jacobian matrices:

  • If these matrices tend to shrink vectors (their largest singular value is below 1), the product decays roughly like r^k with r < 1. After 30 steps even r = 0.9 leaves about 4% of the signal. This is the vanishing gradient problem: the loss at step t provides almost no learning signal about inputs far in the past, so the network cannot learn long-range dependencies.
  • If they tend to stretch vectors (r > 1), the product grows like r^k. This is the exploding gradient problem: one update is enormous and the training jumps to a terrible place, often ending in nan.

The tanh makes vanishing more likely, since its derivative 1 - hยฒ is at most 1 and drops toward 0 when a unit saturates. The experiment below measures the gradient norm after going back k steps in a 60-step sequence, for several settings. The first column of numbers is one step back; the last is 59 steps back.

Python Measuring gradient norms through time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table row by row. With tanh and a recurrent matrix of spectral radius 0.7 the gradient is 4e-4 after 10 steps and about 1e-20 after 59: nothing survives. Even a radius of 1.0 vanishes (1e-13 at the end), because the tanh derivative is below 1 and the matrix is not an ideal rotation. A radius of 1.6 with large inputs saturates the units, which shuts the tanh down, so the gradient vanishes again, only more slowly. With tiny inputs the units stay in their linear range, and the same radius 1.6 finally produces growth (8 at the end, after first shrinking, since the norm of a product of matrices does not grow monotonically). The linear row, with a radius of 1.1 and no tanh at all, grows steadily to about 90. These are the two failure modes in one table, and note how narrow the stable zone is.

Gradient clipping

Exploding gradients have a cheap, robust cure. If the overall gradient norm exceeds a threshold, rescale the whole gradient so its norm equals the threshold. The direction stays the same; only the step length is limited:

if norm > max_norm: grad = grad * (max_norm / norm)

Here norm is computed over all parameters together (the global norm), so the relative sizes of the pieces are preserved. Clipping does nothing on normal steps and acts only on the rare giant ones, which is why it is a standard ingredient of RNN training and also of large transformer training.

The next example trains the same network twice from an initialization with large recurrent weights (standard deviation 1.0, which makes the gradient explode early on), with the same learning rate, with and without clipping at a norm of 1.0.

Python Training with and without gradient clipping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The largest gradient norm is about 29,000, and 191 of the 200 steps have a norm above 1, so this initialization explodes almost all the time. Without clipping, each of those huge steps throws the weights far away, and the loss ends near 2.4, worse than guessing (the loss for a uniform guess over the five letters is about 1.6). With clipping every step has a norm of at most 1 and the same learning rate ends at a loss around 0.38, far lower. Compare the two curves in the plot: the unclipped one jumps around, the clipped one trends downward.

Clipping fixes exploding gradients only. It cannot bring back a vanished signal, because rescaling a tiny gradient upward would amplify noise as well. Vanishing gradients need a different architecture, which is the subject of the next lesson.

Truncated BPTT

Backpropagating through a whole book would be slow and memory-hungry: every hidden state of every step must be stored, and the vanishing problem makes the early parts useless anyway. Truncated BPTT cuts the sequence into chunks of k steps. For each chunk you run forward and backward as usual, but you start the next chunk from the final hidden state of the previous one without passing gradient back across the boundary. The state (memory) keeps flowing through the whole sequence; only the gradient path is limited to k steps. You have already been doing this in the training loops above.

The price is that dependencies longer than k cannot be learned, because no gradient ever connects the cause to the effect. The experiment below uses the text abcabdabeabf..., in which the character after ab depends on the letter three positions earlier. A model without memory can reach only 75%, since it must guess among c, d, e and f after each b. We train with the same budget of 1,500 characters and different truncation lengths k, and report the mean accuracy over three random seeds.

Python How short can the truncation window be?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With k = 1 no gradient flows between steps at all, and the accuracy (78%) stays near the 75% of a model without memory. With k = 2 it is no better. At k = 4, longer than the three-step dependency, accuracy jumps to 89% on average, and at k = 8 it is perfect. The exact numbers vary with the seed and the budget, but the pattern is the point: the truncation window must be at least as long as the dependency you want to learn.

In frameworks such as PyTorch, truncated BPTT is done by calling detach() on the hidden state between chunks; in TensorFlow/Keras you choose the sequence length of each batch and keep the state between batches (a stateful layer). Picking k is a trade: larger k captures longer dependencies but costs memory and time.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does backpropagation through time do with the gradients of the shared weight matrix Whh from different time steps?

    The same Whh is used at every step, so the total gradient is the sum of the contributions from all steps.
  2. 2. A gradient is sent back 50 steps through an RNN whose Jacobians each shrink vectors by about 0.8. Roughly what happens?

    The factor is applied once per step, so the signal decays exponentially: 0.8^50 is about 0.00001. That is the vanishing gradient problem.
  3. 3. What does gradient clipping by global norm do?

    Clipping multiplies all gradients by max_norm / norm when the norm is too large. Direction is preserved and only the step length is limited.
  4. 4. In truncated BPTT with chunk length k, which statement is correct?

    Truncation limits the gradient path, not the memory. Dependencies longer than k steps cannot be learned because no gradient connects them.
  5. 5. Which problem does gradient clipping NOT solve?

    Clipping limits large gradients. It cannot restore a signal that has already shrunk to nearly zero.

Summary

  • BPTT is backpropagation on the unrolled network: go backward step by step, pass dh_next = draw @ Whh.T to the past, and add each step's contribution to the shared weights.
  • A gradient check confirms that the BPTT implementation is correct, even though one weight affects many steps.
  • Going back k steps multiplies the gradient by k Jacobians, so it tends to shrink (vanishing) or grow (exploding) exponentially with k.
  • Gradient clipping by global norm tames explosions but cannot revive vanished gradients.
  • Truncated BPTT limits the gradient path to k steps while carrying the hidden state forward; dependencies longer than k cannot be learned.
  • Vanishing gradients motivate gated architectures that give the gradient a more direct path.

Next: LSTM and GRU.