Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 63 ยท Neural NLP and Transformers

Recurrent Networks and LSTMs

Learn how recurrent networks read text word by word, why gradients vanish, and how LSTM gates keep memory. Includes runnable NumPy forward passes.

Averaging word vectors, as in the last lesson, treats a sentence as a bag: "the dog bites the man" and "the man bites the dog" look identical. Language is a sequence, and meaning depends on order. A recurrent neural network (RNN) reads the words one at a time and carries a running summary, the hidden state, from one step to the next. After the last word, the hidden state is a vector that depends on every word and on the order they came in.

Plain RNNs have a famous flaw: they struggle to remember anything from far back, because the learning signal fades as it travels backward through time. The LSTM (long short-term memory) was invented to fix that with gates that decide what to keep, write and reveal. This lesson builds both in NumPy so you can see the hidden state change, measure the fading gradient, and watch gates hold a memory.

The recurrent idea

An RNN applies the same small network at every position. At step t it takes the current word vector x_t and the previous hidden state h_(t-1), and produces a new hidden state:

TEXT
h_t = tanh( x_t ยท Wxh + h_(t-1) ยท Whh + b )

Wxh and Whh are weight matrices, and they are shared across all steps: that is why the model handles sentences of any length with a fixed number of parameters. The first hidden state h_0 is a vector of zeros.

flowchart LR
    H0["h0 = 0"] --> A["RNN cell"]
    X1["the"] --> A
    A -->|h1| B["RNN cell"]
    X2["dog"] --> B
    B -->|h2| C["RNN cell"]
    X3["barks"] --> C
    C -->|h3| O["Summary vector"]

The same cell is reused at every step; the hidden state is the memory handed along.

For a classification task, you take the final hidden state (or the average of all of them), feed it to a softmax layer, and train everything with backpropagation, now called backpropagation through time (BPTT) because the gradient flows back along the unrolled chain. For tagging tasks such as part-of-speech tagging you read off a prediction at every step instead.

Order sensitivity

The first example runs a randomly initialized RNN over two sentences with the same words in a different order. Every word has a fixed random vector, standing in for an embedding. Nothing is trained: the point is that the architecture itself is sensitive to order.

Python A recurrent hidden state

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first block prints the first four of eight hidden values after each word: they change at every step, because each word is mixed into the running state. The key comparison comes next. The mean of the word vectors is the same for both sentences, so its cosine is exactly 1.00. The final RNN states are different (cosine 0.49): the network has encoded that dog came before man in one sentence and after it in the other. Finally, a 20-word sentence also ends in an 8-number state, so the summary size does not grow with the input.

Vanishing and exploding gradients

Training an RNN means sending the error signal backward through every step. At each step the gradient is multiplied by the matrix Whh and by the slope of tanh (a number between 0 and 1). After T steps the gradient has been multiplied by T such factors. If those factors tend to be smaller than 1, the product shrinks toward zero exponentially: the vanishing gradient problem. Words from far back then have almost no influence on the learning, and the network cannot learn long-range dependencies such as the subject at the start of a long sentence that decides the verb form at the end. If the factors tend to be larger than 1, gradients can instead grow without bound: exploding gradients.

The second example measures this directly. It builds a tanh RNN whose recurrent matrix is a rotation (which preserves vector length) multiplied by a gain, then multiplies the step-to-step Jacobian matrices to see how strongly the first state can still influence the state T steps later.

Python Measuring a fading gradient

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a gain of 0.9, the influence of the first state falls from 0.90 after one step to 0.55 after five and 4.1e-03 after 50 steps: by 100 steps it is 2.2e-05, essentially nothing. A gain of 1.0 holds up better (0.30 after 100 steps), but needs luck and fine-tuning; with a gain of 1.5 the influence grows (70 at 100 steps), though erratically, because saturated tanh units flatten some directions. In real training, exploding gradients are tamed by gradient clipping (rescale the gradient when its norm exceeds a limit). Vanishing is harder, and that is what the LSTM is designed for.

The numbers vary with the random rotation and the starting state, which are fixed here by the seed. The pattern is what counts: a recurrent matrix that shrinks vectors makes gradients fade exponentially with distance.

The LSTM: gates and a cell state

The LSTM adds a second vector, the cell state c, which acts like a conveyor belt running along the sequence. Information can ride along it unchanged, and gates decide what is added and removed. A gate is a sigmoid layer, so its values lie between 0 (closed) and 1 (open), and it multiplies a vector element by element.

The four parts of an LSTM

Forget gate How much old memory to keep.
Input gate How much new content to write.
Candidate The new content itself.
Output gate How much memory to reveal.

Each gate looks at the current word and the previous hidden state.

1 / 2

The cell-state update is mostly addition rather than repeated matrix multiplication, so the gradient can travel back along it without being squashed at every step. That is the reason LSTMs learn dependencies hundreds of steps long where plain RNNs give up after ten or so. A GRU (gated recurrent unit) is a lighter variant with two gates and no separate cell state; it often performs about as well.

The third example implements one LSTM step. A real network learns its weights, but here a single hidden unit is wired by hand to make the mechanism visible. The input has two parts: a signal, and a "write" flag. The input gate is built to open only when the flag is on, and the forget gate to stay almost open. The first step writes a value of 1.0, then 39 steps of random noise follow with the flag off. A plain RNN with the same inputs is shown alongside.

Python An LSTM gate holding a memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At step 0 the input gate is open (0.993), and the cell stores 0.96. From then on the input gate drops to 0.007, so noise barely enters, and the forget gate stays at 0.993, so the memory decays only slowly: it is still 0.84 after 20 steps and 0.74 after 39. The plain RNN has no way to protect its state. It starts at the same 0.96, but the stream of noise keeps pushing it around, and by step 20 it has been overwritten (-0.65) and bears no relation to the value it first saw. In a trained LSTM the gates are not hand-set; the network discovers when to open and close them, for example holding on to the grammatical number of a subject until its verb arrives.

RNNs in practice

  • Bidirectional RNNs run one network forward and another backward and join their states, so every position sees both left and right context. They are standard for tagging.
  • Stacked layers. The hidden states of one layer can feed the next, giving a deeper model.
  • Sequence-to-sequence. An encoder RNN summarizes the source sentence into a vector, and a decoder RNN generates the translation from it. Squeezing a long sentence into one vector is a bottleneck, which motivated attention.
  • Cost. Each step needs the previous one, so training cannot be parallelized across positions. This is the main reason transformers replaced RNNs for most NLP work.

A sentiment classifier with an LSTM, in PyTorch (read-only: PyTorch is not available in the browser):

Python
import torch
from torch import nn

class LSTMClassifier(nn.Module):
    def __init__(self, vocab_size, dim=100, hidden=128, classes=2):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, dim, padding_idx=0)
        self.lstm = nn.LSTM(dim, hidden, batch_first=True, bidirectional=True)
        self.out = nn.Linear(2 * hidden, classes)

    def forward(self, ids, lengths):
        packed = nn.utils.rnn.pack_padded_sequence(
            self.embed(ids), lengths.cpu(), batch_first=True, enforce_sorted=False)
        _, (h_n, _) = self.lstm(packed)            # h_n: (2 directions, batch, hidden)
        final = torch.cat([h_n[0], h_n[1]], dim=1)  # forward and backward final states
        return self.out(final)

# training uses the same loop as before: loss.backward() performs BPTT;
# nn.utils.clip_grad_norm_(model.parameters(), 1.0) tames exploding gradients

pack_padded_sequence lets the LSTM skip padding, so the final state belongs to the last real word. The clipping call is the standard guard against exploding gradients.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What makes a recurrent network able to tell "the dog bites the man" from "the man bites the dog"?

    Each step mixes the new word into the running hidden state through shared weights, so different word orders produce different states, unlike a mean of word vectors.
  2. 2. Why do plain RNNs have trouble learning long-range dependencies?

    Backpropagation through time multiplies the gradient by the recurrent matrix and the tanh slope at every step, so signals from distant words can fade to almost nothing.
  3. 3. In an LSTM, what does a forget gate value near 1 do?

    The cell update is c = forget * old + input * candidate, so a forget gate near 1 preserves the old memory, and gradients can flow through it.
  4. 4. What is gradient clipping used for?

    When the gradient norm exceeds a limit, it is scaled down, so one huge update cannot wreck the weights.
  5. 5. Why did transformers largely replace RNNs for NLP?

    RNN steps depend on the previous one, so training cannot parallelize across positions, and long dependencies stay hard. Attention lets every position look at every other directly.

Summary

  • An RNN reads a sequence one word at a time, updating a hidden state with the same weights at every step; the final state depends on word order.
  • Training uses backpropagation through time; gradients tend to vanish (or explode) over long distances.
  • Gradient clipping handles exploding gradients; gated architectures address vanishing ones.
  • An LSTM adds a cell state with forget, input and output gates; mostly additive updates let information and gradients flow far.
  • A GRU is a lighter gated variant; bidirectional and stacked RNNs add context and depth.
  • Sequential computation and the single-vector bottleneck led to attention and transformers.

Next, see how attention lets every word look at every other word in Attention and the Transformer.