Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 73 ยท Sequences, Attention and Transformers

LSTM and GRU

Understand LSTM and GRU gates and the cell state, build an LSTM forward and backward pass in NumPy, and forecast a noisy sine wave with it.

A plain RNN rewrites its entire hidden state at every step. That makes it easy to forget, and, as you saw in Backpropagation Through Time, it makes gradients vanish over long chains. The fix, proposed in the 1990s, is to give the network a separate memory lane that information can travel along almost untouched, plus gates that decide what to write to it, what to erase, and what to reveal.

The result is the long short-term memory network (LSTM). Its younger and leaner cousin is the gated recurrent unit (GRU). For roughly two decades these two were the default tools for text, speech and time series. In this lesson you will see what each gate does, implement a full LSTM with its backward pass in NumPy, check the gradients, and use it to forecast a noisy wave.

Three gates guard the memory

Forget gate How much of the old memory to keep.
Input gate How much of the new candidate to write.
Output gate How much of the memory to show to the outside.

Each gate is a sigmoid, so it outputs numbers between 0 and 1.

1 / 3

The LSTM cell

An LSTM keeps two vectors: the cell state c (the long-term memory) and the hidden state h (the output at this step). At each step, the input x_t and the previous hidden state h_(t-1) are concatenated and multiplied by one big weight matrix. The result is split into four parts:

  • i = sigmoid(...) the input gate
  • f = sigmoid(...) the forget gate
  • o = sigmoid(...) the output gate
  • g = tanh(...) the candidate values to write

Then two lines do all the work:

c_t = f * c_(t-1) + i * g

h_t = o * tanh(c_t)

The first line is the heart of the design. The new cell state is the old one scaled by the forget gate, plus the candidate scaled by the input gate. When f is near 1 and i near 0, the memory passes through unchanged. When f is near 0 the memory is wiped. When i is near 1 something new is stored. The second line decides how much of the memory is visible to the next layer.

The gates are multiplied element by element (*), so each memory slot has its own private forget, input and output control. The next example shows this with a one-unit LSTM whose weights are set by hand to behave like a latch: it stores a number when a write flag is on and ignores everything else. Then it adds the GRU step for comparison and counts parameters.

Python A hand-wired LSTM latch, and a GRU step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table shows the latch at work. The input gate is 0 on every step except step 1, where write = 1 opens it to 1.000, and the cell stores tanh(0.9) = 0.716. After that the forget gate stays at 1.000, so the value 0.716 survives four more steps of unrelated input (-0.8, 0.4, -0.5, 0.7) untouched. The output h is tanh(0.716) = 0.615 (the tiny drop to 0.614 appears because the forget and output gates are 0.99995, not exactly 1). A trained LSTM learns weights like these, in a messier, distributed form.

Why the cell state helps gradients

The key line c_t = f * c_(t-1) + i * g has a different structure from the plain RNN's h_t = tanh(h_(t-1) @ Whh + ...). The derivative of c_t with respect to c_(t-1) is just f, an element-wise number, with no matrix multiplication and no tanh squashing in the way. Going back k steps along the cell state multiplies the gradient by f_t * f_(t-1) * .... If the forget gates sit near 1, the product barely shrinks, so the gradient reaches distant steps. If a gate learns to close, the gradient is deliberately cut. The network chooses its own memory length.

This is why a common trick is to initialize the forget-gate bias to a positive value such as 1, which starts the gates mostly open. The next example measures the effect. It computes how much the final hidden state changes when only the very first input is nudged, for sequences of increasing length, in a plain tanh RNN and in LSTMs with different forget-gate biases. A larger number means the early input still matters at the end.

Python Does the first input still matter at the end?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain RNN's sensitivity falls from 0.75 to about 1e-5 over 40 steps, and the LSTM with a forget bias of 0 is even worse (an LSTM is not magic: with half-open gates it forgets quickly). With a bias of 1 the sensitivity decays slowly, to 9e-3 after 40 steps. With a bias of 3 it does not decay at all: it is about 0.96 after 40 steps, because the gates are around 0.95 and the memory is carried almost intact. The initialization alone changes how far back the network can see.

GRU: the lean alternative

The GRU, introduced in 2014, merges the cell and hidden state into one vector and uses two gates:

  • The update gate z decides how much of the state to replace: h_t = (1 - z) * h_(t-1) + z * candidate. It plays the role of both the forget and input gates at once; when z is 0 the old state passes through unchanged.
  • The reset gate r decides how much of the old state is visible when computing the candidate: candidate = tanh(x @ Wx + (r * h_(t-1)) @ Wh + b).

The (1 - z) * h_(t-1) term gives the same kind of additive shortcut as the LSTM cell state. A GRU has three weight blocks to the LSTM's four, so with 1 input and 12 hidden units it has 504 parameters against the LSTM's 672 (and 168 for a plain RNN, as printed above). In practice, the two perform very similarly on most tasks; the GRU is a little faster and a good first choice for small data, and the LSTM is a bit more expressive when sequences are long. (Papers that define z the other way round, with z * h_(t-1), are equivalent: it is only a naming flip.)

Backward pass and a forecasting task

Writing the LSTM backward pass is BPTT with more bookkeeping. At each step, going from the end, with dh and dc arriving from the future:

  1. do = dh * tanh(c_t)
  2. dc_total = dc + dh * o * (1 - tanh(c_t)ยฒ)
  3. df = dc_total * c_(t-1), di = dc_total * g, dg = dc_total * i
  4. Multiply each by its activation derivative (sigmoid: a * (1 - a), tanh: 1 - gยฒ) and stack to get the gradient of the pre-activations.
  5. The weight gradients add up over steps, and the message to the past is dc = dc_total * f and dh from the concatenated input's gradient.

The next example implements this in the LSTM class with a batch dimension, checks it numerically, and trains a small LSTM (12 units, Adam, 250 steps) to forecast a noisy signal made of two sine waves, with periods of 30 and 11 steps. The input is a window of 20 past values; the target is the next value. We compare the test error with the naive forecast "tomorrow equals today".

Python Forecasting a noisy wave with a NumPy LSTM

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both gradient checks come out near 1e-10, so the backward pass is right. The training error falls from 0.61 to about 0.006 within 250 steps. On the held-out last 80 steps, the LSTM reaches a test MSE of about 0.010, against 0.064 for the naive "same as the last value" forecast, roughly six times better. The noise added to the signal has a variance of 0.0025, which is the floor no model can beat, so there is still some room, and the plot shows the dashed prediction hugging the true curve. This is a one-step-ahead forecast. The forecasting project later compares recurrent and feed-forward models over longer horizons.

Bidirectional RNNs and Bi-LSTMs

A recurrent layer reads left to right, so its hidden state at position t knows only what came before. That is exactly right when you must predict the next word or react to a live sensor stream. But many tasks give you the whole sequence up front: tagging each word with its part of speech, finding names in a sentence, labeling the phonemes of a recording, classifying a whole review. In "I saw her duck", the right reading of "duck" depends on what comes after it as much as on what came before.

A bidirectional RNN runs two independent recurrent layers over the same input. One reads forward, the other reads the sequence in reverse. At every position the two hidden states are concatenated, so the output has twice the hidden size:

TEXT
h_fwd[t] = cell_f(x[t], h_fwd[t-1])          reads x[0], x[1], ..., x[t]
h_bwd[t] = cell_b(x[t], h_bwd[t+1])          reads x[T-1], ..., x[t]
output[t] = [h_fwd[t] ; h_bwd[t]]            size 2H

The two directions have separate weights, so the layer has twice the parameters and costs twice the computation. Swap the cell for an LSTM and you have a Bi-LSTM, which was the standard encoder for sequence labeling and for the encoder half of translation systems before transformers took over (a transformer encoder is bidirectional by construction, since every position attends to every other).

Python Why you need the other direction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The task counts the special neighbors of each position: 1 if the previous token is 2, plus 1 if the next token is 5. A forward RNN can see the previous token but never the next one, and a backward RNN is the reverse, so both top out at about 85%, only a little above the 72.3% of always answering 0. Concatenating the two reaches 99.7%. The weights here are random and fixed, and only a linear readout is trained, so this demonstrates what information each direction makes available, not what a trained network would score. The shape printout shows the signature of the layer: (sequences, steps, 2 x hidden).

When not to use one

A bidirectional layer needs the entire sequence before it can produce any output, because the backward pass starts at the end. That rules it out for:

  • Generation and language modeling. When you predict the next token, the future tokens do not exist yet. Reading them during training would also be cheating: the model would simply copy the answer.
  • Streaming and real-time use. You cannot wait for the end of a live audio stream to label its first word.

For classification of a complete sequence, the usual recipe is to take the final forward state and the final backward state (the one that has read the whole sequence back to position 0) and concatenate them, or to pool the per-position outputs.

In frameworks it is a single flag, and the output size changes accordingly:

Python
# Read-only: PyTorch and Keras are not available in the browser runner
import torch.nn as nn
layer = nn.LSTM(input_size=64, hidden_size=128, num_layers=2, batch_first=True, bidirectional=True)
# output: (batch, steps, 2 * 128); the final states h_n have shape (num_layers * 2, batch, 128)

from tensorflow import keras
layer = keras.layers.Bidirectional(keras.layers.LSTM(128, return_sequences=True))

You can also stack several recurrent layers (num_layers=2), feeding the output sequence of one layer into the next. Two or three layers is typical; deeper recurrent stacks are hard to train without residual connections.

When are LSTMs and GRUs still the right tool?

Transformers (see The Attention Mechanism) have replaced LSTMs for most language tasks, because attention trains in parallel across the sequence and reaches any position directly, while a recurrent network must process steps one after another. Gated recurrent networks still have a place:

  • Small data and small models. An LSTM with a few thousand parameters often beats a transformer on short sensor or financial series.
  • Streaming and on-device inference. An RNN keeps a fixed-size state and takes constant time per new sample, while attention over a growing history costs more. Recent state-space and linear-recurrent architectures revive this idea at large scale.
  • Long, cheap context. Memory use does not grow with the sequence length.

In frameworks, you will rarely write these cells yourself: nn.LSTM, nn.GRU or keras.layers.LSTM run optimized kernels. Understanding the gates still matters when you read the hyperparameters (hidden size, number of layers, bidirectional) and when you debug a model that forgets too quickly.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In an LSTM, what does the equation c_t = f * c_(t-1) + i * g express?

    The forget gate scales the old memory, the input gate scales the candidate, and the two are added. That additive path is what lets gradients travel far.
  2. 2. Why is the forget-gate bias often initialized to a positive value such as 1?

    A positive bias gives a forget gate near 0.7 to 1 at the start, which keeps information (and gradient) flowing; the network can still learn to close it.
  3. 3. Which statement about the GRU is true?

    A GRU has an update gate and a reset gate and one state vector. It uses three weight blocks instead of the LSTM's four, so it has fewer parameters.
  4. 4. An LSTM has 1 input and 12 hidden units. Which parameter count matches it, using 4 * ((1 + 12) * 12 + 12)?

    Four blocks (three gates and the candidate), each with (inputs + hidden) x hidden weights and one bias per hidden unit: 4 x 168 = 672.
  5. 5. Why have transformers largely replaced LSTMs for language modeling?

    An LSTM must compute step after step, so training is hard to parallelize, and distant positions interact only through a long chain of states.

Summary

  • An LSTM keeps a cell state updated by c_t = f * c_(t-1) + i * g and exposes h_t = o * tanh(c_t) through an output gate.
  • The additive cell-state path lets gradients travel many steps when forget gates are near 1, which is why LSTMs handle longer dependencies than plain RNNs.
  • Starting the forget-gate bias at 1 or higher is a simple and effective initialization.
  • A GRU merges the cell and hidden state and uses update and reset gates, giving roughly three quarters of the LSTM's parameters with similar accuracy.
  • The LSTM backward pass is BPTT with extra bookkeeping, and a gradient check confirms it.
  • A bidirectional layer runs a forward and a backward RNN and concatenates their states (output size 2H), which helps tagging and classification when the whole sequence is known, but it cannot be used for generation or streaming.
  • Gated RNNs remain useful for small datasets, streaming and constant-memory inference, while transformers dominate large-scale language work.

Next: Embeddings.