Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 45 ยท Sequences and Transformers

RNNs, LSTMs and GRUs

Understand recurrent networks in PyTorch: hidden state, nn.RNN, nn.LSTM, nn.GRU, batch_first, padding and packing, and why gradients vanish, with live NumPy stand-ins.

Images have a fixed grid. Sentences, audio, sensor readings and stock prices are sequences: the length varies and the order carries meaning. "Dog bites man" and "man bites dog" contain the same words. A recurrent neural network (RNN) handles this by reading one element at a time and carrying a small memory, the hidden state, from each step to the next. The same weights are reused at every step, just as a convolution reuses its kernel at every position.

In this lesson you will see exactly what an RNN computes at each step, how PyTorch packages it as nn.RNN, nn.LSTM and nn.GRU, what the tensor shapes mean, how to deal with sequences of different lengths, and why plain RNNs struggle with long-range memory. PyTorch cannot run on this site, so the PyTorch code is read-only. The live editors implement an RNN cell, an LSTM cell and a gradient experiment in NumPy. They are stand-ins for the library, built from the same equations.

A recurrent network in one picture

Input x_t The element at step t.
Hidden state h_t A summary of everything seen so far.
Same weights Reused at every time step.
Output One value per step, or only the last.

New state = function of the new input and the old state.

1 / 2

The hidden state

A basic ("Elman") RNN updates its state with one formula:

h_t = tanh(W_x ยท x_t + W_h ยท h_(t-1) + b)

The state h_t is a vector of hidden_size numbers. It starts as zeros (h_0), and at every step it mixes the new input x_t with the previous state. Because h_t depends on h_(t-1), which depends on h_(t-2), and so on, the final state is a function of the entire sequence. To make a prediction you feed h_t (usually the last one) into a Linear layer.

The first editor runs this formula by hand over a short sequence and prints the state after each step. Change the input values and watch how the final state changes: that is the "memory" of the network.

Python An RNN cell, step by step (NumPy stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The parameter count does not depend on the sequence length: whether the input has 4 steps or 4,000, the cell has the same weights.

nn.RNN, nn.LSTM and nn.GRU in PyTorch

PyTorch runs the whole loop over time for you, in optimized code:

Python
import torch
from torch import nn

rnn = nn.RNN(input_size=8, hidden_size=16, num_layers=1, batch_first=True)

x = torch.randn(4, 10, 8)        # (batch, seq_len, features) because batch_first=True
output, h_n = rnn(x)

print(output.shape)              # torch.Size([4, 10, 16])  the state at EVERY step
print(h_n.shape)                 # torch.Size([1, 4, 16])   the final state of each layer

Two shape facts cause most beginner confusion:

  • batch_first: by default PyTorch's recurrent layers expect input shaped (seq_len, batch, features). With batch_first=True it becomes (batch, seq_len, features), which matches how data loaders usually produce batches. The output follows the same convention, but h_n is always (num_layers * num_directions, batch, hidden_size), whatever batch_first says.
  • output versus h_n: output holds the top layer's state at every time step; h_n holds the final state of every layer. For a single-layer, one-directional network, output[:, -1, :] equals h_n[0].

nn.GRU has the same interface. nn.LSTM carries a second state, the cell state, so it takes and returns a pair:

Python
lstm = nn.LSTM(input_size=8, hidden_size=16, num_layers=2, batch_first=True,
               dropout=0.2, bidirectional=True)
output, (h_n, c_n) = lstm(x)

print(output.shape)   # torch.Size([4, 10, 32])  16 forward + 16 backward features
print(h_n.shape)      # torch.Size([4, 4, 16])   2 layers * 2 directions = 4
print(c_n.shape)      # torch.Size([4, 4, 16])

num_layers stacks recurrent layers (each layer's outputs feed the next), dropout is applied between stacked layers (not after the last), and bidirectional=True runs a second RNN backwards over the sequence and concatenates the two, which is useful when the whole sequence is known in advance, as in classification. It is not usable for generating text one token at a time, since the future is not available.

A typical sequence classifier uses the last hidden state:

Python
class SequenceClassifier(nn.Module):
    def __init__(self, n_features, hidden, n_classes):
        super().__init__()
        self.lstm = nn.LSTM(n_features, hidden, batch_first=True)
        self.head = nn.Linear(hidden, n_classes)

    def forward(self, x):
        _, (h_n, _) = self.lstm(x)
        return self.head(h_n[-1])        # last layer's final state: (batch, hidden)

Why plain RNNs forget: vanishing gradients

Training an RNN uses backpropagation through time: the loss gradient flows backwards through every step. Look at the derivative of h_T with respect to an early state h_t. By the chain rule it is a product of T - t Jacobian matrices, each roughly the recurrent weights W_h scaled by tanh' (a number below 1). Multiply many matrices whose largest effective scale is below 1 and the result shrinks exponentially: gradients vanish, and the network cannot learn that an early input mattered. If the scale is above 1 they explode instead, which is why gradient clipping (see Learning Rate Schedules and Gradient Clipping) is standard for recurrent models.

The second editor measures this. It computes the gradient norm of the last state with respect to each earlier state for a few recurrent weight scales.

Python Vanishing and exploding gradients through time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a recurrent scale of 0.5 or 1.0 the gradient reaching step 1 is astronomically small (about 1e-28 and 1e-13 in the run above): an input 40 steps ago has no say in the loss. Only with a large scale (3.0 here) does the signal survive the whole way, and then gradient sizes above 1 show that growth is possible too. The scale a real network sits at depends on initialization and training, and staying in the narrow useful range is fragile. That fragility is the problem the gated cells solve.

LSTM and GRU: gates to the rescue

The LSTM (long short-term memory) adds a second path, the cell state c_t, that is updated mostly by addition instead of repeated matrix multiplication and squashing. Three gates, each a sigmoid between 0 and 1, control it:

  • the forget gate f decides how much of the old cell state to keep;
  • the input gate i decides how much new information to write;
  • the output gate o decides how much of the cell to expose as h_t.

c_t = f * c_(t-1) + i * g and h_t = o * tanh(c_t), where g = tanh(W_x x_t + W_h h_(t-1) + b) is the candidate update. When f is close to 1, the cell state, and its gradient, can flow through many steps nearly unchanged. A common trick is initializing the forget gate's bias to 1 so the network starts out remembering.

The GRU merges the cell and hidden states and uses two gates (update and reset). It has fewer parameters, trains a little faster, and is often as accurate as an LSTM. Choosing between them is an empirical decision: try both on your data.

The next editor implements one LSTM step and shows the gates reacting to different inputs.

Python One LSTM step with its gates (NumPy stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The count of four weight blocks is also why nn.LSTM(input, hidden) has about four times the parameters of nn.RNN(input, hidden) (PyTorch stores two bias vectors per block, so its exact counts are slightly larger than in the stand-in). A GRU has three blocks.

Padding, packing and variable lengths

Real batches contain sequences of different lengths, but a tensor must be rectangular. The usual approach is to pad short sequences with a filler value up to the longest one in the batch. The recurrent layer, however, would happily keep processing the padding and mix meaningless zeros into the state. PyTorch's remedy is packing:

Python
from torch.nn.utils.rnn import pad_sequence, pack_padded_sequence, pad_packed_sequence

seqs = [torch.randn(5, 8), torch.randn(3, 8), torch.randn(7, 8)]
lengths = torch.tensor([len(s) for s in seqs])

padded = pad_sequence(seqs, batch_first=True)              # (3, 7, 8), zero padded
packed = pack_padded_sequence(padded, lengths, batch_first=True, enforce_sorted=False)

packed_out, (h_n, c_n) = lstm_layer(packed)                # padding is skipped entirely
out, out_lengths = pad_packed_sequence(packed_out, batch_first=True)

pack_padded_sequence tells the layer each sequence's true length, so h_n is the state at each sequence's last real element, not after the padding. lengths must be a CPU tensor, and enforce_sorted=False lets you pass sequences in any order. A collate_fn in the DataLoader is the natural place to pad (see Datasets and DataLoaders). An alternative is to leave the padding in, keep a mask, and ignore padded positions when computing the loss or pooling.

Bidirectional networks and input-output patterns

A normal RNN reads left to right, so the state at position t knows only the past. For many tasks the future matters just as much: in "He said he would bank on it" the word after "bank" tells you its meaning. A bidirectional RNN (BiRNN, BiLSTM, BiGRU) runs two independent recurrent layers, one forward and one over the reversed sequence, and concatenates their states at every position. The output width becomes 2 * hidden_size, which is why the earlier example printed 32 features.

Advantages: each position sees both its left and right context, which improves tagging (named entities, part of speech), speech recognition and classification; the backward pass also gives the end of the sentence a shorter path to the start. Challenges: the whole sequence must be available, so bidirectional layers cannot generate text token by token or process a live stream; compute and parameters double; and the two directions cannot be run in a single sequential pass. In PyTorch it is one flag, bidirectional=True; for classification you combine the last forward state and the first backward state:

Python
lstm = nn.LSTM(8, 16, batch_first=True, bidirectional=True)
output, (h_n, c_n) = lstm(x)                  # h_n: (2, batch, 16): [forward, backward]
features = torch.cat([h_n[0], h_n[1]], dim=1) # (batch, 32): the summary of both directions
Python Forward and backward states see different contexts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The forward state at position 2 cannot know about the last element, but the backward state can. That is the whole point of reading in both directions.

One-to-one, one-to-many, many-to-one, many-to-many

Recurrent models are described by how many inputs map to how many outputs:

Pattern Example How it is built
One-to-one Plain classification No recurrence needed; an MLP
One-to-many Image captioning: one image, many words Feed the image once, then generate token by token
Many-to-one Sentiment of a review Run over the sequence, classify the final state (h_n)
Many-to-many, equal length Tagging each word with a label Classify every step of output
Many-to-many, unequal length Translation, summarization Encoder-decoder (seq2seq)

For training, the network is unrolled through time: the same cell is copied once per step and gradients flow back through all copies (backpropagation through time, BPTT), which is why long sequences are expensive and why truncating them helps.

Where LSTMs are used, and their limits

LSTMs and GRUs are used for time-series forecasting (energy demand, sensors, finance), speech and handwriting recognition, music, predictive text and anomaly detection in logs or signals. Their weaknesses: they process steps one after another, so training does not parallelize across time like attention does; very long dependencies are still hard; and they need careful clipping and tuning. These limits are why transformers displaced them on large language tasks.

Using recurrent layers well

  • Use gradient clipping (clip_grad_norm_) with recurrent models; it is cheap insurance against exploding gradients.
  • For very long sequences, train on shorter windows and detach the carried hidden state between windows (h = h.detach()), or backpropagation will try to reach into freed graphs, as explained in Autograd in Practice.
  • Today, attention-based models (see Attention from Scratch) have replaced RNNs for most text tasks, but LSTMs and GRUs remain useful for small data, streaming inputs and time series, and they teach the ideas of state and gating.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. You create nn.LSTM(10, 20, batch_first=True) and feed an input of shape (5, 12, 10). What is the shape of output?

    With batch_first=True the output is (batch, seq_len, hidden_size) = (5, 12, 20). The final states h_n and c_n would be (1, 5, 20).
  2. 2. What is the shape convention of h_n for nn.GRU with batch_first=True?

    batch_first only affects the input and output tensors. The final hidden state always has the layer dimension first.
  3. 3. Why do plain RNNs have trouble learning long-range dependencies?

    Backpropagation through time multiplies many Jacobians. The weights are the same at all steps, so the product tends to vanish or explode.
  4. 4. What does an LSTM's forget gate with a value close to 1 do?

    c_t = f * c_(t-1) + i * g. When f is near 1, the old cell state passes through almost unchanged, which lets information and gradients travel across many steps.
  5. 5. Why use pack_padded_sequence for a batch of sequences with different lengths?

    Packing passes the true lengths to the layer, so padding does not contaminate the hidden state.

Summary

  • An RNN reads a sequence step by step, updating a hidden state with the same weights each time: h_t = tanh(W_x x_t + W_h h_(t-1) + b).
  • nn.RNN, nn.GRU and nn.LSTM return output (the top layer's state at every step) and the final state(s) h_n (and c_n for LSTM); use batch_first=True for (batch, seq, features) input, while h_n stays layer-first.
  • Gradients through many time steps vanish or explode; LSTM and GRU gates give information and gradients a more direct path.
  • Pad variable-length sequences and pack them so the layer ignores the padding, or mask padded positions yourself.
  • Clip gradients, detach carried states, and consider attention models for long-range text tasks.

Next, turn words into vectors in Embeddings and Text Data.