Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 40 ยท Sequences and Text

LSTM and GRU Layers

Understand LSTM and GRU gates, build a gated cell in NumPy, count parameters, and use LSTM, GRU, Bidirectional and stacked recurrent layers in Keras models.

A plain RNN squeezes everything it has seen into one hidden vector that is rewritten at every step. Over long sequences it forgets. LSTM (Long Short-Term Memory) and GRU (Gated Recurrent Unit) cells add gates: small learned switches, values between 0 and 1, that decide what to keep, what to write and what to reveal. Because the memory can pass through many steps almost unchanged, gradients survive much longer and the network can learn dependencies across dozens or hundreds of steps.

Keras and TensorFlow are not available in the browser runner. The NumPy blocks compute real gate math; Keras code is read-only.

What a gate does

A gate is a sigmoid layer: g = sigmoid(x Wx + h Wh + b). Its output has one value per hidden unit in the range 0 to 1. Multiplying a vector by a gate keeps the parts where the gate is near 1 and erases the parts near 0.

Python A gate in action

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The LSTM cell

An LSTM keeps two vectors: the cell state c (long-term memory) and the hidden state h (the output). Three gates and a candidate value control them:

  • Forget gate f: how much of the old cell state to keep.
  • Input gate i: how much of the new candidate g to write.
  • Output gate o: how much of the cell state to expose as h.

c_t = f * c_(t-1) + i * g and h_t = o * tanh(c_t)

The cell state is updated by addition and multiplication by a gate, not by being squashed through tanh at every step. That "memory highway" is what protects gradients.

Python One LSTM step in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

An LSTM has four times the parameters of a SimpleRNN of the same size: 4 weight blocks instead of 1. LSTM(16) on 8 features: 4 * (8*16 + 16*16 + 16) = 1600, which model.summary() confirms.

The GRU

A GRU merges the cell and hidden states and uses two gates: an update gate z (how much to replace the state) and a reset gate r (how much of the past to use when proposing a new state). It has about three quarters of an LSTM's parameters, trains a bit faster and often performs about the same.

Python One GRU step in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rule of thumb: try a GRU first when compute matters, an LSTM when sequences are long or you have lots of data. Often the difference is small, so measure on your own validation set.

Using them in Keras

Python
import keras
from keras import layers

model = keras.Sequential([
    keras.Input(shape=(None,), dtype="int32"),
    layers.Embedding(20000, 64, mask_zero=True),
    layers.Bidirectional(layers.LSTM(64, return_sequences=True)),
    layers.Bidirectional(layers.LSTM(32)),
    layers.Dropout(0.3),
    layers.Dense(1, activation="sigmoid"),
])
model.compile("adam", "binary_crossentropy", metrics=["accuracy"])
model.summary()
  • Bidirectional runs one recurrent layer forward and another backward over the sequence and concatenates the results, so each position sees both left and right context. Output size doubles (2 * units). Use it when the whole sequence is available (text classification), not for real-time forecasting.
  • Stacking recurrent layers needs return_sequences=True on every layer except the last.
  • dropout= and recurrent_dropout= regularize inputs and states. (recurrent_dropout disables the fastest GPU kernel, so prefer a Dropout layer between layers.)
Python What Bidirectional does to shapes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For the last-state output the backward layer's "final" state is the one reached after reading back to position 0, which is why the demo takes backward[:, 0].

Which one should I use?

flowchart TD
    A["Sequence task"] --> B{"Short sequences,<br/>small data?"}
    B -->|yes| C["GRU"]
    B -->|no| D{"Need long-range<br/>memory?"}
    D -->|yes| E["LSTM or Transformer"]
    D -->|no| C
    E --> F{"Whole sequence<br/>available?"}
    F -->|yes| G["Add Bidirectional"]
    F -->|no| H["Keep it one-directional"]

A practical decision flow. For very long text, attention usually beats both.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the forget gate of an LSTM control?

    The forget gate multiplies the previous cell state, so values near 0 erase memory and near 1 keep it.
  2. 2. Compared with an LSTM of the same size, a GRU has:

    A GRU uses an update and a reset gate and one state vector, about three quarters of the LSTM's weights.
  3. 3. A Bidirectional(LSTM(32)) layer returns a last-state output of size:

    Forward and backward outputs of 32 units each are concatenated.
  4. 4. When stacking two recurrent layers, the first needs:

    The second layer needs a full sequence as input, not only the last state.

Summary

  • Gates are sigmoid outputs between 0 and 1 that scale vectors: keep, write or reveal information.
  • An LSTM has a cell state plus forget, input and output gates; a GRU has update and reset gates and fewer weights.
  • The additive cell-state path lets gradients survive long sequences.
  • Bidirectional doubles the output size and needs the whole sequence; stacking needs return_sequences=True.
  • Prefer GRU for speed, LSTM for long memory, and compare on validation data.

Next: Attention and Transformers