Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 73 ยท Sequences, Attention and Transformers

Recurrent Neural Networks

Learn how recurrent neural networks keep a hidden state, share weights across time, and predict the next character. Build an RNN cell in NumPy.

A feed-forward network looks at its input once and answers. That is fine for an image or a row of a spreadsheet, but a sentence, a melody or a sensor trace is different: the meaning of the current item depends on what came before it. The word "bank" means one thing after "river" and another after "savings". To handle this, a model needs some form of memory.

A recurrent neural network (RNN) gets memory with one simple trick. It reads the sequence one step at a time and carries a vector, the hidden state, from each step to the next. The hidden state is the network's running summary of everything it has seen so far. In this lesson you will write an RNN cell in NumPy, watch the hidden state evolve, and use it to predict the next character of a sequence that a memory-less model cannot predict. Training the recurrent weights themselves comes in the next lesson; if you have not yet seen how sequences are prepared as arrays, read Working With Sequences first.

An RNN reads one step at a time

Input The current item, such as one character.
Hidden state A vector that summarizes the past.
Update Mix input and old state into a new state.
Output Predict from the state, for example the next character.

The same cell is applied again and again.

1 / 2

The recurrence

At step t, the cell receives the input vector x_t and the previous hidden state h_(t-1). It computes the new state:

h_t = tanh(x_t @ Wxh + h_(t-1) @ Whh + b)

There are two weight matrices. Wxh (input to hidden) has shape (n_in, n_hidden) and decides how the new input affects the state. Whh (hidden to hidden) has shape (n_hidden, n_hidden) and decides how the old state is carried forward. The tanh keeps every unit between -1 and 1 (see Activation Functions). The first state, h_0, is usually all zeros.

Notice that h_t appears on both sides of the idea: today's state is computed from yesterday's state. That feedback loop is what "recurrent" means.

The next example applies the cell to five random input vectors and prints the state after each one. It then runs 1,000 steps with the same cell, and finally feeds the same inputs in reverse order.

Python One RNN cell, applied step by step

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The cell has 32 parameters (12 + 16 + 4), and that number stays 32 whether the sequence has 5 steps or 1,000. This is parameter sharing across time, the same idea that lets a convolution reuse one kernel across an image. Because the weights are shared, a pattern learned at one position is recognized at any other position. The last line prints False: the final state depends on the order of the inputs, which is exactly what a bag of numbers could not give you.

Unrolling through time

The loop above is compact, but it helps to picture it unrolled: the same cell copied once per time step, with the hidden state passed along the chain.

flowchart LR
    h0["h0 (zeros)"] --> C1["cell"]
    x1["x1"] --> C1
    C1 -->|"h1"| C2["cell"]
    x2["x2"] --> C2
    C2 -->|"h2"| C3["cell"]
    x3["x3"] --> C3
    C3 -->|"h3"| O["output layer"]

Unrolled for three steps. All three boxes labeled "cell" are the same weights.

Unrolling turns a recurrent network into a very deep feed-forward network whose layers share weights. A sequence of length 100 becomes a network 100 layers deep. That picture explains both the strength of RNNs (they can use the whole history) and their main difficulty (long chains of multiplications make gradients unstable, the topic of the next lesson).

Outputs: which shape does your task need?

The hidden state at every step is available, so you choose what to read out of it:

  • Many-to-one: read only the last state. Example: classify a whole review as positive or negative.
  • Many-to-many (aligned): read an output at every step. Example: predict the next character after each character, or tag each word with its part of speech.
  • One-to-many: feed one input and generate a sequence from it, feeding each output back as the next input. Example: generating text or music.
  • Many-to-many (encoder-decoder): read the whole input first, then generate a different-length output. Example: translation. This is the setting where attention later made a huge difference.

The readout is usually an ordinary dense layer: logits_t = h_t @ Why + by, followed by softmax for a classification such as "which character comes next?".

Next-character prediction

Character-level language modeling is the classic RNN task: given the characters so far, predict the next one. Consider the repeating sequence abacad abacad .... Look at the letter a: it is followed by b, then c, then d, in turn. A model that sees only the current character cannot know which comes next, because every a looks identical. Only the history (was the character before this a a b, c or d?) disambiguates.

Training a full RNN needs backpropagation through time, which is the next lesson. To see the hidden state's value now, we use a shortcut: keep Wxh and Whh as random, fixed weights (with Whh rescaled so the state does not explode) and train only the final softmax layer on top of the hidden states. A random recurrent layer is a surprisingly good memory. This setup is known as a reservoir or echo state network, and it makes a clean experiment: one classifier sees only the current character, the other sees the RNN's hidden state.

Python Memory makes the next character predictable

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The memory-less model tops out at 66.6%: it gets every b, c and d right (each is always followed by a), and for the a positions it can only guess one of the three letters, so it is right a third of the time. The hidden-state model reaches 100.0%, because the state encodes which character came before the current a. The heatmap shows the state changing from step to step, and because the sequence repeats every six characters, the pattern of colors repeats too.

Only the readout was trained here, and the recurrent weights stayed random. Real RNNs learn Wxh and Whh too, using the gradient flow you will build in the next lesson. The reservoir trick is still used for some cheap time-series tasks.

How long does the memory last?

A hidden state can only remember as much as its recurrence allows. How the old state is carried forward depends mostly on the size of the weights in Whh. The last experiment sends a single event into the network (an input of +1 versus -1 at step 0, then silence) and measures how different the two hidden-state trajectories remain over time, for three scales of Whh.

Python How long a one-time input stays visible

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With scale 0.5 the difference between the two histories shrinks to almost nothing within ten steps: the network forgets quickly. With 0.9 it fades slowly, so some trace remains after 29 steps. With 1.5 the difference never fades at all; the state keeps amplifying small differences (it saturates against the tanh limits), which means the memory is there but is also chaotic and hard to learn from. Forgetting too fast or being unstable are two faces of the same issue, and the next lesson shows what it does to training.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the hidden state of an RNN represent?

    The hidden state is a vector the cell recomputes at every step from the new input and the previous state. It carries information forward through time.
  2. 2. A simple RNN cell has 32 parameters. How many parameters does it have when processing a sequence of 10,000 steps?

    The same weights are reused at every time step (parameter sharing), so the parameter count is independent of the sequence length.
  3. 3. Why can a model that sees only the current character not predict the next one in 'abacad abacad'?

    After a 'b', 'c' or 'd' the next letter is always 'a', but after 'a' it varies. Only the earlier history tells which letter comes next.
  4. 4. What is unrolling an RNN?

    An unrolled RNN is a chain of identical cells, one per step. It is the view used to derive backpropagation through time.

Summary

  • An RNN processes a sequence step by step and carries a hidden state h_t = tanh(x_t @ Wxh + h_(t-1) @ Whh + b) as its memory.
  • The same weights are used at every step, so the parameter count does not depend on the sequence length.
  • The result depends on the order of the inputs; unrolling shows an RNN as a deep network with shared layers.
  • You can read outputs at the last step, at every step, or feed outputs back to generate sequences.
  • Next-character prediction needs memory when the current character alone is ambiguous; a random fixed recurrent layer with a trained readout already solves the toy case.
  • How long information survives depends on the size of the recurrent weights, which sets up the gradient problems in the next lesson.

Next: Backpropagation Through Time.