Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 73 ยท Sequences, Attention and Transformers

Seq2Seq, Teacher Forcing and Beam Search

Build an encoder-decoder RNN in NumPy, see why teacher forcing speeds up training but causes exposure bias, and compare greedy decoding with beam search.

Most of the networks so far map one input to one output: an image to a label, a window of numbers to the next number. Many real tasks map one sequence to another sequence of a different length: a French sentence to an English one, a long article to a summary, a question to an answer, spoken audio to text. The input length does not match the output length, and the output must be produced one piece at a time.

The classic answer is the sequence-to-sequence (seq2seq) model, also called an encoder-decoder. One recurrent network, the encoder, reads the input and compresses it into a vector. A second recurrent network, the decoder, starts from that vector and writes the output one token at a time. Two techniques decide how well this works in practice. Teacher forcing decides how the decoder is trained, and beam search decides how it is used. This lesson builds a small seq2seq model in NumPy that learns to reverse a string of digits, and then examines both techniques.

Encoder, summary, decoder

Encoder Reads the input one token at a time.
Context vector The encoder's last state summarizes the input.
Decoder Writes the output, one token per step.

The decoder stops when it emits an end-of-sequence token.

1 / 2

The encoder-decoder idea

Both halves are ordinary recurrent networks of the kind you built in Recurrent Networks. The encoder updates a hidden state for each input token and throws away every output except the final state. That final state is the context vector. The decoder starts with it as its own initial state, receives a special start token (<s> or SOS), and predicts the first output token. Each predicted token becomes the next input, and the loop ends when the model predicts an end token (</s> or EOS).

flowchart LR
    x1["x1"] --> E1["encoder"] --> E2["encoder"] --> E3["encoder"]
    x2["x2"] --> E2
    x3["x3"] --> E3
    E3 -- "context vector" --> D1["decoder"]
    S["SOS"] --> D1 --> y1["y1"]
    y1 --> D2["decoder"] --> y2["y2"]
    y2 --> D3["decoder"] --> E["EOS"]

The decoder is a language model on the output side, conditioned on the encoder's summary.

Because the input and output are separate sequences, they can have different lengths and even different vocabularies. Translation, summarization, speech recognition and code generation all fit this shape.

Teacher forcing

At training time you know the correct answer. The question is what to feed the decoder at step t. There are two choices:

  • Free running: feed back the token the decoder predicted at step t - 1. This is what happens when the model is used for real. But early in training the predictions are mostly wrong, so the decoder receives garbage inputs, its learning signal is noisy, and training crawls.
  • Teacher forcing: feed the true token from the training example, no matter what the decoder predicted. Every step becomes an independent, well-posed question: "given the correct prefix, what comes next?" The loss at all steps can be computed at once, and training is stable and fast.

Teacher forcing is the default for training nearly every sequence model, including the transformer block and large language models. It has one weakness, called exposure bias. During training the decoder never sees its own mistakes. At test time one wrong token puts it in a situation it has never practiced, and the error can snowball. The next example trains the model with teacher forcing and then measures the same network in both modes. The input is a random string of 4 digits, and the correct output is the digits in reverse order followed by the end token.

Python A seq2seq model that reverses digits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two columns tell the story. After only 10 updates, the decoder predicts 69.2% of tokens correctly when it is handed the true previous tokens, yet only 8.0% of complete answers are right when it must feed on its own guesses: a single wrong digit spoils the whole line, and one early error also feeds a wrong token into every later step. After 20 updates the numbers are 91.6% and 69.3%. As training continues the gap closes (99.8% and 99.3% at step 400), because the decoder stops making the first mistake. The sequence-level score is always the harsher one, and it is the one users actually experience.

Fixes for exposure bias include scheduled sampling (during training, randomly feed the model's own prediction instead of the true token, with a probability that grows over time), training with sequence-level rewards (reinforcement learning, see Neural Networks in Reinforcement Learning), and simply using more data and a larger model. For many tasks plain teacher forcing is good enough.

The bottleneck problem

Notice what the decoder can see: only the context vector, a single list of numbers of fixed size, whatever the length of the input. Squeezing a 5-token string into 64 numbers is easy. Squeezing a 60-word sentence is not, and quality drops quickly as inputs get longer. That limitation motivated attention, which lets the decoder look back at every encoder state at every step instead of relying on one summary, and ultimately the transformer.

At test time the decoder outputs a probability for each possible next token. To turn that into a sentence you must choose. The simplest rule is greedy decoding: always take the most probable token. It is fast, but a locally best choice can lead to a poor sentence. If a likely first word is followed by only weak continuations, a slightly less likely first word might have led to a far better whole sentence.

Beam search keeps the best k partial sentences (the beam width) alive at each step:

  1. Extend every hypothesis with every possible next token, adding the log-probabilities.
  2. Keep only the k highest-scoring extensions.
  3. A hypothesis that ends in the end token is moved to the finished list.
  4. Stop when enough hypotheses have finished, and return the best.

Beam width 1 is greedy decoding. Larger widths search more of the space at proportionally higher cost, and with diminishing returns. Beam search does not guarantee the best possible sentence, only a better one than greedy. The example uses a hand-written next-word table so that you can see the difference with exact numbers.

Python Greedy decoding versus beam search

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Greedy picks "the" because 0.5 beats 0.4, and then has to choose between three weak continuations, ending at probability 0.170. Beam search with width 2 keeps both "the" and "a" alive, discovers that "a cat" is 0.9 likely once "a" is chosen, and ends at 0.360, more than twice as probable. The last lines show a subtle trap: because every extra token multiplies in a probability below 1, long outputs always score lower than short ones. The usual remedy is length normalization, dividing the log-probability by the length (raised to a power), so the search does not prefer terse answers.

For open-ended text such as stories or chat, beam search tends to produce dull, repetitive output, and people instead sample from the distribution, often with a temperature or a top-k/top-p filter (see How Large Language Models Work). Beam search is the right tool when there is one correct answer: translation, transcription, code or a structured output.

In a framework you write much less, because the library supplies the recurrent layers and the decoding helpers:

Python
# Read-only: PyTorch is not available in the browser runner
import torch.nn as nn

class Seq2Seq(nn.Module):
    def __init__(self, vocab, hidden=128):
        super().__init__()
        self.embed = nn.Embedding(vocab, 64)
        self.encoder = nn.GRU(64, hidden, batch_first=True)
        self.decoder = nn.GRU(64, hidden, batch_first=True)
        self.out = nn.Linear(hidden, vocab)

    def forward(self, src, tgt_in):                      # tgt_in = [SOS, y1, y2, ...]: teacher forcing
        _, context = self.encoder(self.embed(src))       # context: the encoder's final state
        states, _ = self.decoder(self.embed(tgt_in), context)
        return self.out(states)                          # logits for every step at once

Everything in the NumPy example is real and was run; the PyTorch class is only for orientation.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does teacher forcing feed the decoder at step t during training?

    With teacher forcing the decoder always receives the true previous token, so every step is a clean next-token prediction problem and the loss for all steps can be computed together.
  2. 2. What is exposure bias?

    Training feeds true prefixes, but inference feeds the model's own outputs. The mismatch means early errors put the decoder in situations it has not practiced.
  3. 3. Why does a plain encoder-decoder struggle with long inputs?

    Every detail of the input has to pass through one fixed-size vector. Attention removes that bottleneck by letting the decoder look at all encoder states.
  4. 4. In the grid of next-word probabilities, greedy decoding ended at probability 0.170 while beam search found 0.360. Why?

    Greedy commits to the locally best first word. Beam search keeps several candidates and so can find a better overall sequence.
  5. 5. Why is length normalization used with beam search?

    Each additional token multiplies the sequence probability by a number below 1, so without a correction the search drifts toward very short answers.

Summary

  • A seq2seq model has an encoder RNN that summarizes the input into a context vector and a decoder RNN that generates the output token by token, starting from a start token and stopping at an end token.
  • Teacher forcing feeds the decoder the true previous token during training, which makes training fast and stable.
  • The price is exposure bias: at test time the decoder consumes its own outputs, so sequence-level accuracy is lower than token-level accuracy under teacher forcing, especially early in training.
  • A fixed-size context vector is a bottleneck for long inputs, which led to attention.
  • Greedy decoding takes the best token at each step; beam search keeps the best k partial sequences and usually finds a more probable output. Length normalization stops it from preferring short answers.
  • For open-ended generation, sampling is more common than beam search.

Next: The Attention Mechanism.