Skip to content
Tutorials Center
My learning

Topics

Lesson 70 of 73 ยท Projects

Project: Machine Translation with a Transformer

Build a tiny encoder-decoder transformer with cross-attention and a full backward pass in NumPy, train it to translate a toy language pair, and check it with gradient checking.

Machine translation was the problem that produced the transformer. The 2017 paper "Attention Is All You Need" introduced the architecture for translation, and the same design, in decoder-only form, now powers large language models. Translating a sentence means reading the whole source in one language and then writing the target in another, often in a different word order, so it needs exactly the three abilities this course has been building: representing a sentence (embeddings and an encoder), aligning words across languages (cross-attention), and generating fluent output one token at a time (a decoder).

In this project you build the full encoder-decoder transformer in NumPy, including every line of the backward pass, check the gradients numerically, and train it to translate between two toy languages that differ in vocabulary and in word order. The model is the one-block decoder-only network of Training a Tiny Language Model extended with an encoder and a new kind of attention, cross-attention.

The encoder-decoder transformer

Encoder Self-attention over the whole source sentence.
Decoder Masked self-attention over what it has written so far.
Cross-attention Each target position looks into the encoder output.
1 / 2

The architecture

Each part uses the building blocks from The Transformer Block and Self-Attention and Multi-Head Attention.

  • The encoder embeds the source tokens, adds position embeddings, and applies self-attention (every source position looks at every other) followed by a feed-forward network, each wrapped in a residual connection and layer normalization. Its output is one vector per source token, now aware of its context.
  • The decoder embeds the target tokens written so far (starting with a start token) and runs three sub-layers. First masked self-attention: position t may look only at positions up to t, so training cannot cheat by reading the answer. Second cross-attention: the queries come from the decoder, but the keys and values come from the encoder output, so each target position can gather what it needs from the source. Third a feed-forward network.
  • A final linear layer and softmax turn each decoder position into a probability distribution over the target vocabulary. During training the targets are shifted by one (teacher forcing, as in Seq2Seq, Teacher Forcing and Beam Search), so position t predicts word t.

Cross-attention is the descendant of the attention that rescued the seq2seq model from its fixed-size bottleneck. Nothing is compressed into one vector: the decoder can consult any source position at any step. Compared with the earlier decoder-only model, the new parts are one extra attention function that takes its keys and values from a different sequence, and a second set of embeddings.

Writing the backward pass by hand is the best way to understand it. The attention function has a forward and a backward that you reuse three times (encoder self, decoder self, decoder cross). The cross-attention backward returns two gradients, one for the decoder queries and one for the encoder output, and the second flows back through the whole encoder. As always we verify the result with finite differences (Gradient Checking) on a tiny model before trusting it.

Python An encoder-decoder transformer with a full gradient check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every parameter matrix agrees with the numerical gradient to about 1e-9 relative error (worst 3.4e-9), including the encoder's embeddings and attention matrices, whose gradients travel back through the cross-attention. That check covers all 26 parameter tensors and is the reason to trust the training that follows. The loss at initialization, 2.285, is close to ln(9) = 2.197, the loss of a model that guesses uniformly over the 9 target tokens, as it should be.

The task: a toy language pair

A real translation data set needs megabytes of text. For a model that runs in your browser we invent a small pair of languages that still exercises the important mechanisms. The source has the shape determiner adjective noun verb determiner adjective noun (a old fish likes the red cat), built from 18 words. The target uses 18 different words and puts the adjective after the noun, as Spanish does: determiner noun adjective verb determiner noun adjective (un pez viejo quiere el gato rojo). So the model must translate word by word and reorder. The grammar is deliberately simplified: the toy target ignores gender agreement (un casa is not real Spanish), and each source word has exactly one translation. The space of possible sentences is large (over 10,000), and we train on 1,000 of them and test on others.

Training uses Adam with a short warmup, gradient clipping, a batch of 24 sentences and 100 steps. To translate, the decoder starts from the start token, picks the most probable next word, appends it and repeats (greedy decoding).

Python Train the transformer and translate new sentences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model learns quickly. The teacher-forced accuracy on unseen sentences climbs from 5.5% at the start to 36.2% after 25 steps, 63.6% after 50 and 100.0% after 100, and greedy decoding then translates all 150 test sentences completely correctly, with the adjective reordering included (a old house finds the old tree becomes un casa viejo encuentra el arbol viejo). The final experiment is about generalization. The model is trained again on sentences that never contain the phrase "green fish", and tested on 100 sentences that do. It translates all of them correctly, including a green tree likes a green fish -> un arbol verde quiere un pez verde. It has learned the rules (translate words, swap noun and adjective) rather than memorizing whole sentences, which is the property that matters in real translation. Take care with the interpretation: the toy language is regular and tiny, so 100% is not a claim about real translation, which has ambiguity, idiom, agreement and long-distance dependencies.

From the toy to real translation

What changes at scale:

This project A real system
Data 1,000 invented pairs millions of sentence pairs (WMT, OPUS, Multi30k for small experiments)
Tokens whole words, fixed lengths subwords (BPE or SentencePiece), padding and masks for variable length
Depth 1 encoder and 1 decoder block, single head 6 or more of each, 8 or more heads
Training 100 steps hundreds of thousands of steps, label smoothing, warmup and decay schedule
Decoding greedy beam search with length normalization (see the seq2seq lesson)
Evaluation exact match BLEU, chrF or COMET, plus human evaluation

Important details that a real system adds:

  • Subword tokenization. Word vocabularies cannot cover every word, so sentences are split into subwords, often with one vocabulary shared by both languages (see Hugging Face Transformers Pipelines).
  • Padding masks. Batches contain sentences of different lengths, so padded positions must be masked out of the attention, in both the encoder and the cross-attention.
  • Multiple heads. Each head attends to a different aspect (position, agreement, alignment) in a lower-dimensional space, as described in Self-Attention and Multi-Head Attention.
  • Label smoothing and warmup. Targets of 0.9 instead of 1.0 reduce overconfidence, and the learning rate rises for the first few thousand steps and then decays.
  • Beam search and BLEU. See the seq2seq and image captioning lessons for decoding and for how BLEU works and where it misleads.

In practice, you rarely train a translation model from scratch. Pretrained translation models are a download away. This code is read-only (it needs the transformers and sentencepiece packages and downloads a model):

Python
# Read-only: translation with a pretrained transformer
from transformers import pipeline

translate = pipeline("translation_en_to_de", model="Helsinki-NLP/opus-mt-en-de")
translate("The old cat sees a big dog.")[0]["translation_text"]

# The same idea at the PyTorch level: nn.Transformer wires up the encoder, the decoder and cross-attention for you
import torch.nn as nn
model = nn.Transformer(d_model=256, nhead=8, num_encoder_layers=3, num_decoder_layers=3,
                       dim_feedforward=1024, batch_first=True)
# memory = model.encoder(src_embeddings, src_key_padding_mask=pad_mask)
# out = model.decoder(tgt_embeddings, memory, tgt_mask=causal_mask)   # tgt_mask is the causal mask

To go further with a real data set, start with a small one such as Multi30k (about 30,000 English-German image captions), use a library tokenizer, train a small transformer for a few epochs on a GPU, and evaluate with BLEU on held-out data. Starting from a pretrained model and fine-tuning it on your own domain's sentence pairs is almost always better than training from scratch.

Ideas to extend the project

  • Add gender agreement to the target (el gato, la casa) so the model must pick the determiner from the noun: this needs the decoder to use information from a different source position.
  • Make sentences variable length (optional adjectives) with padding and masks.
  • Add a second attention head and compare training speed.
  • Implement beam search and check whether it changes accuracy on harder, ambiguous data.
  • Remove the encoder's self-attention and see which sentence patterns break.
  • Compare with the earlier Seq2Seq model (an RNN encoder-decoder) on the same task and with longer sentences.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In cross-attention, where do the queries, keys and values come from?

    The decoder asks questions (queries) about the source sentence, and the encoder's output provides the keys and values it looks up.
  2. 2. Why does the decoder's self-attention use a causal mask?

    During training the whole target is available, so without the mask the model could read the answer. The mask makes training match generation.
  3. 3. What did the gradient check in the lesson verify?

    A relative error around 1e-9 shows the analytic gradients are correct, which is essential before trusting training results.
  4. 4. The model translated sentences containing 'green fish' correctly without ever seeing that phrase in training. What does this show?

    Generalization to unseen combinations means the model learned the structure, not just whole sentences. The toy language is regular, so this is easier than real translation.
  5. 5. Why is subword tokenization used in real translation systems?

    Subwords keep the vocabulary a manageable size while letting the model spell out rare words from familiar pieces.

Summary

  • A transformer for translation has an encoder (self-attention over the source), a decoder (masked self-attention over the target so far) and cross-attention that lets each target position read the encoder output.
  • Training uses teacher forcing with shifted targets and a causal mask; generation feeds the model's own tokens back, greedily or with beam search.
  • The backward pass reuses one attention function three times; cross-attention returns gradients to both the decoder and the encoder, and a gradient check (worst error 3.4e-9) confirmed it.
  • On a toy language pair with vocabulary changes and adjective-noun reordering, the model reached 100% sentence accuracy on unseen sentences after 100 steps, and also on sentences containing a phrase never seen in training.
  • Real systems add subword tokens, padding masks, multiple heads, label smoothing, beam search and BLEU-style evaluation, and usually start from a pretrained model.

Next: Practice Exercises.