Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 37 Β· Transformer Internals

Multi-Head Attention and Masking

Split attention into several heads, combine them with an output projection, and apply a causal mask so a decoder never sees future tokens. NumPy code included.

A single attention pattern per layer is limiting: a token may need to look at the previous word for grammar, at a distant noun for meaning, and at a delimiter for structure, all at once. Multi-head attention runs several smaller attentions in parallel, each with its own projections, and then combines the results. A second ingredient, the causal mask, makes the same machinery usable for generating text left to right.

Multi-head attention

Suppose the model width is d_model and we want h heads. Each head works in a smaller space of size d_head = d_model / h. For every head there are separate Wq, Wk, Wv matrices, so each head learns its own notion of "what to look for". The head outputs are concatenated back to width d_model and passed through an output matrix Wo that lets the heads mix.

In practice the projections for all heads are one big matrix multiply, followed by a reshape that splits the last dimension into (heads, d_head).

Python Multi-head attention in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The total cost of h heads of size d_model/h is about the same as one full-size head; you get several different attention patterns for free. The block has 4 Β· d_modelΒ² parameters: four square matrices.

What different heads learn

Studies of trained models find heads with distinct jobs: some attend mostly to the previous token, some to the first token (acting like a sink), some link a closing bracket to its opener, and some copy earlier tokens in repeated text ("induction heads", which are central to in-context learning). Many heads are redundant and can be pruned. The design simply gives the network the freedom to specialize.

The causal mask

A decoder trains on a whole sequence at once, predicting the token at position i+1 from the tokens up to i. If position i could attend to position i+1, it could simply copy the answer. The fix is a causal mask: before softmax, scores for any key position later than the query position are set to -∞, so their weights become exactly 0.

Python Apply a causal mask

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The lower-triangular weight matrix means token 0 can only attend to itself, token 1 to tokens 0 and 1, and so on. Every row still sums to 1 because softmax renormalizes over the allowed positions.

Proving no information leaks

A good test of a masked implementation is that changing a future token must not change the output at earlier positions. You can verify it directly.

Python Changing the future leaves the past unchanged

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Positions 0 to 3 are unchanged; positions 4 and 5 (whose own inputs changed) differ. This property is what allows training on all positions in parallel: one forward pass yields n valid next-token predictions.

Padding masks and combined masks

A second mask handles batches. When sequences in a batch have different lengths, padded positions must be hidden from attention. The padding mask and the causal mask are simply combined: a position is blocked if either mask blocks it. Encoder models use only the padding mask. Encoder-decoder models add cross-attention, in which queries come from the decoder and keys and values come from the encoder output, so the decoder can read the source sentence.

Python Combine padding and causal masks

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Heads and masks

Several heads Parallel attentions with their own projections.
Concatenate Join the head outputs, then multiply by Wo.
1 / 2

Check your understanding

Quick check Β· 4 questions

of 4 answered You scored / 4 Β· perfect!

  1. 1. If d_model is 512 and there are 8 heads, what is d_head?

    d_head = d_model / h = 512 / 8 = 64.
  2. 2. Why does a decoder need a causal mask during training?

    Without it the model could read the answer from later positions.
  3. 3. How is masking implemented before softmax?

    exp(-inf) = 0, so the masked weights vanish and the rest renormalize.
  4. 4. What is the role of the output matrix Wo?

    After concatenating the heads, Wo lets information from different heads combine.

Summary

  • Multi-head attention splits the width into several heads that attend in parallel, then concatenates and mixes them with Wo.
  • Attention blocks have about 4Β·d_modelΒ² parameters.
  • Heads specialize in patterns such as previous-token and induction behavior.
  • A causal mask sets future scores to -inf, enabling parallel next-token training.
  • Padding and causal masks combine; cross-attention connects a decoder to an encoder.

Next: Feed-Forward Layers, Residuals and LayerNorm