Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 73 Β· Sequences, Attention and Transformers

Self-Attention and Multi-Head Attention

Build self-attention and multi-head attention in NumPy, learn why order needs positional encodings (sinusoidal, learned, rotary), and visualize heads.

In the previous lesson the queries came from one sequence (the English words being written) and the keys and values from another (the French sentence). Self-attention applies the same machinery to a single sequence: every token asks questions about every other token in the same sequence and collects information from them. In "the animal did not cross the street because it was tired", the word "it" can look back and pull in information from "animal". After a self-attention layer, each token's vector is no longer just the token, but the token in context.

Two ingredients make this work in practice. First, self-attention by itself has no sense of word order, so we must inject position information. Second, one attention pattern per layer is too restrictive, so transformers run several in parallel as multi-head attention. This lesson covers both and ends with a look inside the heads.

Self-attention in one sentence

Same source Queries, keys and values all come from the same tokens.
Three projections Learned matrices Wq, Wk and Wv create the three views.
Context Each output mixes information from the whole sequence.
1 / 2

Self-attention step by step

Take a sequence of T tokens, each represented by a vector of size d_model (stacked into a matrix X of shape (T, d_model), usually from an embedding layer). Three learned weight matrices produce the three roles:

  • Q = X @ Wq, K = X @ Wk, V = X @ Wv, each of shape (T, d_k).

Then apply the attention formula from the last lesson. The score matrix has shape (T, T): row i holds how much token i attends to each token j. The output is (T, d_k): one context-aware vector per token.

Using one matrix X to make all of Q, K and V is the only difference from the previous lesson, and it has one striking consequence, shown in the next example. We build the sentences "man bites dog" and "dog bites man" from the same three word vectors and compare the outputs.

Python Self-attention ignores word order

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two sentences mean opposite things, but the vector produced for "man" is exactly the same in both (identical? True). Self-attention treats its input as a set: shuffle the tokens and the outputs are shuffled in exactly the same way (a property called permutation equivariance). That is a problem for language, and the fix is to give every token a position signal before the attention layers.

Positional encodings

The simplest repair is to add a position vector to each token embedding: x_i = embedding(token_i) + pos(i). There are three popular designs.

Sinusoidal encodings (the original Transformer). Position p gets a fixed vector whose entries are sines and cosines of different frequencies: pos(p)[2i] = sin(p / 10000^(2i/d)) and pos(p)[2i+1] = cos(p / 10000^(2i/d)). Low dimensions oscillate quickly and high dimensions slowly, like the digits of a number in a mixed base. Nothing is learned, so it can in principle be computed for any length. One useful property: the dot product between the encodings of positions p and q depends only on the offset p - q, so "three words apart" looks the same everywhere.

Learned encodings. Treat positions as ordinary tokens: a table of shape (max_len, d_model), one trainable row per position, like an embedding table. This is what BERT and GPT-2 use. It is flexible and simple, but it cannot represent positions beyond max_len.

Rotary encodings (RoPE). Instead of adding a vector, rotate the query and key by an angle proportional to their position. Each pair of dimensions (x_2i, x_2i+1) is rotated in its own plane by angle p * theta_i. When the rotated query at position m is multiplied with the rotated key at position n, the rotations combine into one that depends only on m - n. So the attention score encodes relative distance directly. RoPE is used by many recent open language models.

The following example builds sinusoidal encodings, plots them, checks the offset property, and verifies the rotary property numerically.

Python Sinusoidal and rotary position signals

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The encodings of positions 0 and 1 start [0, 1, 0, 1, ...] and [0.84, 0.54, 0.53, 0.85, ...]: the lowest dimensions change quickly from one position to the next. The three dot products for an offset of 3 are all 12.27, whatever the absolute position, which is the relative-position property. The score of a position with itself is 16.0, because each of the 16 sine-cosine pairs contributes sinΒ² + cosΒ² = 1. It falls as the distance grows (15.3 at distance 1, 12.3 at 3, 10.1 at 10), though not perfectly smoothly, since oscillations at different frequencies interfere. The rotary scores for offset 3 are identical to four decimals (2.7754) at positions (5, 2), (15, 12) and (103, 100), while offset 5 gives a different score. The picture on the left shows the typical striped pattern.

Positional encodings are added once, to the token embeddings at the bottom of the network (sinusoidal and learned) or applied inside every attention layer (rotary). Without them, the model behaves like the "bag of tokens" you saw above.

Multi-head attention

A single attention operation produces one weight pattern per token. But a token often needs several kinds of information at once: the subject of the verb, the previous word, a matching bracket, a word with the same meaning. Multi-head attention runs h attention operations in parallel, each in a smaller subspace:

  1. Project X with Wq, Wk, Wv of shape (d_model, d_model), exactly as for one big head.
  2. Split the last axis into h chunks of size d_k = d_model / h, giving (h, T, d_k) per sequence.
  3. Run scaled dot-product attention independently in each head. The score tensor is (h, T, T): one attention pattern per head.
  4. Concatenate the head outputs back to (T, d_model).
  5. Multiply by an output matrix Wo of shape (d_model, d_model) so that the heads can mix.

Splitting heads is a reshape, not extra computation. With a batch dimension the whole thing is a handful of lines. The next example implements it, checks it against an explicit loop over heads, and prints all the shapes.

Python Multi-head attention with shapes

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The shapes tell the story: (2, 6, 16) in, (2, 4, 6, 4) for Q (four heads, each 4-dimensional), (2, 4, 6, 6) for the scores (four separate attention maps per example), and (2, 6, 16) out again. The result matches the explicit loop. The parameter count is 4 * d_modelΒ² no matter how many heads you use: more heads split the same budget into more, smaller views. The cost of attention also stays the same, since each head works in a space h times smaller.

Looking inside the heads

Trained transformers show heads that specialize. Some attend to the previous token, some to the same word elsewhere in the text, some to the start of the sentence, and some (in deeper layers) track grammar or copy a name from earlier. Researchers visualize heads exactly as you plotted weights before: a grid, tokens on both axes, color for the weight.

We cannot train a transformer in the browser here, so the next example wires three heads by hand, using the same mechanism (projection matrices), to show what such patterns look like. The input vector is a token one-hot concatenated with a position one-hot. Head 1's query asks "what is at position i - 1?", head 2's asks "which positions hold the same word as mine?", and head 3's asks "which positions are within one step of me?".

Python Three hand-wired attention heads

Press Run (or Ctrl + Enter) to see the result here.

Program input Β· what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Head 1 has one bright cell per row just below the diagonal: each token looks at the one before it. Head 2 spreads the weight equally over the three "the"s (and over the two "cat"s), so a repeated word finds its twin. Head 3 forms a band around the diagonal, looking at itself and its two neighbors. In a real model these projection matrices are learned from data and the patterns are fuzzier, mixing several behaviors, but the principle is the same. After the heads run, concatenation and Wo combine all three views into one new vector per token.

Check your understanding

Quick check Β· 5 questions

of 5 answered You scored / 5 Β· perfect!

  1. 1. What makes attention 'self'-attention?

    In self-attention one input sequence X is projected into Q, K and V, so every token can gather information from every token of the same sequence.
  2. 2. Why do transformers need positional information?

    Without positions, 'man bites dog' and 'dog bites man' produce identical vectors for each word. Position signals break that symmetry.
  3. 3. Which property is true of rotary position embeddings (RoPE)?

    After rotating the query at position m and the key at position n, their dot product depends only on m - n, so relative distance is built into the scores.
  4. 4. A model has d_model = 512 and 8 heads. What is d_k, and what is the shape of the score tensor for a batch of 4 sequences of 10 tokens?

    d_k = d_model / heads = 64. Each of the 8 heads has its own T x T score matrix, so the tensor is (batch, heads, T, T) = (4, 8, 10, 10).
  5. 5. How does the number of heads affect the number of parameters in the Q, K, V and output projections (with d_model fixed)?

    Heads split the d_model-wide projections into smaller pieces; the matrices keep the same total size.

Summary

  • Self-attention computes Q, K and V from the same sequence with learned matrices, so each token's output is a context-aware mix of all tokens.
  • Without position information, attention is permutation-equivariant and cannot tell word order; sinusoidal, learned and rotary encodings fix that.
  • Sinusoidal and rotary encodings make scores depend on relative offsets; learned tables are simple but limited to a maximum length.
  • Multi-head attention splits d_model into h subspaces, runs attention in each, concatenates the results and mixes them with Wo; the parameter count does not change with h.
  • Score tensors have the shape (batch, heads, T, T), and plotting them shows which tokens each head reads.
  • Heads in trained models specialize in patterns such as the previous token, repeated words or syntax.

Next: The Transformer Block.