Skip to content
Tutorials Center
My learning

Topics

Lesson 45 of 63 ยท Neural NLP and Transformers

Attention and the Transformer

Understand scaled dot-product attention, multi-head attention and positional encoding by building them in NumPy, and see how they form the Transformer.

A recurrent network reads a sentence one word at a time and squeezes everything it has seen into a single hidden state. That state is a bottleneck: by the time the network reaches the end of a long paragraph, details from the start have faded. It is also slow, because step 50 cannot begin before step 49 has finished.

Attention removes both problems. Instead of passing a summary along a chain, every word looks directly at every other word, decides which ones matter for its own meaning, and takes a weighted mix of them. The Transformer is an architecture built almost entirely from this one operation, and it powers BERT, GPT and nearly every modern language model. In this lesson we build the core pieces in plain NumPy so that nothing stays magical.

The idea: a soft lookup

Think of a dictionary lookup. You have a query, you compare it with every key, and you return the value of the matching entry. Attention is the same thing made soft: instead of returning one value, it returns a blend of all values, weighted by how well each key matches the query.

Every word vector is turned into three new vectors by three learned matrices:

  • a query (Q): what this word is looking for,
  • a key (K): what this word offers to others,
  • a value (V): the information this word hands over when someone attends to it.

Three roles for every word

Query What am I looking for?
Key What do I contain?
Value What do I pass on?
1 / 2

In one formula, with d the size of the key vectors:

TEXT
Attention(Q, K, V) = softmax(Q Kแต€ / sqrt(d)) V

Scaled dot-product attention in NumPy

The code below takes four words with random 8-number vectors (a real model would use learned embeddings, see word embeddings), projects them to queries, keys and values, and prints the attention weights. Row i shows how word i distributes its attention over all four words. Because the numbers are random, the pattern means nothing linguistically; a trained model learns matrices that make the pattern useful.

Python Scaled dot-product attention

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every row sums to 1, because softmax turns scores into a probability distribution. The output has one 4-number vector per word: each is a blend of the value vectors, so each word's new representation now contains information from its neighbors. This is how context enters the picture. The word bank in "river bank" and in "bank loan" starts with the same embedding, but after attention it is mixed with different neighbors.

Why divide by the square root of d?

Dot products grow with the size of the vectors. If query and key entries have unit variance, their dot product has a variance of about d. Large scores push softmax into a state where one weight is nearly 1 and the others nearly 0; there the gradient is almost zero and learning stalls. Dividing by sqrt(d) brings the scores back to a sane range. The next demo shows the effect, and also adds a causal mask, which language models such as GPT use so that a word cannot peek at words that come after it.

Python Why scale, and what a causal mask does

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With d = 64, the unscaled softmax gives the winning key a weight of 0.94, nearly a hard choice, while the scaled version gives 0.38 and keeps the distribution soft. At d = 512 the picture is the same (0.93 against 0.31). In the masked matrix, the entries above the diagonal are exactly zero: the first word can only attend to itself, the last word to all four. Setting a score to a huge negative number before softmax makes its weight vanish.

An encoder model such as BERT lets every word see every other word (no mask). A decoder model such as GPT uses the causal mask, because at generation time the future words do not exist yet. Pre-trained language models covers the difference.

Multi-head attention

One attention pattern is not enough. In a sentence, a word may need to find its subject, its nearest neighbor and a distant pronoun reference all at once. Multi-head attention runs several attentions in parallel, each with its own projection matrices working on a slice of the vector, then concatenates the results and mixes them with one more matrix Wo. Different heads can specialize, and trained models do show heads that track syntax, position or rare patterns.

Positional encoding

Attention has an odd property: it ignores word order. If you shuffle the input rows, the output rows are shuffled in exactly the same way, and nothing else changes. "Dog bites man" and "man bites dog" would look identical to the layer. To fix this, the Transformer adds a positional encoding to each word vector, a pattern that is different for every position. The original paper used sine and cosine waves of different frequencies:

TEXT
PE(pos, 2i)   = sin(pos / 10000^(2i / d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))

Many newer models learn positions or use rotary embeddings, but the goal is the same. The following example builds multi-head attention, proves the shuffling property, and shows that positional encodings make order matter.

Python Multi-head attention and positional encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two heads produce different weight patterns for the same word (0.59 of head 0's attention goes to the third word, while head 1 spreads it out more evenly). The permutation check prints True: without positions the layer cannot tell order. Once positional encodings are added, shuffling changes the result. The last two lines show a nice property of the sine and cosine design: the similarity between two positions depends on how far apart they are. Position 0 is most similar to itself (4.0) and gets less similar as the distance grows, and position 2 has equal similarity to positions 1 and 3.

From attention to the Transformer

A Transformer block wraps attention with a few more ingredients:

  1. Multi-head self-attention, as above, so each word gathers context.
  2. Residual connection and layer normalization: the input is added back to the output and normalized, which keeps training stable in deep stacks.
  3. Feed-forward network: a small two-layer network applied to each word separately, which adds nonlinear processing.
  4. A second residual connection and normalization.

Stack a dozen or a hundred of these blocks, put token embeddings plus positional encodings at the bottom and a prediction layer at the top, and you have a Transformer. Because attention is a set of matrix multiplications over the whole sequence at once, training parallelizes beautifully on GPUs. That, more than any single trick, is why Transformers replaced recurrent networks. The price is that attention compares every pair of words, so cost grows with the square of the sequence length, which is why context windows are limited and long-context models need extra engineering.

FNet: mixing tokens with a Fourier transform

Self-attention is powerful but costly: it compares every pair of tokens and has learned parameters for the projections. FNet (Google, 2021) asks how much of that is really needed. It keeps the Transformer block, but replaces the self-attention sublayer with a fixed 2D Fourier transform, applied over the sequence and the hidden dimensions, keeping the real part. The feed-forward layers, residual connections and layer normalization stay. Nothing in the mixing step is learned, and each output position still depends on every input position, so information flows across the whole sentence.

The trade-off is speed against accuracy: reported FNet models train faster and use less memory, particularly with long sequences (the FFT scales as n log n instead of nยฒ), and reach most of BERT's accuracy on classification benchmarks, but they lag behind on tasks that need precise token-to-token focus. FNet therefore works best for encoder tasks; text generation with FNet is possible when it is used as the encoder of an encoder-decoder or inside a larger model, but plain autoregressive generation needs a causal mask, which a global Fourier mix does not have. In practice, FNet-style layers are a speed-quality experiment rather than a replacement for attention.

Python Attention mixing versus Fourier mixing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The attention step needs two learned matrices (128 parameters here) and produces weights that sum to 1 in every row. The Fourier step has no parameters and produces an output of the same shape. Editing a single value in token 5 changes the output at all 6 positions, which shows the mixing is global, and the output of doubling the input doubles, which shows that it is a fixed linear map rather than an input-dependent weighting like attention.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In attention, what do the weights for one word add up to?

    Softmax turns the scores for one query into positive numbers that sum to 1, so the output is a weighted average of the value vectors.
  2. 2. Why are the scores divided by the square root of the key size?

    Large dot products push softmax toward a one-hot output with almost no gradient. Scaling keeps the scores in a range where learning works.
  3. 3. What problem does positional encoding solve?

    Attention treats its input as a set, so shuffling the words only shuffles the outputs. Adding a position-dependent pattern lets the model use word order.
  4. 4. Why does a decoder model such as GPT use a causal mask?

    When generating text, later words do not exist yet. Masking them during training makes training match generation.
  5. 5. What is the benefit of multiple attention heads?

    Heads work on separate slices with separate projections, so different heads can specialize in different relations, and their results are combined by an output matrix.

Summary

  • Attention lets every word look directly at every other word and take a weighted mix of their value vectors.
  • Queries, keys and values come from learned projections; the formula is softmax(Q Kแต€ / sqrt(d)) V.
  • Scaling by sqrt(d) keeps softmax from saturating; a causal mask blocks future words in decoders.
  • Multi-head attention runs several attentions in parallel and combines them.
  • Attention ignores order, so positional encodings are added to the embeddings.
  • A Transformer stacks attention, feed-forward layers, residual connections and normalization; it trains in parallel, but its cost grows with the square of the sequence length.

Next, see how these blocks are trained at massive scale in Pre-trained Language Models: BERT, GPT and Beyond.