Skip to content
Tutorials Center
My learning

Topics

Lesson 38 of 73 ยท Sequences, Attention and Transformers

The Attention Mechanism

Understand attention as a soft dictionary lookup. Implement scaled dot-product attention in NumPy, plot weights, apply masks and see why we divide by sqrt(d).

A recurrent network squeezes everything it has read into one hidden vector. By the time it reaches the end of a long sentence, details from the beginning are blurred. Attention removes that bottleneck. Instead of relying on a single summary, the model keeps all the vectors it computed and, at each step, looks back at them and picks the ones that matter, taking a weighted average of the relevant ones.

Attention was first added to RNN translation models in 2014 and 2015 and gave a large jump in quality on long sentences. In 2017 the Transformer paper showed that attention alone, with no recurrence, was enough, and almost every modern language, vision and speech model has used it ever since. The core operation is only a few lines of NumPy. This lesson builds it and shows what each ingredient does.

Attention is a soft dictionary lookup

Query What I am looking for.
Keys A label on every stored item.
Values The content that is returned.

A normal dictionary returns one value. Attention returns a blend of all of them.

1 / 2

Queries, keys and values

Think of a library with a card catalog. Every book has a key (its title and subject on the card) and a value (the book itself). You walk in with a query (what you want to read about). A strict lookup would return the one book whose key matches exactly. Attention is the soft version: it compares your query with every key, gets a similarity score for each, converts the scores to weights, and hands you a blend of all the books, with the best matches contributing most.

In formulas, for a query matrix Q of shape (n_queries, d), a key matrix K of shape (n_keys, d) and a value matrix V of shape (n_keys, d_v):

Attention(Q, K, V) = softmax(Q @ K.T / sqrt(d)) @ V

Walk through the shapes. Q @ K.T has shape (n_queries, n_keys): one score for every pair of a query and a key. Softmax along the last axis turns each row into weights that are positive and sum to 1. Multiplying by V gives (n_queries, d_v): for each query, the weighted average of the value rows. (The sqrt(d) has a job of its own, shown below.)

The first example implements this function and uses it as a soft dictionary with three keys that are one-hot vectors. The query decides which values dominate.

Python Attention as a soft dictionary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The query [0, 6, 0] matches key 1, so 94% of the weight lands on it and the output is close to value 1 ([0, 10]). The query [4, 4, 0] splits its weight between keys 0 and 1 and returns the average of their values, [5, 5]. The vague query [1, 1, 1] matches all keys equally and returns the plain average of all values. That is the "soft" in a soft lookup. Because everything is matrix multiplication and softmax, the whole operation is differentiable, so the network can learn what to ask and how to label its stored items.

A picture of the weights: word alignment

Machine translation gives the clearest example. When a model writes the English word "black", it should look at the French word "noir", not at "le". The next example hand-builds tiny vectors in which words with the same meaning share a direction, uses the English words as queries and the French words as keys and values, and plots the attention weights as a heatmap. Real models learn such vectors, with hundreds of dimensions.

Python Attention weights as a heatmap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each row is one English word and sums to 1. The weights are 0.69 on the matching French word and 0.15 on each of the others: "the" looks at "le", "black" at "noir", "cat" at "chat". The English order (adjective before noun) differs from the French order (noun before adjective), and attention handles the reordering without any special machinery: the diagonal is simply permuted. Add a larger scale to the queries and the rows get sharper.

Why divide by the square root of d?

The scaling factor looks like a detail, but removing it quietly breaks training. A dot product of two vectors with d independent entries of unit variance is a sum of d terms, so its variance is d and its standard deviation is sqrt(d). As the key size grows, the scores grow too, and a softmax over large numbers becomes extremely sharp: one weight gets almost all the mass and the others almost none. When softmax is that saturated, its derivative is close to zero, so almost no gradient flows back, the same kind of trouble as saturating sigmoids in Activation Functions.

Dividing by sqrt(d) brings the scores back to unit variance whatever d is. The next example measures all of this on random vectors.

Python What happens without the sqrt(d) scaling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The standard deviation of the raw score grows like sqrt(d) (1.9 at d = 4, 31.6 at d = 1024), while the scaled score stays at 1. With 10 keys, uniform attention would give a top weight of 0.10, and the scaled version stays near 0.3 for every d. The unscaled version rises to 0.96 at d = 1024: attention has become a hard, nearly one-hot choice. The last column shows what that costs: the size of the softmax derivative falls from 0.34 to 0.05, so the gradients are about seven times smaller, while the scaled version stays at about 0.35 for every size.

Masks: controlling who can look at whom

Sometimes some positions must not be attended to. A mask is a boolean matrix of the same shape as the score matrix; wherever it says "not allowed", we replace the score by a huge negative number (such as -1e9) before the softmax, so the weight becomes zero. Two masks matter in practice:

  • Padding mask. Sequences in a batch are padded to the same length (see Working With Sequences). Padding tokens carry no information, so no query should attend to them.
  • Causal mask. When a model generates text, position t may only look at positions 0..t, never at the future. The mask is lower triangular: np.tril(np.ones((T, T))). Without it, a language model could cheat during training by reading the word it is supposed to predict.
Python Padding and causal masks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the unmasked heatmap, every token (including real words) spends some weight on <pad>. With the padding mask, the <pad> column turns to exact zeros (the printout shows 0. for all four real tokens), and the other weights renormalize to still sum to 1. With the causal mask, everything above the diagonal is zero: the first token I can only see itself, and deep sees I, like and deep. Notice in the third plot that row 5 (the pad token as a query) is still computed. It is simply ignored later by the loss.

Attention in the wild

The version here has one query, key and value space and is used with an explicit pair of sequences. A transformer uses a refined variant: it computes queries, keys and values from the same sequence with learned projections (self-attention), and it runs several of them in parallel with different projections (multi-head attention). Those are the topics of the next lesson. Two properties are worth remembering already:

  • Direct paths. Any position can attend to any other in a single step, so the distance a gradient has to travel does not grow with the sequence length, unlike in an RNN.
  • Quadratic cost. The score matrix has n x n entries, so time and memory grow with the square of the sequence length. This is why long context windows are expensive and why many efficiency tricks target attention.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In scaled dot-product attention, what are the shapes if Q is (4, 8), K is (6, 8) and V is (6, 16)?

    Q @ K.T is (4, 8) @ (8, 6) = (4, 6), one score per query-key pair. Multiplying the (4, 6) weights by V (6, 16) gives (4, 16): one blended value per query.
  2. 2. Why are the scores divided by sqrt(d)?

    Dot products of d-dimensional vectors have a standard deviation of about sqrt(d). Without scaling, large d produces near one-hot softmax outputs with tiny gradients.
  3. 3. What does a causal mask do?

    A lower-triangular mask zeroes the weights of future positions, so an autoregressive model cannot read the word it has to predict.
  4. 4. What value is typically added or substituted in masked score positions before the softmax?

    exp of a huge negative number is essentially zero, so the masked position receives zero weight and the remaining weights still sum to 1. Using 0 would still give the position a positive weight.
  5. 5. Compared with an RNN, which is a cost of attention?

    Every position scores every other position, producing an n x n matrix. The benefit is a direct path between any two positions.

Summary

  • Attention is a differentiable, soft dictionary lookup: queries are compared with keys, and the resulting weights mix the values.
  • The formula is softmax(Q @ K.T / sqrt(d)) @ V; each row of weights is positive and sums to 1.
  • Scaling by sqrt(d) keeps scores at unit variance so the softmax does not saturate and gradients keep flowing.
  • Masks (padding, causal) set forbidden scores to a huge negative number so their weights become exactly zero.
  • Attention gives every position a direct path to every other, at a cost that grows with the square of the sequence length.
  • Heatmaps of the weights show which inputs the model used, which makes attention easy to inspect.

Next: Self-Attention and Multi-Head Attention.