Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 37 ยท Transformer Internals

Self-Attention Step by Step

Compute scaled dot-product self-attention by hand: queries, keys, values, scores, softmax and weighted sums, with a NumPy implementation and a heatmap.

Self-attention is the operation that lets each token gather information from the other tokens in the same sequence. It is the heart of the Transformer, and it is short enough to write in a dozen lines of NumPy. This lesson builds it from the question it answers: "for this token, which other tokens matter, and how much?"

Queries, keys and values

Each token's vector x is projected three times by learned matrices:

  • a query q = x ยท Wq: what this token is looking for;
  • a key k = x ยท Wk: what this token offers to be matched against;
  • a value v = x ยท Wv: the information it passes along if it is selected.

A library analogy helps: the query is your search phrase, keys are the titles on the shelf, and values are the book contents. You compare your query with every title and then read a blend of the books in proportion to how well they matched.

Attention in four moves

Score Query dot key for every pair.
Scale Divide by the square root of the key size.
1 / 2

The formula

For all tokens at once, stack the vectors into matrices Q, K, V (one row per token):

Attention(Q, K, V) = softmax(Q Kแต€ / โˆšdโ‚–) V

Q Kแต€ is an n ร— n matrix of scores: entry (i, j) says how much token i wants to look at token j. Softmax is applied row by row, so each token gets a probability distribution over the tokens it can attend to. Multiplying by V produces, for every token, a weighted average of all the value vectors.

Python Self-attention from scratch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the weight matrix row by row: row i shows how token i spreads its attention over all four tokens. The output for token i is the weighted blend of the value vectors.

Why divide by the square root of d

If query and key entries have unit variance, their dot product over d dimensions has variance d, so the scores grow with the key size. Large scores push softmax toward a one-hot distribution with tiny gradients, which slows learning. Dividing by โˆšd keeps the scores at a stable scale.

Python The effect of scaling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

As d grows, the unscaled softmax becomes almost a hard choice (maximum weight near 1.00) while the scaled version stays spread out. The exact numbers depend on the random seed, but the trend is stable.

Seeing attention

A heatmap of the weight matrix is the standard way to inspect attention. Each row is a token deciding where to look.

Python Plot the attention matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

These weights come from random matrices, so they show no linguistic meaning. In a trained model the same picture shows structure, such as a pronoun attending to the noun it refers to. Training is what turns the projection matrices into useful lookups.

What attention is doing

Three properties are worth keeping in mind.

  1. Content-based routing. Which tokens are attended to depends on the content of the query and keys, not on fixed positions.
  2. Weighted averaging. The output is a mixture of values, so it is a smooth, differentiable lookup.
  3. Parallelism. All n ร— n scores are one matrix multiplication, which is what makes Transformers efficient to train on GPUs.

The cost is that the score matrix has nยฒ entries, so doubling the sequence length quadruples the work and memory of this step. See Attention Variants for ways to reduce that.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does entry (i, j) of Q Kแต€ represent?

    It is the dot product between the query of token i and the key of token j.
  2. 2. Across which axis is softmax applied to the attention scores?

    Each query row becomes a probability distribution over the keys.
  3. 3. Why divide the scores by the square root of the key dimension?

    Dot products grow with dimension; scaling prevents near-one-hot softmax and vanishing gradients.
  4. 4. What is the shape of the attention weight matrix for a sequence of n tokens?

    Each of the n tokens has a weight for each of the n tokens, which is why cost grows with n squared.

Summary

  • Each token is projected into a query, key and value.
  • Scores Q Kแต€ / โˆšd are turned into weights by a row-wise softmax; the output is the weighted sum of values.
  • Scaling keeps softmax from saturating.
  • The weight matrix is n ร— n, giving parallel computation but quadratic cost.
  • Training shapes the projection matrices so the weights become meaningful.

Next: Multi-Head Attention and Masking