Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 53 ยท Inside the Transformer

Attention from Scratch

Learn self-attention step by step: queries, keys, values, scaled dot-product, softmax weights and causal masking, built in NumPy with a heatmap.

In the previous lesson a bigram model could only look one token back. Real language needs more: in "The trophy did not fit in the suitcase because it was too big", the word "it" must be linked to "trophy", several tokens away. Attention is the mechanism that lets every token look at every other token and decide which ones matter for its own meaning.

Attention is the heart of the transformer, and it is smaller than its reputation. It is a few matrix multiplications and a softmax. In this lesson you build it in NumPy, look at the weights as a heatmap, and add the causal mask that makes a model able to generate text one token at a time. You need basic NumPy (see NumPy Basics) and the idea of embeddings.

Three roles for every token

Query What am I looking for?
Key What do I contain, as a label?
Value What do I hand over if chosen?

Each is a learned linear projection of the token's vector.

1 / 2

Queries, keys and values

Start with a sequence of n tokens, each already turned into a vector of length d_model (the embedding plus, as we will see in the next lesson, position information). Stack them into a matrix X of shape (n, d_model).

Attention makes three new matrices from X with three learned weight matrices:

  • Q = X @ W_q: one query per token, "what information do I need?"
  • K = X @ W_k: one key per token, "what do I offer?"
  • V = X @ W_v: one value per token, "here is my content if you pick me."

Think of a library. Your query is the question in your head, each book's key is its catalog card, and the value is the book's content. You compare your query with every card, then read mostly from the books whose cards matched well. The trick is that the model learns the projections, so it discovers for itself which relationships are worth tracking: pronouns to nouns, verbs to subjects, closing brackets to opening ones.

Scaled dot-product attention

The score between token i and token j is the dot product of query i with key j. All scores at once are Q @ K.T, an (n, n) matrix. We then:

  1. Scale by sqrt(d_k), where d_k is the key length. Dot products of long random vectors have a variance that grows with d_k; large scores push softmax into saturation, where one weight is nearly 1 and gradients vanish. Dividing keeps the scores in a friendly range.
  2. Apply softmax to each row, so each token gets weights that are positive and add up to 1.
  3. Multiply the weights by V. Each output row is a weighted average of the value rows.

In one line: attention(Q, K, V) = softmax(Q @ K.T / sqrt(d_k)) @ V. If softmax is new, see Logits, Softmax and Temperature.

Python Self-attention in 15 lines of NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The weights are random here because the projections are random; in a trained model they would encode real relationships. What matters is the structure: the output for each token has the same shape as its value vector, and it is built from all tokens in proportion to the weights.

Why scaling matters

You can see the saturation problem directly. Draw random queries and keys of increasing length, compute the largest softmax weight with and without scaling, and watch the unscaled version collapse toward a one-hot choice.

Python What happens without the square-root scaling

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With scaling, the largest weight stays about the same however long the vectors are. Without it, the model would be forced into "pick one token and ignore the rest" for large dimensions, and training would stall.

Causal masking

A model that generates text must not cheat. When it predicts token 5, it may only use tokens 1 to 4. During training, however, we feed the whole sequence in at once for speed, so we need to block each position from seeing the future. The causal mask does this: before the softmax, set every score above the diagonal (where j > i) to minus infinity. exp(-inf) is 0, so those positions get weight exactly 0 and the remaining weights still sum to 1.

flowchart LR
  S["Scores QยทKแต€ / โˆšd"] --> M["Mask future<br/>(set to -inf)"]
  M --> SM["Softmax per row"]
  SM --> W["Weights"]
  W --> O["Weights @ V"]

Masking happens before the softmax, so the blocked positions end up with weight exactly zero.

Models that read a whole input before answering (for example an encoder that classifies text) leave the mask off, so each token can see both directions. Models that write text, such as the GPT family, use the causal version.

Python Causal attention heatmap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The right heatmap is lower-triangular: the first token can only attend to itself, the second to the first two tokens, and so on. That triangle is the signature of every text-generating transformer. It also gives a cost you will meet again in Context Windows and the KV Cache: the weights matrix is n ร— n, so attention work grows with the square of the sequence length.

Attention weights are a useful picture, but not a full explanation of what a model "thinks". A high weight tells you where information was read from, not how it was used afterwards by later layers.

What attention buys you

Compare with the n-gram model. An n-gram model with n=3 sees two tokens; attention sees the whole prefix, and the weights are computed from content, so the same model can look back two tokens in one sentence and 200 in the next. The projections W_q, W_k and W_v are shared across all positions, so the number of parameters does not grow with sequence length. And because every position is computed with matrix multiplication, a GPU can process all positions at once during training, which is a large part of why transformers scaled so well.

One head of attention can only focus on one pattern at a time, though. Real models run many heads in parallel and combine them, wrap attention in residual connections and normalization, and follow it with a feed-forward network. That assembly is the transformer block.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In self-attention, what is computed from the dot product of a query and the keys?

    Scores = Q @ K.T. After scaling and softmax they become weights that say how much each token reads from each other token.
  2. 2. Why do we divide the scores by sqrt(d_k)?

    Dot products grow with vector length. Without scaling, softmax becomes extremely peaked and gradients become tiny.
  3. 3. What does the causal mask do before the softmax?

    Positions where j > i are masked with -inf. exp(-inf) is 0, so a token cannot read from the future.
  4. 4. What are the values (V) used for?

    Output = weights @ V. Queries and keys decide where to look; values are what is actually read.
  5. 5. Why does attention cost grow with the square of the sequence length?

    The score matrix has n x n entries, so doubling the context quadruples that part of the work.

Summary

  • Attention lets each token build its output as a weighted mix of all tokens' values.
  • Queries, keys and values are learned linear projections of the token vectors.
  • softmax(Q @ K.T / sqrt(d_k)) @ V is the whole computation; the scale keeps softmax from saturating.
  • A causal mask sets future scores to minus infinity so a generating model cannot peek ahead.
  • The weight matrix is n ร— n, which is why long contexts get expensive.

Next: put attention together with the other parts in The Transformer Block.