Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 45 ยท Sequences and Transformers

Attention from Scratch

Understand scaled dot-product attention, queries, keys and values, masks, multi-head attention and positional encodings, with PyTorch code and live NumPy stand-ins.

A recurrent network squeezes everything it has read into one hidden state, passed along one step at a time. Attention takes a different approach: when the model processes one token, it looks at all tokens at once and decides how much each one matters. The word "it" in "the cat did not cross the street because it was tired" can look straight back at "cat". There is no long chain of steps for information or gradients to travel through, and every position can be computed in parallel. That is the key idea behind Transformers, and therefore behind modern language models.

In this lesson you will build attention from its formula, learn what queries, keys and values are, see how masks keep padding and the future out of the picture, and extend it to several heads and positional encodings. PyTorch cannot run on this site, so the PyTorch code is read-only. The live editors implement every piece in NumPy, with a heatmap of the attention weights. They are stand-ins for the library, built from the same equations.

Queries, keys and values

Query What this token is looking for.
Key What each token offers to be found by.
Value The content that actually gets passed on.

Match query against keys, then take a weighted mix of values.

1 / 2

Scaled dot-product attention

Suppose a sequence of T tokens has already been turned into vectors (see Embeddings and Text Data). Three linear layers produce, for every token, a query q, a key k and a value v. Stack them into matrices Q, K, V, each of shape (T, d_k). Attention is

Attention(Q, K, V) = softmax(Q ยท K^T / sqrt(d_k)) ยท V

Read it from the inside out:

  1. Q ยท K^T is a (T, T) matrix of dot products. Entry (i, j) measures how well token i's query matches token j's key.
  2. Dividing by sqrt(d_k) keeps the scores in a sensible range. Dot products of long random vectors grow in size with the dimension, and large scores push softmax into a nearly one-hot state with tiny gradients. The scaling prevents that.
  3. softmax over each row turns the scores into attention weights: non-negative, summing to 1.
  4. Multiplying by V gives each token a weighted average of all value vectors.

The output has shape (T, d_k): the same shape as the input, but every row now contains information gathered from the whole sequence. Notice that nothing in this formula depends on the sequence length T as far as parameters go; the cost is the (T, T) score matrix, which grows quadratically with length, and this is the main cost of long contexts.

In PyTorch the formula is a built-in function that picks an efficient kernel for you:

Python
import torch
import torch.nn.functional as F

q = torch.randn(2, 5, 16)      # (batch, tokens, d_k)
k = torch.randn(2, 5, 16)
v = torch.randn(2, 5, 16)

out = F.scaled_dot_product_attention(q, k, v)
print(out.shape)               # torch.Size([2, 5, 16])

The first editor implements the same computation in NumPy on four tokens and prints the weights, then shows why the scaling matters.

Python Scaled dot-product attention in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The standard deviation of raw scores grows like the square root of the dimension, while the scaled scores stay near 1, which keeps softmax out of saturation.

Self-attention in a module

When queries, keys and values all come from the same sequence, it is called self-attention. In cross-attention the queries come from one sequence (say, the output being generated) and the keys and values from another (say, the encoded input). Here is single-head self-attention as a module:

Python
import math
from torch import nn

class SelfAttention(nn.Module):
    def __init__(self, d_model):
        super().__init__()
        self.q = nn.Linear(d_model, d_model)
        self.k = nn.Linear(d_model, d_model)
        self.v = nn.Linear(d_model, d_model)

    def forward(self, x, mask=None):                       # x: (batch, T, d_model)
        q, k, v = self.q(x), self.k(x), self.v(x)
        scores = q @ k.transpose(-2, -1) / math.sqrt(q.size(-1))   # (batch, T, T)
        if mask is not None:
            scores = scores.masked_fill(~mask, float("-inf"))
        weights = scores.softmax(dim=-1)
        return weights @ v

Masks: padding and the future

Two kinds of masks are used all the time.

A padding mask hides the filler tokens that make batches rectangular, so no token attends to padding. A causal mask (also called a look-ahead mask) hides future positions, so token i may only attend to tokens 0..i. A language model that predicts the next word must not see the answer, and a causal mask enforces that: the mask is a lower triangular matrix of "allowed" entries. Blocked scores are set to minus infinity (or a huge negative number) before the softmax, so they get exactly zero weight and the rest renormalize.

Python
T = 5
causal = torch.tril(torch.ones(T, T, dtype=torch.bool))     # True = allowed to attend
out = F.scaled_dot_product_attention(q, k, v, attn_mask=causal)
# or simply:
out = F.scaled_dot_product_attention(q, k, v, is_causal=True)

Boolean mask conventions differ. In F.scaled_dot_product_attention, attn_mask=True means "this position takes part". In nn.MultiheadAttention, a boolean attn_mask or key_padding_mask set to True means "this position is not allowed" (it is masked out). Read the docs for each function you use, and test with a tiny example.

Heatmaps, causal masks and positional encodings

The next editor draws the attention weights as a heatmap, with and without a causal mask, and then draws a sinusoidal positional encoding, which we explain right below it. Dark cells mean strong attention.

Attention itself has no sense of order: if you shuffle the input tokens, the outputs are the same set of vectors, shuffled in the same way. "Dog bites man" and "man bites dog" would look identical. To fix this, the model adds positional information to the token embeddings. The original Transformer used fixed sinusoidal patterns, where position pos and dimension pair i get

PE(pos, 2i) = sin(pos / 10000^(2i / d_model)) and PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model)).

Each dimension oscillates at a different frequency, so every position gets a unique pattern, and relative offsets are easy to compute from it. Many models instead learn a position embedding with nn.Embedding(max_len, d_model), and modern large language models often use rotary embeddings (RoPE), which rotate queries and keys by an angle that depends on position. The goal is always the same: let the model tell where a token is.

Python Attention heatmaps: free, causal, and positional encodings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Multi-head attention

One attention pattern is limiting: a token might need to look at its grammatical subject, the previous word and a related name at the same time. Multi-head attention runs several attention operations in parallel, each in its own lower-dimensional subspace, and joins the results.

With d_model = 64 and h = 8 heads, each head works with d_k = 64 / 8 = 8 dimensions. The recipe:

  1. Project the input with W_q, W_k, W_v (each d_model x d_model).
  2. Reshape (batch, T, d_model) into (batch, h, T, d_k): that is, split the feature axis into heads.
  3. Run scaled dot-product attention independently in every head.
  4. Reshape back to (batch, T, d_model) (concatenating the heads) and apply an output projection W_o.

The total cost is about the same as a single wide head, but the model can learn several different attention patterns at once. In PyTorch you rarely write this yourself:

Python
mha = nn.MultiheadAttention(embed_dim=64, num_heads=8, batch_first=True)

x = torch.randn(4, 10, 64)                                 # (batch, tokens, d_model)
pad_mask = torch.zeros(4, 10, dtype=torch.bool)
pad_mask[:, 8:] = True                                     # here True = ignore these keys (padding)

out, weights = mha(x, x, x, key_padding_mask=pad_mask)     # self-attention: query = key = value = x
print(out.shape)       # torch.Size([4, 10, 64])
print(weights.shape)   # torch.Size([4, 10, 10]) averaged over heads by default

Pass need_weights=False when you only need the output: it lets PyTorch use faster fused kernels. The next editor performs the split-attend-merge procedure and verifies the shapes and a key property: heads do not interact until the final output projection.

Python Multi-head attention: split, attend, merge

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Putting it into perspective

  • Cost: the score matrix is T x T, so memory and time grow quadratically with sequence length. Fused kernels (as used by scaled_dot_product_attention) avoid storing the full matrix, and research on sparse and linear attention tries to reduce the cost further.
  • No recurrence: all positions are computed in parallel, which is why Transformers train so efficiently on GPUs compared with RNNs (see RNNs, LSTMs and GRUs).
  • Interpretability caution: attention weights show where information is read from, but they are not a complete explanation of the model's decision.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In scaled dot-product attention, what are the softmax weights computed from?

    The weights come from softmax(Q K^T / sqrt(d_k)). They are then used to take a weighted average of the values V.
  2. 2. Why is the causal mask applied before the softmax rather than zeroing weights afterwards?

    Masking scores first means softmax renormalizes over the allowed positions. Zeroing weights afterwards would leave rows that no longer sum to 1.
  3. 3. A model has d_model = 512 and 8 heads. What is the dimension of each head?

    Each head works in d_model / num_heads = 512 / 8 = 64 dimensions, and the heads are concatenated back to 512.
  4. 4. Why do Transformers need positional encodings?

    Without positional information, permuting the tokens simply permutes the outputs, so the model could not tell 'dog bites man' from 'man bites dog'.
  5. 5. In nn.MultiheadAttention, what does key_padding_mask=True mean for a position?

    For boolean masks in nn.MultiheadAttention, True marks positions that are NOT allowed. This is the opposite of the attn_mask convention in F.scaled_dot_product_attention.

Summary

  • Attention lets every token look at all others: softmax(Q K^T / sqrt(d_k)) V, with queries, keys and values coming from linear projections.
  • The sqrt(d_k) scaling keeps scores in a range where softmax still has useful gradients.
  • Padding masks and causal masks set blocked scores to minus infinity before the softmax; watch the opposite boolean conventions of F.scaled_dot_product_attention and nn.MultiheadAttention.
  • Multi-head attention splits the feature axis into heads, attends in each, concatenates and projects, so several patterns are learned at once.
  • Attention is order-blind, so positional information (sinusoidal, learned or rotary) is added.
  • Cost grows quadratically with sequence length, but computation is fully parallel.

Next, combine attention with residual connections and feed-forward layers in Building a Transformer Block.