Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 40 ยท Sequences and Text

Attention and Transformers

Learn scaled dot-product attention, multi-head self-attention, positional encoding and the Transformer encoder block, with NumPy demos and Keras MultiHeadAttention code.

Recurrent networks read a sentence one word at a time and must squeeze everything they know into a hidden state. Attention takes a different approach: each word looks directly at every other word and decides how much each one matters for its own meaning. There is no step-by-step bottleneck, and all positions are processed in parallel, which makes training fast on GPUs. Models built from attention layers, Transformers, power modern language models, translation and many vision systems.

TensorFlow is not available in the browser runner. The attention math below is run in NumPy; the Keras layers are shown read-only.

Queries, keys and values

Each token's vector is projected three ways:

  • a query q: "what am I looking for?"
  • a key k: "what do I contain?"
  • a value v: "what do I give you if you attend to me?"

The attention score between token i and token j is the dot product q_i ยท k_j. Scores are scaled, turned into weights with softmax (so they sum to 1 for each token), and used to take a weighted average of the values:

Attention(Q, K, V) = softmax(Q Kแต€ / sqrt(d)) V

The division by sqrt(d) (d = key size) keeps dot products from growing with dimension, which would push softmax into saturation.

Python Scaled dot-product attention

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Row i of the weight matrix says how much token i pulls from each token, including itself. Because the queries, keys and values all come from the same sequence this is self-attention.

A readable example

Weights are easiest to understand when the vectors are meaningful. Here two tokens have similar keys, so a query that matches them splits its attention between those two.

Python Who attends to whom?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The query asks for nouns, so "cat" and "mat" get most of the weight and "sat" and "the" almost none.

Masking

A padding mask stops tokens from attending to padding. A causal mask stops a token from attending to later tokens, which is required when a model generates text one token at a time. Masked scores are set to a very negative number before softmax so their weights become 0.

Python Causal mask

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Multi-head attention

One attention pattern is not enough: a word may need to attend to its subject, its neighbor and a distant reference at once. Multi-head attention runs several attention operations in parallel, each with its own projections of smaller size, and concatenates the results. With model size 64 and 4 heads, each head works in 16 dimensions.

Python
import keras
from keras import layers

mha = layers.MultiHeadAttention(num_heads=4, key_dim=16)
x = keras.random.normal((2, 10, 64))       # (batch, tokens, features)
y = mha(query=x, value=x, key=x)           # self-attention
print(y.shape)                             # (2, 10, 64)

Position information

Attention treats its input as a set: shuffling the tokens would just shuffle the outputs. To use word order we add position information to the embeddings, either with fixed sine and cosine waves or with a learned position embedding.

Python Sinusoidal positional encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The Transformer encoder block

A block combines attention with a small feed-forward network, each wrapped with a residual connection and layer normalization:

flowchart TB
    A["Input embeddings + positions"] --> B["Multi-head self-attention"]
    B --> C["Add & LayerNorm"]
    A --> C
    C --> D["Feed-forward (Dense, Dense)"]
    D --> E["Add & LayerNorm"]
    C --> E
    E --> F["Output (same shape)"]

Residual "Add" connections let gradients bypass each sub-layer, so many blocks can be stacked.

Python
class TransformerBlock(layers.Layer):
    def __init__(self, dim, heads, ff_dim, rate=0.1):
        super().__init__()
        self.att = layers.MultiHeadAttention(num_heads=heads, key_dim=dim // heads)
        self.ff = keras.Sequential([layers.Dense(ff_dim, activation="relu"),
                                    layers.Dense(dim)])
        self.norm1 = layers.LayerNormalization(epsilon=1e-6)
        self.norm2 = layers.LayerNormalization(epsilon=1e-6)
        self.drop = layers.Dropout(rate)

    def call(self, x, training=False):
        a = self.att(x, x)
        x = self.norm1(x + self.drop(a, training=training))
        f = self.ff(x)
        return self.norm2(x + self.drop(f, training=training))

A classifier stacks one or more blocks after Embedding plus positional information, then applies GlobalAveragePooling1D and a Dense head. Custom layers are explained in Custom Layers and Models.

The cost of attention grows with the square of the sequence length (every token against every token), so long inputs need tricks such as local or sparse attention. For small tasks, a one-block Transformer or a plain GRU often performs equally well.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In attention, what do the softmax weights for one token sum to?

    Softmax turns the scores into a probability distribution over the tokens.
  2. 2. Why divide the scores by sqrt(d)?

    Dot products grow with dimension; scaling keeps softmax gradients healthy.
  3. 3. What does a causal mask do?

    It is needed for left-to-right generation so predictions cannot peek at the future.
  4. 4. Why do Transformers add positional encodings?

    Self-attention is order-agnostic, so position information must be injected explicitly.

Summary

  • Attention computes softmax(Q Kแต€ / sqrt(d)) V: each token takes a weighted average of values, weighted by query-key match.
  • Self-attention uses one sequence for Q, K and V; masks block padding or future tokens.
  • Multi-head attention runs several smaller attentions in parallel.
  • Positional encodings (sinusoidal or learned) add order information.
  • A Transformer block is attention plus feed-forward, each with residual connection and layer normalization; cost grows quadratically with length.

Next: tf.data Pipelines