Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 53 · Inside the Transformer

The Transformer Block

Assemble a transformer block: multi-head attention, positional encoding, residual connections, layer norm and feed-forward layers, with a NumPy forward pass.

Attention lets tokens exchange information, but it is only one ingredient. A large language model is a tall stack of identical transformer blocks. Each block takes a matrix of token vectors and returns a matrix of the same shape, a little more informed about its context. GPT-style models with billions of parameters are this one block repeated dozens of times, with different learned weights in each copy.

In this lesson you meet each part of the block, in the order data flows through it: position information, multi-head attention, the residual connection, layer normalization and the feed-forward network. Then you run a full forward pass with random weights in NumPy and count where the parameters live. Random weights cannot produce meaningful text, but they do show the shapes and the arithmetic exactly as a real model has them.

One block, five parts

Position Tell the model token order.
Attention Tokens read from each other.
Residual Add the input back in.
Layer norm Keep numbers well scaled.

Then a feed-forward network, with its own residual and norm.

1 / 2

Position: telling the model where tokens are

Attention by itself has no sense of order. If you shuffle the input rows, the output rows are shuffled the same way and nothing else changes. But "dog bites man" and "man bites dog" must differ, so the model needs position information. Two common approaches:

  • Sinusoidal encodings (the original transformer): add a fixed vector to each token embedding, built from sines and cosines of different frequencies. For position pos and dimension pair i: sin(pos / 10000^(2i/d)) and cos(...). Low dimensions oscillate fast and high ones slowly, like the digits of a counter, so every position gets a distinct pattern.
  • Learned positions: a trainable table with one vector per position, used by the GPT-2 family. It cannot go beyond the sizes it was trained for.
  • Rotary embeddings (RoPE): instead of adding a vector, rotate the query and key vectors by an angle that depends on position. Because rotating by a and by b differ by a - b in the dot product, the attention score depends on the relative distance between tokens. Many modern open models use RoPE.
Python Sinusoidal positional encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The picture shows fast stripes in the low dimensions (top) and slow ones at the bottom. The model adds this matrix to the token embeddings before the first block.

Multi-head attention

One attention head computes one set of weights, so it can only pick one pattern of "who looks at whom" per position. Multi-head attention runs several heads side by side, each with its own W_q, W_k and W_v, and concatenates the results. In practice one big projection is split into h slices: if d_model is 768 and there are 12 heads, each head works in 64 dimensions. The concatenated output (back to 768 dimensions) goes through one more learned matrix W_o.

Different heads tend to specialize: in trained models, some track the previous token, some link a verb to its subject, some look for earlier repeats of the current token. You do not design these roles; training finds them. Multi-head attention costs the same as one big head, because the slices are smaller, and the number of parameters is 4 · d_model² (for W_q, W_k, W_v, W_o).

Residual connections and layer normalization

Deep networks are hard to train because signals and gradients can fade or blow up as they pass through many layers. Two simple devices make stacks of dozens of blocks work.

A residual connection adds a sub-layer's input to its output: x + sublayer(x). The sub-layer only has to learn a change to apply, and the original signal always has a clear path through the whole network.

Layer normalization rescales each token's vector to have mean 0 and variance 1 across its features, then applies a learned scale and shift. This keeps activations in a stable range. Most modern models use the pre-norm layout, normalizing the input to each sub-layer: x = x + attention(norm(x)), then x = x + ffn(norm(x)). Many recent models also swap LayerNorm for the cheaper RMSNorm, which skips the mean subtraction.

The feed-forward network

After attention has mixed information between tokens, each token passes independently through a small two-layer network: expand to a larger width (usually 4 · d_model), apply a non-linearity such as GELU, then project back down. Every token position uses the same weights. Attention moves information between positions; the feed-forward network processes it within each position, and it holds roughly two thirds of a block's parameters. Researchers often describe these layers as where much of the model's stored knowledge lives, though that picture is still being refined.

flowchart TB
  X["Input x (n × d)"] --> N1["LayerNorm"]
  N1 --> A["Multi-head attention"]
  A --> R1(("+"))
  X --> R1
  R1 --> N2["LayerNorm"]
  N2 --> F["Feed-forward (d → 4d → d)"]
  F --> R2(("+"))
  R1 --> R2
  R2 --> Y["Output (n × d)"]

A pre-norm block: two sub-layers, each wrapped in normalization and a residual add. Input and output have the same shape.

A forward pass in NumPy

Now we assemble everything. This code builds a causal multi-head attention, the feed-forward network, layer norm and residuals, then stacks four blocks. The weights are random (scaled so the numbers stay sensible), so the output vectors mean nothing, but every shape and operation matches a real model.

Python Stack transformer blocks with random weights

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the shapes: the matrix stays (6, 32) through every block. That uniform shape is what allows blocks to be stacked to any depth, and every block adds an (n, n) attention matrix per head.

Counting parameters

The weights are where a model's size comes from, so it is worth being able to count them. Per block (ignoring small bias and norm parameters):

  • Attention: W_q, W_k, W_v, W_o = 4 · d²
  • Feed-forward with width 4d: d · 4d + 4d · d = 8 · d²
  • Total per block: about 12 · d²

Add the token embedding table (vocab · d) and you get the full size. The usual approximation is params ≈ 12 · layers · d² + vocab · d.

Python Estimate model size from its shape

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The estimates land close to the published sizes: about 124 million for GPT-2 small, about 1.5 billion for GPT-2 XL, and about 6.5 to 7 billion for a 7B-class model. Modern designs tweak the recipe (gated feed-forward layers, grouped-query attention with fewer key/value heads, no tied embeddings), so treat the formula as a rule of thumb, not an exact count. Notice that nearly all parameters sit in the blocks, and that doubling d_model quadruples a block's size.

Memory for the weights alone is parameters × bytes per parameter: a 7-billion-parameter model in 16-bit precision needs about 14 GB. You will use this in LoRA, Quantization and Efficient Models.

From stack to language model

To turn the stack into a language model, add two things at the ends. At the start, a lookup table converts token IDs to embeddings. At the end, a final layer norm and a linear layer map each output vector to one score per vocabulary token, called logits. For the last position, those logits go through softmax and temperature to become the next-token probabilities, exactly the quantity the n-gram model estimated by counting. Training adjusts all the weights, by gradient descent, so those probabilities match real text, as described in How LLMs Are Trained.

Check your understanding

Quick check · 5 questions

of 5 answered You scored / 5 · perfect!

  1. 1. Why does a transformer need positional information?

    Without positions, 'dog bites man' and 'man bites dog' would look identical to attention. Sinusoidal, learned or rotary methods add the order.
  2. 2. What is the purpose of a residual connection?

    x + sublayer(x) keeps a direct path for the signal and gradients, which makes dozens of stacked blocks trainable.
  3. 3. What does the feed-forward network in a block do?

    Attention mixes across positions; the feed-forward network transforms each position on its own and holds most of the block's parameters.
  4. 4. A model has 24 layers and d_model = 1024. Roughly how many parameters are in its blocks (12 · layers · d²)?

    12 × 24 × 1024² is about 302 million, before counting the embedding table.
  5. 5. Why use multiple attention heads instead of one big head?

    Separate heads, each with their own projections, can specialize in different relations, and their outputs are concatenated and mixed by W_o.

Summary

  • A transformer is a stack of identical blocks that keep the (tokens × width) shape.
  • Position information (sinusoidal, learned or rotary) is needed because attention ignores order.
  • Multi-head attention runs several attention patterns in parallel and merges them with an output projection.
  • Residual connections and layer normalization make deep stacks trainable.
  • The feed-forward network processes each token independently and holds most parameters; a block has about 12 · d² of them.
  • Embeddings at the input and a logits layer at the output turn the stack into a language model.

Next: see why long prompts cost memory and time in Context Windows and the KV Cache.