Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 73 ยท Sequences, Attention and Transformers

The Transformer Block

Build a transformer block in NumPy: attention, MLP, residual connections and layer norm, pre-norm versus post-norm, causal masking and parameter counts.

You have met the two main ingredients already. Self-attention lets every token read from every other token, and layer normalization keeps the numbers flowing through a deep network in a healthy range. A transformer block is what you get when you wire those parts together with two more simple pieces, a small feed-forward network and residual connections, and then stack the result many times.

Almost every modern language model, image model and speech model is a tall stack of nearly identical blocks. Once you understand one block, you understand the architecture. In this lesson you will build one in NumPy, see why the order of its parts matters, and learn the single change (a mask) that turns an "encoder" block into a "decoder" block.

One block, four parts

Attention Tokens exchange information with each other.
MLP Each token is processed on its own, in parallel.
Residuals Each part adds to the signal instead of replacing it.
Layer norm Keeps the scale steady before each part.

Stack 12, 24 or 96 of these and you have a large model.

1 / 2

The recipe

A block takes a sequence of vectors with shape (T, d): T tokens, each described by d numbers. It returns a sequence of the same shape. That is the key design choice: because the shape never changes, you can stack blocks as deep as you like.

Inside, there are two sublayers:

  1. Multi-head self-attention. Every token gathers information from the other tokens.
  2. A position-wise MLP (also called the feed-forward network). Two linear layers with a non-linearity in between, applied to each token separately. The hidden layer is usually four times wider than d, and the activation is usually GELU.

Each sublayer is wrapped in a residual connection (the output is added to the input) and a layer norm. The residual connection is what lets gradients flow straight through dozens of blocks, much like the skip connections in ResNets, and it means a block only has to learn a correction to its input, not a whole new representation.

The attention sublayer is the only place where information moves between positions. The MLP never looks sideways. That clean split is why people describe a transformer as alternating "communication" (attention) and "computation" (MLP).

Building a block in NumPy

The code below defines the pieces from scratch: softmax, layer norm, GELU, multi-head attention with an optional causal mask, the MLP, and a block in the pre-norm layout (normalize first, then run the sublayer, then add the result back). Weights get a fixed seed, so every run prints the same numbers.

The weights here are random and untrained. That is fine for studying shapes and information flow, which is what we are doing.

Python A transformer block, piece by piece

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output from the top. The block returns the same shape it was given, (6, 16), and the attention weights have shape (4, 6, 6): one 6 x 6 table per head. In row 2 of head 0, token 2 spreads its attention over tokens 0, 1 and 2 and puts exactly zero on the later tokens. The last two lines are the real test. When only the final token is replaced, a causal block changes only the final output. The non-causal block changes all six. The next section explains why that difference matters.

Encoder, decoder and the mask

The block above has a causal switch, and that one switch is the difference between the two classic block types.

  • An encoder block lets every token look at every other token, in both directions. That suits tasks where you have the whole input in hand and want a rich understanding of it: classifying a sentence, tagging words, or turning an image into features. BERT-style models are stacks of encoder blocks.
  • A decoder block applies a causal mask: token i may attend only to tokens 0..i. That suits generation. A language model is trained to predict the next token, and if token i could peek at token i+1, it would simply copy the answer. GPT-style models are stacks of decoder blocks.

The mask works by setting the forbidden scores to a huge negative number before the softmax. After the softmax, exp(-1e9) is zero, so the forbidden positions get exactly zero weight and the allowed ones still sum to one. Masking after the softmax would be wrong because the remaining weights would no longer sum to one.

The original 2017 transformer was an encoder-decoder model for translation: an encoder stack reads the source sentence, and a decoder stack writes the translation, with an extra cross-attention sublayer in each decoder block whose queries come from the decoder and whose keys and values come from the encoder output. Today most large language models drop the encoder entirely and use decoder-only stacks, which is also what we build in the next lesson.

Python Encoder mask versus decoder mask

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The encoder grid is solid. The decoder grid is a lower triangle: each row (a query) lights up only the columns at or before its own position. Of the 36 possible token pairs, a decoder allows 21.

A causal mask also gives training a useful property. Because token i can never see the future, one forward pass over a sequence of length T produces T valid next-token predictions at once, one per position. You get T training examples for the price of one pass.

Pre-norm versus post-norm

There are two ways to place layer norm relative to the residual connection.

Post-norm is the layout of the original paper. You run the sublayer, add the result to the input, and then normalize the sum:

x = layer_norm(x + sublayer(x))

Pre-norm normalizes the input to the sublayer and leaves the residual stream alone:

x = x + sublayer(layer_norm(x))

In a pre-norm model, a final layer norm is usually applied once after the last block, just before the output layer, because the residual stream itself is never normalized inside the stack.

The difference sounds cosmetic, but it changes training. In pre-norm, the residual path from the input to the output is a clean identity, with no normalization sitting on it. Gradients flow down that highway unchanged, so very deep stacks train stably, usually even without the careful learning-rate warmup that post-norm needs. In post-norm, every block's output is re-normalized, so the original signal is squeezed through a layer norm at every step, gradients near the bottom of a deep stack can become small, and warmup is close to mandatory. Post-norm can reach slightly better results when it trains successfully, but pre-norm is the safer default and the one almost all large models use.

You can see the effect of the identity path directly by running the same random input through 24 blocks in each layout and asking how much of the original input survives.

Python 24 blocks: pre-norm versus post-norm

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Post-norm pins the size of every vector at exactly 1.0 (the root-mean-square, or rms, of a layer-normalized vector with no scale parameters), so you cannot see the stack accumulating anything. Pre-norm lets the residual stream grow as each block adds its contribution, from about 1.1 after the first block to about 2.4 after the last. The similarity score, a cosine between the normalized input and normalized output, is the telling number: 0.55 for pre-norm versus 0.18 for post-norm. After 24 blocks, the pre-norm output is still strongly correlated with what went in, because the identity path carried it along. In post-norm most of that trace has been washed out. This is the same reason residual connections help CNNs (see Classic CNN Architectures).

Pre-norm has a side effect worth knowing: because the stream's size keeps growing with depth, deep pre-norm models often scale down the initialization of the output projections (weight initialization matters here) so that early training does not start with a huge residual stream.

Counting parameters

Where do a transformer's parameters live? Each block has four d x d attention matrices (Wq, Wk, Wv, Wo), which is 4d^2 numbers. The MLP has d x 4d and 4d x d matrices, which is 8d^2. Add up the pieces and a block holds about 12 d^2 weights; biases and layer-norm scales add only a few multiples of d. The number of heads does not change this at all, because heads split d among themselves instead of adding width.

Python Parameters in a block

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For d = 64 the exact total is 49,728, matching 12 d^2 + 9 d. Scaled up, a stack of 12 blocks at width 768 holds roughly 85 million block parameters, and 32 blocks at width 4096 holds over 6 billion. Real models also have an embedding table and an output layer that add more, but the blocks dominate. Width and depth are the two knobs that scale a transformer.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a causal mask do inside a decoder block?

    Scores for positions after the current one are set to a very large negative number before the softmax, so their weights become zero. That stops a token from seeing the answer it is supposed to predict.
  2. 2. Which sublayer is the only one that moves information between different token positions?

    The MLP, the layer norm and the residual additions all act on one token at a time. Attention is where tokens exchange information.
  3. 3. What is the main difference between pre-norm and post-norm?

    In pre-norm, x = x + sublayer(layer_norm(x)), which keeps a clean identity path for gradients. In post-norm, x = layer_norm(x + sublayer(x)).
  4. 4. Roughly how many weights does one transformer block with model width d contain (ignoring biases)?

    Four d-by-d attention matrices give 4 d^2, and the two MLP matrices with a 4d hidden layer give 8 d^2, for 12 d^2 in total.
  5. 5. Why can a decoder-only language model get many training examples from one pass over a sequence?

    With the mask, position i only sees tokens up to i, so predicting token i+1 from it is a fair task at every position simultaneously.

Summary

  • A transformer block maps (T, d) to (T, d) using two sublayers, multi-head attention and a position-wise MLP, each with a residual connection and layer norm.
  • Attention mixes information across tokens; the MLP transforms each token independently.
  • Encoder blocks attend in both directions; decoder blocks add a causal mask (scores set to a huge negative number before the softmax) so each token sees only the past.
  • Pre-norm (x + sublayer(norm(x))) keeps a clean identity path and trains deep stacks more stably than post-norm (norm(x + sublayer(x))).
  • A block has about 12 d^2 parameters, independent of the number of heads; width and depth are the scaling knobs.

Next: Training a Tiny Language Model.