Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 45 ยท Sequences and Transformers

Building a Transformer Block

Assemble a transformer block in PyTorch: attention, residual connections, LayerNorm, feed-forward layers, a causal LM head and parameter counting.

In the previous lesson you built attention, the operation that lets every position in a sequence look at every other position. Attention alone is not a model, though. A transformer block wraps attention with a few other ingredients (residual connections, normalization and a small feed-forward network) so that dozens of blocks can be stacked and still train. Almost every modern language model is a tall pile of nearly identical blocks.

This lesson shows the block in real PyTorch (read-only, since PyTorch cannot run on this site), then rebuilds the same computation in NumPy so you can run it, test its causal mask and count its parameters. The NumPy block is a stand-in: it has the same math and the same shapes as the PyTorch layer, but random weights and no training.

One block, four parts

Attention Mixes information across positions.
Feed-forward Transforms each position on its own.
Residuals Add the input back to the output.
LayerNorm Keeps activations well scaled.

Attention talks between tokens, the feed-forward network thinks within a token.

1 / 2

The shape of a block

A block takes a tensor of shape (batch, seq_len, d_model) and returns a tensor of exactly the same shape. That is what lets you stack as many as you like. Inside, there are two sub-layers, each wrapped in a residual connection and a LayerNorm.

flowchart TD
    X["x: batch, seq, d_model"] --> N1["LayerNorm"]
    N1 --> A["Multi-head self-attention"]
    A --> P1(("+"))
    X --> P1
    P1 --> N2["LayerNorm"]
    N2 --> F["Feed-forward: Linear, GELU, Linear"]
    F --> P2(("+"))
    P1 --> P2
    P2 --> Y["y: batch, seq, d_model"]

The "pre-norm" layout: normalize, apply a sub-layer, then add the original input back.

The two details that make deep stacks trainable:

  • Residual connections compute x + sublayer(x). The gradient has a direct path through the addition, so early layers still receive a useful signal even when 24 blocks sit above them (this is the vanishing-gradient cure you met in the RNN lesson).
  • LayerNorm normalizes each token's vector over the feature dimension (mean 0, variance 1, then a learned scale and shift). Unlike BatchNorm it does not depend on the batch, so it behaves the same in training and inference. See Initialization, Normalization and Dropout.

The feed-forward network is two linear layers with a nonlinearity between them. It first widens each token's vector (typically to 4 times d_model) and then narrows it back. It is applied to every position independently, with the same weights.

nn.TransformerEncoderLayer

PyTorch ships a ready-made block. This is the usual way to use it:

Python
import torch
from torch import nn

layer = nn.TransformerEncoderLayer(
    d_model=64,          # size of each token vector
    nhead=4,             # attention heads, d_model must be divisible by nhead
    dim_feedforward=256, # width of the hidden feed-forward layer
    dropout=0.1,
    activation="gelu",
    batch_first=True,    # inputs are (batch, seq, d_model)
    norm_first=True,     # pre-norm layout, as in the diagram
)
encoder = nn.TransformerEncoder(layer, num_layers=4)

x = torch.randn(8, 20, 64)   # 8 sequences, 20 tokens, 64 features
y = encoder(x)
print(y.shape)               # torch.Size([8, 20, 64])

Two flags are easy to miss. batch_first=True is not the default (the default expects (seq, batch, d_model)), and norm_first=False is the default, which gives the original "post-norm" layout LayerNorm(x + sublayer(x)). Post-norm works but is touchier to train when deep, so most current models use pre-norm.

Masks make a language model

An encoder lets every token see every other token. A language model predicts the next token, so token t must not see tokens after t, otherwise it could just copy the answer. You enforce that with a causal mask: a triangular matrix that sets the attention scores of future positions to minus infinity before the softmax, so their weights become exactly zero.

Python
seq_len = 20
mask = nn.Transformer.generate_square_subsequent_mask(seq_len)
# a (20, 20) float tensor: 0 on and below the diagonal, -inf above it

y = encoder(x, mask=mask, is_causal=True)

Putting it together gives a tiny GPT-style model: an embedding table, learned position vectors, a stack of masked blocks, a final LayerNorm and a linear head that turns each vector into one score (a logit) per vocabulary word.

Python
class TinyLM(nn.Module):
    def __init__(self, vocab_size, d_model=64, nhead=4, layers=4, max_len=128):
        super().__init__()
        self.tok = nn.Embedding(vocab_size, d_model)
        self.pos = nn.Embedding(max_len, d_model)
        block = nn.TransformerEncoderLayer(
            d_model, nhead, 4 * d_model, dropout=0.0,
            activation="gelu", batch_first=True, norm_first=True,
        )
        self.blocks = nn.TransformerEncoder(block, layers)
        self.norm = nn.LayerNorm(d_model)
        self.head = nn.Linear(d_model, vocab_size, bias=False)
        self.head.weight = self.tok.weight   # weight tying: share the table

    def forward(self, ids):                  # ids: (batch, seq) of integers
        seq = ids.shape[1]
        positions = torch.arange(seq, device=ids.device)
        h = self.tok(ids) + self.pos(positions)
        mask = nn.Transformer.generate_square_subsequent_mask(seq).to(ids.device)
        h = self.blocks(h, mask=mask, is_causal=True)
        return self.head(self.norm(h))       # (batch, seq, vocab) logits

# Training: predict token t+1 from tokens up to t
# logits = model(ids[:, :-1])
# loss = nn.functional.cross_entropy(logits.flatten(0, 1), ids[:, 1:].flatten())

Look at the last two comment lines: the inputs are the sequence without its last token, the targets are the same sequence shifted by one. With the causal mask, a single forward pass trains all positions at once. Weight tying (the head reuses the embedding matrix) saves vocab_size * d_model parameters and usually helps small models.

A block you can run

The editor below is the same block in NumPy: LayerNorm, multi-head causal self-attention, a GELU feed-forward network and two residual additions, in pre-norm order. It then proves the causal mask works: if you change the last token, the outputs for earlier tokens must not move at all.

Python A pre-norm transformer block in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Only the last position reacts to the edit when the mask is on. Without the mask, information flows backwards from the last token to all of the others, which would let a language model cheat during training.

The full encoder-decoder transformer

The original transformer, built for translation, has two stacks. The encoder reads the whole input sentence and turns it into a sequence of context-rich vectors. The decoder generates the output sentence one token at a time, looking both at what it has produced so far and at the encoder's vectors.

flowchart LR
    S["source tokens"] --> EE["embeddings + positions"]
    EE --> ENC["encoder stack<br/>(self-attention + FFN) x N"]
    ENC -->|"keys and values"| DEC
    T["target tokens so far"] --> DE["embeddings + positions"]
    DE --> DEC["decoder stack<br/>(masked self-attention,<br/>cross-attention, FFN) x N"]
    DEC --> L["linear + softmax"]
    L --> O["next token"]

The encoder processes the source once; the decoder runs step by step and consults the encoder through cross-attention.

Inside an encoder block

Each encoder block has multi-head self-attention (every source position looks at every other) followed by a position-wise feed-forward network (FFN), each wrapped with a residual connection and layer normalization. The FFN applies the same two linear layers, FFN(x) = W2 * GELU(W1 * x + b1) + b2, to every position independently. Its inner width is usually four times d_model; it acts like a per-token memory that stores and transforms what attention gathered.

Inside a decoder block

A decoder block has three sub-layers, in this order:

  1. Masked self-attention over the target tokens generated so far. The causal mask forbids looking at later positions, so training with the whole target at once still matches generation, where the future does not exist.
  2. Encoder-decoder (cross) attention. The queries come from the decoder, the keys and values come from the encoder output. This is where the decoder decides which source words matter for the word it is writing now.
  3. A position-wise feed-forward network.

Each sub-layer again has a residual connection and normalization. The decoder ends with a linear layer to vocabulary size and a softmax.

Mathematical form. With queries Q, keys K and values V, one attention step is softmax(Q K^T / sqrt(d_k) + mask) V, covered in Attention. In self-attention Q, K, V are projections of the same sequence; in cross-attention Q = X_dec W_q while K = X_enc W_k and V = X_enc W_v. The attention matrix therefore has shape (target length, source length): one row of source weights per target position. Multi-head attention runs several of these in parallel on slices of d_model and concatenates the results.

PyTorch provides all the pieces:

Python
import torch.nn as nn

d_model, nhead = 256, 8
enc_layer = nn.TransformerEncoderLayer(d_model, nhead, dim_feedforward=1024, batch_first=True)
dec_layer = nn.TransformerDecoderLayer(d_model, nhead, dim_feedforward=1024, batch_first=True)
encoder = nn.TransformerEncoder(enc_layer, num_layers=4)
decoder = nn.TransformerDecoder(dec_layer, num_layers=4)

memory = encoder(src_embedded)                                   # (batch, src_len, d_model)
tgt_mask = nn.Transformer.generate_square_subsequent_mask(tgt_len)
out = decoder(tgt_embedded, memory, tgt_mask=tgt_mask)           # (batch, tgt_len, d_model)

nn.Transformer bundles an encoder and a decoder in one module. At inference you start with a start token, run the decoder, take the most likely next token, append it and repeat until an end token appears. The memory (encoder output) is computed once and reused every step.

Python Cross-attention: decoder queries over encoder outputs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The weights are random here, so which source word gets the most attention means nothing; in a trained translator the row for "chat" would peak on "cat". The shapes and the causal pattern are the real part.

Encoder-only, decoder-only and encoder-decoder models

Family Uses Reads context Examples Best at
Encoder-only Encoder stack Both directions, whole input at once BERT Understanding: classification, tagging, search embeddings
Decoder-only Masked decoder stack Left to right GPT family, most chat models Generating text
Encoder-decoder Both Encoder both ways, decoder left to right The original transformer, T5, BART Translation, summarization

BERT is trained by masking random words and predicting them (masked language modeling), so it uses context from both sides. GPT-style models predict the next token. A Seq2Seq model in general (see Autoencoders) is any encoder-decoder pair, recurrent or attention-based.

Counting parameters

Knowing where the parameters live tells you where memory and compute go. For one block with width d and feed-forward width f:

Part Parameters
Attention: query, key, value projections 3 * d * d + 3 * d
Attention: output projection d * d + d
Feed-forward: two linear layers 2 * d * f + f + d
Two LayerNorms (gain and bias each) 4 * d

The attention weights are 4dยฒ and the feed-forward weights are 2df; with the usual f = 4d that is 8dยฒ, so about two thirds of a block lives in the feed-forward network. The formula below is what PyTorch's TransformerEncoderLayer reports, and you can check it against sum(p.numel() for p in layer.parameters()).

Python Count the parameters of a tiny language model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Counting is not just trivia. A parameter in float32 takes 4 bytes, and the optimizer states of Adam take two more floats per parameter, so training needs roughly 16 bytes per parameter before you count activations. That arithmetic decides whether a model fits on your GPU (see GPUs, Devices and Mixed Precision).

Common mistakes

  • Forgetting batch_first=True and feeding (batch, seq, d) data to a layer that expects (seq, batch, d). Nothing crashes, the model just attends across the wrong axis.
  • Using a language model without a causal mask. Training loss drops suspiciously fast, because every token can read its own target.
  • Choosing d_model not divisible by nhead. PyTorch raises an error because the heads must split the vector evenly.
  • Forgetting positional information. Attention has no sense of order, so without position embeddings the model sees a bag of tokens.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What do the residual connections in a transformer block compute?

    The addition gives gradients a direct path to earlier layers, which is why deep stacks of blocks remain trainable.
  2. 2. Why does a language model need a causal mask?

    Without the mask, position t could look at token t+1 (its own target) and training would learn to copy instead of predict.
  3. 3. In an nn.TransformerEncoderLayer with d_model=64 and nhead=4, how large is each attention head?

    The model width is split across heads: 64 / 4 = 16 features per head, which is why d_model must be divisible by nhead.
  4. 4. Which part of a block holds most of its parameters when the feed-forward width is 4 times d_model?

    Attention has four d-by-d matrices (4d squared), the feed-forward layers have two d-by-4d matrices (8d squared).

Summary

  • A transformer block maps (batch, seq, d_model) to the same shape, so blocks stack freely.
  • Each block is attention plus a feed-forward network, each wrapped in LayerNorm and a residual connection (pre-norm is the modern default, norm_first=True).
  • nn.TransformerEncoderLayer and nn.TransformerEncoder implement it; use batch_first=True and pass a causal mask for language models.
  • A causal LM adds token and position embeddings, a final LayerNorm and a linear head over the vocabulary; targets are the inputs shifted by one.
  • Per block, roughly 12 dยฒ parameters when f = 4d; weight tying saves the head's vocab * d parameters.

Next, learn to adapt a big pretrained model cheaply in Fine-Tuning, PEFT and LoRA, then keep what you trained in Saving and Loading Models.