Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 37 ยท Transformer Internals

Feed-Forward Layers, Residuals and LayerNorm

Learn the rest of a Transformer block: the feed-forward MLP with GELU or SwiGLU, residual connections, LayerNorm and RMSNorm, and pre-norm versus post-norm in NumPy.

Attention moves information between tokens. The rest of a Transformer block does the heavy computation inside each token's vector and keeps training stable. There are three parts: a position-wise feed-forward network, residual connections, and normalization. Together with attention they form the block that is repeated dozens of times in an LLM.

The feed-forward network (FFN)

After attention, each token's vector is processed independently by a small two-layer network: expand to a wider space, apply a nonlinearity, project back.

FFN(x) = W2 ยท activation(W1 ยท x + b1) + b2

The hidden width is typically 4 times d_model (or about 2.7 times in gated variants). Because it is applied to each position separately and uses the same weights for every position, it is also called a position-wise MLP. This is where most of a model's parameters live (8 ยท dยฒ per layer versus 4 ยท dยฒ for attention), and research suggests it stores much of the factual knowledge: it acts like a key-value memory in which the first matrix detects patterns and the second writes out associated content.

Python A position-wise feed-forward layer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The final check confirms the layer has no interaction between tokens: that job belongs entirely to attention.

Activation functions

Early Transformers used ReLU, then GELU, a smooth variant. Many recent models use a gated form such as SwiGLU: two parallel up-projections, one passed through the Swish (SiLU) function and used as a gate for the other, then multiplied elementwise and projected down. It consistently trains a bit better. To keep the parameter count comparable the hidden size is reduced to about two thirds of 4 times d.

Python Compare activations

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Residual connections

Each sub-layer is wrapped so that its output is added to its input:

x = x + Sublayer(x)

This residual (skip) connection gives gradients a direct path from the loss back to the early layers, which is what makes networks with dozens or hundreds of layers trainable. It also gives a useful mental model: the vector for a token is a residual stream that each layer reads from and writes small updates into, rather than something each layer replaces.

Python Signal through many layers, with and without a residual

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The plain stack collapses toward zero (information and gradient vanish), while the residual stack keeps the original signal and adds to it.

Normalization

Activations can drift to very different scales across layers and tokens. Layer normalization rescales each token's vector to zero mean and unit variance across its features, then applies a learned scale ฮณ and shift ฮฒ:

LayerNorm(x) = ฮณ ยท (x โˆ’ mean) / sqrt(variance + ฮต) + ฮฒ

Modern LLMs often use RMSNorm, which skips the mean subtraction and divides by the root-mean-square only. It is slightly cheaper and works just as well.

Python LayerNorm and RMSNorm

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pre-norm versus post-norm

The original Transformer applied normalization after the residual addition (post-norm). Almost all modern LLMs apply it before the sub-layer (pre-norm):

  • post-norm: x = Norm(x + Sublayer(x))
  • pre-norm: x = x + Sublayer(Norm(x))

Pre-norm leaves the residual stream untouched by normalization, which keeps gradients healthy in very deep models and lets training use higher learning rates without careful warm-up. A final norm is applied after the last block, before the output projection.

flowchart TB
    X["x"] --> N1["Norm"]
    N1 --> A["Attention"]
    A --> P1(("+"))
    X --> P1
    P1 --> N2["Norm"]
    N2 --> F["Feed-forward"]
    F --> P2(("+"))
    P1 --> P2
    P2 --> Y["x (next block)"]

A pre-norm block: two sub-layers, each with a norm before and a residual add after.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the position-wise feed-forward network do?

    Interaction between tokens is attention's job; the FFN transforms each token on its own.
  2. 2. Why are residual connections important in deep networks?

    x + Sublayer(x) preserves the signal and keeps gradients flowing through many layers.
  3. 3. What is RMSNorm compared to LayerNorm?

    RMSNorm drops mean-centering and is slightly cheaper.
  4. 4. Where does pre-norm place the normalization?

    Pre-norm computes x + Sublayer(Norm(x)), leaving the residual stream un-normalized.

Summary

  • The FFN expands each token to about 4ร—d, applies GELU or SwiGLU, and projects back; it holds most parameters.
  • Residual connections let each layer add updates to a residual stream and keep gradients healthy.
  • LayerNorm or RMSNorm keeps activations at a stable scale.
  • Modern LLMs use pre-norm blocks with a final norm.

Next: The Decoder-Only Architecture End to End