Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 37 · Transformer Internals

The Decoder-Only Architecture End to End

Assemble embeddings, causal attention, feed-forward layers and an output head into a complete decoder-only Transformer in NumPy, then count its parameters exactly.

You now know every part of a GPT-style model: token embeddings, position information, masked multi-head attention, feed-forward layers, residuals and normalization. This lesson wires them together and runs a full forward pass, from token ids to a probability for every possible next token. The model here has random weights (training comes in the next chapter), but the computation is exactly what a real decoder does, only smaller.

The full stack

flowchart TB
    T["token ids"] --> E["Embedding lookup"]
    E --> B1["Block 1<br/>norm, causal attention, add<br/>norm, feed-forward, add"]
    B1 --> B2["Block 2 ... Block L"]
    B2 --> FN["Final norm"]
    FN --> H["Output head<br/>(vocab scores = logits)"]
    H --> S["Softmax<br/>next-token probabilities"]

Data flows up through L identical blocks. Only the last position's output is used to choose the next token.

Each position produces its own logits. During training all positions are scored at once; during generation we only need the last one.

Python A complete tiny decoder, forward pass

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With random weights the distribution is nearly uniform (every token has probability close to 1/20 = 0.05). Training is the process of moving it to something sharp and correct.

Counting parameters exactly

Every learned array in the model is counted by adding its sizes. For one block with width d and FFN width d_ff:

  • attention: 4 · d² (Wq, Wk, Wv, Wo; biases omitted as in many modern models)
  • FFN: 2 · d · d_ff
  • norm scales: a few d (tiny)

The embedding table adds V · d (and max_len · d if positions are learned). With a tied output head the output layer is free.

Python Count parameters in code and by formula

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For the 7B shape the two figures are identical, because with d_ff = 4d the block parameters are exactly 4d² + 8d² = 12d². Production models deviate slightly (gated FFNs, grouped-query attention, untied output matrices), which is why published counts differ a little.

Generating text with the forward pass

Generation repeats the forward pass: run the model on the tokens so far, take the logits of the last position, choose a token, append it, repeat. This naive version recomputes everything each time; real systems cache the keys and values of earlier positions (the KV cache, a topic of the inference chapter) so each step costs only one new position.

Python Autoregressive generation loop

Press Run (or Ctrl + Enter) to see the result here.

Program input · what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output is meaningless because the weights are random, but the loop is real. Everything a chatbot does is this loop plus a trained set of weights and a better way to choose tokens.

A decoder in one slide

Input Token ids plus positions.
L blocks Attention then feed-forward, each with norm and residual.
1 / 2

Check your understanding

Quick check · 4 questions

of 4 answered You scored / 4 · perfect!

  1. 1. Which position's logits are used to choose the next token during generation?

    The last position has seen all the tokens so far and predicts what comes next.
  2. 2. With d_ff = 4d, how many parameters does one block have (no biases)?

    Attention contributes 4d² and the FFN 2·d·4d = 8d², totaling 12d².
  3. 3. What does an untrained model's next-token distribution look like?

    Random weights carry no information, so every token is roughly equally likely.
  4. 4. What does weight tying save?

    The output head reuses the transpose of the token embedding matrix.

Summary

  • A decoder is embeddings plus L pre-norm blocks (causal attention, feed-forward), a final norm, and an output head producing logits.
  • Training scores all positions in parallel; generation uses the last position's logits and loops.
  • Block parameters are 4d² for attention plus 2·d·d_ff for the FFN, so about 12d² when d_ff = 4d.
  • An untrained model is near uniform; learning sharpens the distribution.

Next: Attention Variants: GQA, FlashAttention and Mixture of Experts