Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 38 ยท Getting Started

The Transformer Architecture in One Lesson

Understand how transformers work: embeddings, self-attention, encoder, decoder and encoder-decoder models, with a runnable NumPy attention example.

Before you load pretrained models it helps to know what is inside them. You do not need the full mathematics. You need four ideas: tokens become vectors, attention mixes information between tokens, stacked layers refine the result, and a small head turns the final vectors into the answer you want.

From text to vectors

A model cannot read text. A tokenizer cuts the text into pieces and maps each piece to an integer id. An embedding table then replaces each id with a learned vector of, say, 768 numbers. Because attention by itself does not know word order, the model also adds position information to each vector.

flowchart LR
  A["Text"] --> B["Tokenizer"]
  B --> C["Token ids"]
  C --> D["Embeddings + positions"]
  D --> E["N transformer layers"]
  E --> F["Hidden vectors"]
  F --> G["Task head"]
  G --> H["Prediction"]

Every Hugging Face model follows this path; only the layers and the head change.

Self-attention

Each token produces three vectors: a query (what am I looking for?), a key (what do I contain?) and a value (what do I give away?). A token compares its query with all keys using a dot product, turns the scores into weights with softmax, and takes a weighted sum of the values. The result is a new vector for that token that blends in context.

The formula is softmax(Q K^T / sqrt(d)) V, where d is the key size. Dividing by sqrt(d) keeps the scores from growing too large.

The runnable code below is a stand-in: a single attention head in plain NumPy with tiny random weights, not a trained model. It shows the mechanics, not meaningful language understanding.

Python Stand-in: scaled dot-product attention

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every row of the weight matrix sums to 1: each token distributes its attention across all tokens, including itself. Real models run many heads in parallel (multi-head attention) so different heads can follow different relationships, for example grammar in one head and references in another.

Masks: what a token may see

A mask blocks some attention scores by setting them to a very negative number before the softmax. Two masks matter:

  • Padding mask: stops tokens from attending to padding added to make a batch rectangular.
  • Causal mask: in a text-generating model, token i may only see tokens up to i, otherwise it would copy the answer from the future.
Python Stand-in: causal mask

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Row 0 can only see itself (weight 1). Row 4 spreads its attention equally over the five visible tokens.

The three layouts

Encoder, decoder, or both

Encoder only Reads the whole text both ways. BERT.
Decoder only Writes left to right. GPT, Llama.
Encoder-decoder Read, then write. T5, BART.

The layout decides which tasks the model suits.

1 / 2
  • Encoder-only models (BERT, RoBERTa, DistilBERT) have no causal mask. Every token sees every other token, which suits classification, tagging and search embeddings.
  • Decoder-only models (GPT-style, Llama, Mistral, Qwen) use the causal mask and predict the next token. Today most large language models use this layout.
  • Encoder-decoder models (T5, BART, Whisper) encode the input, then a decoder generates the output while also attending to the encoder.

The rest of a layer

After attention each layer applies a small feed-forward network to every token separately, with residual connections (the layer's input is added back to its output) and layer normalization to keep training stable. Stack 6, 12, 32 or more such layers and you have a transformer. The task head sits on top: a linear layer producing class scores, or a layer producing a probability over the whole vocabulary for the next token.

Pretraining and fine-tuning

Models are first pretrained on huge unlabelled text with a self-supervised goal: guess masked words (encoders) or the next token (decoders). Then you fine-tune them on your own, much smaller, labelled data. Because the pretrained model already knows language, a few thousand examples can be enough. Chapter 5 shows how.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the softmax in attention produce?

    Softmax turns the query-key scores of a token into positive weights that add up to 1.
  2. 2. Which model family uses a causal mask and predicts the next token?

    Decoder-only models such as GPT and Llama can only attend to earlier positions.
  3. 3. Why is attention divided by the square root of the key size?

    Large scores make softmax almost one-hot with tiny gradients. Scaling keeps them in a useful range.
  4. 4. BERT is best described as:

    BERT stacks encoder layers and sees the whole sentence in both directions.

Summary

  • Text becomes token ids, then embedding vectors plus position information.
  • Self-attention mixes context into each token using queries, keys and values.
  • Masks control visibility: padding masks hide padding, causal masks hide the future.
  • Encoder-only, decoder-only and encoder-decoder layouts suit different tasks; a head on top gives the final output.

Next: Your First Pipeline