Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 37 ยท Transformer Internals

Attention Variants: GQA, FlashAttention and Mixture of Experts

Understand multi-query and grouped-query attention, FlashAttention tiling, sliding-window attention and mixture-of-experts routing, with runnable NumPy examples.

Standard attention is powerful but expensive. Its score matrix grows with the square of the sequence length, and generating text requires remembering a growing cache of keys and values. Much of modern LLM engineering is a set of targeted fixes: share keys across heads, compute attention without ever materializing the full matrix, limit how far each token looks, and make feed-forward layers sparse. This lesson explains each idea and the arithmetic behind it.

The cost of standard attention

For sequence length n and width d, the score matrix has nยฒ entries per head, and computing it costs O(nยฒ ยท d). At 100,000 tokens a single head's score matrix would hold 10 billion numbers. Generation adds a second cost: for each new token the model must read all earlier keys and values, which are stored in the KV cache.

Python How attention memory grows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

At 128,000 tokens one head's matrix would need over 30 GB. This is why real implementations never build it in full.

Multi-query and grouped-query attention

In multi-head attention (MHA) every head has its own keys and values, so the KV cache stores 2 ยท layers ยท heads ยท d_head numbers per token. Multi-query attention (MQA) shares one key head and one value head across all query heads. Grouped-query attention (GQA) is in between: query heads are divided into groups and each group shares one key/value head. GQA is used by most recent open models because it cuts the cache several-fold with little quality loss.

Python KV cache size: MHA, GQA and MQA

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For this 7B-class shape the cache for one 32K-token sequence falls from about 17 GB to about 4 GB with 8 key/value heads. It matters because at serving time the cache, not the weights, often limits how many users a GPU can handle.

Python Grouped-query attention in NumPy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

FlashAttention: same math, less memory traffic

On a GPU, attention is slow mainly because it moves data between large slow memory and small fast memory, not because of arithmetic. FlashAttention computes exactly the same result but processes K and V in tiles, keeping a running maximum and a running sum so that softmax can be finished incrementally (online softmax). The n ร— n matrix is never written to slow memory. The speedup comes from IO awareness, not from approximation. You can verify the key trick, that a tiled online softmax matches the ordinary one:

Python Online (tiled) softmax equals normal softmax

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The difference is on the order of floating-point rounding. FlashAttention applies this idea inside a GPU kernel, so it is exact, faster, and uses memory linear in n.

Sliding-window and sparse attention

Another way to cut cost is to let each token attend only to its last w tokens (sliding-window attention). The cost becomes O(n ยท w). Information still travels farther across layers: after k layers a token can be influenced by tokens up to k ยท w positions back. Some models alternate local layers with a few global layers.

Python Sliding-window mask

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Mixture of experts

In a mixture-of-experts (MoE) layer the single feed-forward network is replaced by E parallel experts plus a small router. For each token the router scores the experts and sends the token to the top k (often 1 or 2); their outputs are combined using the router's weights. Total parameters are about E times larger, but each token only uses k experts, so compute per token stays near that of a small dense model.

Python Top-k expert routing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real MoE training adds an auxiliary loss to keep expert load balanced, otherwise a few experts receive nearly all tokens.

Making attention affordable

GQA Share key and value heads to shrink the KV cache.
FlashAttention Exact attention in tiles with far less memory traffic.
1 / 2

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does grouped-query attention share?

    Several query heads use the same key and value head, shrinking the KV cache.
  2. 2. Is FlashAttention an approximation of attention?

    It uses online softmax in tiles; results match standard attention up to rounding.
  3. 3. What is the compute cost of sliding-window attention with window w?

    Each of n tokens looks at only w tokens.
  4. 4. What is true of mixture-of-experts layers?

    Only the top-k experts run, but all experts must still be stored.

Summary

  • Standard attention costs O(nยฒ) in compute and score memory; generation needs a growing KV cache.
  • MQA and GQA share key/value heads, cutting the cache several-fold.
  • FlashAttention is exact: tiled online softmax avoids writing the nร—n matrix to slow memory.
  • Sliding-window attention limits each token to recent context.
  • Mixture-of-experts adds capacity while activating only a few experts per token.

Next: Pretraining Objective and Cross-Entropy