Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 53 ยท Inside the Transformer

Context Windows and the KV Cache

Learn what a context window is, why attention cost grows with length, why models lose the middle, and how to compute KV cache memory yourself.

Every language model has a context window: the maximum number of tokens it can look at in one go. Your prompt, the conversation so far, any retrieved documents and the answer the model is writing all have to fit inside it. When people say a model has a "128k context", they mean 128,000 tokens shared between input and output.

Why is that number limited, and why does a long prompt cost more and respond more slowly? The answers come from the attention mechanism you met in Attention from Scratch and from a trick called the KV cache. In this lesson you will count the work attention does and calculate the memory a cache needs, with plain Python.

Attention compares every token with every other

In attention, each token produces a query that is scored against the key of every token it is allowed to see. With n tokens that is about n ร— n scores per head and per layer. A causal model (one that only looks backward, like a chat model) needs roughly half of that: n ร— (n + 1) / 2.

Double the prompt length and the number of scores grows four times. This is called quadratic growth, and it is the main reason early models had windows of only a few thousand tokens.

Python How the score count grows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A 128ร— longer prompt means about 16,000 times more raw scores. Real systems use clever kernels (such as FlashAttention) that avoid storing the whole score matrix, plus tricks like sliding windows, so the practical cost is lower. But the trend is the same: longer context means more computation.

What fills the window?

System prompt Rules and persona.
History Earlier turns.
Documents Retrieved or pasted text.
The answer Output tokens count too.

All four share one budget.

1 / 2

Generating one token at a time

A model produces its answer one token at a time. To pick token number 501, it needs the keys and values of the 500 tokens before it. Recomputing them for every new token would waste enormous effort, because the keys and values of old tokens never change.

So the model stores them. This store is the KV cache (K for keys, V for values). Generation then has two phases:

  • Prefill: the whole prompt is processed at once, in parallel, and its keys and values are written to the cache. This is why the pause before the first word grows with prompt length.
  • Decode: each new token computes only its own query, key and value, reads the cache, and appends its new key and value.
flowchart LR
    P["Prompt tokens"] --> F["Prefill:<br/>compute all K and V"]
    F --> C["KV cache"]
    C --> D["Decode one token"]
    D -->|"append new K and V"| C
    D --> O["Output token"]

Prefill fills the cache once; every decode step reads it and adds one entry.

How much memory does the cache need?

For every token, every layer stores one key vector and one value vector for each KV head. The formula is:

bytes per token = 2 ร— layers ร— kv_heads ร— head_dim ร— bytes per number

The 2 is for keys plus values. Total memory is that number times the tokens held in the cache, times the number of conversations being served at the same time.

Python A KV cache calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The medium shape needs 128 KiB per token, so a full 8,192-token conversation holds exactly 1 GiB of cache. A server that handles 50 such conversations at once spends 50 GiB on caches alone, on top of the model weights. That is why providers charge more for long contexts and why they limit how many requests run in parallel.

Why "kv_heads" is smaller than the attention heads

Early transformers used one key/value head per query head (multi-head attention). Modern models often use grouped-query attention (GQA): many query heads share a few KV heads. The quality loss is small and the cache shrinks a lot. The extreme version, a single shared KV head, is called multi-query attention.

Python Attention variants and number formats

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Going from 32 to 8 KV heads cuts the cache by 4ร—; storing the cache in 8-bit numbers halves it again. Both are standard levers in real serving systems.

Context limits are not memory limits

A model can only handle positions it was trained on. Position information (see The Transformer Block) is learned or designed for a certain length, and going beyond it often produces garbage. Techniques that rescale rotary positions can stretch a model to longer inputs, usually with a short extra training phase.

Also, the window is shared. If a model has a 16k window and you send a 15k prompt, you only have about 1k tokens left for the answer. Many APIs expose a separate max_tokens setting for the output, and you must keep input + output under the window.

Lost in the middle

Fitting a document in the window does not mean the model uses it equally well. Experiments with long prompts have repeatedly shown a "lost in the middle" pattern: facts near the beginning and end of the prompt are used more reliably than facts buried in the middle. Newer models are much better, but the effect has not vanished, especially with distracting text around the key fact.

Practical habits that follow from this:

  • Put the instructions and the most important facts at the start or end, not in the middle of a huge paste.
  • Send less, but more relevant text. This is the motivation for retrieval, covered later in the RAG chapter.
  • Repeat the actual question after a long document.
  • Test with your own data: ask for a fact from the beginning, middle and end and compare.

A long context is a budget, not a goal. Every extra token costs money and time, and it can distract the model.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. If you double the prompt length, roughly how many attention scores does a full attention layer compute?

    Every token is compared with every other token, so the work grows with n squared. Doubling n multiplies it by four.
  2. 2. What does the KV cache store?

    Keys and values of past tokens never change, so decoding stores them and only computes the new token's entries.
  3. 3. A model has 32 layers, 8 KV heads, head dimension 128 and stores fp16 values. How big is the cache per token?

    2 x 32 x 8 x 128 x 2 bytes = 131,072 bytes, which is 128 KiB.
  4. 4. Which habit best addresses the lost-in-the-middle effect?

    Less, more relevant text and careful placement make it easier for the model to use the important information.

Summary

  • The context window is the token budget shared by instructions, history, documents and the answer.
  • Attention work grows roughly with the square of the length; long prompts cost more and slow the first token (prefill).
  • The KV cache keeps keys and values of past tokens so each new token is cheap, but it uses memory per token: 2 ร— layers ร— kv_heads ร— head_dim ร— bytes.
  • GQA and lower-precision caches shrink that memory.
  • Models can use the middle of a long prompt less reliably, so send focused context.

Next, see how the model turns its final numbers into a choice of token in Logits, Softmax and Temperature.