Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 38 ยท Retrieval and Context Engineering

The Context Window and Token Budgets

Learn what a context window is, how tokens are counted, and how to split a prompt budget between instructions, context, history and the answer.

A language model does not remember your project, your users or yesterday's chat. It only sees the text you send in one request. That text, plus the answer it writes back, must fit inside the context window. Everything in this chapter, from retrieval to memory, is about one question: what should go into that limited space?

Prompt writing in the earlier chapters was about wording. Context engineering is about selection: which documents, which history, which examples, in what order, and how much of each. A perfect instruction wrapped around the wrong context still gives a wrong answer.

No real model runs on this site. Every code block here is plain Python that shows the logic with a clearly labelled stand-in, so you can see budgets and selection working without an API key.

Tokens, not words

Models read tokens: pieces of text that are often a short word, part of a longer word, a number chunk or a punctuation mark. A rough rule for English is that one token is about four characters, or three quarters of a word. Code, other languages and unusual words use more tokens per character. Providers publish exact tokenizers; the estimate below is only a planning tool.

Python Stand-in: estimating tokens

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real tokenizers disagree with this estimate by 10-30 percent, so always leave a safety margin and use the provider's counting function when exactness matters.

What counts against the window

The window covers input and output together. A model with a 100,000-token window cannot take 100,000 tokens of input and then write a reply. Everything below competes for the same space:

  • the system prompt and standing rules
  • few-shot examples (covered in the techniques chapter)
  • retrieved documents and tool results
  • the conversation so far
  • the user's new message
  • room reserved for the answer

Longer windows do not remove the need for selection. Models can miss details buried in the middle of very long inputs, long prompts cost more and respond slower, and irrelevant text invites irrelevant answers. Treat the window as a budget, not a bucket.

Planning a budget

A budget gives each part a ceiling. Fixed parts (system prompt, answer reserve) come first; the flexible parts share what is left.

flowchart LR
    A["Window: 8000"] --> B["System + rules: 600"]
    A --> C["Answer reserve: 1000"]
    A --> D["User message: 400"]
    A --> E["History: up to 1500"]
    A --> F["Retrieved context: the rest"]

The retrieved context gets whatever remains, so it shrinks when history grows.

Python Allocate a prompt budget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Change WINDOW or the document sizes and watch what gets dropped. The rule that matters: decide what to drop before the model does it for you. A provider that truncates input usually cuts from the start or the end, and the part it cuts may hold your instructions.

Where to put things

Order inside the prompt matters because models weigh the start and the end of the input more reliably than the middle. A common layout:

  1. Role and rules (stable, so it can be cached; see caching)
  2. Reference documents, each wrapped in a labelled block
  3. Conversation history
  4. The user's current request and the output format reminder, last

Put the question after long documents, not before them. Repeat the key constraint at the end when the prompt is long.

Common mistakes

  • Counting only input and forgetting the answer reserve, so replies get cut off mid-sentence.
  • Pasting whole documents when two paragraphs would answer the question.
  • Letting chat history grow without limit until it pushes out the instructions.
  • Measuring in words or characters in production and getting surprised by token bills.
  • Assuming a bigger window means better answers. Measure with a test set before paying for more context.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What must fit inside the context window?

    The window covers everything in the request and the generated reply, so you must reserve room for the answer.
  2. 2. About how many characters is one English token on average?

    A common planning rule is roughly four characters per token for English, though real tokenizers vary.
  3. 3. Why shrink history before shrinking the system prompt?

    Standing rules are fixed and essential; older chat turns are the most expendable part.
  4. 4. Where should the user's actual question go in a prompt with long documents?

    The start and end of a long input are used more reliably than the middle, so put the request last.

Summary

  • The context window holds input and output together; reserve room for the answer.
  • Tokens are sub-word pieces, roughly four English characters each; estimates need a margin.
  • Give each part of the prompt a ceiling and decide what to drop in advance.
  • Put stable rules first and the request last.
  • More context is not automatically better; select, then measure.

Next: Retrieval-Augmented Generation Basics