Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 39 ยท Memory and Retrieval

Short-Term Memory and Context Windows

Learn how an AI agent keeps its working memory inside the context window, how to count tokens, and how to trim or summarize history without losing the plot.

A language model has no memory of its own. Every time your agent calls the model, it sends the whole conversation again: the system prompt, the user's messages, the model's earlier answers, and every tool result. That bundle is the context window, and it is the agent's short-term memory. If something is not in the window, the model cannot see it.

This matters because windows are finite and every token costs money and time. A long-running agent that calls tools thirty times will fill its window with raw tool output unless you manage it. In this lesson you will learn what lives in the window, how to measure it, and three strategies for keeping it small: trimming, summarizing and selective pruning.

The real model APIs are not available on this site. All code below uses small, clearly labelled stand-ins written in plain Python that run in your browser. The ideas (message lists, token budgets, summaries) are exactly what real agents do.

What lives in the context window

An agent's window is a list of messages. A typical one looks like this:

Python
messages = [
    {"role": "system", "content": "You are a helpful travel agent."},
    {"role": "user", "content": "Find me a flight to Lisbon."},
    {"role": "assistant", "content": "", "tool_call": "search_flights('LIS')"},
    {"role": "tool", "content": "[ ... 40 lines of flight data ... ]"},
    {"role": "assistant", "content": "The cheapest option is ..."},
]

Notice that the tool result is often the largest item. Agents spend most of their budget on tool output, not on chat.

flowchart LR
  S["System prompt"] --> W["Context window"]
  H["Chat history"] --> W
  T["Tool results"] --> W
  R["Retrieved facts"] --> W
  W --> M["Model call"]
  M --> A["Answer or next tool call"]
  A --> H

Everything the model can use must pass through the window on every call.

Counting tokens

Models read tokens, not characters. A token is a chunk of text, roughly three-quarters of an English word. Real counts come from the provider's tokenizer, but a rough estimate (characters divided by four) is good enough for budgeting decisions.

Python Estimate the size of a conversation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The tool message dwarfs everything else. That is the typical shape of an agent run, and it tells you where to save space first.

Strategy 1: sliding window trimming

The simplest fix is to keep the system prompt and only the most recent messages that fit a budget. Walk backwards from the newest message and stop when the budget runs out.

Python Trim history to a token budget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Trimming is cheap and predictable, but it forgets. If the user's name or goal appeared early, it vanishes. Two rules help: never drop the system prompt, and never split a tool call from its tool result, because most APIs reject a result without its matching call.

Strategy 2: summarizing old turns

Instead of deleting old messages, replace them with a short summary. In a real agent you would ask the model to write the summary. Here a stand-in keeps the first sentence of each old message so you can see the mechanics.

Python Compact old turns into a summary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Summaries keep the gist of old conversation in a few lines. The risk is that a summary may drop a detail that matters later. Good summaries keep decisions, constraints, names and open tasks, and drop pleasantries and raw data.

Strategy 3: pruning tool output

Often the smartest move is to shrink tool results before they enter the history. Return only the fields the agent needs, truncate long text with a marker, and store the full data outside the window (a file, a database) with a short handle the agent can ask for later.

Python
def shrink_tool_result(rows, keep=("id", "price"), limit=3):
    slim = [{k: r[k] for k in keep} for r in rows[:limit]]
    extra = len(rows) - len(slim)
    note = f"(+{extra} more rows not shown)" if extra > 0 else ""
    return {"rows": slim, "note": note}

Telling the model that more rows exist is important. Without the note, it may assume it saw everything.

Common mistakes

  • Growing without limit. Always set a maximum and test with a long run, not just a short demo.
  • Trimming the goal away. Pin the user's original task near the top, or repeat it in the system prompt.
  • Summarizing too often. Each summary loses detail; summarize in batches, not after every turn.
  • Ignoring cost. Input tokens are billed on every call, so a 20,000-token history re-sent ten times costs 200,000 tokens.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where does an agent's short-term memory live?

    The model is stateless; whatever you put in the message list for this call is all it can use.
  2. 2. What usually takes the most space in an agent's history?

    Tools such as search or file readers return large payloads, so they dominate the window.
  3. 3. Why should a tool result never be kept without its tool call?

    Message formats pair each result with the call that produced it, so trimming must keep them together.
  4. 4. What should a good summary of old turns keep?

    Those details are what the agent needs later; everything else can be dropped.

Summary

  • The context window is the agent's only built-in memory; the model sees nothing outside it.
  • Estimate tokens (about four characters each) to budget the window.
  • Trim by walking back from the newest message, keeping the system prompt.
  • Summarize old turns to keep decisions and facts in fewer tokens.
  • Prune tool output at the source and say when data was cut.

Next: Long-Term Agent Memory