Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 38 ยท Retrieval and Context Engineering

Conversation Memory and Summarization

Manage chat history in prompts: sliding windows, rolling summaries, pinned facts and long-term memory stores, with runnable examples and trade-offs.

A chat model has no memory. When it seems to remember what you said three messages ago, your application resent those messages. Every turn, you rebuild the whole prompt. As a conversation grows, resending everything gets slow, expensive and eventually impossible, so you need a memory strategy: a rule for which parts of the past to keep, shrink or store elsewhere.

This lesson covers four strategies, from simplest to most capable, with runnable stand-ins. The summarizer in the examples is a placeholder function; in real use it is another prompt to the model.

What memory must preserve

Not all history is equal. Ask what the next turn needs:

  • Instructions and rules (always, handled by the system prompt)
  • Stable facts about the user or task: name, goal, constraints, decisions made
  • Recent turns: needed to resolve "it", "that one", "the second option"
  • Old chit-chat and dead ends: usually safe to drop

Strategy 1: keep everything

Simplest and exact. Fine for short sessions. It fails when the budget from the token lesson runs out, and costs grow with every turn because you pay for the whole history each time.

Strategy 2: sliding window

Keep only the last N turns, or the last T tokens. Cheap and predictable, but it forgets early facts completely: the user stated a budget on turn 2, and by turn 20 the model has no idea.

Python A sliding window by token budget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With the small budget the model still sees the battery question but has lost the 800 dollar limit. That is the cost of a pure window.

Strategy 3: rolling summary

When old turns fall out of the window, fold them into a running summary that stays in the prompt. The summary holds decisions and facts, not wording. Each time the window overflows, a summarizer prompt merges the dropped turns into the existing summary.

A good summarizer prompt says what to keep: "Update the summary below with the new turns. Keep user goals, constraints, numbers, decisions and open questions. Drop greetings and repetition. Maximum 120 words."

Python Window plus rolling summary (stand-in summarizer)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The 800 dollar budget now survives inside the summary. Summaries are lossy and the summarizer can drop or distort facts, so test them on conversations where you know what must be remembered. Never summarize anything that must be exact (order numbers, code, legal wording); store it verbatim instead.

Strategy 4: pinned facts and long-term memory

Combine the window with a small structured memory of facts the application extracts and stores outside the prompt, such as a user profile or a list of decisions. Each turn you inject only the facts relevant to the current message, selected the same way as in RAG. This scales across sessions: the assistant can "remember" last week.

Python A tiny fact store injected into the prompt

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ask the model to emit new facts in a structured format (structured output is covered in the previous chapter) and have your code validate and store them. Let users view and delete stored memories: this is both a privacy requirement in many places and a debugging tool.

Choosing a strategy

Situation Use
Short Q&A, few turns Keep everything
Long chat, only recent context matters Sliding window
Long chat with early decisions Window plus rolling summary
Returning users, many sessions Fact store (plus window)

Whatever you pick, put the memory block in a labelled section of the prompt, and tell the model it may be incomplete ("The summary may omit details; ask if unsure").

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does a chat model seem to remember earlier messages?

    The model is stateless; the application rebuilds the prompt with history on every turn.
  2. 2. What is the main weakness of a pure sliding window?

    Anything older than the window disappears, including constraints stated at the start.
  3. 3. What should a rolling summary keep?

    Summaries should hold the information needed for future turns, not the original phrasing.
  4. 4. Which data should never go through a summarizer?

    Summaries are lossy; store exact values verbatim in a fact store or keep them in the window.

Summary

  • The model is stateless; memory is whatever your code puts back into the prompt.
  • Sliding windows are cheap but forget; rolling summaries keep decisions; fact stores persist across sessions.
  • Summaries are lossy: test them, and keep exact values outside them.
  • Label the memory block and let users inspect and delete stored facts.

Next: Building a Prompt Test Set