Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 36 ยท Generation and Prompting

Conversational RAG and Query Rewriting

Handle follow-up questions in a RAG chatbot: rewrite each turn into a standalone query, manage chat history within a budget, and decide when to retrieve again.

Single-shot question answering is easy: the question contains everything the retriever needs. Chat is harder. People write "what about the EU?", "and how much does it cost?" or "can you explain that again?". Embedded on its own, "what about the EU?" is nearly meaningless, so retrieval returns junk and the answer collapses. This lesson shows how to turn each message into a standalone query and how to keep the history under control.

Why follow-ups break retrieval

Consider this exchange:

  1. User: "How long do refunds take for the US plan?"
  2. Assistant: "Five business days."
  3. User: "And for the EU?"

The retriever sees only message 3. The words "refunds" and "how long" are gone. Three strategies exist:

  • Embed the whole conversation. Noisy: old topics drag results away.
  • Embed the last message only. Loses the topic, as above.
  • Rewrite the last message into a standalone question using the history. This is the standard approach, called condense question or query rewriting.

Condense, then retrieve

History Earlier turns
Rewrite Standalone question
1 / 2

In production an LLM performs the rewrite with a prompt such as: "Given the conversation and the follow-up message, rewrite the follow-up as a single self-contained question. If it is already self-contained, return it unchanged." Use a small, fast model; this step adds latency to every turn.

Seeing the problem and the fix

The stand-in rewriter below uses a simple rule: when the follow-up is short and starts with a connector like "and" or "what about", it borrows the topic words from the previous user question. It is far simpler than an LLM, but it demonstrates the effect on retrieval.

Python Follow-up retrieval, before and after rewriting

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Raw, "And for the EU?" matches the EU export chunk (the only word that survives preprocessing is "EU"), which is the wrong topic. After the rewrite, the topic words are back and the EU refund chunk ranks first. A real LLM rewriter generalizes this far beyond hand-written rules: it handles pronouns ("does it support that?"), ellipsis and topic switches.

Managing history

Chat history grows without bound while the context window does not. Options:

Strategy How it works Trade-off
Sliding window Keep the last N turns Simple; forgets early facts
Token budget Keep as many recent turns as fit Predictable cost
Summary memory Summarize older turns into a short note Keeps gist; the summary can drift
Selective memory Store key facts (user region, plan) in a profile Compact and reliable; needs extraction

Also decide what history goes where. The rewriter needs the last few turns. The answer-generation prompt usually needs recent turns for tone and continuity, but older turns can be dropped, because the evidence comes from retrieval, not from chat memory.

Python Keeping history within a token budget

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The oldest message, which states the plan and seat count, is dropped. That is exactly the kind of fact worth saving in a small profile (plan=US, seats=12) that you always include, rather than hoping it survives the window.

Should every turn retrieve?

Not necessarily. "Thanks!", "make it shorter" or "translate that to Spanish" need no new evidence. Common designs:

  • Always retrieve: simplest; wasteful on chit-chat but safe.
  • Router: a cheap classifier or a model call decides between retrieve, answer from history, and small talk.
  • Agentic RAG: the model has a search tool and calls it when it decides it needs evidence (advanced RAG patterns).

When the answer to a follow-up could be in the previous turn's chunks, you can also reuse them: keep the last retrieved chunks and merge them with fresh results so "can you explain that again?" still has evidence.

Pitfalls

  • Rewrite drift: the rewriter adds facts that the user never said. Instruct it to only rephrase, and log both versions.
  • Showing the rewrite: display it for transparency in internal tools; in a consumer chat keep it internal.
  • Latency: the extra model call adds time, so skip it for the first turn and for already self-contained messages.
  • Context leakage across sessions: never mix history between users or conversations.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does embedding only 'And for the EU?' retrieve poorly?

    The follow-up depends on earlier context; without it the query has almost no topical signal.
  2. 2. What is the 'condense question' step?

    The rewrite makes the query self-contained so retrieval works without the chat.
  3. 3. Which is a good place to store durable facts like the user's plan?

    A compact profile survives history trimming and costs few tokens.
  4. 4. Why might you skip retrieval for 'Thanks!'?

    A router can send small talk and meta requests straight to the model.

Summary

  • Follow-up questions need a standalone rewrite before retrieval.
  • Use a small fast model for rewriting and log the rewritten query.
  • Bound the history with a window, a token budget, summaries or a profile.
  • Not every turn needs retrieval; route or let an agent decide.
  • Keep sessions isolated.

Next: Advanced RAG Patterns