Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 36 ยท Production Patterns

Caching, Latency and Cost in RAG Systems

Budget RAG latency stage by stage, cache embeddings and answers (exact and semantic), estimate token cost per query, and apply cheap optimizations safely.

A prototype RAG demo takes eight seconds and costs a cent per question, and nobody minds. In production, those numbers multiply: a thousand users, thousands of questions per day, and a product manager asking why the chat "feels slow". This chapter is about running RAG for real. We start with the two numbers that decide whether a RAG feature survives: latency and cost per answer.

Where the time goes

A typical request passes through these stages:

Stage Typical time Notes
Query rewriting (LLM) 300 to 1000 ms Skip when the message is already standalone
Query embedding 20 to 200 ms API call or local model
Vector / hybrid search 5 to 100 ms Usually the cheapest stage
Re-ranking 50 to 400 ms Scales with candidate count
LLM generation 1 to 10 s Dominates; depends on answer length
Post-checks (citations, guardrails) 10 to 300 ms Depends on checks used

The language model usually dominates. Three consequences: stream the answer so the user sees the first token in a fraction of a second, shorten prompts and answers, and do not optimize the 10 ms stage before the 3 s one. Measure each stage in your own system with timers (see observability).

What costs money

With hosted models you pay per token, in and out, usually with output priced higher. Embeddings, re-rank calls and the vector database add smaller costs. The prompt is mostly your retrieved context, so context size drives cost: five 400-token chunks plus instructions is already about 2,500 input tokens per question.

Python A cost-per-query calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The prices above are invented placeholders so you can see the shape of the problem; plug in real numbers. Dropping from ten chunks to three cuts the prompt by about two thirds, and a re-ranker is often cheaper than the extra tokens you would otherwise send.

Caching layers

Caching is the cheapest win, because many users ask the same things.

  1. Embedding cache: key = hash of the normalized text; stores the vector. Saves embedding calls for repeated queries and, at ingestion, unchanged chunks (re-embedding an unchanged document is pure waste: hash each chunk).
  2. Retrieval cache: key = query plus filters plus index version. Fast and safe if you invalidate on index updates.
  3. Answer cache (exact): key = normalized question plus tenant plus corpus version. Instant answers for repeated questions.
  4. Semantic cache: key = the question's embedding; a new question reuses the answer of a cached question whose embedding is close enough.
  5. Provider prompt caching: many LLM APIs discount repeated prompt prefixes. Put the stable parts (system prompt, fixed instructions) first, variable parts (chunks, question) last.
Python Exact cache plus semantic cache

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first query hits the exact cache, the second is close enough for the semantic cache, the third misses and goes to the full pipeline. A semantic cache is powerful and dangerous: "How do I cancel my plan?" and "How do I cancel my order?" are near neighbours but need different answers. Mitigations: a high threshold (calibrate on pairs you label), include filters (tenant, language, plan, region) in the key, keep a short TTL, never cache answers that depend on user-specific data, and add the corpus version to the key so answers expire when documents change.

Cheap, safe optimizations

  • Stream the response and show retrieved sources immediately.
  • Run stages in parallel: embed the query while rewriting; run keyword and vector search concurrently.
  • Fetch fewer, better chunks with re-ranking; trim whitespace and boilerplate from chunks.
  • Cap answer length in the prompt and with a max-tokens setting.
  • Route by difficulty: a small model for simple lookups, a large model only for hard questions.
  • Skip steps when not needed: rewriting on turn one, re-ranking when the top result is clearly dominant.
  • Batch ingestion embeddings and use the vendor's batch APIs, usually cheaper than synchronous calls.
  • Compress vectors (quantization) when memory dominates the vector database bill; see the indexing chapter of this course.
Python Latency budget: sequential versus parallel stages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Even a perfect optimization of the small stages saves little because generation dominates, but streaming changes how fast the answer feels: the user starts reading after roughly a second instead of after the whole generation.

Limits and reliability

Production calls fail. Set timeouts on every external call, retry with exponential backoff and jitter for transient errors, use a circuit breaker so a failing provider does not exhaust your workers, and define degraded modes: if the re-ranker is down, skip it; if the LLM is down, show the top retrieved passages as search results. Apply rate limits per user to protect cost.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which stage usually dominates RAG latency?

    Generating tokens takes seconds, while indexed vector search takes milliseconds.
  2. 2. What is the main risk of a semantic answer cache?

    Use a high calibrated threshold, put filters in the key and expire on corpus changes.
  3. 3. How does sending fewer, better chunks help?

    Context size drives cost, and extra noisy chunks can hurt quality.
  4. 4. Why put the system prompt first and the retrieved chunks last?

    Prefix caching only works on the unchanged leading part of the prompt.

Summary

  • Measure each stage; generation usually dominates, so stream.
  • Cost is mostly tokens: context size is the lever.
  • Cache embeddings, retrievals and answers; treat semantic caching with care.
  • Add timeouts, retries, circuit breakers and degraded modes.
  • Optimize with data from your own traces, not guesses.

Next: Updates, Deletes and Versioning