Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 36 ยท Production Patterns

Observability and Common RAG Failure Modes

Trace every RAG request, monitor retrieval scores and refusal rate, collect feedback, and use a checklist to diagnose the ten most common RAG failure modes.

You shipped it. Now users ask questions you never imagined, documents change, a model provider updates a model silently, and someone reports "the bot gave a wrong answer yesterday". Without observability, you cannot answer "what did the system retrieve, what prompt did the model see, and why did it say that?". This lesson shows what to record, what to watch, and how to diagnose failures quickly.

Trace every request

A RAG request is a pipeline, so record it as a trace with one span per stage. For each request, store at least:

  • request id, timestamp, user or tenant (pseudonymous), session id,
  • the raw question and the rewritten query,
  • filters applied and index version,
  • retrieved chunk ids with scores (before and after re-ranking),
  • the final prompt (or a hash plus the template version),
  • model name and version, parameters, token counts, finish reason,
  • the answer, validated citations, and refusal flag,
  • timing per stage, errors, retries and cache hits,
  • user feedback (thumbs, comments) attached later by request id.

Tools: OpenTelemetry for general tracing, plus LLM-specific platforms (Langfuse, LangSmith, Arize Phoenix, Helicone and others). A structured log line per stage also works for a small system.

Python A minimal tracing helper

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In production, ship these records to a log store, not stdout, and apply the privacy rules from the security lesson: question texts and chunks can be sensitive, so use retention limits and restricted access.

What to monitor

Signal Why it matters
Latency p50/p95/p99 per stage Finds slow stages and tail latency
Error and timeout rates (LLM, embedding, DB) Provider incidents and bad deploys
Tokens and cost per request Cost regressions after prompt changes
Top retrieval score (mean, p10) Falling scores mean drift: new topics or stale corpus
Share of queries with no result above threshold Coverage gaps
Refusal rate Too high: retrieval gaps. Too low: possible guessing
Citation validity rate Hallucinated or wrong citations (citations)
Cache hit rate Whether caching earns its keep
Feedback (thumbs down rate, reformulations, escalations to human) Real quality signal
Index freshness (age of newest chunk per source) Stale data
Offline evaluation scores on a schedule Regressions from model or corpus changes

Query-level drift detection can be as simple as comparing this week's distribution of top scores to a baseline.

Python Detecting retrieval score drift

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When this alert fires, look at the lowest-scoring queries: they reveal topics your corpus lacks, which is a content-writing to-do list.

The ten common failure modes

Use this as a diagnosis checklist. Look at the trace and find the first stage that went wrong.

  1. Missing content. The answer is not in the corpus. Fix: add documents; make the system refuse.
  2. Not ingested. The document exists but parsing failed (scanned PDF, table, JavaScript-rendered page). Fix: check ingestion logs and parsed text.
  3. Bad chunking. The answer is split across chunks or separated from its heading. Fix: re-chunk with headings, overlap or parent retrieval (advanced patterns).
  4. Vocabulary mismatch. The question says "money back"; the document says "reimbursement". Fix: hybrid search, query expansion, better embedding model (hybrid).
  5. Wrong ranking. The right chunk is at rank 14. Fix: larger candidate set plus a re-ranker.
  6. Filter too strict or wrong. Metadata hides the answer. Fix: log filters; fall back to an unfiltered search.
  7. Context overload. Too many or noisy chunks; the model misses the key one. Fix: fewer chunks, ordering, compression.
  8. Model ignores or contradicts context. Low faithfulness. Fix: stronger grounding prompt, a better model, structured output with citation checks.
  9. Stale or conflicting sources. Old and new policies both retrieved. Fix: versioning, dates in metadata, freshness filters (updates).
  10. Question needs more than retrieval. Calculations, multi-step reasoning, live data. Fix: tools, SQL, decomposition.
Python Triage a failed request from its trace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The order is the point: debug upstream first. Teams often tweak the prompt when the right chunk was never retrieved, and no prompt can fix that.

Feedback loops

  • Add a thumbs up/down button and an optional comment; attach it to the request id.
  • Turn every confirmed failure into a test in your evaluation set.
  • Review a random sample of conversations weekly, not just the flagged ones.
  • Route "I don't know" answers to a content backlog.
  • Use canary releases: send 5 percent of traffic to a new prompt or index and compare metrics before full rollout.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A user reports a wrong answer. What should you look at first?

    The trace shows which stage failed, so you fix the cause instead of guessing.
  2. 2. The mean top retrieval score falls steadily over several weeks. What does it likely indicate?

    Lower scores mean worse matches, which often reflects coverage gaps.
  3. 3. The right chunk was never among the retrieved candidates. Which fix is least relevant?

    A generation prompt cannot use evidence that was never retrieved.
  4. 4. Why log the prompt template version?

    Versioning makes regressions attributable and reproducible.

Summary

  • Trace each stage with ids, scores, prompt version, tokens and timings; protect the logs.
  • Monitor latency, cost, refusal rate, top-score drift, citation validity and feedback.
  • Diagnose upstream first: corpus, ingestion, chunking, retrieval, ranking, context, generation.
  • Feed every failure back into the evaluation set and release changes gradually.

Next: Project: A Support FAQ Search Engine