Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 36 ยท Evaluation

Evaluating Generation: Faithfulness and Answer Quality

Evaluate the answer half of RAG: faithfulness, answer relevance, context precision and correctness, how LLM-as-judge works, and cheap automatic checks.

Good retrieval does not guarantee a good answer. The model can ignore the evidence, mix it with its own memory, over-summarize, or answer a slightly different question. So after retrieval metrics, you need metrics for the generated answer. This is harder than counting hits, because answers are free text and there are many correct wordings.

The RAG triad

A widely used mental model splits quality into three relationships between the question, the retrieved context and the answer:

Check Compares Question
Context relevance (precision) question and context Is the retrieved text actually about the question?
Faithfulness (groundedness) context and answer Is every claim in the answer supported by the context?
Answer relevance question and answer Does the answer address what was asked?

Add two that need a reference answer: context recall (does the context contain the facts needed for the reference answer?) and answer correctness (does the answer match the reference?). Frameworks such as RAGAS, TruLens, DeepEval and promptfoo implement these; the ideas below are what they compute.

The RAG triad

Context relevance Right evidence?
Faithfulness Supported by it?
1 / 2

Diagnose by elimination: low context relevance means fix retrieval; good context but low faithfulness means fix the prompt or model; faithful but irrelevant answers mean the model misread the question.

Faithfulness by claim checking

The standard recipe: (1) split the answer into atomic claims, (2) check each claim against the context, (3) faithfulness = supported claims divided by total claims. In production an LLM or an NLI (natural-language inference) model decides "supported"; a cheap lexical version is useful for smoke tests and regression alarms.

Python A lexical faithfulness score

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The invented "24 hours a day" claim shares almost no content words with the context, so it is flagged. The lexical method fails on paraphrases ("get money back" for "refund") and on claims that reuse context words but reverse their meaning ("Refunds are not available"). That is why serious evaluations use an NLI model or a strong LLM judge, and why you spot-check judge output by hand.

LLM-as-judge

An LLM judge is a model prompted to grade an answer. It scales far beyond human review and correlates reasonably with human labels when set up carefully. A judge prompt is short and explicit; here is a faithfulness example (read-only, because it needs a model call):

TEXT
You are grading a RAG answer for faithfulness.
Context:
{context}

Answer:
{answer}

List every factual claim in the answer. For each claim, say SUPPORTED or
UNSUPPORTED based only on the context. Then output JSON:
{"claims": [{"text": "...", "verdict": "SUPPORTED|UNSUPPORTED"}], "score": <supported/total>}

Good practice for judges:

  • Use a rubric with explicit definitions and a small score scale (for example 1 to 4), not "rate from 1 to 10".
  • Ask for reasoning, then the verdict, or the claim list as above; it improves reliability.
  • Use a different, preferably stronger, model than the generator, since models favour their own style.
  • Randomize order when comparing two answers (position bias), and watch for length bias: judges like longer answers.
  • Calibrate against humans: label 50 to 100 answers yourself, measure agreement with the judge, and revise the rubric until agreement is acceptable.
  • Temperature 0 and a fixed model version for repeatable runs.

Context relevance and recall

Context precision asks how much of what you retrieved is useful, and context recall asks whether everything needed is present. With a reference answer broken into facts, recall is the share of facts found in the context. Both can be approximated lexically for a quick check.

Python Context recall from reference facts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Neither context contains "pro rata", so recall tops out at 2/3: the failure is in the corpus or retrieval, and no prompt can fix it.

Answer correctness against a reference

When you have a gold answer, compare meaning, not wording. Exact match and F1 over tokens work for short factual answers ("30 days"); embedding similarity or an LLM judge works for long ones. Token F1 is simple and standard in question-answering benchmarks.

Python Exact match and token F1

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

F1 gives partial credit to a correct paraphrase but is blind to negation and to synonyms, so use it for short answers and treat it as a rough signal.

Other dimensions worth measuring

  • Refusal quality: include questions the corpus cannot answer; the correct behaviour is to decline (citations lesson).
  • Citation accuracy: share of citations that point at a supporting chunk.
  • Format compliance: valid JSON, language, length limits.
  • Safety and tone: policy compliance, no leaking of other tenants' data.
  • Latency and cost per answer (production lesson).
  • Human feedback: thumbs up or down and free-text comments from real users; the ground truth that offline metrics approximate.

An LLM judge is a model, not an oracle. Never ship a prompt change on a one-point judge improvement without looking at examples. Read 20 to 30 answers yourself every time you change something important.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. The context contains the right facts but the answer includes an invented claim. Which metric drops?

    Faithfulness measures whether every claim in the answer is supported by the context.
  2. 2. Context relevance is low and faithfulness is high. Where should you look first?

    The model stayed true to poor context, so the problem is upstream in retrieval.
  3. 3. Which is a known bias of LLM judges?

    Length and self-preference biases are well documented, so calibrate against human labels.
  4. 4. Why use token F1 rather than exact match for short answers?

    F1 rewards partial overlap, but it still cannot judge meaning or negation.

Summary

  • Evaluate context relevance, faithfulness and answer relevance separately to locate the failing stage.
  • Faithfulness = share of answer claims supported by the context.
  • LLM judges scale review; give them rubrics, calibrate against humans, and beware of bias.
  • Add reference-based checks (context recall, F1) where gold answers exist.
  • Include unanswerable questions and read real outputs.

Next: Building an Evaluation Set