Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Projects and Practice

Project: An Evaluation Harness

Build a small evaluation harness for LLM prompts: test cases, graders, per-category scores, prompt comparison and regression checks, all runnable in your browser.

Every change to a prompt, model or retrieval setting can quietly make some answers worse while making others better. The cure is a harness: a fixed set of test cases, automatic graders, and a report you re-run after every change. In this project you build one in about 80 lines of Python. It works with any model, because the model is just a function from prompt text to answer text.

No LLM runs in this page. The "models" below are two small rule-based functions standing in for two prompt versions. Swap in a real API call later; the harness does not change.

Design

flowchart LR
    C["Test cases: input, expected, grader, category"] --> R["Runner"]
    M["Model function: prompt in, text out"] --> R
    R --> G["Graders: exact, contains, regex"]
    G --> S["Scores by category"]
    S --> D["Compare runs, flag regressions"]

Cases are data, graders are small functions, and the model is a plug-in.

Principles from benchmarks and judging apply here: keep a held-out set, include hard and adversarial cases, and never tune on the exact cases you report.

Step 1: cases and graders

A grader returns True or False. Prefer cheap deterministic graders; use a model judge only where they cannot work.

Python Cases and graders

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 2: runner and report

The runner calls the model on every case, grades the output and stores the full record so you can inspect failures later. Never keep only the final percentage.

Python Run a model and report by category

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading failures is where you learn. Case 2 failed because the answer was a correct number written as words: is the grader too strict or the model unhelpful? Decide deliberately, then fix one or the other.

Step 3: compare two versions and catch regressions

A new prompt may improve the average yet break something that used to work. Compare per case, not just per total.

Python Version comparison and regression check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last lines are the most important lesson: with six cases, a gain of two cases is not evidence. Grow the set (dozens to hundreds of cases per category), and for non-deterministic models run each case several times and average.

Extensions

  1. Model-judge grader for open-ended answers, calibrated against a few human labels (judging).
  2. Repeated trials with sampling (decoding) and report mean and spread.
  3. Cost and latency columns next to quality, so tradeoffs are visible.
  4. Store every run (JSON lines) with the prompt version, model name and date.
  5. Hallucination cases: questions with no answer in the context, where the right output is "I don't know" (safety).
  6. Run it in CI so a prompt change cannot merge if the gate fails.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why store full per-case records instead of only the final percentage?

    Failures explain what to fix, and per-case data reveals regressions hidden by an average.
  2. 2. A new prompt raises the overall score but breaks three previously passing cases. What should you do?

    Regressions may matter more than the average; do not hide them by removing cases.
  3. 3. Which grader is best for 'the answer is exactly a date in YYYY-MM-DD form'?

    A deterministic regex is cheap, exact and reproducible for rigid formats.
  4. 4. Why is a 6-case test set weak evidence?

    Small samples give wide uncertainty intervals; add more cases and repeated trials.

Summary

  • A harness is cases plus graders plus a runner plus a report, with the model as a plug-in function.
  • Prefer deterministic graders, keep full records, and break scores down by category.
  • Compare versions per case to catch regressions, and use enough cases that differences are not noise.
  • Re-run it after every prompt, model or retrieval change.

Next: Project: Question Answering over Documents