Skip to content
Tutorials Center
My learning

Topics

Lesson 48 of 53 ยท Projects

Project: An Evaluation Harness

Build a small evaluation harness: test case format, runners, scorers, a pandas report and a regression comparison between two prompt versions.

Changing a prompt without measuring it is guessing. Prompt A looks better on the three examples you tried, but did it break the other thirty? An evaluation harness answers that question in seconds: it runs every test case through your system, scores the outputs, prints a table and compares versions. It is the single most useful tool you can build for an LLM application.

In this project you write one in about 100 lines of Python. It follows the ideas of Evaluating LLM Applications and Debugging and Improving Prompts. The model is a fake function, so results are deterministic; swapping in a real model changes one line.

The design

Four small pieces

Test cases Input, expected value, tags.
Runner Sends each input to the system.
Scorers Turn each output into a pass or a number.
Report A table, a total and a diff between versions.

Keep each piece separate, so you can change one without breaking the rest.

1 / 2

Our toy task is support ticket classification: label a message as billing, shipping or technical. The "system under test" is a function that takes a prompt version and a message and returns a label, which is exactly the shape of a real LLM call wrapped in your code.

Step 1: test cases and a fake system

A test case is plain data: an input, an expected output and tags that let you slice the results (for example easy and tricky). Store cases in a file in real life (JSON or CSV), and keep them in version control next to your prompts.

Our fake model has two prompt versions. Version 1 uses simple keywords. Version 2 handles more phrases but, as often happens, also introduces a mistake: it treats the word "damaged" as a technical problem.

Python Step 1: cases and two prompt versions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 2: runner and scorers

The runner loops over cases and records the raw output of every run. A scorer takes a case and an output and returns a score, usually 1 or 0. Keep them separate: you can add a new scorer and re-score old outputs without calling the model again, which matters when calls cost money.

We use two scorers: exact match on the label, and a "format" check that the output is one of the allowed labels. Later you can add regex checks, semantic similarity or an LLM judge in the same shape.

Python Step 2: runner, scorers and results table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Always look at the failures, not just the average. A table of every case with its expected and actual output is what lets you see that t07 and t08 fail for a reason you can fix. Slicing by tag shows where the weakness is: easy cases pass, tricky ones do not.

Step 3: compare two prompt versions

The point of a harness is the regression comparison. Run the same cases with both versions, join the results on the case id and list what changed: cases that got fixed, cases that broke and the net effect. A higher average can hide a regression, which is why you read the lists of changes and not only the totals.

Python Step 3: v1 versus v2 regression report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The totals alone say v2 is better (80% against 70% looks like a win). The lists reveal something extra: v2 fixed two cases that v1 missed, but it also broke one that v1 got right. The new "damaged" rule hijacked a shipping ticket (t04, a damaged parcel). In a real project you would fix the prompt, rerun and only ship when nothing important regressed.

Making the harness trustworthy

Before you trust your harness, check these points:

  • Enough cases. Ten cases are a demo. For decisions, you want dozens or hundreds, including edge cases and past production failures. With few cases, a difference of one case is noise (Metrics: Retrieval, Benchmarks and Statistics).
  • Real model randomness. Real models can answer differently on each run. Run each case several times or set temperature to 0, and report the average.
  • Held-out cases. If you tune the prompt until every case passes, you have overfit your test set. Keep a separate set you do not tune on.
  • Cheap and fast. Run the suite on every prompt change, ideally in your CI pipeline, and save results with the prompt version (Prompt Templates and Versioning).
  • Right scorer for the task. Exact match suits labels. Free-text answers need rubric checks or an LLM judge, validated against human review.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why keep runners and scorers separate?

    Model calls cost time and money. Saving raw outputs lets you try new scorers on them for free.
  2. 2. Version 2 has a higher overall accuracy than version 1. What else should you check before shipping it?

    An average can hide regressions. The lists of fixed and broken cases show whether v2 harms something important.
  3. 3. What is the risk of tuning a prompt until every test case passes?

    Optimizing against the same cases you report on inflates the score. Keep a held-out set you do not tune against.
  4. 4. Why slice results by tags such as 'easy' and 'tricky'?

    Group-level scores show patterns, for example that all failures are tricky cases, which tells you what to fix.

Summary

  • A harness has four parts: test cases, a runner, scorers and a report.
  • Test cases are plain data with an input, an expected value and tags; keep them in version control.
  • Save raw outputs so you can re-score them without paying for new model calls.
  • Compare versions by listing fixed and broken cases, not only by comparing averages.
  • Use enough cases, repeat noisy runs, keep a held-out set and choose a scorer that fits the task.
  • Swapping the fake model for a real one only changes the system function.

Next: give your model hands in Project: A Tool-Using Agent.