Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 38 ยท Evaluation and Testing Prompts

Regression Testing and Comparing Prompts

Compare two prompt versions case by case, spot regressions, handle non-deterministic outputs with repeated runs, and decide when a gain is real or noise.

You have a test set and a metric. Now you change the prompt, and the score moves from 82 to 85 percent. Ship it? Maybe. Three points could be a real gain, or one case that flipped by luck, or a gain in one category paid for with a loss in another. Regression testing is the discipline of comparing versions carefully enough to tell.

The examples here use a stand-in model with seeded random noise to mimic the run-to-run variation real models show. The comparison code works the same on real outputs.

Compare case by case, not just totals

Two prompts can both score 80 percent while failing on entirely different cases. Look at each case in both versions and sort them into four groups:

  • Fixed: failed before, passes now
  • Broken (regression): passed before, fails now
  • Still failing
  • Still passing

A new version with 5 fixes and 4 regressions has a net gain of 1 but is a risky change. The regressions tell you what the new wording sacrificed.

Python Which cases were fixed or broken?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Same score, very different story. Make a diff like this part of your standard report, and treat each regression as a question: is the old behaviour something we promised? If yes, do not ship until it is back.

Non-determinism: run it more than once

With a temperature above zero, the same prompt can pass a case on one run and fail on the next. Even at temperature zero, providers do not always guarantee identical output. So one run of 40 cases is a noisy measurement.

Two practices help:

  1. Repeat each case several times (3 to 10) and report the pass rate per case. A case at 60 percent is a flaky case, which is itself useful information: the prompt is under-specified for it.
  2. Lower the temperature for tasks that need consistency (classification, extraction), and keep the evaluation settings identical to production settings.
Python Flaky cases: repeat each test

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Prompt B fixes c3 but weakens c2; the flaky c4 is unsettled in both. The right reaction to a flaky case is to improve the prompt for it (clearer instruction, an example), not to rerun until it passes.

Is the difference real?

With few cases, small differences are noise. A simple, honest tool is a paired comparison: only cases where the versions disagree carry information. If B fixes 6 cases and breaks 1, that is convincing. If it fixes 3 and breaks 2, it is a coin flip.

The sign test puts a number on it: if the versions were equally good, each disagreement would be a fair coin toss, so you ask how likely it would be to see at least this many wins for B.

Python Sign test on paired disagreements

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Treat the verdict as guidance, not law: with 20 cases you can only detect large gains, and testing many prompt variants on one set will eventually produce a lucky winner. That is another reason to keep a held-out set.

A release checklist for prompt changes

Before replacing a production prompt:

  1. Run the full set on old and new with the same settings.
  2. Read the diff: list every regression and decide each one.
  3. Check slices (tags, languages, long inputs), not just the total.
  4. Compare cost and latency, since a small gain may double tokens.
  5. Run the held-out set once.
  6. Add a test for any bug you fixed, so it stays fixed.
  7. Record the version and the scores (see templates and versioning).
flowchart LR
    A["Edit prompt"] --> B["Run set: old vs new"]
    B --> C{"Regressions?"}
    C -- "yes" --> D["Decide or fix"]
    D --> A
    C -- "no" --> E["Held-out run"]
    E --> F["Ship + record version"]

Every prompt change goes through the same loop.

Model upgrades are changes too

Prompts are tuned to a particular model. When the provider releases a new model version or retires an old one, run the same regression suite before switching. Prompts that worked through careful wording often behave differently on a new model, and your suite is the only thing that tells you before your users do.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Two prompts both score 80 percent. Why can they still differ?

    Totals hide which cases changed; a case-by-case diff reveals fixes and regressions.
  2. 2. A case passes 6 of 10 repeated runs. What does that suggest?

    A flaky case points to ambiguity in the instructions; clarify the prompt or add an example.
  3. 3. Version B fixes 3 cases and breaks 2. What should you conclude?

    Few, nearly balanced disagreements carry little evidence either way.
  4. 4. When should you rerun your regression suite apart from prompt edits?

    A model upgrade can change behaviour as much as an edit can, so the suite guards against both.

Summary

  • Compare versions case by case: fixed, regressed, still failing.
  • Repeat cases to measure flakiness; fix flakiness with clearer prompts, not reruns.
  • Use paired disagreements and a sign test to judge whether a gain is real.
  • Check slices, cost and latency, and keep a held-out set for the final check.
  • Re-run the suite whenever the model or settings change.

Next: LLM-as-Judge and Rubrics