Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 38 ยท Prompting Fundamentals

Iterating and Versioning Prompts

Improve prompts the disciplined way: change one thing at a time, test on a fixed set of inputs, diff versions and keep a changelog. Runnable Python included.

Your first prompt is a draft. The skill is in the revision process: noticing a failure, forming a guess about its cause, changing one thing, and checking whether the guess was right. Done sloppily, iteration is just rewording until something looks good on the one example you tried. Done carefully, it is engineering.

The trap: tuning to one example

You test a prompt on a single input, it fails, you tweak it, and it works. But you have learned only that this input now works. The tweak may have broken three others. The fix is simple: keep a small test set of inputs, including awkward ones, and rerun all of them after every change.

A good starter test set has 10 to 20 inputs:

  • A few typical cases.
  • A few edge cases (empty input, very long input, unusual wording).
  • Cases that failed before (regressions you never want back).

Change one thing at a time

When you edit three things at once and the output improves, you do not know which edit helped, or whether one of them hurt. Isolate changes:

  1. Write down the failure ("summary includes a price the source did not state").
  2. Form a hypothesis ("the prompt never says to avoid unstated facts").
  3. Make one edit ("add: use only facts from the text").
  4. Rerun the whole test set and compare.
flowchart LR
  A["Run test set"] --> B["Find a failure"]
  B --> C["Hypothesis"]
  C --> D["One edit"]
  D --> A

Iteration is a loop with a fixed test set, not a series of one-off tries.

Diff your prompts

Treat prompts like code: you should be able to see exactly what changed between versions. Python's difflib gives you a text diff.

Python Diff two prompt versions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Lines starting with - were removed, + were added. A diff in a code review is how teammates catch a rule that was accidentally deleted.

A mini test harness

A harness runs each prompt version against each test input, applies a check, and reports a score. Because we cannot call a real model here, this one uses a stand-in model whose replies are canned. The point is the structure: versions x inputs x checks.

Python Compare prompt versions on a test set (stand-in model)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that v1 passes on some inputs and fails on others; a single-example test could have fooled you either way. The full chapter on evaluation expands this into scoring, graders and regression suites.

Keep a changelog

A short note for each version answers "why did we change this?" six months later.

TEXT
summarize@1  initial draft
summarize@2  limit to 3 sentences (outputs too long); require stated facts only
             (invented a price in test #4)
summarize@3  add fallback: if no date, write "unknown" (test #7)

Store prompts and changelogs in version control next to your code. Tag the version that is in production, so you can roll back quickly when a new prompt misbehaves.

When to stop iterating

Stop when the prompt meets your criteria on the test set, not when one output looks good. Also stop when edits show diminishing returns: if three tweaks in a row change nothing measurable, the problem may be elsewhere (wrong model, missing context, a task that needs splitting into steps).

When a prompt edit is not the answer

Some failures are not wording problems:

  • Missing information: the model cannot know it. Supply context.
  • Too many jobs at once: split into steps (prompt chaining).
  • Wrong settings: truncated output, high randomness.
  • Task better done in code: arithmetic, sorting, exact counts.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the danger of tuning a prompt on a single example?

    A fixed test set shows whether a change helps overall or only for one input.
  2. 2. Why change only one thing at a time?

    Isolated changes let you attribute results to specific edits.
  3. 3. What should a starter test set include?

    Edge cases and regressions are the inputs most likely to reveal a bad change.
  4. 4. Three prompt tweaks in a row change nothing measurable. What is a good next step?

    Diminishing returns suggest the cause is not wording.

Summary

  • Keep a fixed test set and rerun it after every change.
  • Change one thing at a time with a stated hypothesis.
  • Diff and version prompts like code, with a short changelog and the production version tagged.
  • Stop when the test set meets your criteria; if wording stops helping, look at context, task size, settings or code.

Next: Common Prompt Mistakes