Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 38 ยท Evaluation and Testing Prompts

Metrics for Prompt Outputs

Choose the right metric for each prompt: exact match, accuracy, precision and recall, token F1, rule checks, format validity and latency, with runnable code.

A test set is only as useful as the way you score it. Model output is free text, and free text is awkward to grade: "Shipping", "shipping." and "It's a shipping question" mean the same thing to a human and three different things to ==. This lesson covers the main kinds of metric, when each fits, and how to avoid the classic traps.

The rule of thumb: use the cheapest metric that you trust. Code-based checks are fast, free and repeatable; reserve human or model judges (next lessons) for qualities code cannot see.

Exact match, with normalization

For classification and short answers, compare the output with the expected label. Normalize first: trim, lowercase, strip punctuation. Better still, make the prompt demand a constrained answer ("Reply with exactly one of: billing, shipping, other") so normalization has less to do.

Python Exact match with normalization

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The third output stays wrong under both, and that is correct: the prompt did not follow the format. Do not widen the matcher until every sentence passes; fix the prompt, or switch to a different metric.

Accuracy, precision and recall

Accuracy (fraction correct) misleads when classes are unbalanced. If only 5 percent of messages are fraud, a prompt that never says "fraud" is 95 percent accurate and useless. For a target class, track:

  • Precision: of the items the prompt flagged, how many were right? Low precision means false alarms.
  • Recall: of the items that should be flagged, how many did the prompt find? Low recall means misses.
  • F1: the harmonic mean of the two, a single balanced score.

Choose by cost: a spam filter values precision (do not block real mail); a safety screen values recall (do not miss threats).

Python Precision, recall and F1 by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Accuracy is a respectable 0.80 while fraud recall is only 0.67: one in three real frauds slipped through.

Token overlap (F1) for short free text

For answers like "5 business days" versus "within five to seven business days", exact match is too strict. Token F1 counts shared words between the output and a reference. It is crude (it knows nothing about meaning) but useful for extractive questions, and it is the standard score for many question-answering benchmarks.

Python Token-level F1 for short answers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Verbose answers lose points even when correct, which is why token F1 pairs well with a prompt that asks for short answers.

Rule-based checks

Many requirements can be written as small functions:

  • Is the output valid JSON that matches the schema?
  • Is it under 80 words? Does it contain a required disclaimer?
  • Does it cite only passage ids that were provided?
  • Does it avoid forbidden words, URLs or personal data?
  • Is every item of a list in the required language?
Python A rule-based checker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rule checks pair naturally with structured output: the same validator can score tests offline and block bad answers in production.

Operational metrics

Quality is not the only number. Track, per prompt version:

Metric Why it matters
Input and output tokens Direct cost
Latency (median and 95th percentile) User experience; averages hide slow tails
Format failure rate How often downstream code would break
Refusal rate Over-cautious prompts refuse valid requests
Retries needed Hidden cost of unreliable prompts

Common traps

  • One metric to rule them all. Report several: accuracy and recall and cost.
  • Optimizing the metric, not the goal. Pushing token F1 up can reward copying the reference style.
  • Ignoring variance. A single run is one sample; for non-deterministic settings, run each case several times (regression testing).
  • No baseline. A score means little until you know what the old prompt or a trivial rule scores.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A fraud detector says 'ok' to everything and is 95 percent accurate. What does that show?

    Always predicting the majority class gets high accuracy while finding none of the cases you care about.
  2. 2. Which metric should a safety screen prioritize?

    Missing a real threat is costly, so you accept some false alarms to catch more true ones.
  3. 3. What does token F1 measure?

    It counts shared tokens; it is cheap and crude, and penalizes verbose answers.
  4. 4. Why use the 95th percentile latency as well as the average?

    A few very slow responses barely move the mean but ruin the experience for those users.

Summary

  • Use the cheapest metric you trust: exact match, rule checks, then token overlap, then judges.
  • Normalize before comparing, but fix the prompt rather than loosening the matcher endlessly.
  • Use precision, recall and F1 when classes are unbalanced; choose by the cost of each error.
  • Track cost, latency, refusals and format failures alongside quality.

Next: Regression Testing Prompts