Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 38 ยท Evaluation and Metrics

Text Generation Metrics: BLEU, ROUGE, Perplexity

Learn how BLEU, ROUGE, perplexity, exact match and token F1 are computed, what they miss, and how to evaluate generated text, with runnable pure-Python implementations.

Scoring a classifier is easy: the label is right or wrong. Scoring generated text is hard, because many different sentences can be equally good. "The cat sat on the mat" and "A cat was sitting on the mat" mean the same thing but share few exact words. Automatic metrics are therefore proxies: cheap numbers that correlate, imperfectly, with quality. Knowing what each one actually measures lets you use it properly and distrust it at the right moments.

This lesson builds the most common ones from scratch: BLEU (translation), ROUGE (summarization), exact match and token F1 (question answering) and perplexity (language modeling).

The real evaluate implementations add tokenization options, smoothing and multiple references. The versions below are simplified, written in plain Python so you can run and inspect them. Do not use them to compare against published numbers; use the standard libraries for that (for BLEU, sacrebleu).

BLEU: precision of n-grams

BLEU asks how much of the generated text appears in the reference. It counts matching n-grams (word sequences of length 1 to 4), clipped so a word cannot be credited more often than it appears in the reference, takes the geometric mean of the four precisions, and multiplies by a brevity penalty that punishes outputs shorter than the reference (otherwise "the" alone would have perfect precision).

Python A small BLEU implementation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how the repeated "the" scores near zero thanks to clipping, how the short fragment is hurt by the brevity penalty, and how a valid paraphrase gets a mediocre score. BLEU is decent for comparing similar systems on corpus level with several references, and poor for judging one sentence or creative text.

ROUGE: recall of n-grams

ROUGE is the mirror image: how much of the reference is covered by the generated text. ROUGE-1 and ROUGE-2 use unigram and bigram overlap; ROUGE-L uses the longest common subsequence, which rewards words in the right order without needing them to be adjacent. Summarization papers usually report the F-measure of each.

Python ROUGE-1, ROUGE-2 and ROUGE-L

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last candidate is a word salad containing every reference word. It earns a perfect ROUGE-1 but a lower ROUGE-2 and ROUGE-L, which is exactly why the higher-order variants exist. Even so, a fluent paraphrase scores low and a fluent but wrong summary can score high: ROUGE does not know facts.

Exact match and token F1 (question answering)

Extractive QA models return a span of the context. Two metrics dominate: exact match (EM), which is 1 only if the normalized prediction equals a gold answer (lowercase, strip punctuation and articles), and token F1, the overlap between predicted and gold tokens, which gives partial credit. Systems report the maximum over several gold answers.

Python Exact match and token F1

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Perplexity: how surprised is the model?

For language models you can evaluate without any reference text beyond the test corpus: perplexity is exp(average negative log-likelihood per token). Intuitively, a perplexity of 20 means the model is as uncertain as if it were choosing uniformly among 20 equally likely next tokens. Lower is better. In Transformers the Trainer's eval_loss for a language model is already the average negative log-likelihood, so math.exp(eval_loss) is the perplexity.

Python Perplexity from token probabilities

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Perplexity compares models only if they use the same tokenizer and the same text: a model with a larger vocabulary spreads probability over more tokens per word. It also does not measure whether the output is true, helpful or safe.

Beyond n-gram overlap

  • BERTScore compares contextual embeddings of candidate and reference tokens, so paraphrases score higher than with BLEU or ROUGE.
  • Learned metrics such as COMET (translation) are trained on human judgments.
  • LLM-as-judge: a strong model scores or ranks outputs against a rubric. Useful and cheap, but biased toward long or confident answers and should be validated against a sample of human ratings.
  • Task-specific checks: unit tests for generated code, exact-answer checks for maths, schema validation for structured output.
  • Human evaluation remains the reference for open-ended quality. Do at least a small sample.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the brevity penalty in BLEU prevent?

    Without it, a one-word output that appears in the reference would have perfect precision.
  2. 2. Which statement about ROUGE is true?

    ROUGE is recall-oriented n-gram (or subsequence) overlap with the reference.
  3. 3. A language model has eval_loss 2.0. What is its perplexity?

    Perplexity = exp(loss) = exp(2.0), about 7.39.
  4. 4. Why might two language models' perplexities not be comparable?

    Perplexity is per token, so a different tokenization changes the number being averaged.
  5. 5. Exact match for a QA prediction 'the tower' versus gold 'Tower' (after normalization) is

    Normalization lowercases and removes the article 'the', so both become 'tower' and match exactly.

Summary

  • Generation metrics are proxies: BLEU measures n-gram precision with a brevity penalty, ROUGE measures n-gram or subsequence recall.
  • QA uses exact match and token F1 after normalization; language models use perplexity, exp(loss).
  • Overlap metrics miss paraphrases and facts; add BERTScore, learned metrics, task checks, LLM judges and human review.
  • Use standard libraries (sacrebleu, evaluate) for numbers you will publish.

Next: Error analysis, calibration and benchmarks