Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท Evaluation and Benchmarks

Perplexity and Language Metrics

Compute cross-entropy, perplexity and bits per character from model probabilities, and learn exact match, token F1, BLEU-style overlap and when each metric misleads.

How good is a language model? The question has no single answer, because "good" depends on the job. Evaluation therefore uses layers of metrics: intrinsic ones that measure how well the model predicts text (perplexity), task metrics that compare outputs with references (exact match, F1, BLEU), and higher-level judgments covered in the next lessons. This lesson starts at the bottom, with the quantity that training itself minimizes.

Cross-entropy and perplexity

A language model assigns a probability p(token | previous tokens) to every token. Given a test text of N tokens, the cross-entropy (average negative log-likelihood) is

Text
H = -(1/N) * sum_i log p(token_i | context_i)

and the perplexity is PPL = exp(H) (if logs are natural). Perplexity has a friendly reading: the model is as uncertain as if it were choosing uniformly among PPL equally likely tokens at each step. A perplexity of 1 means perfect prediction; a perplexity equal to the vocabulary size means the model knows nothing.

Python Compute perplexity from probabilities

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A single very unlikely token (the 0.05 and 0.1 here) pulls perplexity up a lot, because the logarithm punishes confident mistakes severely.

Comparing models fairly

Perplexity is only comparable when the text and the tokenization are the same. A model with a larger vocabulary splits text into fewer tokens, so its per-token perplexity is not comparable with another tokenizer's. Two fixes:

  • Bits per character (BPC) or bits per byte (BPB): total negative log-likelihood divided by the number of characters or bytes, which is tokenizer-independent.
  • Evaluate on the same tokenizer or convert: BPB = total_bits / total_bytes.
Python Why per-token perplexity misleads across tokenizers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both models are equally good at predicting the text, yet their per-token perplexities differ wildly. Bits per byte gives the same answer for both.

Perplexity is not quality

Low perplexity means the model predicts the dataset well. It does not guarantee helpful, truthful or safe answers. An instruction-tuned or aligned model often has higher perplexity on web text than its base model, even though people prefer it. Use perplexity to monitor training, compare quantization levels (quantization), and detect data problems, not to rank assistants.

Reference-based metrics for generated text

When there is a reference answer, you can score overlap.

  • Exact match (EM): the normalized answer equals the reference. Strict, ideal for short factual answers.
  • Token F1: harmonic mean of precision and recall over tokens shared between answer and reference. Gives partial credit.
  • BLEU: geometric mean of n-gram precisions with a brevity penalty, designed for machine translation.
  • ROUGE: recall-oriented n-gram and longest-subsequence overlap, used for summarization.
  • chrF: character n-gram F-score, robust for morphologically rich languages.
  • Embedding-based: BERTScore and similar compare meanings through embeddings.
Python Exact match, token F1 and a simple BLEU

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

"Paris" is correct but scores low on overlap metrics; "It is located in the French capital" is also correct and scores near zero. That is the weakness of surface metrics: they punish valid paraphrases. They work best for constrained outputs such as numbers, labels, entities and code that can be executed.

Choosing a metric

Output type Good choice
Classification label accuracy, F1 per class
Short factual answer normalized exact match, token F1
Code unit-test pass rate, pass@k (Benchmarks)
Translation chrF, BLEU, plus human or model judging
Open-ended text pairwise preference, rubric scoring (judges)
Language modeling itself perplexity or bits per byte

Always report the sample size and an uncertainty estimate. A 2-point gain on 50 test questions is usually noise.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A model has perplexity 8 on a text. What does that roughly mean?

    Perplexity is the exponential of cross-entropy, the effective number of equally likely choices per token.
  2. 2. Why can't you compare per-token perplexity between models with different tokenizers?

    Use bits per byte or bits per character, which normalize by text length instead of token count.
  3. 3. Which statement about perplexity and assistant quality is correct?

    Alignment changes behavior away from pure web-text prediction, so perplexity is a poor ranking tool for assistants.
  4. 4. Why do exact match and BLEU fail on open-ended answers?

    Surface overlap does not capture meaning, so valid answers phrased differently score low.

Summary

  • Cross-entropy is the mean negative log-likelihood; perplexity is its exponential.
  • Compare across tokenizers with bits per byte, not per-token perplexity.
  • Overlap metrics (EM, F1, BLEU, ROUGE) suit constrained outputs, and miss valid paraphrases.
  • Perplexity monitors training and compression, but does not rank assistants.

Next: Benchmarks and Contamination