Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 37 ยท Evaluation and Benchmarks

Benchmarks and Contamination

Learn how LLM benchmarks work, how to attach bootstrap confidence intervals to accuracy, compute the unbiased pass@k estimator, and detect test-set contamination.

A benchmark is a fixed set of tasks with a scoring rule, so different models can be compared on equal terms. Leaderboards turn that into a single number per model, which is convenient and dangerous. This lesson explains the common benchmark types, how to put honest error bars on a score, how pass@k works for code, and how test data leaks into training sets.

Types of benchmarks

Kind Example task format Scoring
Knowledge, multiple choice questions with options A to D across many subjects accuracy
Reasoning, math word problems with a final numeric answer exact match on the answer
Code generation write a function from a docstring, then run hidden unit tests pass rate, pass@k
Reading comprehension passage plus question EM, F1
Instruction following prompts with checkable constraints ("use exactly 3 bullets") rule-checked pass rate
Truthfulness and safety questions designed to elicit common falsehoods or unsafe replies accuracy, refusal rate
Long context find a fact in a very long document retrieval accuracy
Agentic multi-step tasks in a sandbox, such as fixing software issues task success rate
Human preference arenas users compare two anonymous answers Elo or Bradley-Terry ratings

A score is an estimate

If a model scores 82% on a test of 100 questions, the true accuracy on all similar questions could easily be 75% or 89%. Treat the score as a sample statistic and attach a confidence interval. A simple, assumption-light method is the bootstrap: resample the per-question results with replacement many times and look at the spread of the mean.

Python Bootstrap a confidence interval for accuracy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The interval narrows roughly with 1/sqrt(n). At n = 50 it spans well over 10 points, so a leaderboard gap of 2 points is meaningless. When comparing two models on the same questions, use a paired bootstrap on the per-question differences, which is far more sensitive because question difficulty cancels.

Python Paired comparison of two models

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

pass@k for code

Code benchmarks run generated programs against hidden tests. Because sampling is random, you can ask for n samples per problem, count how many (c) pass, and estimate the chance that at least one of k random samples would pass. The unbiased estimator, per problem, is

Text
pass@k = 1 - C(n - c, k) / C(n, k)

and the benchmark score is the average over problems. Computing the product form directly avoids huge binomials.

Python Unbiased pass@k

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

pass@1 is what a user experiences with one try; pass@10 rewards models that can generate diverse candidates, useful when you can run tests and keep the one that passes.

Contamination

Contamination (or leakage) means the benchmark's test items, or their answers, appeared in the model's training data. The model then looks smarter than it is: it has memorized rather than reasoned. Web-scale pretraining makes this very likely for any public benchmark: test sets get copied to GitHub, forums and blog posts.

Detection methods:

  • N-gram overlap: flag test items that share long n-grams (for example 13 words) with training documents.
  • Canary strings: unique markers placed in benchmark files; a model that can complete them has seen the files.
  • Perplexity tests: unusually low perplexity on test items, or on them in their original order.
  • Paraphrase tests: rewrite questions; a large accuracy drop suggests memorization.
  • Fresh benchmarks: build tests after the model's data cutoff or keep a private held-out set.
Python N-gram overlap contamination check

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A shared 8-gram is strong evidence of copying. Real pipelines hash n-grams so they can check billions of documents.

Other benchmark problems

  • Saturation: top models score 90%+ and differences vanish; harder successors replace old tests.
  • Label errors: a noticeable share of questions in popular sets have wrong or ambiguous answers.
  • Prompt sensitivity: changing option order, format or few-shot examples shifts scores by points.
  • Overfitting to the leaderboard: teams tune on the public test set.
  • Mismatch with real use: a multiple-choice score tells you little about your application. Build a small task-specific evaluation.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A model scores 80% on 50 questions. What is the best way to describe the result?

    With only 50 items sampling noise is large; report an interval, and avoid ranking models on small gaps.
  2. 2. What does pass@10 measure?

    It estimates success when you can draw ten attempts and keep any that pass.
  3. 3. Which finding suggests contamination?

    Memorized items fail when surface wording changes, while true skill would transfer to paraphrases.
  4. 4. Why use a paired bootstrap when comparing two models on the same test set?

    Both models face the same questions, so their per-item difference isolates the model effect.

Summary

  • Benchmarks fix tasks and scoring so models are comparable, but the scores are noisy estimates: add bootstrap intervals and use paired tests.
  • pass@k estimates the chance that one of k samples passes the tests.
  • Contamination inflates scores; detect it with n-gram overlap, canaries and paraphrase checks, and prefer fresh or private tests.
  • Public benchmark results seldom predict your own application's quality.

Next: LLM-as-a-Judge and Human Evaluation