Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Evaluation and Benchmarks

LLM-as-a-Judge and Human Evaluation

Evaluate open-ended LLM answers with humans and model judges: rubrics, Cohen's kappa, position bias, pairwise comparisons, and Bradley-Terry or Elo rankings in NumPy.

For open-ended output (an essay, advice, a summary, a chat reply) there is no single correct string to compare against. You need a judgment. The gold standard is careful human evaluation; the scalable alternative is to ask a strong model to act as the judge. Both have well-known weaknesses, and a good evaluator measures them.

Designing the judgment

Three common formats:

  • Pointwise: rate one answer against a rubric (for example 1 to 5 for accuracy, helpfulness, tone). Easy to scale, but raters and judges drift in what a "4" means.
  • Pairwise: compare answers A and B and pick the better, or call a tie. More reliable, because relative judgments are easier than absolute ones. Used by arena-style leaderboards.
  • Reference-guided: the judge sees a reference answer or checklist of required facts.

Write the rubric with concrete criteria and examples for each score level, not "rate quality". Ask for a short justification before the verdict when using a model judge, and parse the final label in a fixed format.

Measuring agreement: Cohen's kappa

Before trusting any rater (human or model), check how often two raters agree beyond chance. If both always say "good" 90% of the time, they agree 81% of the time by accident. Cohen's kappa corrects for that:

Text
kappa = (p_observed - p_expected) / (1 - p_expected)

Values near 0 mean chance-level agreement; above 0.6 is commonly called substantial; above 0.8 almost perfect.

Python Cohen's kappa for two raters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Raw agreement looks excellent (90%) but kappa is much lower, because both raters say "acceptable" most of the time and would agree often by chance. Always report kappa (or a similar chance-corrected score) when classes are imbalanced.

Known biases of model judges

  • Position bias: many judges favor the first (or second) answer shown. Mitigation: run each comparison twice with the order swapped and count only consistent verdicts.
  • Verbosity bias: longer answers win even when they are not better.
  • Self-preference: a model tends to prefer text that resembles its own output.
  • Format bias: confident tone, bullet lists and Markdown can sway the judge.
  • Limited ability: a judge cannot reliably grade facts or math it cannot itself solve; give it references, or run code, where possible.
Python Swap positions to cancel position bias

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Verdicts that flip when the order flips are position noise. Keeping only the consistent ones (or counting flips as ties) raises accuracy, at the cost of discarding data.

From pairwise votes to a ranking

Given many pairwise outcomes, we can fit scores per model with the same Bradley-Terry model used for reward models (RLHF and DPO): P(i beats j) = sigmoid(s_i - s_j). Elo ratings are an online variant. Chat arenas use this to turn millions of votes into rankings.

Python Fit Bradley-Terry scores from match results

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The fitted scores recover the true order and approximate gaps. A rating is only meaningful with uncertainty: bootstrap the votes to get an interval per model, and do not rank models whose intervals overlap.

Running a good human evaluation

  1. Pick raters who understand the domain; for medical or legal answers you need experts.
  2. Pilot the rubric on 30 items and fix unclear criteria before the main run.
  3. Blind and randomize: hide model names, shuffle order.
  4. Use 2 to 3 raters per item and measure kappa; adjudicate disagreements.
  5. Include gold items with known answers to catch careless raters.
  6. Keep a frozen test set and version the rubric so results stay comparable over time.
  7. Calibrate the model judge against humans: label a few hundred items by hand, run the judge, and report agreement. A judge with low kappa against your humans is not an evaluation.

When not to use a model judge

Do not use a model judge as the only measure for safety-critical claims, for tasks needing verified facts, or to evaluate the same model that produced the answers. Where an answer can be checked mechanically (tests, schema validation, exact values), do that first and use judgment only for what is left. See the evaluation harness project.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Two raters agree on 90% of items. Why can kappa still be low?

    Kappa subtracts the agreement expected by chance from the observed agreement, which matters with imbalanced labels.
  2. 2. How do you reduce position bias in a pairwise model judge?

    If the verdict flips with order, it reflects position noise rather than quality.
  3. 3. Why are pairwise comparisons often preferred to 1-5 ratings?

    Raters disagree on what a 4 means, but usually agree on which of two answers is better.
  4. 4. What must you do before trusting a model judge?

    Calibration against human judgments shows whether the judge measures what you care about.

Summary

  • Open-ended output needs human or model judgments, preferably pairwise and with explicit rubrics.
  • Measure agreement with chance-corrected statistics such as Cohen's kappa.
  • Model judges show position, verbosity and self-preference biases; swap orders and calibrate against humans.
  • Bradley-Terry or Elo turns pairwise votes into rankings, and rankings need confidence intervals.

Next: Prompting Patterns