Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 53 ยท Evaluation

LLM-as-Judge and Human Review

Use a model to grade open-ended answers: write rubrics, run pairwise comparisons, spot position and verbosity bias, and measure agreement with Cohen's kappa.

Exact match and regex checks cannot tell you whether a summary is faithful, an explanation is clear or a reply is polite. For open-ended output, you need a grader that understands language. Two graders are available: people, who are slow and expensive but the reference standard, and LLM judges, which are fast and cheap but imperfect.

An LLM-as-judge is simply a second model call that reads the input, the answer and a set of criteria, and returns a score or a preference. This lesson shows how to write a good judge prompt, which biases to expect, and how to check a judge against human labels with Cohen's kappa. Everything here follows from Evaluating LLM Applications.

Rubrics: tell the judge what "good" means

"Rate this answer from 1 to 10" produces noise, because the scale has no meaning. A rubric defines each level in observable terms:

TEXT
You grade a support reply. Score each criterion 0 or 1.

accurate:  every factual claim is supported by the provided policy text
complete:  the reply addresses every question the customer asked
tone:      polite, no blame, no jargon

Policy text: {policy}
Customer message: {message}
Reply to grade: {reply}

First write one short reason per criterion, then output JSON:
{"accurate": 0|1, "complete": 0|1, "tone": 0|1}

Good rubric habits:

  • Split criteria. Several yes/no questions beat one vague overall score.
  • Use small scales. Pass/fail or 1 to 3 is easier to apply consistently than 1 to 10.
  • Ask for the reason first, then the score. Writing reasons tends to improve the grade and helps you debug the judge.
  • Give the reference. If you have a golden answer or source text, include it.
  • Parse strictly. Ask for JSON and validate it, as in Getting Structured Output.

Pairwise comparison

People and models are better at saying "A is better than B" than at assigning absolute numbers. In a pairwise eval, the judge sees two answers to the same input (for example the old prompt and the new prompt) and picks a winner or a tie. Your win rate is the share of cases the new version wins. It is a direct answer to "did my change help?"

Judge biases to expect

Position Prefers the answer shown first (or last).
Verbosity Prefers longer answers, even padded ones.
Self-preference Favors text written by itself or its family.
Style Rewards confident tone over correctness.

Measure and reduce these. Do not assume they are absent.

1 / 2

The most reliable fix for position bias is cheap: run every pair twice with the order swapped. If the judge picks the same underlying answer both times, count it as a win; if it flips, count a tie.

Python Swapping order to cancel position bias

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With the naive setup the new answer "wins" every time simply because it was shown first. Judging both orders turns the close calls into ties and keeps only the clear results.

Verbosity and other biases

Judges often reward longer, more detailed-looking answers. Test this directly: take a good answer, pad it with filler sentences, and see whether its score rises. If it does, add a rule to the rubric ("do not reward length; penalize repetition") and consider scoring against a reference answer. Also be careful when the judge and the system under test are the same model, because self-preference can inflate scores. Where you can, use a different model as judge, and keep the judge's own prompt and version fixed so scores stay comparable over time.

Is the judge any good? Agreement and Cohen's kappa

Before trusting a judge, compare it with humans on a labeled sample (50 to 100 cases is a start). Raw percent agreement is misleading when one label dominates: if 90% of answers are good, a judge that always says "good" agrees 90% of the time while knowing nothing.

Cohen's kappa corrects for agreement expected by chance:

TEXT
kappa = (observed agreement - chance agreement) / (1 - chance agreement)

Chance agreement is computed from how often each rater uses each label. Kappa is 1 for perfect agreement, 0 for no better than chance, and negative if they disagree systematically. A common (informal) reading: 0.4 to 0.6 is moderate, 0.6 to 0.8 substantial, above 0.8 near-perfect. The right threshold depends on the stakes.

Python Percent agreement versus Cohen's kappa

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The lazy judge scores 85% agreement but a kappa of exactly 0, because it adds no information. Measuring human-to-human kappa tells you how subjective the task is: a judge cannot reasonably be expected to agree with people more than they agree with each other.

Human review

Humans stay in the loop in three places:

  • Building the reference. Label a sample to calibrate the judge and to create golden answers.
  • Auditing. Regularly read a random sample of judge decisions, plus every case where the judge and a cheap check disagree.
  • High-stakes calls. Medical, legal and safety decisions need a person, not an automatic score.

To get consistent human labels: write guidelines with examples of each score, label a pilot batch together, discuss disagreements, update the guidelines, and have at least two people label overlapping items so you can compute kappa. Keep labelers blind to which system produced an answer.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is a rubric with several yes/no criteria usually better than 'rate 1 to 10'?

    Concrete, narrow criteria reduce the vagueness that makes broad numeric scales noisy.
  2. 2. How do you reduce position bias in a pairwise judge?

    If the preference follows the position instead of the answer, swapping the order exposes it.
  3. 3. A judge that always says 'good' agrees with humans 85% of the time on a dataset that is 85% good. What is its Cohen's kappa?

    Kappa subtracts the agreement expected from label frequencies alone, which leaves nothing here.
  4. 4. What does human-to-human kappa tell you?

    If trained people agree only moderately, the task is ambiguous and a judge cannot be expected to do better.

Summary

  • Use model judges for qualities code cannot check; define the criteria in a small-scale rubric and ask for reasons before scores.
  • Pairwise win rates answer "is the new version better?" more reliably than absolute scores.
  • Expect position, verbosity and self-preference bias; swap order, control length and prefer a different judge model.
  • Validate a judge against human labels with Cohen's kappa, not raw percent agreement.
  • Keep humans for calibration, audits and high-stakes decisions, with written guidelines and overlapping labels.

Next: put numbers and error bars on your results in Metrics: Retrieval, Benchmarks and Statistics.