Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 39 ยท Evaluation and Safety

Evaluating Agents

Learn to measure AI agents: build a test set, score outcomes and trajectories, use LLM judges carefully, run repeated trials and catch regressions with a runnable harness.

An agent that worked on your three favorite examples may fail on the fourth. Because language models are probabilistic and agents take many steps, intuition is a poor guide to quality. Evaluation (evals) replaces "it seems fine" with numbers: a fixed set of tasks, a way to score each run, and a report you can compare before and after every change.

This lesson covers what to measure, how to build a test set, how to score outcomes and trajectories, when to use an LLM as a judge, and how to handle randomness. You will write a small eval harness that runs here.

The real agents and judge models are not available on this site. The agent in the code is a deterministic stand-in with a deliberate weakness, so you can see how an eval catches it. The harness, metrics and report are what you would reuse with a real agent.

What to measure

Agents have several dimensions of quality. Pick the ones that matter for your product.

What to measure

Success Did it finish the task correctly?
Trajectory Did it use the right tools in a sane order?
1 / 2
  • Task success is the headline number: the percentage of tasks completed correctly.
  • Trajectory quality looks at the path: how many steps, whether it called a forbidden tool, whether it repeated itself.
  • Cost and latency decide whether the agent is affordable and pleasant to use.
  • Robustness asks whether small changes in wording change the outcome.

Build a test set

A good test set is a list of cases: an input, and a way to decide success. Start with 20 to 50 cases drawn from real or realistic requests, and include:

  • typical requests that must always work,
  • edge cases (empty input, ambiguous wording, long inputs),
  • adversarial cases (prompt injection, requests the agent must refuse),
  • cases from past failures. Every bug you fix becomes a permanent test.

Keep the set versioned alongside your code. Do not tune your prompt against the same cases you report on, or you will overfit; hold some cases back.

Scoring outcomes

When the answer is checkable, check it with code. This is the cheapest and most trustworthy scorer: exact match, number within tolerance, JSON schema valid, file contains a string, unit tests pass.

Python A small eval harness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The eval exposes the weakness at once: the subtraction case fails. Fix the agent, re-run, and the report shows whether anything else broke. That cycle is the heart of agent development.

Scoring trajectories

Sometimes the final answer is right for the wrong reasons, or wrong because of one bad step. Record each run's trace: the list of tool calls. Then assert properties of it.

Python Assert properties of a tool-call trace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Trajectory checks catch inefficiency and policy breaches that outcome checks miss.

LLM as a judge

Many outputs have no single right answer: a summary, an email, a plan. A common approach is to have another model grade the output against a rubric (for example, "is every claim supported by the sources? score 1 to 5"). It scales, but use it with care:

  • Write a specific rubric, not "is this good?".
  • Ask for a short reason before the score.
  • Calibrate: grade 30 cases by hand and compare. If the judge disagrees with humans, fix the rubric.
  • Watch for biases: judges favor longer answers and answers that resemble their own style.
  • Use a different or stronger model than the agent where possible.

Dealing with randomness

Run each case several times (for example five trials) and report the pass rate per case, not one lucky run. A case that passes 3 of 5 times is a flaky behavior worth fixing. Setting temperature to 0 reduces but does not remove variation. Track two numbers: pass@1 (probability a single try succeeds) and pass^k (all k tries succeed), the second being what reliability-sensitive products need.

Offline and online evaluation

Offline evals run on your fixed set before release; they are repeatable and cheap to run in CI. Online evaluation watches real traffic: thumbs up or down, task completion, escalations to humans, plus sampled reviews. Use offline evals to prevent regressions and online signals to discover new failure types, then add those as new test cases.

Common mistakes

  • Testing only happy paths.
  • Changing the test set and the agent at the same time, so scores are not comparable.
  • Trusting an LLM judge without ever checking it against humans.
  • Reporting an average that hides one catastrophic category. Break scores down by case type.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why are agent evals needed?

    A fixed test set gives repeatable numbers to compare before and after changes.
  2. 2. When an answer can be checked by code, you should...

    Programmatic checks are deterministic; use a judge only when no exact check exists.
  3. 3. What does a trajectory check catch?

    It inspects the sequence of tool calls, not just the final text.
  4. 4. Why run each case multiple times?

    Pass rate across trials reveals flaky behavior.
  5. 5. What should you do after fixing a production bug?

    Every real failure should become a permanent eval case.

Summary

  • Measure success, trajectory, cost, latency and safety on a fixed, versioned test set.
  • Include typical, edge, adversarial and past-failure cases; hold some back.
  • Prefer code-based scoring; use LLM judges with rubrics and human calibration.
  • Run multiple trials and report pass rates per category.
  • Pair offline evals (CI) with online monitoring, and feed new failures back as tests.

Next: Agent Guardrails and Prompt Injection