Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 39 ยท Evaluation and Safety

Agent Observability and Tracing

Make AI agents debuggable: record traces and spans for every model and tool call, track tokens, cost and latency, spot loops and replay failures with runnable Python.

When a normal program fails you read a stack trace. When an agent fails, the bug is spread across many steps: a model call that picked the wrong tool, a tool result that was empty, a prompt that grew too long. Without a record of what happened, you can only guess. Observability is that record: structured traces of every run so you can see, measure and replay what the agent did.

This lesson introduces traces and spans, shows how to build a tiny tracer, and explains the metrics (tokens, cost, latency, errors) and alerts that keep an agent healthy in production. It also covers privacy, because traces contain user data.

Production teams use tracing platforms and standards such as OpenTelemetry. None can run on this site, so the tracer below is a minimal hand-written stand-in that stores spans in a list. The data model (trace, span, attributes) matches what those tools use.

Traces and spans

A trace is the whole story of one agent run. It is made of spans: timed steps such as "model call", "tool call: search" or "retrieve documents". Spans nest: the run span contains steps, and a step contains its tool call.

flowchart TD
  R["Trace: answer user question (2.4 s)"] --> M1["Span: model call 1 (0.9 s)"]
  R --> T1["Span: tool search (0.6 s)"]
  R --> M2["Span: model call 2 (0.8 s)"]

Each span records start, duration, inputs, outputs and any error.

For each span, store the name, start time and duration, inputs and outputs (possibly redacted), status (ok or error) and attributes such as model name, token counts and tool arguments.

A minimal tracer

A context manager makes spans easy to add around any step. It records the duration even when the step raises an error.

Python Trace model and tool steps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Printing a span list is the beginner version of a trace viewer. Real viewers draw a timeline so you can see at a glance that most of the time went into one slow tool.

Metrics worth tracking

From traces you can compute the numbers that matter:

  • Latency: per step and overall; watch the 95th percentile, not just the average.
  • Tokens and cost: input and output tokens times price per token, per run and per user.
  • Error rate: tool failures, malformed model output, timeouts.
  • Steps per run: a rising average means the agent is wandering.
  • Success signals: completed tasks, thumbs up or down, escalations to humans.
Python Cost and latency report from spans

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The prices in the code are placeholders. Always use your provider's current rates and keep them in configuration.

Detecting loops and waste

Agents sometimes repeat the same tool call forever. A simple detector flags the same call with the same arguments more than a few times, so you can alert or stop the run.

Python
from collections import Counter

def find_repeats(tool_calls, limit=2):
    counts = Counter((c["tool"], str(c["args"])) for c in tool_calls)
    return [key for key, n in counts.items() if n > limit]

Combine it with hard limits (maximum steps, maximum cost per run) enforced in code, so a bad run stops itself.

Logging for replay

The best debugging aid is the ability to replay a failed run: save the exact messages sent to the model and the tool outputs received. Re-running with the same inputs (and temperature 0) often reproduces the failure, and you can then change the prompt and compare. Replays also turn real failures into test cases for your eval set.

Privacy and retention

Traces contain what users typed and what tools returned, which may include personal data. Redact emails, keys and card numbers before storing, limit who can read traces, set a retention period (for example 30 days) and honor deletion requests. Never log secrets such as API keys. Use the redaction idea from guardrails.

Alerts and dashboards

Start with a small dashboard: runs per day, success rate, median and p95 latency, cost per run, error rate by tool. Add alerts for sudden jumps: cost per run doubling, error rate above a threshold, a user making hundreds of runs an hour. Review a sample of real traces weekly; reading ten runs teaches you more than any chart.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a span?

    A trace is made of spans; each records timing, inputs, outputs and status.
  2. 2. Why watch p95 latency and not only the average?

    A few very slow runs can ruin the experience while the average looks fine.
  3. 3. What helps you reproduce a failed run?

    Replaying identical inputs lets you debug and then turn the case into a test.
  4. 4. What should you do with sensitive data in traces?

    Traces hold user data, so treat them with the same care as the data itself.

Summary

  • A trace records one run as nested spans with timing, inputs, outputs and status.
  • Track latency (including p95), tokens, cost, errors, steps per run and success signals.
  • Detect loops and enforce step and cost limits in code.
  • Save enough to replay failures and convert them into eval cases.
  • Redact sensitive data, restrict access and set a retention period.

Next: The Agent Framework Landscape