Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 32 ยท Evaluation and Deployment

Tracing and Observability

See inside LLM apps: traces, runs, spans, metadata, tags, latency and token usage. Enable LangSmith tracing, add callbacks and build a tiny tracer in Python.

When a normal program misbehaves you read a stack trace. When an agent misbehaves, there may be no error at all: it just gave a strange answer after seven model calls and four tool calls. To debug that you need to see every step: the exact prompt sent, what the model replied, which tools ran with which arguments, how long each took and what it cost. That record is a trace, and collecting traces is called observability. For LangChain the standard tool is LangSmith, but the ideas apply to any tracing system (OpenTelemetry-based tools, Langfuse, Phoenix and others).

LangSmith needs an account and an API key, so it cannot run on this site. Real code is read-only. The runnable examples build a tiny tracer in plain Python so you can see what a trace contains.

Traces, runs and spans

A trace is the record of one request through your app. It is a tree of runs (also called spans): the agent run contains model runs and tool runs, a retriever run contains an embedding run, and so on. Each run stores:

  • its name and type (chain, llm, tool, retriever);
  • input and output;
  • start time, end time (so latency);
  • token counts and cost for model runs;
  • errors, if any;
  • metadata and tags you attach.
flowchart TD
    A["agent run (4.2 s)"] --> M1["llm run (1.1 s)"]
    A --> T1["tool: search_docs (0.4 s)"]
    A --> M2["llm run (2.6 s)"]
    T1 --> R["retriever (0.3 s)"]

A trace is a tree of timed runs.

Turning on LangSmith

For LangChain and LangGraph code, tracing is switched on by environment variables. No code changes are needed.

Terminal
# read-only: set these in your shell or .env
export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your key>"
export LANGSMITH_PROJECT="support-bot-dev"

After that, every invoke of a chain, agent or graph appears in the LangSmith web UI under the named project, with the tree of runs, the inputs and outputs of each step, timings and token usage. You can click any model call to see the exact prompt, which is often all you need to understand a bad answer.

Never put real API keys in source code or in lesson-style examples; keep them in environment variables or a secrets manager.

Tracing your own functions

Code that is not LangChain (a helper, a database query) can join the trace with a decorator:

Python
# Real LangSmith (read-only)
from langsmith import traceable

@traceable(name="load_customer", tags=["db"], metadata={"source": "postgres"})
def load_customer(customer_id: str) -> dict:
    return {"id": customer_id, "plan": "pro"}

Calls to load_customer made inside a traced run show up as child runs. You can also attach metadata and tags per request, such as the user id, app version or experiment name, which makes filtering possible later ("show me failed runs for version 2.3 on the pro plan").

Python
# Real LangChain (read-only)
agent.invoke(
    {"messages": [{"role": "user", "content": "Where is my order?"}]},
    config={
        "tags": ["support", "v2.3"],
        "metadata": {"user_tier": "pro"},
        "run_name": "support-agent",
    },
)

Build a tiny tracer

This stand-in records the same kind of data: a decorator that logs name, inputs, output, duration and parent/child structure.

Python Stand-in: a decorator that records runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real tracers do this with timings and send the result to a server. Notice the tree: answer is the parent of retrieve and call_model.

What to look for in a trace

Open a bad trace and ask:

  1. Was the right context retrieved? Many wrong answers come from missing or irrelevant documents, not from the model.
  2. Was the prompt what you expected? Variables sometimes render empty or duplicated.
  3. Which tool did it choose, with what arguments? Wrong tool means unclear descriptions.
  4. How many loops? Many repeated calls suggest a stuck agent.
  5. Where did the time go? Often one slow tool or a long generation.
  6. Where did the tokens go? A huge tool output copied into every later call is a common cost leak.

Metrics, logs and alerts

Traces explain single requests; production also needs aggregates.

  • Latency percentiles (p50, p95), time to first token for streaming.
  • Token usage and cost per request, user and feature.
  • Error rate, timeouts, rate-limit responses.
  • Quality signals: thumbs up/down, "regenerate" clicks, evaluation scores on sampled traffic.
  • Tool statistics: calls per request, failure rate per tool.

LangSmith dashboards and monitors cover many of these, and you can feed the same numbers to your own metrics system.

Python Stand-in: aggregate latency and cost from runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The single slow, expensive, failed run dominates p95 and the cost, which is exactly what you would investigate in its trace.

Privacy

Traces contain user messages and tool outputs, possibly personal data or secrets. Decide what to record: mask emails and card numbers before logging, set retention periods, restrict who can open traces, and check your provider's data-handling terms. For regulated data, consider self-hosting a tracing backend.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a trace?

    A trace records each step (model, tool, retriever) with inputs, outputs, timing and errors.
  2. 2. How do you enable LangSmith tracing for LangChain code?

    Tracing is switched on by environment variables; no code changes are required.
  3. 3. A wrong answer is often caused by what, visible in a trace?

    Checking the retriever run shows whether the model had the right documents.
  4. 4. Why mask personal data in traces?

    Traces record real user content, so privacy controls and retention policies matter.

Summary

  • A trace is a tree of runs with inputs, outputs, timing, tokens and errors.
  • LangSmith tracing needs only environment variables for LangChain and LangGraph code; @traceable adds your own functions.
  • Tags and metadata make traces searchable; aggregates give latency, cost and error rates.
  • Read traces by checking retrieval, prompts, tool choices, loops, time and tokens.
  • Protect privacy in what you record.

Next: Serving with FastAPI and Streaming