Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 37 Β· APM and Tracing

APM Overview

Learn what Datadog APM does, how a request becomes a trace made of spans, and how to read latency, error rate and throughput for each service.

Metrics tell you that something is slow. Logs tell you what one process wrote down. Application Performance Monitoring (APM) tells you where a single request spent its time as it crossed your code, your database and your other services. Datadog APM collects that data as traces and turns it into service pages, a service map and alerts.

This lesson explains the vocabulary you will use for the rest of the chapter. The Datadog product cannot run on this site, so the code below is a clearly labelled stand-in that models the same ideas in plain JavaScript. Menu names and settings describe the real product; I never show output you could not reproduce.

Traces and spans

A trace is the story of one request. It is made of spans. A span is one timed operation: an HTTP handler, a SQL query, a call to another service. Spans form a tree: each span has a span_id and (except the first) a parent_id, and all spans of the request share one trace_id.

Every span carries:

  • service: the name of the application that did the work (checkout-api).
  • operation name: the kind of work (http.request, postgres.query).
  • resource: the specific thing, such as GET /cart/{id} or SELECT * FROM carts WHERE id = ?. Resources group similar spans together.
  • start and duration, an error flag, and tags (key-value pairs such as env:prod).
flowchart LR
  B["Browser request"] --> W["web-frontend<br/>http.request 480 ms"]
  W --> A["checkout-api<br/>http.request 410 ms"]
  A --> D["postgres.query 35 ms"]
  A --> P["payments-api<br/>http.request 300 ms"]

One trace: the longest branch (payments-api) explains most of the latency.

The three golden numbers

For every service and every resource, Datadog APM computes the same three trace metrics:

Metric Meaning Example metric name
Requests (throughput) How many spans per second trace.http.request.hits
Errors How many spans had an error trace.http.request.errors
Latency How long spans took (average, p75, p90, p95, p99) trace.http.request.duration

These come from all your traffic, even when only part of it is stored as full traces. That is why you can build monitors on them reliably. You will see the sampling details in Sampling, Retention and Profiling.

What the product shows you

In the Datadog app, open APM in the left navigation. The main pages are:

  • Service Catalog: every service that reports data, with owners, health and links.
  • Service page: graphs for requests, errors and latency, plus the busiest resources and the dependencies the service calls.
  • Traces explorer: search stored traces by tag (service:checkout-api status:error), then open a flame graph that lays the spans out on a time axis.
  • Service Map: a graph of services and the calls between them.

Everything is filtered by env (for example prod or staging), so tag your data well. The Metrics and Tags chapter covers tagging in depth.

A stand-in: build and read a trace

The code below is not the Datadog library. It builds a small trace as plain objects and prints the time spent in each span, excluding children (the "self time"). Self time is what a flame graph makes visible.

JS Stand-in: self time per span

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that the root span looks slow (480 ms) but most of that time is spent waiting for children. The span with the most self time is the real bottleneck.

JS Stand-in: the three golden numbers

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The average hides the 900 ms request. That is why you should always watch percentiles such as p95 and p99, not only the average.

When a service looks slow, check percentiles first. A fast average with a terrible p99 means a small group of users has a bad experience.

How data gets to Datadog

Your application runs a tracing library (the Datadog tracer, also called dd-trace). The library sends spans to the Datadog Agent on the same host, usually on port 8126. The Agent batches them, computes the trace metrics, and forwards them to Datadog. You install the tracer in Instrumenting an Application, and the Agent is covered in the Agent and Integrations chapter.

How a trace travels

Tracer Library in your app creates spans
Agent Receives on port 8126
Datadog Stores, indexes, graphs

The app never talks to Datadog directly.

1 / 2

Check your understanding

Quick check Β· 4 questions

of 4 answered You scored / 4 Β· perfect!

  1. 1. What is a trace?

    A trace groups all spans that share one trace_id, so it describes a whole request.
  2. 2. A span has a duration of 300 ms and one child span of 280 ms. What is its self time?

    Self time is the duration minus the time of its children: 300 - 280 = 20 ms.
  3. 3. Why watch p95 and p99 latency in addition to the average?

    A few very slow requests barely move the average but matter to real users.
  4. 4. Where do tracer libraries normally send spans?

    The Agent receives spans (port 8126 by default), batches them and forwards them to Datadog.

Summary

  • A trace is a tree of spans that share a trace_id; each span has a service, operation name, resource, duration and error flag.
  • APM computes hits, errors and latency per service and resource from all traffic.
  • Self time (duration minus children) shows where time is really spent.
  • Tracers send spans to the local Agent, which forwards them to Datadog.
  • Use percentiles, not only averages.

Next: Instrumenting an Application