Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 37 ยท APM and Tracing

Traces, Spans and the Service Map

Search traces in Datadog, read a flame graph, follow distributed tracing across services with context headers, and use the service map to find dependencies.

Once traces arrive, the work is reading them. This lesson covers the Traces explorer, the flame graph, how a trace continues from one service to the next (distributed tracing), and the Service Map. The Datadog UI is not available here, so you get precise descriptions and runnable stand-ins for the underlying logic.

Searching traces

In APM > Traces, you search with tag filters. Useful queries:

TEXT
service:checkout-api env:prod status:error
resource_name:"POST /orders" @duration:>2s
@http.status_code:500 version:2.14.0
  • status:error finds spans marked as errors.
  • @duration:>2s filters by span duration (attributes you add yourself use @).
  • Click a facet on the left (service, resource, version) to narrow quickly.

You can switch between Spans and Traces views. A trace is listed once; the span view lists every span that matches, which is useful when you look for a specific database query.

The flame graph

Opening a trace shows a flame graph. The x-axis is time. Each bar is a span, nested under its parent and coloured by service. Look for:

  • A wide bar with nothing under it: that span itself is slow (CPU or I/O inside the service).
  • Many thin identical bars: the N+1 query pattern (one query per row).
  • A gap between a parent and its children: time spent in code that is not instrumented.
  • Sequential bars that could run in parallel.

The tabs next to the graph show the span list, the tags of the selected span, the error message and stack trace, and a link to the related logs and host metrics.

Distributed tracing

When service A calls service B over HTTP, the tracer in A adds headers so the tracer in B joins the same trace instead of starting a new one. Datadog's own headers are x-datadog-trace-id, x-datadog-parent-id and x-datadog-sampling-priority. The tracer can also speak the W3C traceparent format, which is helpful when you mix vendors. This works automatically for supported HTTP clients and servers. For message queues the IDs travel in message headers or attributes; for custom protocols you must inject and extract them yourself.

sequenceDiagram
  participant A as web-frontend
  participant B as checkout-api
  A->>B: GET /orders (x-datadog-trace-id: 77, x-datadog-parent-id: 5)
  Note over B: new span, parent=5, trace=77
  B-->>A: 200 OK

Service B reuses the trace ID and uses A's span as its parent.

Stand-in: propagate a trace context

This stand-in injects and extracts a context with plain objects, the way a tracer does for HTTP headers.

JS Stand-in: inject and extract trace headers

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A request without the headers starts a brand-new trace, which is why a missing proxy or a dropped header shows up as "broken" traces with no frontend.

The Service Map

APM > Service Map draws every service as a node and every observed call as an arrow, with colours showing monitor status. Use it to:

  • find who calls a failing dependency (blast radius),
  • spot a service nobody owned (an orphan),
  • find unexpected dependencies, such as a batch job hitting the production database.

Databases, caches and queues appear as their own nodes, taken from client spans such as postgres.query. Calls to unmonitored third parties show up as inferred services, named after the host.

Stand-in: find the critical path

The critical path is the chain of spans that determines the total time. This stand-in walks a trace and prints the longest-lasting chain.

JS Stand-in: critical path of a trace

Press Run (or Ctrl + Enter) to see the console output here.

No output.

If you want to make the checkout faster, optimising the database query on this trace would barely matter. The fraud scoring call is where the time goes.

Linking to other signals

From a span you can jump to:

  • Logs with the same trace_id (needs log injection),
  • Infrastructure: the host or container metrics for the time of the span,
  • Profiles, if the profiler is on (Sampling, Retention and Profiling),
  • Error Tracking, which groups errors by stack trace so a thousand identical exceptions become one issue.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In a flame graph, many thin identical database spans under one parent usually mean:

    One query per row, repeated in a loop, is the classic N+1 problem.
  2. 2. How does service B join the trace started by service A?

    Trace ID and parent span ID travel in request headers such as x-datadog-trace-id.
  3. 3. What does a request that arrives without trace headers do?

    With no context to extract, the tracer creates a new trace with a new trace ID.
  4. 4. What is an inferred service on the Service Map?

    Datadog infers nodes for databases and third-party hosts from the client spans that call them.

Summary

  • Search traces with tag filters; use @ for custom attributes.
  • Flame graphs reveal slow spans, N+1 patterns and uninstrumented gaps.
  • Distributed tracing passes trace ID and parent ID in headers.
  • The Service Map shows dependencies and helps you judge blast radius.
  • Jump from spans to logs, host metrics and profiles.

Next: Sampling, Retention and Profiling