Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 37 ยท APM and Tracing

Sampling, Retention and Profiling

Understand ingestion sampling, retention filters and trace metrics in Datadog APM, plus how the continuous profiler finds slow code in production.

Storing every span of every request is expensive and mostly useless: ninety-nine identical healthy requests teach you little. Datadog therefore separates three ideas: what your tracer sends, what Datadog ingests, and what it retains (indexes) for search. This lesson explains that pipeline, then adds the continuous profiler, which answers "which line of code is slow?".

Datadog is not available on this site; the calculators below are labelled stand-ins that show the same maths.

The pipeline: send, ingest, retain

flowchart LR
  T["Tracer<br/>sample decision"] --> A["Agent<br/>trace metrics from all data"]
  A --> I["Ingestion<br/>(sampled traces)"]
  I --> R["Retention filters<br/>indexed spans"]
  I --> L["Live search<br/>(15 minutes)"]

Metrics are computed before sampling, so they stay exact.

  1. Sampling at the source. The tracer decides when a trace starts (head-based sampling) and passes the decision to downstream services in the sampling-priority header. That is how every service in one trace keeps or drops it together.
  2. Ingestion controls. The Agent and Datadog apply rules to keep a useful mix: by default, a share of normal traces plus errors and rare resources. You can tune this in APM > Setup & Configuration > Ingestion Control, or with tracer settings like DD_TRACE_SAMPLE_RATE and sampling rules by service and resource.
  3. Retention filters. Ingested spans are searchable live for a short window. For longer history, retention filters decide which spans are indexed, for example "all spans with errors" or "1% of everything in checkout-api". Indexed spans cost money per million.

Defaults, windows and prices differ by plan and change over time. Treat the numbers in this lesson as examples and read your organisation's APM usage page for real values.

Why metrics stay accurate

The Agent computes request counts, errors and latency from all the traffic it sees before sampling drops anything. So trace.http.request.hits is exact while stored traces are a sample. This is why you build monitors on trace metrics, not by counting traces in the explorer.

Stand-in: how a sample rate affects what you see

JS Stand-in: sampling and rare errors

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With a plain random sample, a rare error can disappear. That is why Datadog keeps errors and rare resources on top of the base rate, and why you should create a retention filter for status:error.

Stand-in: estimate the cost of retention

JS Stand-in: retention budget calculator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The price in the code is made up. The takeaway is the structure: cost scales with ingested spans for ingestion and with indexed spans for retention, so filters that target errors, slow traces and key services save money without blinding you.

Good retention filters

  • Keep all errors (status:error) for a week or more.
  • Keep slow traces (@duration:>2s) for important services.
  • Keep a small random share for baseline comparisons.
  • Keep traces for critical business flows (resource_name:"POST /orders").
  • Exclude health checks (resource_name:"GET /health") from ingestion entirely.

The continuous profiler

A trace tells you which span is slow. The continuous profiler tells you which code inside it is slow. It samples your running process (for example, every few milliseconds it records the call stack) and aggregates the samples into a profile. Overhead is small, which is why it can stay on in production.

Turn it on with an environment variable:

Terminal
DD_PROFILING_ENABLED=true ddtrace-run python app.py

In APM > Profiles, you see:

  • a flame graph of CPU time by function,
  • other profile types depending on the language: wall time, memory allocation, lock contention, I/O,
  • a comparison between two versions, which highlights a regression after a deploy.

Because the profiler uses the same service, env and version tags, you can jump from a slow span to the profile of that exact request window ("code hotspots").

Stand-in: a sampling profiler in miniature

JS Stand-in: aggregate stack samples

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The function at the top of that list (escapeHtml in this invented sample) is the hotspot. Real profilers do the same aggregation, then draw the result as a flame graph.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why are trace metrics such as hit counts still accurate when traces are sampled?

    The Agent calculates trace metrics before any sampling decision drops spans.
  2. 2. What does a retention filter control?

    Retention filters decide which ingested spans become indexed and searchable.
  3. 3. What is head-based sampling?

    The first tracer decides and passes the decision along so a trace is kept or dropped as a whole.
  4. 4. What does the continuous profiler add that traces do not?

    Traces show slow spans; profiles show which functions inside the process cost the time.

Summary

  • Spans move from tracer to ingestion to retention; each stage can drop data and cost money.
  • Trace metrics come from all traffic, so build monitors on them.
  • Retention filters should keep errors, slow traces and key business flows.
  • The continuous profiler is cheap enough for production and finds hot functions and regressions between versions.

Next: Dashboards and Widgets