Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 37 ยท Metrics and Tags

Datadog Metric Types: Count, Rate, Gauge, Histogram, Distribution

Understand Datadog metric types and aggregation: gauge, count, rate, histogram and distribution, time versus space aggregation, rollups, and why averaging percentiles is wrong.

A metric is a named series of numbers over time, such as system.cpu.user or checkout.orders. Behind every graph is a choice that is easy to miss: what type the metric is. The type decides how Datadog combines points, which functions make sense, and whether a graph tells the truth. This lesson gives you the five types and the two kinds of aggregation that every query performs.

The type behavior is modeled with Python stand-ins here, because Datadog itself does not run on this site.

Anatomy of a metric point

Each point has a metric name, a timestamp, a value and a set of tags. The series is the combination of a name and one exact set of tags, so requests.count{env:prod,service:web} and requests.count{env:prod,service:api} are two different series (more in tags).

The metric types

Type What it represents Example Natural combination
Gauge A value at a moment, can go up or down memory used, queue depth, temperature average, last, max
Count Number of events in an interval orders placed, errors sum
Rate Events per second requests per second average of rates
Histogram Distribution of values seen by one Agent in an interval request duration percentiles computed per Agent
Distribution Distribution of values across all hosts, aggregated server-side request duration of a service across 50 pods global percentiles

In the product (Metrics Summary) every metric is stored as gauge, count, rate or distribution. The Agent and DogStatsD have a few submission types (c, g, h, ms, d, s) that map onto them; see DogStatsD.

Gauge

Use a gauge for anything you would read from a dial. Averaging gauges across hosts is meaningful (average CPU), summing is meaningful when the quantity adds up (total memory used).

Count

A count is "how many happened in this interval". The interval matters: a count of 120 over a one-minute bucket is different from 120 in an hour. Graphs of counts are meaningful as sums and often shown with the .as_count() modifier.

Rate

A rate is a count divided by time. Rates are good for comparing intervals of different sizes. If you submit a count but want per-second, use .as_rate() in the query or per_second().

Histogram

A histogram metric lets one Agent submit many values (durations) during the interval, and the Agent reports a set of aggregations as separate metrics: .avg, .count, .median, .max and .95percentile. The percentiles are computed on that Agent only. If you have 50 pods, you get 50 separate percentiles, and combining them is mathematically wrong.

Distribution

A distribution sends the raw values (or a sketch of them) to Datadog, which computes percentiles over all the data globally. You can ask for p50, p95, p99, avg, max, min, sum, count across any tags you choose, even ones that did not exist when you sent the data, because the sketch keeps the shape. You enable percentile aggregations per metric in Metrics Summary > Metric details > Advanced. Use distributions for latency and sizes in any service with more than one instance.

Two aggregations in every query

A graph query combines data in two directions:

  • Time aggregation (rollup). Raw points are grouped into time buckets (for a 4-hour view, maybe 1-minute buckets) and reduced to one value per bucket with a method such as avg, sum, max or last. Datadog chooses the bucket size from the time range; you can force it with .rollup(avg, 60).
  • Space aggregation. Series from different tag combinations are combined with avg, sum, min or max, optionally keeping groups with by {host}.

So avg:system.cpu.user{env:prod} by {host} means: for each host, take the series, roll up in time, and average across any other tags. When the type and the aggregation clash (summing a gauge of percentages across hosts), the graph is wrong even though it renders fine.

The percentile trap

Averaging per-host percentiles is the classic mistake. Suppose host A serves 1000 requests with p95 = 100 ms, and host B serves 10 requests with p95 = 900 ms. The average of the two p95s is 500 ms, but the real p95 of all 1010 requests is close to 100 ms, because B hardly matters. Use distributions (or compute percentiles on the raw data) instead.

Python Stand-in: gauge, count and rate in a 60-second bucket

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: why averaging p95s is wrong

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second example shows the gap: the average of the two percentiles is far from the true percentile of all 1,010 requests.

Units, metadata and naming

Every metric can have a unit (bytes, milliseconds, percent) and a description set in Metrics Summary, so graphs label axes sensibly. Naming conventions worth following:

  • Dots separate namespaces: shop.checkout.latency. Start with a short owner prefix.
  • Lower case, letters, numbers, underscores and dots; start with a letter; keep it under 200 characters.
  • Put the unit and meaning in the name or metadata (.bytes, .seconds), not the dimension values (put region in a tag, not in latency_eu_west).
  • Do not put high-cardinality values (user ID, request ID) in tags.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which type should you use for latency across many pods when you need a global p99?

    Distributions are aggregated server-side across all hosts, so global percentiles are correct.
  2. 2. A graph shows requests as 120 per bucket and the bucket size changes with the time range. Which modifier makes the value comparable across ranges?

    A rate is normalized per second, so changing the bucket size does not change the value.
  3. 3. What is space aggregation?

    Space aggregation combines different series; time aggregation combines points over time.
  4. 4. Why is averaging per-host p95 values wrong?

    The average ignores how many requests each host served; use a distribution for a true global percentile.

Summary

  • Gauge, count, rate, histogram and distribution are the metric types; choose by what the number means.
  • Every query does time aggregation (rollup) and space aggregation (across tags).
  • Use distributions for latency across many instances; never average percentiles.
  • Name metrics with namespaces, put dimensions in tags, set units.

Next: Tags and Unified Service Tagging