Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 37 ยท Metrics and Tags

Datadog Metrics Explorer and Query Language: Scope, Group By, Functions

Write Datadog metric queries: aggregator, metric, scope, by {group}, formulas, rollup, as_count and as_rate, rate and derivative functions, moving averages and top lists.

Every Datadog graph and metric monitor runs a query. The query language is small, and learning it is the fastest way to become productive. This lesson starts at the Metrics Explorer and builds up from the basic form to functions and formulas. Since Datadog does not run here, the stand-ins parse and evaluate the same shapes over small sample data so you can see how each part changes the result.

Metrics Summary and Explorer

  • Metrics > Summary lists every metric name. Click one to see its type, unit, the tag keys it carries and its estimated cardinality.
  • Metrics > Explorer is a quick graph. Choose a metric, a from scope (filter), an aggregation and a group by. Behind the buttons is the query text, which you can edit directly via the </> toggle.

Widgets on dashboards and monitors use the same editor.

The basic form

TEXT
<space aggregation>:<metric name>{<scope>} by {<group>}

Example:

TEXT
avg:system.cpu.user{env:prod,role:web} by {host}
  • avg is the space aggregator: how to combine series that share the same group. Choices: avg, sum, min, max (and count in some contexts).
  • system.cpu.user is the metric.
  • {env:prod,role:web} is the scope: only series that have both tags. A comma means AND. {*} means everything.
  • by {host} gives one line per host. Without it, everything is combined into one line. You can group by several keys: by {service,env}.

Scope variations:

TEXT
{env:prod,!service:batch}       # exclude a tag
{host:web-*}                    # wildcard in the value
{env:prod AND (service:a OR service:b)}   # boolean scope in the newer syntax

Time aggregation: rollup

Datadog picks a bucket size from the time range. Override it:

TEXT
avg:system.cpu.user{*}.rollup(avg, 60)    # one-minute buckets, average in each
sum:shop.orders{*}.rollup(sum, 300)       # five-minute totals

The rollup method is separate from the space aggregator. Two common modifiers on count/rate metrics:

  • .as_count() shows event counts per bucket.
  • .as_rate() shows events per second.

Functions

Functions wrap a query. A few you will use constantly:

Function What it does
per_second(q) Converts a count to a per-second rate
diff(q) Difference between consecutive points (for counters submitted as gauges)
derivative(q) Rate of change per second
abs, log2, log10 Math functions
moving_rollup(q, 300, 'avg') Moving window average (a smoothed line)
top(q, 5, 'mean', 'desc') Show only the five highest series
timeshift(q, -3600) Overlay the same metric from one hour ago
anomalies(q, 'basic', 2) Highlight points outside the expected band
outliers(q, 'dbscan', 3) Flag series that behave differently from the group
forecast(q, 'linear', 1) Extend a trend (monitor feature)

Functions can be chained, for example top(avg:system.cpu.user{env:prod} by {host}, 5, 'mean', 'desc').

Formulas

A formula combines several queries by name, a, b, c: errors / requests * 100 for a percentage. In the widget editor you add queries a and b, then write a / b * 100. The error rate query then looks like:

TEXT
a = sum:trace.http.request.errors{service:checkout}.as_count()
b = sum:trace.http.request.hits{service:checkout}.as_count()
formula: a / b * 100

Formulas are evaluated per group, so by {service} in both queries gives a rate per service. Make sure both queries use the same time aggregation (as_count), or the ratio will be off.

Reading what you graphed

  1. Check the metric type in Summary first (a gauge sums differently from a count).
  2. Confirm the rollup interval; hover the graph to see the bucket start.
  3. Use the tag value counts in the editor to see which tags exist.
  4. If the line is empty, remove tags from the scope one at a time to find which filter excludes everything.
Python Stand-in: parse a metric query into its parts

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: scope filter, group by and aggregator over sample series

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Python Stand-in: error-rate formula with per-group evaluation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Tips for readable, cheap queries

  • Always scope by env and service to keep graphs fast.
  • Group by low-cardinality tags (service, region), not by pod_name or container_id.
  • Prefer top() over graphing 500 lines.
  • Use the same query text in a dashboard widget and its monitor so what you see is what alerts.
  • Save repeatable pieces as template variables on dashboards (see the dashboards chapter).

For the metric semantics behind these queries, return to metric types and aggregation, and for where scope tags come from, tags.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does {env:prod,role:web} mean in a query scope?

    A comma between tags in the scope means AND.
  2. 2. What does by {host} do?

    Grouping keeps a separate series for each value of the tag.
  3. 3. Which modifier converts a count into events per second?

    as_rate() (or per_second()) normalizes by the bucket length.
  4. 4. How do you compute an error percentage across two metrics?

    Formulas combine named queries; use the same aggregation on both.

Summary

  • A metric query is aggregator:metric{scope} by {group}; the scope uses tags (comma = AND, ! excludes).
  • Rollup sets the time bucket; as_count() and as_rate() change how counts are shown.
  • Functions such as top, diff, derivative, timeshift and anomalies wrap queries; formulas combine them.
  • Scope and group by low-cardinality tags to keep queries fast and readable.

Next: Log Collection