Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 37 ยท Dashboards and Monitors

Dashboards and Widgets

Build useful Datadog dashboards: timeseries, query value, top list and heatmap widgets, the metric query format, template variables, and layout rules that work.

A dashboard is a page of widgets that answers a question quickly: "Is checkout healthy right now?" Datadog has two layouts, Dashboards (a grid or free-form board, saved and shared) and Notebooks (documents that mix text and graphs, good for investigations and post-mortems). This lesson focuses on dashboards.

The Datadog app is not available on this site. I describe the real menus and query syntax precisely, and the runnable code is a labelled stand-in for the query format.

Create a dashboard

In the app: Dashboards > New Dashboard, give it a name, choose a layout (the grid layout is the best default), then add widgets from the palette on the right or press + Add Widgets. Every widget has an edit form with three parts: graph your data (the query), set display preferences (colours, axes, markers) and title.

The metric query format

Most widgets use this form:

TEXT
avg:system.cpu.user{env:prod,service:checkout-api} by {host}

Read it left to right:

Part Meaning
avg Space aggregator: how to combine series from many hosts (avg, sum, min, max)
system.cpu.user The metric name
{env:prod,service:checkout-api} Scope: only series with these tags (comma means AND)
by {host} Group by: one line per host

You can chain functions: .rollup(avg, 60) sets the time bucket, .as_rate() converts counts to rates, derivative(), moving_average(), top() and so on. You can also combine queries with arithmetic, for example errors divided by hits for an error rate.

Core widgets

  • Timeseries: lines, bars or area over time; the workhorse. Use markers for deploys or thresholds.
  • Query Value: one big number (current error rate), with conditional colours.
  • Top List: ranked values, for instance the ten slowest endpoints.
  • Heatmap: distribution over time; ideal for latency.
  • Table: several metrics side by side per tag value.
  • Change and Distribution: compare to a previous period, show a histogram.
  • Group, Note and Image: structure and explanations.
  • Monitor Summary, SLO, Alert Graph: show monitors and SLOs on the board.
  • Log Stream, List widgets: show log or event streams.

Stand-in: a mini query evaluator

This code parses the scope and grouping of a query and groups example data the same way a widget would. It is a teaching stand-in, not Datadog's engine.

JS Stand-in: scope, group by and space aggregation

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Template variables

A template variable lets viewers change the scope of every widget at once. Define $env on the dashboard (tag key env, default prod), then write widget queries as avg:system.cpu.user{$env,$service}. A viewer picks staging from a dropdown at the top, and all widgets follow. Saved views store favourite combinations.

JS Stand-in: substituting template variables

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Design rules that make boards useful

  1. Start with a question. "Is the service healthy?" produces a different board from "why did last night's batch fail?".
  2. Put the answer at the top. Query values for error rate, latency and traffic first; details below.
  3. Follow the golden signals: latency, traffic, errors, saturation (the four signals from site reliability practice).
  4. Group widgets with headers: Overview, Dependencies, Infrastructure, Logs.
  5. Use consistent units and time frames; avoid mixing seconds and milliseconds on one axis.
  6. Add markers for deploys (from events) so a change in the line has an explanation.
  7. Limit widgets to what you actually read. Thirty graphs are rarely more useful than twelve.
flowchart TB
  O["Overview: error rate, p95, requests"] --> S["Service details: top resources, versions"]
  S --> D["Dependencies: database, cache, queue"]
  D --> I["Infrastructure: CPU, memory, pods"]

A top-down layout lets an on-call engineer stop reading as soon as they find the cause.

Sharing and managing

  • Share a read-only public link only if your policy allows it; otherwise share inside the organisation.
  • Dashboard lists and Teams keep boards organised; mark important ones as favourites.
  • Dashboards as code: you can export JSON and manage dashboards with the API or Terraform (API, Terraform and Automation).
  • Use Powerpacks to reuse a group of widgets across boards.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In avg:system.cpu.user{env:prod} by {host}, what does by {host} do?

    Group by splits the result into one series per distinct tag value.
  2. 2. Which widget best shows how latency is distributed over time?

    A heatmap shows the distribution of values for each time bucket.
  3. 3. What is the benefit of a template variable such as $env?

    Widgets that reference $env follow the selected value, so one board serves many environments.
  4. 4. Which are the four golden signals?

    Latency, traffic, errors and saturation are the classic signals to put on a service dashboard.

Summary

  • A metric query is aggregator, metric, scope and group by, with optional functions.
  • Choose widgets for the question: timeseries for trends, query values for the current state, top lists for ranking, heatmaps for distributions.
  • Template variables rescope a whole dashboard.
  • Put the answer at the top, group widgets, add deploy markers and keep boards small.

Next: Monitor Types and Thresholds