Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 27 ยท Grafana

Designing Dashboards That Help

Design Grafana dashboards people can act on: layout, RED and USE patterns, units and thresholds, templating, annotations, dashboards as code and common mistakes.

It is easy to build a dashboard with sixty panels. It is much harder to build one that someone can read at 3 a.m. and still know within ten seconds whether the service is healthy and where to look next. A dashboard is a question and answer interface; every panel should answer a question that somebody actually asks.

This lesson covers layout, the RED and USE dashboard patterns, units and thresholds, templating and annotations, keeping dashboards in Git, the mistakes that make dashboards useless, and a short overview of Grafana's own alerting. The mechanics of panels and variables are in Grafana: Data Sources and Dashboards, and the metrics themselves come from RED, USE and the Golden Signals.

Grafana cannot run on this site, so the JSON and YAML below are read-only snippets. They are meant to show the shape of real configuration; check exact option names against your Grafana version. The two runnable examples are JavaScript stand-ins: a unit and threshold formatter, and a small linter for the dashboard design rules in this lesson.

Read a dashboard top to bottom

Health first Is the service OK right now?
Symptoms Rate, errors, latency.
Causes Resources and dependencies.

Put the answer to "is it broken?" at the top left.

1 / 2

Layout: the top answers, the bottom explains

People scan a dashboard from the top left. Use that:

  1. Top row: overall health. A few large Stat panels (current error ratio, p95 latency, requests per second, up targets) with colors that turn red when something is wrong. This row alone should tell you whether to keep reading.
  2. Next: the symptom graphs. Request rate, error ratio and latency over time, side by side with the same time axis.
  3. Below: causes and dependencies. CPU, memory, queue depth, database latency, grouped in rows (collapsible sections) with clear titles.

Keep one dashboard for one audience and one purpose. An overview dashboard for on-call, a details dashboard per service, and a resource dashboard per node type are better than one giant page. Link them: a panel title or a data link can jump from the overview to the details with the same variable values and time range preserved.

Aim for a screenful of panels, not a scroll of fifty. If panels are rarely looked at, remove them; you can always put them back from Git.

RED for services, USE for resources

Two simple structures cover most dashboards (the signals are explained in RED, USE and the Golden Signals):

  • A RED dashboard is for anything that serves requests: Rate (requests per second), Errors (failing share of requests) and Duration (latency percentiles). One column per signal, one row per service or route. This is the first dashboard to build for every service.
  • A USE dashboard is for resources such as CPUs, disks, memory and network links: Utilization (how busy), Saturation (how much work is waiting), Errors. Saturation is the signal people forget: a disk at 50 percent utilization with a growing queue is already hurting.

A well-designed RED panel set uses the queries from Quantiles, Histograms and SLI Queries: the rate, an error ratio, and p50, p95 and p99 on one graph so the gap between median and tail is visible.

Units, axes and thresholds

A number without a unit is a riddle: is 0.25 a quarter of a second, a ratio or a quarter of a gigabyte? Set the unit on every panel. In JSON, units are short IDs in fieldConfig:

JSON
{
  "type": "timeseries",
  "title": "p95 latency",
  "datasource": { "type": "prometheus", "uid": "prometheus" },
  "targets": [
    {
      "expr": "histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job=\"web\"}[$__rate_interval])))",
      "legendFormat": "p95"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "unit": "s",
      "min": 0,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          { "color": "green", "value": null },
          { "color": "orange", "value": 0.3 },
          { "color": "red", "value": 1 }
        ]
      }
    }
  }
}

Commonly used unit IDs are s (seconds), ms, bytes, Bps (bytes per second), percent (0-100), percentunit (0-1, so a ratio of 0.05 shows as 5%) and reqps (requests per second). With the right unit Grafana scales the display automatically: 1536000 bytes appears as 1.54 MB.

Thresholds color a value (or draw a line on a graph) when it crosses a limit. Base them on your SLO, not on a feeling: if the target is "95 percent under 300 ms", the orange line belongs at 0.3. Also fix the axis. Start counters and rates at 0 ("min": 0) so a small wobble does not look like a catastrophe, and use the same axis for panels you want to compare.

JS Units and thresholds

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The stand-in uses decimal (1000-based) units; Grafana offers both decimal and binary (bytes vs bits and IEC variants), so pick the one your team expects and keep it consistent.

Templating and annotations

Variables let one dashboard serve many services (see Grafana: Data Sources and Dashboards). Design rules: put the variables at the top in a sensible order (environment, then cluster, then service), give them clear names, and default them to a safe value such as prod. Do not use a variable that can return thousands of values; it makes the dropdown and the queries unusable.

Annotations draw vertical markers on graphs for events, and answer the question people ask first during an incident: "what changed?". A deploy marker next to a latency jump is worth ten panels. Annotations can come from queries against Prometheus (for example a metric that changes whenever a new version starts), from other data sources, or from Grafana's HTTP API, which a CI/CD pipeline can call after each release. Show them on every time series panel by enabling the built-in annotation query or adding your own in the dashboard settings.

Dashboards as code

Dashboards edited by hand drift, get overwritten, and cannot be reviewed. Treat them like source code:

  • Keep the JSON model in Git and load it through provisioning (the provider file shown in the previous lesson). Reviewers then see which panel and query changed in a pull request.
  • For many similar dashboards, generate them: Grafana publishes libraries such as Grafonnet (Jsonnet) and SDKs for several languages, and the Terraform provider can manage dashboards and folders. A generator turns "one RED row per service" into a loop instead of a copy-and-paste.
  • Set "uid" explicitly and keep it stable, so links and provisioning keep working across environments.
  • Prevent accidental UI edits on provisioned dashboards (allowUiUpdates: false), or export from the UI as a deliberate step.

A minimal dashboard skeleton shows the fields that matter for version control:

JSON
{
  "uid": "web-red",
  "title": "Web service: RED",
  "tags": ["web", "red"],
  "timezone": "browser",
  "refresh": "1m",
  "time": { "from": "now-6h", "to": "now" },
  "templating": { "list": [] },
  "panels": []
}

Common mistakes

  • Too many panels, no hierarchy, no clear question per panel.
  • No units, or the wrong ones, and thresholds that are arbitrary.
  • Averages only. An average latency hides the tail; show percentiles.
  • Stacked graphs for ratios. Stacking only makes sense for parts of a whole (CPU modes), not for percentages of different things.
  • Inconsistent time ranges and axes across panels you want to compare.
  • Hard-coded values (a service name inside a query) instead of variables.
  • Vanity panels: graphs nobody has ever used to make a decision.
  • Dashboards instead of alerts. If a condition needs a human, an alert should tell them; nobody should have to watch a graph.

The second stand-in turns some of these rules into a tiny linter that reads a simplified dashboard model.

JS A dashboard design linter

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The linter is crude (the regular expressions are simple pattern checks, not a PromQL parser), but real teams do run similar checks on exported dashboard JSON in CI.

A short note on Grafana alerting

Besides the Prometheus and Alertmanager pair from Alerting Rules and Alertmanager, Grafana has its own alerting system. You define alert rules in Grafana (a query, a condition and a pending period, much like expr and for), group them into folders and evaluation groups, and route them with notification policies (a tree similar to Alertmanager's routes) to contact points (email, Slack, PagerDuty, webhooks). It can alert on data from many sources, not only Prometheus, and it has its own silences and mute timings.

Which to choose? Rules stored in Prometheus (and routed by Alertmanager) are evaluated next to the data, keep working when Grafana is down and fit naturally into Git. Grafana-managed rules are convenient when the data comes from several sources or when a team wants to manage alerts in the UI. Either way, avoid running the same alert in both systems, or you will be notified twice.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Where should the panels that answer 'is the service healthy right now?' go?

    Readers scan from the top left. A health row at the top lets someone decide in seconds whether to dig deeper.
  2. 2. What does the RED method measure?

    RED is for request-serving services; USE (Utilization, Saturation, Errors) is its counterpart for resources.
  3. 3. A panel shows an error ratio of 0.05 with the unit percentunit. What does Grafana display?

    percentunit treats the value as a fraction of 1, so 0.05 is shown as 5%. The percent unit expects values already in 0-100.
  4. 4. Why use annotations for deploys on latency graphs?

    A marker for a release next to a latency spike often explains the incident immediately.
  5. 5. What is a good reason to keep dashboard JSON in Git and provision it?

    Version control gives review, history and repeatable setups, avoiding drift from manual edits.

Summary

  • Design for a reader under stress: health at the top left, symptoms next, causes below, one purpose per dashboard.
  • Use RED for services and USE for resources, and show percentiles rather than averages.
  • Set units, a sensible axis and thresholds based on your SLOs on every panel.
  • Use variables to reuse dashboards and annotations to show deploys and other changes.
  • Keep dashboards as code (JSON, generators, provisioning) with stable uid values.
  • Grafana has its own alerting with contact points and notification policies; avoid duplicating the same alert in two systems.

Next: Scaling, Retention and Long-Term Storage