Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: Build a Service Health Dashboard with the RED Method

Design and build a service health dashboard in Grafana using the RED method: requests, errors and duration, with variables, thresholds, annotations, drilldown links and JSON export.

In this project you plan, build and package a service health dashboard that an on-call engineer can read in ten seconds. You use the RED method (Rate, Errors, Duration), the dashboard variables, thresholds, annotations and links from the earlier lessons, and you finish by exporting the JSON so it can be provisioned.

Grafana and Prometheus do not run on this site. The Grafana steps are written as precise instructions you follow on your own installation, and the query text is shown as code. The TryIt blocks are Python stand-ins: one generates sample request data and computes the three RED numbers, the other assembles the dashboard JSON. The data is synthetic and the numbers are produced by our own code, not by Grafana.

What we are building

A dashboard named Service health for a fictional web shop whose services (cart, checkout, search) export the usual Prometheus metrics:

  • http_requests_total{service, status, method}: a counter.
  • http_request_duration_seconds_bucket{service, le}: a histogram.

The finished dashboard has this layout on the 24-column grid:

Row Panels
Top (y: 0, h: 4) Three Stat panels, each w: 8: request rate, error ratio, p95 latency
Middle (y: 4, h: 8) Time series for request rate by status class (w: 12) and latency percentiles (w: 12)
Bottom (y: 12, h: 8) Table of services with rate, error ratio and p95, with a data link to a detail dashboard (w: 24)
flowchart TD
    V["Variables<br/>$env, $service"] --> S["Stat row<br/>rate | errors | p95"]
    V --> T["Time series row<br/>rate by status | latency"]
    V --> B["Table<br/>per service + data link"]

Variables feed every panel; the table drills down to a detail dashboard.

The RED method

RED answers the three questions users care about, for every service:

  • Rate: how many requests per second?
  • Errors: what share of those requests fail?
  • Duration: how long do they take (use a high percentile, not the average)?
Python Stand-in: compute RED from a synthetic request log

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Run it twice: because the seed is fixed, the numbers are identical, which is handy when you want to compare a change to the dashboard against stable data.

Step 1: variables

Create two variables (Settings > Variables):

  1. env: type Custom, values prod,staging, default prod.
  2. service: type Query on the Prometheus data source, query label_values(http_requests_total{env="$env"}, service), Multi-value and Include All on, sorted alphabetically. Set Custom all value to .*.

Use label matchers with =~ in every query: {env="$env", service=~"$service"}.

Step 2: the stat row

Add three Stat panels (w: 8, h: 4), Instant queries, Reduce > Last (not null):

  • Rate: sum(rate(http_requests_total{env="$env", service=~"$service"}[$__rate_interval])), unit requests/sec (reqps).
  • Error ratio: sum(rate(http_requests_total{env="$env", service=~"$service", status=~"5.."}[$__rate_interval])) / sum(rate(http_requests_total{env="$env", service=~"$service"}[$__rate_interval])), unit Percent (0.0-1.0), thresholds green below 0.01, orange from 0.01, red from 0.05.
  • p95: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{env="$env", service=~"$service"}[$__rate_interval]))), unit seconds (s), thresholds green below 0.3, orange from 0.3, red from 0.8.

Set Color mode to Background so a glance tells you the state. Add a value mapping for "No data" to read n/a, not a blank box.

  • Time series Rate by status class: sum by (status) (rate(http_requests_total{...}[$__rate_interval])), legend {{status}}, stacking set to Normal and an override that colors 5xx series red.
  • Time series Latency percentiles: three queries (0.5, 0.9, 0.99) with histogram_quantile, legend p50, p90, p99, unit s, and a threshold line (Show thresholds: As lines) at your SLO value.
  • In Dashboard settings > Annotations, add a query for deploys (for example from the table your pipeline writes) with tag deploy, so every release is visible on both graphs.

Step 4: the table with drilldown

A Table panel with three queries (rate, error ratio, p95), each Format: Table, Instant, sum by (service). Then transformations (see transformations):

  1. Join by field on service (outer).
  2. Organize fields: rename Value #A to Rate, Value #B to Error ratio, Value #C to p95, hide Time.
  3. Sort by Error ratio, reverse.

Set field overrides: unit and thresholds per column, Cell type: Colored background for error ratio. Finally add a data link on the service column: /d/svc-detail/service-detail?var-service=${__value.raw}&var-env=${env}&${__url_time_range}.

Step 5: assemble and export

Set the dashboard time range to the last 3 hours and refresh to 30s (respecting the admin's minimum), time zone UTC, tags red and production. Save in the folder Team. Then Export > Export as JSON and keep the file in Git. The stand-in below builds the same skeleton and checks that the layout fits the grid.

Python Stand-in: assemble the dashboard skeleton

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Review checklist

Before you call the dashboard done, check:

  • The most important numbers are top left and read at a glance.
  • Every panel has a title, a unit, and sensible decimals.
  • Thresholds match your SLO, not a guess.
  • The dashboard works with All services and with a single one.
  • Deploy annotations show up.
  • Data links keep the time range and variables.
  • The JSON is in Git with a fixed uid, and provisioned with allowUiUpdates: false (see provisioning).

Extensions: add a repeated row per service (see repeating panels and rows), a logs panel using a Loki query with the same $service, or a saturation row (CPU, memory) to extend RED into the "four golden signals".

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What do the three letters of the RED method stand for?

    RED covers request rate, error share and request duration for each service.
  2. 2. Why use a percentile such as p95 for duration instead of the average?

    A few very slow requests barely move the mean but dominate user experience, so tail percentiles are more honest.
  3. 3. Which query type suits the Stat panels in this project?

    A Stat panel shows a single current value, so an instant query is enough and cheaper.
  4. 4. What does the custom all value .* do for the service variable?

    When All is chosen, the regex wildcard replaces a long list of values.

Summary

  • RED gives a consistent, small set of signals per service: rate, error ratio and a tail latency percentile.
  • Variables (env, service with Include All) feed every query through =~ matchers.
  • Stat panels with thresholds give the glance view; time series show trends with deploy annotations; a table with data links provides drilldown.
  • Export the JSON, keep it in Git and provision it with fixed uids.
  • A review checklist keeps dashboards honest and usable at 3 a.m.

Next: Project: An Alerting Pipeline for Your Service