Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 27 ยท Projects

Project: Instrument a Web Service and Build Its Dashboard

Add request rate, error and duration metrics to a web service, scrape it with Prometheus, write the PromQL and Grafana panels, and track an availability SLO.

In this project you take one small web service from "no visibility" to a working RED dashboard: it counts requests by route and status, measures latency in a histogram, is scraped by Prometheus, and ends up on a Grafana dashboard with an availability objective and its error budget. It uses the ideas from RED, USE and the Golden Signals, Instrumenting Your Code, PromQL rates and histogram queries.

Prometheus and Grafana cannot run on this site, so the application code, configuration, queries and dashboard JSON are read-only. The runnable parts are Node.js stand-ins on the site's server runner: one is a miniature metrics registry that prints a real-looking /metrics page, and one takes two scrapes of simulated traffic and evaluates the project's queries with simplified arithmetic. They show how the numbers are produced; they are not Prometheus.

The plan

flowchart LR
  A["Web service<br/>counts and times requests"] --> B["/metrics<br/>text page"]
  B --> C["Prometheus<br/>scrape every 15 s"]
  C --> D["Recording rules<br/>and alerts"]
  C --> E["Grafana<br/>RED dashboard"]
  D --> E

The service only counts. Prometheus does the scraping, rates and storage; Grafana only asks questions.

The finished project has four parts: instrumentation, a scrape job, a set of queries, and a dashboard with an SLO panel. Choose route names and buckets carefully now, because changing metric names later breaks every dashboard and alert built on them.

Step 1: instrument the service

The service is a Node.js Express app with three routes: /products, /search and /checkout. The official-style prom-client library provides the registry and the metric types. This listing is read-only: install the package with npm install prom-client express in your own project to run it.

JavaScript
import express from "express";
import client from "prom-client";

client.collectDefaultMetrics();   // process CPU, memory, event loop lag, GC

const requests = new client.Counter({
  name: "http_requests_total",
  help: "HTTP requests handled.",
  labelNames: ["method", "route", "code"],
});

const duration = new client.Histogram({
  name: "http_request_duration_seconds",
  help: "HTTP request latency in seconds.",
  labelNames: ["route"],
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5],
});

const app = express();

app.use((req, res, next) => {
  const stop = duration.startTimer();
  res.on("finish", () => {
    // use the route TEMPLATE, never the raw URL, so labels stay bounded
    const route = req.route ? req.route.path : "unmatched";
    requests.inc({ method: req.method, route, code: String(res.statusCode) });
    stop({ route });
  });
  next();
});

app.get("/products", (req, res) => res.json([]));
app.get("/search", (req, res) => res.json([]));
app.post("/checkout", (req, res) => res.status(201).end());

app.get("/metrics", async (req, res) => {
  res.set("Content-Type", client.register.contentType);
  res.end(await client.register.metrics());
});

app.listen(3000);

Three choices matter here. The route label uses the pattern (/products/:id), not the URL, so the series count is bounded (see Security, Cardinality and Common Pitfalls). The histogram buckets surround the latency you care about, such as a 300 ms target, so the quantile estimate is accurate there. And requests that match no route share one unmatched label, so a scanner probing random URLs cannot create endless series.

Stand-in: a registry and its /metrics page

This program implements the part of the library that matters: a counter keyed by label set, a histogram with cumulative buckets, and the text rendering Prometheus reads. It records five requests and prints the page.

Node.js A tiny registry and its /metrics output

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the bucket lines closely. Each le bucket counts requests at or below that bound, so the values never decrease from one bucket to the next, and the +Inf bucket equals the _count. The /products histogram says one request took 0.05 s or less, two took 0.1 s or less and all three took 0.5 s or less.

Step 2: scrape it

Add a job for the service. The 15 second interval is a common default; every range window in the queries below is at least four times as long.

YAML
# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - rules/shop-api.yml

scrape_configs:
  - job_name: shop-api
    static_configs:
      - targets: ["localhost:3000"]
        labels:
          env: dev

Validate and start it, then open the Targets page and check that shop-api is UP. A quick query to try first is up{job="shop-api"}, which should return 1.

Terminal
promtool check config prometheus.yml
prometheus --config.file=prometheus.yml

Step 3: write the queries

Each RED signal becomes one query. Keep the window at 5m for the dashboard.

PROMQL
# Rate: requests per second, per route
sum by (route) (rate(http_requests_total[5m]))

# Errors: share of requests that failed (5xx)
sum(rate(http_requests_total{code=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

# Duration: 95th percentile latency in seconds
histogram_quantile(
  0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

Two details are easy to miss. The sum by (le) keeps the le label that histogram_quantile needs and merges everything else, and sum must come before the quantile, never after. And the error query is a ratio of two sums, which is correct; averaging per-route ratios would let a quiet route outweigh a busy one.

Dashboards and alerts will reuse these expressions many times, so precompute them with recording rules, named in the level:metric:operation style from Recording Rules and Query Performance.

YAML
# rules/shop-api.yml
groups:
  - name: shop-api-red
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job_route:http_requests:rate5m
        expr: sum by (job, route) (rate(http_requests_total[5m]))
      - record: job:http_request_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_request_duration_seconds:p95_rate5m
        expr: |
          histogram_quantile(0.95,
            sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

Stand-in: evaluate the queries on two scrapes

This stand-in runs the service model, takes a scrape at second 0 and another at second 60, parses the text (the way Prometheus reads it), and computes the three RED values plus the SLO figures with the same arithmetic. It uses the plain difference of two samples for rate, ignoring counter resets and extrapolation, and linear interpolation inside the bucket for the quantile, which is the method histogram_quantile uses.

Node.js Two scrapes in, RED and SLO out

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

In this run the one-minute window narrowly misses the 99.5 percent objective and spends more than the whole error budget, which is exactly the kind of reading the SLO panel is designed to show. The numbers would change with different traffic; what you should take from the program is the shape of the calculation, not these particular values. (A real SLO is judged over weeks, not one minute.)

Step 4: build the dashboard

Create a Prometheus data source in Grafana, then add a dashboard with these panels, laid out left to right in the order a reader scans: is it OK, how much traffic, how often does it fail, how slow is it.

Panel Type Query Unit
Availability (30 d) Stat the SLO query below Percent (0.0-1.0)
Requests per second Time series sum by (route) (rate(http_requests_total[$__rate_interval])) requests/sec
Error ratio Time series job:http_request_errors:ratio_rate5m Percent (0.0-1.0)
p95 latency Time series job:http_request_duration_seconds:p95_rate5m seconds
Slowest routes Bar gauge topk(3, histogram_quantile(0.95, sum by (route, le) (rate(http_request_duration_seconds_bucket[$__rate_interval])))) seconds

$__rate_interval is a Grafana variable that picks a safe range window from the panel's resolution and the data source's scrape interval, so one dashboard works at any zoom level. A single panel in the dashboard's JSON looks like this; you can build the whole dashboard in the UI and export it, or store JSON like this in version control (see Designing Dashboards That Help).

JSON
{
  "type": "timeseries",
  "title": "Requests per second",
  "datasource": { "type": "prometheus", "uid": "prometheus" },
  "targets": [
    {
      "refId": "A",
      "expr": "sum by (route) (rate(http_requests_total[$__rate_interval]))",
      "legendFormat": "{{route}}"
    }
  ],
  "fieldConfig": { "defaults": { "unit": "reqps" } }
}

Set a threshold on the error ratio panel at your objective's limit, for example 0.5 percent, so the line turns red when the objective is at risk.

The SLO panel

An availability objective of 99.5 percent over 30 days means that at most 0.5 percent of requests may fail. The ratio and the share of the error budget used come straight from counters:

PROMQL
# availability over the last 30 days
1 - (
  sum(increase(http_requests_total{code=~"5.."}[30d]))
  /
  sum(increase(http_requests_total[30d]))
)

# share of the error budget consumed (1.0 = budget used up)
(
  sum(increase(http_requests_total{code=~"5.."}[30d]))
  /
  sum(increase(http_requests_total[30d]))
)
/ (1 - 0.995)

A 30 day window over raw data is a heavy query, so in practice record a daily or hourly ratio and aggregate that. The next project builds alerts on this same budget.

Verify the project

  • up{job="shop-api"} is 1 and the target page shows no scrape errors.
  • Send some traffic (for example with a shell loop of curl calls) and watch the requests panel move.
  • Produce a few errors on purpose, for example by making /checkout return 500, and confirm the error ratio panel reacts within a couple of scrapes.
  • Look at the series count for http_requests_total: it must stay small however many different URLs you request.

Extend the project

  • Add a Gauge for in-flight requests and graph saturation next to the RED panels.
  • Add a route variable to the dashboard so one panel set serves every route.
  • Put the service behind Docker Compose with Prometheus and Grafana (see Project: A Complete Monitoring Stack with Docker Compose).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the middleware use req.route.path instead of the raw request URL as the route label?

    A route template such as /products/:id has a small, fixed set of values. Raw URLs would create new series for every ID.
  2. 2. In histogram_quantile(0.95, sum by (le) (rate(..._bucket[5m]))), why is sum by (le) needed?

    histogram_quantile needs the le label to interpolate between buckets. Summing by le combines everything else, giving one overall distribution.
  3. 3. The service's histogram shows le="0.5" with value 3 and le="1" with value 3. What does that mean?

    Buckets are cumulative: each counts requests at or below its bound. Equal consecutive values mean no request fell between the two bounds.
  4. 4. With a 99.5 percent availability objective, what fraction of requests may fail in the window before the budget is spent?

    The error budget is 1 minus the objective: 1 - 0.995 = 0.005, or 0.5 percent of requests.

Summary

  • RED instrumentation needs only two metrics: a request counter labelled by method, route and status code, and a latency histogram labelled by route.
  • Use route templates for labels and choose histogram buckets around the latency you care about.
  • Rate, error ratio and p95 are three short queries; sum before you take a quantile, and compute error ratios from summed rates.
  • Recording rules make the dashboard and alerts cheap and consistent.
  • An availability SLO is one minus the failure ratio over the window; the error budget is what the objective leaves over.
  • Build the dashboard in reading order: is it OK, traffic, errors, latency.

Next: Project: Alerts with Burn Rates and Routing.