Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 27 ยท Projects

Project: Alerts with Burn Rates and Routing

Build SLO alerts that page only when the error budget burns fast: multi-window burn-rate rules, severity routing in Alertmanager and runbook annotations.

Most alerting starts with a threshold such as "error rate above 1 percent for 5 minutes". It works until it does not: a small blip pages someone at 3 a.m., while a slow leak that will exhaust your reliability target next week stays silent. In this project you replace that rule with burn-rate alerts, which ask the question that matters: "at the current pace, how fast are we spending the error budget?"

You will add recording rules for the error ratio over several windows, write fast and slow alerts, route pages and tickets to different receivers in Alertmanager, and attach runbooks so the person who gets the alert knows what to do first. It continues the service from Project: Instrument a Web Service and Build Its Dashboard and uses Alerting Rules and Alertmanager and RED, USE and the Golden Signals.

Prometheus and Alertmanager cannot run on this site, so the rule files and configuration are read-only. The runnable TryIts are Python stand-ins in the browser: one computes the burn-rate table, and one replays a simulated incident through the alert conditions. They use the same arithmetic as the real rules; they are not Prometheus.

The idea in four slides

Budget first

The objective For example 99.9% of requests succeed over 30 days.
The budget The 0.1% that may fail without breaking the promise.

Alerts protect the budget, not a number someone guessed.

1 / 4

The burn rate is the observed error ratio divided by the budget. With a 99.9 percent objective the budget is 0.001. If 0.5 percent of requests are failing, the burn rate is 5: at that pace the whole 30 day budget is gone in 6 days. A burn rate of 1 uses the budget exactly over the window, and anything below 1 is sustainable.

Step 1: the numbers

A good starting set of alerts, popularized by the Google SRE workbook, uses four pairs of windows. The first two page someone, the last two open tickets. Each limit is burn rate x budget, and the fraction of budget spent when the alert fires is burn rate x long window / 30 days.

This Python stand-in prints the table for a 99.9 percent objective, so you can check the arithmetic and try another objective by editing the first line.

Python Burn-rate table for a 99.9% objective

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The table shows the logic of the whole design. A page means "you will lose the budget in two days if nothing changes" (2 percent gone in an hour), and the slowest ticket means "you are spending it exactly as fast as you earn it, which leaves no room for a bad day".

Step 2: record the error ratio over each window

Computing a 3 day ratio on every evaluation is expensive, and writing the same expression into six alerts invites typos. Record each window once, using the ratio from the earlier project (naming follows Recording Rules and Query Performance).

YAML
# rules/shop-api-slo.yml
groups:
  - name: shop-api-error-ratio
    interval: 30s
    rules:
      - record: job:http_request_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{job="shop-api", code=~"5.."}[5m]))
          / sum by (job) (rate(http_requests_total{job="shop-api"}[5m]))
      - record: job:http_request_errors:ratio_rate30m
        expr: |
          sum by (job) (rate(http_requests_total{job="shop-api", code=~"5.."}[30m]))
          / sum by (job) (rate(http_requests_total{job="shop-api"}[30m]))
      - record: job:http_request_errors:ratio_rate1h
        expr: |
          sum by (job) (rate(http_requests_total{job="shop-api", code=~"5.."}[1h]))
          / sum by (job) (rate(http_requests_total{job="shop-api"}[1h]))
      - record: job:http_request_errors:ratio_rate2h
        expr: |
          sum by (job) (rate(http_requests_total{job="shop-api", code=~"5.."}[2h]))
          / sum by (job) (rate(http_requests_total{job="shop-api"}[2h]))
      - record: job:http_request_errors:ratio_rate6h
        expr: |
          sum by (job) (rate(http_requests_total{job="shop-api", code=~"5.."}[6h]))
          / sum by (job) (rate(http_requests_total{job="shop-api"}[6h]))
      - record: job:http_request_errors:ratio_rate1d
        expr: |
          sum by (job) (rate(http_requests_total{job="shop-api", code=~"5.."}[1d]))
          / sum by (job) (rate(http_requests_total{job="shop-api"}[1d]))
      - record: job:http_request_errors:ratio_rate3d
        expr: |
          sum by (job) (rate(http_requests_total{job="shop-api", code=~"5.."}[3d]))
          / sum by (job) (rate(http_requests_total{job="shop-api"}[3d]))

If a window is too heavy to evaluate on your data volume, build the long ones from the short ones (for example, an average of the recorded 1h ratios), at the cost of a little accuracy.

Step 3: the alert rules

Each alert combines a long and a short window with and. The limit is written as burn x budget so the intent stays visible: 14.4 * 0.001.

YAML
# rules/shop-api-slo.yml (continued)
  - name: shop-api-burn-rate
    rules:
      - alert: ErrorBudgetBurnFast
        expr: |
          job:http_request_errors:ratio_rate1h{job="shop-api"} > (14.4 * 0.001)
          and
          job:http_request_errors:ratio_rate5m{job="shop-api"} > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          team: shop
        annotations:
          summary: "shop-api is burning its error budget fast"
          description: "{{ $labels.job }} error ratio is {{ $value | humanizePercentage }} over the last hour. At this pace the 30 day budget lasts about two days."
          runbook_url: "https://runbooks.example.com/shop-api/error-budget-burn"
          dashboard: "https://grafana.example.com/d/shop-api"

      - alert: ErrorBudgetBurnModerate
        expr: |
          job:http_request_errors:ratio_rate6h{job="shop-api"} > (6 * 0.001)
          and
          job:http_request_errors:ratio_rate30m{job="shop-api"} > (6 * 0.001)
        for: 5m
        labels:
          severity: page
          team: shop
        annotations:
          summary: "shop-api is steadily burning its error budget"
          description: "{{ $labels.job }} error ratio is {{ $value | humanizePercentage }} over the last 6 hours."
          runbook_url: "https://runbooks.example.com/shop-api/error-budget-burn"

      - alert: ErrorBudgetBurnSlowDay
        expr: |
          job:http_request_errors:ratio_rate1d{job="shop-api"} > (3 * 0.001)
          and
          job:http_request_errors:ratio_rate2h{job="shop-api"} > (3 * 0.001)
        for: 30m
        labels:
          severity: ticket
          team: shop
        annotations:
          summary: "shop-api error budget is draining over days"
          description: "Error ratio over the last day is {{ $value | humanizePercentage }}."
          runbook_url: "https://runbooks.example.com/shop-api/error-budget-burn"

      - alert: ErrorBudgetBurnSlowThreeDay
        expr: |
          job:http_request_errors:ratio_rate3d{job="shop-api"} > (1 * 0.001)
          and
          job:http_request_errors:ratio_rate6h{job="shop-api"} > (1 * 0.001)
        for: 1h
        labels:
          severity: ticket
          team: shop
        annotations:
          summary: "shop-api is spending its budget as fast as it earns it"
          description: "Error ratio over three days is {{ $value | humanizePercentage }}."
          runbook_url: "https://runbooks.example.com/shop-api/error-budget-burn"

Notice what each part does. The and joins two vectors on matching labels, so both sides have to be present and above the limit. The {{ $value }} in the description comes from the left side, the long window. The for durations are short for pages and longer for tickets; with a multi-window rule you do not need long for values to avoid noise, because the short window already confirms the problem is ongoing.

Stand-in: replay an incident

This stand-in reproduces the logic over a simulated eight hours. The service has a background error ratio of 0.02 percent. In the third hour a real incident raises it to 3 percent for an hour, and later a five minute blip reaches 5 percent. The program evaluates a naive "last 5 minutes above 1.44 percent" rule and the multi-window rules, ignoring for for simplicity, and prints when each is true.

Python Incident and blip: which alert fires?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The naive rule reacts within two minutes to the real incident, but it also pages for the seven minute blip that spends only a small slice of the budget. The multi-window rule stays quiet for the blip. Its price is detection time: a 3 percent error ratio is a burn rate of 30, and it takes about half an hour of that before the one hour average passes 1.44 percent. A total outage crosses the limit within about a minute, because the more severe the problem, the less time the long window needs. If half an hour is too slow for your service, tighten the objective or add a faster pair of windows.

Step 4: route alerts with Alertmanager

Alertmanager receives the firing alerts and decides who hears about them. Pages go to the on-call tool and tickets go to a chat channel or queue. Matching uses the severity label set by the rules.

YAML
# alertmanager.yml
route:
  receiver: team-tickets          # default if nothing below matches
  group_by: [alertname, job]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: oncall-pager
      repeat_interval: 1h
    - matchers:
        - severity="ticket"
      receiver: team-tickets

receivers:
  - name: oncall-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-routing-key
        description: "{{ .CommonAnnotations.summary }}"
  - name: team-tickets
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook-url
        channel: "#shop-alerts"
        send_resolved: true
        title: "{{ .CommonAnnotations.summary }}"
        text: "{{ .CommonAnnotations.description }} Runbook: {{ .CommonAnnotations.runbook_url }}"

inhibit_rules:
  - source_matchers:
      - severity="page"
    target_matchers:
      - severity="ticket"
    equal: [job]

The inhibit rule is a courtesy: while a page is active for a job, tickets for the same job stay quiet, because the person handling the page already knows the budget is draining. Keep the secrets in files that are not committed, as in Security, Cardinality and Common Pitfalls. Check the file before loading it:

Terminal
amtool check-config alertmanager.yml

Step 5: runbooks that help

The runbook_url annotation is the most valuable single line in the alert, because it turns "something is wrong" into "do this first". A useful runbook for the burn-rate alerts is short:

  1. Confirm the impact. Open the dashboard from the alert: which routes and status codes are failing?
  2. Check recent changes. A deployment, a config change or a dependency incident explains most budget burns. Roll back first if a deploy lines up with the start.
  3. Find the failing layer. Compare errors by route, by instance and by dependency; check saturation (CPU, connections).
  4. Mitigate, then diagnose. Scale, fail over or disable the feature; investigate root cause afterwards.
  5. Record the outcome. Note the cause and the budget spent, and fix the alert if it was too noisy or too slow.

Review the alert's history regularly and compare how often it paged with how often someone had to act.

Test and extend

Unit-test the rules with promtool test rules, feeding synthetic series as shown in the earlier lesson, and include a test for the blip case so a future edit cannot make it page again. A natural next step is the same pair of alerts per route or per customer tier, which needs a recorded ratio with the extra label. You can also add a latency objective with the same burn-rate idea, using the share of requests slower than a threshold from the histogram buckets.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. With a 99.9 percent objective, what error ratio corresponds to a burn rate of 6?

    The budget is 0.001. The burn rate multiplies it: 6 x 0.001 = 0.006, which is 0.6 percent of requests failing.
  2. 2. Why does each burn-rate alert combine a long window and a short window with and?

    A long window alone keeps firing long after an incident has ended. A short window alone reacts to blips. Requiring both avoids both problems.
  3. 3. What does the inhibit rule in the Alertmanager configuration do?

    An inhibition silences target alerts (tickets) while a matching source alert (a page) is active, here matched on the job label.
  4. 4. A five minute blip spends a tiny part of the error budget. Which rule pages for it?

    A short window alone is sensitive to blips. The multi-window rule also needs the one hour average to exceed the limit, which a short blip does not cause.
  5. 5. What is the most useful annotation to add to a paging alert?

    A runbook link turns an alert into an action plan, which shortens response time, especially at night.

Summary

  • Burn rate is the error ratio divided by the error budget; a burn rate of 1 spends the budget exactly over the SLO window.
  • Use pairs of windows (long and short) joined with and: 1h/5m at 14.4x and 6h/30m at 6x page, 1d/2h at 3x and 3d/6h at 1x open tickets.
  • Record the error ratio once per window and build alerts from the recorded series.
  • Route by severity in Alertmanager, group by alert and job, and use an inhibit rule so tickets do not pile on top of a page.
  • Put a runbook link and a dashboard link in every alert's annotations.
  • The trade-off is detection time for mild problems, in exchange for far fewer pointless pages.

Next: Project: Monitor a Fleet of Servers.