Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 37 ยท Projects and Practice

Project: SLOs and an Alerting Plan

Design SLIs, SLOs, error budgets and a multi-window burn rate alerting plan for a service, and test it against simulated traffic with a runnable simulator.

Good alerting starts with a promise to users. In this project you write that promise as SLOs for shop-api and build the alerting plan that protects it. You will use a simulator to see how different alert rules react to a slow leak and to a sudden outage. Datadog is not available on this site, so the SLO definitions are shown as specifications and the simulation is a labelled stand-in.

If you need a refresher, read Notifications and SLOs first.

Step 1: choose user-visible SLIs

Pick indicators that users feel. For a shop API:

Journey SLI (good events / valid events) Target
Browse products Requests answered with non-5xx status in under 400 ms 99.5% over 30 days
Checkout Orders created successfully (2xx) 99.9% over 30 days
Search Requests under 800 ms 99% over 30 days

Exclude things that are not the service's fault: client errors (4xx) and health checks are not valid events for availability.

In Datadog, create a metric-based SLO: numerator sum:trace.http.request.hits{service:shop-api,resource_name:"POST /orders",!http.status_class:5xx}.as_count(), denominator the same query without the status filter. Add tags team:storefront and journey:checkout.

Step 2: compute the budget

A 99.9% target over 30 days at 50 requests per second:

JS Stand-in: error budget for the project

Press Run (or Ctrl + Enter) to see the console output here.

No output.

About 43 minutes of full outage per month is the whole budget. This is a useful number to put in front of a product manager when they ask for "five nines".

Step 3: alert on burn rate

Use two alert windows per severity, a long one to prove the problem is real and a short one to prove it is still happening. A widely used pattern for a 30-day SLO:

Alert Long window Short window Burn rate Action
Fast burn 1 hour 5 minutes 14.4 Page (2% of the budget gone in an hour)
Medium burn 6 hours 30 minutes 6 Page or urgent ticket
Slow burn 3 days 6 hours 1 Ticket

The numbers come from budget maths: a burn rate of 14.4 for 1 hour uses 14.4 / 720 = 2% of a 30-day budget.

Step 4: simulate the rules

The simulator below feeds per-minute error ratios into the alert rules and reports when each rule would have fired. Scenario A is a sudden outage, scenario B a slow leak.

JS Stand-in: multi-window burn rate simulator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The fast rule catches the outage within a few minutes. It never fires on the leak, because 0.25% errors is a burn rate of 2.5, which is below 14.4. The medium rule (threshold 6) stays silent too. Only the slow rule (threshold 2, with 12-hour and 1-hour windows; in production you would use longer windows such as 3 days and 6 hours) notices it. That is why you keep a slow-burn rule: leaks are real incidents too, they just move slowly.

Step 5: write the alerting plan

Produce a one-page document per SLO:

  1. SLO definition, window and target.
  2. Alert rules table (windows, burn rates, destinations).
  3. Message template with the SLO name, remaining budget, link to the dashboard and runbook.
  4. Owner and escalation policy.
  5. Review cadence: look at the SLO monthly, adjust targets after reality checks.
  6. Error budget policy: what the team does when the budget is spent (for example, freeze risky releases and prioritise reliability work until the SLO recovers).
flowchart TB
  A["Budget healthy"] --> B["Ship features normally"]
  C["Budget low"] --> D["Slow down releases, add tests"]
  E["Budget spent"] --> F["Freeze risky changes, fix reliability"]

The error budget policy turns the number into a decision.

Extension ideas

  • Add a latency SLO with a distribution metric and a percentile threshold.
  • Show the SLO and its burn rate on the dashboard from Project: Observability for a Web Service.
  • Replay a real past incident through the simulator to see what the rules would have done.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use both a long and a short window in a burn rate alert?

    Both windows must exceed the burn rate, so brief spikes and already-recovered problems do not page.
  2. 2. Which events should be excluded from an availability SLI?

    Only valid events that reflect service reliability belong in the denominator.
  3. 3. A 99.9% target over 30 days allows roughly how much full outage?

    30 days is 43,200 minutes; 0.1% of that is 43.2 minutes.
  4. 4. What is an error budget policy?

    It links the number to decisions such as freezing risky releases.

Summary

  • Choose SLIs users feel; define valid events carefully.
  • The error budget is the allowed failure; a 99.9% monthly target is about 43 minutes of outage.
  • Combine long and short windows to build fast, medium and slow burn alerts.
  • Simulate rules on outage and leak patterns, then document owners, messages and an error budget policy.

Next: Project: Incident Drill