Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 37 ยท Dashboards and Monitors

Notifications and SLOs

Write monitor messages with handles and conditional variables, route alerts to Slack and PagerDuty, and define SLOs with error budgets and burn rate alerts.

An alert that nobody understands is noise. This lesson covers the notification message of a monitor (who is told, what they read) and Service Level Objectives (SLOs), which turn "is it healthy?" into a number you can promise and measure. The product is not available on the site, so message syntax is shown read-only and the maths is demonstrated with runnable stand-ins.

Monitor messages

A monitor has a title and a message in Markdown-like text. The message supports:

  • Handles that start with @: @slack-oncall-payments, @pagerduty-checkout, @[email protected], @webhook-name.
  • Conditional blocks: {{#is_alert}} ... {{/is_alert}}, {{#is_warning}}, {{#is_recovery}}, {{#is_no_data}}.
  • Template variables: {{value}}, {{threshold}}, {{host.name}}, {{service.name}}, and tag variables such as {{host.ip}} for multi alerts.
TEXT
{{#is_alert}}
**Checkout error rate is {{value}}% (limit {{threshold}}%)** on {{service.name}}.
Runbook: https://wiki.example.com/runbooks/checkout-errors
Dashboard: https://app.datadoghq.com/dashboard/abc-123
@slack-payments-oncall @pagerduty-checkout
{{/is_alert}}

{{#is_recovery}}
Recovered. @slack-payments-oncall
{{/is_recovery}}

Good messages answer four questions: what is wrong, how bad, where to look (dashboard, runbook, query) and who owns it. Put the handles inside the conditional blocks so a warning does not page someone at night.

Set a team: tag on monitors and use it in notification rules or in search, so you can list all monitors of one team in seconds.

Stand-in: render a message

This code mimics how conditional blocks and variables are resolved for a given state.

JS Stand-in: monitor message renderer

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Routing and fatigue

Send urgent, human-actionable alerts (Alert state) to a pager, and informational ones (Warn) to a chat channel. Use notification rules to apply a standard routing by tag instead of hand-typing handles in every monitor. Avoid alerts nobody acts on; they teach people to ignore the channel (Downtimes and Alert Hygiene).

SLOs in one paragraph

An SLI (service level indicator) is a measurement, such as the share of requests that succeed in under 500 ms. An SLO is a target for that measurement over a window: "99.9% over 30 days". The error budget is what is left to fail: 0.1% of requests. Datadog's SLO page tracks the SLI, the remaining budget and the status.

Two ways to define SLIs in Datadog:

  • Metric-based: good events divided by total events, such as sum:trace.http.request.hits{service:checkout-api,!http.status_class:5xx} over all hits.
  • Monitor-based (time slices): the share of time a monitor was OK.

Error budget maths

If the SLO is 99.9% over 30 days and you serve 10 million requests, you may fail 10,000 of them. If you have already failed 6,000, then 60% of the budget is spent.

JS Stand-in: SLO and error budget calculator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Burn rate alerts

Waiting until the budget is gone is too late. The burn rate is how fast you spend the budget compared to an even spend over the window. Burn rate 1 means you will use exactly the whole budget by the end of the window; burn rate 10 means ten times too fast (the budget is gone in 3 days of a 30-day window).

Common practice is two alerts: a fast burn (for example rate above 14 over 1 hour and 5 minutes, page someone) and a slow burn (rate above 2 over 1 day, open a ticket). Datadog has an SLO Alert monitor type with burn rate and error budget variants.

JS Stand-in: burn rate

Press Run (or Ctrl + Enter) to see the console output here.

No output.

An hourly burn rate of 12 would drain a month of budget in 2.5 days, so it is worth paging.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where do you put @pagerduty handles so that only critical alerts page someone?

    Conditional blocks make a handle fire only for the state it is placed in.
  2. 2. An SLO of 99.9% over 30 days leaves what error budget?

    The budget is the complement of the target: 100 - 99.9 = 0.1%.
  3. 3. A burn rate of 10 means:

    Burn rate compares the current error rate with the allowed error rate.
  4. 4. Which pair of alerts is a common practice for SLOs?

    Different burn speeds deserve different urgency.

Summary

  • Messages use @handles, state blocks such as {{#is_alert}} and variables like {{value}}.
  • A good message says what is wrong, how bad, where to look and who owns it.
  • An SLO is a target for an SLI; the error budget is the allowed failure.
  • Burn rate alerts catch budget loss early; use fast and slow windows.

Next: Downtimes and Alert Hygiene