Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 37 ยท Alerting

Grafana Notification Policies: Routing, Grouping and Timing

Route Grafana alerts with the notification policy tree: root policy, label matchers, nested policies, continue matching, group by, group wait, group interval and repeat interval.

When an alert fires, who should be told, in which channel, and how often? Grafana does not hard-wire alert rules to people. Instead, every alert instance carries labels, and a notification policy tree matches those labels to contact points. Changing who is on call means editing the tree, not hundreds of rules.

Grafana does not run here, so the steps are described in words. The TryIt blocks are Python stand-ins that implement the same matching and timing logic, so you can test a policy tree on paper.

The policy tree

Open Alerting > Notification policies. You will see a tree:

  • The default (root) policy at the top. It has a default contact point and catches every alert that no other policy matches.
  • Nested policies below it, each with matching labels (matchers), a contact point, and optional grouping and timing settings.

An alert instance starts at the root and walks down the tree. At each level the first matching child policy takes it; the search continues inside that policy's own children. If no child matches, the current policy handles the alert with its contact point.

Matchers

A matcher compares a label to a value with an operator:

Operator Meaning
= equals
!= does not equal
=~ matches regular expression
!~ does not match regular expression

For example severity = critical, team =~ (payments|checkout), env != dev. A policy with several matchers needs all of them to match.

Continue matching

Normally the first matching sibling wins and the search stops. Tick Continue matching subsequent sibling nodes to let the alert also be tested against the next siblings, so one alert can notify two contact points (for instance, the team's Slack and the pager).

Python Stand-in: route alerts through a policy tree

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first alert matches the critical policy (page the on-call) and, because continue is on, also the payments policy. The second matches nothing and falls to the root contact point.

Grouping

If ten pods fail at once, ten separate notifications is noise. Group by collects alerts that share the same values for the listed labels into one notification that lists them all. The default is grafana_folder and alertname. Common choices:

  • alertname, cluster: one message per alert type per cluster.
  • ... (a special value): disable grouping and send every alert separately.

Think of a group as "one conversation". Good grouping puts things that a person would handle together in one message, and keeps unrelated problems separate.

Timing options

Three timers shape the stream of notifications for each group:

  • Group wait (default 30s): how long to wait after the first alert of a new group, to collect more alerts before the first notification goes out.
  • Group interval (default 5m): how long to wait before sending an update when new alerts join the group or some resolve.
  • Repeat interval (default 4h): how long to wait before re-sending a notification for alerts that are still firing and unchanged.

Shorter values give faster notice but more noise. A critical policy might use group wait 10s, group interval 2m, repeat 1h; a low-priority one might use 1m, 15m and 24h.

Python Stand-in: when are notifications sent?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first alert at t=0 triggers a notification at 30 s. The one at 40 s joins the group, but an update is only sent after the group interval (t=330 s). The third alert at 700 s triggers another update. An hour later, the repeat interval sends a reminder.

Mute timings and the policy

Each policy can reference mute timings, windows during which that policy sends nothing (nights, weekends, maintenance windows). Alerts still fire and stay visible in the UI; only the notifications are held back. Mute timings are explained in silences and mute timings.

Alertmanager and multiple tenants

Under the hood, Grafana ships with an embedded Alertmanager. The tree you see is the same structure that Prometheus Alertmanager uses (route, matchers, group_by, group_wait, group_interval, repeat_interval, continue), so knowledge transfers directly. You can also point Grafana at external Alertmanagers and use the Alertmanager selector at the top of the Alerting pages to edit one or the other. For a Grafana-managed setup, alert rules can alternatively select a contact point directly ("simplified routing"), skipping the tree for simple cases; the tree is the better choice when many rules share routing logic.

Design tips

  • Route on a small, stable set of labels: severity, team, env, service.
  • Keep the tree shallow; two or three levels is usually enough.
  • Always keep a root contact point that a real person reads, so unmatched alerts are not lost.
  • Test with a labeled test alert and the preview routing feature in the rule form (it shows which policies a label set would hit).
  • Use continue sparingly; document why an alert should notify two places.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens to an alert that matches no nested policy?

    The root policy catches everything that no child policy matches.
  2. 2. What does Group by do?

    Alerts that share the chosen label values are sent together as one grouped notification.
  3. 3. Which timer controls how often an unchanged, still-firing alert is re-sent?

    Repeat interval is the reminder period for alerts that are still firing without changes.
  4. 4. How can one alert notify two contact points?

    With continue matching on, the alert is also tested against the following sibling policies.

Summary

  • A notification policy tree matches alert labels to contact points; the root policy is the catch-all.
  • Matchers use =, !=, =~, !~; all matchers of a policy must match; continue allows several matches.
  • Group by, group wait, group interval and repeat interval control volume and timing.
  • Mute timings stop notifications for a policy during chosen windows.
  • It is the Alertmanager model, so Prometheus Alertmanager knowledge applies.

Next: Contact Points and Templates