Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 27 ยท Recording Rules and Alerting

Alerting Rules and Alertmanager

Write Prometheus alert rules with for durations, labels and annotations, then route, group, inhibit and silence them with Alertmanager receivers.

A dashboard only helps if somebody is looking at it. Alerts are how monitoring reaches people who are asleep, in a meeting or fixing something else. In the Prometheus world the work is split in two: Prometheus decides when something is wrong by evaluating alerting rules, and Alertmanager decides who hears about it, how, and how often.

This lesson covers both halves: the structure of an alerting rule and its pending and firing states, and the Alertmanager routing tree with grouping, inhibition, silences and receivers such as email, Slack and PagerDuty. It also covers the habit that matters more than any syntax: alert on symptoms people feel, and keep causes for dashboards. The rule-file basics come from Recording Rules and Query Performance.

Prometheus and Alertmanager cannot run on this site, so all configuration here is read-only. The two runnable examples are JavaScript stand-ins: a small state machine that models pending and firing, and a router that models label matching and grouping. They teach the logic; they are not the real programs, and the real ones have more options.

Two programs, two jobs

Prometheus Evaluates rules, decides what is firing.
Alertmanager Groups, routes, silences and notifies.

Prometheus never sends emails itself.

1 / 2

An alerting rule

Alert rules live in the same rule files and groups as recording rules. Each one has an expression that returns the series that are in trouble: if the expression returns nothing, nothing is wrong. Here is a rule that fires when more than 5 percent of requests fail:

YAML
groups:
  - name: service-alerts
    rules:
      - alert: HighErrorRatio
        expr: job:http_request_errors:ratio_rate5m > 0.05
        for: 10m
        keep_firing_for: 5m
        labels:
          severity: page
          team: payments
        annotations:
          summary: "{{ $labels.job }}: {{ $value | humanizePercentage }} of requests fail"
          description: "Error ratio above 5% for 10 minutes."
          runbook_url: "https://runbooks.example.com/high-error-ratio"

The fields do different jobs:

  • expr is any PromQL expression. A comparison like > 0.05 filters the vector, keeping only series that exceed the threshold.
  • for is how long the expression must stay true before the alert fires. Until then the alert is pending. It filters out blips: a ten-second spike is not worth waking someone.
  • keep_firing_for (optional, recent versions) keeps the alert firing for a while after the expression stops being true, which prevents flapping when a metric hovers around the threshold.
  • labels are attached to the alert and are what Alertmanager routes on. severity and team are the usual two.
  • annotations are free text for humans, expanded with Go templates: $labels gives the series labels and $value the sample value. Always add a runbook_url, so whoever gets the page knows the first three things to check.

Every series the expression returns becomes its own alert, with its labels copied. That is why aggregating first (one series per job, not one per instance) controls how many notifications you create.

The states

An alert instance is inactive when the expression returns nothing for it, pending from the first evaluation where it appears until for has elapsed, and firing after that. Prometheus sends firing alerts to Alertmanager on every evaluation, so Alertmanager sees them repeated and treats them as a stream. When the expression stops returning the series, Prometheus sends the alert again marked as resolved, and Alertmanager sends a "resolved" notification if configured to.

You can see the current alerts under Alerts in the web UI, and as a series: ALERTS{alertstate="firing"}.

JS A pending and firing state machine

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The first spike (minutes 2 to 4) ends before three minutes pass, so it is silently dropped: this is exactly what for is for. The second run is long enough to fire. In this simple model the check minute - activeSince >= forMinutes is how the timer works. Real Prometheus compares evaluation timestamps against for, and a longer for means fewer false alarms but a slower reaction.

Alert on symptoms

The most common reason teams drown in alerts is alerting on causes: CPU at 85 percent, memory above 70 percent, a pod restarted. These may be perfectly healthy. The rule of thumb is to page a human only for something a user would notice or that will become noticeable soon:

  • error ratio, latency percentile or availability too far from the SLO (see Quantiles, Histograms and SLI Queries);
  • a service is entirely absent (up == 0, or absent(...) for series that vanished);
  • a resource that will run out soon, for example a disk predicted to fill within hours.

Keep the cause-level checks (CPU, queue depth, restarts) as dashboard panels and low-severity tickets. A page should be urgent, actionable and rare. If an alert fires and nobody does anything, delete it or lower its severity. See Security, Cardinality and Common Pitfalls for more on alert fatigue.

Alertmanager: the routing tree

Alertmanager receives alerts from one or more Prometheus servers and does five things: deduplicates them, groups them, routes them to receivers, inhibits noisy ones, and respects silences. Its configuration is a tree. Every alert enters at the root route, and walks down into the first child whose matchers fit.

YAML
global:
  smtp_smarthost: "smtp.example.com:587"
  smtp_from: "[email protected]"

route:
  receiver: team-chat
  group_by: ["alertname", "cluster"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="page"
      receiver: pagerduty
    - matchers:
        - team="db"
      receiver: db-email
      continue: false

inhibit_rules:
  - source_matchers:
      - severity="page"
    target_matchers:
      - severity="warning"
    equal: ["alertname", "cluster"]

receivers:
  - name: team-chat
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-url
        channel: "#alerts"
        send_resolved: true
  - name: pagerduty
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pagerduty-key
  - name: db-email
    email_configs:
      - to: "[email protected]"

Taking it apart:

  • Root route and default receiver. An alert that matches no child goes to team-chat. Always have a catch-all, so nothing is lost silently.
  • Child routes are checked in order; the first match wins, unless it sets continue: true, in which case matching goes on and the alert can reach several receivers. A critical page that must also reach the chat channel uses that.
  • Grouping (group_by) merges alerts that share those labels into one notification. If fifty pods in one cluster throw the same alert, you get one message listing fifty, not fifty messages. group_wait is how long Alertmanager waits to collect more alerts for a new group before the first message; group_interval is the wait before sending updates about a group; repeat_interval controls reminders for alerts that are still firing.
  • Inhibition suppresses a target alert while a source alert is firing and both share the labels in equal. In the config above, a warning is muted while a page with the same alert name and cluster is active, since the page already tells the story. The classic use: do not send "service unreachable" alerts for every service when the whole node is down.
  • Silences are temporary mutes created in the Alertmanager UI or with amtool, matched by labels, with an expiry and an author. Use them for planned maintenance. Unlike inhibition they are set by humans at run time, not by config.
  • Receivers are the destinations. Each one lists integrations such as email_configs, slack_configs and pagerduty_configs. Keep secrets in files (the *_file options) rather than in the config.

You can validate the file before loading it with amtool check-config alertmanager.yml, and test where a label set would go with amtool config routes test --config.file=alertmanager.yml severity=page team=db.

Routing, in code

This stand-in models first-match routing with continue and grouping. It is deliberately smaller than the real matcher language (it only supports exact label equality).

JS A tiny Alertmanager router

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Two alerts with the same alertname and cluster collapse into one notification. The page alerts match the first route, and because no later route matches them they reach only pagerduty. The default receiver is used only when nothing matched at all; it is not an extra recipient. Add a second page route with continue: true earlier in the list to see one alert land in two groups.

Operating alerts well

  • Test alert rules before shipping them with promtool test rules, which replays synthetic series against your rules (see Security, Cardinality and Common Pitfalls).
  • Make Alertmanager highly available by running several instances that gossip with each other, and point every Prometheus at all of them; Alertmanager deduplicates.
  • Watch your own pipeline. Add an always-firing "watchdog" alert routed to an external dead man's switch service, so you notice when alerts stop flowing.
  • Review regularly. Count how often each alert fired and how many led to action.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the purpose of the for field in an alerting rule?

    Until the duration has elapsed the alert is pending. If the condition clears first, nothing is sent.
  2. 2. Which component decides who is notified and groups alerts together?

    Prometheus evaluates rules and sends firing alerts. Alertmanager groups, deduplicates, routes, inhibits and notifies.
  3. 3. Fifty pods in one cluster fire the same alert, and group_by is [alertname, cluster]. How many notifications do you get for the first batch?

    Alerts that share the group_by labels are merged into a single notification.
  4. 4. What does an inhibition rule do?

    Inhibition avoids redundant noise, for example muting warnings while the related page for the same cluster is already active.
  5. 5. Which alert is the better candidate for paging a human at night?

    Symptom-based alerts describe what users feel and are actionable. Cause-level metrics are better as dashboard panels.

Summary

  • Prometheus evaluates alert rules (expr, for, labels, annotations); alerts are inactive, pending or firing, and only firing ones reach Alertmanager.
  • Labels drive routing, and annotations (with a runbook link) help the person who receives the page.
  • Alertmanager routes through a tree (first match wins unless continue: true), groups alerts into fewer notifications, inhibits redundant ones and honors silences.
  • Receivers such as email, Slack and PagerDuty are configured separately, with secrets kept in files.
  • Alert on symptoms users feel; keep causes on dashboards, and test rules and routes before relying on them.

Next: Grafana: Data Sources and Dashboards