Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Alerting

Grafana Alert Rules: Queries, Conditions, Pending Period and States

Create Grafana-managed alert rules: queries plus Reduce and Threshold expressions, evaluation groups, pending period, labels, annotations, No Data and Error handling, and alert states.

Dashboards need someone looking at them. Alerting watches the data for you and tells people when something needs attention. Grafana's alerting system has four main parts: alert rules that decide when something is wrong, notification policies that decide who hears about it, contact points that say how to reach them, and silences and mute timings that pause notifications. This lesson is about the first part: rules.

Grafana does not run on this site, so the screens are described in words and the TryIt blocks are Python stand-ins for the evaluation logic. They model the behavior; they are not Grafana's alerting engine.

Grafana-managed rules

Grafana can evaluate two families of rules:

  • Grafana-managed alert rules run inside Grafana. They can query almost any data source and combine queries with expressions. They are created in Alerting > Alert rules > New alert rule.
  • Data source-managed rules are Prometheus (or Loki) alerting and recording rules that live in the data source's ruler; Grafana can show and edit them.

This lesson covers Grafana-managed rules, which are the ones most people start with.

The anatomy of a rule

The new alert rule form has numbered sections:

  1. Name: a clear name such as High error rate checkout. It becomes the alertname label.
  2. Define query and alert condition: one or more queries (A, B...) plus expressions. The default shape is: query A returns a series, a Reduce expression (B) collapses it to one number (Last, Mean, Max...), and a Threshold expression (C) compares it to a limit. The rule fires when the condition expression returns true. See expressions.
  3. Add folder and labels: choose the folder that stores the rule (permissions apply) and add labels (severity=critical, team=payments). Labels are essential: notification policies route on them.
  4. Set evaluation behavior: choose an evaluation group (a named set of rules with one evaluation interval, such as 1m), and the pending period.
  5. Configure notifications: choose a contact point directly, or let notification policies route by labels.
  6. Configure notification message: annotations such as summary, description, runbook_url, and a link to a dashboard and panel.

Rules in the same group are evaluated one after the other at the group's interval. Pick intervals that make sense for the data: evaluating every 10s a metric scraped every minute wastes resources; many installations enforce a minimum evaluation interval in the server settings.

Pending period

The pending period (also called for) prevents flapping. When the condition first becomes true, the alert does not fire at once. It enters the Pending state and only becomes Alerting when the condition has stayed true for the whole pending period. With a 1m interval and a 5m pending period, you need five consecutive true evaluations. A single bad sample resets the clock. A pending period of 0s fires immediately.

Alert states

Each alert instance (one per label set the query produced) has a state:

  • Normal: the condition is false (healthy).
  • Pending: the condition is true, but not yet for the whole pending period.
  • Alerting: the condition has been true for the pending period; notifications are sent.
  • No Data and Error: special states when the query returned nothing or failed; see below.

Newer versions add a Recovering stage, a configurable keep-firing period that delays resolution, which helps with flapping; check your version's documentation.

When a firing alert becomes false again, it goes back to Normal and Grafana sends a resolved notification.

Python Stand-in: pending period state machine

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Compare the two runs: with pending_evals=1 every spike fires; with 3, only the sustained run (97, 98, 99, 99) reaches Alerting, and the short spike at the start never does.

No Data and Error handling

A rule can fail in two ways, and each has a setting in the evaluation section:

  • If no data: what to do when the query returns no series. Choices: No Data (create an alert named DatasourceNoData), Alerting, Normal, or Keep Last State.
  • If execution error or timeout: Error, Alerting, Normal or Keep Last State.

Pick deliberately. "No data" is the symptom of a dead exporter, so many teams treat it as Alerting for critical services; but for rarely emitting metrics, Normal is more honest. Silent failure of monitoring is itself an outage.

Multi-dimensional alerts

If a query returns many series (one per host), a single rule creates one alert instance per series, each with its own state and labels. The labels from the series (instance="web-1") are copied onto the alert. A rule node CPU above 90% therefore alerts per host, and notification policies can group them. This is called multi-dimensional alerting. To avoid instance explosions keep the query's labels lean (aggregate away pod_template_hash, id, and other noisy labels).

Labels and annotations

  • Labels identify and route the alert: severity, team, service. Labels with templated values such as {{ $labels.instance }} should be avoided in rule labels because they change alert identity.
  • Annotations carry human information: summary ("High error rate on {{ $labels.service }}"), description, runbook_url, and the dashboard UID plus panel ID so the notification links straight to the graph. Annotation templates use Go template syntax with $labels and $values (for example {{ $values.B }} for the reduced value of expression B); see contact points and templates.
Python Stand-in: one rule, many instances

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Good rules

  1. Alert on symptoms users feel (error rate, latency, saturation, an empty queue that should not be empty) rather than every cause.
  2. Always include a runbook link and a clear summary.
  3. Keep a pending period long enough to ignore blips but short enough to matter.
  4. Use severity labels consistently so the routing in notification policies is simple.
  5. Test the rule: use Preview in the rule form to see the query result and the state it would produce.
stateDiagram-v2
    [*] --> Normal
    Normal --> Pending: condition true
    Pending --> Alerting: true for pending period
    Pending --> Normal: condition false
    Alerting --> Normal: condition false (resolved)

Alert instance lifecycle. No Data and Error are handled by settings that map to one of these states.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the pending period do?

    An alert stays Pending until the condition has been true for the whole pending period.
  2. 2. A query returns one series per host. How many alert instances can the rule create?

    Each series becomes its own alert instance with its own state and labels.
  3. 3. Which expressions form the default alert condition?

    Reduce collapses the series to a number and Threshold compares it with the limit.
  4. 4. Why is No Data handling important?

    Choose No Data, Alerting, Normal or Keep Last State deliberately for each rule.

Summary

  • Grafana-managed rules combine queries and expressions; the condition is usually Reduce plus Threshold.
  • Rules live in folders and evaluation groups; the group sets the interval.
  • States: Normal, Pending, Alerting, with settings for No Data and Error.
  • A query with many series makes one alert instance per series.
  • Labels route alerts, annotations explain them; add summaries and runbook links.

Next: Notification Policies