Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 37 ยท Projects and Practice

Project: An End-to-End Grafana Alerting Pipeline and Practice Set

Design a complete alerting pipeline in Grafana: SLO-based alert rules, notification policy tree, contact points, mute timings and silences, then practice with review exercises.

The previous project built a dashboard people read. This one builds the system that calls people when they are needed: alert rules, a notification policy tree, contact points, a mute timing for a maintenance window and a silence procedure. At the end there is a practice set to review everything in this part of the course.

Grafana is not available on this site, so the real setup is described as steps and file excerpts for your own installation. The TryIt block is a Python stand-in that simulates the whole pipeline from sample data to notification, using the same ideas you have learned (pending period, label routing, grouping, silences, mute timings). It models the behavior; it is not Grafana's engine.

Requirements

You operate the cart, checkout and search services. The business rules:

  1. Page the on-call engineer when users see a lot of errors: error ratio above 5% for 5 minutes, for checkout and cart (severity critical).
  2. Notify the team's chat channel when p95 latency is above 800 ms for 10 minutes (severity warning).
  3. If a service stops reporting data, treat it as an alert (severity critical).
  4. No warnings at night or on weekends; critical alerts always notify.
  5. A planned maintenance window must be silenceable in one minute, with a comment.
  6. All of it must be reproducible from files (see provisioning).

Step 1: the rules

In Alerting > Alert rules, create one evaluation group service-health with a 1m interval in the folder Team, and two rules.

Rule High error ratio: query A is the error ratio PromQL from the dashboard project, sum by (service) (...) / sum by (service) (...); expression B is Reduce (Last) of A; expression C is Threshold B is above 0.05. Pending period 5m. If no data: Alerting. If execution error: Error. Labels: severity=critical, team=payments. Annotations: summary = Error ratio above 5% on {{ $labels.service }}, runbook_url, and a link to the dashboard panel.

Rule High p95 latency: similar, threshold 0.8 (seconds) with pending period 10m, labels severity=warning, team=payments, no-data handling Normal (a missing latency series is covered by the first rule's no-data behavior).

Because the queries are grouped by service, each rule is multi-dimensional: one alert instance per service (alert rules).

Step 2: contact points

In Contact points create:

  • oncall-pager: a PagerDuty (or Opsgenie) integration with the integration key stored in a secure field, plus a short template title [{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}.
  • payments-slack: a Slack integration for #payments-alerts with a body template listing .Alerts and the panel and silence URLs (contact points).
  • team-email: the fallback, an email address everyone reads.

Use Test on each. A pager that has never been tested is a pager that does not work.

Step 3: policies

Set the root policy to team-email with group by alertname, service; group wait 30s, group interval 5m, repeat 4h. Add nested policies, in this order:

  1. severity = critical to oncall-pager, group wait 10s, repeat 1h, continue matching on (so the chat also hears it).
  2. team = payments to payments-slack, with the mute timing nights-and-weekends attached.

Now the warnings reach Slack only during working hours, and criticals page at any time (the first policy does not have the mute timing). Create the mute timing with time range 20:00 to 08:00 and a second interval for saturday:sunday, in your team's time zone (notification policies, silences and mute timings).

Step 4: the maintenance procedure

Write a runbook line: "For planned work, open Alerting > Silences > Add silence, matchers service=checkout, duration 2h, comment with the change ticket ID." Practice it once in staging.

Simulate the pipeline

This stand-in feeds a series of per-minute error ratios through the pending-period logic, routes the resulting alert through the policy tree, then applies a mute timing and a silence.

Python Stand-in: from samples to notifications

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output carefully. The alert only becomes Alerting at the fifth consecutive bad minute, so the first four samples stay Pending. Daytime notifications reach both contacts; at 23:00 the pager still rings while the chat is muted; and a silence on service=checkout stops both.

Step 5: test and tune

  1. In the rule form use Preview to check the query and state.
  2. Fire a test alert (lower the threshold temporarily on staging) and verify each contact point, the grouping and the repeat behavior.
  3. Look at Alerting > Alert rules state filters and the History to see state changes.
  4. After a week review: which alerts fired, which led to action, which were noise (silences and mute timings).
flowchart LR
    R["Rules<br/>error ratio / latency"] --> P{"Policy tree"}
    P -- "critical" --> C1["oncall-pager"]
    P -- "team=payments" --> M{"Mute timing<br/>night/weekend?"}
    M -- no --> C2["payments-slack"]
    M -- yes --> X["Held back"]
    P -- "no match" --> C3["team-email"]

The pipeline you built: rules, routing, mute timing and three contact points.

Practice set

Try these without looking back, then check the lessons.

  1. A panel has max data points 1,000 and a 7-day range. Roughly what step does Grafana choose, and where do you see the final query? (query editor)
  2. Write the PromQL that graphs the per-second rate of jobs_failed_total per queue, using the best range variable. (PromQL)
  3. You have two queries returning (host, cpu) and (host, memory). Which transformations produce a table sorted by memory? (transformations)
  4. A variable lists regions with Multi-value on. How do you write the Prometheus matcher? How would a SQL query use it? (variables)
  5. What time range expressions mean "the whole of last month"? (time ranges)
  6. An alert flaps between Normal and Alerting every few minutes. Name two settings that reduce it. (alert rules)
  7. An alert must page the on-call and also post in chat. How do you configure it? (policies)
  8. Which provisioning file keys keep secrets out of Git? (provisioning)
  9. A contractor should read one folder only. List the steps. (permissions)
  10. What can a lint step catch in dashboard JSON before deployment? (dashboards as code)

Suggested answers: (1) about 10 to 15 minutes (604800 / 1000 is roughly 605 s, rounded up to a nice step) and the Query inspector's Query tab; (2) sum by (queue) (rate(jobs_failed_total[$__rate_interval])); (3) Join by field on host, then Sort by memory reversed; (4) region=~"$region" and region IN ($region); (5) From now-1M/M, To now-1M/M; (6) a longer pending period (and keep-firing or grouping/repeat settings); (7) a critical policy with Continue matching on, followed by a team policy; (8) secureJsonData with ${ENV_VAR} references; (9) a Viewer (or no-role) user, a team, View permission on the folder only; (10) duplicate ids, overlaps, missing uid, title or datasource, plaintext secrets.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the critical policy have Continue matching on?

    Without continue, the first matching sibling would end the search and the chat would never hear it.
  2. 2. Where is the mute timing attached so that critical pages still go out at night?

    Mute timings belong to policies; attaching it to the team policy holds back chat but not the pager.
  3. 3. What should happen to a service that stops reporting data in this design?

    Setting If no data to Alerting turns silence from a dead exporter into a notification.
  4. 4. What is the first thing to do after creating a contact point?

    Testing proves the integration works before a real incident depends on it.

Summary

  • A complete pipeline has rules (with pending period and no-data handling), policies, tested contact points, a mute timing and a silence procedure.
  • Route on severity and team labels; use continue matching deliberately; attach mute timings to the policies they should affect.
  • Keep everything in files so the setup is reproducible and reviewable.
  • Review alert history regularly and remove alerts nobody acts on.
  • You now have the full path from query to dashboard to alert to provisioning; return to the Grafana track page to revisit any chapter.

Next: Back to the Grafana track