Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Alerting

Grafana Contact Points, Webhooks and Notification Templates

Set up Grafana contact points for email, Slack, PagerDuty and webhooks, read the webhook payload, and write notification templates with Go template syntax and custom template blocks.

A contact point is a named destination for notifications: one or more integrations such as email, Slack, Microsoft Teams, PagerDuty, Opsgenie, Telegram, a generic webhook and many others. Notification policies choose a contact point; the contact point decides how the message looks and where it goes. This lesson covers contact points, the webhook payload, and templates that customize the text.

Grafana cannot run here, so interface steps are in words. The TryIt blocks are Python stand-ins: a payload builder and a tiny renderer that mimics a small part of Go templates, clearly not the real template engine.

Creating a contact point

Go to Alerting > Contact points > Add contact point. Give it a Name (for example payments-oncall), choose an Integration type, fill in its settings, and click Test to send a sample notification before saving. A contact point can have several integrations, so one contact point can email the team and post to Slack at once.

Typical settings:

  • Email: addresses separated by semicolons or commas. Needs SMTP configured in grafana.ini ([smtp] section). Message and subject can be templated.
  • Slack: an incoming webhook URL, or a bot token plus channel. Optionally title and text templates, mentions, and colors for firing and resolved.
  • PagerDuty: an integration key and severity; deduplication uses the alert group key.
  • Webhook: a URL, an HTTP method (usually POST), optional basic auth or bearer token, extra headers. Grafana sends JSON.
  • Others (Teams, Telegram, Discord, Opsgenie, VictorOps, Google Chat, Pushover, Kafka REST proxy...) follow the same pattern: credentials plus optional templates.

Disable resolve message stops the "resolved" notification for that integration. Secrets (tokens and passwords) are stored encrypted and never shown again after saving.

The webhook payload

A webhook integration sends a JSON document modeled on Prometheus Alertmanager's. The main fields:

Field Meaning
receiver Name of the contact point
status firing or resolved for the group as a whole
alerts List of alert objects (see below)
groupLabels Labels the group was formed by
commonLabels / commonAnnotations Labels and annotations shared by all alerts in the group
externalURL Link to the Grafana instance
groupKey Identifier of the group
title, message Rendered title and text (Grafana adds these)

Each element of alerts has status, labels, annotations, startsAt, endsAt, generatorURL, fingerprint, and Grafana adds dashboardURL, panelURL, silenceURL and valueString. Receivers such as incident tools or chat bots parse this JSON.

Python Stand-in: build and read a webhook-like payload

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notification templates

By default, each integration uses a built-in template. You customize text with notification templates (Alerting > Contact points > Notification templates). Templates use the Go text/template language. The data you work with:

  • .Alerts: all alerts in the notification; .Alerts.Firing and .Alerts.Resolved are filtered lists.
  • .Status, .Receiver, .GroupLabels, .CommonLabels, .CommonAnnotations, .ExternalURL.
  • Inside a loop, each alert has .Labels, .Annotations, .StartsAt, .EndsAt, .ValueString, .DashboardURL, .PanelURL, .SilenceURL.
  • Labels are maps: {{ .Labels.service }}; list them with {{ range .Labels.SortedPairs }}{{ .Name }}={{ .Value }} {{ end }}.

A template block is named with define, and can be reused by other templates with template:

TEXT
{{ define "alert.title" -}}
[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }} ({{ .Alerts.Firing | len }} firing)
{{- end }}

{{ define "alert.body" -}}
{{ range .Alerts }}
* {{ .Labels.service }}: {{ .Annotations.summary }}
  Started {{ .StartsAt.Format "15:04 UTC" }}
{{ end }}
{{- end }}

Then, in the Slack integration's Title field you write {{ template "alert.title" . }} and in Text {{ template "alert.body" . }}. The dot passes the current data to the block. Click Preview in the template editor to render it with sample alerts and see syntax errors before saving.

Notes:

  • In alert rule annotations (not notification templates), the data is different: {{ $labels.instance }} and {{ $values.B }}, {{ $value }}.
  • Whitespace control: {{- trims space before and -}} trims space after.
  • Slack and Teams may need escaping; use the built-in functions such as toUpper, toLower, title, join, reReplaceAll, stringSlice, tmpl.
  • A template that fails to render can produce an empty or error notification, so test with Preview.
Python Stand-in: a tiny template renderer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Testing and good practice

  • Use the Test button on each integration, and Preview on each template.
  • Keep titles short and actionable; put details and runbook links in the body.
  • Include a link to the dashboard panel (.PanelURL) and the silence URL, so recipients can investigate or mute in one click.
  • Use different contact points for different urgency, so a Slack channel for warnings does not drown the pager's critical alerts.
  • Do not put secrets in templates; use the integration's secure fields.
flowchart LR
    R["Alert rule<br/>labels + annotations"] --> P["Notification policy<br/>matchers"]
    P --> C["Contact point"]
    C --> T["Template<br/>title + body"]
    T --> I1["Slack"]
    T --> I2["Email"]
    T --> I3["Webhook (JSON)"]

One alert, one contact point, several integrations, one template.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a contact point?

    A contact point groups integrations that receive notifications from policies.
  2. 2. Which webhook payload field holds the labels shared by all alerts in the group?

    commonLabels lists the labels that every alert in the notification has with the same value.
  3. 3. Which Go template construct reuses a named block?

    define names a block and template calls it, passing the current data with the dot.
  4. 4. Where do you check that a template renders without errors before saving?

    Preview renders the template against sample alerts and reports syntax errors.

Summary

  • Contact points hold integrations (email, Slack, PagerDuty, webhook and more); use Test before saving.
  • Webhook payloads follow the Alertmanager model: status, alerts, groupLabels, commonLabels, commonAnnotations.
  • Notification templates use Go templates with .Alerts, .Labels, .Annotations, plus define and template.
  • Rule annotation templates use $labels and $values instead.
  • Link to the dashboard panel and silence URL, and separate urgency levels into separate contact points.

Next: Silences and Mute Timings