Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 39 ยท Dashboards and Alerts

Checks, Alerts and Notification Rules

Learn how InfluxDB checks turn metrics into ok, info, warn and crit statuses, how deadman checks catch silence, and how notification rules avoid alert spam.

A dashboard only helps if someone is looking at it. Alerting watches the data for you and tells you when something is wrong. InfluxDB 2.x builds alerting from three pieces: checks that decide a status, notification endpoints that say where to send messages, and notification rules that say when to send them.

You will learn the status levels, the difference between a threshold check and a deadman check, and why raw "value above limit" alerts are noisy. The server is not available here, so the real definitions are shown read-only, and the logic is rebuilt in SQL and JavaScript so you can run it.

Checks and statuses

A check is a query that runs on a schedule (for example every minute) and assigns each result one of four levels:

Level Meaning
ok Everything normal
info Worth knowing
warn Needs attention soon
crit Needs attention now

Every evaluation writes a status point into the internal _monitoring bucket, with the level and a message. Two kinds of check exist:

  1. Threshold check. You define conditions such as "crit when usage is above 90", "warn above 75". Thresholds can also be "below" or "inside/outside a range".
  2. Deadman check. It fires when a series stops reporting for a set time, for example "no data from host=db-1 for 5 minutes". A silent dead server produces no high values, so a threshold check never notices it. Deadman checks catch that.

Underneath, InfluxDB checks are generated Flux. A threshold check looks roughly like this (read-only, simplified):

JavaScript
import "influxdata/influxdb/monitor"

data = from(bucket: "raw")
  |> range(start: -5m)
  |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_user")
  |> aggregateWindow(every: 1m, fn: mean, createEmpty: false)

data
  |> monitor.check(
    data: {tags: {}, _check_id: "check-cpu", _check_name: "CPU too high", _type: "threshold", _source_measurement: "cpu"},
    messageFn: (r) => "CPU is ${r.level}: ${string(v: r.usage_user)}%",
    crit: (r) => r.usage_user > 90.0,
    warn: (r) => r.usage_user > 75.0,
    ok: (r) => r.usage_user <= 75.0,
  )

You normally build this in the UI rather than writing it by hand.

Endpoints and rules

A notification endpoint is a destination: Slack, PagerDuty, or a generic HTTP endpoint (webhook). A notification rule connects a stream of statuses to an endpoint: "every minute, look at statuses, and when a status changes from ok to crit, send this message to the on-call Slack channel".

flowchart LR
  D["Metrics"] --> C["Check<br/>every 1m"]
  C --> S["Statuses<br/>ok / info / warn / crit"]
  S --> R["Notification rule<br/>on level change"]
  R --> E["Endpoint<br/>Slack, PagerDuty, HTTP"]

A check produces statuses, a rule watches them, and an endpoint delivers the message.

Rules usually fire on a change of level, not on every evaluation. Otherwise a long incident sends one message per minute.

Stand-in: levels with SQL

The runnable version uses CASE to assign levels and LAG to detect changes. This is the core of what a check and a rule do.

SQL Stand-in: assign levels and find the transitions

Press Run (or Ctrl + Enter) to see the result here.

Eight readings produced three messages (ok to warn, warn to crit, crit to ok), not eight. One reading sequence, many evaluations, few notifications.

Stand-in: a deadman check

A deadman check asks "when did I last hear from each host?". This JavaScript version evaluates "now" against each host's last timestamp.

JS Stand-in: detect hosts that stopped reporting

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Designing alerts people trust

  1. Alert on symptoms users feel, such as error rate and latency, plus a few causes such as disk almost full.
  2. Use durations. "CPU above 90 for 5 minutes" is better than a single spike. Do this with a wider aggregate window in the check.
  3. Separate warn and crit. Warn goes to chat, crit pages a person.
  4. Alert on change, and send a message when the level returns to ok so people know it is over.
  5. Add a deadman for every critical source.
  6. Write useful messages. Include the host, the value, the threshold and a link to the dashboard.
  7. Delete alerts nobody acts on. Unused alerts train people to ignore the real ones.

Alerting in InfluxDB 3

Core and Enterprise do not include checks. Common setups run alert queries from Grafana alerting (see Grafana and InfluxDB), from Kapacitor-style tools, or from a scheduled job that runs a SQL query and posts to a webhook. The design rules above still apply.

Common mistakes

  • Only a threshold check: a crashed server stays silent and nothing fires.
  • Notifying on every evaluation, producing hundreds of messages per incident.
  • Thresholds on raw noisy points rather than on a window mean.
  • Checks that read a bucket whose retention is shorter than the check's range.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which check type notices that a server stopped sending data?

    A deadman check fires when a series has not reported for a set time. A threshold check needs values to compare.
  2. 2. What are the four status levels of an InfluxDB check?

    Checks assign ok, info, warn or crit.
  3. 3. Why do notification rules usually fire on a level change?

    Messages on change keep alerts useful and avoid spam.
  4. 4. Which makes an alert less noisy?

    Using a window mean or duration filters out one-off spikes.

Summary

  • Checks assign ok, info, warn or crit; deadman checks catch missing data.
  • Endpoints say where, and rules say when, to notify.
  • Notify on status changes, with useful messages and a recovery message.
  • Use windows and durations to avoid spikes triggering pages.
  • InfluxDB 3 relies on external alerting such as Grafana.

Next: Grafana and InfluxDB