Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 37 ยท Triggers and Alerts

Zabbix Trigger Severity, Recovery Expressions, Hysteresis and Flapping

Control alert noise in Zabbix: the six severity levels, problem and OK events, recovery expressions for hysteresis, multiple problem events, operational data, tags and manual close.

A trigger that is technically correct but alerts at the wrong moment is a bad trigger. If it fires and recovers every few minutes (flapping), people stop reading alerts. If it never says how important it is, everything looks equally urgent. This lesson covers the settings that make triggers useful: severity, recovery expressions (hysteresis), event generation modes, operational data, tags and manual close.

Zabbix does not run on this site. The configuration is described in words, and the Python stand-ins simulate trigger behaviour.

Severity

Every trigger has one of six severities. Use them consistently, because actions and dashboards filter on them.

Severity Typical use Frontend color
Not classified Unknown importance gray
Information Something happened; no action needed light blue
Warning Something may become a problem; look when convenient yellow
Average A function is degraded; handle during working hours orange
High A service is down or at risk; act now light red
Disaster Business-critical outage; page people at any hour red

A workable policy: Warning and Information go to dashboards and mail; Average goes to chat; High and Disaster page the on-call. Fewer, more meaningful pages beat many ignored ones. You can change the severity names and colors in Administration > General > Trigger displaying options.

Problem and OK events

Each time a trigger changes state, Zabbix creates an event: OK to PROBLEM creates a problem event (and a visible problem), PROBLEM to OK creates a recovery event. Events are stored for audit (Housekeeping controls how long). Problems appear in Monitoring > Problems with the duration; you can acknowledge, comment, change severity, suppress or close them.

OK event generation has three options:

  • Expression (default): the trigger is OK when its problem expression is false.
  • Recovery expression: the trigger is OK only when a separate expression is true.
  • None: the problem never recovers by itself; someone closes it manually (useful for one-off events such as log lines or security alerts).

Hysteresis with a recovery expression

Think of a thermostat: it turns heating on below 19 degrees and off above 21, not at one value. Without that gap it would switch constantly around 20. A trigger has the same issue: if the problem expression is last(//vfs.fs.size[/,pused])>90 and the disk oscillates between 89 and 91, you get a problem, a recovery, a problem... every minute.

A recovery expression adds the gap. Problem: last(//vfs.fs.size[/,pused])>90. Recovery: last(//vfs.fs.size[/,pused])<85. Once in PROBLEM, the trigger stays there until the value is below 85. Between 85 and 90 nothing changes.

The two expressions together define a state machine:

stateDiagram-v2
    [*] --> OK
    OK --> PROBLEM: problem expression true
    PROBLEM --> OK: recovery expression true
    PROBLEM --> PROBLEM: in between: no change

With a recovery expression the trigger holds its state until the recovery condition is met.

Python Stand-in: hysteresis stops flapping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The state string uses O for OK and P for PROBLEM. Without hysteresis there are many changes; with the recovery expression only the real ones remain.

Smoothing with windows

Hysteresis is one tool; windows are another. min(//system.cpu.util,5m)>90 demands 5 minutes of sustained load. For recovery, you might use max(//system.cpu.util,10m)<70 to demand 10 calm minutes. Combining both gives very stable alerts. See trigger expressions.

Event generation mode: single or multiple

By default a trigger generates one problem and no more until it recovers (Single). With Multiple each evaluation that is true (for example each new log line matching an error) creates a separate problem event. Multiple mode suits log triggers: every error line is its own event, and each must be closed manually if recovery is set to None.

Operational data

The trigger name should be stable (Disk space is low on {HOST.NAME}), but you want to see the current value next to it in the Problems list. The Operational data field takes a text with macros, shown in the problem list: Used: {ITEM.LASTVALUE1}, free: {ITEM.LASTVALUE2}. Putting values in the name is discouraged because a changing name affects event correlation and notification text; keep names constant and values in operational data.

Python Stand-in: expanding macros

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Tags on triggers

Trigger tags (key/value pairs, with macros) are attached to the problems they create. Use them to describe what is affected and who owns it: service: checkout, scope: availability, team: {$TEAM}. Tags let you filter the Problems view, route alerts in actions ("only problems with tag team=db") and map problems to Services for SLA reports. Matching tags with the same name between a problem and its recovery lets event correlation pair them.

Manual close and acknowledgement

  • Allow manual close lets operators close a problem even though the trigger is still true. Without it, a problem with recovery mode None can never go away.
  • Acknowledge records that someone took ownership; it can also add a comment, change severity, or suppress the problem for a time. Acknowledgement can itself trigger an action ("update operations") that notifies the team that somebody is on it.

A review checklist for new triggers

  1. Does it describe a symptom that needs a human?
  2. Is the severity honest?
  3. Does it use a sustained window or a recovery expression to avoid flapping?
  4. Does the name stay constant and the details live in operational data?
  5. Do tags identify owner and service?
  6. Can the operator act on it, with a clear description, or a URL to a runbook (the trigger's URL field)?

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a recovery expression add to a trigger?

    The trigger stays in PROBLEM until the recovery expression is true, which prevents flapping.
  2. 2. When should you use OK event generation 'None'?

    None means the problem does not recover automatically.
  3. 3. Where should the current value of an item be shown in the Problems list?

    Operational data shows current values while the trigger name stays stable.
  4. 4. Which severity fits a business-critical outage at night?

    Disaster is the top level and normally pages the on-call engineer.

Summary

  • Six severities (Not classified to Disaster) steer who gets alerted; define a policy.
  • Problem and recovery events are created on every state change.
  • Recovery expressions give hysteresis; windows (min, max, #N) give sustained conditions.
  • Multiple event generation creates one problem per matching event, ideal for logs.
  • Use operational data, tags, URL and manual close to make problems actionable.

Next: Actions and media types