Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 37 ยท Triggers and Alerts

Zabbix Maintenance, Trigger Dependencies and Event Correlation

Reduce alert noise in Zabbix: maintenance periods with and without data collection, trigger dependencies for network outages, problem suppression, and event correlation basics.

A mature monitoring system is quiet. If every planned reboot sends fifty pages, or a failed switch produces three hundred "host unreachable" alerts, people stop trusting it. Zabbix offers three tools to keep alerts honest: maintenance (planned work), trigger dependencies (one cause, many symptoms) and event correlation (automatic matching and closing of events). This lesson covers all three, plus the equivalent features in Nagios.

Zabbix does not run on this site. The configuration is described in words and the Python stand-ins model how suppression works.

Maintenance periods

Data collection > Maintenance > Create maintenance period defines a window in which problems are handled differently.

Fields:

  • Name and Maintenance type:
    • With data collection: Zabbix keeps collecting values and evaluating triggers, but problems created during the window are suppressed: they are visible only if you tick "Show suppressed problems" and actions with the condition "problem is not suppressed" do not notify. Use this for most planned work, since you still get graphs and the history.
    • No data collection: Zabbix stops collecting data for the affected hosts, so no problems, no values and gaps in graphs. Use it if collecting is harmful (a database being restored).
  • Active since / Active till: the overall validity.
  • Periods: one time, daily, weekly or monthly schedules, with start time and length. For example "Sundays 02:00 for 3 hours".
  • Host groups and Hosts to which it applies, optionally filtered by tags (matching problem tags, e.g. only service: checkout problems).

When the window ends, suppressed problems that are still open become visible and trigger their actions, so you learn if the maintenance did not fix things.

Python Stand-in: suppressed problems

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You can also use problem suppression manually: acknowledge a problem and choose Suppress until a time or indefinitely, when someone knows about it and wants it out of the way. Suppression is on the problem, not on the host.

Maintenance and the API

Scripts often create maintenance windows around deployments: maintenance.create with the groups and time. A deployment pipeline that calls the API before it starts and removes the period at the end stops deployment noise automatically. See the integration chapter.

Trigger dependencies

A dependency says: trigger B depends on trigger A; do not create a problem for B while A is in PROBLEM. The classic case is the network:

  • Trigger A: "Switch core-sw-01 is unreachable" (ICMP ping fails).
  • Triggers B for 40 servers behind that switch: "Host is unavailable".

If B depends on A, then when the switch dies you get one problem, not 41. Once the switch is back, the servers' own triggers are evaluated again.

Configure it in the trigger form: Dependencies > Add, pick the parent trigger. A trigger can have many dependencies; Zabbix follows them recursively, and refuses circular chains.

flowchart TD
    SW["Switch unreachable<br/>(root cause)"] --> H1["web-01 unavailable"]
    SW --> H2["web-02 unavailable"]
    SW --> H3["db-01 unavailable"]

Child triggers stay quiet while the parent is in PROBLEM.

In templates you can define dependencies between triggers of the same template, or from a template trigger on a host's own trigger of another template. Dependencies across hosts are made on a specific host, since a template cannot name a host. For large networks use proxy or parent trigger conventions: one "ping" trigger per site gateway and dependencies for everything at the site.

Python Stand-in: dependencies silence child problems

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the child problems reappear when the parent is OK and they are still true; that is correct behaviour, because their own condition is still bad.

Event correlation

Data collection > Event correlation defines rules that match events and close old problems automatically. Two kinds:

  1. Global correlation rules: conditions on event tags, such as "old event tag equals new event tag" with the operation "close old event". Used when different triggers describe the same state: a log line "link down" opens a problem, and the later "link up" line should close it. Both events carry the tag interface: {#IFNAME}, so the correlation matches them.
  2. Trigger-level tag correlation: in the trigger form OK event closes: All problems if tag values match for the specified tag name, so one trigger's OK event closes many matching problems of itself (multiple-event triggers).

This is how you handle events that do not have a natural "recovery" state, like individual log events.

Nagios equivalents

  • Scheduled downtime: a Nagios command (or the web interface) sets a window for a host or service; notifications stop and availability reports account for it.
  • Parent hosts: the parents directive lets Nagios distinguish a DOWN host from an UNREACHABLE one and notify only for the root cause.
  • Service dependencies and notification dependencies: objects servicedependency and hostdependency suppress checks or notifications when a master service fails.
  • Flap detection: Nagios marks objects that change state too often as "flapping" and mutes notifications for them.

Practical advice

  1. Put a maintenance period in your change process, and use the API for automatic windows.
  2. Prefer with data collection unless collecting data is itself harmful.
  3. Build dependency chains where the topology is a real tree (site, switch, server, service).
  4. Alert on the cause, not each symptom: dependencies make that natural.
  5. Review noisy triggers monthly: the top ten alert generators are your improvement list (Reports > Top 100 triggers).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens to problems during a 'with data collection' maintenance?

    Collection and evaluation continue; the problems are suppressed so actions with the 'not suppressed' condition stay quiet.
  2. 2. A trigger B depends on trigger A. What happens when A is in PROBLEM?

    Dependencies suppress child problems while the parent problem exists.
  3. 3. What is event correlation mainly used for?

    Correlation rules close problems when a matching recovery or related event occurs.
  4. 4. Which Nagios feature distinguishes DOWN from UNREACHABLE?

    Parent relationships tell Nagios when a failure is only a consequence of an upstream device.

Summary

  • Maintenance periods suppress alerts for planned work; choose with or without data collection.
  • Trigger dependencies reduce a failure that affects many hosts to one root-cause problem.
  • Event correlation closes problems automatically using tags.
  • Nagios offers scheduled downtime, parent hosts, dependencies and flap detection.
  • Review the noisiest triggers regularly; quiet monitoring is trusted monitoring.

Next: Templates and Template Inheritance