Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: Reviewing and Reducing Alert Noise

Analyze a month of alerts, find flapping and duplicate notifications, apply hysteresis, dependencies and routing, and measure the improvement with simple scripts.

The most common failure of monitoring is not a missed outage. It is a flood of alerts that people learn to ignore. When 200 messages arrive per day and 195 need no action, the five that matter get lost. In this project you will review an alert log the way a reliability engineer does: measure, classify, fix the biggest sources, and measure again. Zabbix and Nagios cannot run on this site, so the data is a built-in sample and all analysis runs as Python.

The method

  1. Export the data. In Zabbix: Reports > Action log (notifications sent) or the API (alert.get, event.get). In Nagios: the notification entries of nagios.log, or the history CGI.
  2. Count by source. Which triggers, hosts and severities cause the most notifications?
  3. Classify each top source: actionable, informational, flapping, duplicate or symptom of another problem.
  4. Choose the fix: change the threshold, add hysteresis, add a dependency, lengthen the evaluation window, route to a different team, lower the severity, suppress during maintenance, or delete the trigger.
  5. Measure after the change with the same query.
graph LR
  A["Export alerts"] --> B["Count by trigger"]
  B --> C["Classify top 10"]
  C --> D["Apply fix"]
  D --> E["Re-measure"]
  E --> B

Noise reduction is a loop, repeated every few weeks.

The sample data

The following records come from a made-up month. Each entry is a problem: trigger name, host, minutes open, whether anyone acknowledged it.

Python Find the noisiest triggers (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output with these questions: Which triggers produce most messages? Which of those are never acknowledged (nobody cared)? Which are short-lived (they recover by themselves in minutes)? "CPU load high" on web01 and web02 is the textbook case: hundreds of short, ignored problems.

Fix 1: evaluation windows and hysteresis

A trigger such as last(/web01/system.cpu.load)>4 fires on every spike. Change it to a window and a recovery level:

TEXT
Problem:   avg(/web01/system.cpu.load[all,avg1],10m)>{$CPU.LOAD.MAX}
Recovery:  avg(/web01/system.cpu.load[all,avg1],10m)<{$CPU.LOAD.MAX}*0.8

Use min() when a condition must hold for the whole period, avg() for sustained load, count() for "N failures in M minutes", and nodata() for silence. Nagios has a similar knob: max_check_attempts and flap_detection_enabled with low_flap_threshold and high_flap_threshold, and notification_interval to avoid repeats.

Python Compare trigger styles on spiky data (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The windowed version reports the sustained incident and ignores most single-sample spikes.

Fix 2: dependencies and symptom alerts

When a core switch fails, twenty access points report "Ping lost". Those are symptoms of one cause. In Zabbix, use trigger dependencies (the AP trigger depends on the switch trigger), or the event correlation and "root cause" marking in 7.0 and later. In Nagios use parent hosts (parents sw-core-1), which move child hosts to UNREACHABLE instead of DOWN, and host_dependency and service_dependency objects.

Python Suppress symptoms with dependencies (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Fix 3: routing and severity

Not every problem deserves a page. Assign severities honestly:

  • Disaster / High: customers affected now, act immediately.
  • Average: will affect customers soon, act during working hours.
  • Warning / Information: record it on a dashboard, no notification.

Then route: an action for severity High and above pages on-call by SMS; an action for Average sends chat messages to the team channel in business hours only (set the user media time period 1-5,08:00-18:00). Add tags to ease filtering. See webhooks and media types.

Fix 4: remove what nobody reads

If a trigger has never produced an action in six months, delete it or turn it into a dashboard item. Delete disabled devices, lab hosts and unused templates. Document why each remaining trigger exists in its description or in a runbook link ({TRIGGER.URL}).

Measure the result

Python Before and after (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Success is not just fewer alerts. The proportion of alerts that required action should rise, and real incidents must still be detected. Replay last quarter's incidents against the changed triggers to confirm they would still alert.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which alert source should you fix first?

    High volume with no response means ignored noise.
  2. 2. How do dependencies reduce noise?

    Dependent triggers stay silent while the parent trigger is in problem.
  3. 3. What does a recovery expression with a lower threshold add?

    Problem and recovery thresholds differ, so small fluctuations do not toggle the state.
  4. 4. Which Nagios setting turns a core outage into UNREACHABLE for child hosts?

    Parent relationships mark hosts behind a down parent as unreachable.

Summary

  • Measure alerts by trigger, host and acknowledgement, then fix the biggest sources.
  • Use windows, hysteresis, dependencies, honest severities and routing.
  • Delete triggers nobody acts on, and verify the changes against past incidents.
  • Repeat the review regularly; noise creeps back.

Next: Practice Questions and Scenarios