Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Integration and Operations

Webhooks, Media Types and Nagios Notifications

Send Zabbix alerts to chat and ticket systems: media types, user media, JavaScript webhook scripts, message templates, escalation tips, and Nagios notification commands.

An alert nobody sees is not an alert. Zabbix can deliver notifications by email, SMS, scripts and, most flexibly, webhooks that call any HTTP API: Slack, Microsoft Teams, PagerDuty, Jira, ServiceNow or your own service. This lesson explains media types, how a webhook script is written, and how Nagios sends notifications by commands. Zabbix and Nagios are not available on this site, so scripts are shown as read-only and the runnable stand-ins build and test the payloads in plain JavaScript and Python.

How notifications flow

graph LR
  T["Trigger fires"] --> A["Action<br/>conditions + operations"]
  A --> M["Media type<br/>email, webhook, script"]
  M --> U["User media<br/>address, severity, hours"]
  U --> X["Slack, Teams,<br/>PagerDuty, mail"]

An action chooses recipients; the media type delivers to each user's media entry.

  1. A trigger changes state and creates a problem event.
  2. An action (Alerts > Actions > Trigger actions) matches conditions such as host group or tag, and defines operations: send a message to users or groups through a media type, run a remote command, escalate after 30 minutes.
  3. A media type (Alerts > Media types) defines how to deliver: Email (SMTP settings), SMS (modem), Script (executes a file), Webhook (JavaScript).
  4. The recipient user needs a media entry (Users > Media): the address or ID, the severities that user wants, and the active hours.

Message text comes from the action's templates with macros: {TRIGGER.NAME}, {HOST.NAME}, {EVENT.SEVERITY}, {EVENT.TIME}, {EVENT.TAGS}, {TRIGGER.URL}. Separate templates exist for problem, recovery and update messages.

Built-in webhook media types

Zabbix ships ready webhook integrations: Slack, Microsoft Teams, Telegram, PagerDuty, Opsgenie, Jira, ServiceNow, Mattermost, Discord and others. For most, you import nothing: open the media type, fill in parameters (a bot token, a webhook URL), and test with the Test button. Create a user for the chat channel, add a media entry with the channel name or ID, and use that user in an action.

Writing a webhook script

A webhook media type has parameters (name and value pairs, with macros) and a script body in JavaScript (Duktape engine in older versions, a modern engine later). The script receives the parameters as a JSON string in the value variable and returns a result or throws an error.

JavaScript
// Read-only example: send a Slack-style message. Not runnable on this site.
var params = JSON.parse(value);
var request = new HttpRequest();
request.addHeader('Content-Type: application/json');

var payload = {
  text: '*' + params.severity + '* ' + params.subject + '\n' + params.message
};
var response = request.post(params.url, JSON.stringify(payload));

if (request.getStatus() !== 200) {
  throw 'Slack returned HTTP ' + request.getStatus() + ': ' + response;
}
return 'OK';

Helper objects available: HttpRequest (get, post, put, delete, setProxy), CurlHttpRequest alias in old versions, Zabbix.log(level, message) for the server log, XML, CookieJar and a few more. Limits: execution timeout (up to 60 seconds), and memory limits for the script.

Tips:

  • Pass secrets as macros of type secret text ({$SLACK.TOKEN}), not as plain text in the script.
  • Set a Process tags and Event menu entry option when the target system returns a ticket number, so you can show a clickable link to the ticket on the problem.
  • Update the external ticket on recovery and acknowledgement, using the "operation type" parameter ({EVENT.UPDATE.STATUS}, {EVENT.VALUE}).

Script and email media types

A script media type runs an executable from AlertScriptsPath, passing parameters as arguments ({ALERT.SENDTO}, {ALERT.SUBJECT}, {ALERT.MESSAGE}). Use it only when no HTTP API exists, because shell scripts are harder to secure and to test. Email needs an SMTP server, authentication, and a "send from" address; use STARTTLS and an app password rather than plain credentials.

Reducing noise before it is sent

Better delivery cannot fix bad alerts. Combine media with:

  • Severity filters per user: on-call engineers get Average and above; managers get Disaster only.
  • Action conditions on tags and host groups: the database team gets database problems.
  • Escalation steps: step 1 notify on-call, step 2 after 15 minutes notify the lead, step 3 after an hour notify management.
  • Maintenance and problem suppression for known work.
  • Event correlation and trigger dependencies so a failing switch does not page for every host behind it.

See alert noise review for a worked exercise.

Nagios notifications

In Nagios Core a notification is simply a command that runs with macros filled in:

TEXT
define command {
    command_name    notify-service-by-email
    command_line    /usr/bin/printf "%b" "Service: $SERVICEDESC$\nHost: $HOSTALIAS$\nState: $SERVICESTATE$\nInfo: $SERVICEOUTPUT$\n" | /usr/bin/mail -s "** $NOTIFICATIONTYPE$ $SERVICEDESC$ is $SERVICESTATE$ **" $CONTACTEMAIL$
}
define contact {
    contact_name                    oncall
    service_notification_period     24x7
    service_notification_options    w,u,c,r
    service_notification_commands   notify-service-by-email
    email                           [email protected]
}

Options w,u,c,r mean warning, unknown, critical, recovery. Escalations are separate serviceescalation objects with first_notification, last_notification, notification_interval and contact_groups. For chat tools, you write a command that calls curl with the macros, or use an add-on. Remember that Nagios can send a notification only if the state is hard (after max_check_attempts), and notification intervals avoid repeats.

Stand-in: build a chat payload

This JavaScript (run in the browser console) mimics what a webhook script does: it takes parameters and builds a payload while validating its input. It sends nothing.

JS Webhook payload builder (JavaScript stand-in)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Stand-in: escalation timeline

Python Escalation steps over time (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In Zabbix you can also stop escalations on acknowledgement, which the last line models.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What defines who receives an alert and through which channel?

    Actions choose recipients; the user's media entries hold the address and filters.
  2. 2. In what language are Zabbix webhook scripts written?

    Webhook media types run JavaScript with an HttpRequest helper.
  3. 3. Where should a bot token be stored for a webhook?

    Secret macros keep tokens out of scripts and exports.
  4. 4. What do the notification options w,u,c,r mean in Nagios?

    They select the states and recoveries that trigger notifications.

Summary

  • Triggers lead to actions, which use media types to notify users through their media entries.
  • Webhook media types run JavaScript against HTTP APIs and can create and update tickets.
  • Protect secrets with secret macros; reduce noise with severity filters, tags and escalations.
  • Nagios notifications are commands with macros, controlled by contact options and escalations.

Next: Security, Encryption and Permissions