Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 37 ยท Monitoring Concepts

Nagios Core: Objects, Plugins, Return Codes and Soft/Hard States

Learn how Nagios Core is configured: hosts, services, commands, contacts and time periods, plugin exit codes and thresholds, soft and hard states, and NRPE remote checks.

Nagios Core is the classic open-source monitoring daemon. Even if you choose Zabbix, you will meet Nagios concepts constantly: plugin exit codes, soft and hard states, host and service checks. Many other tools (Icinga, Naemon, Sensu, Checkmk) reuse the plugin format. This lesson teaches the Nagios model in detail.

Nagios Core does not run on this site. Configuration files and commands are shown read-only, and TryIt blocks are Python stand-ins that implement the same rules.

Objects

Nagios is configured with plain text files that define objects. Each object is a block define <type> { ... }.

Object Purpose
host A machine or device to monitor
hostgroup A named set of hosts
service One thing to check on a host (ping, disk, HTTP)
servicegroup A named set of services
command How to run a plugin: the command line, with macros
contact A person or address to notify
contactgroup A set of contacts
timeperiod Time ranges (24x7, workhours) for checks and notifications

A minimal configuration:

CFG
define command {
    command_name    check_ping
    command_line    $USER1$/check_ping -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$ -p 5
}

define host {
    use                 generic-host
    host_name           web-01
    alias               Web server 1
    address             192.0.2.10
    check_command       check-host-alive
    max_check_attempts  3
    contact_groups      admins
}

define service {
    use                     generic-service
    host_name               web-01
    service_description     PING
    check_command           check_ping!100.0,20%!500.0,60%
}

use generic-host means template inheritance: the object copies settings from a template defined with register 0. $HOSTADDRESS$ and $USER1$ are macros ($USER1$ is set in resource.cfg to the plugin directory, normally /usr/local/nagios/libexec). The ! splits a check command's arguments ($ARG1$, $ARG2$).

The main file is nagios.cfg; it points to object files with cfg_file= or whole directories with cfg_dir=. Always verify before reloading:

Terminal
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
sudo systemctl reload nagios

Plugins: the contract

A Nagios plugin is any executable that follows a simple contract:

  1. Print one line of status text, optionally followed by | and performance data.
  2. Exit with a return code:
Code State Meaning
0 OK Everything is fine
1 WARNING Beyond the warning threshold
2 CRITICAL Beyond the critical threshold
3 UNKNOWN The plugin could not decide (bad arguments, timeout, error)

Example output: DISK OK - free space: / 10240 MB (45%) | /=12288MB;18432;20480;0;22528

The text after | is performance data in the form label=value[unit];warn;crit;min;max, which add-ons turn into graphs. Because the contract is so simple, you can write a plugin in Bash, Python, Perl or Go in a few minutes.

The standard plugin package (nagios-plugins, or monitoring-plugins) provides check_ping, check_http, check_tcp, check_disk, check_load, check_procs, check_swap, check_users, check_dns, check_ssh, check_smtp, check_snmp, and more. Run any plugin with -h to see its options:

Terminal
/usr/local/nagios/libexec/check_http -H www.example.com -S -w 1 -c 3 --sni

Threshold ranges

-w and -c take a range whose format is the same across all plugins:

Written Alerts when the value is
10 < 0 or > 10 (outside 0 to 10)
10: < 10
~:10 > 10
10:20 < 10 or > 20 (outside the range)
@10:20 between 10 and 20, inclusive (inside the range)

This notation is easy to get backwards. Note that -w 80 -c 90 for a "percent used" check alerts above 80 and 90, but for "percent free" you would write -w 20: -c 10: to alert when the value is below 20 and 10.

Python Stand-in: Nagios threshold ranges

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A plugin in a few lines

Python Stand-in: write a plugin

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a real plugin, the last line would be print(text); sys.exit(code).

Soft and hard states

A single failed check should not wake anybody up. Nagios therefore re-checks before it declares a problem:

  • When a check fails, the state is SOFT. The attempt counter starts at 1.
  • Nagios retries every retry_interval minutes, until max_check_attempts is reached.
  • When the failure persists for all attempts, the state becomes HARD, and notifications are sent.
  • A return to OK is a hard recovery (and sends a recovery notification if a problem notification was sent).

With check_interval 5, retry_interval 1, max_check_attempts 4, a failing service is noticed within 5 minutes, then confirmed in 3 more minutes, so the first notification arrives after roughly 8 minutes worst case.

Python Stand-in: soft and hard states

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first failure was soft and cleared. The later run of failures became hard on the third failure and notified exactly once.

Hosts versus services

A host check decides whether the machine is UP or DOWN (usually by ping); service checks (disk, HTTP, ...) run only on hosts that are not DOWN. When a switch fails, all hosts behind it appear unreachable. Nagios distinguishes DOWN from UNREACHABLE through parents relationships, so you get one notification for the switch and not hundreds for servers behind it. Zabbix achieves a similar effect with trigger dependencies.

Remote checks: NRPE

Plugins like check_disk measure the local machine, so for remote hosts Nagios uses NRPE (Nagios Remote Plugin Executor): an NRPE daemon on the target (port 5666) runs allow-listed commands, and check_nrpe on the Nagios server asks for them. Alternatives are SSH, SNMP and the Nagios Cross Platform Agent (NCPA). In Zabbix terms the NRPE daemon plays the role of the agent.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which exit code means CRITICAL?

    0 OK, 1 WARNING, 2 CRITICAL, 3 UNKNOWN.
  2. 2. What does the range @10:20 alert on?

    The @ prefix inverts the range, so it alerts inside it.
  3. 3. When are Nagios notifications normally sent?

    A problem is SOFT until max_check_attempts is reached, then it is HARD and notifies.
  4. 4. What follows the | character in plugin output?

    Performance data uses label=value;warn;crit;min;max.

Summary

  • Nagios is configured with objects: hosts, services, commands, contacts, time periods and groups; templates use use and register 0.
  • Plugins print one line plus optional performance data and return 0, 1, 2 or 3.
  • Threshold ranges follow one format across plugins; @ inverts.
  • Failures are SOFT until max_check_attempts, then HARD and notified.
  • NRPE runs plugins on remote hosts; verify configuration with nagios -v.

Next: Availability and SLA math