Skip to content
Tutorials Center
My learning

Topics

Lesson 12 of 37 ยท Hosts and Items

Zabbix Agent Item Keys Reference: CPU, Memory, Disk, Network, Processes, Logs

A practical guide to the most used Zabbix agent item keys for CPU, memory, filesystems, network, processes, services and logs, plus UserParameter for custom metrics.

The Zabbix agent ships with several hundred built-in item keys. You rarely need all of them, because templates already contain the common ones. But you should know the main families, their parameters and units, because you will adjust templates, test keys with zabbix_get, and create custom items. This lesson is a guided tour.

The agent does not run on this site. Examples show real commands and stand-in Python for parsing and calculation.

Availability and identity

Key Returns
agent.ping 1 if the agent is available (use for availability triggers)
agent.version Agent version string
agent.hostname Hostname from configuration
system.hostname Hostname reported by the OS
system.uname Output like uname -a
system.uptime Seconds since boot
system.boottime Boot time (Unix timestamp)
system.localtime Local time as timestamp (use to detect clock drift)

CPU and load

Key Notes
system.cpu.load[all,avg1] Load average (also avg5, avg15). Divide by core count to compare hosts.
system.cpu.util[,user] CPU percentage by type: user, system, idle, iowait, steal, nice, softirq... Omit the type for total utilization.
system.cpu.num Number of CPUs (cores).
system.cpu.switches Context switches counter; use Change per second.
system.cpu.intr Interrupts counter.

Memory

Key Notes
vm.memory.size[total] Total RAM in bytes
vm.memory.size[available] RAM available to applications (includes reclaimable cache)
vm.memory.size[pavailable] Available as a percentage
vm.memory.size[used] Used memory
system.swap.size[,pfree] Percent of swap free

Use available, not free: Linux uses spare RAM for caching, so "free" is almost always small, and a trigger on it creates false alarms.

Filesystems

Key Notes
vfs.fs.size[/,total], [/,used], [/,free] Bytes; modes pfree and pused give percent
vfs.fs.inode[/,pfree] Percent of free inodes (a full inode table looks like a full disk)
vfs.fs.discovery JSON used by discovery rules
vfs.dev.read[sda,sps], vfs.dev.write[...] Disk activity
vfs.file.exists[/var/run/app.pid] 1 if the file exists
vfs.file.size[...], vfs.file.cksum[...], vfs.file.contents[...] File checks

The vfs.fs.* and vfs.dev.* families are often used with low-level discovery, so each mounted filesystem gets its own items automatically. See templates and discovery.

Network

Key Notes
net.if.in[eth0,bytes] Inbound bytes counter; use Change per second and the multiplier 8 for bits
net.if.out[eth0,bytes], net.if.in[eth0,errors], net.if.in[eth0,dropped] Outbound, errors, drops
net.if.discovery Interface list JSON
net.tcp.listen[443] 1 if the local port is listening
net.tcp.port[192.0.2.5,3306] 1 if a TCP connection works (checked from the agent's machine)
net.tcp.service[ssh] Checks SSH service handshake
net.dns[,example.com,A], net.dns.record[...] DNS lookups from the agent host

Processes and services

Key Notes
proc.num[nginx] Number of processes named nginx
proc.num[,www-data,run] Processes of a user in a state
proc.cpu.util[nginx] CPU used by that process
proc.mem[java,,sum,rss] Memory used
service.info[W32Time,state] (Windows) Service state
systemd.unit.get[nginx.service] (agent 2) Systemd unit details

Logs and Windows events

Log items require active checks and the type "Zabbix agent (active)" with type of information Log:

TEXT
log[/var/log/nginx/error.log,"\[error\]",,,skip]
logrt["/var/log/app/app-.*\.log",ERROR]
eventlog[Application,,"Error|Warning"]

The first parameter is the file, the second a regular expression, then encoding, maxlines per second and the mode (all to read from the start, or skip to read only new data). Log items read incrementally and remember their position.

Windows-specific

perf_counter["\Processor(_Total)\% Processor Time"], wmi.get[root\cimv2,select Name from Win32_OperatingSystem], service_state[Spooler], registry.get[...] (agent 2). On Windows, template names like Windows by Zabbix agent active use these keys.

UserParameter: your own keys

If no built-in key does what you need, add a UserParameter line to the agent config (or an included file in zabbix_agent2.d/):

INI
# fixed key
UserParameter=app.queue.size,/usr/local/bin/queue_size.sh
# key with a parameter: $1 is replaced by the argument
UserParameter=app.file.lines[*],wc -l < "$1"

Then restart the agent and test: zabbix_get -s HOST -k "app.file.lines[/etc/hosts]". A UserParameter command must be fast (under the agent timeout), print one value to stdout, and run as the zabbix user. Be careful with parameters: with UnsafeUserParameters=0 (default) characters such as \ ' " \ * ? [ ] { } ~ $ ! & ; ( ) < > | # @` are refused in arguments, which protects you from command injection.

Python Stand-in: check UserParameter safety

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Doing the math the agent leaves to you

Many keys return counters or raw bytes. These conversions are done in preprocessing or in trigger expressions. The stand-in below shows a typical set.

Python Stand-in: bytes, rates and percentages

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Agent 2 plugins

Agent 2 includes built-in plugins for specific systems, with their own keys: mysql.ping[...], pgsql.ping[...], redis.ping[...], docker.info, memcached.ping, mongodb.ping, oracle.ping, systemd.unit.discovery, and more. Their connection settings (sessions, credentials) are given with Plugins.<Name>.Sessions.<name>.* options or macros in templates. These templates are listed under "Template DB MySQL by Zabbix agent 2" and similar names.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which key should you use to know how much memory applications can still use on Linux?

    Available includes reclaimable cache, so it reflects real headroom.
  2. 2. What is needed for a log[...] item to work?

    Log monitoring is only available as active checks.
  3. 3. How does a UserParameter with [*] receive its argument?

    The first key parameter replaces $1 in the command line.
  4. 4. Which key shows percent of free inodes on a filesystem?

    Inode exhaustion makes a disk appear full even though bytes remain; monitor vfs.fs.inode.

Summary

  • Key families: agent, system., vm., vfs., net., proc.*, log, and Windows-specific keys.
  • Prefer available over free for memory, and watch inodes as well as bytes.
  • Counters (net.if.*, system.cpu.switches) need Change per second.
  • Log items need active mode; UserParameter adds custom keys safely when arguments are restricted.
  • Agent 2 plugins add database and container keys.

Next: Value types and preprocessing