Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 37 ยท Templates and Discovery

Low-Level Discovery (LLD)

Understand Zabbix low-level discovery: how a discovery rule returns JSON, how LLD macros feed item prototypes, and how filters and lost-resource handling work.

Servers have a different number of filesystems, network cards, disks and services each. Writing one item per filesystem by hand does not scale, and the list changes over time. Low-level discovery (LLD) solves this: Zabbix asks the monitored system "what do you have?", receives a list, and creates items, triggers and graphs for every entry automatically.

This lesson explains the JSON contract, the macros, and the life cycle of discovered entities. A hands-on look at prototypes follows in discovery rules and prototypes. Zabbix cannot run here, so the examples use Python to parse and expand the same JSON.

The idea in three steps

  1. A discovery rule is an item with a special value type. It runs on a schedule (every hour is typical) and returns JSON.
  2. Every object in that JSON has fields named like LLD macros: {#FSNAME}, {#FSTYPE}, {#IFNAME}.
  3. Prototypes (item, trigger, graph, host prototypes) contain those macros in their keys and names. For each object, Zabbix substitutes the macros and creates a real entity.
graph LR
  R["Discovery rule<br/>vfs.fs.discovery"] --> J["JSON list<br/>of objects"]
  J --> P["Prototypes<br/>with {#MACROS}"]
  P --> E["Real items, triggers,<br/>graphs per object"]

A discovery rule feeds prototypes; each JSON entry becomes real entities.

The JSON format

Since Zabbix 4.2 the rule may return a plain JSON array of objects. Older agent keys still return an object with a data array, which Zabbix also accepts. A filesystem discovery result looks like this:

JSON
[
  {"{#FSNAME}": "/",     "{#FSTYPE}": "ext4"},
  {"{#FSNAME}": "/boot", "{#FSTYPE}": "vfat"},
  {"{#FSNAME}": "/data", "{#FSTYPE}": "xfs"}
]

Keys must be written as {#NAME} using capital letters, digits, underscores and dots. Each entry must be a flat JSON object. The built-in agent keys include vfs.fs.discovery, net.if.discovery, vfs.dev.discovery and system.cpu.discovery; SNMP uses discovery[{#IFDESCR}, ifDescr]; there is also ODBC, JMX and script based discovery.

Item prototypes

A prototype is an item definition with macros:

Field Value
Name Free space on {#FSNAME}
Key vfs.fs.size[{#FSNAME},pfree]
Type Zabbix agent

For the JSON above Zabbix creates three items, one per filesystem. The trigger prototype last(/Template OS/vfs.fs.size[{#FSNAME},pfree])<{$FS.FREE.MIN:"{#FSNAME}"} then makes three triggers, each with a threshold looked up by context, so /data can have a different limit than /.

Filters

Not everything discovered is worth monitoring. A rule can have filters that use a macro and a regular expression:

  • {#FSTYPE} matches ^(ext4|xfs|btrfs)$ keeps real disks.
  • {#FSTYPE} does not match ^(tmpfs|devtmpfs|squashfs)$ excludes virtual filesystems.

Entries that fail the filter are simply not created. If you later widen the filter, they appear on the next run.

What happens when something disappears

If a filesystem is missing in the next run, Zabbix does not delete its items at once. It marks them as lost and keeps them for the period set in "Keep lost resources period" (default 30 days). During that time the items show a warning icon and a deletion date. If the entry returns, the items continue with their history. After the period they are removed, history included. A short value (such as 0) deletes at once, which hides history you may want later, so think before shortening it.

Zabbix also has "Disable lost resources" in newer versions: instead of deleting, items are disabled after a chosen time.

Parsing and expanding discovery output (stand-in)

The Python below is not part of Zabbix. It parses a discovery payload, applies a filter, and expands item prototypes the way the server does.

Python Expand item prototypes (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Validating output from a custom script

Custom discovery scripts are common (user parameters, external checks). A malformed payload makes the rule go "not supported", so check it before deploying.

Python Discovery payload validator (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Common mistakes

  • Using a macro that the rule does not return in a prototype key: the item stays unsupported.
  • Returning two entries that produce the same key: Zabbix reports a conflict and creates only one.
  • Running discovery every minute on a system that changes rarely: wasted work. An hour is plenty.
  • Forgetting a filter, so that every docker overlay mount gets its own trigger.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a discovery rule return?

    The JSON entries carry macros like {#FSNAME} that prototypes substitute.
  2. 2. A filesystem vanishes from discovery output. What happens by default?

    Lost resources are kept for the configured period (default 30 days) before deletion.
  3. 3. Which text is a valid LLD macro name?

    LLD macros use {#NAME}. {$NAME} is a user macro.
  4. 4. How do you skip tmpfs filesystems?

    Filters on discovered macros exclude entries before entities are created.

Summary

  • LLD turns a changing list of objects into real items, triggers and graphs.
  • The rule returns JSON; each entry supplies {#MACROS}.
  • Filters decide which entries are used; lost entries are kept for a while before deletion.
  • Validate custom discovery output: flat objects, valid macro names, no duplicates.

Next: Discovery Rules and Prototypes