Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 37 ยท Projects and Practice

Project: A Custom Discovery Script and Template

Build a custom low-level discovery for application queues: a script that outputs discovery JSON, a user parameter, a data collector, prototypes, and tests.

Built-in templates cover operating systems and popular software. Your own application is not in them. In this project you will build a complete custom monitoring integration for an imaginary job-queue service that exposes its state as JSON: a discovery script, a collector script, a user parameter file, and the template that uses them. Zabbix cannot run on this site, so configuration files are shown read-only, and the scripts' logic runs as Python in the lesson so you can test it for real.

The scenario

The application writes /var/lib/jobs/status.json:

JSON
{
  "queues": [
    {"name": "email",   "depth": 42,   "workers": 4, "oldest_age_s": 18},
    {"name": "reports", "depth": 3,    "workers": 1, "oldest_age_s": 3},
    {"name": "images",  "depth": 1500, "workers": 8, "oldest_age_s": 940}
  ]
}

Goal: one discovery rule finds queues, and for each queue you get depth, workers and oldest job age, with triggers for a growing backlog and for stuck queues.

Design decisions

  • One read per minute of the file (a master item) and dependent items for everything else, to keep load low.
  • Discovery and data come from the same source, so they cannot disagree.
  • Names from the application may contain odd characters; the discovery script must emit valid JSON no matter what.
  • Scripts run as the zabbix user with a time limit; they must be fast and must not print anything but the result.

Step 1: the agent configuration

In /etc/zabbix/zabbix_agentd.d/jobs.conf:

INI
UserParameter=jobs.status,cat /var/lib/jobs/status.json
Timeout=10

In a real project you would restrict the user parameter to read exactly this file, and make sure zabbix has read permission on it. Avoid unrestricted parameters such as UserParameter=jobs.get[*],/usr/local/bin/jobs $1, because flexible arguments let a compromised server run anything (see security).

Step 2: the template

  1. Master item: key jobs.status, type Zabbix agent, value type Text, interval 1m, history 1h (only needed to feed the dependents).
  2. Discovery rule: type Dependent item, master jobs.status, key jobs.queues.discovery. Preprocessing: JavaScript that returns the list of {#QUEUE} objects (see below).
  3. Item prototypes (all dependent on jobs.status):
    • jobs.depth[{#QUEUE}], JSONPath $.queues[?(@.name=="{#QUEUE}")].depth.first(), numeric unsigned.
    • jobs.workers[{#QUEUE}] and jobs.age[{#QUEUE}] the same way, unit s for age.
  4. Trigger prototypes:
    • min(/Template App Jobs/jobs.depth[{#QUEUE}],10m)>{$JOBS.DEPTH.MAX:"{#QUEUE}"} severity Average.
    • last(/Template App Jobs/jobs.age[{#QUEUE}])>{$JOBS.AGE.MAX:"{#QUEUE}"} severity High, "oldest job is waiting too long".
  5. Macros with defaults: {$JOBS.DEPTH.MAX} = 1000, {$JOBS.AGE.MAX} = 300. Override with context for a big queue: {$JOBS.DEPTH.MAX:"images"} = 5000.
  6. Tags: component:queue, queue:{#QUEUE}.

The discovery preprocessing script

In the rule's preprocessing, a JavaScript step receives the status text in value:

JavaScript
// read-only: runs inside Zabbix
var data = JSON.parse(value);
return JSON.stringify(data.queues.map(function (q) {
  return { '{#QUEUE}': String(q.name) };
}));

Step 3: test the logic before deploying

Use the stand-ins below to test the same transformations offline.

Python Discovery transform and JSONPath-like extraction (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Step 4: test the triggers with macros and context

Python Trigger prototype simulation (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The images queue passes the depth check because of its context macro, but still trips the age trigger: its oldest job is waiting 940 seconds, so workers are not keeping up.

Step 5: failure handling

Decide what should happen in the awkward cases, and test each one:

  • The file is missing or empty: the master item becomes unsupported; add a trigger on nodata(/host/jobs.status,5m) so you hear about it.
  • JSON is invalid: the preprocessing step fails, items stay unsupported. Check error messages in "Not supported items".
  • A queue is removed: its items become lost resources and are deleted after the keep period.
  • Two queues with the same name: discovery would create conflicting keys; make the script de-duplicate.

Step 6: document and ship

Export the template (configuration.export or the UI) into Git, write a README with the macros, the trigger meanings and a runbook entry per trigger, and version the template (Template App Jobs v1). Test on one host before linking widely. For the many-hosts case, use the API to link the template in bulk.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why use a master item with dependent items here?

    One read feeds all queues, lowering load and keeping values consistent.
  2. 2. How do you give the images queue a higher depth limit?

    Context macros let one prototype carry per-object thresholds.
  3. 3. What should happen if the status file disappears?

    A nodata trigger turns silent failure into a visible problem.
  4. 4. Why avoid UserParameter keys that accept arbitrary arguments?

    Flexible arguments widen what a compromised server can execute.

Summary

  • Build custom monitoring from a master item, a dependent discovery rule and dependent prototypes.
  • Use context macros for per-object thresholds and tags for routing.
  • Test transformations and triggers offline, then plan for missing data and invalid input.
  • Export the template to version control and keep user parameters minimal.

Next: Project: Reviewing Alert Noise