Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 37 ยท Triggers and Alerts

Zabbix Trigger Expressions: Functions, Time Windows, nodata and Operators

Write Zabbix 7.0 trigger expressions: item references, last(), avg(), min(), max(), count(), nodata(), find(), forecast(), time windows, #N values, logic operators and macros.

Collecting data is only half of monitoring. A trigger turns data into a decision: it is a logical expression that Zabbix evaluates every time a value it depends on changes, and that can be either OK or PROBLEM. When a trigger goes to PROBLEM, Zabbix creates an event and shows a problem on the dashboard; actions can then notify people.

The syntax in this lesson is that of Zabbix 6.0 and 7.0. (Versions before 5.4 used a different format such as {host:key.last()}>5; you will still see it in old blog posts.) Zabbix does not run on this site, so the expressions are shown read-only, and the TryIt blocks are Python stand-ins that implement the same functions.

Anatomy of an expression

TEXT
last(/web-01/system.cpu.load[all,avg1])>5
  • last is a function.
  • /web-01/system.cpu.load[all,avg1] is the item reference: /host name/item key.
  • >5 is a comparison with a constant. The whole expression is true (PROBLEM) or false (OK).

In a template you cannot name the host, so write // for "the host this template is linked to": last(//system.cpu.load[all,avg1])>5.

The time or count argument

Most functions take a second argument saying which values to use:

  • A time period: 5m, 1h, 1d: the values collected in the last 5 minutes.
  • A number of values with a hash: #3: the last three values, whatever their age.
  • An optional time shift: 5m:now-1h means the 5 minutes before one hour ago (for comparing with the past).

Use #N when you want "N consecutive checks" (useful for passive checks with fixed intervals), and a time period when the update interval may change.

The functions you will use daily

Function Meaning Example
last(/h/k) Most recent value (last(/h/k,#2) = second most recent) last(/h/agent.ping)=0
avg(/h/k,5m) Average over the period avg(/h/system.cpu.util,5m)>90
min(/h/k,5m) Minimum: true only if all values exceed a limit when used with > min(/h/system.cpu.util,5m)>90
max(/h/k,5m) Maximum: any value in the period max(/h/net.if.errors,5m)>0
count(/h/k,10m,"gt",90) How many values meet a condition count(/h/k,10m,"gt",90)>=5
change(/h/k) Difference between the last and previous value change(/h/system.hostname)<>0
nodata(/h/k,5m) 1 if no values arrived in the period nodata(/h/k,5m)=1
find(/h/k,,"like","error") 1 if a recent value matches text log items
diff(/h/k) 1 if last and previous differ version changes
delta(/h/k,1h) max - min in the period
forecast(/h/k,1h,1d) Predicted value 1 day ahead using the last hour disk filling
timeleft(/h/k,1h,0) Seconds until the item reaches 0 "disk full in N seconds"
trendavg(/h/k,1M:now/M) Average from hourly trends long-term baselines
fuzzytime(/h/system.localtime,60) 0 if the clock differs by more than 60 s clock drift

Also useful: bitand, date, dayofweek, time, now and percentile.

Choosing between last, avg, min and max

A raw last()>90 on CPU fires on a one-second spike, so it generates noise. avg(5m)>90 smooths it. min(5m)>90 is stricter: it only fires when every value for 5 minutes was above 90, a good "sustained" condition. max() is the opposite: use it with > when one bad value is enough (errors) or with < to require that every value was below something.

Python Stand-in: last, avg, min, max

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

nodata: detecting silence

nodata(/h/k,5m)=1 is true when no value has arrived for 5 minutes. It is the way to detect a dead agent, a stuck job or a script that stopped reporting; it works for trapper and active items too. Choose a period at least a few times longer than the item's interval, otherwise a single delay triggers it. A good safety net is nodata on agent.ping or system.uptime for every host. Beware of a flood: when a proxy or the network drops, nodata triggers can fire for hundreds of items at once, so use dependencies to suppress the noise.

Python Stand-in: nodata

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Operators and combining conditions

  • Arithmetic: + - * /
  • Comparison: < > <= >= = <> (note: = for equal, <> for not equal)
  • Logic: and, or, not (lowercase; and binds tighter than or, so use parentheses)
TEXT
avg(//system.cpu.util,5m)>{$CPU.CRIT} and last(//agent.ping)=1

This fires only if the CPU is high and the agent is responding (so you do not alert on stale data). You can mix several items, even from different hosts: last(/db-01/replica.lag)>60 or last(/db-02/replica.lag)>60.

Macros in expressions

Use user macros for thresholds so you can change them per host without editing the template: avg(//system.cpu.util,5m)>{$CPU.UTIL.CRIT} with {$CPU.UTIL.CRIT} set to 90 on the template and 98 on one noisy host. With a context, {$FS.FREE.MIN.CRIT:"{#FSNAME}"} gives each discovered filesystem its own limit. See hosts and macros.

Prediction triggers

timeleft(//vfs.fs.size[/,pused],6h,100)<86400 is true if, based on the last 6 hours, the disk will reach 100% within a day. Such trigger types give you time to act instead of reporting an already full disk. forecast and timeleft need enough history and work best with items that change roughly linearly.

Python Stand-in: when will the disk fill?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The trigger form

In Data collection > Hosts (or Templates) > Triggers > Create trigger: Name (supports macros, such as High CPU on {HOST.NAME}), Severity, Expression (use the Add helper to build it and the Expression constructor to combine conditions), OK event generation, Recovery, Tags, Allow manual close, Dependencies. More in severity and recovery. The Test button on the expression evaluates it against current data.

Common mistakes

  1. Using last() on noisy data: use avg, min or #N windows.
  2. A nodata period shorter than or close to the item interval.
  3. Forgetting that an unsupported item produces no value: the trigger keeps its old state. Add an internal trigger for unsupported items.
  4. Trigger on free memory instead of available.
  5. Quoting: " inside strings must be escaped; text constants are quoted, numbers are not.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the correct way to reference an item in a template trigger?

    In templates // stands for the host the template is linked to.
  2. 2. Which expression fires only when every CPU value in the last 5 minutes exceeded 90?

    min() > 90 means even the smallest value was above 90.
  3. 3. What does nodata(/h/k,5m)=1 detect?

    nodata returns 1 when no new value arrived within the period.
  4. 4. What does #3 mean as a function argument?

    #N selects the last N values regardless of their age.

Summary

  • A trigger is a logical expression: function(/host/key,period) compared with a limit.
  • Use // in templates; windows are time periods (5m) or value counts (#3).
  • avg, min, max and count filter out noise; nodata detects silence; timeleft and forecast predict.
  • Combine with and, or, not; put thresholds in macros.
  • Test expressions and avoid last() on noisy data.

Next: Severity, recovery and hysteresis