Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 37 ยท Integration and Operations

Backup, Upgrade and Troubleshooting

Back up Zabbix configuration and database, plan version upgrades with proxies and agents, and diagnose unsupported items, queue growth and log errors step by step.

Operating a monitoring system means keeping it alive: backups you can restore, upgrades that do not lose data, and a method for the day something stops working. This lesson gives checklists and diagnostic routines. The products are not runnable on this site, so commands are read-only examples; the Python stand-ins practice the log and queue analysis.

What to back up

  1. The database: the essential piece. Configuration tables are small and priceless; history and trend tables are huge and sometimes optional.
  2. Configuration files: zabbix_server.conf, zabbix_proxy.conf, zabbix_agentd.conf, the web frontend zabbix.conf.php, web server and PHP configuration, TLS keys and PSK files.
  3. Scripts: AlertScriptsPath, ExternalScripts, user parameter files.
  4. Templates as code: configuration.export output stored in Git.

A MySQL example that dumps everything except the large history tables, plus a separate full dump:

Terminal
# Configuration only (read-only example)
mysqldump --single-transaction --routines --triggers \
  --ignore-table=zabbix.history --ignore-table=zabbix.history_uint \
  --ignore-table=zabbix.history_str --ignore-table=zabbix.history_text \
  --ignore-table=zabbix.history_log --ignore-table=zabbix.trends \
  --ignore-table=zabbix.trends_uint zabbix | gzip > zabbix-config.sql.gz

For PostgreSQL, use pg_dump with --exclude-table-data for the same tables, or physical backups with WAL archiving for large databases. Practice the restore on a separate machine at least once; an untested backup is only a hope.

Because history is large, physical backups (Percona XtraBackup, snapshots, pgBackRest) are often used for the full database, while logical dumps cover configuration daily.

Upgrade planning

Zabbix has LTS releases (such as 6.0 and 7.0) with five years of support, and short-lived standard releases. The usual path is LTS to the next LTS. Rules:

  • Read the upgrade notes and "known issues" for your exact path. Skipping major versions is generally supported (for example 5.0 to 6.0 to 7.0 directly from LTS to LTS), but check the manual.
  • Upgrade order: server first, then proxies, then agents. A server supports proxies of the same major version, and the previous LTS in compatibility tables; agents are backward compatible with older servers and newer servers with older agents within a supported range.
  • Back up the database before starting, stop the server, install new packages, start the server once; it upgrades the database schema automatically and logs progress. Large databases can take hours, so schedule a window.
  • Upgrade the frontend packages with the server; clear the PHP opcache.
  • Check the release notes for changed defaults, removed features (screens, old maps, legacy functions) and new requirements (PHP, database versions).
  • Test on a clone of production data first.

Agents and proxies can be rolled out gradually, because they are compatible across a version window. Use Ansible or your package manager and monitor the "agent version" (agent.version) item to see progress.

Troubleshooting method

Follow the data path from target to screen, and stop at the first broken step:

  1. Is the item supported? In Data collection > Hosts > Items, filter by state "Not supported". The error message (red icon) names the cause: Timeout while executing a shell script, Cannot connect, Unsupported item key, Value of type "string" is not suitable for value type "Numeric (unsigned)".
  2. Can the server reach the agent? From the server host: zabbix_get -s 10.0.0.11 -p 10050 -k agent.ping for passive checks. For SNMP: snmpwalk -v2c -c community host OID.
  3. Is the agent configured correctly? On the target: zabbix_agentd -t "vfs.fs.size[/,pfree]" tests a key locally; check Server=, ServerActive=, Hostname= and the agent log /var/log/zabbix/zabbix_agentd.log.
  4. Does data arrive? In Latest data, check the time of the last value. For active checks watch the server log for "no active checks on server".
  5. Is the server overloaded? Open the Queue (Administration > Queue): items delayed by more than a few minutes signal busy pollers, slow targets or network problems. Check internal items (poller busy, cache free, DB sync).
  6. Is the trigger right? Test the expression in the trigger editor ("Expression constructor"), look at the event details, and remember function windows (last(/host/key,#3) needs three values).
  7. Did the notification go out? Reports > Action log shows each attempt, status and error text from the media type.

Increase debug level temporarily with runtime control, for example zabbix_server -R log_level_increase=poller (and log_level_decrease afterwards); permanent debug level 4 or 5 fills disks fast.

Typical problems and causes

Symptom Likely cause
Item "not supported" after template link Key not available on this OS, missing user parameter, timeout
Active agent never reports Hostname mismatch, firewall on 10051, ServerActive missing
Large queue Too few pollers, slow SNMP devices, unreachable hosts
"Cache is full" warnings CacheSize or ValueCacheSize too small for the item count
Frontend slow Database slow, PHP max_execution_time, too many widgets with short refresh
Triggers flapping No hysteresis; use max(/h/k,5m) or recovery expressions
Disk full on database Housekeeper behind, history retention too long (see database lesson)

For Nagios, the equivalents are nagios -v (config check), tail -f /usr/local/nagios/var/nagios.log, the "Process Info" page, check_nrpe -H host -c check_load run by hand, and checking that check_external_commands=1 and the command file permissions allow passive results.

Stand-in: analyze a server log

This runs on a sample log text. It counts messages by process and level and surfaces the repeated errors that deserve attention.

Python Summarize a log (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: queue age check

The queue page groups delayed items by age (more than 5 seconds, 10 seconds, 30 seconds, 1 minute, 5 minutes, 10 minutes). This checks a set of delays against it.

Python Queue buckets (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In what order should you upgrade Zabbix components?

    The server goes first; proxies and agents follow within the supported version window.
  2. 2. Which tool tests a passive check from the server side?

    zabbix_get asks an agent for one key value.
  3. 3. What does a large queue usually indicate?

    Items delayed beyond their interval point to collection problems.
  4. 4. What is the most important property of a backup?

    Only a tested restore proves that a backup works.

Summary

  • Back up the database, configuration files, scripts and template exports; test restoring.
  • Upgrade LTS to LTS, server first then proxies and agents, with a database backup and a staging test.
  • Troubleshoot along the data path: item state, reachability, agent config, data arrival, queue, trigger, notification.
  • Use runtime log level control instead of permanent debug logging.

Next: Project: Monitor a Linux Web Stack