Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 37 ยท Projects and Practice

Project: Monitor a Linux Web Stack

Plan monitoring for a Linux web stack with nginx, PHP-FPM and a database: what to collect, thresholds, triggers, templates, dashboards and a checklist to verify.

In this project you design monitoring for a small but realistic production service: two web servers behind a load balancer, one database server, and a public website. You will decide what to measure, how to turn measurements into useful alerts, and how to package it as templates. Zabbix cannot run on this site, so the deliverable is a written plan and a set of runnable checkers that validate your design with Python. Work through the steps in order and keep your answers in a text file.

The system

graph TD
  U["Users"] --> LB["Load balancer"]
  LB --> W1["web01<br/>nginx + PHP-FPM"]
  LB --> W2["web02<br/>nginx + PHP-FPM"]
  W1 --> DB["db01<br/>MySQL"]
  W2 --> DB
  Z["Zabbix server"] -.-> LB
  Z -.-> W1
  Z -.-> W2
  Z -.-> DB

The monitoring server watches each layer, plus the site from outside.

Step 1: write the questions first

Monitoring answers questions. For this stack:

  1. Can users load the site right now, and how fast?
  2. Is any server about to run out of CPU, memory, disk or connections?
  3. Is the database healthy and replicating (if it has a replica)?
  4. Did a certificate or a domain quietly approach its end date?
  5. When something breaks, who is told, and how soon?

Every item you create should help answer one of these. If you cannot say which question an item serves, skip it.

Step 2: choose the items

Layer Items (examples of keys and sources)
Operating system system.cpu.util, vm.memory.size[available], vfs.fs.size[/,pfree], vfs.fs.inode[/,pfree], net.if.in[eth0], system.swap.size[,pfree] through an OS template
nginx stub_status page read with the HTTP agent, then dependent items for active connections, accepted and handled, requests per second
PHP-FPM the status page /fpm-status?json: active processes, listen queue, max children reached
MySQL Threads_connected, Threads_running, Slow_queries, Innodb_buffer_pool_wait_free, replication lag, uptime
Website A web scenario from the server or a proxy outside: step 1 home page, step 2 login page, required string, response time, HTTP status
Certificates A check of the TLS expiration date (agent key web.certificate.get, or a script)
Zabbix itself The "Zabbix server health" template

Use dependent items: one HTTP request to the stub status page feeds five items.

Step 3: design triggers with care

A trigger should describe a condition that needs a human. Examples:

  • High: the site scenario fails for 3 consecutive checks.
  • Average: filesystem free below 10 percent and the predicted time to full is under 24 hours (timeleft(/web01/vfs.fs.size[/,pfused],1h,100)<1d).
  • Warning: available memory below 10 percent for 15 minutes, max(/web01/vm.memory.size[pavailable],15m)<10.
  • Average: PHP-FPM "max children reached" increased in the last hour.
  • Average: certificate expires in under 21 days.
  • Disaster: database unreachable from both web servers.

Use thresholds as macros, and hysteresis with a recovery expression so triggers do not flap: problem when avg(5m) > 90, recovery when avg(5m) < 80.

Step 4: structure as templates

TEXT
Template Stack Web
  linked: Template OS Linux by Zabbix agent
  items: nginx via HTTP agent, PHP-FPM via HTTP agent
Template Stack MySQL
  linked: Template OS Linux by Zabbix agent
  items: MySQL via agent
Template Stack Site (linked to the load balancer host)
  web scenario, certificate check

Link Template Stack Web to web01 and web02, Template Stack MySQL to db01, and Template Stack Site to a "website" host, which has the public URL as its interface.

Step 5: dashboard and alert routing

Build one dashboard with five rows: Problems (tag service=shop), site availability and response time graph, top hosts by CPU, database connections, and an SLA widget. Route actions by tag: problems tagged component:db go to the DBA group; everything else goes to the ops group. Escalate after 15 minutes if unacknowledged.

Check 1: capacity planning with a stand-in

Before enabling everything, estimate the load you add. The calculator below prices your design.

Python Estimate NVPS for the plan (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check 2: threshold sanity with hysteresis

Verify that your problem and recovery thresholds do not flap on noisy data.

Python Hysteresis against noisy data (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check 3: validate the template design

Python Template design linter (Python stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The linter reports agent.ping, because three templates define it. Keep it only in the base template.

Deliverables

  1. A list of items with interval, history and trend retention for each layer.
  2. Six to eight triggers with severity, expression (with macros) and recovery expression.
  3. The template structure and host assignment.
  4. A dashboard sketch listing the widgets in the order of importance.
  5. A short runbook line per trigger: what to check first.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which check best answers 'can users load the site?'

    Synthetic checks test the user's path rather than a server's internals.
  2. 2. Why use a recovery expression?

    Different problem and recovery thresholds add hysteresis.
  3. 3. How can one HTTP request to nginx's status page feed several items?

    A master item collects the text; dependent items extract values from it.
  4. 4. Where should the database team's problems be routed?

    Tag-based action conditions route problems to the right group.

Summary

  • Start from questions, then choose items that answer them.
  • Combine OS, application, database and synthetic website checks; use dependent items to save requests.
  • Write triggers that need action, with macros and hysteresis.
  • Package the design as small templates and route alerts by tag.

Next: Project: A Custom Discovery Script