Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 39 ยท Projects and Practice

Project: Server Monitoring Pipeline

Design a complete server monitoring setup, from Telegraf collection and schema to rollups, top-N queries, percentiles and alert rules, with runnable SQL stand-ins.

In this project you design the monitoring system for a small fleet of web and database servers, end to end. You will decide the schema, the collection config, the retention and rollup plan, the dashboard queries and the alert rules. It pulls together almost everything in this course: schema design, Telegraf, downsampling, dashboards and alerts.

InfluxDB itself cannot run in this browser course. The collection and server parts are read-only configuration. The analysis queries run as portable SQL on generated data, so you can try every step.

Requirements

  • 40 servers (30 web, 10 database) in two regions.
  • CPU, memory, disk and request latency every 10 seconds.
  • Dashboards show the last 6 hours live, and 90 days of trends.
  • Alert when CPU stays high, disk is nearly full, or a host stops reporting.

Step 1: schema

Measurement Tags Fields
cpu host, role, region usage_percent
mem host, role, region used_percent
disk host, role, region, path used_percent
http host, role, region latency_ms, requests

Estimated series: about 40 hosts x 4 measurements, with a few disk paths each: well under 1,000. Cardinality is no concern, and nothing unbounded (like a request ID) is a tag.

Step 2: collection

Every host runs Telegraf with global tags for role and region (read-only):

TOML
[global_tags]
  role = "web"
  region = "eu-west"

[agent]
  interval = "10s"
  flush_interval = "10s"
  metric_buffer_limit = 20000

[[inputs.cpu]]
  totalcpu = true
  percpu = false
  fieldpass = ["usage_user", "usage_system"]

[[inputs.mem]]
[[inputs.disk]]
  mount_points = ["/", "/var/lib/data"]

[[outputs.influxdb_v2]]
  urls = ["https://influx.example.com:8086"]
  token = "${INFLUX_WRITE_TOKEN}"
  organization = "ops"
  bucket = "raw"

Step 3: retention plan

  • raw: 7 days, 10-second points.
  • hourly: 400 days, one row per host per hour with mean, max and count (a scheduled task).

The raw volume is 40 hosts x 4 measurements x 8,640 points per day, around 1.4 million points per day: small.

Step 4: analysis queries (runnable)

We generate 6 hours of CPU data for 6 hosts with different personalities (one noisy, one hot, one that goes silent) and run the dashboard queries on it.

SQL Generate data, then top hosts and a window mean

Press Run (or Ctrl + Enter) to see the result here.

Step 5: percentiles

Averages hide tail behaviour. SQLite has no built-in percentile, but a window function ranks values so you can pick the 95th. InfluxDB 3 SQL provides approx_percentile_cont; Flux has quantile().

SQL 95th percentile with ranks, and time above a threshold

Press Run (or Ctrl + Enter) to see the result here.

Step 6: alert rules

Translate your requirements into checks (see checks and notifications):

Alert Type Rule Level
CPU high Threshold 5-minute mean above 85 crit; above 70 is warn
Disk almost full Threshold used_percent above 90 crit
Host silent Deadman No cpu points for 3 minutes crit
Latency Threshold 5-minute p95 above 500 ms warn

The SQL stand-in below finds the hosts that would fire the CPU and silence alerts at the end of the range. The "now" time is fixed so the result is repeatable.

SQL Which hosts would alert at the end of the range?

Press Run (or Ctrl + Enter) to see the result here.

Step 7: deploy and review

  1. Create buckets raw and hourly and tokens: a write-only token per collector group, a read-only token for dashboards.
  2. Roll Telegraf out with a config management tool, using --test first.
  3. Build the dashboard with variables for region and host.
  4. Create the checks, an endpoint (Slack for warn, paging for crit) and rules that notify on change.
  5. Monitor the monitor: alert if Telegraf itself stops sending or if InfluxDB's disk fills.
  6. After two weeks, review the alerts that fired. Tune thresholds, delete the noisy ones.

Extensions

  • Add a service tag for application metrics and chart error rate by service.
  • Add the Grafana dashboards for non-InfluxDB sources.
  • Estimate growth with the capacity planner from scaling.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is it fine to tag with host, role and region in this project?

    Low-cardinality, stable identifiers are the right use of tags.
  2. 2. Which alert catches a crashed server that sends nothing?

    Only a deadman check reacts to the absence of data.
  3. 3. Why also keep max and count in the hourly rollup?

    Max preserves peaks; count lets later roll-ups weight values correctly.
  4. 4. What is the benefit of separate read-only and write-only tokens?

    Each component only gets the access it needs.

Summary

  • Start from requirements, then design schema, collection, retention and alerts together.
  • Keep tags small and stable; keep raw data short-lived and rollups long-lived.
  • Use windows, percentiles and time-above-threshold for honest dashboards.
  • Combine threshold and deadman alerts, and notify on change.
  • Review and tune alerts after real use.

Next: Project: IoT Climate Analysis