Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 27 ยท Monitoring Fundamentals

What Monitoring Is and Why It Matters

Learn what monitoring and observability are, how metrics, logs and traces differ, why alert on symptoms, and where Prometheus and Grafana fit in.

Software that runs on a server is never "done". Disks fill up, a dependency gets slow, a release introduces a bug that only shows under real traffic. Monitoring is the practice of collecting data about a running system so that you notice these problems quickly, ideally before your users do, and so that you can find their cause.

This tutorial teaches monitoring with two popular open-source tools: Prometheus, which collects and stores numeric data and evaluates alert rules, and Grafana, which draws that data as dashboards. First, though, this lesson explains the ideas that any monitoring tool is built on.

Why monitor at all?

Notice early Find problems before users complain.
Diagnose Narrow down where and when it broke.
Plan See growth and spot capacity limits.
Improve Measure whether a change helped.

Without data, every outage is guesswork.

1 / 2

Monitoring versus observability

Monitoring means watching things you decided in advance were worth watching: "is the website up?", "is the error rate below 1%?", "is the disk less than 80% full?". You define the questions, set thresholds, and get alerted when an answer is bad.

Observability is a property of the system: can you understand its internal state from the data it emits, including answering questions you did not think of beforehand? A system with rich, well-labelled data lets you ask "are only users in one country affected, and only on the new version?" without shipping new code. In practice the two terms overlap. Good monitoring is the part you automate, and observability is the depth you need when the automation says "something is wrong" but not why.

Metrics, logs and traces

Three kinds of data are usually collected, and each answers a different question.

  • Metrics are numbers sampled over time, such as requests per second, memory in use or the 95th-percentile latency. They are small and cheap, and they aggregate well, so they are the best fit for dashboards and alerts. They tell you that something is wrong and roughly how much.
  • Logs are timestamped records of individual events: "user 8 failed to log in", a stack trace, a slow database query. They are detailed but large, so they answer why once you know where to look.
  • Traces follow a single request through many services and record how long each step took. They answer where the time went in a distributed system. See Distributed Tracing for how traces work.

Prometheus is a metrics system. It does not store logs or traces. For those you pair it with other tools, for example Loki or Elasticsearch for logs and Jaeger or Tempo for traces. A typical investigation starts at a metric ("errors jumped at 14:02"), then moves to logs and traces for that service and time window.

Why metrics are cheap

The next stand-in records the same 500 simulated requests two ways: as one log line per request, and as a handful of counters. Real systems are far more complex, but the size difference is the reason metrics are the first signal you reach for.

JS Logs grow with traffic, metrics barely do

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The metrics answer "what is the error rate?" (1.6%) and "how many requests were slow?" (161 took over 500 ms) in 86 bytes, but they cannot tell you anything about request number 7. The log line can, at the price of roughly 52 bytes per request. Ten times the traffic means ten times the logs, while the counters stay the same size. This is also why metrics can be kept for months or years.

Symptoms versus causes

The most important monitoring idea is to separate symptoms from causes.

A symptom is something a user feels: pages fail, checkout takes ten seconds, the API returns errors. A cause is something inside the system that may produce a symptom: CPU at 90%, a full disk, a crashed worker, a slow database.

Alert on symptoms. Causes are useful as graphs for diagnosis, but a high CPU is often perfectly fine, and a real outage can happen with every cause metric green. If you page a human only when users are affected, every page is worth waking up for. Alerting on dozens of causes produces noise, and people learn to ignore noisy alerts, which is called alert fatigue and is covered in Security, Cardinality and Common Pitfalls.

What good alerting aims for

An alert is a request for a human to act. A good alert is:

  • Actionable. Someone can do something about it right now.
  • Urgent. It cannot wait until morning. Less urgent problems become tickets or dashboard items instead.
  • Precise. It fires for a real problem and stays quiet otherwise, and it says what is broken, where, and where to start looking.
  • Rare enough to be trusted.

You will build alerts like this in Alerting Rules and Alertmanager.

Where Prometheus and Grafana fit

Here is the pipeline you will build in this tutorial.

flowchart LR
    A["Your app<br/>and servers"] -->|"exposes /metrics"| B["Prometheus<br/>scrapes and stores"]
    B -->|"alert rules"| C["Alertmanager<br/>notifies people"]
    B -->|"PromQL queries"| D["Grafana<br/>dashboards"]

Applications expose numbers, Prometheus collects and stores them, Alertmanager sends alerts and Grafana draws charts.

  • Prometheus repeatedly asks your applications for their current numbers ("scraping"), stores them as time series, lets you query them with a language called PromQL and evaluates alert rules.
  • Alertmanager receives firing alerts from Prometheus, groups and deduplicates them and sends them to email, Slack or a paging service.
  • Grafana is a separate application that queries Prometheus (and many other data sources) and displays the results as dashboards.

Prometheus itself ships with a basic web page for running queries, but most teams build their dashboards in Grafana. The next lessons look at each part in turn, starting with RED, USE and the Golden Signals.

A health check you can run

Real monitoring needs servers, so the stand-in below shows the core loop in a few lines: probe a fake service 40 times, compute the error rate and latency percentile, and turn them into a status using thresholds. Prometheus does the same kind of thing at scale, with much better storage and a query language. The random numbers use a fixed seed so the output is repeatable.

JS Sample a fake service and summarize its health

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Two of 40 probes failed, a 5.0% error rate, which is above the 4% threshold in the code, so the status is DEGRADED even though the mean and p95 latencies look fine. The error rate is the symptom that matters here. Try raising the failure probability (the 0.07) and rerun to watch the status change.

What is real and what is the stand-in

Prometheus, Grafana and Alertmanager are real programs that run as servers, so none of them can run in this page. The commands and configuration in this tutorial are shown as read-only code, and each lesson includes small runnable JavaScript or Python stand-ins that reproduce how the idea works inside: sampling, counting, parsing, querying and alert state machines. The stand-ins are simplified teaching models, not the real implementation.

You can follow the whole tutorial without installing anything, but installing Prometheus is easy and worth doing. Installing Prometheus and Your First Scrape walks through it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which kind of signal is best for dashboards and alerts because it is small and aggregates well?

    Metrics are compact numbers over time, so they stay cheap however much traffic you serve.
  2. 2. What is a symptom in monitoring terms?

    Symptoms are user-visible problems. CPU and disk are possible causes, better kept as diagnostic graphs.
  3. 3. Why should you page a human only for urgent, actionable problems?

    If most pages are noise, people stop trusting them, and the real outage gets missed.
  4. 4. What does Prometheus store?

    Prometheus is a metrics system. Logs, traces and dashboards come from other tools such as Loki, Tempo and Grafana.

Summary

  • Monitoring collects data about a running system so you can detect and diagnose problems.
  • Metrics (numbers over time), logs (events) and traces (one request's path) answer different questions.
  • Alert on symptoms users feel, and keep causes as graphs for diagnosis.
  • A good alert is actionable, urgent, precise and rare.
  • Prometheus collects, stores and queries metrics; Alertmanager sends notifications; Grafana draws dashboards.

Next: RED, USE and the Golden Signals.