Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 27 ยท Prometheus Basics

How Prometheus Works

Understand the Prometheus architecture: the pull model, scrape targets, exporters, the time series database, retention, service discovery and Alertmanager.

Prometheus is a monitoring system and time series database that began at SoundCloud and is now a graduated project of the Cloud Native Computing Foundation. Before you install it, it helps to know which parts exist, what each one does and, most importantly, how data gets in. Prometheus does something unusual: it pulls.

This lesson is a map of the pieces. Later lessons zoom into each one.

The Prometheus ecosystem

Targets Apps and exporters that expose /metrics.
Prometheus server Scrapes, stores, queries, evaluates rules.
Alertmanager Routes alerts to people.
Grafana Draws dashboards from queries.
1 / 2

The pull model

Most older systems made applications push their numbers to a central server. Prometheus reverses this: every few seconds the server sends an HTTP GET request to each configured target, usually the path /metrics, and the target replies with a page of plain text containing its current values. This act is called a scrape.

Pulling has practical benefits:

  • Health for free. If a scrape fails, Prometheus knows the target is unreachable and records it (the built-in up metric is 0).
  • Easy debugging. You can open http://host:8080/metrics in a browser and see exactly what Prometheus would see.
  • Central control. The server decides what to scrape and how often, so a buggy application cannot flood it.
sequenceDiagram
    participant P as Prometheus
    participant A as Your app
    loop every scrape_interval
        P->>A: GET /metrics
        A-->>P: text with current values
        P->>P: store samples with a timestamp
    end

Prometheus keeps asking, the application only has to answer.

Pull does not suit everything. A short batch job may finish before it is scraped. For that, there is the Pushgateway, covered in Exporters: Node, Blackbox and Friends, together with the reasons to use it sparingly.

Targets and exporters

A target is anything that serves metrics in Prometheus's text format. There are two ways to get one:

  1. Instrument your own code with a client library, so the application exposes /metrics itself (Instrumenting Your Code).
  2. Run an exporter: a small program that sits next to software you cannot change, reads its internal statistics and translates them to the Prometheus format. Examples: node_exporter for Linux hosts, mysqld_exporter for MySQL, blackbox_exporter for probing URLs.

The Prometheus server

One Prometheus binary does several jobs:

  • Scraper. Fetches targets on the configured interval.
  • Time series database (TSDB). Stores everything on local disk. Each value is a sample: a timestamp and a number. Samples that share a metric name and labels form one time series. The storage format compresses these very well, and the Prometheus documentation puts it at roughly one to two bytes per sample.
  • Query engine. Evaluates PromQL (PromQL Basics) for the web UI, for Grafana and for rules.
  • Rule evaluator. Periodically runs recording rules and alerting rules (Recording Rules and Query Performance).

The server is deliberately simple: a single process, local storage and no required cluster. That makes it reliable, because it has no hard dependency on the network being healthy, but it also means storage is limited by one machine, which Scaling, Retention and Long-Term Storage addresses.

Alertmanager

Prometheus decides when an alert should be firing, based on a rule such as "error rate above 5% for 10 minutes". It does not decide who gets told. It forwards firing alerts to Alertmanager, a separate program that groups related alerts, removes duplicates, respects silences, and sends notifications to email, Slack, PagerDuty and similar services. See Alerting Rules and Alertmanager.

Service discovery

Hard-coding a list of targets works for three servers but not for a cluster where machines appear and vanish. Service discovery lets Prometheus find targets automatically from files, DNS, cloud APIs (AWS, GCP, Azure) or Kubernetes. Discovery produces a list of targets with metadata, and relabeling rules decide which to keep and what labels to attach. This is the topic of Service Discovery and Relabeling.

Retention

Local disk is finite, so Prometheus keeps data for a limited time. The default is 15 days, set with a command-line flag. Older blocks are deleted automatically. Typical launch flags look like this (read-only, as Prometheus cannot run on this page):

Terminal
prometheus \
  --config.file=prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=30d

--config.file points at the configuration, --storage.tsdb.path is where the database lives, and --storage.tsdb.retention.time is how long to keep samples. There is also --storage.tsdb.retention.size to cap disk usage.

Sizing disk from the numbers

How much disk do you need? Multiply retention time by the ingestion rate by bytes per sample. The stand-in below does the arithmetic for four scenarios, assuming 2 bytes per sample (the upper end of the usual one to two) to be conservative. The real figure depends on your data.

Node.js Estimate disk usage from retention and scrape settings

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Scraping three times as often triples the cost (every 15 s to every 5 s: 2.8 GB to 8.3 GB), and keeping a year instead of 15 days multiplies storage by about 24. That is why scrape intervals of 15 to 60 seconds are normal, and why long retention is usually handled by a separate long-term storage system.

A scrape loop you can run

The stand-in below is a tiny version of the scraper and database. Two fake targets return /metrics text; one of them is "down" between 30 and 59 seconds. The loop scrapes every 15 virtual seconds, stores samples per series, and records up as 1 or 0 for every attempt, exactly like Prometheus.

Node.js A tiny scrape loop with an up metric

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Look at web-2. While it was down, its up series shows 0 at 30 s and 45 s, and its http_requests_total series has no samples at those times: a failed scrape stores no data, only up=0. Meanwhile web-1 has a complete history. Each combination of metric name and labels is its own series: six series for two targets, three each. Prometheus also attaches the job and instance labels to every sample automatically, which is how you tell targets apart.

What is real and what is the stand-in

The pull model, the up metric, the flags, the Alertmanager split and the disk formula are real. The scrape loop above is a heavily simplified stand-in: real Prometheus uses HTTP, parses the full text format (Metrics, Labels and Time Series), writes to a compressed on-disk database with a write-ahead log, and scrapes targets concurrently with small random offsets so they do not all hit at once.

Prometheus uses its own clock for timestamps. The target does not send a time with its values, so the scrape time is what gets stored.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does Prometheus normally get metrics from an application?

    Prometheus pulls: it sends HTTP GET requests to each target on a schedule.
  2. 2. What is an exporter?

    Exporters such as node_exporter translate the statistics of other software into Prometheus's format.
  3. 3. What does a failed scrape of a target record?

    Prometheus stores up=0 for the failed attempt, so you can query and alert on unreachable targets.
  4. 4. Which component sends notifications to Slack or email?

    Prometheus evaluates alert rules, but Alertmanager groups, deduplicates and delivers the notifications.

Summary

  • Prometheus pulls metrics: it scrapes each target's /metrics endpoint on a schedule.
  • Targets are instrumented applications or exporters; each scrape also produces the up metric.
  • The server scrapes, stores samples in a local time series database, answers PromQL queries and evaluates rules.
  • Alertmanager is a separate component that routes and delivers alerts.
  • Service discovery finds targets automatically; retention (15 days by default) limits how long data is kept.

Next: Installing Prometheus and Your First Scrape.