Skip to content
Tutorials Center
My learning

Topics

Lesson 60 of 78 ยท Operations and Observability

Metrics and Monitoring with Prometheus

Monitor Kubernetes with metrics-server, Prometheus, ServiceMonitor and Grafana. Learn counters, rates, the four golden signals and how alert rules fire.

A running cluster is a black box until you measure it. Is the checkout service slow? Is a node running out of memory? Did the last deploy increase errors? Metrics answer these questions with numbers sampled over time, cheap enough to keep for weeks and small enough to draw on a dashboard or check in an alert rule.

Kubernetes has two layers of metrics. A small built-in pipeline, metrics-server, gives you current CPU and memory so that kubectl top and the autoscalers work. For history, dashboards and alerts, almost everyone adds Prometheus, usually with Grafana on top. This lesson explains how the pieces fit together and the one idea you must understand to read any Prometheus graph: counters and rates.

The cluster commands and manifests below are read-only: this site has no cluster, so you cannot run them here. The Python examples are honest stand-ins that run in your browser and show the math Prometheus does for you.

Two pipelines, two jobs

Two metric pipelines

metrics-server Latest CPU and memory only. No history.
Prometheus Stores history and answers queries.

Autoscalers use the first. Dashboards and alerts use the second.

1 / 2

metrics-server collects resource usage from each kubelet every ~15 seconds and serves it through the aggregated API metrics.k8s.io. It keeps only the latest value in memory. It is not a monitoring system, but it is required for kubectl top and for the Horizontal Pod Autoscaler (see the HPA).

Terminal
kubectl top nodes
kubectl top pods -n shop --sort-by=memory
kubectl top pod web-6d9c7b8f5-x2k4p --containers

If kubectl top says "Metrics API not available", metrics-server is not installed or cannot reach the kubelets.

How Prometheus works

Prometheus is a pull-based time-series database. Every few seconds (the scrape interval, often 15 or 30 s), it makes an HTTP GET request to each target's /metrics path and stores what it finds. The format is plain text:

TEXT
# HELP http_requests_total Requests handled.
# TYPE http_requests_total counter
http_requests_total{method="GET",code="200"} 18423
http_requests_total{method="GET",code="500"} 31
# TYPE queue_depth gauge
queue_depth 7

Each line is a series: a metric name plus a set of labels (the same idea as Kubernetes labels) and a number. There are four metric types, and two matter most:

  • A counter only goes up (and resets to zero when the process restarts): requests served, errors, bytes sent.
  • A gauge goes up and down: queue depth, memory in use, number of connections.
  • A histogram counts observations in buckets (for example, request durations), so you can calculate percentiles.
  • A summary is an older cousin of the histogram that computes percentiles inside the application.

Prometheus does not know where your pods are. It asks the Kubernetes API through service discovery, so new pods appear in its target list automatically, and relabeling rules turn pod labels into metric labels.

Telling Prometheus what to scrape

Most clusters run the Prometheus Operator (often installed as the kube-prometheus-stack Helm chart; see Helm basics). It adds custom resources so scrape configuration lives next to your app. A ServiceMonitor selects Services by label and names the port to scrape:

YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: web
  namespace: shop
  labels:
    release: prometheus      # the operator only picks up monitors with its label
spec:
  selector:
    matchLabels:
      app: web
  endpoints:
    - port: http-metrics     # a named port on the Service
      path: /metrics
      interval: 30s

A common mistake is a ServiceMonitor that selects nothing: the label in selector does not match the Service, or the port name does not match. Then the target simply never appears in the Prometheus UI under Status, Targets.

Counters and rates

A raw counter value is rarely useful: "18,423 requests since the process started" says nothing about how busy the service is now. What you want is the rate: requests per second over a recent window. In PromQL, the query language, that is:

TEXT
rate(http_requests_total[5m])

rate takes the samples inside the last five minutes, adds up the increases and divides by the time covered. It also handles counter resets. When a pod restarts, its counter drops to 0; a naive "last minus first" would give a negative or tiny number. Prometheus assumes that a decrease means a reset and treats the new value as the increase since zero.

The stand-in below does the same calculation on samples scraped every 15 seconds. The service restarts at second 45.

Python Counter increase and rate with a reset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The reset-aware rate is 3.6 requests per second, while the naive version reports 0.67, a number that hides the traffic completely. The real PromQL rate also extrapolates to the window edges, so exact values can differ slightly, but the reset logic is the same idea.

Always apply rate() (or increase()) to a counter before graphing or alerting on it, and always aggregate after: sum(rate(...)) by (code). Taking rate of a sum breaks the reset handling.

The four golden signals

With hundreds of metrics, what should you watch first? Google's SRE book names four golden signals that describe the health of almost any request-driven service.

The four golden signals

Traffic How many requests per second.
Errors What share of requests fail.
Latency How long requests take, at p95 or p99.
Saturation How full the resources are.

If these four look healthy, users are probably happy.

1 / 2

Averages hide pain: a service where 5% of requests take ten seconds can still have a fine mean. That is why latency is reported as percentiles: p95 means "95% of requests were faster than this". The next stand-in computes the signals from a list of requests and shows the gap between mean and p95.

Python Golden signals from raw requests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The mean looks like a mild 107 ms, yet the p95 shows 900 ms: one request in twenty is slow. The fourth signal, saturation, comes from gauges such as CPU use compared with the limit, memory working set or queue length, and on Kubernetes also from container CPU throttling.

In a real cluster you get node and pod metrics from kube-state-metrics (the state of objects: replicas wanted versus available, pod phases, restarts) and node-exporter (CPU, memory, disk and network of each machine). The kubelet's built-in cAdvisor exposes container usage. Your own application adds the traffic, error and latency metrics through a client library.

Dashboards and alerts

Grafana reads from Prometheus and draws graphs. You rarely build everything yourself: kube-prometheus-stack ships dashboards for nodes, namespaces, pods and the control plane. Dashboards are for investigating; alerts are for waking someone up.

An alert is a PromQL expression plus a duration, stored in a PrometheusRule:

YAML
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: web-alerts
  namespace: shop
  labels:
    release: prometheus
spec:
  groups:
    - name: web
      rules:
        - alert: HighErrorRatio
          expr: |
            sum(rate(http_requests_total{code=~"5.."}[5m]))
              / sum(rate(http_requests_total[5m])) > 0.05
          for: 10m
          labels:
            severity: page
          annotations:
            summary: "More than 5% of requests are failing"

The for: 10m matters. The condition must be true on every evaluation for ten minutes before the alert fires; until then it is pending. This stops a one-minute blip from paging someone. Fired alerts go to Alertmanager, which groups, deduplicates, silences and routes them to Slack, email or a paging service.

Python An alert rule with a for duration

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Minute 3 is above the threshold, but the next minute drops back, so the counter resets and nothing fires. The sustained run from minute 5 onward becomes pending for two minutes and then FIRING at minute 7. At minute 12 the condition clears and the alert resolves.

Good alerts page on symptoms users feel (error ratio, latency) rather than on causes (CPU is at 80%), and every page should have a clear action. Put cause-level signals on dashboards.

What is real and what is a stand-in

Everything in the YAML and the kubectl top commands is how real clusters are set up, but none of it runs on this page. The Python examples implement the same arithmetic as rate(), nearest-rank percentiles and the for clause. Real Prometheus also estimates histogram percentiles from buckets with histogram_quantile, since it never sees individual requests.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does metrics-server provide?

    metrics-server keeps only recent resource usage in memory and serves it via the metrics.k8s.io API. It is not a monitoring database.
  2. 2. Why do you apply rate() to a counter instead of graphing it directly?

    A raw counter says how many events happened since the process started. rate() converts it to events per second over a window and treats a drop as a restart, not as negative traffic.
  3. 3. A service has a mean latency of 100 ms but a p95 of 2 seconds. What does this tell you?

    Percentiles reveal the slow tail that averages hide. 95% of requests were faster than 2 seconds, so up to 5% were slower.
  4. 4. What does the for: 10m setting do in an alerting rule?

    The alert is pending while the condition holds and fires only after it has held for the whole duration, which avoids paging on short blips.

Summary

  • metrics-server gives current CPU and memory for kubectl top and autoscaling; Prometheus stores history for dashboards and alerts.
  • Prometheus pulls /metrics from targets found through Kubernetes service discovery; a ServiceMonitor selects Services by label.
  • Counters only grow: use rate() or increase(), which handle resets when pods restart.
  • Watch the four golden signals: traffic, errors, latency (as percentiles) and saturation.
  • Alert on symptoms with a for duration, and let Alertmanager group and route notifications.

Next: Tracing and Service Meshes, where you follow one request across many services.