Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 27 ยท Production Practice

Monitoring Kubernetes and Containers

Monitor a Kubernetes cluster with Prometheus: kube-state-metrics, cAdvisor, pod discovery, ServiceMonitor and the Prometheus Operator, plus resource queries.

Kubernetes changes the monitoring problem in one important way: nothing stays where you put it. Pods are created, killed and rescheduled all day, their IP addresses change, and the question "is my application healthy?" now includes questions such as "are all replicas running?", "is a container restarting?" and "is it being throttled by its CPU limit?". A fixed list of scrape targets cannot keep up, so Prometheus asks the Kubernetes API who is running right now.

This lesson shows the pieces you need: the sources of metrics in a cluster, how Prometheus discovers pods, how the Prometheus Operator turns scrape configuration into Kubernetes objects, and the queries behind a useful resource dashboard. You should know service discovery and relabeling, the exporter idea and PromQL joins.

A Kubernetes cluster cannot run on this site, so every manifest, command and query against real metrics is read-only. The runnable TryIts are small JavaScript stand-ins: one applies discovery rules to a list of pods, and one joins usage with requests the way a PromQL query does. They model the logic and say so; they are not Kubernetes or Prometheus.

Four sources of metrics

Where cluster metrics come from

cAdvisor CPU, memory, network and disk use per container.
kube-state-metrics What the API says should exist, and its status.
node_exporter The machines under the pods.
Your apps Their own /metrics endpoints.

Each source answers a different question. You need all four.

1 / 2

cAdvisor is built into the kubelet on every node. It measures containers, and Prometheus scrapes it from the kubelet's /metrics/cadvisor path. The series you will use most are container_cpu_usage_seconds_total (a counter of CPU time), container_memory_working_set_bytes (the memory the kernel counts against the container's limit) and container_cpu_cfs_throttled_periods_total (how often the container was held back by its CPU limit).

kube-state-metrics is a small separate Deployment. It does not measure anything on the nodes. It watches the Kubernetes API and turns object state into metrics: kube_deployment_spec_replicas is what you asked for, kube_deployment_status_replicas_available is what you got, kube_pod_container_status_restarts_total counts restarts, and kube_pod_container_resource_requests holds the CPU and memory requests from your manifests.

node_exporter runs as a DaemonSet (one pod per node) and reports the host's CPU, memory, disk and network, as in Exporters. Do not confuse any of these with metrics-server: that component feeds kubectl top and autoscaling with short-lived numbers and keeps no history, so it is not a replacement for Prometheus.

Letting Prometheus discover pods

Inside the cluster, Prometheus uses kubernetes_sd_configs. Each role lists a different kind of object: node, pod, service, endpoints, endpointslice or ingress. For every discovered object Prometheus creates a target with many __meta_kubernetes_* labels (namespace, pod name, pod IP, container port, every label and annotation), and your relabel_configs decide which to keep and how to name the result.

A widely used convention lets teams opt in with annotations on their pods. It is only a convention: Prometheus does not know the annotation names, your relabeling gives them meaning.

YAML
scrape_configs:
  - job_name: kubernetes-pods
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # keep only pods that opted in
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      # honour a custom metrics path
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      # build the address from the pod IP and the annotated port
      - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (.+);(.+)
        replacement: $1:$2
        target_label: __address__
      # copy useful metadata onto every series
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

Prometheus itself needs permission to ask the API. That is a ServiceAccount plus a ClusterRole that can get, list and watch nodes, services, endpoints, pods and endpoint slices, and may read the non-resource URL /metrics.

Stand-in: discovery rules on a list of pods

This program takes a list of pods like the ones the API returns and applies the same decisions as the config above: keep opted-in pods, default the path to /metrics, build IP:port and attach the namespace and pod labels. Real relabeling is regex based and much more general; the stand-in keeps only the steps used here.

JS Which pods become scrape targets?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Pods that start or stop change the list at the next discovery refresh, with no edit to prometheus.yml. Notice the last pod: a half-finished annotation silently produces no target, which is why you should watch the Targets page and the up metric after changing conventions.

The Prometheus Operator

Writing and reloading scrape configuration by hand does not scale across many teams. The Prometheus Operator moves the configuration into Kubernetes objects (custom resources) and runs a controller that generates the real configuration for you. The main ones are:

  • Prometheus and Alertmanager: describe a server and its replicas, retention and storage.
  • ServiceMonitor: scrape the pods behind Services that match a label selector.
  • PodMonitor: scrape pods directly, without needing a Service.
  • PrometheusRule: alerting and recording rules as an object.
  • Probe and ScrapeConfig: blackbox-style probes and raw scrape settings.

The usual way to install all of this is the community Helm chart kube-prometheus-stack, which also deploys Grafana, kube-state-metrics, node_exporter and a large set of ready-made rules and dashboards.

Terminal
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

An application team then ships a ServiceMonitor with its own manifests:

YAML
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cart
  namespace: shop
  labels:
    release: monitoring        # must match the Prometheus object's selector
spec:
  selector:
    matchLabels:
      app: cart                # selects the Service with this label
  endpoints:
    - port: metrics            # the NAME of the Service port, not the number
      interval: 30s
      path: /metrics

Two details cause most "my target does not appear" problems. The ServiceMonitor must carry the label that the Prometheus object's serviceMonitorSelector looks for (with the Helm chart, the release name by default). And port refers to the name of a port in the Service, so the Service must define name: metrics on it. Rules follow the same pattern:

YAML
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cart-alerts
  namespace: shop
  labels:
    release: monitoring
spec:
  groups:
    - name: cart
      rules:
        - alert: CartReplicasMissing
          expr: kube_deployment_status_replicas_available{namespace="shop", deployment="cart"} < kube_deployment_spec_replicas{namespace="shop", deployment="cart"}
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Cart has fewer ready replicas than requested"

Queries for a resource dashboard

These are read-only PromQL expressions against the metrics above. Filter container!="" to drop the per-pod summary series that cAdvisor also exports, and container!="POD" to drop the pause container.

PROMQL
# CPU cores used per pod
sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total{container!="", container!="POD"}[5m])
)

# Memory in use per pod
sum by (namespace, pod) (
  container_memory_working_set_bytes{container!="", container!="POD"}
)

# CPU used compared with what the pod requested
sum by (namespace, pod) (rate(container_cpu_usage_seconds_total{container!=""}[5m]))
/ on (namespace, pod)
sum by (namespace, pod) (kube_pod_container_resource_requests{resource="cpu"})

# Containers restarted in the last hour
increase(kube_pod_container_status_restarts_total[1h]) > 0

# Share of CPU periods that were throttled
sum by (namespace, pod) (rate(container_cpu_cfs_throttled_periods_total[5m]))
/ sum by (namespace, pod) (rate(container_cpu_cfs_periods_total[5m]))

Memory deserves care: reaching a memory limit gets the container killed (OOMKilled) rather than slowed, so alert on working set against the limit, while CPU only slows down, so look at throttling. Group dashboards by namespace and workload, and use the variable techniques from Grafana basics.

Stand-in: usage divided by requests

The third query above is a join: two sets of series are matched on their namespace and pod labels and divided. This stand-in does the same matching with a Map, so you can see which series are paired and which fall out.

JS Join usage with requests

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A pod without a request, like the debug pod, disappears from the result of a one-to-one join. That is correct PromQL behaviour, and a reason to give every workload explicit requests.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which component turns the state of Kubernetes objects, such as desired and available replicas, into metrics?

    kube-state-metrics watches the API and exposes object state. cAdvisor measures container resource use, node_exporter measures hosts, and metrics-server only serves short-lived numbers to kubectl top and autoscaling.
  2. 2. A ServiceMonitor is created but its targets never show up in Prometheus. Which cause is common?

    The Prometheus object only picks up ServiceMonitors that match its serviceMonitorSelector, often a release label. The endpoint port must also be the Service port's name.
  3. 3. Why do the resource queries filter with container!=""?

    cAdvisor also reports a pod-level series with an empty container label. Summing it together with the per-container series would double count.
  4. 4. What does the annotation prometheus.io/scrape mean to Prometheus itself?

    Prometheus exposes annotations as meta labels. A keep rule in relabel_configs is what makes the annotation opt a pod in.

Summary

  • Four sources cover a cluster: cAdvisor (containers), kube-state-metrics (object state), node_exporter (hosts) and your applications.
  • kubernetes_sd_configs discovers pods, services and nodes; relabeling decides what to keep and which labels to attach.
  • The Prometheus Operator turns scrape and rule configuration into ServiceMonitor, PodMonitor and PrometheusRule objects; kube-prometheus-stack installs the whole set.
  • Match the selector label and use the Service port name when a ServiceMonitor shows no targets.
  • Watch CPU throttling, memory against limits, restarts and desired versus available replicas, and give every workload requests.

Next: Security, Cardinality and Common Pitfalls.