Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 27 ยท Instrumenting Applications

Service Discovery and Relabeling

Let Prometheus find its targets automatically with file_sd, DNS, Kubernetes and cloud discovery, then rewrite labels with relabel_configs and metric_relabel_configs.

A static_configs list of host:port pairs is fine for three servers. It stops working the moment machines come and go: containers restart on new IP addresses, autoscaling adds instances at night, and somebody forgets to update the file. Service discovery (SD) lets Prometheus ask a source of truth, such as DNS, a file, Kubernetes or a cloud API, "which targets exist right now?" and keep its target list in step automatically.

Discovery only finds where things are. It hands Prometheus a list of addresses with a pile of metadata labels. What you scrape, what the targets are called and which labels end up on the stored series is decided by relabeling, a small rule language that appears in two places and trips up almost everyone at first. This lesson explains both halves, and finishes with a runnable stand-in of the relabeling engine.

Where discovery fits

flowchart LR
    SRC["Source of truth<br/>(file, DNS, Kubernetes, cloud)"] --> SD["Service discovery<br/>targets + __meta_* labels"]
    SD --> RL["relabel_configs<br/>choose and rename targets"]
    RL --> SCRAPE["Scrape each target"]
    SCRAPE --> MRL["metric_relabel_configs<br/>filter scraped series"]
    MRL --> TSDB[("Time series database")]

Targets are relabeled before the scrape; series are relabeled after it, just before storage.

Remember the order. Rules in relabel_configs work on targets before anything is scraped. Rules in metric_relabel_configs work on every scraped sample after the scrape and before it is stored.

Ways to discover targets

Pick a discovery method

static A fixed list in prometheus.yml.
file_sd JSON or YAML files that tools can rewrite.
DNS Names that resolve to many addresses.
Kubernetes Pods, services, nodes and endpoints.
1 / 2

Static

You have already seen this one. It is the simplest and needs a reload (or restart) to change:

YAML
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["web-1:9100", "web-2:9100"]
        labels:
          env: production

file_sd

file_sd_configs points at one or more files that Prometheus re-reads when they change, with no reload. Configuration management, a deploy script or a cron job writes the file; Prometheus just follows it. This is the most flexible method when your platform has no built-in integration.

YAML
scrape_configs:
  - job_name: node
    file_sd_configs:
      - files: ["/etc/prometheus/targets/*.yml"]
        refresh_interval: 5m
YAML
# /etc/prometheus/targets/web.yml
- targets: ["web-1:9100", "web-2:9100"]
  labels:
    env: production
    team: shop
- targets: ["db-1:9100"]
  labels:
    env: production
    team: data

Each group of targets shares its labels. Write the file atomically (temporary file, then rename) so Prometheus never reads half of it. The fleet monitoring project builds on this.

DNS

dns_sd_configs resolve names on a schedule. With type: A (or AAAA) every resolved IP becomes a target with a port you give; with type: SRV the records supply hostnames and ports:

YAML
scrape_configs:
  - job_name: api
    dns_sd_configs:
      - names: ["api.internal.example.com"]
        type: A
        port: 9102
        refresh_interval: 30s

DNS is a good fit for load-balanced services behind a name, and for Consul's DNS interface.

Kubernetes

kubernetes_sd_configs talk to the cluster API and discover objects by role: node, pod, service, endpoints, endpointslice or ingress. Each target comes with many __meta_kubernetes_* labels:

YAML
scrape_configs:
  - job_name: pods
    kubernetes_sd_configs:
      - role: pod

Labels include __meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_ip, one __meta_kubernetes_pod_label_<name> per Pod label and one __meta_kubernetes_pod_annotation_<name> per annotation. See Monitoring Kubernetes and Containers for the complete picture, including the Prometheus Operator, which hides most of this relabeling behind ServiceMonitor objects.

Cloud and others

There are SD mechanisms for AWS EC2, Google Compute Engine, Azure, DigitalOcean, Hetzner, OpenStack, Consul, Docker and more. They work the same way: you give credentials and a region, and each instance appears with __meta_* labels such as tags and instance types. Check the configuration reference of your Prometheus version for the exact names, because they differ per provider.

Meta labels and special labels

Labels that start with __ are special. Discovery fills in __meta_* labels, which exist only during relabeling and are removed afterwards. These special labels also steer the scrape itself:

Label Meaning
__address__ The host:port to scrape. Becomes the instance label if you do not set one
__scheme__ http or https
__metrics_path__ The path to scrape, /metrics by default
__param_<name> A URL query parameter for the scrape
job Set from job_name before relabeling

After relabel_configs finishes, every label whose name starts with __ is dropped. If you want a piece of discovery metadata to appear on your series, you must copy it to a normal label name.

relabel_configs: rewriting targets

A relabeling rule has this shape:

YAML
- source_labels: [label_a, label_b]   # which labels to read
  separator: ";"                      # joins their values (default ";")
  regex: "(.*)"                       # matched against the joined text
  target_label: some_label            # where replace writes
  replacement: "$1"                   # may use capture groups
  action: replace                     # default action

The values of source_labels are joined with the separator, then matched against regex. The regex is anchored at both ends: prod matches the whole text prod, not production. The actions you will use most:

Action What it does
replace (default) If the regex matches, writes replacement into target_label
keep Keeps the target only if the regex matches; otherwise drops it
drop Drops the target if the regex matches
labelmap Copies every label whose name matches regex to a new name built with replacement
labeldrop, labelkeep Removes labels whose names match (or keeps only those)

Three classic recipes for Kubernetes pods follow. Scrape only pods that opt in with an annotation, rewrite the address to use a port from another annotation, and turn Pod labels into series labels:

YAML
relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: "true"
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    regex: "([^:]+)(?::\\d+)?;(\\d+)"
    replacement: "$1:$2"
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    target_label: namespace

Read the second rule carefully: the joined text looks like 10.0.1.11:8080;9102, the regex captures the IP and the new port, and the replacement 10.0.1.11:9102 overwrites __address__. If the annotation is missing the joined text ends with ;, the regex does not match, and the address stays as it was. A replace whose regex does not match leaves everything unchanged.

metric_relabel_configs: filtering what you store

After the scrape, metric_relabel_configs run on each sample. The metric name is itself a label called __name__, so you can match on it. Typical uses are dropping expensive metrics that nobody uses and removing noisy labels:

YAML
scrape_configs:
  - job_name: app
    static_configs:
      - targets: ["app:9102"]
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "go_gc_.*|promhttp_metric_handler_.*"
        action: drop
      - regex: "pod_template_hash"
        action: labeldrop

This is your emergency brake for cardinality problems: if a deploy suddenly produces a flood of series, you can drop the label or metric at ingestion without touching the application. Two warnings apply. Dropping a label that was the only difference between two series makes them collide into one, and Prometheus rejects the duplicates. And relabeling cannot repair broken data: after a labeldrop the series have simply lost information for good.

A relabeling engine you can run

This is a stand-in written in plain JavaScript, not Prometheus's own code. It implements replace, keep, drop, labelmap, labeldrop and labelkeep with the same anchored-regex and joined-label rules, and applies the pod recipe above to four discovered pods. It writes digits as [0-9] so that the regex needs no escapes.

Node.js Relabeling discovered targets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Trace it: pg-0 has no prometheus.io/scrape annotation, so the keep rule removes it. dns-1 has the annotation but lives in kube-system, so the drop rule removes it. The cart pod is rewritten to port 9102 using its annotation, while search has no port annotation and keeps 8080. All __meta_* labels are gone from the final label sets, and app appeared through labelmap.

The same rules after the scrape

Now the same engine works on scraped samples, where the metric name is the label __name__. This version drops a metric family, drops a noisy label and shows what happens to series that become identical.

JS metric_relabel_configs on scraped samples

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The two /cart samples differed only by pod_template_hash. After dropping that label they have the same identity, so only one of them can be stored. In a real setup the fix is to keep the label, or aggregate with PromQL later instead of at ingestion.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which settings run BEFORE a target is scraped?

    relabel_configs rewrite or filter targets and their labels before the scrape. metric_relabel_configs run on the scraped samples afterwards, just before storage.
  2. 2. A regex in a relabeling rule is prod. The joined value is production. What happens?

    Prometheus anchors relabeling regexes at both ends, so use prod.* or production to match.
  3. 3. What happens to labels such as __meta_kubernetes_pod_name after relabel_configs finish?

    Labels starting with __ are dropped after target relabeling. Copy the ones you want to a normal label with replace or labelmap.
  4. 4. You need to stop storing a noisy metric family that already arrives from a running application, without redeploying it. What do you use?

    metric_relabel_configs see each scraped sample and can drop it by matching the metric name label. Target relabeling cannot filter individual metrics.
  5. 5. Which discovery method is best when your own tooling can write a list of targets to disk and Prometheus should follow changes without a reload?

    file_sd re-reads its files when they change, which makes it the glue for any platform without a built-in integration.

Summary

  • Service discovery supplies target addresses and __meta_* labels; static, file_sd, DNS, Kubernetes and cloud providers are the common sources.
  • relabel_configs rewrite and filter targets before the scrape; metric_relabel_configs filter and rewrite samples after it.
  • A rule joins source_labels, matches an anchored regex, then applies an action such as replace, keep, drop, labelmap or labeldrop.
  • Labels starting with __ disappear after relabeling, so copy what you want to keep; __address__ becomes instance.
  • Dropping labels can make series collide, so test relabeling on a copy before you rely on it.

Now that targets are known and clean, learn to query their data in PromQL Basics.