Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 27 ยท Production Practice

Security, Cardinality and Common Pitfalls

Secure Prometheus endpoints with TLS and auth, keep /metrics private, avoid label explosions and alert fatigue, and test rules with promtool before production.

A monitoring system that works in a demo can still fail in production in three quiet ways. It can leak information, because metrics endpoints are plain HTTP pages that anyone on the network can read. It can drown, because one careless label multiplies the number of time series until memory runs out. Or it can lose the trust of the people on call, because it pages them for things that do not matter and stays silent when something does.

This lesson covers each of these, then the small gotchas around clocks and staleness that surprise nearly everyone once, and finally how to test configuration and alert rules with promtool before they reach production. It builds on Metrics, Labels and Time Series, alerting and rate and counter math.

Prometheus, Alertmanager and promtool cannot run on this site, so the configuration and commands below are read-only. The runnable TryIts are JavaScript stand-ins that make two of the ideas visible: how fast the series count grows, and why a series that does not exist can never trigger an alert. They are models of the behaviour, not Prometheus.

The security picture in one slide

What can go wrong

Open endpoints Anyone who can reach /metrics can read it.
Admin APIs Delete and reload endpoints must stay protected.
Label explosions Unbounded labels exhaust memory.
Alert fatigue Noisy pages train people to ignore them.

Most incidents come from defaults nobody reviewed.

1 / 2

Securing endpoints

A fresh Prometheus server has no login and no encryption: the web UI, the query API and every target page are open to whoever can reach the port. The first rule is network placement: keep Prometheus, Alertmanager and exporters on a private network or behind firewall rules or Kubernetes network policies, and never publish them on the public internet.

Beyond that, Prometheus can protect its own web server with a web configuration file, passed with --web.config.file. It offers TLS and basic authentication, with passwords stored as bcrypt hashes. Exporters built with the same toolkit (node_exporter, Alertmanager and many others) accept the same file.

YAML
# web.yml
tls_server_config:
  cert_file: /etc/prometheus/tls/server.crt
  key_file: /etc/prometheus/tls/server.key
basic_auth_users:
  # bcrypt hash, for example created with: htpasswd -nBC 10 "" | tr -d ':\n'
  admin: "$2y$10$replace-with-a-real-bcrypt-hash"
Terminal
prometheus --config.file=prometheus.yml --web.config.file=web.yml
promtool check web-config web.yml

On the scraping side, each job says how to talk to a protected target, with scheme, tls_config and either basic_auth or a bearer authorization block. Read secrets from files rather than writing them into the main config.

YAML
scrape_configs:
  - job_name: orders
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/tls/ca.crt
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/secrets/orders-token
    static_configs:
      - targets: ["orders.internal:9443"]

Never set insecure_skip_verify: true outside a quick experiment: it turns TLS into encryption without identity.

Exposing /metrics safely

Application metrics are easy to publish by accident, because many frameworks serve /metrics on the same port as the public site. Safer options, roughly from best to acceptable:

  • Serve metrics on a separate internal port that the load balancer does not expose.
  • Allow /metrics only from the monitoring network in the reverse proxy, and return 404 to everyone else.
  • Require authentication when the first two are not possible.

Check what the endpoint reveals. Metric names and label values can include customer names, internal host names, query text or paths with identifiers. Treat the page as internal documentation of your architecture.

Two Prometheus flags deserve special care. --web.enable-admin-api adds endpoints that can delete series and create snapshots, and --web.enable-lifecycle allows remote reload and shutdown. Both are off by default; turn them on only behind authentication.

Cardinality: the cost of every label

Every unique combination of a metric name and label values is a separate time series, and every series costs memory, disk and query time. The number of series for one metric is the product of the number of values of each label. Five methods, six status codes and forty routes already make 1,200 series, before multiplying by the number of instances.

The classic mistake is a label whose values are unbounded: a user ID, an email address, a request ID, a full URL with identifiers, an error message. Each new value creates new series that live for as long as the retention period, even if they were used once. The rule is simple: label values must come from a small, known set. Details about individual users or requests belong in logs and traces. Histograms multiply the cost, because each label combination also gets one series per bucket plus _sum and _count.

Stand-in: counting series

This program simulates 2,000 requests and counts the distinct series produced by two designs of the same counter: one labelled with the raw request path, one with the route template. Then it multiplies out a histogram to show the effect of buckets. The traffic is generated deterministically, so the output is always the same.

JS How many series does a label create?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The raw-path version keeps growing with traffic, while the route version stays at a fixed size whatever happens. That difference is the whole lesson. As a rough guide, each active series costs on the order of a few kilobytes of memory in the Prometheus server, so a million series is a large server, and ten million is a design problem.

To find offenders on a real server, use the TSDB status page in the web UI (it lists the metrics and labels with the most series), or analyze the data directory offline:

Terminal
promtool tsdb analyze /var/lib/prometheus

To stop one bad target from hurting everyone, add limits to the scrape job. A scrape that exceeds a limit fails as a whole, and up becomes 0, so the problem is visible.

YAML
scrape_configs:
  - job_name: orders
    sample_limit: 20000        # max samples per scrape after relabeling
    label_limit: 30            # max labels per series
    body_size_limit: 10MB
    static_configs:
      - targets: ["orders.internal:9100"]

You can also drop an unwanted label or metric at ingestion with metric_relabel_configs, as shown in Service Discovery and Relabeling, but fixing the instrumentation is better than cleaning up afterwards.

Alert fatigue

An alert is a request to interrupt a human. Every alert that fires without needing action makes the next real one less likely to be taken seriously. Practical rules:

  • Alert on symptoms users feel, such as error ratio and latency, and keep cause-level signals (CPU, disk, restarts) for dashboards and tickets.
  • Every page must be actionable now. If the answer to "what do I do?" is "wait", it is a ticket or a dashboard, not a page.
  • Use for so that a single bad scrape or short spike does not page anyone, and use a window long enough to smooth noise.
  • Add a runbook link and context in annotations, and route by severity so only urgent alerts reach a phone.
  • Review regularly. Count how often each alert fired and what was done; delete or fix the ones that never led to action.

Clock and staleness gotchas

Prometheus gives each sample the timestamp of the scrape, taken from the Prometheus server's clock, so targets with wrong clocks usually do not corrupt data. Server clocks still matter: run NTP (or chrony) on the Prometheus host, because rule evaluation and alert timing follow its clock, and compare times in UTC.

The following behaviours surprise newcomers:

  • Lookback and staleness. An instant query uses the newest sample within a five-minute lookback window. When a target disappears or a series is no longer exposed, Prometheus writes a staleness marker, so the series ends cleanly rather than hanging around for five minutes.
  • Range windows need samples. rate() needs at least two samples inside the window. With a 15 second scrape interval, a [30s] window is fragile: one missed scrape leaves no result. Use a window of at least four times the scrape interval.
  • Missing series do not compare. errors_total > 0 returns nothing when no series named errors_total exists, so the alert never fires, and neither does it say anything is wrong. Initialize counters to zero when your process starts, and use absent() or up == 0 for "the thing is gone".
  • Counter resets. A restart sets a counter back to zero. rate() and increase() handle this, but raw subtraction does not.
  • Two replicas, two views. In a highly available pair, the two servers scrape at slightly different moments and can differ a little. Compare ratios, not exact values.

Stand-in: a series that does not exist cannot alert

This stand-in stores series in a map and evaluates the rule errors > 0 the way an instant query would: by looking at the series that exist. It runs the rule on two services, one of which has not registered the series yet.

JS Why a missing series stays silent

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The honest conclusion is that "no data" and "zero" are different things, and the rule must say which one it means.

Testing rules and configuration with promtool

promtool ships with Prometheus and checks files before you deploy them. Make these commands part of your CI pipeline so a typo cannot take down alerting.

Terminal
promtool check config prometheus.yml     # syntax, referenced files, rule files
promtool check rules alerts.yml          # validate rule files
promtool check web-config web.yml        # validate the web configuration
promtool test rules tests.yml            # run unit tests for rules

Unit tests feed synthetic series into your rules and assert which alerts fire. The notation 0+10x15 means "start at 0, add 10 each step, for 15 steps" (16 values), and each step is one interval.

YAML
# alerts.yml
groups:
  - name: api
    rules:
      - alert: HighErrorRate
        expr: sum by (job) (rate(http_requests_total{code=~"5.."}[5m])) > 0.05
        for: 5m
        labels:
          severity: page
        annotations:
          summary: "API 5xx rate is high"
YAML
# tests.yml
rule_files:
  - alerts.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{job="api", code="500"}'
        values: '0+10x15'          # 10 errors per minute
      - series: 'http_requests_total{job="web", code="500"}'
        values: '0+0x15'           # no errors
    alert_rule_test:
      - eval_time: 10m
        alertname: HighErrorRate
        exp_alerts:
          - exp_labels:
              severity: page
              job: api
            exp_annotations:
              summary: "API 5xx rate is high"
      - eval_time: 2m
        alertname: HighErrorRate
        exp_alerts: []             # still pending, not firing yet

The second case is just as valuable as the first: it proves the for duration holds the alert back. Add a test for every alert you would be unhappy to find broken.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which flag file lets Prometheus serve its own web UI over TLS with basic authentication?

    The web configuration file passed with --web.config.file defines tls_server_config and basic_auth_users. The main config file does not secure the web server.
  2. 2. A counter has labels method (5 values), status (6 values) and route (40 values). How many series does it create at most?

    Series multiply across labels: 5 x 6 x 40 = 1,200, and that is per target. A label with unbounded values, such as a user ID, would make the count unlimited.
  3. 3. Why does the rule errors_total > 0 stay silent for a service that never created the series?

    An instant query works on existing series. Initialize counters to zero and use absent() or up == 0 to detect a missing target.
  4. 4. What is the best response to an alert that fires often but never needs action?

    Alerts that do not lead to action cause fatigue. Make them actionable, move them to a lower severity, or remove them.

Summary

  • Keep Prometheus and exporters on private networks; add TLS and basic authentication with --web.config.file, and use tls_config and credentials files for protected targets.
  • Serve /metrics on an internal port or restrict it, and leave the admin and lifecycle APIs off unless protected.
  • Series count is the product of label values: never use unbounded values such as IDs or raw paths, and add sample_limit and label_limit as safety nets.
  • Page on symptoms, make each page actionable, use for, and review alerts that never lead to action.
  • Know the gotchas: five-minute lookback, rate windows of at least four scrape intervals, missing series that never compare, and counter resets.
  • Run promtool check and promtool test rules in CI.

Next: Project: Instrument a Web Service and Build Its Dashboard.