Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 27 ยท Instrumenting Applications

Exporters: Node, Blackbox and Friends

Monitor servers, containers, websites and databases you cannot modify with Prometheus exporters, use the textfile collector, and know when Pushgateway is wrong.

You can add a client library to code you write. But your Linux kernel, your MySQL server, your Redis cache and the nginx in front of everything do not speak Prometheus. An exporter is a small program that sits next to such a system, asks it for its numbers in whatever way it supports, and translates them into the text exposition format on an HTTP /metrics endpoint.

From Prometheus's point of view an exporter is just another scrape target. It does not know or care that the numbers came from /proc files or a SQL query. This lesson walks through the exporters you will meet in nearly every setup, then covers two ways to get metrics from things that cannot be scraped directly: the textfile collector and the Pushgateway.

None of the real exporters can run on this site, so their commands and configs are read-only. The runnable examples are stand-ins that show the pattern every exporter follows.

How an exporter works

One exporter, one job

Ask Read the system on every scrape.
Translate Turn the answer into metrics.
Serve Return text at /metrics.
1 / 2

A well-behaved exporter collects on demand: when a scrape arrives, it reads the system's current state and builds the response. It does not poll in the background and cache the answer, because then the timestamp of the data would be wrong and a hung background loop would keep serving stale values. Many exporters also add a metric that says whether the last collection worked, such as mysql_up or node_scrape_collector_success, so a broken connection to the database shows up as a value instead of as a missing scrape.

The usual deployment rule is one exporter per machine for host metrics, and one exporter per database instance placed next to it (or on a host that can reach it). Exporters should run as a normal user with only the permissions they need: a MySQL exporter needs a monitoring account, not root.

node_exporter: the host

node_exporter reports hardware and operating system metrics for Linux and other Unix-like systems. It reads files such as /proc/stat, /proc/meminfo and the filesystem statistics from the kernel. Its name comes from "node" meaning machine, and has nothing to do with Node.js.

Run it directly or as a service, then point Prometheus at port 9100:

CONSOLE
$ ./node_exporter
$ curl -s localhost:9100/metrics | grep -E '^node_(load1|memory_MemAvailable_bytes) '
node_load1 0.42
node_memory_MemAvailable_bytes 6.231400448e+09
YAML
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ["web-1:9100", "web-2:9100", "db-1:9100"]

The metrics you will use most:

Metric Type What it tells you
node_cpu_seconds_total{cpu, mode} counter CPU time per core and mode (idle, user, system, iowait...)
node_memory_MemAvailable_bytes gauge Memory that could be given to programs
node_filesystem_avail_bytes{mountpoint} gauge Free space for unprivileged users
node_load1, node_load5, node_load15 gauge Load averages
node_network_receive_bytes_total{device} counter Bytes received per interface
node_disk_io_time_seconds_total{device} counter Time the disk was busy

CPU usage is not a metric you read directly, you compute it with PromQL from the idle counter; you will do that in rate, increase and Counter Math. Collectors can be switched on and off with flags such as --collector.systemd or --no-collector.hwmon, which also reduces the number of series.

On Windows the equivalent exporter is windows_exporter, and for macOS the same node_exporter works for a subset of metrics.

cAdvisor: containers

cAdvisor (Container Advisor) reports resource usage per container: CPU time, memory, network and block I/O. Docker has no built-in Prometheus metrics for individual containers, so cAdvisor is the usual answer on a Docker host. It runs as a container itself and mounts a few host paths read-only:

CONSOLE
$ docker run -d --name cadvisor -p 8080:8080 \
    -v /:/rootfs:ro -v /var/run:/var/run:ro \
    -v /sys:/sys:ro -v /var/lib/docker/:/var/lib/docker:ro \
    gcr.io/cadvisor/cadvisor:latest

Typical metrics are container_cpu_usage_seconds_total, container_memory_working_set_bytes and container_network_receive_bytes_total, each labelled with the container name. On Kubernetes, cAdvisor is built into the kubelet, so you scrape it from there; that is covered in Monitoring Kubernetes and Containers. Pin the image to a concrete version in real deployments instead of latest.

blackbox_exporter: probing from the outside

Most exporters look inside a system. The blackbox_exporter does the opposite: it behaves like a user and probes an endpoint over HTTP, HTTPS, TCP, ICMP or DNS, then reports whether it worked and how long it took. It answers "is my site reachable from here?", which no internal metric can fully answer.

Probes are defined as modules in the exporter's own config file:

YAML
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      preferred_ip_protocol: ip4
  tcp_connect:
    prober: tcp
    timeout: 5s

The exporter does not know what to probe in advance. Prometheus passes the target as a URL parameter, which is why the scrape config is unusual. It uses relabeling (covered in Service Discovery and Relabeling) to turn the "target" into a parameter and send the scrape to the exporter itself:

YAML
scrape_configs:
  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com
          - https://example.com/login
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Each scrape of /probe?target=https://example.com&module=http_2xx returns metrics such as probe_success (1 or 0), probe_duration_seconds, probe_http_status_code and, for HTTPS, probe_ssl_earliest_cert_expiry. That last one lets you alert two weeks before a certificate expires.

Database and service exporters

Databases and brokers each have a community or official exporter that connects with a normal client protocol and runs its status queries:

Software Exporter Needs
MySQL / MariaDB mysqld_exporter A monitoring user with PROCESS, REPLICATION CLIENT and SELECT on needed schemas
PostgreSQL postgres_exporter A connection string, ideally a read-only role
Redis redis_exporter The Redis address (and password if set)
nginx nginx-prometheus-exporter The stub_status page enabled
Kafka, RabbitMQ, MongoDB Dedicated exporters Network access to the broker

Some software now exposes Prometheus metrics itself (RabbitMQ and etcd, among others), so check whether the product has a built-in endpoint before adding another process. Pass credentials through environment variables or files, not command-line flags, because flags show up in process lists. For example, a PostgreSQL exporter commonly reads its connection string from an environment variable:

CONSOLE
$ export DATA_SOURCE_NAME='postgresql://monitor:secret@localhost:5432/postgres?sslmode=disable'
$ ./postgres_exporter

The textfile collector

Some facts do not come from a running process at all: when did last night's backup finish, how many rows did a cron job import, is a file present? A short-lived script cannot keep an HTTP endpoint open for Prometheus to scrape.

The textfile collector of node_exporter solves this. Start the exporter with a directory:

CONSOLE
$ ./node_exporter --collector.textfile.textfile.directory=/var/lib/node_exporter/textfile

Any script can write a file ending in .prom into that directory, in the text exposition format, and node_exporter includes its contents in every scrape:

Terminal
#!/bin/sh
# at the end of a backup job
tmp=/var/lib/node_exporter/textfile/backup.prom.$$
cat > "$tmp" <<EOF
# HELP backup_last_success_timestamp_seconds Unix time of the last good backup.
# TYPE backup_last_success_timestamp_seconds gauge
backup_last_success_timestamp_seconds $(date +%s)
EOF
mv "$tmp" /var/lib/node_exporter/textfile/backup.prom

Two rules matter. First, write to a temporary name and mv it into place, because a rename is atomic and a scrape never sees a half-written file. Second, record a timestamp of success instead of a status such as "ok". Then a PromQL alert on time() - backup_last_success_timestamp_seconds > 90000 catches a job that stopped running, while a stored "ok" would stay "ok" forever.

Pushgateway and when it is wrong

The Pushgateway is a small service (port 9091) that accepts metrics pushed by short-lived batch jobs and holds them so Prometheus can scrape them. It exists for jobs that finish before a scrape can reach them.

CONSOLE
$ cat <<EOF | curl --data-binary @- http://pushgateway:9091/metrics/job/nightly_import
# TYPE import_rows_total gauge
import_rows_total 48213
# TYPE import_last_success_timestamp_seconds gauge
import_last_success_timestamp_seconds 1767225600
EOF

It is the right tool for service-level batch jobs, and nothing else. It is wrong for:

  • Ordinary services. Pull-based scraping tells you automatically whether a target is up. With pushes, a dead service leaves its last values in the gateway for ever and looks perfectly healthy.
  • Many instances of the same thing. The gateway has no idea an instance disappeared, and it becomes a single point of failure and a bottleneck.
  • Per-machine jobs. Use the textfile collector for those.

The gateway never forgets: pushed values stay until someone deletes them through its API. The last stand-in shows why that is dangerous, and why you should push a timestamp.

Stand-in: a simple exporter

This stand-in is a toy exporter written in plain JavaScript, not a real one. A fake "system" has a load average that changes and a counter of network bytes. A collect() function reads that state at scrape time and renders text, and it also reports whether the collection worked, like real exporters do.

JS A toy exporter that collects on every scrape

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice how every scrape returns the state as it is now (the counter has grown by 250,000 bytes between scrapes 1 and 2), and how a failure turns into toy_scrape_success 0 instead of a missing page. Real exporters follow this recipe with proper libraries, with hundreds of metrics.

Stand-in: why the Pushgateway keeps stale data

This stand-in imitates a Pushgateway as a simple key-value store. A nightly job pushes last_success metrics, then stops running. A check based on the pushed value stays happy for ever, while a check based on the pushed timestamp notices the problem.

JS A gateway that never forgets

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The gateway also records push_time_seconds for each group itself, so you can write the age alert as time() - push_time_seconds{job="nightly_import"} > 93600 (26 hours) in real PromQL.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main job of an exporter?

    Exporters sit next to software that does not speak Prometheus, read its state and expose it as scrapeable text. Storage is Prometheus's job, alerts are Alertmanager's, and dashboards are Grafana's.
  2. 2. You want to know whether your public website answers from the outside and when its TLS certificate expires. Which exporter fits?

    The blackbox exporter probes endpoints over HTTP, TCP, ICMP or DNS and reports probe_success, durations and certificate expiry. The others report internals of a host, containers or a database.
  3. 3. A cron job on a server should publish when it last succeeded. What is the recommended way?

    The textfile collector is made for per-machine jobs. The atomic rename prevents half-written files, and a timestamp lets a query detect that the job stopped.
  4. 4. Why is the Pushgateway a poor choice for a normal long-running web service?

    With pull-based scraping, a dead target makes up become 0. A gateway keeps serving the old values until they are deleted, which makes a dead service look healthy.

Summary

  • An exporter translates the numbers of software you cannot change into the exposition format; Prometheus scrapes it like any target.
  • Collect at scrape time and report a success metric so failures are visible.
  • node_exporter covers hosts, cAdvisor covers containers, database exporters cover MySQL, PostgreSQL, Redis and more; check whether the product already exposes metrics itself.
  • blackbox_exporter probes from outside with /probe?target=...&module=... and uses relabeling to pass the target.
  • The textfile collector publishes results of cron jobs through .prom files written atomically; store timestamps, not statuses.
  • Use the Pushgateway only for service-level batch jobs, never for ordinary services.

Next, learn how Prometheus finds all these targets automatically: Service Discovery and Relabeling.