Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท Operations

Consul Telemetry, Logs and Monitoring

Monitor a Consul cluster: enable telemetry for Prometheus, watch the key Raft and Autopilot metrics, read logs with consul monitor and set practical alerts.

A Consul cluster that nobody watches fails in surprising ways: a disk slowly fills, a certificate expires, a leader flaps. Good monitoring turns those into alerts days earlier. Consul exposes three signals: metrics (telemetry), logs and health APIs. This lesson shows how to enable them and what to watch.

The product cannot run on this site. Commands and configuration are read-only; the runnable examples are stand-ins that parse metrics and evaluate alert rules.

Metrics with Prometheus

Consul can serve its metrics in Prometheus format. Enable it in the agent configuration:

HCL
telemetry {
  prometheus_retention_time = "60s"
  disable_hostname          = true
}

Then scrape the agent's HTTP endpoint:

Terminal
curl "http://127.0.0.1:8500/v1/agent/metrics?format=prometheus"

A Prometheus job targets every agent's port 8500 (or 8501 with TLS), path /v1/agent/metrics, parameter format=prometheus. If ACLs are on, the scraper needs a token with agent_prefix "" { policy = "read" }. Setting disable_hostname keeps metric names stable across nodes; the hostname is better as a Prometheus label from the scrape target.

Other sinks exist: StatsD (statsd_address), DogStatsD (dogstatsd_addr), and in-memory telemetry visible with consul monitor-style signals on SIGUSR1. Pick the one your platform already runs.

For Envoy sidecars, proxies expose their own metrics: proxy-defaults can set envoy_prometheus_bind_addr = "0.0.0.0:20200" so Prometheus scrapes request counts and latencies per service. Those give you the golden signals (rate, errors, duration) for every service in the mesh without changing application code.

The metrics that matter

Metric names use dots in StatsD and underscores in Prometheus (consul_raft_leader_lastContact).

Metric Meaning Worry when
consul.raft.leader.lastContact Time since followers last heard from the leader p99 above 200 ms regularly
consul.raft.commitTime Time to commit a write to the log Rising trend, over 50 ms
consul.raft.state.leader / candidate Leadership changes More than a few per hour means flapping
consul.autopilot.healthy 1 if all servers are healthy 0
consul.serf.member.flap / member.failed Agents joining and failing repeatedly Spikes
consul.rpc.query / consul.rpc.request RPC load Sudden jumps
consul.runtime.alloc_bytes, heap_objects, num_goroutines Process memory and goroutines Steady growth (leak)
consul.acl.blocked.* Denied ACL checks Spikes from a misconfigured app
consul.catalog.service.query Lookup rate Unexpected hot loops

Add system metrics for the servers: disk space and fsync latency, CPU and network, and certificate expiry from a blackbox exporter or file check.

Python Stand-in: parse Prometheus-format metrics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Alerts worth having

Start small. A handful of alerts catch most outages:

  1. No leader for more than 30 seconds: no write is possible.
  2. Autopilot unhealthy (consul_autopilot_healthy == 0) for 2 minutes.
  3. Leader changes: consul_raft_state_candidate increasing more than twice in 10 minutes.
  4. High commit time or last contact p99 sustained over thresholds.
  5. Server count below the planned number (consul members or the Raft peer count).
  6. Certificate expiry under 14 days (CA, server, mesh root).
  7. Disk usage over 80% on the data volume.
  8. ACL denial spikes that suggest a broken deployment or an attack.

Each alert needs a runbook: what it means, how to check and who acts.

Python Stand-in: evaluate alert rules

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Logs

Consul logs to standard output by default (and to files or syslog if configured). Set the level and format in configuration:

HCL
log_level = "INFO"
log_json  = true
log_file  = "/var/log/consul/"
log_rotate_duration  = "24h"
log_rotate_max_files = 14

INFO is the right default. Levels, from noisiest: TRACE, DEBUG, INFO, WARN, ERR. Switch a running agent to debug without restarting:

Terminal
consul monitor -log-level=debug

consul monitor streams the agent's log to your terminal until you stop it; it needs agent:read. Debug logs are verbose, so use them briefly and never leave them on in production by accident. JSON logs ship cleanly to log aggregators such as Loki, Elasticsearch or a cloud logging service.

Messages worth recognizing:

  • No cluster leader: quorum or election trouble.
  • rpc error: ... ACL not found / Permission denied: a token problem, not a network problem.
  • memberlist: Failed to send ping / suspect ... has failed: network or overloaded agent.
  • x509: certificate has expired or is not yet valid: time or certificate issues.
  • heartbeat timeout reached, starting election: leader instability.

Health endpoints and the UI

  • curl http://127.0.0.1:8500/v1/status/leader returns the leader's address; an empty string means no leader.
  • /v1/status/peers lists the Raft peers.
  • /v1/operator/autopilot/health reports each server's health (needs operator:read).
  • The web UI (enabled with ui_config { enabled = true }) shows services, nodes, KV, intentions and ACLs; it is a convenient dashboard, but an alerting system is still required.

For mesh traffic, Consul can also show service topology and metrics in the UI when a metrics provider such as Prometheus is configured with ui_config { metrics_provider = "prometheus" }.

Tracing and debugging bundles

consul debug collects a time-boxed capture of metrics, logs, goroutine profiles and host info into an archive for support or post-mortems:

Terminal
consul debug -duration=1m -interval=15s -output=consul-debug

It needs operator:read and agent:read and may include sensitive data, so check before sharing.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which setting makes an agent serve Prometheus-format metrics?

    A non-zero Prometheus retention time enables format=prometheus on /v1/agent/metrics.
  2. 2. Which metric tells you whether all servers are healthy according to Autopilot?

    consul.autopilot.healthy is 1 when every server meets the health thresholds.
  3. 3. How can you temporarily watch debug logs of a running agent?

    consul monitor streams logs at the chosen level without changing the agent's configuration.
  4. 4. An empty answer from /v1/status/leader means what?

    The endpoint returns the leader's address; empty means no leader is elected.
  5. 5. Why alert on certificate expiry?

    An expired CA or server certificate can take down the whole cluster; alert days before.

Summary

  • Enable Prometheus telemetry (or StatsD) and scrape /v1/agent/metrics; use envoy_prometheus_bind_addr for sidecar metrics.
  • Watch Raft last contact, commit time, leader changes, Autopilot health, server count, certificate expiry and disk usage.
  • Keep logs at INFO in JSON, use consul monitor for temporary debugging and consul debug for capture bundles.
  • Every alert needs a runbook.

Next: Backups, Upgrades and Troubleshooting