Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Operations and Monitoring

Monitoring and Prometheus

Monitor RabbitMQ with the rabbitmq_prometheus plugin: key metrics for queues, consumers, memory, disk and connections, scrape setup, Grafana dashboards and practical alert rules.

A queue-based system fails quietly. Nothing crashes; a backlog just grows, consumers stall, or a disk fills, and by the time a user complains the problem is hours old. Monitoring turns those silent states into numbers and alerts. RabbitMQ has first-class support for Prometheus (a metrics database) and Grafana (dashboards), and the official plugin exposes everything you need without polling the heavy management API.

RabbitMQ and Prometheus cannot run on this page. The configuration and queries below are read-only. The runnable blocks are plain Python stand-ins that parse Prometheus text and evaluate alert rules, using sample data, so you can practice the logic behind dashboards.

Enable the metrics endpoint

Text
rabbitmq-plugins enable rabbitmq_prometheus

Metrics are then served as plain text at http://node:15692/metrics. Add a scrape job to prometheus.yml:

YAML
scrape_configs:
  - job_name: rabbitmq
    scrape_interval: 15s
    static_configs:
      - targets: ["rabbit1:15692", "rabbit2:15692", "rabbit3:15692"]

Scrape every node: each reports its own view, and an aggregated endpoint (/metrics) is cheap, while /metrics/per-object (queue and channel level labels) is expensive on big brokers, so enable it only if needed. Official Grafana dashboards (RabbitMQ Overview, Erlang distribution, RabbitMQ Quorum Queues Raft) can be imported by id from the dashboard library or from the plugin's documentation page.

The format of the metrics

Prometheus text format is one sample per line: name{label="value"} number, with # HELP and # TYPE comment lines. Counters only go up (use rate() in queries); gauges go up and down.

Python Stand-in: parse a Prometheus scrape

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Which metrics matter

Four golden questions

Is work piling up? Ready messages and age.
Is it being processed? Consumers, acks, unacked.
Is the node healthy? Memory, disk, file descriptors.
Are clients happy? Connections, blocked, churn.
1 / 2
Metric Type Why it matters
rabbitmq_queue_messages_ready gauge Backlog waiting for consumers. Growth means consumers are too slow or absent.
rabbitmq_queue_messages_unacked gauge In flight. A large, flat number suggests stuck consumers or missing acks.
rabbitmq_consumers gauge Zero consumers on a busy queue is an outage.
rabbitmq_channel_messages_published_total counter Incoming rate; use rate().
rabbitmq_channel_messages_acked_total counter Completed work rate; compare with publish rate.
rabbitmq_connections, rabbitmq_channels gauge Leaks show as steady growth; churn as spikes in opened/closed counters.
rabbitmq_process_resident_memory_bytes gauge Compare to the memory high watermark (see Flow Control and Resource Alarms).
rabbitmq_disk_space_available_bytes gauge Compare to disk_free_limit.
rabbitmq_process_open_fds / rabbitmq_process_max_fds gauge File descriptor exhaustion refuses new connections.
rabbitmq_raft_term_total counter Frequent increases mean unstable Raft leader elections.

Metric names can differ slightly across versions, so check the /metrics output of your own broker before writing alerts.

Useful queries and alerts

Text
# backlog growth over 10 minutes, per node
deriv(rabbitmq_queue_messages_ready[10m])

# in versus out
sum(rate(rabbitmq_channel_messages_published_total[5m]))
  - sum(rate(rabbitmq_channel_messages_acked_total[5m]))

# queues with messages and no consumers (requires per-object metrics)
rabbitmq_detailed_queue_messages_ready > 0 and rabbitmq_detailed_queue_consumers == 0

Alert rules that earn their keep:

  • Backlog not draining: ready messages above a threshold for 10 minutes and rising.
  • No consumers on a queue that should have some, for 2 minutes.
  • Unacked stuck high while the ack rate is zero.
  • Dead letter queue non-empty (someone must look; see Dead Letter Exchanges and Retries).
  • Memory above 80% of the watermark, disk free below 2x the limit, file descriptors above 80%.
  • Node down / cluster size less than expected, and partition detected.
  • Connection churn above normal, which hints at misbehaving clients opening a connection per message.

Alert on symptoms with a duration instead of instantaneous spikes: a queue with 10,000 messages for five seconds is a burst, and for ten minutes it is an incident.

Python Stand-in: evaluate alert rules over samples

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Logs and other signals

Metrics tell you what; logs tell you why. Watch for connection errors, PRECONDITION_FAILED (acks or queue argument mismatches), consumer timeouts, partition and Raft messages. Ship logs with your usual stack (journald, Loki, ELK). RabbitMQ writes to a file or to stdout in containers (log.console = true, optionally JSON with log.console.formatter = json). For deeper debugging use rabbitmq-diagnostics memory_breakdown, rabbitmq-diagnostics check_local_alarms and, in a pinch, the management UI's per-queue charts (see Management UI, HTTP API and CLI).

Add application metrics too: your own producer and consumer code should count messages handled and failed and measure end-to-end latency by putting a timestamp in the message. Broker numbers alone cannot tell you that a job took 40 minutes to complete.

Python Stand-in: end-to-end latency from message timestamps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. On which port does the rabbitmq_prometheus plugin serve metrics by default?

    15692 is the Prometheus endpoint (/metrics). 5672 is AMQP, 15672 the management UI and 9090 Prometheus itself.
  2. 2. Why use rate() for rabbitmq_channel_messages_published_total?

    Counters grow forever; the rate of change shows current throughput.
  3. 3. Which signal best indicates consumers are stuck rather than slow?

    Messages delivered but never acknowledged, with no acks flowing, point to stuck consumers or missing ack calls.
  4. 4. Why alert on sustained conditions rather than instant values?

    A queue is a buffer; temporary backlog is expected, a backlog that does not drain is an incident.

Summary

  • Enable rabbitmq_prometheus, scrape port 15692 on every node and import the official Grafana dashboards.
  • Watch ready, unacked, consumers, publish and ack rates, memory, disk, file descriptors and connection churn.
  • Alert on sustained symptoms, dead letter queue contents and missing consumers.
  • Add application metrics and end-to-end latency; logs explain what metrics only flag.

Next: Flow Control and Resource Alarms