Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท Operations and Monitoring

Flow Control and Resource Alarms

Understand how RabbitMQ protects itself: credit-based flow control, the memory high watermark, the disk free limit, blocked connections, and how to tune and react to alarms.

A broker has finite memory, disk and CPU. If producers publish faster than consumers drain, something has to give. RabbitMQ does not crash quietly; it pushes back. First it slows individual publishers with flow control. If that is not enough, it raises a resource alarm (memory or disk) that blocks all publishing connections until the problem clears. Knowing these mechanisms lets you tell "RabbitMQ is protecting itself" from "RabbitMQ is broken", and tune accordingly.

RabbitMQ is not available on this page: configuration lines and CLI commands are read-only. The runnable blocks are plain Python stand-ins that model credit flow and alarm thresholds with sample numbers.

Credit-based flow control

Inside a node, messages pass through a pipeline of Erlang processes: connection reader, channel, queue, disk writer. Each stage gives the previous one credit: "you may send me 200 more messages". When a slow stage runs out of credit, the stage in front of it stops and so on back to the TCP socket. The practical effect: a publisher whose messages pile up in a slow queue is simply throttled: its socket reads are delayed, so TCP back-pressure slows the client.

In the management UI a throttled connection shows the state flow (instead of running). That is normal and healthy for short periods; if connections stay in flow, the real cause is downstream: slow queue, slow disk or slow consumers.

Python Stand-in: credit-based back-pressure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

After a short burst, the producer is limited to roughly the consumer's rate. Nothing is lost; the client just waits.

The memory high watermark

RabbitMQ watches the memory used by the Erlang VM. When it passes the high watermark, the broker raises a memory alarm and blocks every connection that is publishing (consumers keep working, which is how the memory is freed). The default watermark is 0.6, i.e. 60% of the available RAM (container limits are detected). Set it in rabbitmq.conf:

Text
vm_memory_high_watermark.relative = 0.6
# or an absolute value:
# vm_memory_high_watermark.absolute = 4GiB

Why only 60%? Garbage collection and other operations (paging, snapshots) can temporarily need as much memory again, and the operating system needs room too. Do not push it above 0.7 and never to 1.0.

Memory is used by: messages in queues (the bigger part for backlogs), connections and channels (each costs tens of KB, so tens of thousands add up), metadata and ETS tables, binary heap, and plugins. To see the breakdown:

Text
rabbitmq-diagnostics memory_breakdown --unit "MB"
rabbitmq-diagnostics check_local_alarms
rabbitmqctl status

The disk free limit

The second alarm guards the disk where the broker keeps its data. If free space drops below disk_free_limit (default 50 MB, which is far too low for production), the disk alarm fires and publishers are blocked. Set it relative to memory so a full memory dump (paging or a core file) can always be written, a typical recommendation being 1.0x to 2x the RAM:

Text
disk_free_limit.relative = 1.5
# or: disk_free_limit.absolute = 8GB

The disk alarm also prevents corruption: a full disk is the leading cause of broker data damage. Alert before the limit is reached (see Monitoring and Prometheus).

Python Stand-in: when do alarms fire?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What blocked publishers experience

When an alarm is active:

  • Each node with an alarm tells all nodes in the cluster, and publishers connected to any node are blocked. The alarm is cluster-wide.
  • The broker sends connection.blocked to clients that support the capability (most do), and stops reading from publishing connections. Their publishes sit in client buffers or time out.
  • Consumers continue; when memory or disk recovers, the broker sends connection.unblocked.

Client code must handle it. With pika, set a callback and a timeout so a blocked connection does not hang forever:

Python
params = pika.ConnectionParameters("localhost", blocked_connection_timeout=300)
connection = pika.BlockingConnection(params)
connection.add_on_connection_blocked_callback(lambda conn, frame: log.warning("blocked: %s", frame))
connection.add_on_connection_unblocked_callback(lambda conn, frame: log.info("unblocked"))

A common mistake is publishing and consuming on the same connection. When publishing is blocked, the broker stops reading from the connection entirely, including your ack frames, so consumption stalls too and the memory never drops. Use separate connections for publishers and consumers.

Reacting to an alarm

  1. Check rabbitmq-diagnostics check_local_alarms and the management Overview to see which alarm and which node.
  2. Find the cause. A growing queue? rabbitmqctl list_queues name messages memory. Dead consumers? Missing acks? A sudden rise in connections?
  3. Drain. Start more consumers, fix the stuck ones, or (for disposable data) purge a queue: rabbitmqctl purge_queue name.
  4. Fix the capacity. Add RAM or disk, move to quorum queues with length limits (max-length, overflow: reject-publish) so a runaway producer is rejected early instead of exhausting memory.
  5. Resist raising the watermark as the only fix: it postpones the crash, it does not prevent it.
Python Stand-in: a length limit protects the broker

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other limits worth setting

  • File descriptors: every connection and every queue file uses one. Run with at least 64,000 (ulimit -n, systemd LimitNOFILE), and check rabbitmq_process_open_fds.
  • Connection and channel limits per vhost or user (rabbitmqctl set_vhost_limits -p shop '{"max-connections": 500}') stop a buggy client from opening thousands of connections.
  • Per-channel prefetch and consumer timeouts protect against runaway deliveries (Consumer Acknowledgements and Prefetch).
  • Message size: the default maximum is 16 MiB (max_message_size); huge messages hurt every queue sharing the node. Store large blobs elsewhere and send a reference.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens when the memory high watermark is exceeded?

    A memory alarm blocks publishers cluster-wide. Consumers continue, which lets the backlog drain and memory fall.
  2. 2. What is the default vm_memory_high_watermark?

    The default is 0.6; leaving 40% accounts for GC spikes and the OS.
  3. 3. Why should publishers and consumers use separate connections?

    When blocked, the broker stops reading from the connection, including consumer acknowledgements.
  4. 4. Which setting is better than just raising the watermark to prevent a runaway producer from filling memory?

    Bounding the queue makes the broker reject excess publishes early so producers see the problem.

Summary

  • Credit-based flow control throttles fast publishers; the UI shows state flow.
  • The memory high watermark (default 0.6) and disk_free_limit (raise it from 50 MB) trigger cluster-wide alarms that block publishers.
  • Handle connection.blocked, separate publish and consume connections and bound queues with length limits.
  • Diagnose with check_local_alarms and memory_breakdown, drain the backlog, and fix capacity rather than only the limit.

Next: Security: Users, Virtual Hosts and TLS