Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 37 ยท AMQP and Architecture

Inside the Broker: Erlang, Nodes and Storage

A tour of RabbitMQ internals: Erlang processes, nodes and the metadata store, how messages are stored, and the memory and disk alarms that throttle publishers.

You can use RabbitMQ for years without reading its source, but a mental picture of the inside explains many behaviors: why it handles thousands of connections, why a crashed queue does not take the broker down, and why a "memory alarm" suddenly blocks publishers.

This lesson describes real RabbitMQ architecture in prose. The runnable blocks are small Python stand-ins for ideas such as independent mailboxes and threshold alarms; they are not RabbitMQ code.

Erlang and the BEAM

RabbitMQ is written in Erlang and runs on the BEAM virtual machine. Erlang was designed for telephone switches that must handle huge numbers of simultaneous, independent activities and keep running when parts fail. Three properties matter to you:

  • Lightweight processes. An Erlang process is far smaller than an operating system thread. RabbitMQ gives roughly every connection, every channel and every queue its own process, so thousands of them can coexist.
  • Message passing, no shared memory. Processes talk by sending messages to each other's mailboxes. There is no shared state to lock, so one process cannot corrupt another.
  • Supervision. Processes are watched by supervisors that restart them when they crash ("let it crash"). A faulty channel is cleaned up without stopping the broker.

The stand-in below shows the actor idea: each queue has its own mailbox, a failure in one is contained, and a supervisor restarts it.

Python Independent mailboxes and a supervisor (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

(In real RabbitMQ, durable queues recover their persistent contents from disk when restarted; this stand-in does not model that.)

Nodes and the cluster

One running RabbitMQ instance is a node, identified by a name such as rabbit@host1. Nodes can join into a cluster. Nodes find each other over Erlang's own distribution protocol (port 25672 by default) and authenticate with a shared secret file called the Erlang cookie. If cookies differ, nodes refuse to talk, which is a common first-cluster error. Clusters are covered in a later chapter.

What lives where

Two kinds of data exist:

  • Metadata: users, vhosts, exchanges, queues and bindings, policies, runtime parameters. It is replicated to every node in a cluster. Older versions stored it in Erlang's built-in database Mnesia; RabbitMQ 4 uses a Raft-based store called Khepri by default, which behaves better when the network partitions.
  • Messages: stored by each queue, according to its type (classic, quorum or stream). They are not copied to every node: only replicated queues copy their messages, and only to the nodes that host replicas.

How classic queues store messages

A classic queue keeps messages in memory when it can and writes persistent ones to disk. Messages for durable queues with delivery_mode=2 are written to a message store and an index, so they can be recovered after a restart. Transient messages normally stay in memory but may be paged to disk when memory is tight. In recent versions classic queues mostly keep their data on disk and use memory as a small cache, which makes them stable under large backlogs.

Quorum queues and streams use their own Raft-based write-ahead logs and segment files. The practical advice: put the data directory on fast, reliable storage (SSD, local to the node) and leave plenty of free space.

Alarms: memory and disk

To protect itself, a node watches two resources:

  • Memory high watermark. By default, when the Erlang VM uses more than 60% of the machine's RAM (vm_memory_high_watermark.relative = 0.6), the node raises a memory alarm. All connections that publish are blocked until memory drops.
  • Free disk limit. When free disk space falls below disk_free_limit (by default 50 MB; production setups use a much larger value, often relative to RAM), a disk alarm is raised and publishers are blocked in the same way.

Blocking publishers is a deliberate safety valve: it keeps the broker alive instead of letting it crash. Consumers keep working, so queues can drain and the alarm clears. The management UI shows alarms in red.

Settings live in rabbitmq.conf:

INI
# rabbitmq.conf (read-only example)
vm_memory_high_watermark.relative = 0.5
disk_free_limit.absolute = 2GB
Python Memory and disk alarm logic (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Resource limits you should know

  • File descriptors. Each connection and many files use one. Production nodes need a high limit (tens of thousands) set in the operating system or service file.
  • Erlang processes and atoms have limits that you rarely hit.
  • Channel max per connection, and optional connection limits per vhost or user.

The management UI's node page shows file descriptors, memory and disk usage against their limits.

The life of a published message

Putting it together for a classic durable queue:

  1. The client sends frames on a channel; the channel process assembles a complete message.
  2. The channel asks the exchange logic which queues match (routing happens in the channel's process, using metadata).
  3. The message is handed to each target queue process.
  4. The queue stores it (and writes it to disk if persistent), then offers it to a consumer.
  5. When the consumer acknowledges, the queue removes it.
flowchart LR
    CH["Channel process"] -->|"route"| Q["Queue process"]
    Q -->|"persist"| D[("Disk")]
    Q -->|"deliver"| CO["Consumer channel process"]
    CO -->|"ack"| Q

Several small processes cooperate; none of them shares memory.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why can RabbitMQ handle thousands of connections and queues?

    The BEAM runs huge numbers of cheap processes that communicate by messages, so each connection, channel and queue can have its own.
  2. 2. What happens when a node's memory high watermark is exceeded?

    The memory alarm blocks publishers so the node can recover. Consumers continue and drain queues.
  3. 3. What must nodes share to be able to form a cluster?

    Erlang distribution authenticates with a shared secret cookie. Different cookies mean the nodes refuse to talk.
  4. 4. Which statement about message storage is true?

    Metadata is replicated to all nodes, but message data belongs to the queue and is replicated only for replicated queue types.

Summary

  • RabbitMQ runs on Erlang's BEAM: tiny independent processes, message passing and supervision.
  • A node is one broker instance; nodes cluster using the Erlang cookie.
  • Metadata is replicated across the cluster (Khepri in RabbitMQ 4); message data belongs to queues.
  • Memory and disk alarms block publishers to protect the node.
  • Give nodes fast disks, free space and enough file descriptors.

Next: Queues in Depth