Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 37 ยท Clustering and High Availability

Clustering Fundamentals

Learn how a RabbitMQ cluster works: nodes, the Erlang cookie, shared metadata, queue leaders, required ports, joining nodes with rabbitmqctl and why you want an odd node count.

One RabbitMQ node is a single point of failure: when the machine or the process stops, every queue on it is unavailable. A cluster joins several nodes into one logical broker. Clients can connect to any node, and the cluster shares its metadata (users, virtual hosts, exchanges, queues, bindings, policies) so every node knows what exists. What a cluster does not do automatically is copy your messages: whether a queue survives a node failure depends on the queue type, covered in Quorum Queues.

RabbitMQ cannot run on this page. The shell commands are read-only, and the runnable blocks are plain Python stand-ins for the ideas: replicated metadata, queue leaders and majorities. They model the concepts, not RabbitMQ's Erlang internals.

What a cluster shares

flowchart TB
    subgraph Cluster
        N1["rabbit@node1<br/>metadata + queue A leader"]
        N2["rabbit@node2<br/>metadata + queue B leader"]
        N3["rabbit@node3<br/>metadata + queue C leader"]
    end
    N1 --- N2
    N2 --- N3
    N1 --- N3
    C1[Client] --> N1
    C2[Client] --> N3
  • Shared by all nodes: virtual hosts, users and permissions, exchanges, bindings, queue definitions, policies, parameters.
  • Not shared: the message bodies of a classic queue. A classic queue lives on one node (its home). Clients connected to other nodes reach it through that node, over the network.
  • Replicated per queue type: quorum queues and streams copy messages to several nodes (a replica set) and elect a leader.

RabbitMQ 3.x stored metadata in Mnesia, an Erlang database. RabbitMQ 4.x is moving metadata to Khepri, a Raft-based store that tolerates network partitions better. Which one is used depends on your version and feature flags; the operator-visible rules below apply to both.

Requirements before nodes can join

  1. Same Erlang cookie. Nodes authenticate each other with a shared secret in the file .erlang.cookie (in the RabbitMQ user's home or /var/lib/rabbitmq). It must be identical on every node and kept private: anyone with it can control the cluster. Mismatch shows up as Authentication failed in the logs.
  2. Resolvable node names. A node is named rabbit@hostname. Every node must be able to resolve the others' hostnames (DNS or /etc/hosts). Use stable names and the same naming style (short or long) on all nodes.
  3. Open ports. Nodes talk through these ports; block them from the outside, allow them between nodes:
Port Used for
4369 epmd, the Erlang port mapper
25672 Inter-node and CLI communication (distribution port)
35672 to 35682 CLI tools talking to a node
5672 / 5671 AMQP 0-9-1 clients (plain / TLS)
15672 Management UI and HTTP API
5552 Stream protocol
  1. Same RabbitMQ and Erlang versions (a short rolling-upgrade window is the only exception).

Joining nodes

Text
# on node2 and node3, after the same cookie is in place
rabbitmqctl stop_app
rabbitmqctl reset                 # forget any previous standalone state
rabbitmqctl join_cluster rabbit@node1
rabbitmqctl start_app

rabbitmqctl cluster_status

reset deletes all data on that node, so only run it on a fresh node. After joining, cluster_status lists the nodes, their versions and whether alarms are active. To remove a node: rabbitmqctl forget_cluster_node rabbit@node3 (run on a remaining node when the dead node cannot do it itself). For repeatable setups you use the peer discovery plugins (DNS, Kubernetes, AWS, Consul, etcd) or a classic config list:

Text
# rabbitmq.conf
cluster_formation.peer_discovery_backend = classic_config
cluster_formation.classic_config.nodes.1 = rabbit@node1
cluster_formation.classic_config.nodes.2 = rabbit@node2
cluster_formation.classic_config.nodes.3 = rabbit@node3

Nodes started with that file form the cluster on their own.

Old guides mention RAM nodes, which keep metadata only in memory. They are discouraged: they speed up almost nothing and are risky. Use disc nodes only.

Why three (or five) nodes

Replicated features such as quorum queues and Khepri use Raft, which needs a majority (quorum) of members to agree. With n members the majority is floor(n/2) + 1. Notice that an even number does not help:

Nodes Majority needed Failures tolerated
1 1 0
2 2 0
3 2 1
4 3 1
5 3 2

Two nodes tolerate no failure at all, and four nodes tolerate no more than three. Use 3 nodes for most workloads and 5 when you must survive two failures.

Python Stand-in: majority arithmetic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Queue leaders and where clients connect

Each queue has one leader (classic queues: their single home node). All publishes and consumes for a queue are processed by its leader, so a client connected to another node is proxied to it. That costs an extra network hop but works. To keep traffic balanced, RabbitMQ spreads new queue leaders across nodes (the queue_leader_locator setting, balanced by default in recent versions). Put a load balancer (HAProxy, a cloud TCP load balancer) in front of all nodes so clients do not need a node list; see Network Partitions and Client Failover.

Python Stand-in: leaders, replicas and a node failure

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rolling maintenance

Upgrades and restarts are done one node at a time: stop (rabbitmq-upgrade drain first in newer versions, so clients and queue leaders move away), do the work, start, wait until rabbitmq-diagnostics check_running and the replicas are in sync, then continue. Never stop two nodes of a three-node cluster together. Also enable the same feature flags after the whole cluster runs the new version (rabbitmqctl enable_feature_flag all).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What must be identical on all nodes before they can form a cluster?

    Nodes authenticate each other with the shared Erlang cookie. Resolvable hostnames and matching versions are required as well.
  2. 2. How many node failures can a 4-node cluster survive for majority-based features?

    The majority of 4 is 3, so only one node can be down. That is the same as a 3-node cluster, which is why odd sizes are recommended.
  3. 3. What is replicated across every node of a cluster by default?

    Metadata is shared; classic queue contents live on one node unless you use a replicated queue type.
  4. 4. Which command prepares a fresh node to join node1's cluster (after stop_app and reset)?

    join_cluster is run on the joining node, followed by start_app.

Summary

  • A cluster shares metadata between nodes; message replication depends on queue type.
  • Joining needs the same Erlang cookie, resolvable hostnames, open inter-node ports (4369, 25672) and matching versions.
  • Majority-based replication means odd sizes: 3 nodes survive one failure, 5 survive two.
  • Put a load balancer in front, upgrade one node at a time and use disc nodes only.

Next: Quorum Queues