Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 37 ยท Clustering and High Availability

Network Partitions and Client Failover

Learn what a RabbitMQ network partition is, how pause_minority and autoheal work, why majority protocols help, and how clients and load balancers reconnect after a node failure.

A cluster spans machines, and machines are connected by networks that sometimes fail. A network partition happens when the cluster nodes lose contact with each other while still running: node A cannot see B and C, and B and C can still see each other. Each side may believe the other is dead. If both keep accepting writes, their data diverges, the classic split brain. This lesson explains how RabbitMQ handles the problem and, from the client side, how applications survive node failures with reconnection and load balancing.

RabbitMQ cannot run here, so configuration and client snippets are read-only; the runnable blocks are plain Python stand-ins that show the decision rules and a reconnection loop. They model the logic, not Erlang's actual failure detector.

What a partition looks like

flowchart LR
    subgraph Minority
        A[node1]
    end
    subgraph Majority
        B[node2]
        C[node3]
    end
    A -. link down .- B
    A -. link down .- C
    B --- C

RabbitMQ nodes notice peers are gone when the Erlang net tick timeout expires (about 60 seconds by default, net_ticktime). A short blip can pass unnoticed; a longer one triggers the partition rules. In the management UI and rabbitmqctl cluster_status you will see a "Network partition detected" warning.

Partition handling strategies

With the Mnesia metadata store (RabbitMQ 3.x and 4.x until the move to Khepri), the setting cluster_partition_handling in rabbitmq.conf decides what the nodes do:

Strategy Behavior Choose when
ignore Do nothing; nodes continue independently. You repair by restarting nodes manually. Reliable network, you prefer availability and handle repairs
pause_minority A node that cannot see more than half of the cluster pauses itself (stops serving) until it reconnects. Three or more nodes; the safe default for most clusters
autoheal Both sides keep running; after the link returns, the side with fewer connections is restarted and loses its changes. Availability matters more than the few messages that may be lost
Text
# rabbitmq.conf
cluster_partition_handling = pause_minority

pause_minority uses the same majority arithmetic as clustering fundamentals. It needs an odd cluster of at least three nodes: with two nodes neither side has a majority and both pause. Clients of the paused minority see their connections closed and reconnect to a healthy node.

Python Stand-in: which side keeps running?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The 2 vs 2 case shows why even sizes are poor: with pause_minority both halves pause and the whole cluster stops.

Newer behavior with Khepri and quorum queues

Quorum queues and the Khepri metadata store both use Raft, so they are partition-safe by design: a minority side simply cannot make progress and cannot diverge. When the link returns, the minority replays the leader's log. This is why new designs depend on quorum queues rather than partition-handling modes. Note that classic queues and the Mnesia store are the parts that need pause_minority.

Typical causes of partitions are cloud network glitches, overloaded nodes missing heartbeats (a node paused for a long GC or disk stall looks dead), and VM live migrations. Fix them at the source, and avoid stretching one cluster across distant data centers; use federation or the shovel plugin to move messages between sites instead.

Client failover

Even a perfect cluster is useless if clients stay attached to a dead node. Three ingredients give failover:

  1. Several addresses. Give the client all nodes, or a single load-balanced DNS name that points at all of them. Most official clients accept an address list.
  2. Automatic connection recovery. The Java and .NET clients recover connections, channels, exchanges, queues, bindings and consumers on their own. The Python pika library does not; you write a reconnect loop (or use aio-pika/pika with a recovery wrapper).
  3. Idempotent processing. After recovery, unconfirmed publishes and redelivered messages repeat work, so handlers must tolerate duplicates.
Python
import time
import pika

NODES = ["rabbit1.internal", "rabbit2.internal", "rabbit3.internal"]

def connect():
    params = [pika.ConnectionParameters(host=h, heartbeat=30,
                                        blocked_connection_timeout=60,
                                        connection_attempts=1) for h in NODES]
    return pika.BlockingConnection(params)   # pika tries the list in order

while True:
    try:
        connection = connect()
        run_consumer(connection)
    except pika.exceptions.AMQPConnectionError:
        time.sleep(2)                          # backoff, then try the next node

A TCP load balancer such as HAProxy in front of the nodes hides the list and can health-check each node (port 5672, or rabbitmq-diagnostics check_running through an agent). Use balance leastconn and long timeouts (hours), because AMQP connections are long-lived and idle channels should not be cut.

Python Stand-in: reconnect with backoff across nodes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pair blocked_connection_timeout and heartbeats with connection_attempts/retry_delay so a dead node does not hang the client for minutes. Consumers should also re-declare their queues on reconnect: declaring an existing durable queue with identical arguments is harmless.

A failure-day checklist

  • Run three or five nodes, spread across failure zones.
  • Use quorum queues for important data, pause_minority for what is left on Mnesia.
  • Set heartbeats (10 to 60 seconds) and sensible client timeouts.
  • Test: stop one node (rabbitmqctl stop_app) during a load test and watch that clients recover with no lost confirms.
  • Alarm on partitions and on queues without a leader (see Monitoring and Prometheus).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does pause_minority do?

    The minority side suspends itself so that only the majority side accepts writes, preventing split brain.
  2. 2. Why is a 4-node cluster a poor choice with pause_minority?

    A majority of 4 is 3; neither half of a 2-2 split has it.
  3. 3. Which client library feature is missing from pika and must be written by you?

    The Java and .NET clients recover automatically; with pika you write a reconnect loop.
  4. 4. Why must consumers be idempotent after failover?

    At-least-once delivery means a message can be delivered more than once after reconnection.

Summary

  • A partition splits a cluster while nodes still run; unmanaged, both halves diverge.
  • pause_minority (odd node counts) is the safe strategy for Mnesia; quorum queues and Khepri are Raft-based and partition-safe.
  • Clients need multiple addresses or a load balancer, reconnect logic with backoff, and idempotent processing.
  • Avoid stretching a cluster over a WAN; use federation or shovel.

Next: Work Queues and Fair Dispatch