Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 36 ยท Consensus and Coordination

Failure Detectors and Heartbeats

Learn how distributed systems notice dead nodes: heartbeats, timeouts, false positives, the phi-accrual idea and SWIM-style gossip membership, with simulations.

Every distributed system must answer a question that sounds trivial: is that node alive? If a node has crashed, you want to stop sending it requests, promote a replacement, and reassign its work. But from the outside, a crashed node and a node that is merely slow or unreachable look identical: you simply receive nothing. Over an asynchronous network (see system models), no detector can be both perfectly fast and perfectly accurate.

A failure detector is the component that makes a best guess. In this lesson you will build one from heartbeats and timeouts, see why it produces false alarms, learn how the phi-accrual idea replaces a yes/no answer with a suspicion level, and simulate a SWIM-style gossip membership protocol that scales to large clusters.

Heartbeats and timeouts

The simplest detector: every node sends a small heartbeat message to a monitor every interval ms. The monitor remembers when it last heard from each node. If no heartbeat has arrived for timeout ms, the node is suspected to have failed.

A timeout is a guess

Short timeout Fast detection, but more false alarms.
Long timeout Few false alarms, but slow reaction.
Network jitter Delays vary, so no value fits all.

A suspected node may be alive; a silent node may only be slow.

1 / 2

Choosing the timeout is a trade-off between completeness (every dead node is eventually suspected) and accuracy (live nodes are not suspected). A too-short timeout declares healthy but slow nodes dead. That is costly: a wrongly declared leader may be replaced while it still believes it is leader (see leader election), data gets re-replicated unnecessarily, and a busy cluster, now missing a "dead" node, becomes even busier and triggers more false alarms.

Simulating false positives

Network delay is random. The simulation below sends heartbeats every 100 ms from a healthy node whose delays follow an exponential-like distribution with occasional spikes. We count how often a fixed timeout wrongly suspects it, and then do the same for a node that really dies at time 5,000 ms and measure how long detection takes.

Python Fixed timeouts: false alarms versus detection time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Shorter timeouts detect crashes sooner but raise many false alarms; longer ones are quiet but slow. No value is "right": it depends on how expensive a false alarm is compared with a slow detection.

Phi accrual: suspicion as a number

Instead of a binary alive/dead verdict, the phi-accrual failure detector (used by Cassandra and Akka) outputs a continuous suspicion level ฯ†. The detector learns the recent distribution of heartbeat inter-arrival times. For the current silence, it computes how unlikely it is that a live node would be silent this long. The higher ฯ†, the less plausible "still alive": roughly, ฯ† = 1 means about a 10% chance the node is alive, ฯ† = 2 about 1%, ฯ† = 3 about 0.1%.

Two benefits follow. First, the detector adapts to the network: on a jittery link it learns that long gaps are normal. Second, different applications can use different thresholds for the same detector, for example "stop routing traffic at ฯ† โ‰ฅ 3, start rebuilding data at ฯ† โ‰ฅ 8".

Python A simple phi-accrual detector

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same 160 ms of silence gives ฯ† = 92 on the steady network (certainly dead) but only about 2 on the jittery one (a mild doubt). The huge numbers on the steady network appear because its heartbeats are almost perfectly regular. A fixed timeout cannot express that.

Why heartbeats to one monitor do not scale

If every node pings a central monitor, the monitor becomes a bottleneck and a single point of failure. If every node heartbeats every other node, traffic grows with nยฒ, which is hopeless for hundreds of machines. Large clusters use gossip-based membership instead.

SWIM: scalable membership

SWIM (Scalable Weakly-consistent Infection-style Membership) protocols, which inspired Consul/Serf and many others, split the job into two parts:

  1. Failure detection. In each protocol period, a node picks one random peer and pings it. If no reply comes, it asks k other random members to ping the target on its behalf (an indirect probe). If still nothing, the target is marked suspect.
  2. Dissemination. Membership changes (joined, suspect, dead) are piggybacked on the ping messages, so news spreads like an epidemic (see gossip protocols).

The indirect probe is the clever part. If the direct path between two nodes is flaky but the target is fine, a third party will usually reach it, so a single bad link does not cause a false positive. A suspect node also gets a grace period to refute the suspicion before it is declared dead.

sequenceDiagram
    participant A as Node A
    participant B as Node B (target)
    participant C as Node C
    participant D as Node D
    A->>B: ping
    Note over A,B: no ack (link problem)
    A->>C: ping-req(B)
    A->>D: ping-req(B)
    C->>B: ping
    B-->>C: ack
    C-->>A: ack for B
    Note over A: B is alive, no suspicion

An indirect probe saves B from a false suspicion caused by a bad A-to-B link.

Simulating SWIM-style detection

The simulation has 20 nodes in synchronous rounds. Each message is lost with 10% probability. One node crashes at round 5. We compare a direct-only detector (suspect after a failed ping) with SWIM's indirect probes (3 helpers) and count false suspicions of live nodes and rounds until everyone knows about the crash.

Python Direct probes versus SWIM indirect probes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Indirect probes cut false suspicions from 140 to 7 in this run. The price is slower detection of the real crash (round 17 instead of round 10), because a probe of the dead node must now fail on every path before it counts. Each node still sends only a handful of messages per round, regardless of cluster size. Real SWIM adds a suspect state with a timeout and incarnation numbers, so that a node that is wrongly suspected can announce "I am alive" and override the rumor.

Practical advice

  • Treat a failure verdict as a hint, not a fact. Make actions on suspicion safe and reversible; use fencing tokens so that a node wrongly declared dead cannot do harm.
  • Use adaptive detection (phi-accrual) or generous timeouts plus a suspect state on networks with high variance.
  • Detect failures at several layers: process health checks, network probes, and application-level health.
  • Watch out for gray failures: nodes that answer heartbeats but are too slow to do real work. Heartbeat on a path that exercises real functionality.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why can't a failure detector be perfect in an asynchronous network?

    All you observe is missing messages. You cannot tell whether the node is dead or whether its messages are delayed.
  2. 2. What is the effect of making a heartbeat timeout shorter?

    A short timeout reacts quickly but misjudges nodes that are merely slow.
  3. 3. What does the phi-accrual detector output?

    ฯ† grows as the silence becomes less plausible given the learned heartbeat distribution, so applications can choose their own threshold.
  4. 4. What problem do SWIM's indirect probes address?

    If a direct ping fails, other nodes probe the target too. If any of them reaches it, the target is not suspected.

Summary

  • Failure detectors guess from missing messages; they cannot be perfectly accurate and fast.
  • Heartbeat + timeout is simple, but the timeout trades detection time against false positives.
  • Phi accrual adapts to network behavior and gives a graded suspicion level.
  • SWIM scales by probing one random peer per round, using indirect probes to avoid false alarms, and spreading news by piggybacked gossip.
  • Treat suspicion as a hint: make reactions safe, use fencing, and beware of gray failures.

Next: Leader Election.