Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 36 ยท Projects

Project: A Mini Raft Cluster

Build a five-node Raft simulation with elections, log replication, crashes and recovery, driven by a deterministic scheduler and invariant checks.

Raft makes a group of machines agree on one ordered log of commands, even when some of them crash and messages are lost, delayed or reordered. In Raft Step by Step you learned the rules. Now you will implement them. The result is a five-node Raft cluster in about 170 lines of Python, running inside a deterministic scheduler: simulated time ticks forward, a seeded random generator decides message delays, drops and timeouts, and after every tick the code checks the core safety invariants.

This is the same style real teams use to test consensus code, as described in Testing Distributed Systems. It is a teaching model: no disk, no snapshots, no membership changes and clients simply hand commands to the current leader. But the heart of Raft (terms, votes, append entries, the commit rule) is all here.

The moving parts

Election Randomized timeouts, one vote per term.
Replication The leader appends and pushes entries.
Commit A majority holds the entry, from the leader's own term.
Repair Followers drop conflicting tails.
1 / 2

The design in one page

  • Nodes keep persistent state (current term, who they voted for, the log) and volatile state (role, commit index, leader bookkeeping). A crash and recovery keeps the first group and loses the second.
  • Log entries are (term, command) tuples. Indexes start at 1, so entry i is log[i - 1].
  • Messages travel through a list with a delivery time. The scheduler delivers messages in time order, and each tick lets every live node act on its timers.
  • Election: a follower that hears nothing for 12 to 24 ticks becomes a candidate, increments its term and asks for votes. A node grants at most one vote per term, and only to a candidate whose log is at least as up to date as its own (higher last term wins, then longer log).
  • Leader: on winning a majority it appends a no-op entry for its new term, then sends append messages (heartbeats double as replication) every 3 ticks.
  • Replication: each append carries the previous index and term. A follower rejects it if its log does not match there, and the leader steps next back by one and retries. When the log matches, the follower deletes any conflicting entries and appends the new ones.
  • Commit rule: the leader commits index i when a majority stores it and the entry belongs to the leader's own term. Older entries are committed indirectly. This is why the no-op exists.

Our nodes "send" messages by adding to a list and "receive" in the scheduler loop. Real Raft would use RPCs over a network and persist term, vote and log to disk before replying. The logic you see here is unchanged by those details.

Step 1: the cluster and a crash story

The Cluster class holds the nodes, the message list, the clock and the invariant checker. The first scenario shows a whole life cycle: elect a leader, replicate three commands, crash the leader, elect a new one, replicate two more commands, bring the old leader back, and verify that every node ends up with the same applied commands.

Python Raft cluster: election, crash and recovery

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the event log first. Node n2 times out first (tick 13) and wins term 1 at tick 16. Node n1 also started an election for term 1 at the same tick, but it had already lost: a node votes once per term, and n2 collected the majority. When n2 crashes at tick 78, a follower times out, and n4 becomes leader of term 2. When n2 comes back at tick 150, it still thinks it is in term 1, but the first append from n4 carries term 2, so it steps down and catches up.

Then look at the final table. All five nodes have applied the same sequence, including the noop entries that each new leader adds at the start of its term. The assertion problem is None after every tick means the invariants held at all 210 ticks of the story.

The invariants

The checker in Cluster.check tests three properties from the Raft paper. They are the properties that, if violated, would make the cluster wrong.

  1. Election safety: at most one leader per term. Two leaders in one term would mean two different histories could both claim to be official.
  2. Log matching: if two logs contain an entry with the same index and term, the logs are identical up to that index. This is why the "previous index and term" check in append is enough to keep logs consistent.
  3. State-machine safety (committed entries never change): once any node has committed index i, every node's entry at i must equal it, forever. The checker remembers the first committed entry per index and compares.

Step 2: chaos tests and negative controls

Now to the key question: would we notice if the code was wrong? The chaos test runs many seeds. In each run, random events crash and recover nodes, drop 10% of messages and submit commands. After every tick the invariants are checked. Then the test heals the cluster (recover everybody, no more drops) and requires that all logs converge and a leader exists. That checks liveness as well as safety.

Then the test does something you should always do with a test: it breaks the implementation on purpose and checks that the test notices:

  • no-log-check makes nodes vote for any candidate, even one with an out-of-date log. A leader without the committed entries can then overwrite them.
  • commit-early lets the leader commit once only one node (itself) has the entry, instead of a majority. A new leader that never saw the entry will then replace it.
Python Chaos testing with invariants and injected bugs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The correct implementation shows no violations across 80 random fault schedules, and every run converges after healing. Each buggy variant is caught on some of the seeds, with a message such as committed entry 9 changed, which is exactly the failure Raft's rules exist to prevent. The two bugs are caught by different seeds, a reminder that you need many varied schedules, and that the number of seeds you can afford determines how rare a bug you can find. A failing seed is a replayable bug report: run chaos(2, "commit-early") again and you get the identical sequence of events.

Try this in the editor: change drop=0.1 to 0.3 or the crash probability from 0.03 to 0.1. The correct version should still pass every invariant. Heavy faults slow progress, since elections keep failing, but they never break safety. That distinction between safety and liveness is central to consensus (Consensus and Paxos).

Extensions

  • Persist and restart: save term, vote and log to a dict on crash and rebuild from it. Check that losing the vote (voted_for) on restart can break election safety.
  • Pre-vote: stop a partitioned node from inflating its term and disrupting the cluster when it returns.
  • Client sessions: give each command an ID so a retried command is not applied twice (see Idempotency, Retries and Backoff).
  • Snapshots: truncate the log once a prefix is applied by every node.
  • Network partitions: extend send to drop messages between two groups. Which side can elect a leader and commit?
  • Linearizable reads: a leader must confirm it is still leader (for example with a heartbeat round) before answering a read.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does a new leader append a no-op entry for its own term?

    The commit rule only counts replicas for entries of the leader's current term. Committing the no-op also commits everything before it.
  2. 2. What does a follower do when an append's previous index and term do not match its log?

    The mismatch tells the leader to back up. Once the logs match at some index, conflicting entries after it are replaced.
  3. 3. Which bug does the 'no-log-check' variant introduce?

    The up-to-date check ensures that a winning candidate holds every committed entry.
  4. 4. Why does the test deliberately run broken versions of the code?

    A test that never fails proves little. Catching known bugs shows that the checker has teeth.

Summary

  • A Raft node keeps term, vote and log across crashes; leaders are elected by majority with randomized timeouts and one vote per term.
  • Leaders replicate by appending with previous-index checks; followers drop conflicting tails, and commit requires a majority plus an entry from the leader's own term.
  • A deterministic scheduler with a seeded RNG makes distributed behavior reproducible and explorable.
  • Check invariants after every tick: one leader per term, log matching, committed entries never change.
  • Inject known bugs to prove your tests catch them, and heal the cluster at the end to test liveness.

Next: Project: Reliable Job Queue.