Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 36 ยท Replication

Replication: Leaders and Followers

Learn single-leader replication: why we copy data, synchronous vs asynchronous replication, replication lag and failover pitfalls with a toy store.

Replication means keeping a copy of the same data on several machines. It is the first tool you reach for when one database server is not enough, and it solves three different problems at once: it keeps data available when a machine dies, it puts copies closer to users to cut latency, and it lets many read requests be spread across many machines.

The price is that copies can disagree. This lesson covers the most common design, single-leader replication, and shows with a runnable toy store where lag, lost writes and failover surprises come from.

Why replicate

Three reasons to copy data

Availability Survive a dead machine.
Latency Keep a copy near each user.
Read scale Spread reads over many copies.

Replication does NOT help you store more data than one machine holds. That is sharding.

1 / 2

Partitioning data across machines to hold more than one machine can store is a separate idea, covered in Partitioning and Sharding. Real systems usually combine both: every shard is replicated.

Single-leader replication

One node is the leader (also called primary or master). All writes go to the leader. The leader writes the change to its own storage and also sends it, as an ordered replication log, to the followers (replicas, secondaries). Followers apply the log in the same order, so they converge to the same state. Reads can go to the leader or to any follower.

flowchart LR
    C1["Client: write"] --> L["Leader"]
    L -->|"replication log"| F1["Follower 1"]
    L -->|"replication log"| F2["Follower 2"]
    C2["Client: read"] --> F1
    C3["Client: read"] --> F2

Writes enter at one place; the followers copy the leader's log and serve reads.

The big win of one leader is simplicity: every write is ordered by one node, so there are no conflicting writes to resolve. (Systems with several writers, covered in Multi-Leader and Leaderless Replication, face that problem.)

Synchronous vs asynchronous

When does the leader tell the client "your write succeeded"?

  • Synchronous: the leader waits until the follower confirms it has the write. The follower is guaranteed to be up to date, but if that follower is slow or down, writes stall.
  • Asynchronous: the leader confirms to the client immediately, and ships the change to followers in the background. Writes are fast and survive a dead follower, but followers lag behind, and a leader crash can lose confirmed writes.
  • Semi-synchronous: one follower is synchronous, the rest are asynchronous. If the sync follower fails, another is promoted to sync. This guarantees at least two up-to-date copies without letting any one slow follower block writes.

Replication lag

With asynchronous replication a follower is always some time behind the leader. That delay is replication lag. Usually it is milliseconds, but under load, during a network problem or while a follower catches up after being down, it can reach seconds or minutes.

The effect is that a read from a follower can return old data. The next lesson, Consistency Models, names the guarantees you can ask for. First, let's see lag happen.

Python A toy leader-follower store

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Followers apply the same log in the same order, so a lagging follower is never wrong in a random way. It is a correct but older version of the leader. That is eventual consistency in action: stop writing and all followers converge.

Failover and its pitfalls

When the leader dies, the system needs a new one. Failover has three steps: detect that the leader is dead (usually by timeout, see Failure Detectors and Heartbeats), choose the new leader (the most up-to-date follower is the natural pick), and reconfigure clients and the other followers to use it. Each step has traps:

  • Lost writes. With async replication the old leader may have acknowledged writes that never reached any follower. The new leader doesn't have them. If the old leader later returns with those writes, they conflict with the new history and are usually discarded.
  • Split brain. If the old leader was only slow, not dead, two nodes may both believe they are the leader and accept writes. Data diverges. Solutions involve fencing and consensus, see Leader Election.
  • Wrong timeout. Too short and a temporary slowdown triggers a needless failover that adds load; too long and users wait through a long outage.
  • Stale coordination. Other systems that kept the old leader's ID (caches, an auto-increment counter reused by the new leader) can behave wrongly.
Python Failover: async loses writes, sync does not

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the async run the clients were told five writes succeeded, but the promoted follower only has three of them. In the sync run nothing was lost, but when the follower is down the sync store refuses to write at all. That is the core trade-off: durability against availability and latency.

Real databases implement the replication log in different ways: shipping SQL statements, shipping physical disk changes, or logical row-level changes. Row-level logical logs are the most robust. Statement-based logs break on nondeterministic functions such as NOW() or RAND().

Adding a new follower

You can't just copy the leader's files while it is being written to. The usual recipe: take a consistent snapshot of the leader at a known log position, copy it to the new node, and then let the new node request every log entry since that position. When it has caught up it becomes a normal follower.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In single-leader replication, where do writes go?

    All writes are ordered by the leader and shipped as a replication log. This avoids write conflicts at the cost of a single write entry point.
  2. 2. What is the main risk of asynchronous replication?

    The leader acknowledges before followers have the write. If it dies in that gap, the promoted follower never had those writes.
  3. 3. What is the downside of fully synchronous replication to one follower?

    The leader waits for the follower on every write, so one slow or dead follower blocks all writes. Semi-synchronous setups limit this.
  4. 4. What is split brain?

    If the old leader is slow rather than dead, it can keep accepting writes after a new leader is elected. The data diverges.

Summary

  • Replication keeps copies on several machines for availability, lower latency and read scaling; it doesn't add storage capacity.
  • In single-leader replication all writes go through the leader and flow to followers as an ordered log.
  • Synchronous replication is safer but blocks on slow followers; asynchronous is fast but can lose acknowledged writes; semi-synchronous mixes them.
  • Replication lag means follower reads can be stale; followers converge when writes stop.
  • Failover (detect, elect, reconfigure) can lose writes, cause split brain, and misfire on bad timeouts.

Next: Consistency Models