Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 65 ยท Databases at Scale

Database Replication

Learn database replication for system design: leader-follower, multi-leader and leaderless setups, sync versus async, replication lag and failover, with Python demos.

Replication means keeping copies of the same data on several machines. It is the first tool you reach for when one database server is not enough, for three reasons: availability (another copy takes over if one dies), read scaling (spread reads across copies) and latency (put a copy near your users).

The hard part is that copies must be kept in step while data keeps changing. This lesson walks through the common topologies, the sync versus async choice, replication lag and what happens during failover.

Leader-follower replication

In the most common setup, one node is the leader (also called primary or master). All writes go to it. It records each change in a log, and followers (replicas, secondaries) apply that log in the same order. Reads can go to any node.

flowchart LR
  C[Clients] -->|writes| L[(Leader)]
  L -->|replication log| F1[(Follower 1)]
  L -->|replication log| F2[(Follower 2)]
  C -->|reads| F1
  C -->|reads| F2

One leader accepts writes; followers copy its log and serve reads.

PostgreSQL, MySQL, MongoDB and Redis all work this way by default. It is simple because there is a single source of truth for the order of writes, so there are no write conflicts.

Synchronous or asynchronous

When does the leader tell the client "done"?

  • Synchronous: the leader waits until followers have confirmed the write. No acknowledged write can be lost if the leader dies, but a slow or dead follower blocks writes.
  • Asynchronous: the leader answers right after writing locally and ships the change later. Writes are fast and survive follower outages, but a leader crash can lose the most recent writes.
  • Semi-synchronous: wait for one follower, copy the rest asynchronously. This is a popular middle ground: at least two copies exist for every acknowledged write.
Python Sync versus async: what survives a leader crash

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result is the central trade-off of replication: the more copies you wait for, the safer your writes and the slower (and less available) they become.

Replication lag and its anomalies

With asynchronous followers, a follower can be a little behind. That delay is replication lag. A client reading from a lagging follower may see stale data, and some of the behavior is surprising:

  • Read-your-writes violated: you update your profile, reload, and see the old one because the read went to a lagging replica.
  • Monotonic reads violated: you see a comment, refresh, and it disappears because the second read hit a more stale replica.

Standard fixes: read from the leader for data the user just changed, or pin a user's session to one replica, or send the last write's log position with each request and only read from a replica that has caught up. The theory is in Consistency Models.

Python Routing reads to avoid stale data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Failover

When the leader fails, someone must take over. Failover has three steps: detect the failure (usually by missed heartbeats), choose a new leader (the follower with the most up-to-date log), and reconfigure clients and other followers to use it. It sounds easy, and it goes wrong in familiar ways:

  • Lost writes: the new leader never received the last asynchronous writes.
  • Split brain: the old leader was only slow, comes back, and now two nodes both accept writes. Prevent this by fencing the old leader (cutting its storage or network access) or by using consensus-based election.
  • Wrong timeout: too short causes needless failovers during load spikes; too long means long outages.

Other topologies

Multi-leader replication lets several nodes accept writes, typically one leader per data center, so users write locally and fast. The price is write conflicts: two leaders can change the same record at once, and you need a resolution rule such as last-write-wins, merging or app-level logic. Leaderless replication (Dynamo, Cassandra) sends each write to several nodes and uses quorums: with N replicas, write to W and read from R such that W + R > N, and reads overlap with the latest write. Both are covered in depth in Multi-Leader and Leaderless Replication.

Topology Writes go to Conflicts Typical use
Leader-follower One leader None Most web apps
Multi-leader One leader per region Yes, must resolve Multi-region apps, offline editing
Leaderless Several replicas via quorum Yes, via versions Highly available key-value stores

What replication does not solve

Replication copies everything to every node. It scales reads and survives failures, but it does not increase how much data you can hold or how many writes you can take, because every replica still has to apply every write. For that you need partitioning (sharding), covered next in Sharding and Partitioning.

Replicas are not backups. A bad DELETE or a corrupting bug is replicated to every follower within seconds. Keep point-in-time backups too.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the risk of asynchronous replication?

    With async replication the leader acknowledges before followers have the data, so the newest writes may exist only on the dead leader.
  2. 2. A user saves a setting but sees the old value after a reload. Which fix is the most targeted?

    The read hit a replica that had not applied the write yet. Routing the user's reads to the leader, or only to caught-up replicas, restores read-your-writes.
  3. 3. What is split brain?

    If the old leader returns after a failover, both it and the new leader accept writes and the data diverges. Fencing prevents this.
  4. 4. Which problem does replication NOT solve?

    Every replica holds the full data set and applies every write, so capacity and write throughput do not grow. Sharding does that.

Summary

  • Replication copies data to several nodes for availability, read scaling and low latency.
  • Leader-follower is the default; multi-leader and leaderless trade simplicity for write availability and need conflict handling.
  • Sync replication is safe but slow, async is fast but can lose writes, semi-sync is a common compromise.
  • Replication lag causes stale and out-of-order reads; fix with leader reads, session pinning or log positions.
  • Failover can lose writes or cause split brain; replicas are not backups.

Next: Sharding and Partitioning