Skip to content
Tutorials Center
My learning

Topics

Lesson 50 of 70 ยท Replication, sharding and operations

Replica Sets

Understand MongoDB replica sets: primary and secondaries, the oplog, elections and failover, arbiters, rollbacks and replica set connection strings.

A single database server is a single point of failure. Disks die, machines reboot, data centers lose power, and an application that depends on one mongod process goes down with it. MongoDB's answer is the replica set: a group of mongod processes that all keep a copy of the same data. If one member fails, the others notice and carry on, usually within seconds and without anyone logging in at 3 a.m.

This lesson explains who does what inside a replica set, how the oplog keeps copies in step, how an election picks a new primary, what a rollback is and why it can lose writes, and how your application connects to the whole set. Replica sets are also the foundation for transactions, change streams and sharding, so the ideas here come back later.

You cannot start a replica set on this site. The mongosh listings are real commands and are read-only. The runnable examples use a small simulation written in plain JavaScript: members with logs, a majority rule and a simplified election. It teaches the logic, but real elections also involve priorities, heartbeats and timeouts that the simulation only hints at.

Roles in a replica set

Primary The only member that accepts writes.
Secondaries Copy the primary's oplog and apply it.
Arbiter Votes in elections but stores no data.

Reads go to the primary by default; read preference can change that.

1 / 2

Primary and secondaries

In a replica set exactly one member is the primary at any time. It receives every write. The other members are secondaries. Each secondary copies the primary's operations and applies them to its own data, so after a short delay it holds the same documents. Your application normally talks only to the primary, and a secondary is a hot standby that is ready to take over.

A set is created once with rs.initiate() and then inspected with rs.status():

JavaScript
rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "db1.example.com:27017" },
    { _id: 1, host: "db2.example.com:27017" },
    { _id: 2, host: "db3.example.com:27017" }
  ]
})

rs.status().members.map(m => ({ name: m.name, state: m.stateStr }))

After a moment the output shows one PRIMARY and two SECONDARY members. rs.conf() prints the configuration, and rs.add() and rs.remove() change membership while the set runs.

Members can also have special roles. A hidden member holds data but is invisible to applications, which is handy for backups and reporting. A delayed member applies operations a fixed time late (an hour, for example) so that a mistaken deleteMany can be recovered from it. A priority setting makes some members more likely to become primary, and priority 0 means a member can never be elected, which suits a distant data center.

The oplog: how copies stay in step

The primary records every write in a special capped collection called the oplog (local.oplog.rs). Each entry is a small, idempotent description of one change, with a timestamp. Secondaries continuously read the oplog of the primary (or of another secondary) and apply the entries in order.

Two properties of the oplog matter in practice:

  • It is capped. Old entries are overwritten when the oplog is full, so it covers a limited window of time. A secondary that is down for longer than that window cannot catch up from the oplog and must be completely resynchronized. The "oplog window" is therefore a number to watch, and you can resize the oplog on a running member.
  • Entries are idempotent. Applying the same entry twice gives the same result. That is what lets a secondary safely resume after a crash and also what lets change streams resume from a token.

The delay between a write on the primary and its appearance on a secondary is replication lag. It is usually milliseconds, but a slow disk or a heavy bulk load can stretch it to seconds or minutes. Reading from a lagging secondary returns stale data, so read preference and read concern deserve a deliberate choice.

Elections and the majority rule

Members send each other heartbeats every two seconds. If the secondaries do not hear from the primary for the election timeout (ten seconds by default), one of them calls an election. The election protocol is similar to the Raft algorithm: the candidate increases a number called the term, asks every member for its vote, and becomes primary if it gets votes from a majority of the voting members.

A member grants its vote only if the candidate's copy of the data is at least as up to date as its own. This rule is what keeps the most advanced member in front. And because there can only be one majority at a time, two primaries can never win in the same term.

The majority rule has a direct consequence for how many members you need:

Voting members Majority Failures tolerated
1 1 0
2 2 0
3 2 1
4 3 1
5 3 2
7 4 3

Two members tolerate no failure at all (if either dies, the survivor has one vote out of two), and four members tolerate no more than three do. That is why replica sets normally have an odd number of voting members, and why three is the usual minimum for production. A set can have up to 50 members, but only 7 of them may vote.

A primary that can no longer reach a majority steps down on its own and becomes a secondary. This is the safety valve for network partitions: the side of the cut that holds a majority elects a primary, and the minority side does not keep one.

Arbiters

An arbiter is a member that votes but stores no data. It exists to make the voter count odd when you cannot afford a third data-bearing server, for example two data centers plus one tiny arbiter machine. Arbiters are cheap but come with real costs. They add no durability, because there is no third copy. And with a primary, a secondary and an arbiter, losing the secondary leaves only the primary able to acknowledge writes, so writes with w: "majority" can stall. Current MongoDB documentation recommends data-bearing members instead of arbiters wherever possible.

The first TryIt: elections and the majority

The simulation below models members with logs. tick() stands for "enough time passed for heartbeats to time out": primaries without a majority step down, and any side with a majority and no primary holds an election. Members with the longest log win. Run it and follow what happens as members crash and return.

JS Crash members and watch elections

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Follow the story. A wins the first election and takes two writes. When A crashes, B and C hold two of three votes, so B is elected. When C also dies, B is alone: one of three votes is not a majority, so B steps down and writes are refused, even though B is healthy. Only when A returns does a majority exist again. B holds the newest data, so B is elected again and A catches up from it. This is exactly why the two-failure case on a three-member set is an outage, not a data-corruption risk.

Rollback: when acknowledged writes disappear

There is one subtle failure mode. Suppose the primary accepts a write, acknowledges it with the default w: 1 and then gets cut off from the network before any secondary copies it. The rest of the set elects a new primary that never saw the write and carries on. When the old primary comes back, its log has an operation that nobody else has. The member cannot keep it, because the new history won, so it undoes that operation in a rollback. MongoDB saves the rolled-back documents in a rollback folder as BSON files so that an operator can inspect or re-apply them.

This is not a bug; it follows from acknowledging a write before a majority has it. The fix is a write concern: with w: "majority" the primary only confirms a write once most members have it, and a write confirmed that way is never rolled back. Majority is the default in recent MongoDB versions for most deployments, but check yours, and see Read and Write Concerns for details and trade-offs.

The second TryIt: a partition and a rollback

This simulation cuts the primary off from the rest of the set. It shows a w: 1 write that is acknowledged and later lost, a w: "majority" write that is correctly refused, and the rollback when the network heals.

JS Partition, failover and rollback

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The w: 1 write on the isolated member reports success, with one acknowledgement. The w: "majority" write reports that it timed out, since only one of the two required members has it. In real MongoDB the write would stay in the old primary's log in both cases, and both ops, op2 and op3, are rolled back when the member rejoins. The acknowledged op2 is the dangerous one: your application was told it succeeded.

Member types and a deployment walkthrough

So far the members were interchangeable. In practice a replica set mixes members with different jobs, and each job is a few fields in the replica set configuration (rs.conf()). This section lists the member types, then walks through deploying a three-member set and handling failures.

Member types

Type Configuration What it does
Regular (data-bearing) priority: 1 (default), votes: 1 Holds data, votes, and can become primary.
Lower or higher priority priority: 0.5, priority: 2 Higher priority wins elections more often; use it to prefer the strongest server.
Priority 0 priority: 0 Holds data and can vote, but can never become primary. Good for a member in a distant data center.
Hidden priority: 0, hidden: true Invisible to applications (drivers do not route reads to it). Use it for backups, reports and analytics without disturbing live traffic.
Delayed priority: 0, hidden: true, secondaryDelaySecs: 3600 Applies the oplog one hour late, so it is a safety net against a bad deployment or an accidental drop.
Non-voting votes: 0 (with priority: 0) Holds data but does not vote. Needed once a set has more than 7 members.
Arbiter arbiterOnly: true Votes only, no data (see the Arbiters section above for the costs).

The limits to remember: a replica set can have up to 50 members, but only 7 voting members. Every non-voting member needs priority: 0.

Applications that need to read from specific members use tags and read preferences (for example reading from a member tagged as a nearby data center), and buildIndexes: false skips index building on a pure backup member. Both are advanced settings you can add later.

Deploying a three-member replica set

  1. Prepare three servers with the same MongoDB version, reachable by name from one another (use stable DNS names, not IP addresses that change), with the replication port open only between them and your application (see Security Hardening).
  2. Set the replica set name and an authentication key in every mongod.conf:
YAML
replication:
  replSetName: rs0
security:
  keyFile: /etc/mongodb/rs0.key    # the same file on every member, mode 400
  authorization: enabled
net:
  bindIp: 0.0.0.0                  # restrict with a firewall or a private network
  1. Start mongod on all three servers. Each runs alone and refuses elections until initiated.
  2. Initiate from exactly one member (once), then add the others:
JavaScript
rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "db1.internal:27017", priority: 2 },
    { _id: 1, host: "db2.internal:27017" },
    { _id: 2, host: "db3.internal:27017" }
  ]
})

rs.add({ host: "db4.internal:27017", priority: 0, hidden: true })   // a hidden backup member
rs.addArb("arb1.internal:27017")                                      // only if you truly need an arbiter
rs.remove("db4.internal:27017")
  1. Check health with rs.status() (one PRIMARY, the others SECONDARY, and health: 1 for each) and rs.conf() for the configuration.
  2. Create the admin user on the primary, then connect applications with a replica set connection string (mongodb://db1,db2,db3/?replicaSet=rs0), which lets the driver find the current primary after a failover.

To change a member's role later, edit the configuration and apply it. rs.reconfig must run on the primary, and it validates the result: a primary cannot be given priority: 0 while it is the only electable member, and a hidden member must have priority 0.

JavaScript
const cfg = rs.conf()
cfg.members[2].priority = 0
cfg.members[2].hidden = true
rs.reconfig(cfg)

When something fails

  • A secondary goes down. Nothing visible happens to applications. When it returns, it replays the oplog to catch up. If it was down longer than the oplog window (the time span the oplog covers), it can no longer catch up and does an initial sync: it clears its data and copies everything from another member. Size the oplog (replSetResizeOplog) to cover a day or more of downtime.
  • The primary goes down. The remaining members hold an election (see the TryIts above) and a new primary is chosen within seconds. Retryable writes in modern drivers resend an interrupted write once.
  • A planned restart or upgrade. Restart secondaries one by one, then ask the primary to hand over with rs.stepDown() and restart it last, so you get a controlled election rather than a surprise one.
  • A majority is lost (two of three members down). There is no primary, so writes stop. First try to restore the members. As a last resort on the surviving member, rs.reconfig(cfg, { force: true }) can forcibly shrink the configuration; this risks losing writes that were not replicated, so treat it as emergency surgery.

The model below checks a proposed configuration against these rules before you "apply" it. It is a stand-in written for this lesson and is not rs.reconfig, but the rules it enforces are real.

JS Validate a replica set configuration

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The standard set tolerates one failure. The delayed member adds a safety net without adding a voter. The bad configurations are rejected for the same reasons a real server would reject them (the even-voter check is advice rather than an error, since MongoDB allows an even count but gains nothing from it).

Connecting to a replica set

Applications should list several members and name the set, so the driver can discover the whole topology and follow the primary after a failover:

JavaScript
mongodb://db1.example.com:27017,db2.example.com:27017,db3.example.com:27017/shop?replicaSet=rs0

The driver connects to any listed member, asks for the member list and the current primary, and keeps monitoring them. When an election happens, the driver finds the new primary on its own, and with retryable writes (on by default in current drivers) a single write that failed because of a failover is retried once automatically. Your code should still handle errors for operations that outlast the failover.

Atlas and many hosted services use the mongodb+srv:// form, where one DNS name stands for the whole set. The driver looks up the member hosts and options in DNS records (Installing MongoDB and Using mongosh shows the form). Never connect your application to one member only, because then a failover turns into an outage.

Useful options in the connection string include readPreference (for example secondaryPreferred), w and retryWrites. Run rs.status() often while learning: the stateStr, optimeDate and health fields tell you who is primary and how far behind each member is.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A replica set has five voting members. How many can fail while the set still elects a primary?

    The majority of five is three, so up to two members can be lost. With three failures only two voters remain, which is not a majority.
  2. 2. What is the oplog?

    The oplog (local.oplog.rs) records every write as an idempotent entry. Secondaries copy and apply it, and a member that falls out of its window must resync.
  3. 3. Why can a write acknowledged with w: 1 be lost during a failover?

    w: 1 confirms after the primary alone has applied the write. If the primary is cut off before replication, a rollback removes it later. w: majority prevents this.
  4. 4. Why does a three-member set with one arbiter and one data-bearing secondary make majority writes risky?

    An arbiter votes but holds no data, so it cannot acknowledge a write. With one data member down, w: majority writes can stall even though elections still work.

Summary

  • A replica set keeps copies of the data on several members: one primary takes writes, secondaries replay its oplog.
  • The oplog is a capped log of idempotent operations; a member that stays down longer than the oplog window must resync.
  • Elections need votes from a majority of voting members, so use an odd number (three is the usual minimum) and expect n - majority tolerated failures.
  • A primary that sees no majority steps down; the majority side elects a new primary within seconds.
  • Writes acknowledged before a majority has them can be rolled back after a failover; w: "majority" avoids that.
  • Arbiters vote without storing data and are best avoided; connect with several hosts and replicaSet=, or an SRV string, so drivers follow failovers.

Next, learn how MongoDB spreads one dataset across many replica sets in Sharding.