Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 42 ยท High Availability and Scaling

High Availability with Sentinel

Learn Redis Sentinel: monitoring, SDOWN and ODOWN, quorum versus majority, automatic failover, how clients find the master and how split-brain is limited.

Replication gives you copies of your data, but it does not promote anyone by itself. If the master dies at 3 a.m., the replicas just sit there waiting. Redis Sentinel is the supervisor that fills that gap. It is a set of separate processes that watch a master and its replicas, agree that the master is really gone, elect one of the replicas as the new master, reconfigure the others, and tell clients where the new master lives.

Sentinel does not shard data: every Redis node still holds the whole dataset. It gives you high availability for a single dataset. When the data no longer fits on one machine, you need Redis Cluster instead.

Redis and Sentinel cannot run on this page, so the configuration and redis-cli snippets are read-only. The runnable examples are plain JavaScript stand-ins for the two ideas that matter most: how sentinels vote, and how a replica is picked for promotion. They model the rules, not the Sentinel network protocol.

What Sentinel does

Monitor Pings the master and replicas.
Agree Sentinels vote that it is down.
Fail over Promotes the best replica.
Announce Clients ask where the master is.
1 / 2

A minimal setup

A Sentinel is a normal Redis binary started in a special mode (redis-sentinel sentinel.conf or redis-server sentinel.conf --sentinel). It listens on port 26379 and needs a config file it is allowed to rewrite, because it stores what it learns there:

Text
# sentinel.conf (the same on all three sentinels)
port 26379
sentinel monitor mymaster 10.0.0.5 6379 2
sentinel auth-pass mymaster the-master-password
sentinel down-after-milliseconds mymaster 5000
sentinel failover-timeout mymaster 60000
sentinel parallel-syncs mymaster 1

The important line is sentinel monitor <name> <ip> <port> <quorum>. You only list the master; Sentinel finds the replicas by asking the master, and finds the other sentinels through a publish/subscribe channel on the master. down-after-milliseconds is how long the master may stay silent before one sentinel considers it down. parallel-syncs limits how many replicas resynchronize with the new master at once, so that you do not take every read replica offline at the same moment.

From SDOWN to ODOWN to failover

Failure detection happens in two steps, and the difference between them is the heart of Sentinel.

  1. Subjectively down (SDOWN). One sentinel stops getting valid replies from the master for down-after-milliseconds. That is only one observer's opinion; the problem may be on that sentinel's own network.
  2. Objectively down (ODOWN). The sentinel asks the others. If at least quorum sentinels report the master as down, the master is declared objectively down and a failover may start.

The quorum only decides when a master is considered failing. Actually performing the failover needs more: the sentinel that wants to run it must be elected leader by a majority of all configured sentinels, not just by the quorum. With five sentinels and quorum 2, two sentinels can agree the master is down, but three votes are still needed to authorize the failover. This is what stops a small isolated group from promoting its own master.

flowchart TD
    A["Sentinel misses replies"] --> B["SDOWN: my own opinion"]
    B --> C{"quorum sentinels<br/>agree?"}
    C -- no --> D["Keep watching"]
    C -- yes --> E["ODOWN"]
    E --> F{"majority of all<br/>sentinels vote for me?"}
    F -- no --> G["Another sentinel or a later epoch tries"]
    F -- yes --> H["Promote a replica and reconfigure"]

Quorum detects the failure, but a majority of all sentinels must authorize the failover.

The TryIt below applies exactly those rules to five scenarios. Each sentinel has a list of sentinels it can talk to and a flag for whether it can reach the master.

JS Quorum versus majority

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Study the fourth scenario. With quorum 1, every isolated sentinel declares ODOWN on its own, yet none of them can win an election, so no promotion happens. Sentinel is protected against that mistake by the majority rule; a low quorum just makes it jumpy about noticing failures. A common choice is the number of sentinels divided by two, rounded up (2 of 3, 3 of 5).

Choosing the new master

Once a sentinel wins the election, it picks a replica. It first removes candidates that are unsuitable: replicas that were disconnected from the old master for too long (so their data is too old), and replicas with replica-priority 0, which means "never promote me". It then ranks the rest by, in order:

  1. lowest replica-priority (default 100),
  2. highest replication offset, meaning the replica that has processed the most of the master's stream and so has lost the least data,
  3. smallest run ID, as a deterministic tie-breaker.

The winner receives REPLICAOF NO ONE, the failover increments the configuration epoch (a version number that makes the newest configuration win everywhere), the other replicas are told to follow the new master (parallel-syncs at a time), and the old master is marked as a replica so that it rejoins as one when it returns.

JS Picking the replica to promote

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice that replica-c has the best priority (10), so it wins even though its offset is lowest and it will lose the most data. Priority is an operator decision that overrides the offset, so use it deliberately, for example to prefer a replica in the same data center as your application servers.

How clients find the master

Applications must not hard-code the master's address. A Sentinel-aware client library is configured with the list of sentinel addresses and the master name. On connect it asks a sentinel:

Text
127.0.0.1:26379> SENTINEL get-master-addr-by-name mymaster
1) "10.0.0.5"
2) "6379"
127.0.0.1:26379> SENTINEL replicas mymaster

and then connects to that address. A good client also verifies the node is actually a master (ROLE), reconnects through Sentinel when a connection drops, and may subscribe to sentinel's +switch-master event to react immediately. Libraries such as ioredis, Lettuce, Jedis and redis-py support this natively; see Using Redis from Node.js, Python, PHP and Java. A plain client that only knows one IP will keep talking to the dead master, so Sentinel is useless without a Sentinel-aware client or a proxy that does the discovery.

Split-brain and its limits

A split-brain happens when a network partition leaves the old master alive on a small side while the sentinels on the other side promote a replica. Now two masters accept writes, and when the partition heals the old master is demoted and its writes made during the partition are discarded. Sentinel cannot prevent that completely, because replication is asynchronous. You can shrink the damage on the master with:

Text
min-replicas-to-write 1
min-replicas-max-lag 10

With these, a master that cannot reach any replica for 10 seconds stops accepting writes, so a stranded old master loses at most a few seconds of writes and then becomes read-only, at the cost of refusing writes whenever its replicas are away.

Operating notes

  • Test failover on purpose: SENTINEL FAILOVER mymaster forces one without a crash, and killing the master process shows the real timing (detection time is roughly down-after-milliseconds plus the election).
  • Put each sentinel on a different machine or availability zone, and keep down-after-milliseconds large enough that a slow BGSAVE or garbage in the network does not cause false failovers.
  • Sentinel keeps no application data and uses little memory, but it does need its config file to be writable.
  • Secure sentinels like Redis itself: they listen on a TCP port that can reconfigure your topology (see Securing Redis).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the difference between SDOWN and ODOWN?

    A single sentinel can only say 'I cannot reach it'. When quorum sentinels say so, the master is objectively down and a failover can start.
  2. 2. With 5 sentinels and quorum 2, how many sentinels must vote for the one that runs the failover?

    The quorum detects the failure, but electing the failover leader needs a majority of all configured sentinels.
  3. 3. Which replica does Sentinel prefer for promotion, assuming it is eligible?

    Lower replica-priority wins, then the larger replication offset (less data lost), then the smaller run ID. Priority 0 means never promote.
  4. 4. Why does a Sentinel-managed setup need Sentinel-aware clients?

    After a failover the master has a new address. Clients learn it with SENTINEL get-master-addr-by-name or by following sentinel events.
  5. 5. What does Sentinel NOT provide?

    Every node still holds the whole dataset. Sharding is the job of Redis Cluster.

Summary

  • Sentinel is a group of processes (port 26379) that monitor a master and its replicas, run the failover and tell clients the new address.
  • One sentinel notices a failure (SDOWN); the quorum makes it ODOWN; a majority of all sentinels must elect the leader that performs the failover.
  • Run an odd number (at least three) on separate hosts.
  • The new master is chosen by replica-priority, then replication offset, then run ID; the configuration epoch makes the newest decision win.
  • Clients must discover the master through Sentinel; min-replicas-to-write limits the writes lost to a split-brain.
  • Sentinel gives high availability, not sharding.

Next, spreading the data itself over several masters: Redis Cluster and Sharding.