Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 65 ยท Load Balancing and Traffic

Health Checks and Failover

Learn active and passive health checks, liveness vs readiness, active-passive and active-active failover, failure detection and split brain, with simulations.

Redundancy only helps if the system notices a failure and reacts. Having three servers is useless if the load balancer keeps sending a third of the traffic to the dead one. Two mechanisms close the gap: health checks detect that something is wrong, and failover moves the work to something healthy.

Load balancing versus failover

The two ideas are related but different.

  • Load balancing spreads traffic across several servers that are all serving at the same time. The goal is capacity and efficiency, and it incidentally tolerates failures.
  • Failover keeps a standby ready and switches to it when the primary fails. The goal is continuity, not extra capacity.

In practice you combine them: a pool of active servers is balanced, and each pool or component also has a way to replace failed members.

Load balancing Failover
Purpose Share load, scale Survive failures
Standby servers idle? No, all serve Often yes (active-passive)
Trigger Every request A detected failure
Capacity gain Yes Not by itself

Failover designs

Active-passive vs active-active

Active-passive One serves, a standby waits. Simple, standby is idle.
Active-active All serve at once. Efficient, needs spare capacity.
Switch time Cold standby minutes, warm seconds, hot near zero.

Faster failover costs more money and complexity.

1 / 2
  • Active-passive (primary and standby): one node serves; a standby replicates its state and takes over if the primary fails, often by taking over a shared virtual IP. Simple and avoids conflicting writes, but the standby capacity is idle.
  • Active-active: every node serves traffic and absorbs the load of a failed node. Better utilization, but you must keep headroom (if two nodes each run at 80%, one failing overloads the other) and, for data, resolve conflicting writes.
  • Cold, warm and hot standby describe how ready the spare is. Faster failover needs more preparation and money.
  • Multi-zone and multi-region: spread across failure domains so a whole data center loss is survivable.

Health checks

A health check is a small test the balancer or orchestrator runs regularly to decide whether a server should receive traffic.

Active checks: the balancer sends a probe every few seconds: a TCP connect, an HTTP GET /health expecting 200, or a deeper check that runs a tiny database query.

Passive checks: the balancer watches real traffic. If a server returns many errors or times out, it is marked unhealthy, without any special probe. Passive checks react to real failures; active checks find problems even when no traffic arrives. Use both.

Liveness and readiness

Container platforms distinguish two questions:

  • Liveness: "is the process alive, or stuck and in need of a restart?"
  • Readiness: "is it ready to serve traffic right now?" A server that is starting, warming caches, or temporarily overloaded can be alive but not ready: it should stay in the pool's waiting room, not be restarted.

Keep health endpoints cheap and honest. A check that only returns 200 without testing anything hides real faults. A check that does a heavy query makes problems worse under load. And do not make the check depend on a shared dependency (such as the database) for liveness: if the database is down, killing every app server in a restart storm helps nobody.

Thresholds and flapping

A single failed probe should not eject a server (the network can hiccup). Typical rules: mark unhealthy after 3 consecutive failures, mark healthy again after 2 consecutive successes. This hysteresis stops a borderline server from flapping in and out of service. The price is detection delay: with checks every 5 s and 3 failures needed, up to 15 seconds of traffic can reach a dead server, so clients also need timeouts and retries to cover that window.

Python Detecting a failed server with health checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

B dies at t=20 but stays in the pool for three probe rounds (the probes at t=20, 25 and 30, so about 10 seconds), during which about a third of requests fail. Once removed, errors stop. When B is repaired it re-enters only after two successful probes. A retry with a different server would have hidden most of those failures from users.

Failure detection in distributed systems

Deciding that a node is dead is genuinely hard: a missing reply could mean a crash, a slow network, or an overloaded process. Common approaches:

  • Heartbeats: each node regularly sends "I am alive"; silence for longer than a timeout means suspect.
  • Timeouts: choose them from measured latency (a few times the p99), not by guess. Too short causes false alarms; too long delays recovery.
  • Phi-accrual and gossip detectors: compute a suspicion level from past heartbeat timing, or let nodes share their observations so a majority agrees.

You can never be sure a node is dead, only that you have not heard from it; designs must tolerate being wrong.

Split brain and fencing

If the standby decides the primary is dead while it is merely unreachable, both may act as primary: split brain. Two nodes accepting writes can corrupt data. Defenses:

  • Quorum or consensus: a node may become leader only if a majority of an odd-sized group agrees (for example, three or five nodes).
  • Fencing tokens or STONITH ("shoot the other node in the head"): ensure the old primary is cut off from storage or power before the new one starts.
  • Leases: a primary holds a time-limited lease and must stop acting when it expires.
Python Failover with a lease and fencing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The old primary wakes up believing it is in charge, but the storage rejects its stale token. This is how correct systems stay safe even when failure detection is imperfect.

Failover for each layer

Layer Typical mechanism
Load balancer Pair with a floating IP, or DNS or anycast across several
App servers Health-checked pool; a failed instance is replaced automatically
Cache Replicas or re-hashing, with the database as a fallback
Database Primary with replicas and automatic promotion
Region DNS or global traffic manager shifts users to another region

Two practices make failover trustworthy: practice it (game days and chaos testing, because untested failover usually fails when needed), and automate it while keeping a manual override. Also remember failback, the return to normal after the repair, which must be as careful as the failover itself. See also availability.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the key difference between load balancing and failover?

    Balancing is for capacity; failover is for continuity after failure, though the two are combined.
  2. 2. Why require several consecutive failed probes before removing a server?

    Thresholds add hysteresis, trading a little detection delay for stability.
  3. 3. What is the difference between liveness and readiness?

    A starting or overloaded server can be alive but not ready, so it should wait rather than be restarted.
  4. 4. What is split brain?

    Both nodes accept writes and can corrupt data; quorum, leases and fencing prevent it.
  5. 5. In active-active, why keep headroom?

    If every node runs near its limit, losing one overloads the rest and can cascade.

Summary

  • Redundancy needs detection and switching: health checks find failures, failover moves work.
  • Load balancing is about sharing load; failover is about continuity; real systems use both.
  • Combine active and passive health checks, use thresholds, and separate liveness from readiness.
  • Choose active-passive or active-active and a standby temperature by cost and recovery time.
  • Beware split brain: use quorum, leases and fencing tokens, and rehearse failover regularly.

Next: Rate Limiting Algorithms