Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 65 ยท Reliability and Operations

High Availability and Failover

Learn availability in system design: nines and downtime math, series versus parallel components, redundancy, active-passive and active-active failover, with Python calculators.

A system is available when it can serve requests correctly at the moment someone makes one. High availability (HA) means designing so that individual failures, which are certain to happen, do not become outages. Disks die, servers reboot, networks drop packets, whole data centers lose power. HA is the discipline of surviving all of that with little or no visible downtime.

This lesson gives you the math (the famous "nines"), the key idea (redundancy with no single point of failure) and the main failover designs.

Measuring availability

Availability is the fraction of time a system works:

availability = uptime / (uptime + downtime)

It is usually quoted in "nines":

Availability Downtime per year Downtime per month
99% (two nines) 3.65 days about 7.3 hours
99.9% (three nines) 8.76 hours about 43.8 minutes
99.99% (four nines) 52.6 minutes about 4.4 minutes
99.999% (five nines) 5.26 minutes about 26 seconds

Each extra nine cuts allowed downtime by ten times and typically costs much more engineering effort. Choose a target based on what an outage costs the business, not on bragging rights, and remember it is a promise measured over a period (an SLO, see Monitoring, Logging and Tracing).

Python From nines to allowed downtime

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two related terms: MTBF (mean time between failures) and MTTR (mean time to repair). Availability equals MTBF / (MTBF + MTTR). You can raise availability by failing less often or, usually more cheaply, by recovering faster: automated detection and failover shrink MTTR from hours to seconds.

Series and parallel components

A request often passes through many components. The math differs depending on how they are arranged.

  • In series (all must work): multiply availabilities. Two 99.9% components give 0.999 ร— 0.999 = 99.8%. Every added dependency lowers availability.
  • In parallel (any one is enough): the system fails only if all copies fail. Two 99% servers give 1 - 0.01 ร— 0.01 = 99.99%.
Python Why redundancy raises availability

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The lesson of the arithmetic: find the weakest link, and give it redundancy. After the servers are replicated, the single database or load balancer becomes the limit. That is the single point of failure (SPOF): any component whose failure takes the system down. HA design is a hunt for SPOFs.

Redundancy

Levels of redundancy

Server Several app instances behind a load balancer.
Zone Spread across data centers in a region.
Region Another city, for disasters.

Higher levels cost more and protect against bigger failures.

1 / 2

Redundancy is only useful when failures are independent. Two servers in the same rack on the same power supply fail together. Spread copies across availability zones (separate data centers with independent power and network) and, for the highest tiers, across regions.

Failover: active-passive and active-active

Failover is switching work from a failed component to a healthy one. It needs three things: redundancy, a way to detect failure (health checks, heartbeats) and a way to redirect traffic.

Active-passive (hot standby). One node serves traffic; a standby waits. When health checks fail, the standby is promoted (for a database) or a floating IP moves. It is simple and avoids conflicts, but the standby is idle capacity, and failover takes seconds to minutes. A warm standby runs but does not serve; a cold standby must be started or restored, which is slow.

Active-active. All nodes serve traffic, so capacity is used fully and failover just means the load balancer stops sending to the dead node. It is the natural model for stateless web servers. For stateful data, it needs multi-leader or leaderless replication and conflict handling (see Database Replication).

flowchart TB
  C[Clients] --> LB{Load balancer<br/>health checks}
  LB --> A[App 1 โœ”]
  LB --> B[App 2 โœ”]
  LB -. removed .-> X[App 3 โœ˜]

The balancer stops routing to App 3 after it fails its health checks.

Python Health checks and automatic failover

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the gap: until the checks accumulate enough misses, some requests still reach the dead server. Tuning check interval and threshold trades faster detection against false alarms during load spikes. Clients should therefore also use retries and timeouts (see Retries, Timeouts and Dead Letter Queues).

Eliminating state from the app tier

Failover is trivial for stateless servers: any instance can serve any request. Keep sessions, uploads and caches in shared stores (a database, Redis, object storage) rather than on the server's disk or memory. Stateful tiers (databases, caches, queues) need replication, leader election and careful failover.

Beyond failover: designing for the whole failure

  • Graceful degradation: keep core features alive when parts fail: show cached recommendations if the recommender is down.
  • Disaster recovery (DR): define RPO (how much data you can afford to lose) and RTO (how long until you are back). They decide whether you need synchronous replication, hot standbys or just backups. Test restores regularly.
  • Chaos engineering: deliberately kill instances in production-like environments to prove failover works before a real outage does.
  • Avoid correlated failures: staggered deployments, no shared config pushed to every region at once.

An untested failover is a hope, not a feature. Many outages come from a standby that was never promoted in practice, or a failover script that depended on the very system that failed.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Two components, each 99.9% available, are both required for a request to succeed. The combined availability is about:

    In series, availabilities multiply: 0.999 x 0.999 = 0.998, so each extra dependency lowers the total.
  2. 2. Two independent servers, each 99% available, are in parallel. What is the availability?

    The system fails only if both fail: 0.01 x 0.01 = 0.0001, so availability is 99.99%.
  3. 3. What is a single point of failure?

    HA design means finding and duplicating every SPOF.
  4. 4. Which is simplest to fail over?

    Stateless servers can be removed from the pool and any other instance can take over immediately.

Summary

  • Availability is uptime over total time, quoted in nines; each nine cuts allowed downtime tenfold.
  • Series dependencies multiply (worse); parallel redundancy multiplies failure probabilities (better).
  • Find and remove single points of failure; spread copies over independent zones and regions.
  • Active-passive is simple with a recovery delay; active-active uses all capacity and suits stateless tiers.
  • Define RPO and RTO, degrade gracefully and test failover regularly.

Next: Retries, Timeouts and Dead Letter Queues