Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 65 ยท Thinking in Systems

Availability, Reliability and Maintainability

Learn the nines of availability, MTBF and MTTR, how redundancy raises uptime in series and parallel systems, and why maintainability matters, with Python.

Three words are used loosely in design discussions: availability, reliability and maintainability. They are related but different, and each has a number you can compute. Together they describe whether people can depend on your system.

  • Availability: the fraction of time the system is up and able to answer requests.
  • Reliability: the probability that it works correctly for a given period, without failing or returning wrong results.
  • Maintainability: how easily the system can be repaired, changed and operated.

A system can be available but unreliable (it answers, but sometimes with wrong data), or reliable but briefly unavailable (it never corrupts data, but goes down for maintenance every night).

Availability and the nines

Availability is usually written as a percentage of uptime, and engineers speak of "nines":

Availability Name Downtime per year Per month
99% two nines 3.65 days 7.3 hours
99.9% three nines 8.8 hours 43.8 minutes
99.99% four nines 52.6 minutes 4.4 minutes
99.999% five nines 5.3 minutes 26 seconds

Each extra nine cuts the allowed downtime tenfold and costs much more. Five nines leaves no room for slow human response or a manual failover; everything must be automatic. Choose the number from the business need: an internal report tool can tolerate three nines, a payment system needs more.

Python Downtime budget for each availability level

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "error budget" idea follows: if the SLO is 99.9%, you have about 43 minutes of failure a month to spend on risky deployments, experiments and surprises. Spend it wisely; once it is gone, freeze risky changes.

MTBF, MTTR and the availability formula

Two measurements describe failures:

  • MTBF (mean time between failures): average uptime before a failure.
  • MTTR (mean time to repair or recover): average time to restore service.

Availability = MTBF / (MTBF + MTTR)

This has an important lesson. You can raise availability by failing less often (hard and expensive) or by recovering faster (often cheaper). A server that fails once a month (MTBF about 720 hours) and takes 2 hours to fix is 99.7% available. If automation restores it in 5 minutes, it is about 99.99% available, without making the server any more reliable.

Python Failing less vs recovering faster

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Faster recovery matches or beats ten times better hardware here. This is why monitoring, automatic restarts, health checks and rehearsed runbooks are as important as quality components.

Redundancy: series and parallel

A request usually passes through several components. If they are all required, the system is a series, and availabilities multiply, so the whole is worse than any part. If components are interchangeable copies, they form a parallel group, which fails only when all copies fail, so the whole is better than any part.

  • Series: A = A1 x A2 x A3 ...
  • Parallel (n copies each with availability a): A = 1 - (1 - a)^n
flowchart LR
  U["User"] --> LB["Load balancer<br/>99.99%"]
  LB --> W1["Web 99.9%"]
  LB --> W2["Web 99.9%"]
  W1 --> DB[("Database<br/>99.95%")]
  W2 --> DB

Figure: two web servers in parallel; the balancer, the web pair and the database are in series.

Python Series and parallel availability

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two lessons appear. First, ten services at 99.9% each are only 99.0% together: every extra dependency lowers availability, which is a strong argument for keeping the critical path short. Second, redundancy helps only where you add it: the single load balancer and the single database now limit the whole chain, so they need redundancy too (see failover).

The parallel formula assumes failures are independent. Copies that share a power feed, a bug, a bad deploy or a rack fail together. Spread replicas across zones and regions, and stagger deployments.

High availability techniques

  • Redundancy: no single point of failure at any layer.
  • Failover: automatic switch to a healthy replica.
  • Health checks and automatic recovery: detect and replace bad instances.
  • Multiple zones and regions: survive data-center-level failures.
  • Graceful degradation: lose a feature (recommendations) but keep the core (checkout).
  • Safe deployments: rolling or canary releases with fast rollback.
  • Timeouts, retries and circuit breakers so one slow dependency does not stall everything.

Reliability

Reliability asks whether the system does the right thing consistently. Failures come from hardware (disks and machines fail), software (bugs, memory leaks) and people (a bad config change causes more outages than anything else). Reliable design assumes everything fails and plans for it: replicate data, validate inputs, make operations idempotent (safe to repeat) so retries do not duplicate work, and test failure on purpose.

Durability is the reliability of stored data: "once acknowledged, never lost". It is achieved by replication, checksums and backups, and is a separate target from availability.

Maintainability

Most of a system's cost is not building it; it is running and changing it for years. Maintainability has three parts:

  • Operability: easy for operators to keep running, with good monitoring, logs, runbooks and automation.
  • Simplicity: easy for new engineers to understand; avoid needless complexity.
  • Evolvability: easy to change as requirements change, with clear module boundaries and good tests.

Practical habits: small, well-named components with narrow interfaces; automated tests and deployments; dashboards and alerts that match the SLOs; documentation of why decisions were made. A design that is slightly less efficient but much easier to understand often wins over the long run. See also modularity for how boundaries help.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Roughly how much downtime per year does 99.99% availability allow?

    0.01% of a year is about 52.6 minutes.
  2. 2. A service has MTBF of 1000 hours and MTTR of 1 hour. Which change raises availability most cheaply?

    Availability = MTBF/(MTBF+MTTR), so cutting repair time is often easier than preventing every failure.
  3. 3. Three components, each 99.9% available, are required in series. The system availability is about:

    0.999 x 0.999 x 0.999 is about 0.997. Series dependencies reduce availability.
  4. 4. Why can redundant copies still fail together?

    The parallel formula assumes independent failures; shared causes break that assumption.
  5. 5. Which is part of maintainability?

    Maintainability is about how easily the system can be run, understood and changed.

Summary

  • Availability is uptime; reliability is correctness over time; maintainability is how easily the system is run and changed.
  • Each extra nine cuts downtime tenfold and costs more; choose the target from the business need.
  • Availability = MTBF / (MTBF + MTTR): fast recovery is as valuable as fewer failures.
  • Series dependencies multiply availability down; parallel copies raise it, if failures are independent.
  • Design for failure, keep the critical path short, and keep the system simple to operate.

Next: Communication Protocols