Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 37 ยท Resilience

Fault Tolerance and Chaos Engineering

Build resilient microservices: redundancy and failover, finding critical components with FMEA, chaos experiments, self-healing, backups and disaster recovery targets.

Circuit breakers and retries handle individual bad calls. Fault tolerance asks a bigger question: when parts of the system fail, as they will, does the system as a whole keep working? A resilient system keeps delivering its important functions during failures and recovers quickly afterward, instead of assuming failures will not happen.

This lesson covers the design techniques (redundancy, isolation, self-healing), how to find out which components matter most, how to test resilience on purpose with chaos engineering, and how to plan for the failures you cannot absorb with disaster recovery.

Faults, failures and the goal

A fault is something going wrong inside a component (a crashed process, a full disk, a bad deployment). A failure is when users notice. Fault tolerance means faults do not become failures. The aims are to minimize downtime, limit the blast radius (how much is affected), recover automatically, and degrade gracefully when full service is not possible. Reliability is usually measured as availability: uptime / total time. A service that is down two hours in a week is up (168 - 2) / 168, about 98.8% of the time; "four nines" (99.99%) allows about four minutes per month.

Techniques

  • Redundancy. Run several copies of each service in different failure domains (servers, racks, availability zones, regions). Stateless services are easy: put replicas behind a load balancer (Service Discovery and Load Balancing). Data needs replication.
  • Active and passive replicas. In active-active, all copies serve traffic, so a failure just removes capacity. In active-passive (hot standby), a replica waits and takes over on failover; it is simpler for data but failover takes time and must be tested.
  • Multi-zone and multi-region deployment. Spreading across zones survives a data-center failure; spreading across regions survives a regional outage but adds latency, cost and consistency challenges.
  • Isolation and containment. Bulkheads, separate thread pools, separate databases per service (Data Ownership) and per-tenant limits stop one failure from spreading.
  • Health checks and self-healing. An orchestrator replaces crashed containers and removes unhealthy ones from rotation automatically (Containers, Docker Compose and Kubernetes).
  • Graceful degradation. Serve cached or reduced responses when a dependency is down.
  • Safe change management. Many outages come from deployments: use canary releases and fast rollbacks (Deployment Strategies and CI/CD).

Redundancy costs money and adds consistency problems, so apply it where it matters. That brings us to finding the critical parts.

Finding critical components

Not every service deserves the same investment. Techniques for deciding:

  • Dependency mapping. Draw which services call which; the ones many paths depend on are critical, and so are single points of failure (one database, one DNS provider, one gateway).
  • Business impact analysis (BIA). Ask what the business loses per hour of each function being down. Checkout may matter far more than recommendations.
  • Failure Mode and Effects Analysis (FMEA). List how each component can fail and score each mode for severity (how bad), occurrence (how likely) and detection (how hard to notice, higher is worse), each from 1 to 10. Multiply them to get a Risk Priority Number. Fix the highest numbers first.

Chaos engineering

Resilience that is never tested is a hope. Chaos engineering is the discipline of running controlled experiments on a system to learn how it behaves under failure, before a real incident does the experiment for you. The method:

  1. Define steady state: a measurable healthy condition, such as "99% of catalog calls succeed" or "p95 latency under 300 ms".
  2. Form a hypothesis: "if one catalog replica dies, steady state holds."
  3. Inject a realistic failure: kill an instance, add latency, drop packets, fail a dependency, exhaust CPU or fill a disk.
  4. Observe the metrics and compare with steady state.
  5. Fix what broke, and repeat. Start in a test or staging environment, with a small blast radius and a way to stop immediately, then move toward production with care.

The script below does a miniature experiment with a seeded random generator, so the result is repeatable. It kills one of the catalog service's replicas, compares one replica against three, and then ranks failure modes with an FMEA score.

JS A chaos experiment and an FMEA ranking

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With one replica, killing it breaks the hypothesis; with three, the other two absorb the traffic. The FMEA list shows that a rare but severe, hard-to-detect failure (a bad configuration pushed everywhere) can outrank a frequent, noisy one.

Tools exist to run experiments: Chaos Mesh and LitmusChaos for Kubernetes, AWS Fault Injection Service, Gremlin, and the original Chaos Monkey that randomly terminates instances. A game day is a scheduled exercise where the team injects failures and practices its response, including the people and runbooks, not just the software.

Backup and disaster recovery

Some failures cannot be absorbed by redundancy: an operator deletes a table, a bug corrupts data in every replica, a region is lost, ransomware encrypts everything. Replication copies mistakes too, so you also need backups: regular, versioned, stored separately, and tested by actually restoring them. Two targets drive the design:

  • RPO (recovery point objective): how much data you can afford to lose, measured in time. An RPO of five minutes means backups or replication must be at most five minutes behind.
  • RTO (recovery time objective): how long recovery may take. An RTO of one hour means the restore and failover process must reliably finish within an hour.

Tighter targets cost more: continuous replication to another region versus nightly backups restored by hand. Write the plan down, automate it, and rehearse it. A backup that has never been restored is not yet a backup.

Continuous improvement

Resilience is a loop, not a project: monitor (Logs, Metrics and Observability), run blameless post-incident reviews, turn every incident into a test or an automated fix, and re-run chaos experiments after significant changes. For the system-design perspective and interview framing, see Availability, Reliability and Maintainability.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the difference between a fault and a failure?

    Fault tolerance is about stopping faults from becoming user-visible failures.
  2. 2. In FMEA, how is the Risk Priority Number computed?

    Multiplying the three 1-10 scores ranks failure modes so the riskiest are fixed first.
  3. 3. What is the first step of a chaos experiment?

    Without a measurable steady state you cannot tell whether the injected failure had an effect.
  4. 4. A company can tolerate losing at most 5 minutes of data. Which target is that?

    The recovery point objective is the maximum acceptable data loss measured in time; RTO is the maximum acceptable recovery time.

Summary

  • Fault tolerance keeps faults from becoming failures through redundancy, failover, isolation, health checks and graceful degradation.
  • Rank components with dependency maps, business impact analysis and FMEA; protect single points of failure first.
  • Chaos engineering tests resilience on purpose: steady state, hypothesis, injected failure, observation, fix.
  • Replication does not replace backups; plan disaster recovery around RPO and RTO and test restores.
  • Treat resilience as a loop of monitoring, incident review and repeated experiments.

Next: Containers, Docker Compose and Kubernetes.