Skip to content
Tutorials Center
My learning

Topics

Lesson 44 of 60 ยท Non-functional Testing Concepts

Reliability, Recovery and Disaster Testing

Test how software behaves when things fail: failover, retries, timeouts, chaos experiments and backup restore drills, with a runnable flaky-service stand-in.

Everything breaks eventually: a server crashes, a network link drops, a disk fills up, a partner's API stops answering. Functional testing checks that the software works when conditions are good. Reliability testing asks what happens when they are not, and recovery testing asks how quickly and how completely the system gets back to normal. These are the questions that decide whether an outage is a minor blip or a front-page incident.

This lesson covers the main ideas: reliability measures, failover, timeouts and retries, chaos experiments, and backup and restore drills. The real mechanisms involve servers and infrastructure, so the code is a stand-in: a simulated flaky service with a fake clock and a miniature backup drill. The logic is the same as in real systems; only the machines are imaginary. Reliability is one of the non-functional test types, and it pairs naturally with performance testing, since overloaded systems are often the ones that fail.

Reliability, availability and recoverability

These three words are related but not identical.

  • Reliability: the probability the system works correctly over a period of time. A common measure is MTBF, the mean time between failures.
  • Availability: the share of time the system is usable. Often quoted in "nines": 99.9 percent allows roughly 8.8 hours of downtime a year, and 99.99 percent about 53 minutes.
  • Recoverability (or maintainability of service): how fast and how fully the system returns after a failure. Common measures are MTTR, the mean time to repair, and two targets set for disaster planning: the RTO (recovery time objective, how long an outage may last) and the RPO (recovery point objective, how much recent data you can afford to lose).

A system that fails rarely but takes a day to recover can be less available than one that fails weekly but heals in seconds. Testing both sides matters.

Plan for failure

Things break Servers, networks, disks, partners.
Contain it Timeouts, retries, failover.
Recover Restart, restore, resume.
Prove it Test the failure, not only success.
1 / 2

Failover and redundancy

Redundancy means keeping spare capacity: two database servers instead of one, several application instances behind a load balancer, data stored in more than one place. Failover is the switch from a failed component to its spare. Typical test ideas:

  • Stop the primary database. Does the standby take over, and how long does it take?
  • Kill one of three application instances while users are active. Do requests keep working, and are logged-in sessions kept?
  • After the failed component returns, does the system switch back cleanly, or does it end up with two parts both believing they are in charge?

A failover that has never been tested is a hope, not a feature. Watch for data loss or duplicated work at the moment of the switch, such as an order submitted twice or a payment charged and not recorded.

Timeouts and retries

When one service calls another, the callee may be slow or silent. Two simple protections are standard:

  • A timeout limits how long the caller waits. Without one, a hung dependency can freeze every thread of the caller too and spread the outage.
  • A retry repeats a failed request, because many failures are brief (a dropped packet, a restart in progress).

Retries have rules that testers should check:

  • Retry only temporary errors. Retrying "invalid password" will never help.
  • Limit the number of attempts.
  • Wait between attempts, and increase the wait each time (exponential backoff), usually with a little randomness (jitter) so that thousands of clients do not all retry at the same instant.
  • Make repeated requests safe. If a payment request is retried after the first one actually succeeded, the customer must not be charged twice. Well-designed APIs use an idempotency key so the second request is recognized as a repeat.
sequenceDiagram
  participant C as Client
  participant S as Service
  C->>S: request (attempt 1)
  S-->>C: error, temporary
  Note over C: wait 100 ms
  C->>S: request (attempt 2)
  S-->>C: error, temporary
  Note over C: wait 200 ms
  C->>S: request (attempt 3)
  S-->>C: success

Retries with a growing wait; each failure is temporary, the third attempt succeeds.

Try it: a flaky service with retry and backoff

The service below fails on a fixed schedule that you choose, so the run is deterministic. The retry function waits longer after each failure, using a fake clock (a plain number) instead of real waiting so the example finishes instantly. It also refuses to retry errors that are not temporary. Change the failure schedule to see when the call succeeds and when the client gives up.

JS Retry with exponential backoff

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The four scenarios are exactly the test cases a tester would write: success first time, recovery after temporary failures, giving up cleanly, and not wasting time on an error that retrying cannot fix. Notice that the last failing attempt does not wait afterward; waiting before giving up would only delay the error report.

Chaos testing

Chaos testing (chaos engineering) deliberately injects failures into a running system to see whether it copes: terminate a server at random, add network delay, make a dependency return errors, fill a disk. The idea is to find weaknesses on your own schedule rather than at 3 a.m. during a real outage.

Good chaos experiments follow a discipline:

  1. State the steady state, such as "orders per minute stay above 90 percent of normal".
  2. Form a hypothesis: "if one cache node dies, users see no errors."
  3. Inject a small, controlled failure, ideally in a test or staging environment first, with a way to stop instantly.
  4. Compare with the steady state, then fix what you learn.

This is advanced work, usually done by engineering teams, and it is never a license to break things at random. As a tester, the useful habit is simpler: for every dependency in a feature, ask "what should the user see if this is down, slow or returns garbage?" and write a test for the answer.

Chaos engineering in practice

The short section above gave the definition and the four-step discipline. This section adds the vocabulary and the practices that turn a clever idea into a safe routine.

Where it came from

Chaos engineering became popular after a large streaming company moved to the cloud and built a tool, Chaos Monkey, that randomly terminated production instances during working hours. If the service survived, engineers learned that redundancy worked; if it did not, they learned it while they were at their desks and able to fix it. The idea grew into a family of tools (a "Simian Army" of monkeys for latency, security and whole-region failures) and then into a discipline described in the Principles of Chaos Engineering. Today commercial and open-source tools inject failures into containers, virtual machines, networks and cloud services.

Key terms

  • Steady state: measurable normal behaviour, such as orders per minute or the 95th percentile response time. It is how you recognise harm.
  • Hypothesis: "if X fails, the steady state stays within Y."
  • Experiment: the deliberate injection of a fault.
  • Blast radius: how much of the system and how many users the experiment can affect. Start tiny (one instance, 1% of traffic, one test account) and grow.
  • Abort condition (kill switch): a rule that stops the experiment automatically if harm exceeds a threshold.
  • Game day: a scheduled session where the team runs experiments together, watches dashboards and practises incident response.

Types of faults to inject

Category Examples
Resource Kill a process, instance or container; fill the disk; use all CPU or memory
Network Add latency, drop packets, cut a dependency, break DNS
Dependency Make a third-party API return errors or time out
State Corrupt or delay data, expire certificates, skew the clock
Infrastructure Lose a zone or region, fail a database primary

A safe experiment, step by step

  1. Pick a question that matters ("what happens if the payment provider is down for ten minutes?").
  2. Define the steady state with metrics you already collect.
  3. Make a hypothesis and write it down, including the expected user experience.
  4. Check that you can observe the system (dashboards, logs, alerts), and that you can stop the experiment immediately.
  5. Start small in a test or staging environment, with a small blast radius and a team ready to respond.
  6. Inject the fault and compare against the steady state.
  7. Stop if the abort condition is hit.
  8. Learn and fix: if the hypothesis failed, you found a weakness. If it held, you gained confidence; increase the blast radius or move to production carefully.
  9. Automate experiments that proved useful and run them regularly.
flowchart LR
  H["Hypothesis<br/>steady state holds"] --> I["Inject small fault"] --> O["Observe metrics"]
  O --> D{"Steady state kept?"}
  D -- yes --> G["Gain confidence,<br/>widen blast radius"]
  D -- no --> F["Weakness found:<br/>fix and repeat"]
  O -. "abort condition hit" .-> K["Stop immediately"]

Every experiment has a way out.

Chaos in CI and delivery pipelines

Chaos experiments are not only for production. Teams add small, deterministic ones to the pipeline:

  • Run the integration tests while a mock dependency returns errors or timeouts and check that fallbacks work.
  • Start the test environment, kill one container and assert that the health check recovers and requests still succeed.
  • Add latency injection to a proxy in front of a service in the test environment, then run the API tests with timeouts.
  • Run a larger scheduled experiment (nightly or weekly) in staging, with results published as a report.

Keep those in CI deterministic and fast; leave random, wide-reaching experiments to game days and scheduled runs.

Cautions

  • Chaos testing needs good monitoring first; without observability you cannot tell what you broke.
  • Do not run experiments on systems that have no redundancy: you already know the result.
  • Get approval and tell the people who could be affected (support, on-call).
  • Treat results as learning, not blame.
  • It complements, and does not replace, recovery tests and restore drills.
JS A chaos experiment on a replicated service

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Experiment B is the classic discovery: the cluster looked redundant, but a missing health check meant a third of the requests still went to the dead node. Experiment C shows why you do not run chaos tests on systems that cannot survive them: the answer is already known.

Backups and restore drills

A backup you have never restored is only an assumption. Recovery testing includes:

  • Backup checks: do backups run on schedule, and do they finish without errors?
  • Restore drills: actually restore a backup into a clean environment and verify that the data is complete and the application works against it.
  • Timing: measure how long the restore takes, because that is your real RTO.
  • Data age: compare the newest restored record with the failure time; the gap is your real RPO.
  • Disaster scenarios: lose a whole site or region, and follow the written runbook with the people who would really do it.

Try it: a restore drill with verification

This stand-in backs up a small data set, simulates losing part of it, restores from the backup, and verifies the result with a simple checksum, the same idea real tools use to detect a corrupt or incomplete restore. It also reports how many records were created after the backup, which would be lost: the RPO in miniature.

JS Backup, damage, restore, verify

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last two lines show why verification matters: a restore that silently differs from the backup fails the check. The report also makes the RPO concrete: the two accounts opened after 02:00 are gone, and the business must decide whether losing that window is acceptable. If it is not, backups need to run more often, or the system needs continuous replication.

Writing recovery test cases

Use the same structure as any other test case, with extra attention to the failure injected and the measurement that decides pass or fail:

Field Example
Failure injected Primary database stopped during checkout
Expected behavior Standby promoted; no completed order lost; user sees a retry message, not a crash
Measure Failover time under 60 seconds; zero duplicate orders
After recovery Original node rejoins as standby; monitoring raises and clears an alert

Because these tests are risky, agree on the environment, the schedule and a rollback plan in your test plan first.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does RPO (recovery point objective) describe?

    RPO is about data: the age of the newest data you can restore. RTO is the one about time to recover.
  2. 2. Which error should a retry mechanism NOT retry?

    Permanent errors such as invalid credentials will fail every time. Only temporary failures are worth retrying.
  3. 3. Why add a growing delay (exponential backoff) between retries?

    Immediate repeated retries can pile onto a service that is already overloaded. Waiting longer each time spreads out the load.
  4. 4. What makes a backup trustworthy?

    Only a successful restore proves a backup is usable. A backup on the same server also fails with that server.

Summary

  • Reliability is about working over time, availability about the share of time usable, and recoverability about how fast and completely the system heals (MTTR, RTO, RPO).
  • Test failover by stopping components on purpose and checking for lost or duplicated work.
  • Timeouts stop callers from hanging; retries should be limited, spaced out with backoff, applied only to temporary errors and safe to repeat.
  • Chaos testing injects small, controlled failures with a hypothesis and a steady-state measure.
  • A backup counts only after a verified restore drill; measure real restore time and data loss.

Next: When and What to Automate.