Skip to content
Tutorials Center
My learning

Topics

Lesson 47 of 57 ยท Architecture and Reliability

High Availability, Backup and Disaster Recovery

Learn SLAs and composite SLA math, availability zones vs regions, Azure Backup, Site Recovery, RPO and RTO, and chaos testing, with calculators to try.

Everything fails eventually: a disk, a rack, a datacenter, a bad deployment, a person with too many permissions. Reliability is the work of deciding which failures you will survive, how quickly you will recover from the rest, and proving it before a real incident does. Three words cover most of the vocabulary: high availability (keep running through small failures), backup (get your data back after damage), and disaster recovery (bring the whole workload back after a large failure).

This lesson covers SLAs and the arithmetic of combining them, availability zones and regions, Azure Backup and Azure Site Recovery, the two numbers every recovery plan needs (RPO and RTO), and chaos testing. The real services need a subscription, so their commands are read-only. The runnable examples are calculators and simulators that use the same math, with invented numbers, so you can check a design on paper before you pay for it. This lesson builds on The Azure Well-Architected Framework and Disks and Redundancy.

SLAs and what they promise

A service level agreement (SLA) is a published availability target for a service, written as a percentage of time in a month. Microsoft states one for most services, and the number often depends on how you configure the service: a virtual machine alone has a lower SLA than the same workload spread over availability zones.

A percentage hides how much time it allows you to lose. Using a 30-day month (43,200 minutes):

SLA Allowed downtime per month Allowed downtime per year
99% about 7.2 hours about 3.65 days
99.9% about 43 minutes about 8.8 hours
99.95% about 22 minutes about 4.4 hours
99.99% about 4.3 minutes about 53 minutes

An SLA is not a guarantee that the service will be up. It is a contract: if Microsoft misses the target, you can claim a service credit, which is a discount on your bill, not compensation for lost revenue. Read the SLA for each service you depend on, because the fine print defines what counts as downtime and what you must configure (for example two or more instances).

Your own target is a service level objective (SLO), and it should come from the business: what does an hour of downtime cost? A 99.99% target costs far more to build than 99.9%, so choose the number you actually need.

Composite SLA math

Real applications use several services, and their availability combines. There are two rules.

Components in a chain. If the application needs the web tier and the database and storage, then all of them must be up. Multiply the availabilities. Three components at 99.95%, 99.99% and 99.9% give 0.9995 x 0.9999 x 0.999, about 99.84%. The result is always lower than the weakest link, and every extra dependency lowers it.

Redundant copies. If either of two independent copies can serve a request, the system fails only when both fail. Multiply the failure probabilities: 1 - (1 - a) x (1 - b). Two copies at 99.84% give about 99.9997%.

flowchart LR
    U["Users"] --> FD["Front door<br/>or Traffic Manager"]
    FD --> R1["Region A stack<br/>99.84%"]
    FD --> R2["Region B stack<br/>99.84%"]
    R1 --> D1[("Data")]
    R2 --> D2[("Data")]

The chain inside each region is multiplied; the two regions are combined as redundant copies; the traffic router sits in front as one more link in the chain.

The redundant-copy formula assumes the copies fail independently and that something moves traffic to the survivor. Two copies in the same zone share power and networking, so they are not independent. The router that fails over is itself a component with its own SLA, so it multiplies into the total. And if the second region needs a manual, rehearsed procedure, the calculation says nothing about how long that takes. Treat composite SLAs as a rough upper bound, never as a promise.

A composite SLA calculator: a stand-in

This program is not Azure. It uses invented component SLAs (the real numbers change, so always read the current SLA for each service) and applies the two rules.

JS Composite SLA and downtime calculator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice how one region misses 99.95% even though the web tier alone meets it, and how the router becomes the new ceiling once you have two regions. That is the practical lesson: redundancy raises the ceiling of the parts you duplicate, but whatever is left in a chain limits you.

Zones and regions

Azure gives you two physical levels of redundancy (see Azure at a Glance):

  • Availability zones are separate datacenter groups inside one region, each with independent power, cooling and networking, connected by a fast low-latency network. A zone-redundant service spreads copies across zones and survives the loss of one zone with little or no action from you. Many services offer a zone-redundant setting, such as zone-redundant storage (Standard_ZRS), zone-redundant App Service plans, or a zone-spread scale set. Some regions have no zones, so check the region you chose.
  • Regions are hundreds of kilometers apart. Using a second region protects you from a region-wide event, at a higher price and complexity, because data must be replicated across a long distance and traffic must be steered between regions. Many regions are paired to receive coordinated platform updates and to prioritize recovery.
Terminal
# Read-only examples: spreading across zones and storing data redundantly
az vm create --resource-group rg-demo --name vm-web-1 --image Ubuntu2204 --zone 1
az vm create --resource-group rg-demo --name vm-web-2 --image Ubuntu2204 --zone 2

az storage account create --resource-group rg-demo --name stdemozrs001 \
  --sku Standard_ZRS --location westeurope

A common ladder, from cheap to expensive: a single instance, several instances in one zone, instances across zones, active-passive in two regions, and active-active in two regions. Pick the lowest rung that meets the business target. Most workloads need zones, and only critical ones need a second region.

Active-active and active-passive

In active-active, both regions serve users all the time. Failover is fast and simple, because the second region is already warm, but data conflicts and double cost are your problem. In active-passive, the second region is idle or scaled down (a "warm standby" runs a small copy; a "cold standby" has only data and templates). It is cheaper but recovery takes longer. Infrastructure as code from Bicep and ARM Templates is what makes a cold standby feasible, because you can recreate the environment from files.

RPO and RTO

Two numbers define a recovery plan:

  • Recovery Point Objective (RPO): how much data you can afford to lose, measured as time. If the last good copy is 15 minutes old when the disaster hits, you lose up to 15 minutes of data. RPO is set by backup frequency or by replication lag.
  • Recovery Time Objective (RTO): how long you can be down before the workload is back. RTO is set by how much has to be rebuilt, restored or switched, and by how much of it is automated.
flowchart LR
    A["Last good copy"] -->|"RPO: data you lose"| B["Failure"]
    B -->|"RTO: time you are down"| C["Back in service"]

RPO looks backwards from the failure; RTO looks forwards.

Different parts of a workload usually have different numbers. A product catalog can be rebuilt (long RPO is fine), an order database cannot (RPO near zero), and the public website may need a short RTO while a reporting dashboard can wait a day. Write the numbers down per component; the cost follows directly from how small they are.

An RPO and RTO checker: a stand-in

The stand-in below simulates one failure at 14:40 and asks each recovery strategy what was lost and how long recovery took. All durations are invented, but the relationships are real: backups lose up to one interval of data, asynchronous replication loses the lag, and a restore takes much longer than a failover.

JS Check strategies against RPO and RTO

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The first two strategies fail the 15-minute RPO for the same reason: they only know about the world as it was at the last backup. To tighten RPO you either back up more often or replicate continuously. To tighten RTO you automate and keep the standby warm.

Azure Backup

Azure Backup is the managed service for point-in-time copies. Backups live in a vault, either a Recovery Services vault or the newer Backup vault, depending on the workload type. It protects virtual machines, Azure Files shares, SQL Server in VMs, blobs, managed disks and more. You attach a backup policy that defines the schedule (for example daily at 02:00) and how long to keep each copy (daily, weekly, monthly, yearly).

Terminal
az backup vault create --resource-group rg-demo --name rsv-demo --location westeurope

az backup protection enable-for-vm \
  --resource-group rg-demo --vault-name rsv-demo \
  --vm vm-web-1 --policy-name DefaultPolicy

az backup protection backup-now \
  --resource-group rg-demo --vault-name rsv-demo \
  --container-name vm-web-1 --item-name vm-web-1

Good habits:

  • Test restores. A backup you have never restored is a hope, not a backup. Schedule a restore drill into a separate resource group.
  • Keep copies away from the blast radius. Use geo-redundant vault storage or cross-region restore for important data, and turn on soft delete so an attacker or an accident cannot erase backups instantly. Immutability options add protection against ransomware.
  • Back up what is not already in code. Databases and file shares need backups; a stateless web tier can usually be redeployed from templates and images instead.
  • Managed databases such as Azure SQL and Azure Database for PostgreSQL have their own automated backups and point-in-time restore, with retention you configure (see Azure SQL Database).

Azure Site Recovery

Azure Site Recovery (ASR) is disaster recovery for machines. It continuously replicates virtual machines (Azure to Azure across regions, and also on-premises servers to Azure) to a target location. When the source fails you start a failover and Azure brings up the replicas, often using a recovery plan that defines the order (database first, then app servers, then web). You can run a test failover into an isolated network without disturbing production, which is how you practice. After the primary is healthy again you fail back.

Because replication is continuous, ASR gives an RPO of seconds to minutes and an RTO of minutes, far better than restoring a nightly backup. It costs per protected machine plus the storage and the standby network. Backup and Site Recovery solve different problems: backup protects against deletion and corruption (you need an older copy), while replication protects against loss of a location (you need a current copy elsewhere). A replicated corruption is still corruption, so serious workloads use both.

Chaos testing

Design documents describe the system you hope you built. Chaos engineering checks the system you actually have by injecting controlled failures: stop a VM, add network latency, fail a dependency, exhaust CPU, cut off a zone. Azure Chaos Studio provides managed faults and experiments you can run against resources such as virtual machines, AKS and Cosmos DB, with a clear scope and a way to stop.

A sensible practice:

  1. State a steady-state hypothesis: "with one zone down, 99% of checkout requests still succeed within two seconds".
  2. Run the experiment first in a test environment, with monitoring open (see Azure Monitor, Log Analytics and Application Insights).
  3. Start small and have an abort switch. Then widen the blast radius and move toward production during a planned window.
  4. Fix what you find, and repeat on a schedule, since architecture drifts.

Pair this with game days: a rehearsed disaster where the team follows the runbook, including a real failover and restore, and times it. The measured time is your real RTO. The number in the design document is only a target.

Site Recovery in practice, and scripting blob backups

Setting up Site Recovery

Azure Site Recovery (ASR) replicates machines to a second location so you can fail over after an outage. It works for Azure VMs (region to region), and also for on-premises VMware VMs, Hyper-V VMs and physical servers (on-premises to Azure). A typical setup:

  1. Create a Recovery Services vault in the target region.
  2. Choose the source and target and enable replication for each machine. For VMware and physical servers, deploy a configuration server (an appliance on your network) that talks to Azure; Azure-to-Azure needs no appliance, only an agent extension on each VM.
  3. Pick a replication policy: how often recovery points are created and how long they are kept. This sets your RPO.
  4. Build a recovery plan: an ordered group of machines with steps, for example "start the database first, run a script, then start the web servers". Recovery plans are the customized recovery part of the service, and can include manual steps and Azure Automation runbooks.
  5. Run a test failover into an isolated network regularly. It costs little and proves the plan works without touching production.
  6. During a real outage, fail over. When the primary site is healthy again, reprotect and fail back. If replication falls behind or breaks, use resynchronize to copy only the differences instead of starting over; check the replication health state and alerts to catch a failure early.

ASR is the disaster-recovery piece of a wider business-continuity plan. Azure Backup covers recovery from deletion and corruption, ASR covers a lost site; you usually want both.

Backing up files to blob storage with a script

Sometimes you want a simple scripted backup of a folder to a blob container, which you can restore later. The real script uses the Azure Storage Blob SDK for Python (install with pip install azure-storage-blob azure-identity). It is read-only here because it needs an account:

Python
import os
from azure.identity import DefaultAzureCredential
from azure.storage.blob import BlobServiceClient

service = BlobServiceClient("https://<account>.blob.core.windows.net",
                            credential=DefaultAzureCredential())
container = service.get_container_client("backups")
if not container.exists():
    container.create_container()

def backup(folder):
    for root, _, files in os.walk(folder):
        for name in files:
            path = os.path.join(root, name)
            blob_name = os.path.relpath(path, folder).replace(os.sep, "/")
            with open(path, "rb") as f:
                container.upload_blob(blob_name, f, overwrite=True)

def restore(folder):
    for blob in container.list_blobs():
        target = os.path.join(folder, blob.name)
        os.makedirs(os.path.dirname(target) or ".", exist_ok=True)
        with open(target, "wb") as f:
            f.write(container.download_blob(blob.name).readall())

Run backup("/data") on a schedule (cron or a timer-triggered Function) and restore("/restore") after a loss. Use Entra credentials rather than account keys, as in Managed Identities and Service Principals. Turn on blob versioning, soft delete and a lifecycle rule (see Storage Accounts and Blob Storage) so old backups age into cooler tiers and a bad overwrite can be undone. If the script fails with ModuleNotFoundError, the SDK is missing from the Python you ran: install it with python3 -m pip install azure-storage-blob so it lands in the same interpreter.

The stand-in below runs the same backup and restore logic against an in-memory "container", so you can see the path handling and the overwrite behavior.

JS Back up and restore with an in-memory container

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A request needs a web tier at 99.95% AND a database at 99.99% to be available. What is the combined availability, roughly?

    Components that are all required are multiplied: 0.9995 x 0.9999 is about 0.9994. The result is below the weakest component, and each added dependency lowers it further.
  2. 2. Which statement about RPO and RTO is correct?

    RPO looks back from the failure to the last good copy. RTO is the time until service is restored. You choose them per workload, and they drive cost.
  3. 3. Why is an Azure Backup copy NOT a replacement for Site Recovery or replication?

    Backup protects against deletion and corruption by giving you an earlier copy; replication keeps a current copy elsewhere for fast failover. Serious workloads often use both.
  4. 4. What does a zone-redundant service protect you from?

    Zones are separate datacenter groups within one region. They do not protect against a region-wide event or against logical errors such as deletion or bugs.
  5. 5. What is the main purpose of a chaos experiment?

    Chaos testing checks a hypothesis about the real system under controlled failure, and shows weaknesses before a real incident does.

Summary

  • An SLA is a contractual target with service credits; your own SLO should come from business impact.
  • Multiply the availabilities of required components in a chain; combine independent redundant copies with 1 - (1 - a) x (1 - b), and remember the traffic router is a link in the chain.
  • Availability zones survive the loss of one datacenter group; a second region survives a regional event at higher cost and complexity.
  • RPO is tolerated data loss and RTO is tolerated downtime; set both per component and design backward from them.
  • Azure Backup gives earlier copies (test the restores, use soft delete); Site Recovery replicates machines for fast failover (use test failovers).
  • Prove the design with chaos experiments and game days, and trust measured recovery times over estimated ones.

Next: Project: Static Web App with a Serverless API.