Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 45 ยท Production Practice

Reliability, Security and Architecture Patterns

Design reliable, secure Google Cloud systems: multi-zone design, backups and disaster recovery, VPC Service Controls and defense in depth, with an availability calculator.

Getting an app running is one skill. Keeping it running when a disk fails, a zone has an outage, a release goes wrong or someone steals a password is another. Google Cloud publishes an Architecture Framework (sometimes called the well-architected guidance) that groups this advice into pillars. You do not need to memorize it. You need to know the questions it asks, because they turn vague worries into concrete design decisions.

This lesson covers the patterns behind those questions: how to survive failures, how to recover from disasters, and how to layer security so one mistake is not fatal. The commands are read-only because they need a real project. The stand-in calculator lets you see why some designs are far more available than others.

The pillars as questions

Questions to ask of any design

Reliability What breaks, and how fast do we recover?
Security Who can reach what, and what if they are wrong?
Cost Are we paying only for what we need?
Operations Can we see, change and repair it safely?

Performance is the fifth pillar, and it often trades off against the others.

1 / 2

Design choices are trade-offs. Extra redundancy costs money, and stricter security costs convenience. The aim is to make those trades deliberately, based on what a failure would cost your business.

Reliability: plan for failure

Everything fails eventually: disks, machines, zones, networks and your own code. Reliability comes from removing single points of failure and from limiting how far a failure spreads.

Key patterns:

  • Spread across zones. A zone is a separate failure domain inside a region (regions and zones). Run at least two instances in different zones behind a load balancer, for example with a regional managed instance group. Cloud Run and many managed services already spread across zones for you.
  • Use health checks and autohealing. The platform replaces instances that fail checks, so recovery needs no human.
  • Keep servers stateless. Store sessions and files in managed services, so any instance can be replaced.
  • Use highly available data services. For example Cloud SQL high availability keeps a standby in another zone, and Spanner can span regions.
  • Fail gracefully. Use timeouts, retries with backoff, and circuit breakers, so one slow dependency does not take everything down.
  • Release carefully. Canary and gradual rollouts (Cloud Build and Cloud Deploy) limit the blast radius of a bad change.

Availability and nines

Availability is the share of time a service works. "Three nines" means 99.9%, which allows about 8.8 hours of downtime in a year. Each extra nine divides the allowance by ten. When components depend on each other in a chain, availability multiplies, so it is lower than the weakest part. When you add independent copies, the chance that all of them fail at once is the product of their failure chances, so availability rises. The stand-in below lets you check this.

Backups and disaster recovery

Redundancy protects against hardware failure. It does not protect against a deleted table, a bad migration or ransomware, because the mistake is copied to every replica instantly. You also need backups: separate, versioned copies you can restore.

Two numbers define a recovery plan:

  • RPO (recovery point objective): how much recent data you can afford to lose, measured in time.
  • RTO (recovery time objective): how long you can afford to be down.

A one-hour RPO means backups or replication at least hourly. The tighter the numbers, the higher the cost, so set them per system and not for the whole company. Common disaster recovery strategies, from cheapest to most expensive: backup and restore (slow recovery), pilot light (a minimal copy in another region that you scale up), warm standby (a smaller running copy) and active-active (full capacity in two regions).

Practical steps:

Terminal
# Enable Cloud SQL automated backups and point-in-time recovery
gcloud sql instances patch my-db \
  --backup-start-time=03:00 \
  --enable-point-in-time-recovery

# Keep old versions of objects in a bucket
gcloud storage buckets update gs://my-bucket --versioning

Flags change over time, so check the current documentation. Most important of all: test your restores. A backup that has never been restored is only a hope. Schedule a practice recovery and time it against your RTO.

Security in layers

Defense in depth means no single control is trusted completely. If one layer fails, the next one stops the attacker. On Google Cloud the layers look like this:

flowchart TB
  A["Identity: IAM, least privilege, MFA"] --> B["Network: VPC, firewall rules, private IPs"]
  B --> C["Perimeter: VPC Service Controls"]
  C --> D["Data: encryption, KMS, backups"]
  D --> E["Detection: audit logs, alerts"]

Each layer protects against a different mistake.

  • Identity. Give each workload its own service account and grant least-privilege roles. Require multi-factor authentication for people.
  • Network. Use a custom VPC, deny by default, avoid public IP addresses where you can, and reach Google APIs with Private Google Access.
  • Perimeter: VPC Service Controls. IAM answers "who may call this API?" VPC Service Controls adds a boundary around projects and services (such as Cloud Storage and BigQuery). Calls that cross the boundary are blocked even if the caller has the right IAM role. That protects against stolen credentials and data being copied to a project you do not control. Rolling it out is delicate: use dry-run mode first to see what would be blocked.
  • Data. Data is encrypted at rest by default. For more control use customer-managed keys in Cloud KMS, and keep secrets in Secret Manager.
  • Detection. Turn on audit logs, route them to a protected place, and alert on risky events (Cloud Logging and Monitoring).

Use organization policies to enforce rules for every project, such as blocking public IP addresses on VMs or restricting resource locations.

Security and reliability meet at blast radius. Separate projects for production and development, separate service accounts per workload, and small deployments each limit how much one mistake can damage.

Scanning a web application for vulnerabilities

Code reviews miss things, so most teams also scan a running application. On Google Cloud the built-in option is Web Security Scanner, part of Security Command Center. It targets applications that run on App Engine, Compute Engine and GKE with a public URL.

It works like a patient, harmless attacker in three steps:

  1. Crawl. The scanner starts from your URLs, follows every link and fills in forms, building a map of pages and inputs.
  2. Simulate attacks. For each input it sends test payloads, such as a harmless script marker where cross-site scripting might happen.
  3. Analyze responses. It checks whether the payload came back unescaped or whether the page leaks something it should not, and reports the finding with the URL, the parameter and a reproduction.

Typical findings are cross-site scripting (XSS), mixed content (HTTPS pages loading HTTP resources), clear-text passwords sent over HTTP, outdated libraries with known vulnerabilities, and misconfigured content types or Flash injection (a legacy issue). There are managed scans that run on a schedule and custom scans that you configure with your own start URLs and credentials for pages behind a login.

Good ways to use it: run a scan as a pre-launch security check on a staging copy (never an unprepared production site, because the scanner submits forms and can create data), and repeat regular security audits, for example on every release or monthly. Its strengths are automation, low cost and one console for findings. Its limits are limited coverage (it cannot understand business logic, and heavy JavaScript apps may not be crawled fully) and occasional false positives that a person must confirm. Other well-known scanners such as OWASP ZAP, Nessus, Acunetix and WebInspect cover more ground or work outside Google Cloud, and a thorough review combines scanning with code review and manual testing.

The toy scanner below shows the core idea: inject a marker into a parameter and see whether it comes back unescaped. The two "pages" are plain Python functions.

Python Detecting reflected XSS

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: an availability calculator

This stand-in is plain JavaScript math, not a Google Cloud tool. It models independent components with a given availability. Real failures are not always independent (a bad release hits every zone), so treat the results as an upper bound.

JS Stand-in: series and parallel availability

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Adding a second VM helps a lot, but after that the single database becomes the weakest link, which is why high-availability database options exist. Reliability work is largely the search for the next weakest link.

Stand-in: RPO and backup schedules

This second stand-in checks whether a backup schedule can meet an RPO. It assumes a worst case: the failure happens just before the next backup would run.

JS Stand-in: does the backup schedule meet the RPO?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

A short checklist

  • At least two zones for anything customers depend on; managed, regional services where possible.
  • Backups that are separate from the primary data, versioned, and restored in a regular drill.
  • Written RPO and RTO for each important system.
  • Least-privilege IAM, no key files, private networking by default.
  • VPC Service Controls (tested in dry-run first) for projects that hold sensitive data.
  • Audit logs on, with alerts for risky changes.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why are replicas alone not a backup strategy?

    Replication copies changes, including bad ones. Backups are separate, versioned copies you can restore from.
  2. 2. What does RTO measure?

    RTO is the recovery time objective. RPO is the amount of data loss you can tolerate, measured in time.
  3. 3. What extra protection does VPC Service Controls add on top of IAM?

    It limits where data can flow, which helps against stolen credentials and data exfiltration.
  4. 4. Two independent VMs each have 99% availability. Running both in parallel gives about...

    Both fail at once with probability 0.01 x 0.01 = 0.0001, so the pair is available about 99.99% of the time, assuming independence.
  5. 5. What is the most important property of a backup?

    An untested backup may be incomplete or unusable. Regular restore drills prove the plan works.

Summary

  • Ask pillar questions of every design: reliability, security, cost, operations and performance.
  • Remove single points of failure with multi-zone designs, health checks, stateless servers and highly available data services.
  • Redundancy is not backup: keep versioned backups, set RPO and RTO per system, and rehearse restores.
  • Layer security: identity, network, VPC Service Controls, data protection and detection, with organization policies as guardrails.
  • Series components multiply availability down, parallel copies raise it, and the weakest link sets the limit.

Next: Common Mistakes and Troubleshooting