Skip to content
Tutorials Center
My learning

Topics

Lesson 63 of 70 ยท Real-world projects

Project: Three-Tier Web App

Build a classic three-tier app across two Availability Zones: VPC subnets, ALB, Auto Scaling, Multi-AZ RDS, chained security groups and Session Manager access.

This project builds the layout that most web applications on AWS still start from: a load balancer, a fleet of application servers and a relational database, spread over two Availability Zones so that losing one data center does not take the site down. You will plan the network, chain the security groups so each tier only talks to its neighbour, choose how to reach private machines without a bastion host, and then simulate two failures: a whole Availability Zone disappearing, and a database failover.

The real resources need an AWS account (and the database and load balancer cost money while they run), so everything that creates infrastructure below is read-only. The two Python stand-ins are simplified simulations of how the load balancer and the database behave under failure. Their timings are invented for teaching and are not measurements. For the building blocks see VPC Fundamentals, Elastic Load Balancing, Auto Scaling Groups and RDS and Aurora.

The target architecture

flowchart TD
    I["Internet"] --> ALB["Application Load Balancer<br/>public subnets a and b"]
    subgraph AZA["Availability Zone a"]
      PA["public 10.0.0.0/24<br/>ALB node, NAT gateway"]
      AA["private app 10.0.10.0/24<br/>EC2 instances"]
      DA["private data 10.0.20.0/24<br/>RDS primary"]
    end
    subgraph AZB["Availability Zone b"]
      PB["public 10.0.1.0/24<br/>ALB node"]
      AB["private app 10.0.11.0/24<br/>EC2 instances"]
      DB["private data 10.0.21.0/24<br/>RDS standby"]
    end
    ALB --> AA
    ALB --> AB
    AA --> DA
    AB --> DA
    DA -. "synchronous replication" .-> DB

Six subnets in a 10.0.0.0/16 VPC: public for the load balancer, private for the app, private again for the data.

The key design decisions:

  • Only the load balancer is public. App servers and the database have no public IP addresses, and the private subnets' route tables have no route to an internet gateway.
  • App servers reach the internet outbound (package updates, third-party APIs) through a NAT gateway in a public subnet (NAT Gateways and VPC Endpoints). For a stricter setup, use VPC endpoints and drop NAT.
  • Each tier lives in two subnets in different Availability Zones, so every component has a counterpart in the other zone.

Security groups form a chain

Security groups are stateful firewalls attached to network interfaces. Instead of listing IP ranges, each tier's group references the previous tier's group as the source. That keeps the rules correct when instances come and go.

Group Inbound rule
alb-sg TCP 443 (and 80, redirected) from 0.0.0.0/0
app-sg TCP 8080 from alb-sg only
db-sg TCP 5432 from app-sg only

There is no rule that lets the internet reach app-sg or db-sg, and no rule that lets the load balancer reach the database. A compromised web server cannot jump straight to anything except what its own group permits. Details are in Connecting to Instances and Security Groups.

Terminal
# Read-only sketch: names and ids are placeholders
ALB_SG=$(aws ec2 create-security-group --group-name alb-sg --description "ALB" \
  --vpc-id vpc-0abc --query GroupId --output text)
APP_SG=$(aws ec2 create-security-group --group-name app-sg --description "App tier" \
  --vpc-id vpc-0abc --query GroupId --output text)
DB_SG=$(aws ec2 create-security-group --group-name db-sg --description "Database" \
  --vpc-id vpc-0abc --query GroupId --output text)

aws ec2 authorize-security-group-ingress --group-id "$ALB_SG" \
  --protocol tcp --port 443 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id "$APP_SG" \
  --protocol tcp --port 8080 --source-group "$ALB_SG"
aws ec2 authorize-security-group-ingress --group-id "$DB_SG" \
  --protocol tcp --port 5432 --source-group "$APP_SG"

The load balancer and the Auto Scaling group

An Application Load Balancer listens on 443 with an ACM certificate and forwards to a target group that runs a health check against a path such as /health. The Auto Scaling group launches instances from a launch template into the two private app subnets, registers them with the target group and keeps the desired count. Using the load balancer's health check as the group's health check means an instance that fails requests is replaced, not just removed from rotation.

Terminal
aws elbv2 create-target-group --name app-tg --protocol HTTP --port 8080 \
  --vpc-id vpc-0abc --health-check-path /health \
  --health-check-interval-seconds 15 --healthy-threshold-count 2 --unhealthy-threshold-count 2

aws autoscaling create-auto-scaling-group \
  --auto-scaling-group-name app-asg \
  --launch-template LaunchTemplateName=app-lt,Version='$Latest' \
  --min-size 2 --max-size 6 --desired-capacity 2 \
  --vpc-zone-identifier "subnet-0app-a,subnet-0app-b" \
  --target-group-arns arn:aws:elasticloadbalancing:eu-west-1:111122223333:targetgroup/app-tg/abc123 \
  --health-check-type ELB --health-check-grace-period 120

aws autoscaling put-scaling-policy --auto-scaling-group-name app-asg \
  --policy-name cpu50 --policy-type TargetTrackingScaling \
  --target-tracking-configuration '{
    "PredefinedMetricSpecification": {"PredefinedMetricType": "ASGAverageCPUUtilization"},
    "TargetValue": 50.0}'

Minimum two instances means one per zone in normal operation. The app should be stateless: sessions go to a shared store such as ElastiCache or a signed cookie, and uploads go to S3, because any instance can disappear at any time.

The database

Terminal
aws rds create-db-subnet-group --db-subnet-group-name data-subnets \
  --db-subnet-group-description "private data subnets" \
  --subnet-ids subnet-0data-a subnet-0data-b

aws rds create-db-instance --db-instance-identifier shop-db \
  --engine postgres --db-instance-class db.m7g.large \
  --allocated-storage 100 --storage-type gp3 --storage-encrypted \
  --multi-az --no-publicly-accessible \
  --db-subnet-group-name data-subnets --vpc-security-group-ids "$DB_SG" \
  --master-username app_admin --manage-master-user-password \
  --backup-retention-period 7

Four flags carry the design: --multi-az keeps a synchronous standby in the other zone, --no-publicly-accessible keeps it off the internet, --storage-encrypted encrypts it at rest, and --manage-master-user-password lets RDS keep the admin password in Secrets Manager so it never sits in a script. The application connects to the endpoint DNS name, never to an IP address, because failover works by moving that name to the standby. Multi-AZ is for availability, not read scaling: for more read capacity add read replicas (RDS and Aurora).

Access without a bastion host

The traditional way to reach private servers is a "bastion" host with a public IP and an open SSH port, which is one more machine to patch and one more target. AWS Systems Manager Session Manager replaces it. Instances get an instance profile with the managed policy AmazonSSMManagedInstanceCore, the SSM agent connects outward to Systems Manager (through NAT or three interface endpoints: ssm, ssmmessages and ec2messages), and you open a shell with IAM permissions. No inbound port is needed, and every session can be logged to CloudWatch Logs or S3.

Terminal
aws ssm start-session --target i-0123456789abcdef0

# Reach the private database from your laptop through an instance, no open ports
aws ssm start-session --target i-0123456789abcdef0 \
  --document-name AWS-StartPortForwardingSessionToRemoteHost \
  --parameters '{"host":["shop-db.abc123.eu-west-1.rds.amazonaws.com"],"portNumber":["5432"],"localPortNumber":["15432"]}'

See Systems Manager and Day-Two Operations. The security group of the app tier needs no inbound SSH rule at all.

Stand-in 1: load balancer health checks and a lost zone

This simulation has four instances, two per zone, behind a round-robin balancer. Each tick, six requests arrive, then the health checker runs. A target leaves rotation after two failed checks and rejoins after two successful ones, and the Auto Scaling stand-in launches replacements that need three ticks to boot. At tick 3, everything in zone b fails. Watch the errors before the balancer notices, and the recovery after replacements arrive.

Python Lose an Availability Zone behind a load balancer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table in three phases. After the zone fails, requests still go to the dead targets until two health checks have failed, so some requests error. Then the survivors carry all traffic with no errors, but with half the capacity, which is why you size each zone to handle the load alone (or let target tracking add instances). Finally, replacements boot and rejoin after passing their checks. Two real-world lessons follow: shorten the health check interval and thresholds for faster detection (at the cost of more false alarms), and make clients retry idempotent requests, because the load balancer can only fix the future, not the requests already in flight.

Stand-in 2: database failover and client retries

When the primary fails, RDS promotes the standby and moves the endpoint's DNS name to it. Writes fail in between. This simulation writes once per tick, with the outage lasting ticks 3 to 6, and compares a client without retries, a client with three quick retries, and one with exponential backoff and four retries. The outage length is invented; real Multi-AZ failovers commonly take one to two minutes.

Python Writes during a Multi-AZ failover

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The retry budget has to cover the outage. Three quick retries still lose the write submitted at tick 3, while exponential backoff reaches far enough into the future to survive it. The caveat is idempotency: a retried write that actually succeeded the first time (the acknowledgement was lost) must not create a duplicate, so use unique request ids or upserts. Also keep connection pools short-lived enough that they re-resolve DNS after a failover.

Verification checklist

  • The ALB DNS name answers over HTTPS and HTTP redirects to HTTPS.
  • nc -zv <instance-ip> 8080 from outside the VPC fails, and the database has no public address.
  • Terminate one instance in the console: capacity returns to the desired count within a few minutes without errors visible to users.
  • Run a Multi-AZ failover in a test environment (aws rds reboot-db-instance --force-failover) and watch how your application behaves.
  • Set alarms on the target group's UnHealthyHostCount and on the database's free storage and CPU (CloudWatch).
  • Remember the bill: the NAT gateway, the load balancer and the Multi-AZ database all cost money per hour even when idle, so tag and budget the stack and tear it down when you finish experimenting.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How should the database security group allow traffic from the application servers?

    Referencing the app tier's security group as the source keeps the rule correct as instances change and exposes nothing else.
  2. 2. What does Multi-AZ on an RDS instance give you?

    Multi-AZ is about availability. Read replicas scale reads, and cross-Region copies are a separate feature.
  3. 3. Why do some requests fail right after an Availability Zone is lost, even with a load balancer?

    Detection takes the health check interval multiplied by the unhealthy threshold, and requests in that window can hit dead targets.
  4. 4. How do you reach a private instance without a bastion host or an open SSH port?

    The SSM agent connects outbound to Systems Manager, so you get audited shell access without any inbound rule.
  5. 5. Why must the application connect to the database endpoint name instead of an IP address?

    After a failover the endpoint resolves to the promoted standby, while the old IP address belongs to the failed primary.

Summary

  • Put only the load balancer in public subnets; run the app and data tiers in private subnets in two Availability Zones.
  • Chain security groups so each tier accepts traffic only from the tier in front of it.
  • Use an Auto Scaling group with ELB health checks, a stateless app and at least one instance per zone.
  • Run RDS with Multi-AZ, no public access, encryption and managed credentials, and connect through the endpoint name.
  • Replace bastion hosts with Session Manager, and give clients retries with backoff and idempotent writes to survive failovers.

Next: Project: Event-Driven Image Pipeline.