Skip to content
Tutorials Center
My learning

Topics

Lesson 55 of 77 ยท DevOps, Tools and System Design

System Design: The Building Blocks

Learn the system design building blocks interviewers expect: load balancers, caches, queues, CDNs, databases and back-of-envelope estimates, with runnable Node models.

A system design interview gives you a vague prompt ("design a photo-sharing service") and 45 minutes. Nobody expects a perfect architecture. They watch whether you can ask good questions, estimate scale, pick standard building blocks, and explain the trade-offs of each choice. This lesson covers those blocks. The next one, rate limiters, feeds and chat, uses them to solve typical problems.

Real load balancers, Redis servers, Kafka clusters and CDNs cannot run on this site. The configuration and architecture snippets are read-only; the runnable examples are small Node stand-ins that model the behavior (round-robin balancing, a cache with a hit rate, a queue with a worker, an estimate calculator). They show the idea, not the performance of real products.

The usual building blocks

Load balancer Spreads requests over servers.
Cache Keeps hot data close and fast.
Queue Lets slow work happen later.
CDN Serves files from near the user.

Add each block only when a requirement asks for it.

1 / 2

How do you start a system design question?

Short answer: Clarify before you draw. Ask what the product must do (functional requirements) and how well it must do it (non-functional requirements): number of users, read/write ratio, latency target, availability, consistency needs, data size. Then state the numbers you will assume, estimate load, sketch the high-level design, and only then dive into the hardest component.

Functional requirements for a URL shortener: create a short link, redirect. Non-functional: redirects must be very fast and highly available, links are never lost, short links are hard to guess. Saying the read-to-write ratio out loud (maybe 100 reads per write) immediately tells you the design should favor reads, with caching.

How do you do back-of-envelope estimates?

You need rough orders of magnitude, not precision. Round aggressively and show your arithmetic. Useful anchors:

  • A day has about 86,400 seconds, so call it ~100,000 (10^5). A million requests per day is about 12 per second on average.
  • Peak traffic is often 2 to 10 times the average.
  • 1 KB = 10^3 bytes, 1 MB = 10^6, 1 GB = 10^9 (using decimal is fine for estimates).
  • Reading from memory is roughly 100 times faster than SSD, which is roughly 100 times faster than a network round trip within a data center; a cross-continent round trip costs tens to hundreds of milliseconds.
  • One ordinary server can handle on the order of thousands of simple requests per second, but a database often becomes the first limit.
Node.js Estimate calculator for a photo service

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The last number is huge, and that is the point of the exercise: it tells you a CDN and image thumbnails are not optional. Estimates exist to reveal which component is the bottleneck.

What does a load balancer do?

A load balancer sits in front of several identical servers and spreads requests across them. It improves capacity and availability, because it stops sending traffic to servers that fail health checks. Layer 4 balancers route by IP and port; layer 7 balancers understand HTTP and can route by path, header or cookie, terminate TLS, and retry.

Common algorithms: round robin (take turns), least connections (pick the least busy server), and hashing on a key such as the client IP or user ID (sticky routing). Sticky sessions are convenient but make scaling uneven; the cleaner design is stateless servers that keep session data in a shared store, so any server can handle any request. The load balancer itself must not be a single point of failure, so you run a pair, or use a managed service.

NGINX
# Read-only: nginx as a layer 7 load balancer
upstream app {
    least_conn;
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    server 10.0.0.13:8080 backup;
}
server {
    listen 80;
    location / { proxy_pass http://app; }
}
Node.js Round robin with health checks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

What is caching, and what can go wrong?

A cache stores the result of expensive work (a database query, a rendered page) in fast storage so later requests are cheaper. The most common strategy is cache-aside: the application checks the cache first; on a miss it reads the database, then writes the value into the cache with a TTL (time to live).

TEXT
Read-only: cache-aside
1. app asks cache for key
2. hit  -> return it
3. miss -> query database -> store result in cache (with TTL) -> return it

Other strategies: write-through (write to cache and database together, simple and consistent but slower writes), write-back (write to cache and flush to the database later, fast but risks data loss) and read-through (the cache itself loads on a miss). Where caches live: the browser, a CDN, an in-process map, or a shared cache such as Redis or Memcached.

Know the failure modes:

  • Stale data: the database changed but the cache did not. Fix with TTLs or by deleting the key on writes (invalidation, famously hard).
  • Cache stampede: a hot key expires and thousands of requests hit the database at once. Fix with request coalescing (one request refills), jittered TTLs or early refresh.
  • Cache penetration: requests for keys that do not exist always miss. Cache the "not found" result briefly.
  • Eviction: when memory fills, an eviction policy such as LRU (least recently used) chooses what to drop.
Node.js Cache-aside and its hit rate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Why use a message queue?

A queue decouples the part of the system that creates work (the producer) from the part that does it (the consumer). The user's request returns quickly ("we got your upload"), while slow work (resizing images, sending email, charging a card) happens in the background. Queues also absorb spikes: a burst of 10,000 jobs waits in line instead of crashing the workers, and you can add consumers to catch up.

Things interviewers probe:

  • Delivery guarantees: at-most-once (may lose), at-least-once (may duplicate; the most common) and exactly-once (really "at-least-once plus idempotent consumers"). Because duplicates happen, consumers must be idempotent.
  • Retries and dead-letter queues: failing messages are retried with backoff, then parked in a dead-letter queue for inspection.
  • Ordering: a single queue can keep order; partitioned logs such as Kafka keep order only within a partition.
  • Queue vs pub/sub: a queue delivers each message to one worker; publish/subscribe delivers it to every subscriber.
Node.js A queue, a worker and a retry

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

What is a CDN?

A content delivery network is a set of servers spread across the world that cache your static files (images, scripts, video segments, even whole pages) close to users. A request goes to the nearest edge server; on a miss, the edge fetches from your origin and keeps a copy. The benefits are lower latency, less load on your origin, and some protection against traffic spikes and attacks. Control freshness with Cache-Control headers, versioned file names (app.3f9a1c.js) and purge APIs.

How do you choose a database?

Start from the access pattern, not fashion.

  • Relational (SQL): structured data, joins, transactions, strong consistency. The default choice that goes a very long way. Details in indexes, normalization and transactions.
  • Key-value / document stores: simple lookups by key at huge scale, flexible schemas. See NoSQL, caching and scaling.
  • Wide-column stores: massive write volumes, time-ordered data.
  • Search engines: full-text queries and ranking.
  • Object storage: large blobs (photos, video) with a database holding only the metadata and URL.

Scaling a database: first add indexes and read replicas (copies that serve reads, with a little replication lag), then partition big tables, and finally shard (split data across servers by a key, which makes cross-shard queries and transactions hard). Mention CAP: during a network partition you must choose between consistency and availability.

A reusable checklist for the deep dive

  1. What is the bottleneck (reads, writes, storage, bandwidth)?
  2. What is the single point of failure and how do we remove it?
  3. What happens when a component fails or a region goes down?
  4. What can be cached or done asynchronously?
  5. What do we monitor (latency percentiles, error rate, queue depth)?
  6. What did we trade away (cost, consistency, complexity)?

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What should you do first when given a system design prompt?

    Requirements and scale drive every later choice. Designing before clarifying risks solving the wrong problem.
  2. 2. One million requests per day is roughly how many requests per second on average?

    A day has about 86,400 seconds, so 1,000,000 / 86,400 is about 11.6. Peaks will be several times higher.
  3. 3. In cache-aside, what happens on a cache miss?

    Cache-aside means the application manages the cache: check, then load and fill on a miss.
  4. 4. A queue delivers messages at-least-once. What must the consumer be?

    At-least-once delivery can repeat a message, so processing it twice must be safe, for example by recording processed IDs.
  5. 5. Why prefer stateless application servers behind a load balancer?

    When session state lives in a shared store, servers are interchangeable, which makes horizontal scaling and failover simple.

Summary

  • Clarify requirements, estimate scale, sketch the design, then dig into the bottleneck.
  • Load balancers spread traffic and remove failed servers; keep servers stateless.
  • Caches (cache-aside plus TTL, LRU eviction) speed reads but bring staleness and stampede risks.
  • Queues decouple work, absorb spikes and need idempotent consumers; CDNs serve static content near users.
  • Choose databases by access pattern; scale with indexes, replicas, partitioning and sharding.

Next: System Design: Rate Limiters, Feeds and Chat.