Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 37 ยท Real-World Projects

Project: Scalable URL Shortener

Design a URL shortener that handles heavy read traffic: read/write split, cache, replicas, sharding, a coordination-free ID generator and analytics events.

A URL shortener looks trivial: store a long address, hand back a short code, redirect when someone visits it. That is exactly why it is a favourite design exercise. The function is simple; the shape of the traffic is what makes it interesting. Links are created rarely and read constantly, often at a ratio of a hundred reads to one write, and a few viral links get most of the visits.

In this project you design the system around those facts and then run a simulation to watch each decision pay off. You will split reads from writes, generate unique codes without a central counter, put a cache in front of replicated and sharded storage, and keep click analytics off the hot path by turning clicks into events.

Requirements and numbers

Start with what the system must do, and with rough numbers; they drive every decision.

  • Create a short code for a long URL. Resolve a code to a URL (redirect). Optionally, expiry and custom aliases.
  • Reads vastly outnumber writes. Assume 100 million new links a year (about 3 per second) and 100 reads for each, so roughly 300 redirects per second on average and several thousand per second at peak.
  • A redirect must be fast (tens of milliseconds) and must stay available when parts fail.
  • Click counts may be a little late; redirect correctness may not.

Four decisions

Split reads and writes Two services, scaled separately.
Coordination-free IDs Generators never share a counter.
Cache the hot links A few links get most of the traffic.
Shard the storage Spread data, add replicas for reads.
1 / 2

Architecture

flowchart LR
    U["Visitor"] --> RS["Read service"]
    RS --> C[("Cache")]
    C -->|miss| R[("Replica")]
    R -->|not yet replicated| P[("Primary")]
    RS -->|click event| Q(("Queue"))
    Q --> A["Analytics consumer"]
    W["Write service"] --> G["ID generators"]
    W --> P
    P -.->|replication| R

Reads go through the cache and replicas; writes go to the primary; clicks flow through a queue to analytics.

Step 1: short codes without a central counter

A code needs to be unique, short and not trivially guessable. Three approaches:

  1. Hash the URL and truncate. Cheap, but short hashes collide (the birthday problem), and two users shortening the same URL get the same code, which is a problem for per-user analytics.
  2. A global database counter. Unique, but every write now depends on one hot, central resource.
  3. Independent generators with non-overlapping ranges. Generator k of n hands out k, k+n, k+2n..., so no two ever produce the same number and none talks to another. Real systems (Snowflake-style IDs) put a timestamp, a worker id and a sequence number into one 64-bit integer, which also solves restarts.

The numbers are then encoded in base62 (digits plus lower- and upper-case letters) to be short. Passing them through a bijective mixer makes neighbouring ids look unrelated without ever causing a collision, because a bijection maps different inputs to different outputs. This is obfuscation, not security: if links must be unguessable, add randomness or signed tokens.

The first stand-in shows the arithmetic: how many links fit in each code length, why hash-and-truncate collides, and that the mixed counter never does.

Node.js Code length, collisions and a collision-free mixer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Seven characters give about 3.5 trillion codes, which at 100 million links a year lasts for thousands of years, so six or seven characters are plenty. The 3-character hash demonstrates the problem on a small scale: collisions appear long before the space is full, and every collision would need extra handling. The counter-and-mixer approach needs none.

Step 2: the whole system, under load

The next program contains both services, two storage shards (each a primary plus a replica that lags), a small LRU cache, an analytics queue with a batch consumer, and a load simulation. Notice these points while reading:

  • shorten (write service) picks a generator round-robin and writes to the shard chosen by hashing the code, so the read service can find the shard knowing only the code.
  • resolve (read service) checks the cache first, then the replica, and only falls back to the primary when the replica has not caught up yet. Without that fallback, a link would answer "not found" for a moment after creation (replication lag).
  • Every successful resolve pushes a click event onto a queue; the queue is drained in batches, so the redirect never waits for counting.
  • Random numbers come from a seeded generator, so every run is identical. The traffic is skewed on purpose: random() ** 3 makes low indexes far more popular, like real viral links.
Node.js Read/write split, cache, shards, replicas and analytics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the results. All 200 codes were distinct, and the data is spread over two shards (94 and 106 links). About 40 percent of the 10,000 reads were answered by the cache alone, and because of skewed traffic a cache of only 30 entries (15 percent of the links) was enough for that. Replicas served almost all the remaining reads, and the primary was needed only 68 times, early on, before replication caught up. The ten requests for zzzzzz returned not found, and exactly 9,990 clicks were counted, one per successful redirect: the queue lost nothing.

What is real and what is simulated? The ideas (ID ranges, base62, LRU, replica fallback, queued analytics, hashing to shards) are what production systems use. The simulation replaces networks, disks and time with objects and loop counters, and replication "catches up" every 100 reads instead of continuously.

Details that matter in production

  • Which redirect status? A 301 (permanent) lets browsers cache the redirect, which lowers your load but hides later clicks from your analytics and makes edits slow to take effect. 302/307 keeps every click visible. Many shorteners choose 302 because analytics is the product.
  • Cache invalidation. If links can be edited or deleted, remove or expire the cached entry (a short TTL is the simplest safety net) and negative-cache "not found" briefly so nonexistent codes cannot hammer the database.
  • Hot keys. One viral code can overload a single cache node. Replicate popular entries, or add a small in-process cache in front of the shared one.
  • Resharding. Hash-modulo placement makes adding a shard move most keys. Consistent hashing or a directory service avoids a mass move.
  • Abuse. Rate-limit creation (Rate Limiting, Load Shedding and Backpressure), check destinations against malware lists, and never redirect to dangerous schemes such as javascript:.
  • Failure. If the cache is down, reads still work (slower); if analytics is down, the queue buffers; if the primary is down, existing links still resolve from replicas. That graceful degradation is the payoff of splitting the system (Circuit Breakers, Bulkheads and Fallbacks).

Why microservices here? The read and write paths have totally different load, scaling and failure needs. They deserve to be deployed and scaled separately. The analytics consumer is a third, independent unit fed by events. A real shortener does not need dozens of services, though; three or four, or even a modular monolith with separate deployables, is plenty (When (and When Not) to Use Microservices).

Extending the project

  1. Add expiry times and a background job that deletes expired links and evicts them from the cache.
  2. Support custom aliases, and decide how to avoid clashes with generated codes (a reserved alphabet or separate namespace).
  3. Add a write-through step so new links are placed in the cache immediately, and compare the hit rate.
  4. Replace hash-modulo shard placement with consistent hashing and measure how many keys move when you add a third shard.
  5. Add per-IP rate limiting to shorten and tests that prove the limit holds.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why split the read and write paths of a URL shortener?

    Reads outnumber writes by a large factor, so they must scale and fail independently.
  2. 2. How do independent ID generators avoid producing the same number?

    With disjoint ranges, no coordination is needed at write time.
  3. 3. Why does the read service fall back to the primary database?

    Replication lag means a replica can briefly miss new data. A fallback avoids false 'not found' answers.
  4. 4. Why are clicks recorded as events on a queue instead of updating a counter during the redirect?

    The redirect only drops a message; counting happens later and can lag without hurting visitors.
  5. 5. Why can a small cache give a high hit rate here?

    When requests follow a skewed (power-law-like) distribution, caching the hottest entries covers a large share of traffic.

Summary

  • Design from the traffic shape: many reads, few writes, a few very hot links.
  • Generate unique ids without a central counter, encode them in base62 and mix them so codes look unrelated.
  • Put a cache in front of replicas, and fall back to the primary to hide replication lag.
  • Shard storage by a hash of the code, and plan resharding before you need it.
  • Treat clicks as events so analytics never slows a redirect.

Next: Project: Ride-Sharing Matching Service.