Skip to content
Tutorials Center
My learning

Topics

Lesson 67 of 77 ยท Projects

Project: URL Shortener

Build a URL shortener in Django: base62 codes, a redirect view, race-free click counting with F(), a cache in front of the database and a test suite.

A URL shortener turns https://example.com/some/very/long/address?with=parameters into something like https://sho.rt/2Bi/. Visiting the short address sends you to the long one. The idea is simple, which makes it a perfect small project: it has one table, one create endpoint and one very hot read path, and it forces you to think about three things that matter in real systems.

  • Unique, compact codes. How do you generate a short identifier that never collides?
  • Counting under concurrency. Two visitors arrive at the same instant. Do both clicks get counted? This is where F() expressions earn their place.
  • Speed. A shortener is read-heavy, so a cache in front of the database pays off immediately.

You will first build the code generator and URL checker as plain Python, then assemble the full Django app with a cache and a test suite.

How the pieces fit

flowchart TD
    A["GET /2Bi/"] --> B{"Target in cache?"}
    B -->|"yes"| E
    B -->|"no"| C["SELECT link by code"]
    C -->|"not found"| D["404"]
    C -->|"found"| F["Store target in cache"]
    F --> E["UPDATE clicks = clicks + 1"]
    E --> G["302 redirect to target"]

A cache hit skips the lookup, but the click is always counted with one atomic UPDATE.

Step 1: short codes with base62

You could make codes random, but then you must check for collisions on every insert. A neater trick is to derive the code from the row's primary key. The key is already unique, so the code is unique too, with no retries. Encode the number in base 62 (digits 0-9, letters a-z and A-Z), and 6 characters already cover more than 56 billion links.

Adding an offset (here 10,000) to the id avoids tiny codes such as 1 and 2 for your first links. Codes derived from sequential ids are guessable, though: someone can walk through 2Bi, 2Bj, 2Bk and discover every link. That is fine for a public shortener but wrong for private links, where you would use random codes instead.

The URL itself also needs checking. Without validation, your shortener could redirect visitors to a javascript: address or to nonsense. The helper below accepts only http and https addresses with a host name.

Python Base62 codes and URL validation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The round trip proves decode(encode(n)) == n, which is the property a code generator must have. The table of capacities shows how fast base62 grows: add one character and you get 62 times more room.

Step 2: the model and F() counters

The Link model has the unique code, the target URL, a clicks counter and a creation time. Two decisions deserve explanation.

The code starts as NULL. The row must exist before its primary key is known. Many NULL values are allowed in a unique column, so a link can be inserted first and its code filled in right after, inside one transaction. (A custom alias such as summer-sale is stored directly.)

Clicks are counted with F(). This is the heart of the lesson. Compare these two ways to add one click:

Python
link.clicks += 1        # read the number into Python, add, write back
link.save()

Link.objects.filter(pk=link.pk).update(clicks=F("clicks") + 1)   # one SQL statement

The first reads a value, changes it in Python, and writes the result. If two requests read clicks = 5 at the same time, both write 6, and one click is lost. The second tells the database to add 1 to whatever the current value is, in a single UPDATE clicks = clicks + 1, which the database executes atomically. You can see the lost update in the first part of the live example below. More on this in Transactions and Concurrency and Lookups, Q and F.

Step 3: the full app, live

The script below contains the whole project: settings (with a locmem cache), the model, the helper functions, two views, the URLs and a test class that runs at the end. Read it in order.

Look at three things as you go:

  • redirect_view caches only the target, not the click count, so the counter stays exact even on cache hits. It uses 302 (temporary) rather than 301 (permanent), because browsers cache a 301 and would stop sending requests, which would stop the counting.
  • create_link runs in transaction.atomic() so a half-created link (a row without a code) can never be seen by other requests.
  • Aliases are checked against a pattern and a list of reserved words, because /api/ must keep working as a route.
Python The URL shortener with cache and tests

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first two printed lines are the key experiment. With the read-modify-write pattern, two clicks produced a count of 1. With F(), two clicks produced 2. On the tour, visit 1 cost two queries (look up the link, then count the click), while visits 2 and 3 cost one (only the count), because the target came from the cache.

The cache here is locmem, which lives inside one process. With several web servers you need a shared cache such as Redis, otherwise each server keeps its own copy. When a link is edited or deleted, remember to call cache.delete(f"link:{code}"), or visitors will keep reaching the old target until the entry expires.

When the counter becomes a bottleneck

One UPDATE per visit is perfectly fine for thousands of visits a day. For a link that goes viral, every visit updating the same row creates a queue of locks. The usual fix is to buffer the clicks and write them in batches. The pattern uses the cache's atomic counters:

Python
# on each visit (very cheap, no database write)
cache.add(f"clicks:{code}", 0, timeout=None)
cache.incr(f"clicks:{code}")

# in a periodic job (a management command or a Celery beat task)
for code in codes_with_pending_clicks():
    pending = cache.get(f"clicks:{code}", 0)
    if pending:
        Link.objects.filter(code=code).update(clicks=F("clicks") + pending)
        cache.decr(f"clicks:{code}", pending)

The totals lag by a minute or so, and a cache failure can lose a few clicks, which is an acceptable trade for analytics. If every click must be recorded exactly (billing, for example), insert one row per click into a separate Click table and count them later with an aggregate. The Docker, Celery and CI lesson shows how to run such a job.

Ideas to extend it

  • Expiry: add an expires field and return 410 Gone after it passes.
  • Owner and dashboard: attach links to users and list their links with click counts.
  • QR codes and custom domains for branded links.
  • Abuse protection: block known malicious domains and rate-limit the create endpoint per IP address.
  • Analytics: record the referrer and country in a Click table, and report with aggregation.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why derive the short code from the row's primary key instead of generating random text?

    Encoding a unique number gives a unique code. The downside is that sequential ids make codes guessable, which matters if the links are private.
  2. 2. Two requests read clicks = 5 at the same moment, then each saves clicks = 6. What went wrong, and what is the fix?

    Both requests wrote the same stale value plus one. An UPDATE with F() makes the database compute clicks = clicks + 1 atomically, so both increments count.
  3. 3. Why does the redirect view respond with 302 instead of 301?

    A permanent redirect may be remembered by the browser. A temporary one makes every visit pass through the server, where it is counted.
  4. 4. What does the view store in the cache?

    Only the rarely changing target is cached. Clicks still go to the database each time, so the count stays exact on cache hits.
  5. 5. Why validate that the scheme is http or https?

    A shortener hides the destination, so it must not become a tool for sending people to harmful or script-executing links.

Summary

  • Derive short codes by encoding the primary key in base62; they are unique without retries, but guessable.
  • Validate user-supplied URLs: allow only http and https and require a host.
  • Count with update(clicks=F("clicks") + 1); reading, adding in Python and saving loses updates under concurrency.
  • Cache the target for each code to skip the lookup, and delete the cache entry whenever a link changes.
  • Use a temporary 302 redirect so every visit is counted, and keep reserved words out of the alias space.
  • For very hot links, buffer counters in the cache and flush them to the database in batches.
  • Tests with assertNumQueries lock in the performance behavior you designed.

Next lesson: Project: Bookstore with Cart.