Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 36 ยท Building Blocks

Distributed Caching

Learn cache-aside and write-through, invalidation, cache stampedes, request coalescing and TTL jitter, with runnable Python simulations.

A cache keeps a copy of data in a place that is faster to reach than the original. In a distributed system the original is usually a database that is slow, far away or expensive to query, and the cache is a fast in-memory store such as Redis or Memcached shared by many application servers. A good cache can cut database load by 90% or more and drop response times from tens of milliseconds to one.

Caching also creates one of the oldest problems in computing: the copy and the original can disagree. This lesson covers the common caching patterns, how to keep the copy reasonably fresh, and two failure modes that only show up at scale: cache stampedes and synchronized expiry. Everything is runnable. If you want the replication background, see replication basics.

Where caches live

Cache layers

Browser and CDN Closest to the user, best for static files.
In-process Fastest, but each server has its own copy.
Shared cache Redis or Memcached, one copy for all servers.

Every layer adds speed and a chance of stale data.

1 / 2

A distributed (shared) cache is usually split across several nodes using sharding, often with consistent hashing, so that adding a node moves only a small share of the keys.

Cache-aside

In cache-aside (also called lazy loading) the application owns the logic:

  1. Look in the cache. On a hit, return the value.
  2. On a miss, read from the database, put the value in the cache, and return it.
  3. On a write, update the database, then invalidate (delete) the cached key.

Only data that is actually requested ends up in the cache, and the cache can fail without breaking the system: the application just becomes slower. The cost is that the first request for a key is always a miss, and there is a window between a database change and the invalidation in which the cache holds old data.

Write-through

In write-through, every write goes to the cache and to the database together. Reads after a write always see the new value, and the cache stays warm. The costs are higher write latency (two stores) and a cache that fills with data that nobody reads. A variant, write-behind, writes to the cache first and flushes to the database later. It is faster, but data can be lost if the cache dies before flushing.

A first cache simulation

The next example runs a tiny cache-aside with a database dictionary. It shows a hit, a miss, a stale read after an unannounced database change, the fix with invalidation, and a write-through update.

Python Cache-aside, staleness, invalidation and write-through

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second read is a hit, so the database was read only once so far. After the database changed behind the cache's back, the cache happily served the old value. Deleting the key forced a reload and fixed it. With write-through there was no stale window at all, and the read was a hit.

Invalidation and TTLs

"There are only two hard things in computer science: cache invalidation and naming things."

Invalidation is hard because the writer and the cache are separate systems, and the steps are not atomic. A classic race: a reader misses and reads the old value from the database, then a writer updates the database and deletes the key, and finally the reader stores its old value in the cache. The stale value now lives until it expires.

Practical defenses:

  • Always set a TTL (time to live). Even if invalidation fails, the entry disappears eventually. The TTL is your upper bound on staleness.
  • Delete rather than update on writes in cache-aside: the next read recomputes the value from the source of truth.
  • Version keys (user:1:v7) for data that changes rarely, so old versions are simply never read.
  • Publish invalidation events through a message queue (see queues and pub/sub) when many caches must drop the same key.

Cache stampede

A cache stampede (or thundering herd) happens when a popular key expires and many requests arrive before anyone has refilled it. Every one of them misses and goes to the database. One query becomes hundreds, the database slows down, loads take even longer, and more requests pile up.

Request coalescing (also called single-flight) fixes this: the first request that misses becomes the loader, and the others wait for its result instead of starting their own loads. Other tools are a short lock around the reload, serving stale data while one request refreshes in the background, and early refresh, where a key is recomputed slightly before it expires.

In this example 40 requests per tick arrive for a hot key that just expired. A database load takes three ticks to finish.

Python Stampede versus request coalescing

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without coalescing, every request in the first three ticks starts its own load: 120 database queries for a single key. With coalescing there is exactly one load, and everyone else shares its result.

TTL jitter

Stampedes can also come from many keys expiring together. If you fill the cache at startup, or after a deployment, with the same TTL for every key, they all expire at the same second and the database is hit by a wave of reloads. The cure is jitter: add a random amount to each TTL so expiries spread out.

Python Fixed TTL versus jittered TTL

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a fixed TTL, all 1,000 keys expire in the same second. With ยฑ10% jitter the busiest second has only 17 of them (the seed is fixed, so you get the same numbers every run), and the reloads are spread over 120 different seconds. Jitter is cheap, and you should add it by default.

Other things to remember

  • Negative caching: cache "not found" answers briefly, or attackers asking for missing keys will bypass your cache and hit the database every time.
  • Hot keys: one very popular key can overload the single cache node that owns it. Replicate it, or add a small in-process cache in front.
  • Big values and serialization: cached objects cost memory and CPU to encode; measure before caching everything.
  • Eviction: when memory is full, the cache removes entries by a policy such as least recently used (LRU).
  • Consistency: a cache is a replica with relaxed guarantees. Read the consistency models lesson to name exactly what your users may see.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In cache-aside, what should the application do after updating the database?

    Deleting the key forces the next read to reload fresh data. Setting a TTL as well limits staleness if the delete is lost.
  2. 2. What is a cache stampede?

    When a popular key expires, every concurrent request misses and reloads from the source, which can overload the database.
  3. 3. How does request coalescing help?

    One load per key instead of one per request keeps the database load flat when a hot key expires.
  4. 4. Why add random jitter to TTLs?

    Spreading expiry times turns one big wave of reloads into a steady trickle.
  5. 5. Which pattern gives readers fresh data right after a write at the price of slower writes?

    Write-through updates the cache and the database together, so reads after a write see the new value, but every write touches two stores.

Summary

  • A shared cache speeds up reads and protects the database, but the copy can be stale.
  • Cache-aside loads on a miss and invalidates on writes; write-through updates the cache and the database together.
  • Always use a TTL as the upper bound on staleness; add jitter so keys don't all expire together.
  • Stampedes are fixed with request coalescing, short locks, stale-while-revalidate or early refresh.
  • Treat the cache as a replica with weaker guarantees, and cache negative results too.

Next: Load Balancing and Distributed Rate Limiting.