Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 65 ยท Caching

Cold and Warm Caches, Stampedes and Warming

Understand cold and warm caches, how to pre-warm them, and how to prevent cache stampedes with locks, request coalescing, TTL jitter and early refresh, with Python.

A cache helps only when it already contains what people ask for. Right after a restart, a new node joining or a deploy that changes the keys, the cache is empty, and every request misses. This lesson covers what that costs you, how to start with a warm cache instead, and the related failure called the cache stampede (or thundering herd), where a single expired entry can knock over a database.

Cold, warm and hot caches

  • A cold cache is empty or nearly so. Hit rate is close to zero, every request goes to the origin, and latency and database load are at their worst.
  • A warm cache has been filled by traffic or pre-loading and serves most requests. The hit rate is at its normal, stable level.
  • A hot cache (sometimes used for the very best case) holds the hottest items in the fastest tier; some people use "hot" and "warm" simply to mean "full" versus "filling".

When do you meet a cold cache?

  • A new cache cluster, a restart or a crash.
  • Adding or replacing a node (its share of keys starts empty).
  • A deploy that changes the key format or flushes the cache.
  • A new region or a new CDN edge going live.
  • A big event whose content was never requested before (a product launch).
  • An entry expiring right when a spike of traffic arrives.

The danger is a self-reinforcing failure. The cold cache sends everything to the database; the database slows down; the slow queries make the cache fill slowly; users retry, which adds more load. A system that runs at 99% hit rate and 100 queries per second at the database can suddenly send 10,000 per second. If the database was sized for the warm cache, it falls over.

Cold cache, warm cache

Cold Empty. Every request goes to the database.
Warming Filling from traffic. Latency improves.
Warm Hit rate at its normal level.
Risk A cold start can overload the origin.

Plan for the cold start, it is a normal event, not a rare one.

1 / 2

Warming strategies

Cache warming means filling the cache before real users arrive.

  1. Pre-load known hot data. Load the home page, top products, popular categories, configuration and reference data at startup or deploy time, from a list or from yesterday's access logs.
  2. Replay traffic. Replay a sample of recent production requests (or the logs' most frequent URLs) against the new cache before routing real users to it.
  3. Warm in the background after deploys. Rebuild expensive entries with a job rather than waiting for users to trigger them.
  4. Copy from a neighbor. A new node can fetch the hottest keys from a replica or a peer before taking traffic.
  5. Gradual traffic shifting. Give a cold node a small share of traffic ("slow start" in a load balancer) and increase as its cache fills.
  6. Refresh ahead. Before a popular entry expires, refresh it in the background so users never see a miss.
  7. Use consistent hashing so scaling moves few keys (consistent hashing); a modulo-based rehash would turn the whole cache cold.

The next simulation compares starting with an empty cache against pre-loading the hottest keys. It measures the database queries during the first 2,000 requests.

Python Cold start vs warmed start

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pre-loading the 1,000 hottest items cuts the early database queries from about 750 to about 320, because a small set of items accounts for much of the traffic. The more skewed your traffic, the more a small preload buys.

Cache stampede

A cache stampede (also called a dog-pile or thundering herd) happens when a popular entry expires or is missing, and many requests arrive before anyone has refilled it. Each one sees a miss, and each one runs the same expensive query. If the query takes two seconds and 5,000 requests per second want that key, 10,000 identical queries hit the database within the recomputation window. The database slows, so the recomputation takes longer, so more requests pile in.

Python A stampede on an expired popular key

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One expiry of one key caused 10,000 identical queries. Scale that to many keys and the database is gone. Here are the main defenses, often combined.

1. Request coalescing (single flight)

Allow one request to rebuild the value while the others wait for that result and share it. Inside one process this is called single flight; across servers you need a small lock in the cache (SET lock:key id NX EX 10). The lock holder rebuilds and stores the value; waiters poll briefly or retry. Always give the lock an expiry so a crashed rebuilder does not block everyone forever.

2. Serve stale while revalidating

Keep the expired value for a grace period and keep serving it while one background request refreshes it. Users see slightly old data but never wait and never stampede the origin. This is stale-while-revalidate for caches (invalidation lesson).

3. TTL jitter

If many keys are created at the same time with the same TTL, they also expire at the same time: an avalanche. Adding a random offset to each TTL spreads the expirations out.

4. Probabilistic early refresh

Let requests voluntarily refresh a key before it expires, with a probability that rises as expiry approaches and with the cost of recomputation. A popular key is then refreshed by some lucky early request while everyone else still hits the valid copy, so no one ever sees an empty cache.

5. Background refresh and "never expire" hot keys

For the very hottest keys, let a scheduled job rebuild them regularly and give them a long TTL (or none), so user requests never trigger the rebuild at all.

6. Rate limit and shed load at the origin

As a last line of defense, limit concurrent queries to the database; when the limit is reached, fail fast or serve a fallback instead of queueing without bound (rate limiting).

Python Single flight and stale-while-revalidate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The naive version sends a database query for every request during the rebuild window. Single flight sends exactly one, and the other requests wait for its answer. Adding stale serving means nobody waits at all: they get the previous value instantly while one rebuild runs. In a real system the lock lives in the cache (SET lock:key id NX EX 10) so it works across servers.

TTL jitter in numbers

Python Expiry avalanche with and without jitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With no jitter all 5,000 keys expire in the same second. A few percent of random variation already flattens the peak dramatically, and 10 to 20% spreads reloads almost evenly across minutes.

Probabilistic early refresh

A neat, coordination-free trick: on each read, compute whether to refresh now using the remaining time and the recompute cost. The closer to expiry and the more expensive the rebuild, the higher the chance. The formula, often called XFetch, is: refresh if time_left < -cost x beta x ln(random()), where random() is uniform between 0 and 1 and beta is about 1.

Python Early refresh so users never see a miss

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without early refresh, a user has to wait for a rebuild every time the entry expires. With it, a busy key is refreshed by a read shortly before the deadline, so users almost never see a miss, at the cost of a few extra rebuilds.

A checklist for a safe cold start and steady state

  1. Know your cold-start load. Estimate database traffic at 0% hit rate and check whether the origin survives it, or can be protected with limits and load shedding.
  2. Warm before switching traffic: preload, replay, or copy.
  3. Slow-start new nodes and shift traffic gradually.
  4. Add TTL jitter to every cache write.
  5. Coalesce rebuilds with single flight or a short-expiry lock.
  6. Serve stale within a grace period for non-critical data.
  7. Refresh hot keys in the background or early.
  8. Use consistent hashing so scaling does not flush the cache.
  9. Monitor hit rate, origin queries per second, and rebuild latency, and alert on sudden hit-rate drops.
  10. Rehearse it: restart the cache in a test environment under load and watch what the database does.

This closes the caching chapter: a cache is not just a fast dictionary. It is a component with its own lifecycle (cold, warm, expiring, failing) and each phase needs a plan.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is a cache stampede?

    The simultaneous identical rebuilds can overload the database and make the problem worse.
  2. 2. Which technique lets only one request rebuild an expired value while others wait or use stale data?

    A lock or single-flight call ensures one rebuilder; waiters share its result or use the stale copy.
  3. 3. Why add jitter to TTLs?

    Keys loaded together would expire together; random offsets prevent an expiry avalanche.
  4. 4. Which is a good way to warm a new cache before it takes traffic?

    Pre-filling the hot working set avoids sending the full load to the origin.
  5. 5. Why should a rebuild lock have an expiry?

    Without an expiry, a failed lock holder would leave the key permanently locked.

Summary

  • A cold cache sends all traffic to the origin; cold starts happen at restarts, scaling events, deploys and launches.
  • Warm caches by preloading hot data, replaying traffic, copying from peers and shifting traffic slowly.
  • A stampede is many simultaneous rebuilds of one expired key; defend with single flight, stale-while-revalidate, jitter, early refresh and background rebuilds.
  • Protect the origin with rate limits and load shedding, and use consistent hashing so scaling does not flush the cache.
  • Rehearse restarts under load: caching is a component with a lifecycle, not just a dictionary.

Next: The System Design track