Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 37 ยท Distributed Caching

Hot Keys and Replication

Find hot keys that overload one cache server and spread them with key replication, suffix sharding and a tiny local cache. Includes a runnable Zipf traffic simulation.

Consistent hashing balances keys, but traffic is never balanced across keys. A home-page banner, a viral product or a feature-flag blob may receive thousands of reads per second while most keys are read once a day. Whichever server owns that key carries the whole load. Its CPU or network card saturates, latency rises for every other key on that server, and the rest of the cluster idles.

This is the hot key problem. Adding servers does not help, because the hot key still lives on exactly one of them. You need to spread the key itself. This lesson simulates realistic skewed traffic and tries three fixes.

Traffic is skewed: the Zipf shape

Web popularity usually follows a Zipf-like curve: the most popular item gets about twice the traffic of the second, three times the third, and so on. The simulation below draws 100,000 requests from such a distribution over 1,000 keys and shows how unbalanced the servers become even with a perfectly balanced ring.

Python Skewed traffic overloads one server

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The server that owns the hottest key receives far more than its fair share, even though the number of keys per server is balanced.

Detecting hot keys

You cannot fix what you cannot see. Options:

  • Client-side sampling: count a random 1 in 100 requests per key and report the top ten.
  • Server stats: Memcached's watch and metadump commands (and in some versions lru_crawler metadump) can reveal key activity, but they have overhead; use them briefly.
  • Network tools: packet sampling on the server shows which keys appear most.
  • Per-server request rates: if one server has twice the cmd_get rate of its peers while memory use is similar, suspect a hot key.

Fix 1: replicate the hot key under several names

Store the value under key#0, key#1, ... key#R-1. Each name hashes to a different server. Readers pick a random replica, so reads are spread over up to R servers. Writers must update (or delete) all replicas, which costs R operations, so apply this only to the few keys that need it.

Python Replicate the hottest keys

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The busiest server's load drops noticeably, and a bigger replica count would flatten it further. Decide the set of hot keys from monitoring; do not replicate everything.

Fix 2: a tiny local cache in front

If one key is read thousands of times per second by each web server, the cheapest read is the one that never leaves the process. A short-lived in-process cache (an "L1") holding the hot value for one or two seconds can remove almost all traffic for that key. The cost is up to a couple of seconds of staleness, which suits banners, feature flags and configuration, not account balances.

Python
import time

_local = {}   # key -> (expires_at, value)

def get_with_local_cache(key, fetch, ttl=2.0):
    now = time.monotonic()
    hit = _local.get(key)
    if hit and hit[0] > now:
        return hit[1]
    value = fetch(key)               # goes to memcached
    _local[key] = (now + ttl, value)
    return value

Multi-layer caching is discussed further in Multi-Layer Caching and CDNs.

Fix 3: split large hot objects

Sometimes the "hot key" is hot because it is big: one 800 KB blob that every request downloads. Break it into pieces (for example the part that changes often and the part that does not) and let each part be fetched only when needed. Smaller values also cost less network bandwidth, which is often the first bottleneck on a hot server.

Trade-offs

Technique Helps with Cost
Key replication Read load of a few keys R writes per update, more memory, risk of replicas disagreeing briefly
Local L1 cache Extreme read rates Staleness for a few seconds, memory in every app process
Splitting values Bandwidth More keys, more code

Replicas can disagree for a moment while an update is in flight: one replica holds the new value and another the old. If your data must be exactly current, delete all replicas first and let the next readers repopulate them (see Stale Data and Race Conditions).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does adding more servers not fix a hot key?

    A key is owned by one server. Extra servers help only if you also spread the key itself, for example by replication.
  2. 2. How do readers use a replicated hot key?

    Random choice among replicas spreads the reads over the servers that own those names.
  3. 3. What is the main cost of replicating a hot key?

    Each update multiplies by the replica count, and replicas can briefly disagree.
  4. 4. When is a short in-process L1 cache a good fit?

    A one or two second local copy removes most traffic, and a few seconds of staleness is fine for such data.

Summary

  • Balanced key counts do not mean balanced load; traffic is skewed.
  • Detect hot keys with sampling, per-server rates and short-lived server-side tools.
  • Replicate hot keys under suffixed names, writing all replicas and reading a random one.
  • A tiny in-process cache removes extreme read rates for tolerant data.
  • Splitting oversized values cuts bandwidth on hot servers.

Next: Stale Data and Race Conditions