Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 65 ยท Caching

Distributed Caches - Redis and Memcached

Design a distributed cache: sharding with hash slots, replication, Redis and Memcached compared, data structures, persistence, hot keys, and a Python sharded cache.

A cache inside one application process is fast, but it has limits. It disappears on every restart, each application server keeps its own copy (so ten servers hold ten partial caches), and it can use only the memory of that machine. A distributed cache moves the cache out into a shared service made of several nodes, so all application servers see the same data, the capacity grows by adding nodes, and the cache survives application deploys.

The two names you will meet constantly are Memcached and Redis. This lesson explains how a distributed cache is built, how the two products differ, and the problems that appear at scale.

flowchart LR
  A1["App server 1"] --> CL
  A2["App server 2"] --> CL
  A3["App server 3"] --> CL
  subgraph CL["Cache cluster"]
    N1[("Node 1")]
    N2[("Node 2")]
    N3[("Node 3")]
  end
  CL -.->|"miss"| DB[("Database")]

Figure: all application servers share one cache cluster; keys are spread over the nodes.

Spreading keys over nodes

With several cache nodes, each key must live on exactly one node (plus replicas) and every client must agree which. Two approaches:

  1. Client-side sharding: each client library holds the node list and maps keys to nodes with consistent hashing. No coordinator, very fast, used by classic Memcached deployments. The drawback: every client must have the same view of the cluster.
  2. Server-side or proxy sharding: clients talk to a proxy or to any node and are redirected. Redis Cluster divides the key space into 16,384 hash slots (the slot is CRC16(key) mod 16384); each node owns a range of slots, and moving slots between nodes rebalances the cluster. Clients learn the slot map and cache it.

Hash slots give the effect of consistent hashing with explicit, movable chunks, so adding a node means migrating some slots instead of rehashing everything.

Python A sharded cache with hash slots

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Adding a node moves roughly the share of data in the slots it takes (about a quarter here), not everything, and every key is still found afterwards because the slot map was updated.

Replication and failure

If a cache node dies, its keys vanish and reads fall through to the database, a sudden surge that can itself overload it. Options:

  • Accept it: a pure cache can be rebuilt; spread the load with many small nodes so one failure loses only a small fraction (virtual nodes or many slots help here).
  • Replicas: each primary has one or more replicas that copy its data. If the primary fails, a replica is promoted automatically (Redis Sentinel or Cluster does this). Replication is usually asynchronous, so a failover can lose the last few writes, acceptable for caches.
  • Multiple zones: put replicas in a different availability zone.

Memcached and Redis compared

Memcached is a pure, simple, multi-threaded in-memory key-value cache. It stores opaque values (strings or bytes), has LRU eviction, no persistence, no replication of its own, and scales by client-side sharding. It is extremely fast and predictable for plain caching of rendered fragments and query results.

Redis is an in-memory data structure server. Beyond strings it provides lists, sets, hashes, sorted sets, streams, bitmaps, HyperLogLog, geospatial indexes, pub/sub and scripting. It has optional persistence, replication, a clustering mode and many eviction policies. Command execution is single-threaded for each shard (which makes operations atomic), with I/O threads in newer versions.

Memcached Redis
Data model Key to opaque value Strings, lists, sets, hashes, sorted sets, streams...
Persistence None Snapshots (RDB), append-only log (AOF)
Replication No built in Yes, with automatic failover
Clustering Client-side Built in (hash slots)
Threading Multi-threaded Mostly single-threaded per shard
Eviction LRU Several: LRU, LFU, random, TTL-based
Best for Simple, large, volatile caches Caches plus counters, queues, leaderboards, sessions

If you only need a cache for blobs, either works; Redis's flexibility makes it the common default, and many teams use it as more than a cache (sessions, rate limiting, queues, real-time ranking), but then you must treat it with more care for durability.

Redis beyond key-value

Sorted set Leaderboards and ranking with scores.
Counters Atomic increments for rate limits and views.
Lists and streams Simple queues and event logs.
Pub/sub Broadcast messages between services.

Atomic commands avoid many race conditions.

1 / 2

What Redis data structures buy you

Choosing the right structure often replaces application code with one atomic command:

  • String + TTL: cached values, locks (SET key value NX EX 30), sessions.
  • Counter (INCR): page views, API quotas; atomic, so no lost updates.
  • Hash: store an object's fields without serializing the whole thing.
  • List: a simple queue (LPUSH / RPOP) or recent-activity feed.
  • Set: unique members, tags, "who liked this".
  • Sorted set: leaderboards and time-ordered data, ranked by score with O(log n) updates.

The next simulation builds a tiny in-memory model of a few of these commands (strings with TTL, counters and a sorted-set leaderboard) so you can see how they behave. It is a teaching model, not the real server.

Python A tiny Redis-like store

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

SET ... NX with an expiry is the building block of simple distributed locks and of "only one request recomputes this value" guards (cache stampede). Because Redis runs commands one at a time, INCR is naturally race-free.

Persistence and memory

Redis can save its data in two ways. RDB snapshots write the whole dataset to disk periodically (compact, fast restart, may lose the last minutes). AOF (append-only file) logs every write command (much less data loss, larger files, slower restart). Use neither for a pure cache; use AOF or both when Redis holds data you cannot easily rebuild.

Memory is the limit. Configure maxmemory and an eviction policy (allkeys-lru, allkeys-lfu, volatile-ttl, noeviction...) so a full cache evicts rather than refusing writes. See eviction policies. Watch the eviction rate: constant evictions mean the cache is too small for the working set.

Problems at scale

  • Hot keys: one very popular key lands on one node and overloads it no matter how many nodes exist. Cures: keep a short-lived copy in each application process (a local near-cache), or replicate the hot key under several names (key#1..key#8) and read a random copy.
  • Big keys: a multi-megabyte value or a list with millions of items blocks the node and the network. Split them, compress them, or avoid storing them.
  • Cache avalanche: many keys expire together, or a node fails, and the database is flooded. Use TTL jitter, replicas and rate limits.
  • Connection storms: thousands of app instances each opening many connections. Use connection pools or a proxy.
  • Network latency: every cache call is a network round trip (about 0.5 ms in a data center). Batch with pipelining or multi-get, and avoid chatty loops of one-key lookups.
  • Serialization cost: encoding large objects costs CPU; prefer compact formats and cache only what you need.
  • Consistency: replicas lag and invalidation races exist (invalidation); never use the cache as the source of truth for money-like data.
Python Taming a hot key by splitting it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The single key puts every request on one node. Eight copies under different names land on different nodes, spreading the read load. Writes must now update all copies, a trade-off that is fine for read-heavy hot data.

Designing a distributed cache: a checklist

  1. Requirements: read/write ratio, acceptable staleness, latency target (usually under 1 to 2 ms), expected hit rate.
  2. Capacity: items x average size x overhead (about 1.5 to 2x). If 100 million items average 2 KB, that is about 200 GB of data and roughly 300 to 400 GB of memory, so around 8 to 12 nodes of 32 to 48 GB each. See capacity estimation.
  3. Throughput: operations per second per node (tens to hundreds of thousands for Redis on good hardware); add nodes if exceeded.
  4. Partitioning: consistent hashing or hash slots; plan resharding.
  5. Replication and failover: replicas, health checks, automatic promotion.
  6. Eviction and TTL policy, with jitter.
  7. Client behavior: timeouts, retries with backoff, circuit breaker, fallback to the database, near-cache for hot keys.
  8. Observability: hit rate, latency percentiles, memory use, evictions, connections, hot-key detection.
  9. Security: private network only, authentication, TLS, never expose the cache publicly.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why use a distributed cache instead of an in-process cache for a fleet of servers?

    A shared cache avoids duplicated, divergent per-server caches and scales independently.
  2. 2. How does Redis Cluster decide which node owns a key?

    CRC16(key) mod 16384 picks a slot, and the slot map says which node owns it.
  3. 3. Which Redis structure suits a real-time leaderboard?

    A sorted set keeps members ordered by score and updates in O(log n).
  4. 4. One key receives most of the traffic and overloads its node. Which fix helps?

    Adding nodes does not help because the key still hashes to one node; spreading reads does.
  5. 5. What is a major difference between Memcached and Redis?

    Memcached stays minimal; Redis is a data structure server with optional durability.

Summary

  • A distributed cache is a shared, scalable, restart-proof cache cluster; keys are spread by consistent hashing or hash slots.
  • Replication and automatic failover protect against node loss; spread data over many nodes to limit the blast radius.
  • Memcached is a simple volatile cache; Redis adds data structures, persistence and clustering.
  • Pick structures that make operations atomic (counters, sorted sets, SET NX).
  • Watch for hot keys, big keys, avalanches and chatty access; size memory from items x size x overhead.

Next: CDN and Edge Caching