Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 37 ยท Consistency and Failure

Node Failure and Failover

Learn what happens to hit ratio and database load when a Memcached node dies, and compare ejecting the node, failing fast and using a gutter pool. Runnable simulation.

Servers fail: a power supply dies, a kernel update reboots a node, a network cable is pulled. Memcached holds no persistent data, so a failure never loses anything you cannot rebuild. What you do lose is the cache hit ratio, and that matters because your database was sized on the assumption that most reads never reach it.

If you run ten cache servers and one fails, about 10% of keys have nowhere to live. Those reads all go to the database. That is manageable. If you run three servers and one fails, a third of your read traffic hits the database at once. This lesson quantifies it and compares the strategies clients use. As elsewhere, the cluster is simulated in Python.

What the client sees

A dead server shows up as one of these:

  • Connection refused (process down, machine up): fails instantly.
  • Timeout (machine down, network partition, overloaded server): each request waits for the full timeout, which can be far worse than a refusal because threads pile up waiting.
  • Slow responses (swapping, a saturated NIC): the "gray failure", which is the hardest because the server is technically alive.

Your client therefore needs short timeouts (tens of milliseconds, not seconds) and a policy for what to do next.

Policy 1: treat failures as misses

The simplest policy: if the server is unreachable, behave as if the key was not in the cache, read from the database and skip the cache write. No new rebuilds hit the pool, and nothing is mapped to the wrong place. The downside is that every request for that server's keys now pays the database cost until the server returns.

Policy 2: eject the node from the ring

After N consecutive failures the client removes the server from its ring for a cool-down period (ketama-style clients call this auto-eject). Its keys are redistributed to the other servers, which start filling with those keys. Hit ratio recovers, but be careful: when the node comes back, the ring changes again and the keys that had moved are now served by the old node, which is cold.

Python Hit ratio when one of six nodes fails

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The toy caches here have no size limit, so the healthy cluster scores 100%. With one node unreachable, about 15% of reads fall through to the database and stay there for as long as the node is down. With the node ejected, the survivors only miss once per moved key while they fill, so the damage is a short dip rather than a lasting loss. A real, size-limited cache would also evict useful data to make room for the newcomers.

Policy 3: a gutter pool

Facebook's published design for large Memcached clusters includes a gutter pool: a small set of spare servers that take over when a regular server fails. Requests for keys from the failed node go to the gutter servers, which cache entries for only a short TTL (seconds). That has two benefits compared to plain ejection:

  • The database is protected: repeated reads of the same keys are absorbed by the gutter.
  • Stale data stays bounded: because the TTL is short, the gutter cannot hold a wrong value for long, and invalidations (which only reach the normal servers) cause no lasting damage.
flowchart LR
  App["Client"] --> Pool["Regular pool"]
  Pool -->|"server unreachable"| Gutter["Gutter pool (short TTL)"]
  Gutter -->|"miss"| DB["Database"]

The gutter is idle in normal times. It only receives traffic for keys whose home server is down.

Restoring a node

A restarted Memcached is empty. Bringing it into the ring at full traffic produces a miss storm for its share of keys (see Cache Stampede Protection). Mitigations:

  • Warm up the node: replay the top keys from another source before sending live traffic.
  • Ramp up gradually, as proxies like mcrouter can do by routing a small percentage of requests first.
  • Replicate important keys across two nodes (in the style of Hot Keys and Replication) so one failure is absorbed by the other.

Designing for failure

  • Capacity: size the database for the worst plausible cache loss, for example one node of N down, not only for the healthy hit ratio.
  • More, smaller nodes: 20 nodes of 4 GB lose 5% when one fails; 4 nodes of 20 GB lose 25%. Many small nodes also mean cheaper replacement.
  • Spread across failure domains: do not put every node in the same rack or availability zone.
  • Monitor: alert on a falling hit ratio, rising get_misses, and the number of ejected servers, as covered in Hit Ratio and Metrics.
  • Rehearse: switch a node off in a quiet period and watch the dashboards. Never let your first failure happen in production peak traffic.

Memcached has no built-in replication or persistence. If you need your data to survive a restart, the cache is the wrong place to keep it as the only copy. Treat every item as rebuildable.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a client lose when one Memcached node dies?

    Only the cached copies on that node vanish. The source of truth is intact, but the database takes extra reads.
  2. 2. Why are short client timeouts important?

    A long timeout on a dead server can exhaust workers and turn a partial outage into a total one.
  3. 3. What is the purpose of a gutter pool?

    The gutter takes traffic for a failed server and uses short TTLs so stale data cannot linger.
  4. 4. Why does a cluster of many small nodes handle a single failure better?

    Losing one of 20 nodes removes 5% of the cache, while losing one of 4 removes 25%.

Summary

  • A dead node costs hit ratio, not data; size the database for that.
  • Use short timeouts and treat unreachable servers as misses.
  • Ejecting a node moves its keys to cold servers; restoring it moves them back to a cold node.
  • A gutter pool with a short TTL absorbs a failed node's traffic safely.
  • Warm up or ramp up restarted nodes, spread nodes over failure domains and rehearse failures.

Next: Consistency Models and CAS