Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 37 ยท Consistency and Failure

Stale Data and Race Conditions

Replay the exact interleavings that leave stale values in Memcached after a database update, and learn which write order, TTLs and deletes keep the damage small.

A cache is a copy, and copies drift. Between the moment the database changes and the moment the cache notices, readers may see the old value. For a product description that is fine. For a stock count at checkout or a permissions flag it may not be. The dangerous case is not the short, expected delay; it is a race that leaves the wrong value in the cache until its TTL expires, possibly for hours.

In this lesson you will replay, step by step, the interleavings that cause such bugs, using a tiny in-memory database and cache. The real Memcached is not in the live editors, so a dictionary stands in for it. The races are identical because they come from the order of operations, not from the cache product.

Setup: cache-aside with a stand-in database

The usual pattern, cache-aside, works like this:

  • Read: look in the cache; on a miss read the database and set the cache.
  • Write: update the database, then invalidate (delete) the cached key.
Python
def read(key):
    v = cache.get(key)
    if v is None:                 # miss
        v = db[key]
        cache[key] = v
    return v

Two requests running at the same time interleave their steps. The simulation below lets us choose the exact order.

Race 1: a slow reader repopulates old data

Reader A misses and reads the old value from the database. Before A writes it into the cache, writer B updates the database and deletes the cache key. Then A finally stores its (now stale) value. The cache now disagrees with the database, and nothing will fix it until the TTL ends.

Python A stale value survives a delete

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This race needs a slow reader and a write in a very narrow window, so it is rare, but at millions of requests per day rare things happen daily. Two defences:

  • A TTL on everything. The wrong value expires eventually. Choose TTLs by how long you can tolerate wrongness, not by how long the data is valid.
  • add instead of set for fills, plus a lease token (see Cache Stampede Protection and Consistency Models and CAS). With a lease, B's delete invalidates A's right to fill.

Race 2: wrong order, delete before update

Some code deletes the key first and then updates the database. That opens a wider window: a reader that arrives between the delete and the update finds a miss, reads the old row and caches it.

Python Why delete-then-update is worse

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The correct order is update the database, then delete the cache. The window for race 1 is then limited to the time of a single read-and-fill by a reader that started before the update, which is much narrower than "the entire duration of the write transaction".

Race 3: write-through with concurrent updates

If writers set the cache with the new value instead of deleting, two concurrent writers can store their values in the opposite order to their database commits:

  1. Writer 1 commits price=110.
  2. Writer 2 commits price=120.
  3. Writer 2 sets cache 120.
  4. Writer 1 sets cache 110 (it was slower).

The database says 120; the cache says 110. Deleting is safer than setting because a delete is idempotent and order-insensitive: it does not matter which writer deletes last, the next read rebuilds from the database.

sequenceDiagram
  participant A as Reader A
  participant C as Cache
  participant D as Database
  participant B as Writer B
  A->>C: get price (miss)
  A->>D: read price = 100
  B->>D: update price = 120
  B->>C: delete price
  A->>C: set price = 100 (stale)

The stale set at the end is the whole bug: A's read happened before B's write but its cache fill happened after B's delete.

How much staleness is acceptable?

Ask for each kind of data:

Data Tolerable staleness Typical approach
Blog post HTML minutes to hours TTL, delete on edit
Product price seconds to minutes delete on change plus TTL
Stock at checkout none read the database, not the cache
Permissions seconds short TTL, delete on change
Session none for logout delete on logout, short TTL

Never use the cache as the source of truth for anything you cannot rebuild. Memcached may evict any item at any time, so it must always be safe to lose a key.

When in doubt, shorten the TTL. A 60 second TTL turns almost every cache bug into a one-minute glitch instead of a day-long incident.

Another reason for stale reads: failures and lists

Stale data also appears when a server is unreachable for a while and the client keeps serving a copy it had cached locally, or when two app servers disagree about the pool (see Node Failure and Failover). Treat it as a sign to measure how old your data can get, not just whether it is correct "most of the time".

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In race 1, why does a stale value stay in the cache?

    The reader's database read happened before the write, but its set happened after the delete, so the old value is stored.
  2. 2. Which write order is safer for cache-aside?

    Deleting after the commit leaves only a narrow window; deleting before it leaves a wide one.
  3. 3. Why is deleting safer than setting the new value from concurrent writers?

    Sets can land out of commit order and leave an older value, but the last delete always leads to a fresh rebuild.
  4. 4. What is the cheapest safety net against races you did not foresee?

    A TTL bounds how long any wrong value can live.

Summary

  • Cache-aside is eventually consistent; races can leave stale values until the TTL ends.
  • Update the database first, then delete the cache key.
  • Prefer deletes to sets when several writers can run at once.
  • Always set a TTL, chosen by tolerable staleness.
  • Never keep data only in the cache; keys can disappear at any time.

Next: Cache Stampede Protection