Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 65 ยท Caching

Caching Fundamentals and Strategies

Learn what a cache is, where caches live, hit rate math and the cache-aside, read-through, write-through, write-around and write-back strategies, with Python.

A cache is a small, fast store that keeps copies of data that is expensive to fetch or compute, so the next request for the same data can be answered quickly. Caching is the single most effective performance tool in system design, and it works because of a simple fact about real traffic: a small part of the data gets most of the requests, and recently used data is likely to be used again.

The price of a cache is staleness: the copy can become out of date. Almost every caching decision balances speed against freshness and complexity, and the rest of this chapter explores those trade-offs: strategies (this lesson), eviction, invalidation, distributed caches, CDNs and warming.

Why caching works

Reading from memory takes about 100 nanoseconds; a database query over the network takes milliseconds, tens of thousands of times longer. Two patterns of access make caches effective:

  • Temporal locality: data used just now will probably be used again soon.
  • Skewed popularity: a few items (the front page, a trending video) are requested far more than the rest, often following the 80/20 pattern.

A cache sits between a client and a slower origin (database, API, disk):

flowchart LR
  A["Application"] -->|"1. look up"| C[("Cache<br/>fast, small")]
  C -->|"hit: return"| A
  C -.->|"miss"| D[("Database<br/>slow, large")]
  D -.->|"2. fetch and store"| C

Figure: a hit is answered from the cache; a miss goes to the origin, and the result is stored for next time.

Where caches live

Caches exist at every layer, and a request may pass several:

  1. Browser cache: files and responses stored on the user's device (controlled by HTTP headers).
  2. CDN and edge cache: copies near the user (CDN lesson).
  3. Reverse proxy or gateway cache: in front of your servers.
  4. Application cache: in-process memory (a dictionary inside the app) for tiny, hot data.
  5. Distributed cache: a shared service like Redis or Memcached.
  6. Database cache: the buffer pool and query cache inside the database engine.
  7. CPU caches and the operating system's disk cache, which you get for free.

Cache layers

Browser Closest to the user. No network at all.
CDN / edge Near the user. Great for static content.
App and distributed cache Shared, fast. Saves database work.
Database buffers Keeps hot pages in memory.

Each layer removes work from the next one.

1 / 2

Hit rate and average latency

The key metric is the hit rate: the fraction of lookups answered by the cache. The average access time is:

average = hit_rate x cache_time + (1 - hit_rate) x origin_time

Because the origin is so much slower, even a small number of misses dominates. A 95% hit rate and a 99% hit rate look similar, but the second sends five times less traffic to the database. When sizing a cache, watch hit rate, miss rate, eviction rate and the origin's load.

Python Hit rate and what it saves

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Going from 90% to 99% hits cuts database queries from 1,000 to 100 per second and average latency from about 5 ms to 1.4 ms. Cache sizing is therefore often about the last few percentage points.

Caching strategies

A strategy decides who talks to the cache and the database, and in which order. The choice sets your consistency, latency and complexity.

Cache-aside (lazy loading)

The application owns the logic: read from the cache; on a miss, read the database, then put the result in the cache. On a write, update the database and then invalidate (delete) the cached entry so the next read reloads it. It is the most common pattern: simple, caches only what is requested, and survives a cache outage (the app just goes to the database). The weaknesses: the first request for any item is always slow, and there is a window in which the cache can briefly be stale.

Read-through

The application talks only to the cache. The cache itself loads missing data from the database, through a loader function. Application code is simpler and the loading logic lives in one place; it needs a cache library or service that supports it.

Write-through

Every write goes to the cache and the database together, and is acknowledged only when both succeed. The cache is always fresh and reads are fast, but writes are slower (two stores), and you may cache data that is never read again.

Write-around

Writes go straight to the database, bypassing the cache; the cache is filled only when the data is read. It avoids filling the cache with write-once data (logs, uploads), but a read right after a write is a miss.

Write-back (write-behind)

Writes go to the cache and are acknowledged immediately. The cache flushes them to the database later, in batches or after a delay. Writes are extremely fast and batching reduces database load, but if the cache fails before flushing, data is lost, so it needs replication or a durable log. It suits high write rates where a small risk is acceptable (counters, metrics).

Strategy Read path Write path Strength Risk
Cache-aside App checks cache, then DB DB, then delete cache key Simple, resilient Stale window, cold misses
Read-through Cache loads from DB DB (often with write-through) Clean code Needs support
Write-through Cache Cache and DB together Always fresh Slower writes
Write-around Cache, DB on miss DB only No cache pollution Miss after write
Write-back Cache Cache, DB later Very fast writes Data loss if cache dies

Implementing the strategies

Here are cache-aside, write-through and write-back in one runnable file, using dictionaries to stand in for the cache and the database, with counters so we can see how many times each store was touched.

Python Cache-aside, write-through and write-back

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Write-back collapsed a thousand updates into a single database write, which is exactly why it exists. If the process had crashed before flush(), those updates would be gone.

A cache-aside simulation with realistic access

Real traffic is skewed. The next simulation draws keys from a skewed popularity distribution and measures the hit rate of a cache holding only a fraction of the data. Even a small cache hits well when popularity is concentrated.

Python Small cache, skewed traffic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A cache holding just 5% of the items answers more than half of the requests, and 20% of the items answer about 70%, because the popular items stay in it. Returns diminish as the cache grows, so size it by measuring the hit rate against memory cost. Which items to remove when it fills is the topic of the next lesson.

What to cache and what not to

Good candidates: results of expensive queries, rendered pages and fragments, API responses from slow third parties, user profile data, configuration, computed aggregates, session data, static files.

Poor candidates: data unique to each request, highly volatile data that must be exact (account balance during a transaction), very large objects that evict many small ones, and anything private that cannot be separated by user in the key.

Rules of thumb:

  • Cache read-heavy data first; measure hit rate afterward.
  • Give every entry a time to live so mistakes eventually heal.
  • Use clear keys that include everything the value depends on (product:42:lang:en).
  • The cache must be optional: if it fails, the system slows down but still works. Never make the cache the only copy of important data unless you use write-back with durability.
  • Treat caching as an optimization you can remove, not as part of your data model.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In cache-aside, what does the application do on a cache miss?

    The application loads from the origin and fills the cache for the next request.
  2. 2. Which strategy risks losing acknowledged writes if the cache crashes?

    Write-back acknowledges after updating only the cache and flushes to the database later.
  3. 3. A hit rate rises from 90% to 99%. About how does database load change?

    Misses fall from 10% to 1% of requests, so the origin sees one tenth of the queries.
  4. 4. Why should a cache never be the only place important data lives (in cache-aside)?

    Entries can be evicted, expire or vanish in an outage; the origin must hold the real data.
  5. 5. Which strategy is best for write-once data that is rarely read again, like logs?

    Write-around skips the cache on writes, avoiding pollution with data nobody reads.

Summary

  • A cache keeps fast copies of slow data; it works because access is local and skewed.
  • Layers range from the browser to CDN, proxy, application, distributed cache and database buffers.
  • Hit rate drives latency and origin load; the last few points matter most.
  • Strategies: cache-aside, read-through, write-through, write-around, write-back, each trading freshness, speed and durability.
  • Keep the cache optional, key it carefully, add TTLs and measure.

Next: Cache Eviction Policies and LRU