Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 37 ยท Caching Concepts

Why Caching Works

Understand why caches speed up systems: the latency gap between memory and databases, repeated reads, read-heavy workloads, and when caching does not help.

A cache is a bet. You bet that the answer you computed once will be asked for again before it goes stale, and that remembering it is cheaper than recomputing it. When the bet pays off, a system gets faster and cheaper at the same time. When it does not, you have added a moving part and a new way to be wrong.

This lesson explains why the bet usually pays off, what numbers make it worthwhile, and how to recognise data that should not be cached at all.

The speed gap

Different storage layers differ by orders of magnitude in access time. The figures below are rounded, typical values; real hardware varies, but the ratios are what matter.

Where time goes

CPU cache about 1 ns
RAM about 100 ns
Memcached about 0.2 to 1 ms
Database about 1 to 100 ms

Each step is roughly 10x to 1000x slower.

1 / 3

A Memcached lookup crosses the network, so it is thousands of times slower than reading a variable, but it is still often 10 to 100 times faster than a database query that touches disk, joins several tables, or renders a page.

Four conditions for a good cache

1. The same data is requested repeatedly. A product page read by 10,000 visitors a day is a perfect candidate. A report that one person opens once a year is not: you would store it and never use it.

2. Reads far outnumber writes. If data changes on every read, every cached copy is stale immediately. A catalogue that changes once an hour and is read ten times a second has a read-to-write ratio of 36,000 to 1.

3. The result is expensive to produce. Caching a value that takes 20 microseconds to compute saves nothing once you add a network trip. Cache things that cost milliseconds or more: aggregated queries, rendered fragments, external API calls.

4. Slightly stale data is acceptable. If a user must see their account balance to the cent the instant it changes, you cannot serve a copy that is 60 seconds old. If a view counter can lag by a few seconds, you can.

Measuring the gain

The simplest model: each request either hits (cost h) or misses (cost m, which includes the slow work). With hit ratio r, the average cost is r * h + (1 - r) * m.

Python Average cost with and without a cache

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at how the gain grows: going from 90% to 99% hits is a bigger jump than going from 0% to 50%. That is why teams obsess over the last few points of hit ratio, a topic for the next lesson.

Offloading the database

Speed is only half the story. A cache also protects the backing store. If a database can serve 2,000 queries per second and your site sends 10,000 requests per second, a 90% hit ratio leaves the database 1,000 queries per second, which it can handle. Without the cache it would fall over.

Python How many requests reach the database?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

There is a danger hidden here: a site that depends on a 99% hit ratio to survive is one cache restart away from an outage. Chapter 6 covers cold caches and how to avoid that.

When caching does not help

  • Unique requests. Search queries with free text, or URLs containing a timestamp, almost never repeat. The hit ratio stays near zero while you pay the cost of storing everything.
  • Per-user private data with low reuse. A page that is different for every user and viewed once has no sharing.
  • Data that must be exact. Stock levels during checkout, payment status, anything where staleness causes real harm.
  • Cheap computations. Adding a network round trip to a one-microsecond calculation makes it slower.

A quick test: estimate how often the same key will be asked for before it expires. If the answer is "less than twice", skip the cache for that data.

Where does this fit with Memcached?

Memcached is one layer in a stack of caches. Browsers cache files, CDNs cache pages near users, your web server and framework cache templates, and the database has its own buffer pool. Memcached is the shared application cache: all your web servers read and write the same copy of the data. You will see how these layers interact in TTL and Expiration and the strategies chapter.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which data is the best candidate for caching?

    Frequently repeated, rarely changing and somewhat costly to produce: all conditions for a cache bet are met.
  2. 2. With a 0.5 ms hit and a 45 ms miss, what happens to the average time as the hit ratio goes from 90% to 99%?

    At 90% the average is about 4.95 ms; at 99% it is about 0.95 ms. The last points matter a lot.
  3. 3. Besides speed, what else does a cache give a database?

    Hits never reach the database, so it can survive traffic far beyond its own capacity.
  4. 4. When is a cache a poor choice?

    If keys never repeat, nothing is ever hit, and you pay storage and network cost for no benefit.

Summary

  • Caching trades a little memory and some staleness for large reductions in latency and load.
  • It works best for repeated, read-heavy, expensive, staleness-tolerant data.
  • Average cost = hit ratio times hit cost plus miss ratio times miss cost; the last points of hit ratio matter most.
  • A cache also shields the database, but do not build a system that collapses without it.
  • Skip caching for unique, exact or cheap data.

Next: Hit Ratio and Average Access Time