Skip to content
Tutorials Center
My learning

Topics

Lesson 13 of 55 ยท Inside the Machine

Memory: RAM, Cache and the Memory Hierarchy

Understand RAM, memory addresses, volatile vs non-volatile storage, CPU cache levels, locality of reference, virtual memory and swap, with runnable simulators.

A processor is only as useful as the data it can reach. Programs and their data have to live somewhere the CPU can read quickly, and there is a basic conflict in hardware: memory that is fast is small and expensive, and memory that is large and cheap is slow. No single technology gives you all three of speed, size and low cost.

Computers solve this by using several kinds of memory arranged in a hierarchy: a little very fast memory close to the CPU, more medium-speed memory further away, and a lot of slow storage at the bottom. This lesson explains each level, why the arrangement works (a property called locality), and what happens when you run out of RAM. It follows The CPU and is followed by Storage: HDD, SSD and Beyond.

Volatile and non-volatile memory

The first distinction is what happens when the power goes off.

  • Volatile memory forgets everything. RAM and CPU registers and caches are volatile. They are fast precisely because they hold data only while powered.
  • Non-volatile memory keeps its contents without power: SSDs, hard disks, USB sticks, and the chip that holds firmware.

That is why you must save a document: until you do, it exists only in RAM. A power cut or a crash and the unsaved work is gone.

RAM and addresses

RAM (random access memory) is the computer's working space. It holds the programs that are running, the documents that are open and the data they use. "Random access" means that any location can be read in about the same time, regardless of where the previous access was, unlike a tape that has to be wound.

RAM can be pictured as a very long row of numbered boxes. Each box holds one byte, and its number is its address. The CPU asks for "the 8 bytes starting at address 1,000,000" and gets them back. A 64-bit CPU can in theory use addresses up to 2^64, far more than any real machine has; in practice, operating systems offer each program a private view of memory that is mapped onto the real RAM (see Memory Management).

Typical sizes: 8 GB to 16 GB in a laptop or phone, 32 GB or more in a workstation, hundreds of GB in a server. More RAM does not make a CPU faster; it lets you keep more things in fast memory at once.

The memory hierarchy

Memory is organized like a pyramid. The higher the level, the faster, smaller and more expensive per byte it is.

flowchart TB
  A["Registers<br/>tens of values, under 1 ns"] --> B["L1 cache<br/>tens of KB, about 1 ns"]
  B --> C["L2 cache<br/>hundreds of KB to MB, a few ns"]
  C --> D["L3 cache<br/>several MB to tens of MB, ~10-20 ns"]
  D --> E["RAM<br/>gigabytes, ~50-100 ns"]
  E --> F["SSD<br/>hundreds of GB, ~50-100 microseconds"]
  F --> G["Hard disk, network storage<br/>terabytes, milliseconds"]

Each level is bigger and slower than the one above it; the numbers are typical orders of magnitude, not exact specifications.

The cache is a small amount of very fast memory, built into the CPU chip, that keeps copies of recently used data from RAM. Modern CPUs have several levels: L1 is the smallest and fastest (often per core), L2 is larger and slower, and L3 is bigger still and usually shared by all cores. When the CPU needs a byte, it checks L1 first, then L2, then L3, and only then goes to RAM.

  • A cache hit means the data was found in the cache: fast.
  • A cache miss means it was not: the CPU waits while the data is fetched from the next level, and a copy is placed in the cache for next time.

The difference between the levels is large. If one register access took one second, a trip to RAM would feel like several minutes, and reading from an SSD would be like waiting days. The TryIt below turns typical latencies into that human-scale picture and also shows how the hit rate controls the average access time.

JS Latency at human scale

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The numbers are illustrative; real chips differ, and each generation shifts them. The shape is what matters: a few percent of misses can dominate the average, so a high hit rate is worth a lot.

Why caches work: locality

A cache would be useless if programs accessed memory at random. Luckily they do not. Two patterns, together called locality of reference, appear in almost every program.

  • Temporal locality: data used recently is likely to be used again soon (a loop counter, a variable in a loop).
  • Spatial locality: data next to recently used data is likely to be needed soon (going through the items of an array one after another).

Because of spatial locality, caches do not fetch one byte at a time. They fetch a whole cache line, typically 64 bytes, so the next several accesses are already waiting in the cache.

The simulator below is a small direct-mapped cache with 8 lines of 16 bytes each. It runs four access patterns of 256 reads and counts hits. Walking through memory in order, a tight loop and jumping around randomly give very different results.

JS A cache hit and miss simulator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Sequential reads miss once per line (every fourth access: 4 integers of 4 bytes fit in 16 bytes) and hit the rest, so the hit rate is 75%. Skipping a full line each time never reuses a line, and random jumps over a region bigger than the cache almost always miss. The loop that reuses a few values hits nearly every time. This is why the order in which a program walks through its data can change its speed several times over, even though it does the same arithmetic.

When RAM runs out: virtual memory and swap

What if the programs you open need more memory than the machine has? Operating systems use virtual memory: each program gets a private address space, which the operating system divides into fixed-size pieces called pages (commonly 4 KiB) and maps onto real RAM as needed. Pages that have not been used recently can be moved out to a reserved area of the disk, called the swap file or page file. If a program touches a page that is no longer in RAM, a page fault occurs: the operating system loads it back from disk, possibly evicting another page.

This keeps the system alive, but disk is thousands of times slower than RAM. When the pages a program actively uses do not fit in RAM, the system spends its time moving pages back and forth instead of doing useful work. This is called thrashing, and it is why a computer that is short of RAM becomes dreadfully slow rather than just slightly slower. Details follow in Memory Management.

The final TryIt models a machine with only 4 pages of RAM that loops through its pages, evicting the least recently used page when it needs room. The 100,000 ns penalty per fault is an illustrative stand-in for an SSD read, not a measured value.

JS Why running out of RAM feels so slow

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With a working set of 4 pages or fewer, only the first touch of each page faults. With 5 pages, the loop evicts exactly the page it will need next, every time, so every access faults and the average cost jumps from about 1,000 ns to over 100,000 ns, roughly a hundredfold. Real systems use smarter replacement than this, but a working set larger than RAM is just as painful.

If a computer becomes sluggish when you open many tabs or programs, check memory use before anything else. Closing programs or adding RAM often helps more than a faster CPU.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens to data in RAM when the computer loses power?

    RAM is volatile memory. Without power it forgets its contents, so unsaved work must be written to non-volatile storage.
  2. 2. What is a cache miss?

    On a miss the CPU waits while the data is brought in from the next level (another cache or RAM), usually along with its neighbors.
  3. 3. Looping through an array from the first item to the last benefits from which kind of locality?

    Neighboring items sit in the same cache line, so fetching one brings the following ones along. That is spatial locality.
  4. 4. Why does a computer with too little RAM become very slow?

    Swapping pages in and out of storage takes thousands of times longer than a RAM access, so useful work is delayed (thrashing).

Summary

  • Fast memory is small and costly; slow memory is large and cheap, so computers use a hierarchy from registers and caches through RAM down to SSDs and disks.
  • RAM is volatile working memory made of byte-sized locations with numeric addresses; storage devices are non-volatile.
  • Caches hold recent data and whole cache lines; temporal and spatial locality are why they work, and the hit rate controls average speed.
  • Virtual memory lets programs exceed RAM by swapping pages to disk, but too much swapping (thrashing) makes a machine crawl.

Next, see where programs and files live permanently in Storage: HDD, SSD and Beyond.