Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 31 ยท Memory

The Memory Hierarchy

Learn the CUDA memory hierarchy: registers, local, shared, global, constant and texture memory, the L1 and L2 caches, plus scope, lifetime, latency and bandwidth.

A GPU can do arithmetic much faster than it can fetch the numbers to do arithmetic on. That single fact explains most of CUDA performance work. A modern GPU has thousands of cores, but all of them share one pool of device memory, and reaching into that pool costs hundreds of clock cycles. If every thread went to that pool for every operand, the cores would spend nearly all their time waiting.

The answer is a hierarchy: a few small, very fast memories close to the cores, and one big, slow memory far away. Your job as a CUDA programmer is to decide which data lives where. This lesson names every memory space, says who can see it and how long it lives, and gives you rough numbers for speed. The next lessons then zoom into the two spaces that matter most: global memory and shared memory.

The real CUDA code and hardware numbers in this lesson are read-only: this site has no GPU. The runnable examples are Python stand-ins that model the ideas, and the cycle counts in them are illustrative round numbers, not measurements of any specific card.

The spaces at a glance

Every variable in a kernel lives in exactly one memory space. The space decides three things: which threads can see the variable (its scope), when it disappears (its lifetime), and how expensive it is to touch.

flowchart TD
    T["Thread<br/>registers + local memory"] --> S["Shared memory<br/>one per block, on-chip"]
    S --> L1["L1 cache<br/>one per SM, on-chip"]
    L1 --> L2["L2 cache<br/>one for the whole GPU"]
    L2 --> G["Device DRAM<br/>global, local, constant, texture data"]
    G --> H["Host memory<br/>across PCIe or NVLink"]

Closer to the top means smaller, faster and more private; closer to the bottom means bigger, slower and more shared.

Here is each space in turn.

Registers

Registers hold the scalar variables you declare inside a kernel, such as int i or float sum. They are the fastest storage on the chip: an instruction reads them with essentially no extra delay. Each thread has its own private copy, and the value disappears when the thread ends.

Registers are a limited resource. An SM (streaming multiprocessor, the unit that runs blocks) has a fixed register file, typically 65,536 32-bit registers, shared by all the threads resident on it. A single thread can use at most 255. The more registers each thread uses, the fewer threads fit on the SM at once, which is why register pressure shows up again in Divergence, Registers and Instruction-Level Tuning.

Local memory

The name is misleading. Local memory is private to one thread, like registers, but it is physically stored in device DRAM, so it is as slow as global memory (the L1 and L2 caches soften it a little). The compiler puts a variable there in three cases:

  • the thread needs more registers than are available, so some values are spilled;
  • you declare an array that the compiler cannot keep in registers, usually because it is indexed with a value known only at run time;
  • the variable is a large structure.

You never write a special keyword for local memory. If a profiler reports local loads and stores, treat it as a hint that registers ran out.

Shared memory

Shared memory is a small, fast scratchpad on the SM itself, declared with __shared__. All threads in one block can read and write it, and no other block can see it. It lives as long as the block does. Its speed is close to the register file when accesses are well behaved, so it is the standard place to stage data that several threads reuse. A typical limit is 48 KB per block by default, with more available on newer GPUs after an explicit opt-in.

Global memory

Global memory is the big device DRAM: gigabytes of it. It is the memory you fill with cudaMalloc and cudaMemcpy. Every thread of every block can read and write it, the host can too (through copies), and it lives until you call cudaFree or the program ends. It is the only space that carries data between kernel launches. It is also the slowest, with latencies in the range of several hundred cycles.

Constant memory

Constant memory is a 64 KB read-only region of device DRAM, declared with __constant__ and filled by the host. Reads go through a dedicated cache. When all threads of a warp read the same address, that cache broadcasts the value in one step, which makes constant memory ideal for filter weights, physical constants and small lookup tables. If threads of a warp read different addresses, the reads are serialized, and constant memory becomes a bad choice.

Texture and read-only data

Texture memory is also ordinary DRAM, but read through a separate cache that is tuned for 2D spatial locality and can do hardware filtering and format conversion. For plain numeric work, the practical modern use is the read-only data path: marking a pointer const __restrict__ (or using __ldg()) lets the compiler load through that cache when it can prove the data does not change during the kernel.

The L1 and L2 caches

Caches are automatic: you do not allocate in them, the hardware fills them as you access global and local memory. L1 sits on each SM and often shares physical space with shared memory (the split is configurable on many GPUs). L2 is one larger cache shared by every SM, and it is where the whole GPU agrees about memory. Writes from different SMs become visible to each other through L2.

Reading the table

Python The memory spaces as a table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is a stand-in: just data and a filter. The values are order-of-magnitude figures that are common in CUDA teaching material (a register costs about one cycle, shared and L1 tens of cycles, DRAM hundreds). Exact numbers differ per architecture and are not guaranteed anywhere, so measure on your own card when it matters.

Latency and bandwidth are different things

Two numbers describe a memory. Latency is how long one request takes to come back. Bandwidth is how many bytes per second the memory can deliver when it is kept busy. Global memory has high latency but also very high bandwidth: depending on the card, from a few hundred GB/s (GDDR) to a few TB/s (HBM). A CPU has far lower bandwidth to its DRAM.

GPUs hide latency instead of avoiding it. While one warp waits for its load, the SM switches to another warp that is ready, so the arithmetic units stay busy as long as enough warps are resident. That is the link to occupancy. But hiding latency only works if you also respect bandwidth: ask for too many bytes, or ask in an inefficient pattern, and the memory system itself becomes the bottleneck.

Why reuse is the whole game

Suppose a kernel computes a 3-point stencil, where each output is the sum of an element and its two neighbours. Every input element is needed by three different threads. If each thread loads its three values from global memory, every element is fetched three times. If a block first copies its slice into shared memory once, and the threads then read from there, global traffic drops by almost a factor of three.

Python Counting the cost of a stencil

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The "+ 2" in the tiled version is the halo: each block needs one extra element on each side of its slice. The model ignores caching, overlap and warp scheduling, so the 2.47x it prints is not a prediction; it shows where the saving comes from. Real hardware caches would already catch some of the naive version's repeated loads, which is why you always measure.

Scope in action

Scope rules are the source of many beginner bugs: a thread cannot read another thread's registers, and a block cannot read another block's shared memory. The simulator below runs a tiny "kernel" with two blocks of three threads. Each thread has its own registers dictionary, each block its own shared dictionary, and there is one global_mem for everyone.

Python Who can see what

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice what the stand-in cannot show: it runs threads one after another, so thread 2 always finds the leader already written. On a real GPU the threads of a block run concurrently, and without a barrier thread 2 might read before thread 0 writes. That timing problem is the subject of Shared Memory and Tiling.

A quick tour in real CUDA

This is how each space looks in source code. It is read-only here; compile it with nvcc on a machine with a GPU.

C++
__constant__ float weights[9];          // constant memory, set by the host

__global__ void demo(const float* __restrict__ in, float* out, int n) {
    __shared__ float tile[256];         // shared memory, one copy per block
    int i = blockIdx.x * blockDim.x + threadIdx.x;   // registers
    float sum = 0.0f;                                // register

    if (i < n) {
        tile[threadIdx.x] = in[i];      // global -> shared
    }
    __syncthreads();

    if (i < n) {
        sum = tile[threadIdx.x] * weights[0];        // shared and constant reads
        out[i] = sum;                   // register -> global
    }
}

Here in and out point into global memory (allocated by the host with cudaMalloc), tile is shared, weights is constant, and i and sum live in registers. No local memory is needed unless the compiler runs out of registers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which memory space is private to a single thread but physically stored in device DRAM?

    Local memory is per-thread in scope, yet it lives in DRAM, so it is as slow as global memory. Spilled registers and dynamically indexed arrays end up there.
  2. 2. Which statement about shared memory is correct?

    Shared memory is allocated per block, lives on the SM, and is released when the block finishes. Other blocks cannot see it.
  3. 3. When is constant memory a good fit?

    The constant cache broadcasts one value to the whole warp in a single step. Different addresses per thread are serialized.
  4. 4. How does a GPU mainly deal with the high latency of global memory?

    Latency hiding by running many warps is the core design. It needs enough resident warps and memory access patterns that do not waste bandwidth.

Summary

  • Registers (per thread), shared memory (per block), and global memory (whole grid and host) are the three spaces you will use constantly.
  • Local memory is per-thread but lives in slow DRAM; it appears through register spills and dynamically indexed arrays.
  • Constant memory is a 64 KB read-only space that is fast when a warp reads one shared value; texture and read-only data paths use a separate cache.
  • L1 is per SM and L2 is shared by the GPU; both are automatic.
  • Latency (cycles per request) and bandwidth (bytes per second) are different; GPUs hide latency with many warps but can still be limited by bandwidth.
  • The goal is data reuse: load once from global memory, then reuse from faster spaces.

Next, see how the pattern of global accesses decides how much of that bandwidth you actually get: Global Memory and Coalescing.