Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 31 ยท Memory

Global Memory and Coalescing

Understand how a warp's global loads become memory transactions, why coalesced access is fast, and how SoA, alignment and indexing change bandwidth.

Global memory is where almost all your data starts and ends, and how you read it often decides whether a kernel runs at 10 percent or 90 percent of the card's bandwidth. The hardware does not serve threads one by one. It looks at the 32 addresses requested by a warp at the same instruction and fetches the smallest set of memory chunks that covers them. When those addresses sit next to each other, a few chunks are enough. When they are scattered, the same instruction triggers many chunks and most of the bytes fetched are thrown away.

This lesson explains that rule, called coalescing, with a small transaction counter you can run. You will see why a[i] is good and a[32 * i] is terrible, why a struct of three floats hurts, and how to lay out a matrix so threads walk along memory instead of across it. If warps are new to you, read Warps, SMs and the Hardware first.

CUDA code in this lesson is read-only because the site has no GPU. The Python examples are a model: they count how many 32-byte sectors a warp touches. Real hardware has more details (caches, replays, different levels of the hierarchy), but the sector count is an excellent first-order predictor of bandwidth use.

Memory moves in chunks

The DRAM and cache system never delivers a single 4-byte float on its own. On current NVIDIA GPUs, global memory traffic between L2 and the SM is organized in 32-byte sectors, and the L1 cache works on 128-byte lines made of four sectors. If your warp needs one float, the hardware still moves a whole sector.

So one warp-wide load instruction costs as many transactions as the number of distinct sectors its 32 addresses fall into. For a warp reading 32 consecutive float values, that is 32 x 4 = 128 bytes, which fills exactly four sectors, and every fetched byte is used. That is the best case.

flowchart LR
    W["Warp: 32 threads<br/>one load instruction"] --> C["Hardware groups the<br/>32 addresses by sector"]
    C --> G["Coalesced:<br/>4 sectors for 128 useful bytes"]
    C --> B["Strided: up to 32 sectors<br/>for the same 128 useful bytes"]

The same warp instruction costs between 4 and 32 sector transactions depending on its address pattern.

Counting transactions

The next example takes the byte address each thread of a warp reads and counts the sectors touched. "Useful" means distinct bytes the threads asked for divided by bytes moved.

Python Transaction counter for access patterns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table from top to bottom. Contiguous access is 100 percent efficient. Stride 2 halves the efficiency because every sector carries four floats but the warp only uses two of them. At stride 8 and beyond, every thread lands in its own sector, so the warp moves 1,024 bytes to use 128. That is 8 times as much traffic, and since the kernel is probably limited by memory bandwidth, it will run about 8 times slower than the contiguous version.

The "broadcast" row is a special case: all threads read the same address, so only one sector is needed. The row's low "useful" percentage is only because a single float is a small part of its sector; the transaction count of 1 is what matters, and it is why broadcast reads are cheap.

Misalignment is mild on modern GPUs

The second row, a[i+1], shows a warp that starts four bytes past a sector boundary: 5 sectors instead of 4, so efficiency drops from 100 to 80 percent. On old GPUs misalignment was a disaster; on current ones it costs one extra sector per warp request. You still want aligned bases. Memory from cudaMalloc is aligned to at least 256 bytes, so the start of an array is fine, but a + 1 or an odd row length in a pitched matrix will shift every warp.

Which index should threads use?

The rule is simple: adjacent threads (consecutive threadIdx.x) should touch adjacent addresses. Compare two ways to add one to every element of a row-major matrix with cols columns:

C++
// Coalesced: consecutive threads handle consecutive columns of one row.
__global__ void good(float* m, int rows, int cols) {
    int col = blockIdx.x * blockDim.x + threadIdx.x;
    int row = blockIdx.y * blockDim.y + threadIdx.y;
    if (row < rows && col < cols) m[row * cols + col] += 1.0f;
}

// Strided: consecutive threads handle consecutive ROWS of one column.
__global__ void bad(float* m, int rows, int cols) {
    int row = blockIdx.x * blockDim.x + threadIdx.x;
    int col = blockIdx.y * blockDim.y + threadIdx.y;
    if (row < rows && col < cols) m[row * cols + col] += 1.0f;
}

The two kernels do identical arithmetic. In good, the thread index x moves along a row, so a warp covers 32 neighbours. In bad, x selects the row, so neighbouring threads are cols floats apart: the same stride pattern as a[32*i] above when cols is large.

Python Row walk versus column walk

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The column walk touches 32 sectors, eight times the row walk. The last lines show alignment: a contiguous warp costs 4 sectors when it starts on a sector boundary (bytes 0 or 32) and 5 otherwise. Starting at byte 124 is the worst alignment for this pattern: the warp covers bytes 124 to 251, which spills over five sectors.

When your algorithm genuinely needs column access (a matrix transpose is the classic example), you do not accept the strided pattern. You stage data through shared memory so that both the global read and the global write are coalesced. Bank Conflicts and Padding finishes that story.

Array of structures versus structure of arrays

Programmers like to group related values in a struct:

C++
struct Particle { float x, y, z; };      // 12 bytes
Particle* particles;                     // array of structures (AoS)
// thread t reads particles[t].x

Consecutive threads now read addresses 12 bytes apart, a stride of three floats. Each thread wants only the x field but the sectors also contain y and z. The alternative stores each field in its own array:

C++
struct Particles { float *x, *y, *z; };  // structure of arrays (SoA)
// thread t reads p.x[t]

Now consecutive threads read consecutive floats, which is perfectly coalesced. Let us count both layouts, first for one field and then for a kernel that reads all three fields of 1,024 particles.

Python AoS versus SoA

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For a single field SoA is three times better. When the kernel uses all three fields the model still says AoS touches three times as many sectors, because each field load is a separate instruction that touches the whole 384-byte span of the warp's structs. Real GPUs soften this: after the first load, the neighbouring fields are probably sitting in L1, so the later loads hit the cache. That is exactly why AoS is "only" a moderate loss in practice and a large loss when only one field is used or when the structs are large. SoA is the safe default for performance-critical data.

If you must keep structs, make them friendly to wide loads. A 16-byte aligned struct such as struct __align__(16) Point { float x, y, z, w; } lets one thread load all four floats with a single 128-bit instruction. The built-in vector types float2 and float4 do the same:

C++
float4 v = reinterpret_cast<const float4*>(in)[i];   // one 16-byte load per thread

A warp executing that load reads 512 bytes in one instruction, that touches 16 sectors, the minimum for that many bytes, so it is fully coalesced.

Practical checklist

  • Map threadIdx.x to the fastest-changing index of your data (the last index in row-major arrays).
  • Use structure-of-arrays for data that threads read field by field.
  • Keep array bases aligned and prefer row lengths that are multiples of the warp size (or let cudaMallocPitch pad rows).
  • Read each value from global memory once; use shared memory if several threads need it (next lesson).
  • Do not trust intuition: the profiler reports "global load efficiency" and sectors per request, which is the same quantity this lesson counted by hand. See Profiling with Nsight.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A warp reads 32 consecutive 4-byte floats starting at a 32-byte boundary. How many 32-byte sectors does that request need?

    32 floats are 128 bytes, which is exactly four aligned 32-byte sectors, and every byte fetched is used.
  2. 2. Thread t reads a[32 * t] (a float array). Compared with a[t], about how much memory traffic does the warp generate?

    Each thread lands in its own sector, so 32 sectors (1,024 bytes) move to deliver 128 useful bytes, which is 8 times the contiguous case.
  3. 3. Why does a structure of arrays usually beat an array of structures for GPU kernels?

    With SoA, thread t reads x[t], so neighbouring threads read neighbouring floats and the warp's request is coalesced.
  4. 4. In a row-major matrix, which mapping gives coalesced global access?

    Consecutive threads (consecutive threadIdx.x) must touch consecutive memory addresses, and in row-major storage that means consecutive columns.

Summary

  • A warp's load becomes as many memory transactions as the number of distinct 32-byte sectors its addresses touch.
  • Contiguous, aligned access is the ideal: 4 sectors for 128 bytes of floats. A stride of 8 floats or more wastes 7 of every 8 bytes fetched.
  • Misalignment costs about one extra sector per request on modern GPUs; broadcast reads (same address) need only one.
  • Map the fastest-moving thread index to the fastest-moving data index.
  • Prefer structure of arrays; use float4 or 16-byte aligned structs when each thread needs several neighbouring values.
  • When the algorithm needs a transposed pattern, route it through shared memory.

Next: how a block can load a tile once and reuse it, in Shared Memory and Tiling.