Skip to content
Tutorials Center
My learning

Topics

Lesson 9 of 31 ยท Memory

Shared Memory and Tiling

Use CUDA shared memory, __syncthreads and tiling to cut global loads. Covers static and dynamic allocation and a tiled matrix multiply with runnable models.

Shared memory is the programmer-controlled cache of the GPU. It is a small, fast block of memory on each streaming multiprocessor that every thread of one block can read and write. Where a hardware cache guesses what you will reuse, shared memory lets you say so explicitly: load a tile of data from slow global memory once, let the whole block work on it, then move on to the next tile. This technique is called tiling, and it is behind most of the big speed-ups in matrix multiplication, convolutions, stencils and many other kernels.

Tiling has two ingredients. First, you must be able to declare the shared space (__shared__). Second, because threads of a block run concurrently and not in lockstep, you need a barrier, __syncthreads(), so nobody reads a tile before everyone has finished writing it. This lesson covers both, then works through tiled matrix multiplication with a model that counts how many global loads each version needs.

CUDA code here is read-only: the site has no GPU. The Python examples run for real and model the ideas (a loop that counts loads, and a tiny scheduler that shows what a barrier prevents). Where numbers appear, they come from those models, not from hardware.

Declaring shared memory

A variable with the __shared__ qualifier lives in shared memory, with one copy per block:

C++
__global__ void reverseBlock(float* data) {
    __shared__ float buf[256];            // static: size known at compile time
    int t = threadIdx.x;
    int i = blockIdx.x * blockDim.x + t;

    buf[t] = data[i];                     // global -> shared
    __syncthreads();                      // wait until the whole block has written
    data[i] = buf[blockDim.x - 1 - t];    // shared -> global, reversed within the block
}
// launch with exactly 256 threads per block:
// reverseBlock<<<numBlocks, 256>>>(d_data);

This is the static form. The size must be a compile-time constant. When the size depends on a launch parameter, use the dynamic form: declare an extern array with no size and pass the byte count as the third launch parameter:

C++
__global__ void reverseAny(float* data) {
    extern __shared__ float buf[];        // size comes from the launch
    int t = threadIdx.x;
    int i = blockIdx.x * blockDim.x + t;
    buf[t] = data[i];
    __syncthreads();
    data[i] = buf[blockDim.x - 1 - t];
}
// reverseAny<<<numBlocks, threads, threads * sizeof(float)>>>(d_data);

There is only one dynamic shared allocation per kernel. If you need several arrays, take one extern pointer and carve it into pieces yourself, for example float* a = buf; float* b = buf + n;.

Shared memory is a scarce resource per SM. By default a block may use up to 48 KB, and newer GPUs allow more after you opt in with cudaFuncSetAttribute. The more shared memory each block takes, the fewer blocks fit on an SM at the same time.

Python How tile size limits resident blocks

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The limits used here (100 KB of shared memory and 2,048 threads per SM) are example values; real GPUs differ, and a real SM also caps blocks per SM and registers. The pattern is what matters: a bigger tile means more reuse, but it also means more bytes per block and fewer blocks to hide latency with.

Why __syncthreads() matters

All threads of a block do not execute in perfect lockstep. A warp may be several instructions ahead of another warp. Without a barrier, a thread might read a shared slot before the thread responsible for filling it has run. __syncthreads() is that barrier: no thread proceeds past it until every thread of the block has reached it, and all shared and global writes made before it are visible to the block afterwards.

The simulator below makes this concrete. Each thread is a Python generator; a yield stands for the barrier. Every thread writes t * 10 into its own slot and then reads its neighbour's slot.

Python What a barrier prevents

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With the barrier every thread sees its neighbour's value. Without it, in the unlucky order where thread 0 runs completely first, threads 0 to 4 read slots that nobody has written yet and get None. Thread 5 is "lucky" because thread 0 already ran. On real hardware the failing order is not fixed: the program may pass many times and fail once, which is what makes a missing barrier such a nasty bug. The chapter on synchronization returns to races.

Two rules about barriers deserve emphasis:

  • Every thread of the block must reach the same __syncthreads(). Putting it inside an if that only some threads enter is undefined behaviour and commonly hangs the kernel.
  • You usually need two per loop iteration when you reload tiles: one after the loads (so everyone sees the tile) and one after the compute (so nobody overwrites the tile while a slower thread is still reading it).

Tiled matrix multiplication

To compute C = A x B for n x n matrices, thread (row, col) needs a dot product of row row of A with column col of B: n multiplications reading 2n values. The naive kernel fetches all of them from global memory, so across n^2 threads it does 2n^3 global loads, even though each element of A is needed by n different threads.

The tiled version splits the dot product into phases. Each block computes a T x T tile of C. In every phase the block loads one T x T tile of A and one of B into shared memory (each thread loads one element of each), waits at a barrier, then every thread does T multiply-adds from shared memory.

flowchart LR
    A["Tile of A<br/>T x T from global"] --> S["Shared memory<br/>As and Bs"]
    B["Tile of B<br/>T x T from global"] --> S
    S --> M["T multiply-adds<br/>per thread"]
    M --> R["Accumulate in<br/>a register"]
    R -->|"next phase"| A

Each phase loads two tiles once and reuses every loaded value T times.

Here is the full kernel, in read-only CUDA C++:

C++
#define TILE 16

__global__ void matmulTiled(const float* A, const float* B, float* C, int n) {
    __shared__ float As[TILE][TILE];
    __shared__ float Bs[TILE][TILE];

    int row = blockIdx.y * TILE + threadIdx.y;
    int col = blockIdx.x * TILE + threadIdx.x;
    float acc = 0.0f;

    for (int t = 0; t < (n + TILE - 1) / TILE; ++t) {
        int aCol = t * TILE + threadIdx.x;
        int bRow = t * TILE + threadIdx.y;
        As[threadIdx.y][threadIdx.x] = (row < n && aCol < n) ? A[row * n + aCol] : 0.0f;
        Bs[threadIdx.y][threadIdx.x] = (bRow < n && col < n) ? B[bRow * n + col] : 0.0f;
        __syncthreads();                       // tiles complete

        for (int k = 0; k < TILE; ++k) {
            acc += As[threadIdx.y][k] * Bs[k][threadIdx.x];
        }
        __syncthreads();                       // done reading before the next load
    }
    if (row < n && col < n) C[row * n + col] = acc;
}
// dim3 block(TILE, TILE); dim3 grid((n + TILE - 1) / TILE, (n + TILE - 1) / TILE);

Notice that all threads, including those outside the matrix, go through both barriers; the bounds checks guard only the loads and the final store, with zeros padding the edges.

Counting the savings

The model below does the same arithmetic in NumPy but counts every element it would fetch from global memory. Each pass of the by/bx loops is a block; each pass of the k0 loop is a phase.

Python Tiled matmul: counting global loads

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result matches the formula. Naive needs 2 n^3 = 8,192 loads for n = 16. A tiled version needs 2 n^3 / T: 4,096 for T = 2, 2,048 for T = 4, 1,024 for T = 8. The reduction equals the tile width, because each loaded value now serves T threads. This is why larger tiles help, up to the limits of shared memory and block size that the earlier table showed. On real hardware, caches already catch some of the naive version's repeated loads, so measured gains are smaller than the model, yet the direction and the reason are the same.

Also remember what the model does not capture. Reading As[ty][k] and Bs[k][tx] is fast only when the accesses spread over the shared memory banks; the next lesson, Bank Conflicts and Padding, shows when they do not.

When is shared memory worth it?

Use it when data is reused by several threads of the same block, or when you need to reorder accesses (for instance to turn a strided global access into a coalesced one). Do not use it for data each thread reads exactly once and in a coalesced way: the extra copy and barrier only add cost. Shared memory is also how threads of a block communicate, as in reductions (Parallel Reduction).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does __syncthreads() guarantee?

    It is a block-wide barrier. It does not synchronize different blocks and it does not involve the host.
  2. 2. How do you pass the size of a dynamic shared array to a kernel?

    The kernel declares extern shared float buf[], and the launch supplies the byte count as the third value in the angle brackets.
  3. 3. A tiled matrix multiply with tile width T needs roughly how many global loads compared with the naive kernel?

    Each value loaded into shared memory is reused by T threads, so global traffic drops by about a factor of T.
  4. 4. Why is a second __syncthreads() needed at the end of each tile loop iteration?

    Without it, a fast thread could start loading the next tile while a slow thread still computes from the current one.

Summary

  • __shared__ declares per-block memory on the SM; use a fixed size, or extern __shared__ with the byte count as the third launch parameter.
  • Threads of a block do not run in lockstep, so use __syncthreads() after writing shared data and before overwriting it; every thread of the block must reach each barrier.
  • Tiling loads a tile into shared memory once and reuses it, cutting global loads by about the tile width in matrix multiplication.
  • Bigger tiles mean more reuse but use more shared memory and may reduce the number of resident blocks.
  • Shared memory has its own performance rule, the bank structure, covered next.

Continue with Bank Conflicts and Padding.