Skip to content
Tutorials Center
My learning

Topics

Lesson 6 of 31 ยท The Programming Model

Warps, SMs and the Hardware

See how a GPU runs your blocks: streaming multiprocessors, warps of 32 threads, SIMT execution, occupancy limits, and why block sizes should be multiples of 32.

You can write correct CUDA without knowing how the chip works, but you cannot write fast CUDA that way. This lesson opens the box just enough to explain the rules of thumb from earlier lessons: why blocks should be a multiple of 32 threads, why launching thousands of threads is good rather than wasteful, and what occupancy means.

Nothing here can be run on real hardware on this site. The numbers used are typical of recent data-center GPUs but vary by chip, so treat them as examples and read your own device's limits with cudaGetDeviceProperties. The Python stand-ins implement simplified models of warps and occupancy.

Streaming multiprocessors

A GPU is built from many copies of one unit: the streaming multiprocessor (SM). Each SM has its own arithmetic lanes, a register file, a pool of fast on-chip memory (shared memory and L1 cache), and schedulers that issue instructions. A mid-range GPU has a few dozen SMs; the biggest have well over a hundred.

When you launch a grid, the hardware hands whole blocks to SMs:

  • A block runs entirely on one SM and never moves.
  • An SM can hold several blocks at once (they are resident), as long as resources allow.
  • When a block finishes, the SM gets another waiting block.
  • Blocks may run in any order, on any SM. That is why blocks must be independent.
flowchart TD
    G["Grid: 8 blocks"] --> Q["Waiting blocks"]
    Q --> S0["SM 0<br/>blocks 0, 2 resident"]
    Q --> S1["SM 1<br/>blocks 1, 3 resident"]
    S0 -. "block finishes, takes next" .-> Q
    S1 -. "block finishes, takes next" .-> Q

Blocks are dealt out to SMs as space frees up. A GPU with more SMs simply finishes the same grid in more parallel waves.

This design is how one compiled program runs well on a small laptop GPU and a giant data-center card: more SMs means more blocks in flight, with no code change.

Warps: the unit of execution

Inside a block the hardware does not treat threads individually. It groups them in warps of 32 consecutive threads (by linear thread index) and executes a warp together. Threads 0-31 of a block form warp 0, threads 32-63 form warp 1, and so on. For a 2D or 3D block the threads are first flattened in x-fastest order, then cut into groups of 32.

This is SIMT: single instruction, multiple threads. All 32 lanes of a warp are issued the same instruction in the same cycle, each operating on its own registers and data. It resembles SIMD vector instructions on a CPU, but each thread has its own program counter state and may take its own path, as explained next.

Divergence

What if the code branches?

C++
if (threadIdx.x % 2 == 0) {
    a();      // even lanes
} else {
    b();      // odd lanes
}

Within one warp, both groups cannot run at once with a single instruction stream. The hardware runs path a() with the odd lanes masked off, then runs path b() with the even lanes masked off. The warp takes the time of both paths. This is warp divergence. Branches that all 32 threads agree on cost nothing extra; branches that split a warp cost the sum of the paths. Different warps can take different paths with no penalty.

Python Masked execution of a diverging warp

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

When every lane agrees, only one pass is needed. A split (even/odd or half and half) requires two passes: the warp pays for both paths. Divergence, Registers and Instruction-Level Tuning returns to this with strategies to avoid it.

Why the block size should be a multiple of 32

A warp always has 32 lanes. If your block has 100 threads, the hardware creates 4 warps, and the last warp has only 4 real threads; the other 28 lanes are switched off but still occupy a scheduling slot. A block of 128 threads makes exactly 4 full warps. The rule of thumb, then, is: choose block sizes that are multiples of 32, so that no lanes are wasted.

Python How a block is cut into warps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A 16 x 4 block has 64 threads and so 2 warps. The first two rows (y = 0 and 1) are warp 0, the next two are warp 1, because the thread index is flattened with x fastest.

Latency hiding and resident warps

An SM has far fewer arithmetic lanes than resident threads. How does it use them? A memory read can take hundreds of clock cycles. Rather than wait, the warp scheduler switches to another resident warp that is ready in the very next cycle, at no cost, because every resident warp keeps its registers on the chip. This is the GPU's replacement for a big cache, and it explains the earlier advice: launch many more threads than there are lanes.

Occupancy

Occupancy is the ratio of resident warps on an SM to the maximum the SM supports. More resident warps give the scheduler more chances to hide latency. Three resources limit how many blocks fit on an SM at once, and the tightest one decides:

  1. Threads (or warps) per SM. For example 2,048 threads (64 warps) on many recent chips.
  2. Blocks per SM. For example a cap of 32 resident blocks.
  3. Registers. Each thread uses some number of registers; the SM's register file (for example 65,536 32-bit registers) is divided among resident threads.
  4. Shared memory. Each block that declares shared memory takes a share of the SM's pool.

The calculator below uses simplified limits resembling a recent data-center GPU. Real hardware also rounds register and shared-memory use up to allocation granularities, which this model ignores, so the official CUDA occupancy calculator and API will differ slightly.

Python A simplified occupancy calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table with the limiter column in mind. With light register use, 256-thread blocks reach 100% because eight of them fill the 2,048-thread budget. Raising registers per thread from 32 to 64 halves occupancy, and 128 registers cut it to 25%, because the register file runs out first. A block that asks for 48 KiB of shared memory lets only three blocks fit (38%). The 64-thread case just reaches both the thread and block caps at once (32 blocks of 64 threads), so small blocks are not automatically a problem here, though on chips with a lower block cap they would be.

Occupancy is a tool, not a goal

High occupancy helps hide latency, but 100% is not required for speed. A kernel that loads data once and does a lot of arithmetic per thread can be fastest at 25-50% occupancy, because it is not waiting on memory. Raising occupancy by forcing fewer registers can cause spilling to slow local memory and make things slower. Treat occupancy as a diagnostic: if profiling shows the SM idle waiting for memory and occupancy is low, raising it may help. The Optimization Workflow shows how to decide.

You can ask the CUDA runtime for a suggestion with the occupancy API (read-only):

C++
int minGrid, blockSize;
cudaOccupancyMaxPotentialBlockSize(&minGrid, &blockSize, vecAdd, 0, 0);
// blockSize: a launch size that maximizes theoretical occupancy

Compute capability and limits

Everything above has per-generation numbers. The compute capability (for example 8.0, 8.6 or 9.0) determines the maximum threads per SM, blocks per SM, registers per SM, shared memory per SM and which instructions exist. Some values are the same across generations (32 threads per warp, 1,024 per block, 255 registers per thread), while others differ (for instance, 2,048 resident threads per SM on some chips and 1,536 on others). Look them up in the CUDA Programming Guide's feature table, or query them at run time:

C++
cudaDeviceProp p;
cudaGetDeviceProperties(&p, 0);
printf("warp size          %d\n", p.warpSize);
printf("max threads/SM     %d\n", p.maxThreadsPerMultiProcessor);
printf("registers per SM   %d\n", p.regsPerMultiprocessor);
printf("shared mem per SM  %zu\n", p.sharedMemPerMultiprocessor);

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is a warp?

    The hardware schedules and executes threads in warps of 32 lanes sharing an instruction stream.
  2. 2. A warp hits if (threadIdx.x % 2 == 0) A(); else B();. What happens?

    A warp has one instruction stream, so divergent paths are serialized with inactive lanes masked.
  3. 3. Why choose block sizes that are multiples of 32?

    A block of 100 threads still occupies 4 warps, and the last one has 28 inactive lanes.
  4. 4. Which of these can limit how many blocks are resident on an SM?

    The SM fits as many blocks as its tightest resource allows: threads, block count, registers or shared memory.
  5. 5. Is maximum occupancy always the fastest configuration?

    Occupancy helps hide latency but is a diagnostic. Forcing it higher can cause register spilling and slow the kernel down.

Summary

  • A GPU is a set of streaming multiprocessors; each block is assigned whole to one SM, and blocks run independently in any order.
  • Threads execute in warps of 32 (SIMT); a branch that splits a warp makes it run both paths one after another.
  • Use block sizes that are multiples of 32 and flatten multi-dimensional blocks with x varying fastest to know the warps.
  • The SM hides memory latency by switching between resident warps, so launch plenty of threads.
  • Occupancy is limited by threads, blocks, registers and shared memory; it is a diagnostic, not a target to maximize blindly.
  • Limits depend on compute capability; query them with cudaGetDeviceProperties instead of memorizing.

Next: learn where data lives and how fast each place is in The Memory Hierarchy.