Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 31 ยท Performance

Divergence, Registers and Instruction-Level Tuning

Cut warp divergence, understand register pressure and spills, use __launch_bounds__ and fast math wisely, and get a first look at tensor cores.

Once your memory accesses are coalesced and your tiles fit in shared memory, the next layer of cost sits inside the instruction stream itself. Three things matter most there: whether the 32 lanes of a warp agree on which branch to take, how many registers each thread needs, and which arithmetic instructions the compiler picks. None of these changes what your program computes, but they can change how fast it runs by a large factor.

This lesson builds on Warps, SMs and the Hardware. The real compiler and GPU are not available on this site, so every CUDA snippet here is read-only. The Python TryIts are honest stand-ins: a small model of how a warp pays for each path it needs, and a toy register model. They show the shape of the costs, not real cycle counts.

Warp divergence, measured

Recall that a warp has one instruction stream. When lanes disagree on a branch, the hardware runs each needed path in turn with the other lanes masked off. A useful way to think about cost is: the number of distinct paths taken inside a warp. A warp where all lanes agree pays for one path; a warp split three ways pays for three.

The stand-in below counts, for every warp of a block, how many distinct branch outcomes it contains, and adds up the cost in "warp passes."

Python Divergence cost of different branch conditions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two branches that split the block at a warp boundary (tid < 128 and (tid // 32) % 2) cost nothing extra: each warp is uniform, only different warps differ. The branches that split lanes inside a warp cost two or even four times as much. The rule is simple: what matters is the pattern within each group of 32 consecutive threads, not how many threads take each path overall.

flowchart LR
    W["Warp: 32 lanes<br/>if (cond)"] --> A["Pass 1: lanes where cond is true<br/>others masked off"]
    A --> B["Pass 2: lanes where cond is false<br/>others masked off"]
    B --> R["Reconverge:<br/>all 32 lanes continue together"]

A divergent branch runs both sides one after another, then the warp reconverges. The cost is the sum of the two sides.

Avoiding divergence

You rarely need to eliminate every branch. A short if guarding a bounds check diverges only in the last warp of the grid and costs almost nothing. Worry about divergence when the branch bodies are long or when the split happens in every warp. Common remedies:

  • Make the condition warp-uniform. If the choice depends on data that is the same for a whole warp (a block index, a tile number, threadIdx.x / 32), no warp diverges.
  • Reorganize the data. If threads process items of different kinds, group items of the same kind next to each other so that neighboring threads take the same branch. Sorting or partitioning the input by category is often worth it.
  • Replace the branch with arithmetic. For tiny bodies, compute both results and select one. The compiler already does this for short branches (predication), where all lanes execute the instructions and a predicate decides which write takes effect.
  • Split the work into separate kernels. One launch per category removes the branch entirely, at the price of an extra launch.
C++
// Branchy version: lanes may disagree
if (x[i] > 0.0f) y[i] = sqrtf(x[i]);
else             y[i] = 0.0f;

// Branch-free: clamp first, then one straight-line computation
y[i] = sqrtf(fmaxf(x[i], 0.0f));

The second form gives the same result with no branch at all: every lane executes the same two instructions. When both sides of an if are cheap, the compiler often makes this kind of transformation itself with predicated or select instructions, so check the profiler before rewriting by hand.

The next stand-in shows the "reorganize the data" idea. Items are either cheap (1 unit of work) or expensive (20 units). A warp pays the cost of every path present in it, so mixing kinds in a warp is costly.

Python Sorting items so each warp takes one path

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With roughly one expensive item in ten, nearly every warp of 32 contains at least one, so in the mixed order almost every warp pays for both paths. After grouping by kind, only one warp (at most) contains both. The model is crude, but the lesson carries over: data layout can remove divergence.

Registers and register pressure

Each thread's local variables live in registers, the fastest memory on the chip. The SM has a fixed register file (for example 65,536 32-bit registers), shared by all resident threads. A kernel that needs 128 registers per thread allows only half as many resident threads as one that needs 64. You saw this effect on occupancy in the hardware lesson.

When a kernel needs more live values than the per-thread limit allows (the architecture caps it at 255), the compiler spills some of them to local memory. Despite the name, local memory is a private slice of ordinary global memory, cached in L1 and L2. A spill costs extra loads and stores, so a few spills are tolerable and many are harmful.

You can ask the compiler to report this without running anything:

Terminal
nvcc -O3 --resource-usage -arch=sm_80 kernel.cu
# ptxas info    : Used 64 registers, 0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads

If "spill stores" is nonzero, the compiler ran out of registers. The remedies are the usual ones: simplify the kernel, reduce unrolling, split it into two kernels, or move some values to shared memory.

The toy model below shows the tradeoff the compiler faces. A cap on registers raises occupancy, but values that do not fit spill. It is a made-up cost function, not a model of any real chip; use it to see why an unlimited push for occupancy can backfire.

Python A toy register cap versus spill model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In this toy, giving the kernel all it wants (96 registers) leaves occupancy at 25% and the relative time is 145. Capping at 64 lifts occupancy to 50% at the price of 32 spilled values, and the total gets worse (258); capping lower makes it worse still. Whether a cap helps on real hardware depends on the kernel, so measure.

Controlling registers: __launch_bounds__

You can give the compiler hints about how the kernel will be launched, so it can choose register counts that make that launch possible:

C++
__global__ void __launch_bounds__(256, 2)      // max 256 threads/block, want >= 2 blocks/SM
heavyKernel(const float* in, float* out, int n) { /* ... */ }

The first argument is the maximum block size you will ever launch; the compiler then knows no more than 256 threads need registers at once. The optional second argument asks for a minimum number of resident blocks per SM, which makes the compiler limit registers to fit. The global switch nvcc --maxrregcount=64 applies one cap to every kernel in the file; __launch_bounds__ is more precise because it is per kernel. If a launch ever exceeds the declared bound, it fails with a launch error, so keep the bound honest.

Instruction-level choices

Fast math

Single-precision math functions such as sinf, expf and 1.0f / x have accurate versions that take several instructions, and faster, less precise hardware approximations. Two ways to opt in:

  • nvcc --use_fast_math replaces many standard functions with their fast intrinsic forms (for example __sinf, __expf, __fdividef) and flushes denormal numbers to zero, for the whole file.
  • Call intrinsics directly, such as __expf(x), only where lower accuracy is acceptable.
C++
float a = expf(x);       // accurate (few ulp error)
float b = __expf(x);     // fast, approximate, error grows with |x|

The fast forms are a good match for graphics and some machine learning work; they can be a poor match for scientific code that needs stable results. As a rule, turn on fast math only after you have a test that checks the error you can tolerate.

Other cheap wins

  • Use float, not double, when you can. Many GPUs have far lower double-precision throughput (consumer chips especially). A stray 1.0 literal turns an expression into double precision; write 1.0f.
  • Prefer multiplication to division and use rsqrtf for reciprocal square roots.
  • Fused multiply-add (FMA). a * b + c usually compiles into one instruction with a single rounding. This is faster and slightly more accurate, but means results can differ from a CPU that rounds twice.
  • Unrolling. #pragma unroll removes loop overhead and exposes independent instructions, at the cost of registers.
  • Integer math. 32-bit integer multiply and index arithmetic is cheaper than 64-bit; use int indices unless arrays exceed two billion elements.

Tensor cores: a first look

Modern NVIDIA GPUs contain tensor cores, units that multiply small matrix tiles in a single operation. Where an ordinary core does one multiply-add per cycle per thread, a tensor core computes a whole small matrix product-and-accumulate (such as a 16x16 tile times a 16x16 tile) with warp-level cooperation. They accept reduced-precision inputs (FP16, BF16, TF32, INT8, and newer FP8) and accumulate in a wider type, which is exactly what deep-learning workloads need.

You can program them directly with the WMMA API (a warp loads fragments and calls mma_sync), with lower-level PTX instructions, or, almost always better, through a library: cuBLAS, cuDNN and CUTLASS use tensor cores when the data types and shapes allow it. Frameworks such as PyTorch switch them on through half-precision or TF32 settings.

C++
#include <mma.h>
using namespace nvcuda::wmma;
// One warp computes a 16x16x16 tile: D = A * B + C
fragment<matrix_a, 16, 16, 16, half, row_major> a;
fragment<matrix_b, 16, 16, 16, half, row_major> b;
fragment<accumulator, 16, 16, 16, float> acc;
fill_fragment(acc, 0.0f);
load_matrix_sync(a, aPtr, 16);
load_matrix_sync(b, bPtr, 16);
mma_sync(acc, a, b, acc);          // one warp-wide matrix multiply-accumulate

You can imitate the numerical effect on a CPU: multiply in low precision, accumulate in high precision.

Python Low-precision inputs, high-precision accumulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Rounding the inputs to 16 bits costs about three orders of magnitude of accuracy (the error goes from about 2e-7 to about 3e-4). Accumulating in 16 bits as well is a little worse again (about 5e-4 here), and the gap widens as more terms are summed, for example in a long dot product. This is why tensor cores pair narrow inputs with a wide accumulator. (Exact figures vary slightly with the NumPy build; the ordering of the three lines is what matters.)

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which branch causes warp divergence in a block of 256 threads?

    Even and odd threads sit in the same warp, so lanes of one warp disagree. The other conditions are the same for every lane of a warp.
  2. 2. What does a divergent warp pay for?

    The warp executes each needed path in turn with the non-participating lanes masked off.
  3. 3. What is a register spill?

    Local memory is per-thread storage backed by global memory and caches, so spills add memory traffic.
  4. 4. What does __launch_bounds__(256, 2) tell the compiler?

    The first argument is the maximum threads per block and the second a minimum number of resident blocks per SM; the compiler budgets registers to match.
  5. 5. Why do tensor cores usually accumulate in a wider type than their inputs?

    Narrow inputs save bandwidth and area, while a wide accumulator keeps the running sum accurate.

Summary

  • Divergence is about lanes inside one warp: a warp pays for every distinct path its lanes take. Branches that are uniform per warp are free.
  • Remove divergence by making conditions warp-uniform, grouping similar items together, using predicated selects for tiny bodies, or splitting work into separate kernels.
  • Registers are shared by resident threads; too many per thread lowers occupancy, and too few makes the compiler spill to slow local memory. Check --resource-usage output.
  • __launch_bounds__ gives the compiler per-kernel launch information; --maxrregcount is a blunt global cap.
  • --use_fast_math and intrinsics trade accuracy for speed; use them only when you can test the error. Write 1.0f to stay in single precision.
  • Tensor cores multiply small matrix tiles per warp using narrow inputs and wide accumulators; reach them through libraries unless you have a special reason.

Next: find out where your time actually goes with Profiling with Nsight.