Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 31 ยท Performance

The Optimization Workflow

A practical CUDA optimization workflow: profile first, tell memory-bound from compute-bound with the roofline model, use arithmetic intensity, and avoid occupancy myths.

Most GPU programs are not slow for the reason their authors expect. Someone unrolls a loop that takes 2% of the time, while the real cost is a transfer, a launch gap or an uncoalesced load that nobody measured. Optimization is a loop of measure, find the limit, change one thing, measure again. Skipping the first step wastes more engineering time than any other mistake in performance work.

This lesson gives you that loop. You will learn how to decide where the time goes, how to tell whether a kernel is limited by memory or by arithmetic using the roofline model and arithmetic intensity, why high occupancy is not the goal, and a checklist of the wins that pay off most often. The next lessons dig into specific techniques (divergence and instruction tuning) and tools (profiling with Nsight).

The commands and CUDA code here are read-only, because this site has no GPU. The runnable TryIts are Python stand-ins: calculators for the roofline and for latency hiding. The hardware numbers in them (20 TFLOP/s, 900 GB/s and so on) describe a made-up mid-range GPU, not a specific product.

The loop

flowchart LR
    A["Measure<br/>whole program first"] --> B["Find the limit<br/>memory, compute or latency"]
    B --> C["Change one thing"]
    C --> D["Verify results and re-measure"]
    D --> A

Each pass should change one variable so you know what helped.

Before touching a kernel, work from the outside in:

  1. Establish correctness and a baseline. Save the output of a trusted (usually CPU) version and a timing you can repeat. Time the whole task, including copies and launches, not only one kernel.
  2. Find the biggest cost. A timeline tool shows whether the time is in kernels, transfers or gaps where the GPU waits for the CPU (see Streams and Events).
  3. For the biggest kernel, find the limit. Is it waiting on memory, on arithmetic, or on something else such as latency or synchronization?
  4. Fix the limit, then repeat. After a fix the bottleneck moves, so measure again.

Amdahl's law: why you profile first

If a part of the program takes a fraction p of the runtime and you speed it up by a factor s, the whole program speeds up by 1 / ((1 - p) + p / s). Making a 5% part infinitely fast gains you at most 5%.

Python Where should the effort go?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The biggest block (kernel B, 45% of the time) is worth a 1.51x overall gain at a 4x local speed-up; the small kernel C returns 1.03x for the same effort. Notice also that the two copies together account for 43% of the time, so overlapping them with computation (see Streams and Events) may beat any kernel tweak.

Memory-bound or compute-bound?

Every kernel moves bytes and does arithmetic. Two hardware limits cap its speed:

  • Peak compute, in FLOP/s (floating-point operations per second). A GPU with a peak of 20 TFLOP/s can do at most 20 ร— 10ยนยฒ operations a second.
  • Peak memory bandwidth, in bytes/s from device memory. A bandwidth of 900 GB/s moves at most 900 ร— 10โน bytes a second.

Which one stops you depends on how much arithmetic you do per byte you fetch. That ratio is the kernel's arithmetic intensity (AI):

TEXT
arithmetic intensity = FLOPs performed / bytes moved to and from device memory

A kernel that adds two float vectors does 1 FLOP per element and moves 12 bytes (two 4-byte loads and one 4-byte store), so its AI is about 0.083 FLOP/byte. It can never keep the arithmetic units busy; it is limited by memory. A kernel that does 200 FLOPs on every value it loads has an AI far above that, and is limited by arithmetic.

The roofline model

The roofline model puts both limits in one formula:

TEXT
attainable FLOP/s = min( peak FLOP/s , AI ร— peak bandwidth )

Plotted against AI, this is a slanted line (the memory roof) that rises until it hits a flat ceiling (the compute roof). The AI where they meet is the ridge point, peak FLOP/s / peak bandwidth. Kernels with an AI to the left of the ridge are memory-bound; those to the right are compute-bound.

flowchart LR
    L["AI below the ridge point<br/>memory-bound<br/>fix: move fewer bytes"] --> R["ridge point<br/>peak FLOPs / peak bandwidth"]
    R --> H["AI above the ridge point<br/>compute-bound<br/>fix: do cheaper arithmetic"]

The ridge point splits the two kinds of kernel and tells you which resource to economize.

Python A roofline calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the table from the bottom up. Vector add can reach only about 0.4% of peak FLOP/s, no matter how well you write it; the best you can do is to reach the bandwidth limit. The polynomial has an AI of 10, still below the ridge point of 22.2, so it too is limited by memory in this model, but it reaches 45% of the peak. And a tiled matrix multiply moves along the roof as the tile grows: that is the reason shared memory tiling works. A tile of 16 gives an AI of 4 and a tile of 32 gives 8; still on the sloped part, which is why high-performance matrix multiplication goes further with register blocking (Project: Fast Matrix Multiplication).

How to use this in practice:

  • Compute the achieved numbers. Divide the FLOPs your algorithm needs by the measured time to get achieved FLOP/s, and divide the bytes it must move by the time to get achieved bandwidth. Compare each with the hardware peak.
  • If achieved bandwidth is near the peak (say 80% or more) the kernel is already doing its job. Reduce the number of bytes (fuse kernels, use half precision, reuse data in shared memory) or accept the result.
  • If both numbers are low, the kernel is limited by something else: latency (too little parallelism), uncoalesced access (Global Memory and Coalescing), divergence, synchronization or launch gaps.
  • If FLOP/s is near the peak and you want more, reduce the work: cheaper math, fast-math intrinsics, or tensor cores (Divergence, Registers and Instruction-Level Tuning).

The hardware peak depends on the instruction mix and clocks, so treat it as an upper bound that you approach rather than expect to reach.

Occupancy is not the goal

Occupancy is the ratio of resident warps on an SM to the maximum it supports (see Warps, SMs and the Hardware). Low occupancy is a symptom worth checking, because an SM needs enough independent work to hide memory latency while some warps wait. But occupancy is a means, not the objective. Past the point where latency is hidden, more occupancy buys nothing, and getting it can cost you: forcing a lower register count to fit more warps may cause register spills, which slow each thread down.

Little's law gives a way to estimate how much parallelism you need: the amount of data in flight must equal latency times bandwidth. To keep the memory system busy, enough requests must be outstanding at once, whether they come from many warps or from several independent loads inside each thread (instruction-level parallelism, ILP).

Python How much occupancy do you really need?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With one outstanding load per thread, this made-up GPU would need about 82% occupancy, while four independent loads per thread need only about 21% to saturate the memory system. The practical conclusion: if a profiler reports 25% occupancy for a bandwidth-bound kernel that already reaches 85% of peak bandwidth, leave it alone. If it reports 25% occupancy and 20% of peak bandwidth, the missing parallelism is the likely culprit, and the fix may be more threads, more independent loads per thread (unroll a loop and issue the loads first), or smaller blocks so more fit per SM.

A checklist of common wins

Roughly in order of how often they matter:

  1. Reduce host-device traffic. Keep data on the GPU between kernels; use pinned memory and overlap copies with compute; avoid copying results you do not need (Moving Data).
  2. Launch enough work. Thousands of threads at least; one block per SM is rarely enough. If the kernels are tiny, fuse them or use graphs.
  3. Coalesce global memory access. Adjacent threads should touch adjacent addresses; prefer structure-of-arrays (Global Memory and Coalescing).
  4. Reuse data. Tile with shared memory, keep values in registers, and fuse kernels so intermediate results never reach global memory.
  5. Avoid bank conflicts and divergence inside the hot loops (Bank Conflicts, Divergence).
  6. Choose the right precision. float instead of double, or half/bfloat16 with tensor cores, where the accuracy allows. Check whether the fast-math flags are acceptable.
  7. Use a library for standard operations; they are tuned per architecture (The CUDA Library Ecosystem).
  8. Only then reach for micro-optimizations such as unrolling, __launch_bounds__ and instruction tuning.

Change one thing at a time and keep each change only if the measured time improves and the output still matches your reference. Beware of timing a single run: warm up first, run several iterations and report the median.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What should you do first when a CUDA program is slower than expected?

    Optimization effort should go where the time is. Transfers, launch gaps or a kernel you never suspected may dominate, and only a measurement shows which.
  2. 2. A kernel adds 1 FLOP per element and moves 12 bytes per element. What is its arithmetic intensity?

    AI is FLOPs divided by bytes: 1 / 12 is about 0.083. That is far below the ridge point of any modern GPU, so the kernel is memory-bound.
  3. 3. What does the ridge point of a roofline model represent?

    Ridge point = peak FLOP/s divided by peak bandwidth. Below it a kernel is memory-bound, above it compute-bound.
  4. 4. A bandwidth-bound kernel runs at 25% occupancy but already achieves 85% of peak memory bandwidth. What is the sensible conclusion?

    Occupancy is a means to hide latency. If the memory system is already almost saturated, the extra warps would add little, and forcing them can cost registers.
  5. 5. Making a part that takes 10% of the runtime infinitely fast speeds the whole program up by at most about...

    By Amdahl's law the best case is 1 / (1 - 0.10), which is about 1.11x.

Summary

  • Optimize in a loop: measure the whole program, find the limit, change one thing, verify and measure again.
  • Amdahl's law says the effort belongs where the time is; copies and launch gaps count as much as kernels.
  • Arithmetic intensity is FLOPs per byte moved; the roofline model caps performance at the smaller of the compute peak and AI times bandwidth.
  • Below the ridge point a kernel is memory-bound (move fewer bytes); above it, compute-bound (do cheaper arithmetic).
  • Occupancy exists to hide latency; Little's law and instruction-level parallelism show that a lot less than 100% is often enough.
  • Work down the checklist: transfers, enough threads, coalescing, reuse, conflicts and divergence, precision, libraries, then micro-tuning.

Next, tune the instruction level itself: Divergence, Registers and Instruction-Level Tuning.