Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 31 ยท Performance

Profiling with Nsight

Learn the nsys then ncu workflow, how to read timelines and key metrics, and how to hunt memory bugs and crashes with compute-sanitizer and cuda-gdb.

Guessing is the slowest way to optimize a GPU program. NVIDIA ships a set of tools that tell you exactly where time goes: Nsight Systems (command line nsys) shows the whole program as a timeline, Nsight Compute (ncu) dissects a single kernel in great detail, compute-sanitizer finds memory errors and races, and cuda-gdb is a debugger for device code. The standard workflow is to start wide with nsys, pick the kernel that matters, and then zoom in with ncu.

None of these tools can run on this site, because there is no GPU and no CUDA runtime. The commands and sample output in this lesson are read-only and the sample numbers are made up for teaching. What does run here are small Python stand-ins that do the same arithmetic the tools do for you: summing a kernel table, finding idle gaps on a timeline, and turning counters into bandwidth and a bound classification. Doing that arithmetic once by hand makes the real reports much easier to read.

flowchart LR
    A["Whole program is slow"] --> B["nsys: timeline<br/>where does time go?"]
    B --> C["Hot kernel found"]
    B --> D["Gaps, copies, launch overhead"]
    C --> E["ncu: one kernel<br/>why is it slow?"]
    D --> F["Fix streams, transfers, batching"]
    E --> G["Fix memory pattern or occupancy"]
    G --> H["Re-measure"]
    F --> H

Start with the timeline, then zoom into a kernel. Always re-measure after a change.

Compile so the tools can help

Profilers work on optimized binaries, so profile a normal -O3 build, not a debug build. One flag is worth adding for source-level views:

Terminal
nvcc -O3 -lineinfo -arch=sm_80 -o app app.cu

-lineinfo records which source line each instruction came from, without turning off optimizations. Do not use -G for profiling: that flag is for debugging and disables optimizations, so timings become meaningless.

Step 1: Nsight Systems (nsys)

Nsight Systems records what the CPU and GPU did over time: kernel launches, memory copies, API calls and your own annotations. Start with:

Terminal
nsys profile --stats=true -o report ./app

This runs the program, writes report.nsys-rep (open it in the Nsight Systems GUI on any machine) and prints summary tables. You can re-print the tables later:

Terminal
nsys stats report.nsys-rep

Useful summaries include cuda_gpu_kern_sum (time per kernel), cuda_gpu_mem_time_sum (time per copy kind) and cuda_api_sum (time inside CUDA API calls on the host). A sample kernel table looks like this (numbers invented for the lesson):

TEXT
 Time (%)  Total Time (ns)  Instances   Avg (ns)    Name
 --------  ---------------  ---------  ----------   -----------------------
     71.4       42,800,000        100     428,000   stencil_kernel
     20.1       12,050,000        100     120,500   reduce_kernel
      8.5        5,100,000          1   5,100,000   init_kernel

The first column already tells you where to focus: in this made-up run stencil_kernel is 71% of GPU kernel time. Optimizing init_kernel by 10x would gain under 8%.

Python Rank kernels from a summary table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last loop is Amdahl's law at work: a 2x speedup of a kernel that takes 71% of the time makes the GPU phase about 1.56x faster, while the same effort on the 8.5% kernel yields about 1.04x. Always optimize the biggest share first.

Reading the timeline

The GUI shows rows of colored bars against time: one row for the CPU threads, one for CUDA API calls, one row per stream of kernels, and rows for memory copies. Things to look for:

  • Gaps between kernels on the GPU row. The GPU is idle, usually because the CPU is slow to launch, or code calls cudaDeviceSynchronize too often.
  • Copies that do not overlap kernels. If a host-to-device copy runs alone while kernels wait, streams and pinned memory can help (Streams and Events).
  • Many tiny kernels. Hundreds of microsecond-long kernels with launch gaps suggest fusing them or using CUDA Graphs.
  • A long cudaMemcpy or cudaMalloc bar inside a loop: move allocation out of the loop.

The stand-in below does the gap arithmetic on a list of kernel start and end times, the same information the GPU row shows.

Python Find idle gaps on a GPU timeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Annotating with NVTX

By default the timeline only knows about kernels and API calls. NVTX ranges let you label phases of your own code so the timeline reads like your program:

C++
#include <nvtx3/nvToolsExt.h>

nvtxRangePushA("preprocess");
preprocess<<<grid, block>>>(d_in, d_tmp, n);
nvtxRangePop();

nvtxRangePushA("solve");
solve<<<grid, block>>>(d_tmp, d_out, n);
nvtxRangePop();

Run nsys profile --trace=cuda,nvtx ./app and the ranges appear as named bars above the kernels. Annotations cost almost nothing when no profiler is attached.

Step 2: Nsight Compute (ncu)

Once you know which kernel matters, ncu replays it several times and collects hardware counters. Because of that replay, a profiled run is much slower than a normal one, so restrict it to the kernel you care about:

Terminal
ncu --kernel-name stencil_kernel --launch-count 1 --set full -o stencil_report ./app

Common options:

  • --kernel-name (or -k) selects kernels by name, --launch-skip N skips the first N launches (useful to skip warm-up), --launch-count N stops after N.
  • --set basic collects a few sections quickly; --set full collects everything.
  • --section SpeedOfLight collects just one section; others include MemoryWorkloadAnalysis, Occupancy, WarpStateStats and LaunchStats.
  • -o name writes name.ncu-rep for the GUI; without it, results print to the terminal. Open a saved report with ncu -i name.ncu-rep.

An abridged, invented "Speed of Light" section reads like this:

TEXT
  stencil_kernel (1024, 1, 1)x(256, 1, 1), Context 1, Stream 7, Device 0, CC 8.0
    Section: GPU Speed Of Light Throughput
    ----------------------- ----------- ------------
    DRAM Frequency          cycle/usecond      1.21
    Duration                      usecond    428.00
    Memory Throughput                   %     88.40
    DRAM Throughput                     %     88.40
    Compute (SM) Throughput             %     21.70
    Achieved Occupancy                  %     63.10

Key metrics and how to read them

  • Duration is the kernel's elapsed GPU time.
  • Memory throughput / DRAM throughput (% of peak) says how close you are to the memory bandwidth limit. A high value (above roughly 70-80%) with low compute throughput means a memory-bound kernel that is already near its ceiling; to speed it up you must move less data.
  • Compute (SM) throughput (% of peak) says how busy the arithmetic pipelines are. High compute and low memory means compute-bound.
  • If both are low, the kernel is latency-bound: not enough parallelism, divergence, poor access patterns or stalls. Look at occupancy, "Warp State Statistics" (why warps are stalled) and the memory chart for uncoalesced access.
  • Achieved occupancy compares resident warps to the hardware maximum, over the time the SM was active. Compare it with the theoretical occupancy shown in the Occupancy section; the gap points to uneven blocks or tail effects.
  • Sectors per request in the memory section reveals coalescing: about 4 sectors per 32-thread float load is ideal; values near 32 mean scattered access (Global Memory and Coalescing).

The following stand-in turns raw numbers into those same percentages and a verdict. The peak figures are invented round numbers; the real tool gets the true peaks from the chip.

Python From counters to a bound classification

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that "saxpy slow" has the same traffic as a vector add but runs at roughly a third of the bandwidth. That is the kind of result that sends you to the memory-chart section looking for uncoalesced accesses or low occupancy. Be careful with the byte counts you feed in yourself: real hardware counters count actual transactions, which can exceed the useful bytes.

Finding bugs: compute-sanitizer

Profilers answer "how fast"; compute-sanitizer answers "is it correct". It runs your program under instrumentation and reports problems with source lines when you compiled with -lineinfo:

Terminal
compute-sanitizer ./app                     # memcheck: out-of-bounds, misaligned access
compute-sanitizer --tool racecheck ./app    # data races in shared memory
compute-sanitizer --tool initcheck ./app    # reads of uninitialized global memory
compute-sanitizer --tool synccheck ./app    # invalid synchronization (e.g. divergent __syncthreads)

A typical memcheck report (invented) looks like:

TEXT
========= Invalid __global__ write of size 4 bytes
=========     at vecAdd(float *, float *, float *, int)+0x90 in vec.cu:12
=========     by thread (255,0,0) in block (39,0,0)
=========     Address 0x7f2a40a00400 is out of bounds

It names the kernel, the line, the exact thread and block, and the kind of access: usually enough to see a missing bounds check at once. The tools slow your program down a lot, so run them on small inputs, and treat a clean report as strong evidence rather than proof.

Stepping through code: cuda-gdb

When a kernel misbehaves and the sanitizer is not enough, compile for debugging and use the CUDA debugger:

Terminal
nvcc -g -G -O0 -arch=sm_80 -o app_debug app.cu
cuda-gdb ./app_debug

Inside the debugger the usual GDB commands work, plus some for the GPU:

TEXT
(cuda-gdb) break vecAdd
(cuda-gdb) run
(cuda-gdb) info cuda kernels            # which kernels are running
(cuda-gdb) info cuda threads            # which threads are at which line
(cuda-gdb) cuda block 3 thread 5        # focus on one thread
(cuda-gdb) print i
(cuda-gdb) print a[i]

Remember that -G turns off device optimizations, so timing and sometimes behavior (especially of race conditions) differ from the release build. Use printf inside kernels for quick checks (output appears when the kernel finishes and the device synchronizes), and reach for the debugger when you need to inspect a single thread.

A repeatable profiling routine

  1. Check correctness first (sanitizer on a small input).
  2. Build with -O3 -lineinfo; run nsys profile --stats=true on a realistic input.
  3. Read the kernel table, then the timeline: pick the biggest share and the biggest gaps.
  4. Run ncu on that one kernel; classify it as memory-, compute- or latency-bound.
  5. Change one thing, re-measure the same way, keep the result only if the numbers improve.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which tool should you normally run first on a slow program?

    Nsight Systems gives the program-wide picture, so you know which kernel or gap is worth a detailed look in Nsight Compute.
  2. 2. A kernel shows DRAM throughput of 90% of peak and SM throughput of 20%. What is the best description?

    The memory system is almost saturated while arithmetic units are idle, so the way to speed up is to move less data.
  3. 3. Why should you not use a -G build for performance measurements?

    -G is for debugging; it turns off optimizations. Profile an -O3 build, optionally with -lineinfo.
  4. 4. Which tool reports an out-of-bounds write in a kernel along with the thread and block that did it?

    Memcheck instruments memory accesses and names the kernel, source line (with -lineinfo), thread and block.
  5. 5. A kernel takes 8% of total GPU time. If you make it infinitely fast, roughly how much faster is the GPU phase?

    By Amdahl's law the best case removes 8% of the time, giving about 1.09x. Optimize the big share first.

Summary

  • Profile an optimized build (-O3 -lineinfo), never a -G debug build.
  • nsys profile --stats=true gives a timeline plus summary tables; use it to find the dominant kernels, idle gaps, copies that do not overlap and swarms of tiny kernels. Add NVTX ranges to name your own phases.
  • ncu collects hardware counters for chosen kernels; compare memory and compute throughput percentages to classify a kernel as memory-, compute- or latency-bound.
  • Occupancy, warp-state stalls and sectors per request point to the cause once you know the class.
  • compute-sanitizer (memcheck, racecheck, initcheck, synccheck) finds correctness bugs; cuda-gdb steps through device code in a -G build.
  • Change one thing at a time and re-measure the same way.

Next: see which ready-made libraries already contain tuned kernels in The CUDA Library Ecosystem.