Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 31 ยท Practice

CUDA Cheat Sheet

A one-page CUDA cheat sheet: launch syntax, index formulas, memory API, qualifiers, an error-check macro, profiling commands and an optimization checklist.

This page is a compact reference for CUDA C++: the syntax and formulas you look up again and again, gathered in one place. It is not a tutorial. Each section links to the lesson that explains the topic properly, so use it as a map and a memory aid once you have worked through the track.

All CUDA code and shell commands on this page are read-only, because the site has no GPU, no nvcc and no driver. What does run here are two small Python helpers that do the arithmetic you repeat in every project: launch sizing and index math, and a quick roofline check. They are stand-ins for your own calculator, not CUDA itself.

Numbers such as "1024 threads per block" are limits of current NVIDIA GPUs. Details vary by compute capability, so confirm them for your card with cudaGetDeviceProperties or the CUDA programming guide before you depend on them.

Launching a kernel

Your First Kernel, Grids, Blocks and Threads

C++
__global__ void scale(float *x, float a, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) x[i] *= a;
}

int threads = 256;                           // a multiple of 32
int blocks  = (n + threads - 1) / threads;   // ceiling division
scale<<<blocks, threads>>>(d_x, 2.0f, n);    // asynchronous
cudaDeviceSynchronize();                     // wait (or use a stream or an event)

The full launch form has four parts:

C++
kernel<<<gridDim, blockDim, dynamicSharedBytes, stream>>>(args...);
Part Type Meaning
gridDim int or dim3 number of blocks
blockDim int or dim3 threads per block
dynamicSharedBytes size_t bytes for extern __shared__ (default 0)
stream cudaStream_t which queue to run in (default 0)

A launch returns immediately. Errors in the launch arguments show up through cudaGetLastError(); errors during execution show up later, at the next synchronizing call.

Index formulas

The built-in variables threadIdx, blockIdx, blockDim and gridDim are all three-component values (.x, .y, .z).

Goal Formula
Global index, 1D blockIdx.x * blockDim.x + threadIdx.x
Blocks needed for n items (n + threads - 1) / threads
Total threads in a 1D grid gridDim.x * blockDim.x
Column and row, 2D x = blockIdx.x*blockDim.x + threadIdx.x, y = blockIdx.y*blockDim.y + threadIdx.y
Row-major offset y * width + x
Thread number inside a block threadIdx.x + threadIdx.y*blockDim.x + threadIdx.z*blockDim.x*blockDim.y
Warp number, lane in warp tid / 32, tid % 32

The grid-stride loop lets a fixed-size grid handle any n:

C++
for (int i = blockIdx.x * blockDim.x + threadIdx.x;
     i < n;
     i += gridDim.x * blockDim.x) {
    out[i] = f(in[i]);
}

For 2D data, use dim3 block(16, 16) (256 threads) and dim3 grid((w + 15) / 16, (h + 15) / 16).

Typical hardware limits (check your device): at most 1024 threads per block; block dimensions up to 1024 x 1024 x 64 (but the product must stay at or below 1024); grid x up to 2^31 - 1, grid y and z up to 65,535; warp size 32.

Try it: launch sizing and index math

This helper does the ceiling division, shows how many threads sit idle, and prints the (block, thread) coordinates for a few global indices, in 1D and 2D.

Python Launch sizing and index calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Memory API

Vector Addition from Start to Finish, Moving Data

C++
float *d_a;
cudaMalloc(&d_a, n * sizeof(float));                          // device memory
cudaMemcpy(d_a, h_a, bytes, cudaMemcpyHostToDevice);          // blocking copy
cudaMemcpy(h_out, d_a, bytes, cudaMemcpyDeviceToHost);
cudaMemset(d_a, 0, bytes);                                    // fill bytes
cudaFree(d_a);

cudaMallocHost(&h_pinned, bytes);                             // pinned (page-locked) host memory
cudaMemcpyAsync(d_a, h_pinned, bytes, cudaMemcpyHostToDevice, stream);
cudaFreeHost(h_pinned);

cudaMallocManaged(&u, bytes);                                 // unified memory
cudaMemPrefetchAsync(u, bytes, device, stream);               // hint: move it there now
cudaFree(u);
Call Use
cudaMalloc / cudaFree device (global) memory
cudaMemcpy blocking copy; direction is HostToDevice, DeviceToHost, DeviceToDevice or Default
cudaMemcpyAsync non-blocking; needs pinned host memory to overlap
cudaMallocHost pinned host memory, faster transfers
cudaMallocManaged one pointer valid on host and device
cudaMemcpyToSymbol fill a __constant__ variable
cudaGetDeviceProperties read limits and capabilities

Rule of thumb: copy once in, compute a lot, copy once out. A transfer over PCIe is far slower than a trip to device memory.

Function and variable qualifiers

The Memory Hierarchy, Shared Memory and Tiling

Qualifier Meaning
__global__ kernel: runs on the device, called from the host with <<<>>>, returns void
__device__ (function) runs on the device, called from device code
__host__ runs on the host (the default); combine with __device__ to compile both
__device__ (variable) variable in global memory, lives as long as the program
__constant__ read-only, cached, 64 KB total, set from the host
__shared__ one copy per block, on-chip, lives as long as the block
extern __shared__ size chosen at launch (third <<<>>> argument)
__restrict__ promise that pointers do not alias
__launch_bounds__(maxThreads, minBlocks) tell the compiler how big blocks will be so it budgets registers
Memory Scope Lifetime Speed
Registers one thread thread fastest
Local (spills, big arrays) one thread thread slow, lives in global memory
Shared block block fast, 32 banks
Global all threads and host until freed large, slow, needs coalescing
Constant all threads program fast when all threads read the same address

Synchronization and warp tools

Synchronization and Atomics, Warp Primitives

C++
__syncthreads();                          // barrier for the block; every thread must reach it
atomicAdd(&sum, v);                       // also atomicMax, atomicMin, atomicCAS, atomicExch
__shfl_down_sync(0xffffffff, v, offset);  // read a value from a lane `offset` higher
__ballot_sync(0xffffffff, predicate);     // 32-bit mask of lanes where predicate is true
__syncwarp();                             // warp-level barrier
cudaDeviceSynchronize();                  // host waits for all device work

Warp-sum pattern:

C++
for (int off = 16; off > 0; off >>= 1)
    v += __shfl_down_sync(0xffffffff, v, off);   // lane 0 now holds the warp sum

Never put __syncthreads() inside a branch that only some threads of the block take.

Error-checking macro

Error Handling, Debugging and Pitfalls

C++
#include <cstdio>
#include <cstdlib>

#define CUDA_CHECK(call)                                                   \
    do {                                                                   \
        cudaError_t err_ = (call);                                         \
        if (err_ != cudaSuccess) {                                         \
            std::fprintf(stderr, "CUDA error %s at %s:%d: %s\n",           \
                         cudaGetErrorName(err_), __FILE__, __LINE__,       \
                         cudaGetErrorString(err_));                        \
            std::exit(EXIT_FAILURE);                                       \
        }                                                                  \
    } while (0)

CUDA_CHECK(cudaMalloc(&d_a, bytes));
kernel<<<blocks, threads>>>(d_a, n);
CUDA_CHECK(cudaGetLastError());          // launch errors
CUDA_CHECK(cudaDeviceSynchronize());     // errors raised while the kernel ran

Kernels return nothing, so you check the launch with cudaGetLastError() and the execution with a synchronizing call. Remove the extra cudaDeviceSynchronize() from release builds if it hurts overlap.

Streams and events

Streams and Events

C++
cudaStream_t s;  cudaStreamCreate(&s);
cudaMemcpyAsync(d_in, h_in, bytes, cudaMemcpyHostToDevice, s);
kernel<<<blocks, threads, 0, s>>>(d_in, d_out, n);
cudaMemcpyAsync(h_out, d_out, bytes, cudaMemcpyDeviceToHost, s);
cudaStreamSynchronize(s);
cudaStreamDestroy(s);

cudaEvent_t t0, t1;  cudaEventCreate(&t0);  cudaEventCreate(&t1);
cudaEventRecord(t0, s);  /* work */  cudaEventRecord(t1, s);
cudaEventSynchronize(t1);
float ms;  cudaEventElapsedTime(&ms, t0, t1);

Work in one stream runs in order; work in different streams may overlap.

Build and profile commands

Setting Up CUDA, Profiling with Nsight

Terminal
nvidia-smi                                   # driver, GPUs, memory use
nvcc --version                               # toolkit version

nvcc -O3 -arch=sm_80 app.cu -o app           # build for compute capability 8.0
nvcc -O3 -lineinfo app.cu -o app             # keep source lines for the profiler
nvcc -G -g app.cu -o app_debug               # device debug build (slow, for cuda-gdb)
nvcc -Xptxas -v app.cu -o app                # print registers and shared memory per kernel

nsys profile -o timeline ./app               # timeline: copies, kernels, gaps
nsys stats timeline.nsys-rep                 # summary tables
ncu --set full -o kernel_report ./app        # per-kernel hardware counters
ncu --kernel-name scale --launch-count 1 ./app
compute-sanitizer ./app                      # out-of-bounds, races, leaks
compute-sanitizer --tool racecheck ./app
cuda-gdb ./app_debug

Use nsys first to see where time goes, then ncu on the one kernel that matters.

Optimization checklist

The Optimization Workflow, Divergence and Instruction Tuning

flowchart TD
    A["Correct and tested?"] --> B["Profile with nsys"]
    B --> C{"Where is the time?"}
    C -->|"copies or gaps"| D["Fewer transfers, pinned memory,<br/>streams, graphs"]
    C -->|"one kernel"| E["Profile with ncu"]
    E --> F{"Memory-bound or<br/>compute-bound?"}
    F -->|"memory"| G["Coalesce, tile in shared memory,<br/>fix bank conflicts"]
    F -->|"compute"| H["Cut divergence, use fast math<br/>or tensor cores"]
    D --> B
    G --> B
    H --> B

Measure, change one thing, measure again; the loop ends when the kernel is near its hardware limit.

  1. Correctness first: check every API call, run compute-sanitizer, compare with a CPU result.
  2. Transfers: keep data on the device, use pinned memory, overlap with streams.
  3. Launch shape: block size a multiple of 32 (128 to 256 is a good start), enough blocks to fill every SM.
  4. Global memory: neighbouring threads read neighbouring addresses; prefer structure of arrays over array of structures.
  5. Reuse: stage reused data in shared memory; pad arrays to avoid bank conflicts.
  6. Divergence: keep threads of one warp on the same branch.
  7. Registers and occupancy: look at -Xptxas -v; use __launch_bounds__ if spills appear.
  8. Libraries: use cuBLAS, cuFFT, Thrust or CUB when they already do the job.
  9. Many tiny launches: use CUDA Graphs.

Try it: a quick roofline check

Whether a kernel is memory-bound or compute-bound depends on its arithmetic intensity (floating-point operations per byte moved) compared with the machine's balance point. This helper computes both. The device numbers are made up for illustration; use your own card's specification.

Python Roofline check for common kernels

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Quick troubleshooting

Symptom Likely cause
invalid configuration argument more than 1024 threads per block, or a zero-sized grid
illegal memory access missing bounds check, or a host pointer used on the device
Kernel seems not to run no error check after launch; read cudaGetLastError()
Result differs slightly from the CPU floating-point addition order changed (atomics, reductions)
Program hangs __syncthreads() inside a branch, or an unmatched warp mask
Speed far below expectation uncoalesced access, bank conflicts, tiny kernels, or transfer time

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You need to process n = 1,000,000 elements with 256 threads per block. How many blocks does the launch need?

    Use ceiling division: (1,000,000 + 255) / 256 gives 3,907 in integer arithmetic. 3,906 blocks would leave the last 64 elements unprocessed.
  2. 2. Which line reports an error that happened while a kernel was running?

    Launches are asynchronous, so execution errors surface at the next call that waits for the device. cudaGetLastError() only catches launch problems.
  3. 3. What does the third argument inside <<<a, b, c, d>>> set?

    The arguments are grid size, block size, dynamic shared memory bytes and the stream, in that order.
  4. 4. A kernel has arithmetic intensity 0.2 FLOP/byte on a GPU whose ridge point is 20 FLOP/byte. What should you work on?

    An intensity far below the ridge point means the kernel waits on memory, so improving memory behavior pays off and arithmetic tricks do not.

Summary

  • Launch with kernel<<<grid, block, sharedBytes, stream>>>(args); compute blocks with ceiling division and guard with if (i < n).
  • Global index in 1D is blockIdx.x * blockDim.x + threadIdx.x; use a grid-stride loop when the grid is smaller than the data.
  • Allocate with cudaMalloc, copy with cudaMemcpy (or async with pinned memory), always free.
  • __global__ runs on the device from the host; __shared__ is per block; __syncthreads() must be reached by every thread of the block.
  • Wrap every API call in an error macro and check cudaGetLastError() after launches.
  • Profile with nsys first, then ncu; fix the biggest bottleneck, measure, repeat.

That completes the track. Revisit the track introduction or review the optimization workflow whenever a kernel is slower than you expect.