Skip to content
Tutorials Center
My learning

Topics

CUDA and GPU Programming Tutorial

Learn CUDA and GPU Programming step by step: 31 lessons in 10 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Why GPUs

  1. 1 What Is GPU Computing Learn what GPU computing is, how a GPU differs from a CPU, which workloads fit data parallelism, how CUDA started, and when a GPU is the wrong tool.
  2. 2 Setting Up CUDA Install the NVIDIA driver and CUDA Toolkit, compile with nvcc, read nvidia-smi, understand compute capability, and try WSL, Linux, Windows or the cloud.

The Programming Model

  1. 3 Your First Kernel Write your first CUDA kernel: learn __global__, the <<<blocks, threads>>> launch, host versus device code, asynchronous launches and an error-check macro.
  2. 4 Grids, Blocks and Threads Understand threadIdx, blockIdx, blockDim and gridDim, compute global indices in 1D, 2D and 3D with dim3, and use the grid-stride loop for any data size.
  3. 5 Vector Addition from Start to Finish Build the classic CUDA vector add program step by step: allocate device memory, copy data, launch the kernel, copy back, free, and choose a block size.
  4. 6 Warps, SMs and the Hardware See how a GPU runs your blocks: streaming multiprocessors, warps of 32 threads, SIMT execution, occupancy limits, and why block sizes should be multiples of 32.

Memory

  1. 7 The Memory Hierarchy Learn the CUDA memory hierarchy: registers, local, shared, global, constant and texture memory, the L1 and L2 caches, plus scope, lifetime, latency and bandwidth.
  2. 8 Global Memory and Coalescing Understand how a warp's global loads become memory transactions, why coalesced access is fast, and how SoA, alignment and indexing change bandwidth.
  3. 9 Shared Memory and Tiling Use CUDA shared memory, __syncthreads and tiling to cut global loads. Covers static and dynamic allocation and a tiled matrix multiply with runnable models.
  4. 10 Bank Conflicts and Padding Learn how CUDA shared memory is split into 32 banks, which access patterns conflict, how padding removes conflicts and why a tiled transpose needs it.
  5. 11 Moving Data: Pinned, Managed and Unified Memory Why PCIe copies dominate many CUDA programs, and how pinned memory, cudaMallocManaged, prefetching and zero-copy change the cost, with a runnable cost model.

Synchronization and Parallel Patterns

  1. 12 Synchronization and Atomics Understand race conditions in CUDA, when __syncthreads is enough, how atomicAdd and atomicCAS work, what contention costs and what memory fences do.
  2. 13 Parallel Reduction Learn parallel reduction in CUDA: tree reduction, interleaved vs sequential addressing, warp shuffles, an atomic final step, and why float sums change with order.
  3. 14 Scan, Histogram and Other Patterns Learn the CUDA prefix sum (Hillis-Steele and Blelloch), stream compaction, privatized histograms and how scan powers GPU sorting, with runnable stand-ins.
  4. 15 Warp Primitives and Cooperative Groups Learn CUDA warp primitives: __shfl_sync, ballot and vote functions, warp-level reduce and scan, then cooperative_groups tiles and grid synchronization.

Concurrency

  1. 16 Streams and Events Learn CUDA streams and events: the default stream, cudaMemcpyAsync, overlapping copy and compute, cudaEvent timing, stream priorities and host callbacks.
  2. 17 CUDA Graphs and Dynamic Parallelism Learn CUDA graphs and dynamic parallelism: stream capture, launch overhead savings, child kernels, when to use each, and the basics of multi-GPU work.

Performance

  1. 18 The Optimization Workflow A practical CUDA optimization workflow: profile first, tell memory-bound from compute-bound with the roofline model, use arithmetic intensity, and avoid occupancy myths.
  2. 19 Divergence, Registers and Instruction-Level Tuning Cut warp divergence, understand register pressure and spills, use __launch_bounds__ and fast math wisely, and get a first look at tensor cores.
  3. 20 Profiling with Nsight Learn the nsys then ncu workflow, how to read timelines and key metrics, and how to hunt memory bugs and crashes with compute-sanitizer and cuda-gdb.

Libraries and Python

  1. 21 The CUDA Library Ecosystem A tour of the CUDA libraries (cuBLAS, cuFFT, cuRAND, cuSPARSE, cuDNN, Thrust, CUB, CUTLASS) and how to decide between a library call and your own kernel.
  2. 22 Python on the GPU: CuPy, Numba and PyTorch Use the GPU from Python with CuPy, Numba @cuda.jit kernels, PyTorch tensors and RAPIDS, see what each is best at, and learn how to choose between them.

Production

  1. 23 Error Handling, Debugging and Pitfalls Check every CUDA call, understand sticky and asynchronous errors, decode illegal address and out-of-memory failures, and avoid the classic beginner bugs.
  2. 24 Multi-GPU, Portability and the Wider Landscape Use several GPUs with cudaSetDevice, peer access and NCCL, understand compute capability and PTX compatibility, ship in containers, and compare HIP, SYCL and OpenCL.

Projects

  1. 25 Project: Image Blur and Convolution Build a CUDA image blur step by step: a 2D grid, a constant-memory filter and halo tiles in shared memory, checked with a runnable thread-grid simulator.
  2. 26 Project: Fast Matrix Multiplication Take CUDA matrix multiplication from a naive kernel to tiled and register-blocked versions, learn the GFLOPS arithmetic and see why cuBLAS wins.
  3. 27 Project: N-Body Simulation Simulate gravity on the GPU: all-pairs forces, shared-memory tiling of bodies, leapfrog integration and an energy check, with runnable NumPy stand-ins.
  4. 28 Project: Monte Carlo Pi and Option Pricing Use cuRAND with one random state per thread, reduce results across blocks and judge statistical accuracy in a Monte Carlo pi and option pricing project.

Practice

  1. 29 Practice Exercises with Solutions Six CUDA exercises on index math, bounds checks, tiling, reduction, coalescing and stream timelines, each solved with runnable thread-grid simulator code.
  2. 30 CUDA Interview Questions Prepare for CUDA and GPU programming interviews with concise answers on warps, divergence, memory, occupancy, streams and correctness, plus runnable models.
  3. 31 CUDA Cheat Sheet A one-page CUDA cheat sheet: launch syntax, index formulas, memory API, qualifiers, an error-check macro, profiling commands and an optimization checklist.