CUDA and GPU Programming Tutorial
Learn CUDA and GPU Programming step by step: 31 lessons in 10 chapters with examples, quizzes, diagrams and projects.
Start learning →
of lessons completed
Why GPUs
- What Is GPU Computing Learn what GPU computing is, how a GPU differs from a CPU, which workloads fit data parallelism, how CUDA started, and when a GPU is the wrong tool.
- Setting Up CUDA Install the NVIDIA driver and CUDA Toolkit, compile with nvcc, read nvidia-smi, understand compute capability, and try WSL, Linux, Windows or the cloud.
The Programming Model
- Your First Kernel Write your first CUDA kernel: learn __global__, the <<<blocks, threads>>> launch, host versus device code, asynchronous launches and an error-check macro.
- Grids, Blocks and Threads Understand threadIdx, blockIdx, blockDim and gridDim, compute global indices in 1D, 2D and 3D with dim3, and use the grid-stride loop for any data size.
- Vector Addition from Start to Finish Build the classic CUDA vector add program step by step: allocate device memory, copy data, launch the kernel, copy back, free, and choose a block size.
- Warps, SMs and the Hardware See how a GPU runs your blocks: streaming multiprocessors, warps of 32 threads, SIMT execution, occupancy limits, and why block sizes should be multiples of 32.
Memory
- The Memory Hierarchy Learn the CUDA memory hierarchy: registers, local, shared, global, constant and texture memory, the L1 and L2 caches, plus scope, lifetime, latency and bandwidth.
- Global Memory and Coalescing Understand how a warp's global loads become memory transactions, why coalesced access is fast, and how SoA, alignment and indexing change bandwidth.
- Shared Memory and Tiling Use CUDA shared memory, __syncthreads and tiling to cut global loads. Covers static and dynamic allocation and a tiled matrix multiply with runnable models.
- Bank Conflicts and Padding Learn how CUDA shared memory is split into 32 banks, which access patterns conflict, how padding removes conflicts and why a tiled transpose needs it.
- Moving Data: Pinned, Managed and Unified Memory Why PCIe copies dominate many CUDA programs, and how pinned memory, cudaMallocManaged, prefetching and zero-copy change the cost, with a runnable cost model.
Synchronization and Parallel Patterns
- Synchronization and Atomics Understand race conditions in CUDA, when __syncthreads is enough, how atomicAdd and atomicCAS work, what contention costs and what memory fences do.
- Parallel Reduction Learn parallel reduction in CUDA: tree reduction, interleaved vs sequential addressing, warp shuffles, an atomic final step, and why float sums change with order.
- Scan, Histogram and Other Patterns Learn the CUDA prefix sum (Hillis-Steele and Blelloch), stream compaction, privatized histograms and how scan powers GPU sorting, with runnable stand-ins.
- Warp Primitives and Cooperative Groups Learn CUDA warp primitives: __shfl_sync, ballot and vote functions, warp-level reduce and scan, then cooperative_groups tiles and grid synchronization.
Concurrency
- Streams and Events Learn CUDA streams and events: the default stream, cudaMemcpyAsync, overlapping copy and compute, cudaEvent timing, stream priorities and host callbacks.
- CUDA Graphs and Dynamic Parallelism Learn CUDA graphs and dynamic parallelism: stream capture, launch overhead savings, child kernels, when to use each, and the basics of multi-GPU work.
Performance
- The Optimization Workflow A practical CUDA optimization workflow: profile first, tell memory-bound from compute-bound with the roofline model, use arithmetic intensity, and avoid occupancy myths.
- Divergence, Registers and Instruction-Level Tuning Cut warp divergence, understand register pressure and spills, use __launch_bounds__ and fast math wisely, and get a first look at tensor cores.
- Profiling with Nsight Learn the nsys then ncu workflow, how to read timelines and key metrics, and how to hunt memory bugs and crashes with compute-sanitizer and cuda-gdb.
Libraries and Python
- The CUDA Library Ecosystem A tour of the CUDA libraries (cuBLAS, cuFFT, cuRAND, cuSPARSE, cuDNN, Thrust, CUB, CUTLASS) and how to decide between a library call and your own kernel.
- Python on the GPU: CuPy, Numba and PyTorch Use the GPU from Python with CuPy, Numba @cuda.jit kernels, PyTorch tensors and RAPIDS, see what each is best at, and learn how to choose between them.
Production
- Error Handling, Debugging and Pitfalls Check every CUDA call, understand sticky and asynchronous errors, decode illegal address and out-of-memory failures, and avoid the classic beginner bugs.
- Multi-GPU, Portability and the Wider Landscape Use several GPUs with cudaSetDevice, peer access and NCCL, understand compute capability and PTX compatibility, ship in containers, and compare HIP, SYCL and OpenCL.
Projects
- Project: Image Blur and Convolution Build a CUDA image blur step by step: a 2D grid, a constant-memory filter and halo tiles in shared memory, checked with a runnable thread-grid simulator.
- Project: Fast Matrix Multiplication Take CUDA matrix multiplication from a naive kernel to tiled and register-blocked versions, learn the GFLOPS arithmetic and see why cuBLAS wins.
- Project: N-Body Simulation Simulate gravity on the GPU: all-pairs forces, shared-memory tiling of bodies, leapfrog integration and an energy check, with runnable NumPy stand-ins.
- Project: Monte Carlo Pi and Option Pricing Use cuRAND with one random state per thread, reduce results across blocks and judge statistical accuracy in a Monte Carlo pi and option pricing project.
Practice
- Practice Exercises with Solutions Six CUDA exercises on index math, bounds checks, tiling, reduction, coalescing and stream timelines, each solved with runnable thread-grid simulator code.
- CUDA Interview Questions Prepare for CUDA and GPU programming interviews with concise answers on warps, divergence, memory, occupancy, streams and correctness, plus runnable models.
- CUDA Cheat Sheet A one-page CUDA cheat sheet: launch syntax, index formulas, memory API, qualifiers, an error-check macro, profiling commands and an optimization checklist.