Skip to content
Tutorials Center
My learning

Topics

Lesson 17 of 31 ยท Concurrency

CUDA Graphs and Dynamic Parallelism

Learn CUDA graphs and dynamic parallelism: stream capture, launch overhead savings, child kernels, when to use each, and the basics of multi-GPU work.

Every kernel launch costs a few microseconds of CPU and driver time, however tiny the kernel is. That cost is irrelevant for a kernel that runs for 10 milliseconds and painful for a pipeline of hundreds of kernels that each run for 5 microseconds. Deep-learning inference, iterative solvers and simulations with many small steps often live in that second world, where the GPU spends more time waiting to be told what to do than doing it.

CUDA has two tools for this. CUDA graphs record a whole sequence of work once and replay it with a single launch. Dynamic parallelism lets a kernel launch other kernels itself, so the decision about what to do next does not have to travel back to the CPU. This lesson explains both, when each helps, and ends with the first steps of using more than one GPU. It continues from Streams and Events.

The CUDA C++ here is read-only, because this site has no GPU. The runnable TryIts are Python stand-ins: a launch-overhead cost model with illustrative numbers (real launch costs depend on your CPU, driver and GPU, so measure on your own system) and a simulation of recursive refinement that counts the kernels a parent would launch.

The launch overhead problem

When you write kernel<<<grid, block, 0, stream>>>(args), the CPU validates the configuration, copies the arguments and pushes a command into the GPU's queue. That takes a few microseconds. If the GPU finishes a kernel faster than the CPU can submit the next one, the GPU idles between kernels and the CPU is the bottleneck. A sequence of 200 kernels of 4 microseconds each should take under a millisecond of GPU time, yet may take over a millisecond just to submit.

Python A launch-overhead cost model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The model shows the rule of thumb. When a kernel is shorter than the launch cost, the CPU sets the pace and a graph helps: about 1.25x at 3 microseconds per kernel and about 2.5x at 1 microsecond, because the graph replaces the per-kernel submission cost with a small per-node cost. When kernels are long (20 or 50 microseconds), launches hide behind execution, and the graph's per-node cost can even make it a hair slower in this model (0.96x and 0.98x). Real gains vary with the GPU, driver and kernel arguments, so treat these numbers as shape, not prediction. Do not reach for graphs until a profile shows gaps between short kernels (see Profiling with Nsight).

CUDA graphs

A graph is a set of nodes (kernel launches, memory copies, memsets, host functions, even child graphs) connected by dependency edges. You build it once, instantiate it into an executable form (the driver does the expensive validation and optimization here), and then launch it as many times as you want.

flowchart LR
    A["Define work<br/>(or capture it from a stream)"] --> B["cudaGraph_t<br/>the template"]
    B --> C["Instantiate once<br/>cudaGraphExec_t"]
    C --> D["Launch many times<br/>one CPU call each"]

The costly steps happen once; every replay is a single cheap launch.

Stream capture: the easy way

You do not have to describe nodes by hand. Put a stream into capture mode, run your normal asynchronous code, and CUDA records the work instead of executing it:

C++
cudaStream_t stream;
cudaStreamCreate(&stream);

cudaGraph_t graph;
cudaGraphExec_t exec;

cudaStreamBeginCapture(stream, cudaStreamCaptureModeGlobal);
    stepA<<<grid, block, 0, stream>>>(d_x, n);
    stepB<<<grid, block, 0, stream>>>(d_x, d_y, n);
    stepC<<<grid, block, 0, stream>>>(d_y, n);
cudaStreamEndCapture(stream, &graph);

cudaGraphInstantiate(&exec, graph, 0);       // CUDA 12 signature

for (int iter = 0; iter < 1000; ++iter) {
    cudaGraphLaunch(exec, stream);           // one launch runs A, B and C
}
cudaStreamSynchronize(stream);

cudaGraphExecDestroy(exec);
cudaGraphDestroy(graph);

Kernels recorded in the same stream become a chain of dependent nodes. If you use several streams joined by events during capture (cudaEventRecord plus cudaStreamWaitEvent), the graph contains parallel branches and the GPU can run independent nodes concurrently.

Things to know about capture:

  • Only asynchronous work can be captured. A blocking call such as cudaMalloc or a plain synchronous cudaMemcpy during capture is an error (or invalidates the capture). Allocate before you capture.
  • The arguments are baked in. A captured launch uses the same pointers and sizes at every replay. To change them, update the executable graph (cudaGraphExecUpdate, or per-node functions such as cudaGraphExecKernelNodeSetParams) instead of rebuilding it.
  • The graph holds the shape of the work, not the data. Copy new input into the same device buffers between replays and the graph processes the new data.
  • During capture, kernels do not run. Verify correctness with a normal run first.

When graphs help

Graphs pay off when the same sequence repeats many times and the individual pieces are short: time steps in a simulation, the layers of a neural network at inference time, or a pipeline run for every incoming frame. They help much less for a single large kernel, for work whose structure changes at every iteration, or when the host must look at results between steps.

Dynamic parallelism

With dynamic parallelism a kernel running on the GPU can launch another kernel with the usual <<<>>> syntax. The launching kernel is the parent, the new kernel is the child, and the child grid is a full grid with its own blocks and threads.

C++
__global__ void child(float* data, int offset, int count) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < count) data[offset + i] *= 2.0f;
}

__global__ void parent(float* data, const int* sizes, int numChunks) {
    int c = blockIdx.x * blockDim.x + threadIdx.x;
    if (c < numChunks) {
        int count = sizes[c];
        int offset = c * 1024;
        child<<<(count + 127) / 128, 128>>>(data, offset, count);   // launched from the GPU
    }
}

Compile code that uses it with relocatable device code and the device runtime library: nvcc -rdc=true app.cu -lcudadevrt. It requires compute capability 3.5 or newer.

The usual reason to use it is irregular, data-dependent work. Examples are a tree or graph traversal where each node decides how many children to expand, adaptive mesh refinement where only difficult regions deserve more resolution, or recursive algorithms such as quicksort. Without dynamic parallelism the CPU would have to read intermediate results, decide, and launch again, paying a round trip over PCIe for every decision.

Python Adaptive refinement: how many child kernels?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The simulation does not run on a GPU, but the structure is the same as a dynamic-parallelism kernel: each "parent" checks its interval and either stops or launches two children. The count shows where the technique wins: the work concentrates around the bump, and a uniform grid would waste effort on the smooth part.

Costs and rules

  • A device-side launch is cheaper than a CPU launch but not free, and each child grid needs enough work (many blocks) to be worth launching. A child of one block is usually a loss.
  • There is a limit on pending launches and on nesting depth. Launching millions of tiny children can run out of the pending-launch buffer and fail.
  • In CUDA 12 and later, the device runtime was redesigned: a parent can no longer wait inside the kernel for its children with cudaDeviceSynchronize(). Instead, children are launched into special streams. A fire-and-forget launch runs independently, and a tail launch runs after the parent grid finishes, which is the way to do "continue after the children are done". A parent grid does not complete until its children complete.
  • Child kernels cannot see the parent's registers or shared memory. Pass data through global memory, and be careful with local variables and shared arrays, which are not valid in the child.

Graphs or dynamic parallelism?

Question Graphs Dynamic parallelism
Is the sequence fixed and repeated? Excellent Not needed
Does the next step depend on data computed on the GPU? Cannot branch (conditional nodes exist in newer toolkits, but are limited) Natural fit
What does it save? CPU launch overhead CPU round trips and hand-off
Main cost Build and instantiate once Device-side launch cost and limits

They also combine: a graph can contain a kernel that uses dynamic parallelism. If your work is regular but made of many small kernels, start with graphs. If it is irregular and recursive, consider dynamic parallelism, or first check whether a flat reformulation using scan and compaction would do the job with less machinery.

Multi-GPU basics

A machine can hold several GPUs. Each one is a separate device with its own memory, its own streams and its own current context. The host thread selects the device that following calls apply to:

C++
int count = 0;
cudaGetDeviceCount(&count);

for (int dev = 0; dev < count; ++dev) {
    cudaSetDevice(dev);                         // all following calls target this GPU
    cudaMalloc(&d_part[dev], partBytes);
    cudaStreamCreate(&stream[dev]);
    cudaMemcpyAsync(d_part[dev], h_part[dev], partBytes,
                    cudaMemcpyHostToDevice, stream[dev]);
    work<<<grid, block, 0, stream[dev]>>>(d_part[dev], partElems);
}
for (int dev = 0; dev < count; ++dev) {
    cudaSetDevice(dev);
    cudaStreamSynchronize(stream[dev]);
}

Because launches are asynchronous, a single host thread can start work on all GPUs one after another and they will then run concurrently. Pointers belong to the device that allocated them; using a pointer on the wrong device fails unless peer access is enabled (covered in Multi-GPU, Portability and the Wider Landscape). The first design question is always how to split the data.

Python Splitting work across devices

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

An even split leaves the fast devices waiting for the slow one. Weighting by speed balances the finish times. Real systems also pay for transfers between GPUs and for combining partial results, which makes the best split less obvious than this model.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What problem do CUDA graphs mainly solve?

    A graph records a fixed sequence once and replays it with one launch, which removes most per-kernel submission cost. It does not change what happens inside a kernel.
  2. 2. During stream capture, which of these is a problem?

    Blocking allocation and synchronous calls are not valid while a stream is being captured. Allocate before capture, and use asynchronous operations inside it. Event joins between streams are the supported way to build parallel branches.
  3. 3. When is dynamic parallelism a good fit?

    Irregular, data-dependent work such as adaptive refinement or tree traversal benefits from launching children without a trip back to the CPU. Fixed repeated pipelines are better served by graphs.
  4. 4. With several GPUs, what must you do before allocating memory for device 1?

    cudaSetDevice selects the device that following allocation, stream and launch calls apply to for the current host thread.

Summary

  • Each kernel launch costs a few microseconds of CPU time; this dominates when kernels are very short.
  • A CUDA graph captures or defines a sequence of work, is instantiated once, and is launched with a single call; arguments are fixed unless you update the executable graph.
  • Stream capture is the easy way to build a graph, but only asynchronous operations may be captured.
  • Dynamic parallelism lets a kernel launch child grids for irregular, data-dependent work; compile with -rdc=true -lcudadevrt, and give each child plenty of work.
  • In recent toolkits, use fire-and-forget and tail launches instead of waiting for children inside a kernel.
  • Multi-GPU code selects devices with cudaSetDevice, uses one stream per device, and needs a sensible way to split the data.

Next, learn how to find where the time actually goes and what to fix first: The Optimization Workflow.