Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 31 ยท Production

Multi-GPU, Portability and the Wider Landscape

Use several GPUs with cudaSetDevice, peer access and NCCL, understand compute capability and PTX compatibility, ship in containers, and compare HIP, SYCL and OpenCL.

One GPU is where every CUDA program begins, but real systems quickly outgrow it. A server may hold eight GPUs joined by fast links, a training job may span many servers, and a research code may need to run on hardware from more than one vendor. This lesson covers the practical edge of CUDA: driving several devices from one program, how binaries stay compatible across GPU generations, how to deploy in containers, and what the alternatives to CUDA look like.

The real hardware, NCCL and the other toolchains cannot run on this site, so their commands and code are read-only. Two Python stand-ins make the central ideas concrete: a model of how data-parallel work is split across devices and combined by a ring all-reduce, and a checker for the compute capability and PTX rules that decide whether a binary will start on a given GPU.

Seeing and selecting devices

A machine can expose several GPUs. The runtime numbers them from 0:

C++
int count = 0;
CUDA_CHECK(cudaGetDeviceCount(&count));
for (int d = 0; d < count; ++d) {
    cudaDeviceProp p;
    CUDA_CHECK(cudaGetDeviceProperties(&p, d));
    printf("GPU %d: %s, compute capability %d.%d, %.1f GiB\n",
           d, p.name, p.major, p.minor, p.totalGlobalMem / 1073741824.0);
}

cudaSetDevice(d) makes device d the current device for the calling host thread. Everything you do afterwards from that thread (allocations, streams, events, kernel launches) targets that device. Two consequences:

  • Memory is per device. A pointer from cudaMalloc on device 0 is not valid in a kernel running on device 1, unless peer access is enabled (below).
  • Streams and events belong to the device that was current when you created them. Launching into a stream of another device is an error.

At the process level, the environment variable CUDA_VISIBLE_DEVICES=2,3 restricts which GPUs the program sees and renumbers them 0 and 1, which is how schedulers and containers hand out GPUs.

The simplest multi-GPU pattern: split the data

When the work is data parallel, give each GPU a slice. Because launches and async copies return immediately, one host thread can start work on every device in turn and then wait for them all:

C++
const int chunk = n / ngpus;
for (int d = 0; d < ngpus; ++d) {
    CUDA_CHECK(cudaSetDevice(d));
    CUDA_CHECK(cudaMalloc(&d_in[d], chunk * sizeof(float)));
    CUDA_CHECK(cudaMemcpyAsync(d_in[d], h_in + d * chunk, chunk * sizeof(float),
                               cudaMemcpyHostToDevice, streams[d]));
    kernel<<<blocks, threads, 0, streams[d]>>>(d_in[d], chunk);
}
for (int d = 0; d < ngpus; ++d) {
    CUDA_CHECK(cudaSetDevice(d));
    CUDA_CHECK(cudaStreamSynchronize(streams[d]));
}

The host buffer should be pinned memory so the copies are truly asynchronous (Moving Data). Many programs instead use one host thread or process per GPU, which scales more easily and is how MPI-based codes and deep-learning frameworks work.

flowchart LR
    H["Host: input array"] --> S{"Split into slices"}
    S --> G0["GPU 0<br/>slice 0"]
    S --> G1["GPU 1<br/>slice 1"]
    S --> G2["GPU 2<br/>slice 2"]
    S --> G3["GPU 3<br/>slice 3"]
    G0 --> C["Combine:<br/>gather or all-reduce"]
    G1 --> C
    G2 --> C
    G3 --> C

Data-parallel multi-GPU work: split, compute independently, then combine. The combine step is where interconnect speed matters.

Peer access and interconnects

Before using GPUs in the same machine together, check whether they can address each other's memory:

C++
int canAccess = 0;
CUDA_CHECK(cudaDeviceCanAccessPeer(&canAccess, 0, 1));   // can device 0 access device 1?
if (canAccess) {
    CUDA_CHECK(cudaSetDevice(0));
    CUDA_CHECK(cudaDeviceEnablePeerAccess(1, 0));         // flags must be 0
}
// Direct copy between devices (goes over NVLink or PCIe, no host round trip when peer access is on)
CUDA_CHECK(cudaMemcpyPeer(d_on1, 1, d_on0, 0, bytes));

With peer access enabled, a kernel on device 0 can even dereference a pointer that lives on device 1. Direct copies avoid staging through host memory. How fast they are depends on the link: PCIe is the common path, while NVLink (and NVSwitch, which connects many GPUs all-to-all) offers several times the bandwidth. nvidia-smi topo -m prints the connection matrix of your machine, and it is worth reading before you design communication.

NCCL: collective communication

Copying buffers by hand does not scale past a few GPUs. NCCL (the NVIDIA Collective Communications Library, pronounced "nickel") implements collective operations (all-reduce, broadcast, reduce, all-gather, reduce-scatter) tuned to the machine's topology, within one server and across servers over InfiniBand or Ethernet. Deep-learning frameworks use it to average gradients: after each GPU computes gradients on its own batch, an all-reduce gives every GPU the sum.

C++
#include <nccl.h>

ncclComm_t comms[ngpus];
int devs[ngpus] = {0, 1, 2, 3};
ncclCommInitAll(comms, ngpus, devs);                      // one communicator per GPU

ncclGroupStart();                                          // issue all calls together
for (int d = 0; d < ngpus; ++d) {
    cudaSetDevice(devs[d]);
    ncclAllReduce(d_grad[d], d_grad[d], count, ncclFloat, ncclSum, comms[d], streams[d]);
}
ncclGroupEnd();
// afterwards, each d_grad[d] holds the sum over all GPUs

NCCL calls are enqueued on a CUDA stream like kernels, so communication can overlap with computation. For several processes or nodes, the processes first exchange a unique ID (typically with MPI) and then each calls ncclCommInitRank.

How a ring all-reduce works

A naive all-reduce sends everything to one GPU, adds it, and sends the answer back, creating a hot spot. A ring all-reduce splits each GPU's buffer into P chunks (P is the number of GPUs) and arranges the GPUs in a ring. In P-1 steps of reduce-scatter, each GPU passes one chunk to its neighbor, which adds it to its own copy; after that each GPU owns one fully summed chunk. Then P-1 steps of all-gather circulate the finished chunks. Every link carries the same amount of data in every step, so bandwidth use is balanced, and the total sent per GPU is about 2 x (P-1)/P times the buffer size, nearly independent of P.

The stand-in below simulates exactly those steps on lists of numbers, counting the elements each GPU sends.

Python A ring all-reduce simulation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two things stand out. The result is correct on every "GPU", and the traffic per GPU stays below twice the buffer size however many GPUs join (1.0x for 2 GPUs, 1.97x for 64), which is why ring (and tree) algorithms scale. NCCL chooses between rings, trees and other algorithms automatically depending on message size and topology.

Portability across GPU generations

A GPU's compute capability (written major.minor, for example 7.5, 8.0, 8.6, 9.0) identifies its architecture generation and the instruction set and features it supports. Your program is compiled into two kinds of code, and the rules for running them are specific:

  • SASS / cubin: machine code for a specific architecture, named sm_XY. A cubin for sm_80 runs on any device with the same major version and a minor version at least 0, so it also runs on 8.6 hardware. It does not run on 7.5 or on 9.0.
  • PTX: a portable intermediate assembly, named compute_XY. If the binary contains PTX for compute_80, the driver can JIT-compile it at load time for any device whose compute capability is at least 8.0, including future GPUs. The first run is slower (the driver caches the result).

nvcc lets you choose both with -gencode:

Terminal
# Native code for two architectures, plus PTX for forward compatibility
nvcc -O3 app.cu -o app \
  -gencode arch=compute_80,code=sm_80 \
  -gencode arch=compute_90,code=sm_90 \
  -gencode arch=compute_90,code=compute_90

# Convenient shortcuts
nvcc -arch=sm_80 app.cu        # code for sm_80 plus PTX compute_80
nvcc -arch=native app.cu       # build for the GPUs in this machine

If the binary contains neither a compatible cubin nor usable PTX, the first CUDA call or kernel launch fails with cudaErrorNoKernelImageForDevice ("no kernel image is available for execution on the device"), as seen in Error Handling, Debugging and Pitfalls. The stand-in below applies these rules.

Python Will this binary run on that GPU?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Read the output with these lessons in mind. The plain sm_80 build runs natively on 8.0 and 8.6, JIT-compiles on newer devices, and fails on the older 7.5. A binary with only SASS and no PTX is fast on its target family but has no future. That is why distributed applications typically ship a few native architectures plus PTX for the newest one. The capability numbers in the table are only examples; consult the CUDA release notes, because support for old architectures is removed from newer toolkits.

Separately, there are two version numbers to keep apart: the driver version (installed on the machine, shown by nvidia-smi) and the toolkit version (the compiler and libraries you built with, shown by nvcc --version). The driver must be new enough for the toolkit you used; a newer driver runs older toolkits, and CUDA offers minor-version compatibility within one major release.

Deploying in containers

Containers are the usual way to ship GPU software, because they pin the toolkit and library versions. The key design: the kernel-mode driver lives on the host; the container carries the toolkit and libraries but not the driver. The NVIDIA Container Toolkit injects the driver's user-space libraries and the device files when you ask for GPUs.

Terminal
# Check the GPUs are visible inside a container
docker run --rm --gpus all nvidia/cuda:<version>-base-ubuntu22.04 nvidia-smi

# Give a container only GPUs 0 and 1
docker run --rm --gpus '"device=0,1"' my-cuda-app:latest

Practical advice:

  • Build with a -devel image (compiler included) and run in a smaller -runtime image (multi-stage Docker build).
  • Keep the container's toolkit version no newer than the host driver supports.
  • Build for the architectures of the GPUs you deploy to, and include PTX for the newest.
  • In Kubernetes, request GPUs as a resource (nvidia.com/gpu) through the NVIDIA device plugin, and let the scheduler assign devices.

The wider landscape

CUDA is NVIDIA-only. Several alternatives target other hardware, and the concepts you learned (grids, blocks, threads, shared memory, coalescing) carry over to all of them under different names.

Technology What it is Notes
HIP / ROCm AMD's CUDA-like C++ API and stack Syntax is nearly identical (hipMalloc, hipMemcpy, <<<>>> launches). The HIPIFY tools translate CUDA source mostly mechanically. HIP can also compile for NVIDIA GPUs, wrapping CUDA. Wavefronts are often 64 lanes wide on AMD data-center GPUs, which affects warp-level code.
SYCL Khronos standard for single-source C++ across vendors Queues, buffers or USM pointers, and parallel_for lambdas. Implementations include Intel oneAPI DPC++ and AdaptiveCpp, which can target NVIDIA, AMD and Intel GPUs. A migration tool (SYCLomatic) converts CUDA code.
OpenCL Older cross-vendor standard C-based kernels compiled at run time from strings, verbose host API, wide device support including CPUs and FPGAs; less active than the above.
Kokkos, RAJA C++ performance-portability libraries Write loops once, choose CUDA, HIP, SYCL or OpenMP as a back end.
OpenMP offload, OpenACC Directive-based offloading Add pragmas to loops; good for incremental porting of existing C/C++/Fortran.
Vulkan compute, Metal, DirectX compute Graphics-API compute Used in games and on Apple or Windows devices; different programming style.

How to choose: if you target NVIDIA only, CUDA gives the best tooling and library coverage. If you need AMD as well, HIP is the smallest step from CUDA code. If you want one codebase for many vendors from the start, SYCL, Kokkos or a framework (PyTorch, JAX) hides the differences. Whatever you pick, keep the algorithm and the memory-access thinking separate from the API, because that part transfers.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does cudaSetDevice(2) affect?

    The current device is a per-host-thread setting, and objects such as streams and allocations belong to the device that was current when they were created.
  2. 2. A binary was built with -arch=sm_80 (cubin and PTX for 8.0). On which device does it run?

    Cubin runs on the same major version with an equal or higher minor, PTX is JIT-compiled for newer capabilities, and nothing covers older ones.
  3. 3. What does an NCCL all-reduce with ncclSum give each GPU?

    All-reduce combines all inputs with the chosen operation and leaves the result on every device, which is how gradients are averaged in data-parallel training.
  4. 4. In a GPU container, where does the kernel-mode driver live?

    Images hold the toolkit and libraries, while the driver comes from the host through the NVIDIA Container Toolkit.
  5. 5. Which option is the smallest step for porting CUDA code to AMD GPUs?

    HIP uses nearly the same names and launch syntax, and HIPIFY tools translate most CUDA source automatically.

Summary

  • cudaGetDeviceCount lists devices; cudaSetDevice sets the current device per host thread; memory, streams and events belong to a device. CUDA_VISIBLE_DEVICES restricts what a process sees.
  • Data-parallel multi-GPU work is split, computed independently, and combined; interconnect (PCIe, NVLink, NVSwitch) decides how costly the combine is. Check with nvidia-smi topo -m.
  • Peer access (cudaDeviceCanAccessPeer, cudaDeviceEnablePeerAccess, cudaMemcpyPeer) allows direct device-to-device transfers.
  • NCCL provides tuned collectives such as all-reduce, using ring and tree algorithms whose per-GPU traffic stays near twice the buffer size.
  • Cubin (sm_XY) runs on the same major version with at least that minor; PTX (compute_XY) is JIT-compiled for newer devices. Ship native code for the GPUs you target plus PTX for the newest.
  • Containers carry the toolkit while the host supplies the driver; HIP, SYCL, OpenCL and portability layers cover other vendors, and the programming ideas carry over.

Next: put everything together in a hands-on build with Project: Image Blur and Convolution.