Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 31 ยท Why GPUs

Setting Up CUDA

Install the NVIDIA driver and CUDA Toolkit, compile with nvcc, read nvidia-smi, understand compute capability, and try WSL, Linux, Windows or the cloud.

Before you can run a kernel you need three things: a CUDA-capable NVIDIA GPU, a driver that talks to it, and a toolkit that turns your .cu files into programs. This lesson explains each piece, how to check that everything works, and what to do if you do not own an NVIDIA GPU.

The site cannot install anything on your machine, so the commands below are read-only. The runnable examples are Python stand-ins that model the same decisions, such as reading a device description and choosing a compile target.

The pieces and who owns them

flowchart TD
    A["Your .cu source file"] --> B["nvcc compiler<br/>(CUDA Toolkit)"]
    B --> C["Executable with<br/>GPU code embedded"]
    C --> D["CUDA runtime library<br/>(Toolkit)"]
    D --> E["NVIDIA driver"]
    E --> F["GPU hardware"]

Your program calls the runtime, the runtime calls the driver, and the driver controls the GPU.

  • The NVIDIA driver is installed with the operating system. It is the only piece that must match your hardware. It includes the tool nvidia-smi.
  • The CUDA Toolkit contains nvcc (the compiler), the runtime and math libraries, headers, and profilers. You install it to develop. Programs built with it only need the driver to run.
  • Compute capability is the version number of a GPU's feature set, written like 8.6. It tells the compiler which instructions and limits to assume.

A key rule: a newer driver runs programs built with an older toolkit, but a toolkit needs a driver at least as new as its minimum. If the driver is too old you get an error such as "CUDA driver version is insufficient for CUDA runtime version".

Step 1: check the driver with nvidia-smi

On Linux, Windows, and WSL, open a terminal and run:

Terminal
nvidia-smi

It prints a table with the GPU name, driver version, temperature, memory use, and the running processes. In the top right it shows a "CUDA Version" field. That number is the highest CUDA version the driver supports, not the version of any toolkit you have installed. Beginners often confuse the two.

Useful variants:

Terminal
nvidia-smi -L                      # list GPUs
nvidia-smi --query-gpu=name,compute_cap,memory.total --format=csv
nvidia-smi -l 1                    # refresh every second

The compute_cap query field is available on reasonably recent drivers; if yours rejects it, deviceQuery (below) prints the same information.

Step 2: install the CUDA Toolkit

  • Linux: use NVIDIA's package repository for your distribution (Ubuntu, Debian, RHEL and others are supported) and install the cuda-toolkit meta package. Install the driver first, or let the repository's cuda package bring one.
  • Windows: download the installer from NVIDIA's CUDA downloads page. You also need Microsoft Visual Studio (the C++ build tools) because nvcc uses the host C++ compiler for the CPU half of your code.
  • WSL 2 (Windows Subsystem for Linux): install the NVIDIA driver on Windows only, not inside Linux. The Windows driver exposes the GPU to WSL. Inside the distribution install only the toolkit, using the package labeled for WSL so that it does not overwrite the driver.
  • macOS: modern Macs have no NVIDIA support, so use a cloud GPU or a different API such as Metal.

After installing, make sure the compiler is on your path:

Terminal
nvcc --version

On Linux the toolkit usually lives in /usr/local/cuda; you may need export PATH=/usr/local/cuda/bin:$PATH.

Windows command-line checklist

On Windows the order matters, because the toolkit installer looks for a host compiler.

  1. Install the latest NVIDIA driver for your card and reboot.
  2. Install Visual Studio (Community is free) with the Desktop development with C++ workload. That provides cl.exe, the host compiler.
  3. Install the CUDA Toolkit from NVIDIA's download page. Its installer integrates with the Visual Studio you just installed.
  4. Open the x64 Native Tools Command Prompt for VS (it sets up the paths for cl.exe), then run nvcc --version and cl to confirm both tools are found.
  5. In that prompt, compile nvcc hello.cu -o hello.exe and run hello.exe. A plain cmd window without the Visual Studio environment usually fails with "Cannot find compiler 'cl.exe' in PATH".

In VS Code, install the C/C++ extension and run the same commands from a terminal started from that Native Tools prompt (or add a task that calls nvcc). Nothing about CUDA itself changes.

Step 3: compile and run

Save this as hello.cu. The extension matters: nvcc treats .cu files as mixed CPU/GPU code.

C++
#include <cstdio>

__global__ void hello() {
    printf("Hello from the GPU, thread %d\n", threadIdx.x);
}

int main() {
    hello<<<1, 4>>>();
    cudaDeviceSynchronize();
    return 0;
}

Build and run it:

Terminal
nvcc hello.cu -o hello
./hello

You can pass useful flags:

Terminal
nvcc -O2 -std=c++17 hello.cu -o hello             # optimize, pick the C++ standard
nvcc -arch=sm_86 hello.cu -o hello                # target one compute capability
nvcc -arch=native hello.cu -o hello               # target the GPU in this machine
nvcc -lineinfo -g -G hello.cu -o hello_debug      # debugging builds (slow code)

-arch=sm_86 means "generate machine code for compute capability 8.6". If you leave the flag out, nvcc picks a default target that may be old, so for serious work always say what you target. -arch=native is available in recent toolkits and is convenient for local development.

What nvcc does with a .cu file

nvcc is a driver that runs several tools, not one compiler. It splits your .cu file into two parts and sends each to the right compiler.

  1. Device compilation. The kernels and __device__ functions go through NVIDIA's own compiler. The result is PTX and, for each target you ask for, SASS machine code, packed into a fat binary.
  2. Host compilation. Everything else, including main and the code that launches kernels, goes to your normal host C++ compiler (GCC or Clang on Linux, cl.exe from Visual Studio on Windows). nvcc rewrites each <<<...>>> launch into ordinary runtime calls first, and embeds the fat binary in the host object file.
  3. Linking. The host objects and the CUDA runtime library become one executable. At run time the runtime loads the matching device code onto the GPU.

That is why you need a supported host compiler installed, and why nvcc -ccbin /path/to/g++ exists: it picks which host compiler to use. To see the steps, run nvcc --dryrun hello.cu. A .cu file can be compiled into an object (nvcc -c hello.cu -o hello.o) and linked later, exactly like C++.

Compute capability, SASS and PTX

nvcc compiles GPU code in two forms. SASS is the real machine code for a specific GPU generation (sm_86). PTX is a portable intermediate assembly that the driver can compile later for a newer GPU. A binary that embeds PTX as well as SASS keeps working on future hardware. The flag -gencode arch=compute_86,code=sm_86 writes it out in full; Multi-GPU, Portability and the Wider Landscape returns to this.

Roughly, compute capability numbers map to generations: 7.0 and 7.5 are Volta and Turing, 8.0 and 8.6 are Ampere, 8.9 is Ada, 9.0 is Hopper, 10.x and 12.x are Blackwell parts. Check NVIDIA's documentation for your exact card instead of trusting a memorized table.

Ask the device about itself

A real program uses the runtime API to list devices:

C++
#include <cstdio>

int main() {
    int count = 0;
    cudaGetDeviceCount(&count);
    for (int d = 0; d < count; ++d) {
        cudaDeviceProp p;
        cudaGetDeviceProperties(&p, d);
        printf("Device %d: %s\n", d, p.name);
        printf("  compute capability %d.%d\n", p.major, p.minor);
        printf("  multiprocessors    %d\n", p.multiProcessorCount);
        printf("  global memory      %.1f GiB\n", p.totalGlobalMem / 1073741824.0);
        printf("  max threads/block  %d\n", p.maxThreadsPerBlock);
    }
    return 0;
}

Because we cannot call a real device, the stand-in below uses a made-up list of GPUs as plain data and prints it in the same layout. The names and figures are invented for illustration, not real products.

Python A simulated device query

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line shows how a compute capability turns into an -arch flag: join the digits after sm_.

Choosing a compile target

A second stand-in models a decision you will make for real: given the GPUs you must support, which targets should you compile? Anything not in the list will rely on PTX being compiled by the driver at first run.

Python Which -gencode flags do I need?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The result has real machine code for each capability in your fleet, plus PTX for the newest one so that even later hardware can run the binary. Each extra target makes the file bigger and the build slower, which is why you list only what you need.

Verify the installation

NVIDIA ships sample programs on GitHub (the cuda-samples repository). Two are useful immediately:

  • deviceQuery prints everything the runtime knows about each GPU.
  • bandwidthTest measures copy speed between host and device.

If nvcc --version works but programs fail, check these in order:

  1. Does nvidia-smi work? If not, fix the driver.
  2. Is the driver new enough for the toolkit? Compare the "CUDA Version" printed by nvidia-smi with nvcc --version.
  3. On WSL, is the Windows driver recent and WSL itself updated?
  4. In a container, did you start it with GPU access (for example docker run --gpus all) and install the NVIDIA Container Toolkit?

No GPU? Use the cloud

You can learn everything in this tutorial on rented hardware.

Google Colab step by step

Colab gives you a free NVIDIA GPU in a notebook, which is enough for every lesson here.

  1. Enable the GPU. Choose Runtime, then Change runtime type, pick a GPU hardware accelerator and save.
  2. Verify the allocation. In a cell run !nvidia-smi. The table should list a GPU model, its memory and the driver version. If it says no device was found, the runtime type was not applied.
  3. Check the compiler. Run !nvcc --version. Colab images normally ship the CUDA toolkit, so this prints a release number.
  4. Install the notebook plugin. Run !pip install nvcc4jupyter. This small package lets a cell hold CUDA source.
  5. Load the extension. Run %load_ext nvcc4jupyter.
  6. Write and run a program. Start a cell with the cell magic %%cuda, then put a complete .cu program under it. The plugin compiles it with nvcc and prints the output below the cell.
Text
%%cuda
#include <cstdio>
__global__ void hello() { printf("Hello from the GPU\n"); }
int main() { hello<<<1, 1>>>(); cudaDeviceSynchronize(); return 0; }

If you prefer files, write the source with %%writefile hello.cu and compile it with !nvcc hello.cu -o hello && ./hello.

Other options

  • Cloud providers (AWS, Google Cloud, Azure, and smaller GPU clouds) rent virtual machines with GPUs by the hour. Their deep learning images usually include the driver and toolkit already. Cloud and Python alternatives are in Python on the GPU.
  • Remember to shut instances down. A forgotten GPU machine is an expensive mistake.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the "CUDA Version" shown at the top right of nvidia-smi tell you?

    It describes the driver's capability. The toolkit you installed is reported separately by nvcc --version.
  2. 2. Which component do you need on a machine that only RUNS a finished CUDA program?

    Development needs the toolkit; running a built program needs only a sufficiently new driver plus any shared libraries it links.
  3. 3. A GPU has compute capability 8.6. Which nvcc flag targets it directly?

    The sm_ prefix followed by the major and minor digits selects the machine code for that capability.
  4. 4. Why does a binary often embed PTX in addition to SASS?

    SASS is tied to specific hardware. PTX is portable assembly that the driver can compile later for a newer architecture.
  5. 5. On WSL 2 where should the NVIDIA display driver be installed?

    The Windows driver exposes the GPU to WSL. Installing a Linux driver inside WSL can break that setup.

Summary

  • You need an NVIDIA GPU, a driver (checked with nvidia-smi), and the CUDA Toolkit (nvcc) to build programs.
  • The driver must be at least as new as the toolkit requires; the "CUDA Version" in nvidia-smi is the driver's maximum, not the installed toolkit.
  • Compute capability (for example 8.6) selects -arch=sm_86; embedding PTX keeps a binary usable on future GPUs.
  • cudaGetDeviceProperties reports what the device offers; here a Python stand-in mimics that output.
  • No GPU? Use Colab or a cloud VM, and remember to turn rented machines off.

Next: write and launch your first GPU function in Your First Kernel.