Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 31 ยท The Programming Model

Vector Addition from Start to Finish

Build the classic CUDA vector add program step by step: allocate device memory, copy data, launch the kernel, copy back, free, and choose a block size.

Vector addition is the "hello world" of GPU computing: c[i] = a[i] + b[i] for every element. It does almost no arithmetic, so it will not show off the GPU's speed, but it contains the complete shape of every CUDA program: allocate on the device, copy inputs in, launch, copy results out, free. Learn this skeleton once and every later lesson is a variation.

The full CUDA C++ program is read-only below. The runnable stand-ins simulate device memory and the launch in Python, and use NumPy to verify the result, exactly as you would verify a real program against a CPU version.

The five-step recipe

flowchart LR
    A["1. Allocate<br/>host and device"] --> B["2. Copy inputs<br/>host to device"]
    B --> C["3. Launch kernel"]
    C --> D["4. Copy result<br/>device to host"]
    D --> E["5. Free memory<br/>and check"]

Every simple CUDA program follows this path.

The CPU and the GPU have separate memories. A pointer returned by cudaMalloc points into device memory and must never be dereferenced by host code; likewise a normal malloc pointer cannot be read by a kernel (with the exception of special memory kinds covered in Moving Data). Data travels only through explicit copies.

The kernel

C++
__global__ void vecAdd(const float *a, const float *b, float *c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        c[i] = a[i] + b[i];
    }
}

One thread computes one output element. Pointers are passed by value: the kernel receives the device addresses that cudaMalloc produced. The const qualifiers are optional but document which arrays are read-only.

The complete program

C++
#include <cstdio>
#include <cstdlib>
#include <cmath>

#define CUDA_CHECK(call)                                               \
    do {                                                               \
        cudaError_t err = (call);                                      \
        if (err != cudaSuccess) {                                      \
            fprintf(stderr, "CUDA error: %s (%s:%d)\n",                \
                    cudaGetErrorString(err), __FILE__, __LINE__);      \
            exit(EXIT_FAILURE);                                        \
        }                                                              \
    } while (0)

__global__ void vecAdd(const float *a, const float *b, float *c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) c[i] = a[i] + b[i];
}

int main() {
    const int n = 1 << 20;                       // 1,048,576 elements
    const size_t bytes = n * sizeof(float);

    // 1. Allocate host and device memory
    float *h_a = (float *)malloc(bytes);
    float *h_b = (float *)malloc(bytes);
    float *h_c = (float *)malloc(bytes);
    float *d_a, *d_b, *d_c;
    CUDA_CHECK(cudaMalloc(&d_a, bytes));
    CUDA_CHECK(cudaMalloc(&d_b, bytes));
    CUDA_CHECK(cudaMalloc(&d_c, bytes));

    for (int i = 0; i < n; ++i) {
        h_a[i] = (float)i;
        h_b[i] = 2.0f * i;
    }

    // 2. Copy inputs to the device
    CUDA_CHECK(cudaMemcpy(d_a, h_a, bytes, cudaMemcpyHostToDevice));
    CUDA_CHECK(cudaMemcpy(d_b, h_b, bytes, cudaMemcpyHostToDevice));

    // 3. Launch
    const int threads = 256;
    const int blocks = (n + threads - 1) / threads;
    vecAdd<<<blocks, threads>>>(d_a, d_b, d_c, n);
    CUDA_CHECK(cudaGetLastError());

    // 4. Copy the result back (this call waits for the kernel to finish)
    CUDA_CHECK(cudaMemcpy(h_c, d_c, bytes, cudaMemcpyDeviceToHost));

    // Verify on the CPU
    for (int i = 0; i < n; ++i) {
        if (fabsf(h_c[i] - 3.0f * i) > 1e-3f * (1.0f + 3.0f * i)) {
            printf("Mismatch at %d: %f\n", i, h_c[i]);
            return 1;
        }
    }
    printf("All %d results correct\n", n);

    // 5. Free everything
    CUDA_CHECK(cudaFree(d_a));
    CUDA_CHECK(cudaFree(d_b));
    CUDA_CHECK(cudaFree(d_c));
    free(h_a); free(h_b); free(h_c);
    return 0;
}

Build with nvcc vecadd.cu -o vecadd. Points to notice:

  • cudaMalloc(&d_a, bytes) takes the address of the pointer and the size in bytes (not elements). Forgetting sizeof(float) is a classic bug that allocates a quarter of what you need.
  • cudaMemcpy(dst, src, bytes, direction) takes the destination first, like memcpy. Using the wrong direction constant is another classic bug.
  • The copy from device to host is blocking on the default stream, so it implicitly waits for the kernel. That is why no cudaDeviceSynchronize() appears here.
  • The verification compares with a tolerance. Floating-point results from the GPU can differ slightly from the CPU for more complex math; for plain addition they match.

A device-memory simulator

The stand-in below mimics cudaMalloc, cudaMemcpy and cudaFree with NumPy arrays hidden behind integer "device pointers". Host code can only reach device data through the copy functions, which mirrors the real rule. It also reuses the thread-grid launcher from Your First Kernel.

Python Vector add on a simulated device

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The same check you would write in C appears as a one-liner here: compare the device result against h_a + h_b, computed by NumPy on the "host". Always keep such a reference implementation; it is how you find bugs in kernels.

Choosing a block size

The grid is derived from the block size, so the real decision is "threads per block". Beginners ask for the perfect number. There is none, but there are firm rules:

  1. Use a multiple of 32. The hardware runs threads in groups of 32 called warps (Warps, SMs and the Hardware). A block of 100 threads occupies 4 warps, wasting 28 lanes.
  2. Stay at or below 1,024, the hardware limit.
  3. 128 to 256 is a sensible default. Smaller blocks risk leaving the multiprocessor underused; very large blocks reduce flexibility and can limit how many blocks fit at once.
  4. Measure when it matters. For simple kernels the difference is small. For heavy kernels, profile different sizes (The Optimization Workflow).

The stand-in below shows how the block size changes the number of blocks, the number of idle threads, and the number of warps wasted by a partly filled last warp.

Python Block size, blocks and wasted threads

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that block sizes like 33 or 100 waste lanes inside every block, while multiples of 32 waste none. The count of idle threads from rounding up (the idle column) is always smaller than one block and is harmless.

Common mistakes in this program

  • Using n instead of n * sizeof(float) in cudaMalloc or cudaMemcpy.
  • Swapping source and destination in cudaMemcpy.
  • Dereferencing a device pointer on the host (for example d_a[0] in main), which crashes with a segmentation fault.
  • Skipping the bounds check, which writes past the end of d_c in the last block.
  • Ignoring errors, so the program prints "correct" for garbage because the kernel never ran. Without CUDA_CHECK and a real comparison against a CPU result, you are guessing.
  • Timing the launch call and believing the kernel is incredibly fast. The launch returns immediately; time with events (Streams and Events).

Why this program is slow-ish

For vector addition the GPU does one addition per three memory operations (two loads and a store). The arithmetic is so cheap that time is dominated by memory, and the copies across the PCIe bus can take longer than the kernel itself. That is a feature of this problem, not of the GPU: speed-ups appear when you do much more work per byte moved or keep data on the device across many kernels. The later lessons on memory and the optimization workflow quantify this.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the size argument of cudaMalloc measure?

    Sizes are in bytes, so allocate n * sizeof(float) for n floats.
  2. 2. Why is no cudaDeviceSynchronize needed before reading results with a blocking cudaMemcpy device-to-host on the default stream?

    Work in one stream runs in order, and the blocking copy does not return until the data has arrived.
  3. 3. Which is a bad default block size and why?

    100 threads fill 3 warps completely and use only 4 lanes of a fourth warp, so 28 lanes do nothing.
  4. 4. What goes wrong if you read d_a[0] directly in main()?

    A pointer from cudaMalloc refers to device memory. The host must copy data back with cudaMemcpy before reading it.
  5. 5. For 1,000 elements and 256 threads per block, how many threads sit idle behind the bounds check?

    4 blocks launch 1,024 threads for 1,000 elements, so 24 threads fail the i < n test.

Summary

  • Every basic CUDA program: allocate host and device memory, copy inputs, launch, copy results back, free.
  • Sizes passed to cudaMalloc and cudaMemcpy are in bytes; device pointers must not be used on the host.
  • blocks = (n + threads - 1) / threads and an if (i < n) guard cover any n.
  • Pick a block size that is a multiple of 32, usually 128 to 256, and measure for performance-critical kernels.
  • Compare against a CPU reference and check errors, or you cannot trust the output.
  • Vector addition is limited by memory and transfers, not arithmetic; the stand-in simulator shows the logic, not the speed.

Next: see how the hardware actually runs your blocks in Warps, SMs and the Hardware.