Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 31 ยท The Programming Model

Your First Kernel

Write your first CUDA kernel: learn __global__, the <<<blocks, threads>>> launch, host versus device code, asynchronous launches and an error-check macro.

A kernel is a function that runs on the GPU. When you launch it, you do not call it once: you ask the GPU to run it many times at once, one copy per thread. Every thread runs the same code, and each one finds out who it is (its index) so that it can work on its own piece of the data.

This lesson shows the smallest complete CUDA program, explains the strange triple-angle-bracket launch syntax, and covers two things that surprise every beginner: launches are asynchronous, and errors do not announce themselves. The real CUDA C++ is read-only. The runnable examples are Python stand-ins, including a tiny thread-grid simulator that you will reuse throughout the tutorial.

Host and device

CUDA programs have two sides.

  • The host is the CPU and its memory. Your main function runs here.
  • The device is the GPU and its own memory. Kernels run here.

One .cu file holds both. Function qualifiers say where each function lives:

Qualifier Runs on Called from
__global__ device host (or device with dynamic parallelism)
__device__ device device
__host__ host host (the default if you write nothing)

A function can be marked __host__ __device__ to compile for both sides. A __global__ function is the entry point of GPU work: it is the kernel. It must return void.

The smallest program

C++
#include <cstdio>

__global__ void hello() {
    printf("Hello from block %d, thread %d\n", blockIdx.x, threadIdx.x);
}

int main() {
    hello<<<2, 3>>>();
    cudaDeviceSynchronize();
    printf("Back on the host\n");
    return 0;
}

Line by line:

  1. __global__ void hello() declares a kernel. It takes no arguments here, but kernels can take any plain data, such as numbers and pointers to device memory.
  2. threadIdx.x and blockIdx.x are built-in variables available only inside device code. They identify the current thread.
  3. hello<<<2, 3>>>(); is the launch. The values between <<< and >>> are the execution configuration: 2 blocks, 3 threads per block. That makes 2 x 3 = 6 threads in total.
  4. cudaDeviceSynchronize() makes the host wait until the GPU has finished.

When you run it, six lines of "Hello" appear. The order in which they appear is not defined: thread 2 of block 1 may print before thread 0 of block 0. That is not a bug. It is the first lesson of parallel programming: threads run independently, and you must not rely on an order unless you add explicit synchronization.

flowchart TD
    L["hello<<<2, 3>>>()"] --> G["Grid: 2 blocks"]
    G --> B0["Block 0"]
    G --> B1["Block 1"]
    B0 --> T00["thread 0"]
    B0 --> T01["thread 1"]
    B0 --> T02["thread 2"]
    B1 --> T10["thread 0"]
    B1 --> T11["thread 1"]
    B1 --> T12["thread 2"]

One launch creates a grid of blocks; each block holds threads. Here 2 x 3 = 6 threads run the same function.

The next lesson, Grids, Blocks and Threads, covers the hierarchy in depth. For now, think of <<<blocks, threadsPerBlock>>>.

A thread-grid simulator

We cannot run a GPU, but we can run the idea: call the kernel function once for every (block, thread) pair. The simulator below does exactly that, one thread after another. Real hardware runs many at once, so use the simulator to understand indexing, not timing.

Python Run a kernel on a simulated grid

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The simulator prints blocks in order. On a GPU the order is unspecified, so write kernels whose result does not depend on it.

A kernel that does work

Printing is for debugging. Real kernels write results into memory. Each thread computes its own global index and handles one element. The CUDA version of "double every element" looks like this (memory allocation is covered in Vector Addition, so assume d_data already points to device memory):

C++
__global__ void doubleAll(float *data, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) {
        data[i] = data[i] * 2.0f;
    }
}

// launch for n elements, 256 threads per block:
int threads = 256;
int blocks = (n + threads - 1) / threads;   // round up
doubleAll<<<blocks, threads>>>(d_data, n);

Two details deserve attention. The formula blockIdx.x * blockDim.x + threadIdx.x gives every thread a unique number. And the if (i < n) guard is essential because the launch usually creates a few more threads than there are elements.

The same kernel in the simulator:

Python A kernel that doubles an array

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three blocks of four threads give 12 threads for 10 elements. The two extra threads (indices 10 and 11) fail the i < n test and do nothing. Remove the guard in the simulator and Python raises an IndexError; in real CUDA the same mistake silently corrupts memory or crashes, as you will see in Error Handling and Debugging.

Function qualifiers and parameters in practice

The three qualifiers decide which compiler handles a function and who may call it.

  • __host__ marks ordinary CPU code. It is the default, so you rarely write it, but it is useful next to __device__ in the combined form below. A host function can orchestrate everything: allocate memory, launch kernels, print results. It cannot be called from the GPU.
  • __device__ marks a helper that runs on the GPU and can only be called from other device code or from a kernel. Calling it from main is a compile error.
  • __host__ __device__ makes the compiler produce two versions of the same function, one for each side. That keeps a formula identical on the CPU and the GPU, which is handy for checking GPU results against a CPU reference.
C++
#include <cstdio>

__host__ __device__ float computeSquare(float x) { return x * x; }   // two compiled versions

__device__ int calculateOffset(int block, int stride) {              // GPU only
    return block * stride;
}

__host__ void initializeSystem() {                                   // CPU only (same as no qualifier)
    printf("The host is initializing the application...\n");
}

__global__ void scaleAndShift(float *out, float factor, int n) {
    int i = calculateOffset(blockIdx.x, blockDim.x) + threadIdx.x;
    if (i < n) {
        out[i] = computeSquare((float)i) * factor;
    }
}

The kernel above also shows how parameters reach the GPU. Kernel arguments are copied to the device when you launch, so:

  • Pass by value (float factor, int n): every thread receives its own copy. Changing it inside the kernel never changes the host's variable. Small structs work too, as long as they are plain data.
  • Pass by pointer (float *out): the pointer is copied, but it points to the same device memory for every thread, so writes through it are visible afterwards. The pointer must refer to device memory (from cudaMalloc or managed memory). A pointer into ordinary host memory is invalid on the device and causes an illegal-address error.
  • C++ references and host containers such as std::vector cannot be passed to a kernel directly. Pass a raw pointer to the data and the element count instead.

Here is the same structure in the simulator. The list out plays the part of the device array, and factor is passed by value.

Python Qualifiers and kernel parameters

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Two blocks of four threads give eight threads for six elements, so the last two do nothing. Because computeSquare exists on both sides, the host can verify a GPU answer with the identical formula.

Launches are asynchronous

Here is the most important behavior to internalize. When the host executes hello<<<2, 3>>>(), it does not wait for the kernel. It hands the work to the GPU and moves on immediately. The GPU runs the kernel whenever it gets to it, in the order launches were issued within the same stream.

This is why the first program needs cudaDeviceSynchronize(): without it, main could return, and the process exit, before the kernel printed anything. Copies such as cudaMemcpy from device to host act as a synchronization point, so a kernel followed by a blocking copy is safe without an explicit synchronize.

The following stand-in models a queue of pending GPU work. It is only a model of the ordering rules, not of any real scheduling.

Python Why you must synchronize

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The host line "doing CPU work meanwhile" appears before either kernel runs. That gap is the opportunity for overlap: the CPU can prepare more data while the GPU computes. Later lessons (Streams and Events) build on it.

Errors do not announce themselves

A failed CUDA call does not crash your program or throw an exception. It returns an error code (cudaError_t) that you can ignore. A kernel launch returns nothing at all; you must ask afterwards. The standard habit is a small macro that checks every call:

C++
#include <cstdio>
#include <cstdlib>

#define CUDA_CHECK(call)                                              \
    do {                                                              \
        cudaError_t err = (call);                                     \
        if (err != cudaSuccess) {                                     \
            fprintf(stderr, "CUDA error %s at %s:%d: %s\n",           \
                    cudaGetErrorName(err), __FILE__, __LINE__,        \
                    cudaGetErrorString(err));                         \
            exit(EXIT_FAILURE);                                       \
        }                                                             \
    } while (0)

int main() {
    float *d = nullptr;
    CUDA_CHECK(cudaMalloc(&d, 1024 * sizeof(float)));

    doubleAll<<<4, 256>>>(d, 1024);
    CUDA_CHECK(cudaGetLastError());        // catches bad launch configuration
    CUDA_CHECK(cudaDeviceSynchronize());   // catches errors while the kernel ran

    CUDA_CHECK(cudaFree(d));
    return 0;
}

There are two checks after a launch because there are two kinds of failure. cudaGetLastError() reports problems detected at launch time, such as asking for 2,000 threads per block. cudaDeviceSynchronize() returns errors that happened while the kernel ran, such as an illegal memory access, but only once the GPU has actually got that far. Use the macro on every call from the start; it will save hours.

A tiny stand-in shows the pattern of "return a code, then check it":

Python The check-every-call habit

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What is real and what is the stand-in

The .cu programs above are genuine CUDA C++ and will compile with nvcc on a machine with a GPU. The Python launch function only imitates the meaning of <<<grid, block>>>: it runs the threads one at a time, so it cannot show parallel speed, races or synchronization effects. Where a lesson needs those, it uses a different model and says so.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does the qualifier global mean?

    A __global__ function is a kernel: it executes on the GPU and the host starts it with the <<<...>>> syntax. It must return void.
  2. 2. How many threads does the launch kernel<<<4, 128>>>() create?

    That is 4 blocks times 128 threads per block, which is 512 threads in total.
  3. 3. Why is an if (i < n) guard usually needed in a kernel?

    The block count is rounded up, so the last block can contain threads whose index is beyond the end of the data.
  4. 4. After kernel<<<b, t>>>(...) returns on the host, what is guaranteed?

    Launches are asynchronous. Use a synchronization point such as cudaDeviceSynchronize or a blocking cudaMemcpy to wait for completion.
  5. 5. Why call both cudaGetLastError() and cudaDeviceSynchronize() after a launch?

    Some errors are detectable at launch; others, like illegal memory accesses, only surface once the GPU has executed the kernel.

Summary

  • A kernel is a __global__ function that runs once per thread; the host launches it with kernel<<<blocks, threadsPerBlock>>>(args).
  • The host is the CPU, the device is the GPU; __host__, __device__ and __global__ say where code runs.
  • A global index like blockIdx.x * blockDim.x + threadIdx.x lets each thread pick its data, and a bounds check protects the extra threads.
  • Thread order is unspecified, and launches are asynchronous, so synchronize before using results.
  • Wrap every CUDA call in an error-checking macro and check after launches.
  • The Python simulator is a stand-in that models indexing, not speed or parallel behavior.

Next: understand how threads are organized in 1D, 2D and 3D in Grids, Blocks and Threads.