Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 31 ยท The Programming Model

Grids, Blocks and Threads

Understand threadIdx, blockIdx, blockDim and gridDim, compute global indices in 1D, 2D and 3D with dim3, and use the grid-stride loop for any data size.

Every CUDA launch creates a grid of blocks, and every block contains threads. Your kernel code never sees a "thread number" handed to it; it computes one from four built-in variables. Getting this index arithmetic right is the foundation of every kernel you will write.

This lesson explains the hierarchy, the formulas for 1D, 2D and 3D layouts, the size limits, and the grid-stride loop, a pattern that lets one kernel handle any amount of data. The CUDA code is read-only; every formula also runs in a Python stand-in so you can poke at it.

The hierarchy

A launch has two shapes:

  • gridDim: how many blocks the grid has, in x, y and z.
  • blockDim: how many threads each block has, in x, y and z.

And each thread has two positions:

  • blockIdx: which block it belongs to, within the grid.
  • threadIdx: its position inside its block.

All four are three-component structures with fields .x, .y and .z. Unused dimensions are 1 for sizes and 0 for indices.

flowchart TD
    G["Grid: gridDim = (3, 1, 1)"] --> B0["blockIdx.x = 0"]
    G --> B1["blockIdx.x = 1"]
    G --> B2["blockIdx.x = 2"]
    B1 --> T["Block: blockDim = (4, 1, 1)<br/>threadIdx.x = 0, 1, 2, 3"]

A 1D launch of 3 blocks with 4 threads each: 12 threads, numbered 0 to 11 globally.

Why two levels instead of one big pool? Because the hardware treats them differently. Threads in the same block can cooperate: they can share fast memory and synchronize with each other (Shared Memory). Threads in different blocks cannot cooperate directly, which lets the GPU run blocks in any order on any available multiprocessor, and lets the same program scale from a small GPU to a huge one (Warps, SMs and the Hardware).

The 1D global index

For a one-dimensional launch the formula is the one you have already met:

C++
int i = blockIdx.x * blockDim.x + threadIdx.x;

Read it as "skip all the threads in earlier blocks, then add my position within my block". With 4 threads per block, block 2 thread 1 is 2 * 4 + 1 = 9.

The simulator below prints the mapping for a small grid.

Python 1D global index table

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every number from 0 to 11 appears exactly once. That uniqueness is the point: it is what lets 12 threads divide 12 pieces of work without overlap.

2D layouts for images and matrices

Many problems are naturally two-dimensional: pixels in an image, entries in a matrix. Use a 2D block and a 2D grid, built with the dim3 type:

C++
dim3 block(16, 16);                     // 16 x 16 = 256 threads per block
dim3 grid((width  + block.x - 1) / block.x,
          (height + block.y - 1) / block.y);

kernel2d<<<grid, block>>>(d_image, width, height);

dim3 holds up to three numbers; omitted ones default to 1. Inside the kernel, compute a column and a row independently:

C++
__global__ void kernel2d(float *img, int width, int height) {
    int col = blockIdx.x * blockDim.x + threadIdx.x;
    int row = blockIdx.y * blockDim.y + threadIdx.y;
    if (col < width && row < height) {
        img[row * width + col] *= 0.5f;     // row-major flat index
    }
}

The last line shows the usual convention. Memory is a flat line of numbers, so a 2D position becomes row * width + col (row-major order, as in C arrays). Choosing x for columns is deliberate: threads that are neighbors in x touch neighboring addresses, which is what makes memory access fast (Global Memory and Coalescing).

Python 2D index mapping

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The width of 6 does not divide evenly into blocks of 4, so the grid has 2 blocks across (and 2 down, since 4 rows need two blocks of 2 rows), and the second block column is only half used. The bounds check discards the idle threads. Each pixel is owned by exactly one block.

3D layouts

Volumes (medical scans, simulations on a 3D mesh) use all three dimensions:

C++
dim3 block(8, 8, 4);
dim3 grid((nx + 7) / 8, (ny + 7) / 8, (nz + 3) / 4);

int x = blockIdx.x * blockDim.x + threadIdx.x;
int y = blockIdx.y * blockDim.y + threadIdx.y;
int z = blockIdx.z * blockDim.z + threadIdx.z;
int idx = (z * ny + y) * nx + x;

The flat index nests the same way: layers of rows of columns. The important thing is that dimensions are a convenience for you. The hardware sees a block as one bundle of threads and flattens it itself.

The limits you must respect

  • A block may hold at most 1,024 threads in total on modern GPUs. dim3 block(32, 32) is 1,024 and legal; dim3 block(64, 32) is 2,048 and fails to launch.
  • Block dimension limits are 1,024 in x, 1,024 in y and 64 in z (and their product must still be at most 1,024).
  • Grid limits are very large in x (2,147,483,647 blocks) and 65,535 in y and z.
  • A launch with a bad configuration does not run and sets an error that cudaGetLastError() reports as invalid configuration argument.

You can confirm these numbers for your own GPU with cudaGetDeviceProperties (see Setting Up CUDA).

How many blocks do I need?

For n elements and threads threads per block, you need ceil(n / threads) blocks. In integer arithmetic that is:

C++
int blocks = (n + threads - 1) / threads;

For n = 1000 and 256 threads this gives (1000 + 255) / 256 = 4 blocks, or 1,024 threads for 1,000 elements: 24 threads sit idle behind the bounds check.

The grid-stride loop

So far one thread handled one element. That has two limits: you cannot launch more than the grid allows, and the launch itself has a cost. A more flexible pattern lets each thread handle several elements by jumping ahead by the total number of threads in the grid:

C++
__global__ void scale(float *x, float a, int n) {
    int stride = blockDim.x * gridDim.x;                  // total threads
    for (int i = blockIdx.x * blockDim.x + threadIdx.x;
         i < n;
         i += stride) {
        x[i] *= a;
    }
}

Advantages: the kernel is correct for any n regardless of grid size; you can tune the grid to the hardware (for example a few blocks per multiprocessor) rather than to the data; and consecutive threads still touch consecutive elements on each pass, so memory access stays efficient.

Python Which thread handles which element?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Eight threads cover twenty elements: thread (0, 0) takes elements 0, 8 and 16; the work is shared out as evenly as 20 divided by 8 allows (some threads do 3 items, some 2). The loop condition i < n doubles as the bounds check.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. For a 1D launch, which expression gives a thread's unique global index?

    Multiply the block number by the block size to skip earlier blocks, then add the position inside the current block.
  2. 2. How many blocks are needed for 1,000 elements with 256 threads per block?

    ceil(1000 / 256) = 4, computed in integers as (1000 + 255) / 256.
  3. 3. Which block shape is invalid on a modern GPU?

    64 x 32 = 2,048 threads, which exceeds the 1,024-threads-per-block limit.
  4. 4. In the 2D mapping, why is x usually used for the column?

    Threads with consecutive threadIdx.x values are grouped together, so mapping them to consecutive addresses gives efficient memory access.
  5. 5. What is the advantage of a grid-stride loop?

    Each thread advances by the total thread count, so the same kernel works whether the grid is smaller or larger than the data.

Summary

  • A launch creates a grid of blocks of threads; gridDim, blockDim, blockIdx and threadIdx identify the shape and the position.
  • Global index in 1D: blockIdx.x * blockDim.x + threadIdx.x; in 2D compute row and column the same way and flatten with row * width + col.
  • Use dim3 for 2D and 3D configurations; a block holds at most 1,024 threads in total.
  • Round the block count up with (n + threads - 1) / threads and guard against extra threads.
  • A grid-stride loop (i += blockDim.x * gridDim.x) makes a kernel work for any data size.
  • Threads in a block can cooperate; blocks are independent, which lets programs scale across GPUs.

Next: put these pieces together in a complete program in Vector Addition from Start to Finish.