Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 31 ยท Practice

Practice Exercises with Solutions

Six CUDA exercises on index math, bounds checks, tiling, reduction, coalescing and stream timelines, each solved with runnable thread-grid simulator code.

Reading about kernels is not the same as being able to reason about them. The skill that separates beginners from confident CUDA programmers is a quick, accurate mental model: which thread touches which element, how many times memory is read, and what runs when. These exercises train exactly that. Each one states a problem, asks you to work out the answer before you look, and then gives a runnable solution that proves it.

Try every exercise on paper first. Then press Run on the solution and compare. If your answer differs, change the numbers in the code and watch what moves.

There is no GPU on this site. The exercises that involve kernels use the thread-grid simulator from earlier lessons, which calls a Python function once for every thread with CUDA-style indices. The memory and timing exercises use small models that count transactions or schedule operations. They are honest stand-ins: the arithmetic is real, but no timing here comes from hardware.

Exercise 1: index math

Problem. An image is 40 pixels wide and is processed by a 2D grid of 4 x 4 blocks. (a) The thread with blockIdx = (2, 1) and threadIdx = (3, 2) handles which pixel (x, y), and what is its index in a row-major array? (b) How many 16 x 16 blocks do you need to cover a 1000 x 700 image, and how many threads are launched but idle?

Remember the formula from Grids, Blocks and Threads: x = blockIdx.x * blockDim.x + threadIdx.x, the same for y, and the row-major index is y * width + x. The number of blocks per dimension is the ceiling of the size divided by the block size.

Python Solution 1: index math, checked by the simulator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For (a) the thread is at pixel (11, 6) and the array index is 6 * 40 + 11 = 251. For (b) you need 63 x 44 blocks, which launches 709,632 threads for 700,000 pixels, so 9,632 threads are idle, about 1.4 percent. Idle threads are cheap, which is why rounding the grid up and adding a bounds check is the standard solution.

Exercise 2: what happens without the bounds check

Problem. A kernel doubles an array of n = 1000 floats and is launched with blocks of 256 threads. How many blocks do you launch, how many threads are idle, and what goes wrong if the kernel has no if (i < n) guard?

Python Solution 2: a simulator that catches out-of-bounds writes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Four blocks of 256 give 1,024 threads, so 24 are idle. Without the guard those 24 threads write past the end of the array. The simulator stops at index 1000 with an error. Real hardware does not stop you: a stray write may silently corrupt a neighbouring allocation, or crash with an illegal-address error on a later, unrelated call. That is why the guard is not optional. The lesson on Error Handling, Debugging and Pitfalls shows how to hunt such bugs with compute-sanitizer.

Exercise 3: tiling arithmetic

Problem. Multiply two 64 x 64 matrices. (a) How many global-memory loads does the naive kernel need (counting one load per operand per multiply-add)? (b) How many does a tiled kernel with 16 x 16 tiles need? (c) How many bytes of shared memory does one block use, and how many such blocks fit into 48 KB of shared memory?

Think in phases: the output is divided into (64/16)^2 = 16 blocks, each block runs 64/16 = 4 phases, and in each phase it loads two 16 x 16 tiles.

Python Solution 3: counting loads and shared memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Naive needs 524,288 loads, tiled 32,768, a factor of 16, the tile width. A 16 x 16 block uses 2,048 bytes of shared memory, so shared memory alone would allow 24 blocks per 48 KB, well above what the thread limits allow. The last table shows why T = 32 is the largest comfortable choice: it already means 1,024 threads in a block, the maximum block size on current GPUs.

Exercise 4: a reduction step by step

Problem. Sum these 16 numbers with a tree reduction using sequential addressing (thread t adds the element stride positions to its right): 3 1 4 1 5 9 2 6 5 3 5 8 9 7 9 3. Write down the array after every step, and say how many steps and how many threads are active in each.

flowchart LR
    A["16 values"] -->|"stride 8<br/>8 threads"| B["8 partial sums"]
    B -->|"stride 4<br/>4 threads"| C["4 partial sums"]
    C -->|"stride 2<br/>2 threads"| D["2 partial sums"]
    D -->|"stride 1<br/>1 thread"| E["total"]

Each step halves the number of values; log2 of 16 gives 4 steps.

Python Solution 4: sequential-addressing tree reduction

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The array shrinks 16, 8, 4, 2, 1, so there are 4 steps and the active thread counts are 8, 4, 2, 1. The total is 80. Because the active threads are always the lowest-numbered ones, whole warps drop out together and the remaining warps do not diverge, which is why this layout beats interleaved addressing (Parallel Reduction). The exercise also shows what the barrier protects: step 2 reads values that step 1 wrote.

Exercise 5: coalescing

Problem. A warp of 32 threads reads 4-byte floats. Assume memory is fetched in 32-byte sectors. How many sectors does the warp need for each of these accesses, and what fraction of the fetched bytes is actually used? (a) a[tid] (b) a[tid + 1] where a starts on a sector boundary, so the access is shifted by 4 bytes (c) a[2 * tid] (d) a[32 * tid] (e) every thread reads a[0].

Python Solution 5: counting sectors for access patterns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The contiguous read (a) needs 128 bytes, which is exactly 4 sectors, so it is 100 percent efficient. The shifted read (b) touches a fifth sector and drops to 80 percent: alignment matters. Stride 2 (c) wastes half of every sector, and stride 32 (d) touches 32 different sectors to use 4 bytes of each: 12.5 percent. Case (e) needs just one sector, but it uses only 4 of its 32 bytes because every thread wants the same value: it is cheap in transactions (a broadcast) and reported as 12.5 percent only because the warp needs so little data. See Global Memory and Coalescing for the full story, including structure-of-arrays layouts.

Exercise 6: a stream timeline puzzle

Problem. A job is split into 3 chunks. Each chunk needs a 4 ms upload, a 6 ms kernel and a 4 ms download. The GPU has one copy engine for each direction and one compute engine, and each engine runs one operation at a time. Operations within a stream run in order. (a) How long does the job take with a single stream? (b) With three streams, one per chunk? Draw the schedule.

Python Solution 6: a tiny stream scheduler

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With one stream everything is serial: 3 x (4 + 6 + 4) = 42 ms. With three streams the upload of chunk 1 overlaps the kernel of chunk 0, and so on: the total is 26 ms. The compute engine is the bottleneck once the pipeline fills: it is idle only for the first 4 ms and from then on runs 6 ms kernels back to back (4-10, 10-16, 16-22), followed by the last 4 ms download. The minimum possible is 4 + 3 x 6 + 4 = 26 ms, so three streams already reach it. More chunks would make the overlap even more valuable. On a real GPU, remember that the host memory must be pinned for copies to overlap with kernels; Streams and Events covers the details.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A kernel is launched with 256 threads per block for n = 1000 elements. How many blocks are needed?

    ceil(1000 / 256) = 4 blocks, which gives 1,024 threads. The last 24 need the i < n guard.
  2. 2. A tiled matrix multiply with 16 x 16 tiles does how many times fewer global loads than the naive kernel?

    Every loaded value is reused by T = 16 threads, so loads fall by the tile width.
  3. 3. A warp reads 4-byte floats with a stride of 32 elements. With 32-byte sectors, how many sectors does it touch?

    Each thread's address is 128 bytes from the next, so every thread lands in its own sector, 32 in total, using 4 bytes of each.
  4. 4. Why can three streams finish the 3-chunk job in 26 ms instead of 42 ms?

    Different engines work at the same time, so the upload of one chunk overlaps with the kernel of another. The 6 ms compute engine becomes the bottleneck.

Summary

  • Compute the global index with blockIdx * blockDim + threadIdx, round the grid up and guard with if (i < n).
  • Without a guard, extra threads write out of bounds and hardware will not necessarily tell you.
  • Tiling with width T cuts global loads by T; shared memory per block and threads per block limit how large T can be.
  • Sequential-addressing reduction takes log2(n) steps, keeps the low threads busy and needs a barrier between steps.
  • Count sectors to judge coalescing: contiguous aligned reads are 100 percent efficient, strides and shifts waste bandwidth.
  • Streams let copies and kernels of different chunks overlap; the busiest engine sets the best possible total time.

Continue with CUDA Interview Questions.