Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 31 ยท Practice

CUDA Interview Questions

Prepare for CUDA and GPU programming interviews with concise answers on warps, divergence, memory, occupancy, streams and correctness, plus runnable models.

CUDA interviews tend to circle the same ideas: how the hardware executes threads, how memory behaves, and how you find out why a kernel is slow. Interviewers rarely want recited definitions. They want to hear that you can reason about a concrete situation: "this kernel reads with stride 32, what happens?" This lesson collects model questions with short answers you can adapt, and three small runnable models so you can check the numbers behind the answers.

Every answer is deliberately compact. If one feels thin, follow the link to the lesson that covers it in depth. As in the rest of the tutorial, no GPU is available here: the Python models are stand-ins that compute the arithmetic of the situation, and the CUDA snippets are read-only.

Execution model

What is a warp? A warp is a group of 32 threads of one block that the hardware schedules and issues instructions for together. The programming model is SIMT (single instruction, multiple threads): all lanes of a warp execute the same instruction at the same time, each on its own data and registers. Warp size is 32 on all NVIDIA GPUs to date, but write warpSize instead of a literal 32 if your code should survive a change.

Why should block sizes be a multiple of 32? Warps are formed from consecutive thread indices. A block of 100 threads still occupies 4 warps, and the last warp has only 4 active lanes, wasting 28 lanes of hardware for the whole kernel. Common sizes are 128, 256 and 512; the maximum is 1,024 threads per block. See Warps, SMs and the Hardware.

What is the hierarchy of threads, and how does it map to hardware? Threads form blocks, blocks form a grid. A block runs entirely on one streaming multiprocessor (SM) and never migrates; one SM can hold several blocks at once; blocks of a grid may run in any order and cannot synchronize with each other inside the kernel. This independence is what allows the same program to scale from a small laptop GPU to a large data-centre one.

What is warp divergence, and how expensive is it? When lanes of one warp take different sides of a branch, the hardware runs each path in turn with the other lanes masked off, so the warp pays for both paths. The cost depends on which threads diverge. Only divergence inside a warp matters; if the branch condition is uniform per warp (for example depends on threadIdx.x / 32), there is no penalty. The next model makes this concrete.

Python Divergence cost for four branch conditions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

tid % 2 and tid % 32 == 0 both diverge in every warp and cost double. tid < 16 diverges only in warp 0, so it costs just 10 percent more. The condition based on the warp number never diverges. A typical interview follow-up is "how do you fix a divergent kernel?" The answer: reorganize the data or the thread mapping so that threads in the same warp take the same path, or replace the branch with arithmetic (see Divergence, Registers and Instruction-Level Tuning).

What happens if __syncthreads() is inside a divergent if? It is undefined behaviour unless all threads of the block take the same path. In practice the kernel may hang or produce wrong data. Put barriers in code that every thread of the block reaches.

Memory

List the CUDA memory spaces and their scope. Registers are private to a thread and the fastest. Local memory is also private but physically lives in global memory (used for register spills and large per-thread arrays). Shared memory is on-chip and shared by one block. Global memory is large, off-chip and visible to all threads and to the host. Constant memory is a 64 KB read-only region with a cache that broadcasts a single address to a warp. Texture/read-only paths use specialized caches. L1 and L2 caches sit in between. See The Memory Hierarchy.

flowchart TB
    R["Registers<br/>per thread, fastest"] --> S["Shared memory<br/>per block, on chip"]
    S --> L1["L1 cache<br/>per SM"]
    L1 --> L2["L2 cache<br/>whole GPU"]
    L2 --> G["Global memory (DRAM)<br/>whole GPU, largest and slowest"]

Closer to the cores means faster and smaller.

What is memory coalescing? When the 32 threads of a warp access consecutive, aligned addresses, the hardware serves them with a few wide transactions. Strided or scattered accesses need many more transactions for the same useful data. The practical rule: let threadIdx.x index the fastest-changing (contiguous) dimension, and prefer a structure of arrays over an array of structures. See Global Memory and Coalescing.

What is a shared-memory bank conflict? Shared memory is divided into 32 banks, and consecutive 4-byte words fall in consecutive banks. If several lanes of a warp access different words in the same bank, the accesses are serialized. Lanes reading the same word is a broadcast and causes no conflict. The standard fix for column access to a 32 x 32 tile is padding: declare tile[32][33] so that column elements fall in different banks. See Bank Conflicts and Padding.

When is shared memory worth using? When data is reused by several threads of a block, or when it lets you reorder accesses to make global reads coalesced. For data read once in a coalesced pattern it only adds copies and a barrier.

Pinned, pageable, managed: what is the difference? Pageable host memory (plain malloc) can be swapped out, so the driver first copies it into a staging buffer, and the copy cannot overlap with kernels. Pinned (page-locked) memory from cudaMallocHost can be read directly by the GPU's copy engine, is faster and is required for truly asynchronous copies, but too much of it hurts the whole system. Managed (unified) memory from cudaMallocManaged gives one pointer valid on both sides, and the driver migrates pages on demand; convenient, but prefetching with cudaMemPrefetchAsync matters for speed. See Moving Data.

Performance

What is occupancy, and is higher always better? Occupancy is the ratio of resident warps on an SM to the maximum it supports. It is limited by threads per block, registers per thread, shared memory per block and the cap on blocks per SM. More resident warps help hide memory latency, but beyond the point where latency is hidden, extra occupancy gives nothing, and forcing it by limiting registers can cause spills that make the kernel slower. The model below computes occupancy for some configurations using example limits (2,048 threads, 32 blocks, 65,536 registers and 100 KB of shared memory per SM; real GPUs differ and also allocate registers in granules, which this model ignores).

Python An occupancy calculator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Doubling the registers per thread from 32 to 64 halves the resident blocks and the occupancy (100 to 50 percent), and 128 registers leaves 25 percent. A block size of 32 is capped by the 32-block limit and reaches only 50 percent. Dynamic shared memory of 32 KB per block permits only 3 blocks. The lesson for the interview: name all four limiters and say that you would check with the occupancy calculator or the profiler.

How do you decide whether a kernel is memory bound or compute bound? Compute its arithmetic intensity (FLOPs per byte moved from DRAM) and compare it with the GPU's ridge point (peak FLOPS divided by memory bandwidth). Below the ridge you are memory bound and should reduce traffic (coalescing, tiling, caching, fusion, lower precision). Above it you are compute bound and should look at instruction mix, divergence and tensor cores. Confirm with a profiler (Nsight Compute). See The Optimization Workflow.

Python Classifying kernels with the roofline model

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Vector addition has an intensity of 0.08 FLOP per byte, so it can never exceed 75 GFLOPS here (under 0.4 percent of peak). That is not a sign of a bad kernel; it is the nature of the operation. Only the large-tile matrix multiply crosses the ridge point. Interviewers like this question because it checks whether you will measure before optimizing.

Why can launching a kernel for a tiny amount of work be slower than the CPU? A launch costs several microseconds, and a copy over PCIe has latency plus a bandwidth cost. If the work takes less time than that overhead, or if the data must cross the bus for every small step, the CPU wins. CUDA Graphs (CUDA Graphs and Dynamic Parallelism) cut launch overhead, and fusing kernels removes round trips.

Concurrency and correctness

What is the difference between __syncthreads() and cudaDeviceSynchronize()? __syncthreads() is a device-side barrier among the threads of one block. cudaDeviceSynchronize() is host code that blocks the CPU until all previously issued GPU work has finished. They solve different problems and cannot replace each other.

Is a kernel launch synchronous? No. The launch returns to the host immediately and the kernel runs later, in order within its stream. Blocking calls such as cudaMemcpy (the synchronous version) or cudaDeviceSynchronize() make the host wait. This is why timing a kernel with a CPU clock without synchronizing measures only the launch; use cudaEvent timers instead (Streams and Events).

What do you need for copy and compute to overlap? Pinned host memory, cudaMemcpyAsync into a non-default stream, a kernel in a different stream, and independent work (data that does not depend on the other operation). Then the copy engine and the SMs run in parallel. The exercises lesson (Practice Exercises with Solutions) works through a timeline.

What is a race condition on a GPU, and how do you fix it? Two threads access the same memory location without ordering, and at least one writes. A classic example is count[bin]++ from many threads: each does read, add, write, and updates get lost. Fix it with atomicAdd, with a per-block private histogram merged at the end, or by restructuring so each thread owns its output. See Synchronization and Atomics.

How would you implement a sum of a billion floats? Each thread sums a grid-stride slice into a register; the block reduces its threads' sums with a tree in shared memory (or with warp shuffles, which need no shared memory or barrier within a warp); one atomic or a second small kernel combines the block results. Mention that the order of addition changes the rounding, so results can differ slightly from a CPU sum. See Parallel Reduction.

How do you handle errors, given that launches are asynchronous? Wrap every runtime call in a checking macro, call cudaGetLastError() right after a launch to catch launch-configuration errors, and call cudaDeviceSynchronize() (in debug builds) to surface errors that occur while the kernel runs. Some errors, such as an illegal memory access, are sticky: the context is corrupted and every later call fails with the same error. Run compute-sanitizer to locate them. See Error Handling, Debugging and Pitfalls.

When should you not use a GPU? For small data sets, branchy or serial algorithms with little parallelism, workloads dominated by transfers rather than computation, and code that needs frequent host decisions. And when a tuned library (cuBLAS, cuFFT, Thrust) already solves the problem, because writing your own kernel then costs more than it earns.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A branch condition depends on (threadIdx.x / 32) % 2. What is the divergence cost?

    All threads of a warp have the same value of threadIdx.x / 32, so the branch is uniform inside each warp and no path is executed twice.
  2. 2. What is the effect of declaring a shared tile as float tile[32][33] instead of [32][32]?

    With a row length of 33 words, consecutive rows start in different banks, so a warp reading down a column hits 32 different banks.
  3. 3. A kernel uses 128 registers per thread with 256 threads per block on an SM with 65,536 registers. How many blocks fit by the register limit?

    65,536 / (128 x 256) = 2 blocks, which is 16 warps out of a possible 64: 25 percent occupancy.
  4. 4. Why does timing a kernel with a CPU clock right after the launch give a wrong answer?

    Without synchronization or events you only measure how long the launch call took, not the execution of the kernel.
  5. 5. Which pair is required for a copy to overlap with a kernel?

    Asynchronous copies need page-locked host memory and a stream different from the one running the kernel, plus independent data.

Summary

  • A warp is 32 threads executing in SIMT; divergence inside a warp serializes both paths, divergence between warps costs nothing.
  • Know the memory spaces by scope and speed, and be able to explain coalescing and bank conflicts with a concrete access pattern.
  • Occupancy is limited by threads, registers, shared memory and block slots; more is not always better.
  • Use arithmetic intensity and the roofline to decide whether a kernel is memory or compute bound, then profile to confirm.
  • Launches are asynchronous: time with events, check errors after launches, and use pinned memory plus streams to overlap copies and compute.
  • Say what you would measure before you say what you would change.

Continue with CUDA Cheat Sheet.