Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 31 ยท Production

Error Handling, Debugging and Pitfalls

Check every CUDA call, understand sticky and asynchronous errors, decode illegal address and out-of-memory failures, and avoid the classic beginner bugs.

CUDA programs fail in a quieter way than most. A kernel launch that is wrong does not crash your program or throw an exception; it simply returns, and the problem may surface milliseconds later at an unrelated call, or never, leaving you with silently wrong numbers. Reliable GPU code therefore starts with a habit: check every call, and know how and when each kind of error is reported.

This lesson is about that habit, about the few error types you will meet most often, and about the beginner bugs behind most of them. The real runtime cannot run here, so CUDA code is read-only. The TryIts are Python stand-ins: a small simulated runtime that reproduces the timing of asynchronous and sticky errors, a grid simulator that catches out-of-bounds indexing the way a sanitizer does, and a short experiment on floating-point differences, which are not errors but are often mistaken for them.

Every call can fail: the check macro

All CUDA runtime functions (cudaMalloc, cudaMemcpy, cudaFree, ...) return a cudaError_t. A value of cudaSuccess means all is well. Wrapping each call in a macro is the standard idiom:

C++
#include <cstdio>
#include <cstdlib>

#define CUDA_CHECK(call)                                                   \
    do {                                                                   \
        cudaError_t err__ = (call);                                        \
        if (err__ != cudaSuccess) {                                        \
            fprintf(stderr, "CUDA error at %s:%d: %s (%s)\n",              \
                    __FILE__, __LINE__,                                    \
                    cudaGetErrorString(err__), cudaGetErrorName(err__));   \
            exit(EXIT_FAILURE);                                            \
        }                                                                  \
    } while (0)

CUDA_CHECK(cudaMalloc(&d_a, bytes));
CUDA_CHECK(cudaMemcpy(d_a, h_a, bytes, cudaMemcpyHostToDevice));

cudaGetErrorName returns the symbol (cudaErrorIllegalAddress) and cudaGetErrorString a sentence ("an illegal memory access was encountered"). Printing the file and line tells you where you noticed the error, which, as the next section shows, is not always where it happened.

Kernel launches need two checks

A kernel launch kernel<<<grid, block>>>(...) returns nothing and starts work that finishes later. Two kinds of failure exist, reported in two different places:

  1. Launch errors, detected immediately when the launch is validated: a bad configuration (for example 2,048 threads per block, over the 1,024 limit), too much shared memory, or a binary built for the wrong architecture. Retrieve them with cudaGetLastError() right after the launch.
  2. Execution errors, which happen while the kernel runs on the GPU: an out-of-bounds access, an assertion failure, a hardware exception. They are reported by a later call, such as cudaDeviceSynchronize() or the next cudaMemcpy.
C++
#define CUDA_CHECK_LAUNCH()                  \
    do {                                     \
        CUDA_CHECK(cudaGetLastError());      \
        CUDA_CHECK(cudaDeviceSynchronize()); \
    } while (0)

myKernel<<<blocks, threads>>>(d_a, n);
CUDA_CHECK_LAUNCH();    // debug builds only: the sync serializes everything

cudaGetLastError() reads and clears the last recorded error (for non-sticky errors); cudaPeekAtLastError() reads without clearing. The synchronize in the macro is valuable while debugging but removes the overlap between CPU and GPU, so guard it with #ifdef DEBUG or a debug flag in production code.

When you cannot tell which launch was guilty, run the program with the environment variable CUDA_LAUNCH_BLOCKING=1. It makes every launch synchronous, so an execution error is reported at the very launch that caused it. Remember to switch it off afterwards: it slows everything down.

Errors surface late: a simulation

The stand-in below implements a tiny fake runtime with a stream of queued kernels. Calls return quickly, execution happens later, and a failing kernel reports at the next synchronizing call. Run it to see the "wrong line number" problem, and what CUDA_LAUNCH_BLOCKING changes.

Python Where does an asynchronous error show up?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the default run, lines 10 to 12 all look fine and the error appears at the memcpy on line 13, which is innocent. With blocking launches the error is attributed to line 11, the kernel that really failed. Lines 12 and 13 then keep returning the same error, which previews the sticky behavior described next. This is why a failing cudaMemcpy often means "some earlier kernel crashed".

Sticky errors

Most errors are recoverable: after cudaGetLastError() clears an invalid-configuration or out-of-memory error, you may continue. But errors that corrupt the GPU context, such as cudaErrorIllegalAddress or a device-side assert, are sticky: every later CUDA call in that process keeps returning the same error, and only terminating the process (not just that kernel) clears it. If one line in your log shows an illegal address and the next hundred calls all fail with the same message, the first one is the root cause. Look at the earliest error, never the last.

The errors you will meet most

Error Usual cause What to do
cudaErrorInvalidConfiguration ("invalid configuration argument") Block over 1,024 threads, a zero-size grid, a grid dimension limit exceeded, too much shared memory Print the launch parameters; check the computed grid for n = 0
cudaErrorIllegalAddress ("an illegal memory access was encountered") Out-of-bounds read or write; host pointer used on the device; use after cudaFree Run compute-sanitizer; check bounds and pointer types
cudaErrorMemoryAllocation ("out of memory") Request larger than free device memory, leaks, or fragmentation Query cudaMemGetInfo; free buffers; process data in chunks
cudaErrorNoKernelImageForDevice ("no kernel image is available for execution on the device") The binary has no code for this GPU's architecture Compile with the right -arch or -gencode; include PTX
"invalid device function" (cudaErrorInvalidDeviceFunction) The same architecture mismatch as the row above, reported by older toolkits when the kernel was compiled for a different sm_ target Rebuild with -arch=native or the exact sm_ of your GPU; check with cuobjdump --list-elf what your binary contains
cudaErrorLaunchTimeout A long kernel on a GPU that also drives a display (OS watchdog) Shorten the kernel, split the work, or use a headless GPU
cudaErrorInsufficientDriver / cudaErrorInvalidDevice Driver older than the toolkit; wrong device index; no GPU visible (containers) Update the driver; check nvidia-smi and CUDA_VISIBLE_DEVICES

Out-of-memory deserves care: the memory a program can allocate is less than the number on the box, because the driver, the context and other processes use some. Always check the return value of cudaMalloc, and when you are near the limit, process in chunks rather than hoping.

Catching out-of-bounds indexing

The most common cause of illegal addresses is plain: the grid is rounded up, an if (i < n) check is missing, and the extra threads write past the end of the array. Writes into unallocated memory fault; writes into allocated-but-unrelated memory (inside another array) silently corrupt data, which is worse.

compute-sanitizer catches both kinds on the real GPU (Profiling with Nsight). The stand-in below does the same job for the Python simulator: it runs a kernel over a thread grid with a checked array wrapper that reports the first bad access, naming the block and thread, just as the sanitizer does.

Python A simulator that catches out-of-bounds access

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first bad access is a read (the thread reads before it writes), by thread 232 of block 3 (global index 1000, the first one past the end of the 1,000-element arrays). On a real GPU the same bug may not crash at all if the arrays happen to sit inside a larger allocation, which is exactly why you run the sanitizer on small test cases.

Floating-point differences are not errors

A different class of surprise is a result that is slightly different from the CPU's, or from the previous run. Common reasons, none a bug in itself:

  • Order of summation. Floating-point addition is not associative. A parallel reduction adds numbers in a different order than a serial loop, and one using atomics may add in a different order every run (Parallel Reduction).
  • Fused multiply-add. The GPU compiler fuses a * b + c into one instruction with one rounding, while many CPU builds round twice. --fmad=false disables fusion (at a speed cost).
  • Precision and fast math. float has about 7 decimal digits; intrinsics and --use_fast_math trade accuracy for speed.
  • Different library algorithms (cuBLAS versus a CPU BLAS) accumulate differently.

So never compare GPU and CPU results for exact equality. Compare with a tolerance relative to the size of the numbers, and decide that tolerance deliberately.

Python Summation order changes the last digits

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The two float32 sums differ from each other in the last digits and both differ from the float64 reference, usually by a little. The tree order is typically the more accurate of the two, which is a pleasant side effect of parallel reduction. The correct test is "close within a tolerance", not "equal".

Classic beginner bugs

  1. No bounds check after rounding the grid up (if (i < n)): illegal addresses or silent corruption.
  2. Wrong byte count. cudaMalloc(&d, n) instead of n * sizeof(float) allocates a quarter of what you need.
  3. Using a host pointer in a kernel (or a device pointer on the host): the first is an illegal address, the second a segmentation fault. A naming convention (h_ and d_) helps.
  4. Forgetting to copy back or to synchronize before reading results or timing. A cudaMemcpy device-to-host synchronizes the default stream implicitly; an async copy does not.
  5. Integer division in the grid size: n / threads drops the remainder; use (n + threads - 1) / threads. For n = 0 this gives a zero-block grid, which is an invalid configuration, so guard it.
  6. Uninitialized device memory. cudaMalloc does not clear memory; use cudaMemset or write before reading.
  7. __syncthreads() inside a branch that only some threads of the block take: undefined behavior, often a hang.
  8. Data races: two threads writing the same location without atomics (Synchronization and Atomics).
  9. Index overflow. int indices overflow past about 2.1 billion elements; use size_t or long long for big arrays, and compute blockIdx.x * blockDim.x in the wider type.
  10. Leaks and double frees. Every cudaMalloc needs one cudaFree; wrap resources in RAII classes in C++.
  11. Ignoring the architecture flag. Compiling for sm_80 and running on an older GPU gives "no kernel image available".

Debugging checklist

  1. Add CUDA_CHECK to every runtime call and cudaGetLastError() after every launch.
  2. Reproduce on a tiny input.
  3. Run compute-sanitizer (memcheck, then racecheck, initcheck, synccheck).
  4. Set CUDA_LAUNCH_BLOCKING=1 to find the first failing launch.
  5. Look at the earliest error in the log; later ones may be sticky echoes.
  6. Compare with a CPU reference using a tolerance.
  7. Use printf in the kernel for a few threads, or cuda-gdb on a -G build.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why can a kernel's out-of-bounds write be reported at a later cudaMemcpy?

    A launch only queues work. The error is raised when the host next waits for the device, which may be a different line.
  2. 2. Which call reports an invalid launch configuration such as 2,048 threads per block?

    Configuration errors are detected at launch time and retrieved with cudaGetLastError().
  3. 3. What makes an error "sticky"?

    Errors such as illegal address leave the context unusable; the first occurrence is the one to investigate.
  4. 4. A GPU sum differs from the CPU sum in the last few digits. What is the most likely explanation?

    Floating-point addition is not associative, so parallel reductions give slightly different, equally valid answers.
  5. 5. What does CUDA_LAUNCH_BLOCKING=1 do?

    With synchronous launches, an execution error is attributed to the correct line, at the cost of speed.

Summary

  • Check every CUDA runtime call with a macro that prints the error name, file and line.
  • After a kernel launch, call cudaGetLastError() for launch errors; execution errors appear at a later synchronizing call. CUDA_LAUNCH_BLOCKING=1 pinpoints the guilty launch.
  • Sticky errors (illegal address, device assert) poison the whole process, so investigate the first error, not the last.
  • Illegal addresses usually come from missing bounds checks, wrong pointers or use after free; compute-sanitizer finds them with the thread and block.
  • Out-of-memory, invalid configuration and missing kernel image errors have specific causes listed above.
  • GPU and CPU floating-point results differ legitimately because of summation order and fused multiply-add; compare with a tolerance.

Next: scale beyond one device and one vendor in Multi-GPU, Portability and the Wider Landscape.