Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 31 ยท Concurrency

Streams and Events

Learn CUDA streams and events: the default stream, cudaMemcpyAsync, overlapping copy and compute, cudaEvent timing, stream priorities and host callbacks.

So far every example has followed the same script: copy the input to the GPU, run a kernel, copy the result back, one step after the other. The GPU has separate hardware for those steps: copy engines that move data across PCIe or NVLink, and the SMs that run kernels. While the SMs compute on one batch, a copy engine can already deliver the next. Doing nothing about that leaves a large part of the machine idle.

CUDA gives you streams to describe which work may overlap, and events to mark and time points inside that work. This lesson explains the default stream, asynchronous copies, how to overlap transfers with computation, how to time GPU work correctly, and how stream priorities and host callbacks fit in. It builds on Moving Data: Pinned, Managed and Unified Memory and on Vector Addition from Start to Finish.

The CUDA C++ here is read-only, because this site has no GPU. The runnable TryIts are Python stand-ins: a small timeline simulator that schedules operations on a compute engine and two copy engines, using the same rules as the real device (order inside a stream, one operation at a time per engine). The durations are made-up round numbers, not measurements.

A stream is an ordered queue

A stream is a queue of GPU operations (kernel launches, copies, event records) that execute in the order you issued them. Operations in different streams have no ordering guarantee between them: the hardware may run them at the same time, or in any order, if it has the resources.

Everything you did in earlier lessons ran in the default stream (stream 0, also written 0 or nullptr). By default it is the legacy default stream, which is special: it synchronizes with every other blocking stream. A kernel launched into the default stream waits until all earlier work in other blocking streams finishes, and later work in those streams waits for it. That makes it safe and slow. Two ways to relax it:

  • create your own streams with cudaStreamNonBlocking, which do not synchronize with the legacy default stream;
  • compile with nvcc --default-stream per-thread, so each host thread gets its own default stream that behaves like an ordinary one.
C++
cudaStream_t s1, s2;
cudaStreamCreate(&s1);                                       // blocking stream
cudaStreamCreateWithFlags(&s2, cudaStreamNonBlocking);       // does not sync with stream 0

myKernel<<<grid, block, 0, s1>>>(d_a, n);    // 4th launch parameter = the stream
cudaMemcpyAsync(h_b, d_b, bytes, cudaMemcpyDeviceToHost, s2);

cudaStreamSynchronize(s1);   // host waits for everything queued in s1
cudaStreamDestroy(s1);
cudaStreamDestroy(s2);

The third launch parameter (here 0) is the dynamic shared memory size; the fourth is the stream. Every launch and every Async copy returns to the host immediately; the host only waits when you call a synchronizing function such as cudaStreamSynchronize or cudaDeviceSynchronize.

Asynchronous copies need pinned memory

cudaMemcpy blocks the host until it completes. cudaMemcpyAsync(dst, src, bytes, kind, stream) only queues the copy. But a copy from ordinary pageable host memory has to be staged through a hidden pinned buffer first, so it cannot truly overlap with other work. For real overlap, the host buffer must be pinned (page-locked), allocated with cudaMallocHost or cudaHostAlloc:

C++
float* h_in;
cudaMallocHost(&h_in, bytes);      // pinned: the DMA engine can read it directly
// ... fill h_in ...
cudaMemcpyAsync(d_in, h_in, bytes, cudaMemcpyHostToDevice, stream);

Free it with cudaFreeHost. Pinned memory is a limited resource: pinning too much starves the operating system.

Overlapping copy and compute

Imagine a job that copies N bytes in, runs a kernel, and copies N bytes out. With one stream, the three steps run one after the other. Now split the data into chunks and give each chunk its own stream. While chunk 1 is computing, chunk 2 is copying in, and chunk 0 is copying out.

C++
const int CHUNKS = 4;
size_t chunkElems = n / CHUNKS;
size_t chunkBytes = chunkElems * sizeof(float);
cudaStream_t streams[CHUNKS];
for (int c = 0; c < CHUNKS; ++c) cudaStreamCreate(&streams[c]);

for (int c = 0; c < CHUNKS; ++c) {
    size_t off = c * chunkElems;
    cudaMemcpyAsync(d_in + off, h_in + off, chunkBytes, cudaMemcpyHostToDevice, streams[c]);
    scale<<<(chunkElems + 255) / 256, 256, 0, streams[c]>>>(d_in + off, d_out + off, chunkElems);
    cudaMemcpyAsync(h_out + off, d_out + off, chunkBytes, cudaMemcpyDeviceToHost, streams[c]);
}
cudaDeviceSynchronize();
flowchart LR
    S0["Stream 0<br/>chunk 0"] --> H["Copy engine<br/>host to device"]
    S1["Stream 1<br/>chunk 1"] --> H
    S0 --> K["SMs<br/>kernels"]
    S1 --> K
    S0 --> D["Copy engine<br/>device to host"]
    S1 --> D

Streams feed three separate engines; each stream keeps its own order while the engines work on different chunks at the same time.

Three hardware rules limit the overlap, and the simulator below enforces them:

  1. Operations in one stream run in order.
  2. One engine runs one operation at a time: copies in the same direction queue behind each other, and so do kernels (unless each kernel is small enough to share the SMs with another).
  3. Host-to-device and device-to-host copies use separate engines on most GPUs, so they can overlap with each other and with a kernel.
Python A timeline simulator: one stream vs four

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In this model, the work takes 80 ms in one piece, because the stages add up (24 + 32 + 24). With four chunks it drops to 44 ms (and to 35 ms with sixteen), so the table shows diminishing returns: the kernel engine becomes the bottleneck, so the best possible time cannot go below the kernel's 32 ms plus the unavoidable pipeline fill and drain. More chunks also mean more launch overhead in real life, which this simple model leaves out. Four to sixteen chunks is a typical range; measure to choose.

The order in which you issue operations matters on some hardware. A "depth-first" loop (copy, kernel, copy for chunk 0, then chunk 1...) as above works well on recent GPUs. On older ones the single hardware queue per engine could create false dependencies, and a "breadth-first" order (all the copies in, then all the kernels, then all the copies out) overlapped better. If a profile shows less overlap than you expect, try the other order.

Events

An event is a marker you drop into a stream. It completes when all work queued in that stream before it has finished. Events do two jobs.

Timing. Because the host returns from a launch immediately, a CPU timer around a launch measures almost nothing. Record events on the GPU timeline instead:

C++
cudaEvent_t start, stop;
cudaEventCreate(&start);
cudaEventCreate(&stop);

cudaEventRecord(start, stream);
myKernel<<<grid, block, 0, stream>>>(d_a, n);
cudaEventRecord(stop, stream);

cudaEventSynchronize(stop);               // wait until `stop` has happened
float ms = 0.0f;
cudaEventElapsedTime(&ms, start, stop);   // milliseconds between the two events
printf("kernel: %.3f ms\n", ms);

cudaEventDestroy(start);
cudaEventDestroy(stop);

The elapsed time has a resolution of roughly half a microsecond. Run the kernel once before timing it, to exclude one-time start-up work such as context creation and module loading, and average several runs.

Dependencies between streams. cudaStreamWaitEvent(streamB, event, 0) makes later work in streamB wait until event (recorded in another stream) completes, without blocking the host. This is how you say "stream B's kernel needs stream A's copy".

C++
cudaEvent_t copied;
cudaEventCreateWithFlags(&copied, cudaEventDisableTiming);   // cheaper if you do not time it
cudaMemcpyAsync(d_x, h_x, bytes, cudaMemcpyHostToDevice, copyStream);
cudaEventRecord(copied, copyStream);
cudaStreamWaitEvent(computeStream, copied, 0);      // compute waits for the copy only
kernel<<<grid, block, 0, computeStream>>>(d_x, n);
Python Why host timers lie, and how events fix it

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The dependency lines at the end encode exactly what cudaStreamWaitEvent means: the waiting stream's clock cannot be earlier than the event's time.

Priorities and callbacks

Stream priorities. You can create a stream that the scheduler favors when several have blocks waiting for an SM:

C++
int leastPriority, greatestPriority;
cudaDeviceGetStreamPriorityRange(&leastPriority, &greatestPriority);
// numerically lower = higher priority; typical range is 0 (lowest) to -1 (highest)
cudaStream_t urgent;
cudaStreamCreateWithPriority(&urgent, cudaStreamNonBlocking, greatestPriority);

Priority only affects which waiting blocks get an SM first; it does not interrupt blocks that are already running. It is useful for latency-sensitive work, such as a small interactive kernel next to a long batch job.

Host callbacks. Sometimes the CPU should react when the GPU reaches a point, for example to release a buffer or submit the next job. cudaLaunchHostFunc queues a CPU function into the stream; it runs after all earlier work in the stream completes, and later work in the stream waits until it returns:

C++
void CUDART_CB onDone(void* userData) {
    printf("chunk %d finished\n", *(int*)userData);   // no CUDA API calls allowed here
}
// ...
cudaLaunchHostFunc(streams[c], onDone, &ids[c]);

Do not call CUDA functions inside the callback, and keep it short, because it holds up the stream. The older cudaStreamAddCallback is deprecated in favor of cudaLaunchHostFunc.

Common pitfalls

  • Using pageable host memory with cudaMemcpyAsync and wondering why nothing overlaps.
  • Launching everything into the legacy default stream, which serializes with all other blocking streams.
  • Reusing a host buffer before the stream that reads it has finished. Async calls return early; synchronize or record an event first.
  • Timing with the host clock without synchronizing.
  • Creating a stream per tiny operation. Streams are cheap but not free, and thousands of tiny streams just add overhead (see CUDA Graphs and Dynamic Parallelism for another way to cut launch cost).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What ordering guarantee does a CUDA stream give?

    A stream is an ordered queue. Different streams are free to overlap or run in any relative order unless you add events or other synchronization.
  2. 2. Why does cudaMemcpyAsync often fail to overlap with a kernel when the host buffer comes from malloc?

    For true asynchronous DMA the host memory must be page-locked (cudaMallocHost or cudaHostAlloc).
  3. 3. Why is a CPU timer around a kernel launch usually wrong?

    Launches are asynchronous. Either synchronize before reading the CPU clock, or record events on the stream and use cudaEventElapsedTime.
  4. 4. What does cudaStreamWaitEvent(streamB, evt, 0) do?

    It adds a dependency on the GPU side, so the host keeps running and streamB's later operations wait.
  5. 5. A lower numeric stream priority means...

    In CUDA, numerically lower priority values mean higher priority; the default stream priority is 0, the lowest.

Summary

  • A stream is an ordered queue of GPU work; different streams may overlap.
  • The legacy default stream synchronizes with other blocking streams; use non-blocking streams or --default-stream per-thread to avoid that.
  • Real copy/compute overlap needs cudaMemcpyAsync, pinned host memory, chunked work and separate streams.
  • Engines are limited: one operation at a time each, and host-to-device and device-to-host copies use separate engines.
  • Events time GPU work (cudaEventElapsedTime) and link streams (cudaStreamWaitEvent).
  • Priorities favor urgent streams when blocks wait; cudaLaunchHostFunc runs CPU code in stream order.

Next, learn how to cut the cost of launching many small operations: CUDA Graphs and Dynamic Parallelism.