Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 31 ยท Synchronization and Parallel Patterns

Warp Primitives and Cooperative Groups

Learn CUDA warp primitives: __shfl_sync, ballot and vote functions, warp-level reduce and scan, then cooperative_groups tiles and grid synchronization.

The 32 threads of a warp execute together on the same hardware, so they can exchange data much faster than threads of a block can through shared memory. CUDA exposes this with warp-level primitives: shuffles that move a register value from one lane to another, and votes that combine a yes/no answer from all lanes. No memory is touched and no __syncthreads() is needed. Together they let you write short, fast reductions, scans and counters.

The second half of the lesson introduces cooperative groups, a C++ library that wraps these operations in objects (a tile of 32 threads, a block, the whole grid), so your code says which group of threads is cooperating. It builds on Warps, SMs and the Hardware and on Parallel Reduction.

The CUDA C++ here is read-only, because this site has no GPU. The runnable TryIts are Python stand-ins that model a warp as a list of 32 lane values and apply each primitive to the whole list. Data movement and results are exact; real hardware executes these instructions in a single step, which a list comprehension only imitates.

Lanes, masks and the _sync suffix

A thread's position inside its warp is its lane, from 0 to 31. You get it with threadIdx.x % 32 for a 1D block. Every modern warp primitive has _sync in its name and takes a mask as its first argument: a 32-bit number whose bit i says "lane i takes part". Almost always you pass 0xffffffff (all 32 lanes).

The mask exists because, since the Volta architecture, threads in a warp are no longer guaranteed to move in lockstep after a branch. Each _sync call makes the named threads meet at that instruction first, so the exchange is safe. Every lane named in the mask must execute the same call with the same mask, or the behavior is undefined, and a hang is possible. The old unsuffixed versions (__shfl, __ballot, __all) were deprecated in CUDA 9 and are unavailable on Volta and newer GPUs for that reason.

Shuffles

A shuffle reads a register of another lane. There are four variants, and in each case v is the value held in the calling lane's register:

C++
unsigned FULL = 0xffffffff;
float a = __shfl_sync(FULL, v, srcLane);       // read lane srcLane directly
float b = __shfl_up_sync(FULL, v, delta);      // read lane (me - delta)
float c = __shfl_down_sync(FULL, v, delta);    // read lane (me + delta)
float d = __shfl_xor_sync(FULL, v, laneMask);  // read lane (me XOR laneMask)

For up and down, a lane whose source would fall outside the warp (a negative lane, or one past 31) receives its own value back unchanged. An optional last argument width (a power of two up to 32) splits the warp into independent segments, which is handy for a smaller tile. The XOR variant creates a butterfly pattern in which every lane trades values with a partner, so all lanes (not just lane 0) end up with the same result.

Python The four shuffles on an 8-lane mini warp

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Warp-level reduction and scan

With shuffles, a reduction needs no memory at all. The down version leaves the total in lane 0 only. The xor version leaves the total in every lane, which is useful when all lanes need the answer (for example to normalize by the sum).

C++
__device__ float warpSumAll(float v) {
    for (int mask = 16; mask > 0; mask >>= 1) {
        v += __shfl_xor_sync(0xffffffff, v, mask);
    }
    return v;   // every lane holds the total
}

__device__ float warpInclusiveScan(float v) {
    int lane = threadIdx.x & 31;
    for (int delta = 1; delta < 32; delta <<= 1) {
        float n = __shfl_up_sync(0xffffffff, v, delta);
        if (lane >= delta) v += n;       // lanes below delta have nothing to add
    }
    return v;   // lane i holds the sum of lanes 0..i
}

The scan is a Hillis-Steele scan (see Scan, Histogram and Other Patterns) done entirely inside registers. The if (lane >= delta) is needed because out-of-range shuffles return the lane's own value, and adding that would count it twice.

Python Warp sum with xor and warp scan with up

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Votes and ballots

Votes combine one boolean per lane:

C++
int allPos = __all_sync(0xffffffff, x > 0);     // 1 if every lane has x > 0
int anyNeg = __any_sync(0xffffffff, x < 0);     // 1 if at least one lane has x < 0
unsigned bits = __ballot_sync(0xffffffff, x > 0); // bit i = lane i's answer

__ballot_sync is the most powerful: it returns a 32-bit mask with one bit per lane, and every lane receives the same mask. Combine it with two helper functions: __popc(m) counts the set bits, and __ffs(m) finds the first set bit (counting from 1). With unsigned lt = (1u << lane) - 1 (all bits below my lane), __popc(bits & lt) tells a lane how many earlier lanes said yes. That is a warp-sized stream compaction in two instructions.

A classic use is the warp-aggregated atomic. Instead of 32 lanes each calling atomicAdd(&counter, 1), one lane does a single atomicAdd(&counter, n) for the whole warp, and every lane computes its own slot from the ballot:

C++
__device__ int appendIfPositive(int* out, int* counter, int x) {
    unsigned FULL = 0xffffffff;
    int lane = threadIdx.x & 31;
    bool keep = (x > 0);
    unsigned votes = __ballot_sync(FULL, keep);
    int warpCount = __popc(votes);
    int base = 0;
    if (lane == 0 && warpCount > 0) base = atomicAdd(counter, warpCount);
    base = __shfl_sync(FULL, base, 0);                 // broadcast lane 0's base
    if (keep) {
        int myRank = __popc(votes & ((1u << lane) - 1u));
        out[base + myRank] = x;
        return base + myRank;
    }
    return -1;
}

One atomic per warp replaces up to 32, which cuts contention exactly as in the histogram example. Recent compilers can apply this transformation themselves in simple cases, but writing it out shows the primitives working together.

Python Ballot, popcount and warp-aggregated append

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other warp operations

  • __syncwarp(mask) is a barrier (and memory fence) for a subset of a warp. Use it when lanes have diverged and must meet again before they use data written by each other through shared memory.
  • __activemask() reports which lanes are currently executing. It is a snapshot, not a guarantee that the lanes will meet, so prefer an explicit mask when you can compute one.
  • __match_any_sync(mask, value) returns a mask of the lanes that hold the same value as the caller; __match_all_sync tests whether all lanes agree. They need compute capability 7.0 or newer and are useful for grouping, such as deduplicating histogram updates.
  • __reduce_add_sync(mask, x) and its relatives do a whole-warp integer reduction in one instruction on compute capability 8.0 and above.

Cooperative groups

The primitives above work on "the warp", but a kernel may want to say "these 4 threads" or "this whole block". The cooperative groups API (header <cooperative_groups.h>) gives each set of threads a handle. A handle knows its size, its thread rank, and how to synchronize and exchange data.

C++
#include <cooperative_groups.h>
#include <cooperative_groups/reduce.h>
namespace cg = cooperative_groups;

__global__ void sumKernel(const float* in, float* total, int n) {
    cg::thread_block block = cg::this_thread_block();           // my block
    cg::thread_block_tile<32> warp = cg::tiled_partition<32>(block);  // my warp

    int i = blockIdx.x * blockDim.x + threadIdx.x;
    float v = (i < n) ? in[i] : 0.0f;

    v = cg::reduce(warp, v, cg::plus<float>());   // warp-wide sum (every lane gets it)

    if (warp.thread_rank() == 0) {
        atomicAdd(total, v);                      // one atomic per warp
    }
    block.sync();                                  // same as __syncthreads()
}

The key types are:

Type Meaning
thread_block All threads of the block; .sync() is __syncthreads().
thread_block_tile<N> A compile-time sized tile of N threads (a power of two up to 32); has shfl, shfl_down, shfl_up, shfl_xor, ballot, any, all and sync().
coalesced_group The currently active threads of a warp, as a group (cg::coalesced_threads()).
grid_group Every thread of the whole grid; .sync() is a grid-wide barrier.

Tiles let you write generic helpers that work for any tile size: a function that takes a cg::thread_block_tile<16> reduces 16 threads, and the same body works for 32. Because the group is a function argument, the code documents which threads must reach the call, and the compiler can check the shape at compile time.

flowchart TD
    G["grid_group<br/>every thread in the launch"] --> B["thread_block<br/>one block, shares shared memory"]
    B --> T["tile of 32 threads<br/>one warp"]
    T --> S["tile of 4 threads<br/>a sub-tile"]

Groups nest from the whole grid down to small tiles; each level can synchronize and share data.

Grid-wide synchronization

Normally blocks cannot wait for each other: the hardware may run only some of them at a time, so a barrier across blocks could deadlock. A grid.sync() is allowed only when you launch with the cooperative launch API, which refuses to start unless every block can be resident on the GPU at the same time:

C++
void* args[] = { &d_in, &d_out, &n };
dim3 grid(numSMs * blocksPerSM), block(256);
cudaLaunchCooperativeKernel((void*)myKernel, grid, block, args);

Inside the kernel, cg::this_grid().sync() then separates phases (for instance, "all blocks finished the first pass" before the second pass reads their results). The price is that the grid size is capped by what fits on the device, so you must size it from the occupancy calculator (cudaOccupancyMaxActiveBlocksPerMultiprocessor). The device must also support cooperative launch. Many problems are easier to solve with two separate kernel launches, which already act as a grid-wide barrier.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does __shfl_down_sync(0xffffffff, v, 2) return for lane 30?

    When the source lane is outside the warp, up and down shuffles return the caller's own value unchanged. Code such as a reduction simply ignores those lanes' results.
  2. 2. What is the main difference between a __shfl_down_sync reduction and a __shfl_xor_sync reduction?

    XOR shuffles pair lanes symmetrically (a butterfly), so each lane accumulates the same total. A down shuffle accumulates toward lane 0.
  3. 3. How does a lane find its rank among the lanes that voted yes in a ballot?

    Masking off the bits at and above the lane leaves only earlier lanes' votes, and __popc counts them. That count is the lane's output slot within the warp's batch.
  4. 4. Why can grid.sync() only be used with cudaLaunchCooperativeKernel?

    If only some blocks were resident, the waiting blocks could block the others from ever starting. The cooperative launch refuses to run unless the whole grid fits.

Summary

  • Warp primitives exchange data between lanes in registers: __shfl_sync, _up_, _down_ and _xor_, plus __all_sync, __any_sync and __ballot_sync.
  • Every _sync call takes a participation mask (usually 0xffffffff), and all named lanes must execute the same call.
  • Shuffle reductions and scans need no shared memory and no barrier; xor leaves the result in all lanes.
  • Ballot plus __popc gives each lane a rank, which makes warp-aggregated atomics and compaction cheap.
  • Cooperative groups wrap thread sets in typed objects (thread_block, thread_block_tile<N>, grid_group).
  • grid.sync() needs a cooperative launch and a grid small enough for the whole GPU to hold at once.

Next, move from threads inside a kernel to work across kernels and copies: Streams and Events.