Skip to content
Tutorials Center
My learning

Topics

Lesson 21 of 31 ยท Libraries and Python

The CUDA Library Ecosystem

A tour of the CUDA libraries (cuBLAS, cuFFT, cuRAND, cuSPARSE, cuDNN, Thrust, CUB, CUTLASS) and how to decide between a library call and your own kernel.

Writing your own kernel is a good way to learn and the only choice for truly custom work, but for the common building blocks of numerical computing it is rarely the right choice in production. NVIDIA and others have spent years tuning kernels for matrix multiplication, Fourier transforms, random numbers, sorting and more, for every GPU generation. A one-line library call can beat a week of hand-tuned code.

This lesson is a map of that ecosystem. The libraries need a GPU and the CUDA Toolkit, so their code here is read-only. To make each idea concrete, the TryIts run NumPy equivalents of the same operations: they compute the same results on the CPU, so you can see exactly what each library call means, including some details (such as column-major storage) that trip up newcomers. The stand-ins are not GPU code and say nothing about speed.

flowchart TD
    A["Your application"] --> B["Frameworks: PyTorch, CuPy, RAPIDS"]
    A --> C["Libraries: cuBLAS, cuFFT, cuRAND,<br/>cuSPARSE, cuDNN, Thrust"]
    B --> C
    C --> D["Building blocks: CUB, CUTLASS"]
    D --> E["CUDA runtime and your own kernels"]

Most code should sit near the top. Go down a layer only when the layer above cannot do what you need.

Choosing: library or kernel?

Use a library when:

  • The operation is standard: matrix multiply, FFT, sort, reduction, scan, convolution, sparse matrix-vector product, random numbers.
  • You want performance portability across GPU generations without retuning.
  • Correctness and numerical behavior matter: libraries are heavily tested.

Write your own kernel when:

  • The operation is not covered, or you can fuse several steps into one pass (saving trips to global memory).
  • The shapes or data layout are unusual enough that general routines are inefficient.
  • You are learning, or prototyping an algorithm.

A common middle path is to use library calls for the heavy standard parts and a few small custom kernels to glue them together.

cuBLAS: dense linear algebra

cuBLAS implements the BLAS routines on the GPU: vector operations (level 1, such as axpy and dot), matrix-vector (level 2) and matrix-matrix products (level 3, gemm). gemm computes C = alpha * A * B + beta * C.

C++
#include <cublas_v2.h>

cublasHandle_t handle;
cublasCreate(&handle);

const float alpha = 1.0f, beta = 0.0f;
// C (m x n) = alpha * A (m x k) * B (k x n) + beta * C   -- all column-major
cublasSgemm(handle, CUBLAS_OP_N, CUBLAS_OP_N,
            m, n, k,
            &alpha, dA, m,      // dA with leading dimension m
                    dB, k,
            &beta,  dC, m);
cublasDestroy(handle);

The famous catch: cuBLAS uses column-major storage, as Fortran does, while C and C++ arrays are row-major. If you call it with row-major arrays without care, you get the transpose of what you wanted. The standard trick uses the identity (A B)^T = B^T A^T. A row-major matrix reinterpreted as column-major is its transpose, so you pass B first and A second and get a row-major C out. The stand-in shows the idea with NumPy, where a flat buffer can be viewed either way.

Python Row-major data, column-major BLAS: the transpose trick

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Other useful pieces of the family: cublasGemmEx and cublasLtMatmul for mixed precision and tensor cores, batched calls (cublasSgemmBatched, strided batched) for many small matrices, and cuSOLVER for factorizations, solvers and eigenvalues.

cuFFT: Fourier transforms

cuFFT computes 1D, 2D and 3D FFTs, real or complex, batched. You create a plan once, reuse it for many transforms, and execute:

C++
#include <cufft.h>

cufftHandle plan;
cufftPlan1d(&plan, N, CUFFT_C2C, 1);                   // N complex points, one transform
cufftExecC2C(plan, d_signal, d_spectrum, CUFFT_FORWARD);
cufftExecC2C(plan, d_spectrum, d_back, CUFFT_INVERSE); // result is N times too large
cufftDestroy(plan);

Like FFTW, cuFFT does not normalize: a forward transform followed by an inverse returns the input multiplied by N, so you divide by N yourself. NumPy's ifft divides for you, so the stand-in below makes the difference explicit.

cuRAND: random numbers

Parallel random numbers need care: if every thread draws from the same generator the results are correlated, and if the threads share state you serialize them. cuRAND offers two interfaces. The host API fills a device array in one call; the device API gives each thread its own generator state inside a kernel (used in the Monte Carlo project later).

C++
#include <curand.h>

curandGenerator_t gen;
curandCreateGenerator(&gen, CURAND_RNG_PSEUDO_DEFAULT);
curandSetPseudoRandomGeneratorSeed(gen, 1234ULL);
curandGenerateUniform(gen, d_random, n);       // n floats in (0, 1]
curandDestroyGenerator(gen);

Note that curandGenerateUniform produces values in (0, 1], which excludes 0 and includes 1, unlike NumPy's [0, 1). For reproducible science, fix the seed; results then repeat on the same library version and hardware setup, but are not guaranteed identical across versions or devices.

cuSPARSE: sparse matrices

When most entries of a matrix are zero (graphs, finite elements, recommendation data), storing and multiplying them densely wastes memory and time. cuSPARSE works on formats such as CSR (compressed sparse row): three arrays holding the non-zero values, their column indices, and where each row starts. Its main operations are sparse matrix-vector multiply (SpMV), sparse-dense matrix multiply (SpMM) and sparse-sparse products. The generic API creates descriptors and calls cusparseSpMV.

The stand-in below builds a CSR matrix by hand and multiplies it by a vector, which is what one thread (or warp) per row does on the GPU. Because rows are independent, SpMV parallelizes naturally across rows.

Python CSR sparse matrix-vector multiply

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Row 1 is empty (row_ptr repeats the value 2), so its result is 0 and costs nothing. Rows with very different numbers of non-zeros cause load imbalance, which is why cuSPARSE offers several algorithms.

cuDNN: deep learning primitives

cuDNN provides tuned building blocks for neural networks: convolutions (forward and backward), pooling, normalization, activation functions, attention and RNN layers. You will almost never call it directly. PyTorch, TensorFlow and JAX call it for you; tuning options such as torch.backends.cudnn.benchmark = True let the library try several algorithms and keep the fastest for your shapes. Call it directly only when you build your own framework or inference engine, using its graph-based frontend API.

Thrust: the C++ standard library for the GPU

Thrust feels like the C++ STL: containers (device_vector, host_vector) and algorithms (sort, reduce, transform, scan, copy_if, count_if) that run on the GPU when given device data. It ships with the CUDA Toolkit and needs no linking. It is the fastest way to a working GPU program when your problem is made of such pieces.

C++
#include <thrust/device_vector.h>
#include <thrust/sort.h>
#include <thrust/reduce.h>
#include <thrust/transform.h>
#include <thrust/functional.h>

thrust::device_vector<int> v(h_data.begin(), h_data.end());   // copy to the GPU
thrust::sort(v.begin(), v.end());
int total = thrust::reduce(v.begin(), v.end(), 0, thrust::plus<int>());
thrust::transform(v.begin(), v.end(), v.begin(), thrust::negate<int>());

Thrust hides allocation and launch details, and the price is less control: temporary buffers are allocated for you, and fusing several steps takes extra care (transform_iterator, zip_iterator).

CUB: building blocks you control

CUB (now part of CCCL, the CUDA C++ Core Libraries, together with Thrust and libcu++) offers the same kind of algorithms at three levels: device-wide functions called from the host, and block-wide and warp-wide primitives you call inside your own kernels (BlockReduce, BlockScan, WarpReduce, BlockRadixSort). The device-wide functions use a two-call pattern to let you control memory: the first call only reports how much temporary storage it needs.

C++
#include <cub/cub.cuh>

void* d_temp = nullptr;
size_t temp_bytes = 0;
cub::DeviceReduce::Sum(d_temp, temp_bytes, d_in, d_out, n);   // 1st call: sizes only
cudaMalloc(&d_temp, temp_bytes);
cub::DeviceReduce::Sum(d_temp, temp_bytes, d_in, d_out, n);   // 2nd call: does the work

Use CUB when you want to allocate temp storage once and reuse it, or when you need a block-level scan or reduction inside a fused kernel without writing it yourself (compare with Parallel Reduction).

CUTLASS: matrix multiply templates

CUTLASS is a header-only C++ template library for GEMM and related operations, structured so that each stage (global-to-shared copies, warp-level tensor core instructions, epilogues) is a reusable component. It is for people who need a matrix-multiply variant cuBLAS does not offer: a fused epilogue (bias, activation), an unusual data type, or a grouped GEMM. Expect heavy templates and a steep learning curve; for a first project, use cuBLAS.

A NumPy tour of the same operations

The final stand-in lines up the calls above with their NumPy meanings, including the FFT normalization difference and an inclusive scan, which thrust::inclusive_scan computes.

Python NumPy equivalents of common library calls

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each GPU library call corresponds to one of these lines, but the GPU versions move data across the PCIe bus, run asynchronously on a stream, and are enormously faster for large inputs. When you read library documentation, always check the three details the stand-ins highlighted: storage order, normalization and value range.

Using libraries well

  • Keep data on the device between calls. Copying back to the host after each library call erases the gain (Moving Data).
  • Reuse handles and plans. Creating a cuBLAS handle or cuFFT plan is expensive; create once, use many times.
  • Use streams. Most libraries let you attach a stream (cublasSetStream, cufftSetStream) so that library work overlaps with copies and your kernels.
  • Check status codes. Every library has its own status type (cublasStatus_t, cufftResult, curandStatus_t); check them just as you check cudaError_t.
  • Link correctly: for example nvcc app.cu -lcublas -lcufft -lcurand -lcusparse. Thrust and CUB are header-only.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. You hold row-major matrices A (m x k) and B (k x n) and call a column-major sgemm. How do you get a row-major C = A * B?

    Reinterpreting a row-major buffer as column-major transposes it, and (A B)^T = B^T A^T, so swapping the operands yields the row-major product.
  2. 2. After cuFFT forward then inverse, how does the result compare with the original signal of length N?

    cuFFT does not normalize, like FFTW. NumPy's ifft divides by N for you.
  3. 3. Which library is the closest to the C++ STL for GPU data (sort, reduce, transform)?

    Thrust offers STL-like containers and algorithms that run on the GPU.
  4. 4. What is the main reason to use a library such as cuBLAS rather than your own kernel for a standard matrix multiply?

    Vendor libraries embody years of per-architecture tuning, so they are usually both faster and more reliable than a first-draft kernel.
  5. 5. Why does the CUB device-wide reduce take two calls?

    Passing a null temp pointer returns the required byte count; you allocate that, then call again to do the work.

Summary

  • Prefer a library call for standard operations: it is tuned, tested and portable across GPU generations. Write kernels for custom or fusible work.
  • cuBLAS (dense linear algebra, column-major), cuFFT (unnormalized FFTs with reusable plans), cuRAND (host and per-thread device generators), cuSPARSE (CSR and other sparse formats) and cuDNN (neural network primitives, normally reached through frameworks) cover the classic needs.
  • Thrust gives STL-style algorithms; CUB gives device-, block- and warp-level building blocks with explicit temporary storage; CUTLASS gives templates for custom GEMMs.
  • Keep data on the GPU, reuse handles and plans, attach streams, and check each library's status codes.
  • Watch storage order, normalization and value ranges: they are where library results differ from your CPU expectations.

Next: reach the same power from Python in Python on the GPU: CuPy, Numba and PyTorch.