Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 31 ยท Libraries and Python

Python on the GPU: CuPy, Numba and PyTorch

Use the GPU from Python with CuPy, Numba @cuda.jit kernels, PyTorch tensors and RAPIDS, see what each is best at, and learn how to choose between them.

You do not have to write C++ to use a GPU. Python has several mature routes, each at a different level: CuPy gives you NumPy arrays that live on the GPU, Numba lets you write CUDA kernels as Python functions, PyTorch gives you tensors with automatic differentiation, and RAPIDS brings pandas- and scikit-learn-style data science to the GPU. Under the hood they use the same ideas as the rest of this tutorial: device memory, kernel launches, streams and the CUDA libraries.

None of them can run on this site, since there is no GPU or CUDA driver here. Their code in this lesson is read-only. The TryIts are stand-ins written in plain NumPy and Python: a drop-in array module pattern, a counter that shows what hidden host-device copies cost, and a small simulation of the @cuda.jit programming style that really executes a kernel function over a grid of threads. The lesson says clearly where the real tool differs.

The big picture

flowchart TD
    Q{"What do you have?"} -->|"NumPy/SciPy code"| C["CuPy<br/>change the import"]
    Q -->|"pandas / scikit-learn code"| R["RAPIDS<br/>cuDF, cuML"]
    Q -->|"A neural network"| P["PyTorch or JAX"]
    Q -->|"A custom loop that vectorizing cannot express"| N["Numba @cuda.jit<br/>or CuPy RawKernel"]

Start from the highest-level tool that fits your problem and drop down only when it cannot express the work.

CuPy: NumPy on the GPU

CuPy mirrors the NumPy (and much of SciPy) API. An array created with CuPy lives in GPU memory, and operations on it run as GPU kernels or library calls (cuBLAS, cuFFT and others):

Python
import cupy as cp

x = cp.random.random((4096, 4096), dtype=cp.float32)   # created on the GPU
y = x @ x.T                                              # cuBLAS
s = cp.sum(y)                                            # reduction kernel
print(float(s))                                          # copies one number back

host = cp.asnumpy(y)       # or y.get(): copies the whole array to a NumPy array
dev = cp.asarray(host)     # copies a NumPy array to the GPU

Two things to remember:

  1. Operations are asynchronous. Python returns as soon as the work is queued. To time something, call cp.cuda.Device().synchronize() first and after; otherwise you measure only the launch.
  2. Conversions are the expensive part. cp.asnumpy, float(...), print(array) and boolean checks all force a copy and a wait. Keep data on the device and convert at the end.

Because the APIs match, you can write one function that accepts either kind of array using the array module pattern: cp.get_array_module(x) returns numpy or cupy depending on where x lives. The stand-in below shows the pattern with a function that works on any array module; here we can only pass NumPy.

Python One function, either array module

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The function body never mentions NumPy or CuPy directly, so the same code runs on either. That is the whole attraction of CuPy: porting a NumPy pipeline can be as small as changing import numpy as np to import cupy as np and then fixing the few places that need the host.

CuPy also lets you go lower when you need to. cp.fuse and cp.ElementwiseKernel build a custom fused element-wise kernel from a C expression, and cp.RawKernel compiles real CUDA C++ source:

Python
add_kernel = cp.RawKernel(r'''
extern "C" __global__
void add(const float* a, const float* b, float* c, int n) {
    int i = blockIdx.x * blockDim.x + threadIdx.x;
    if (i < n) c[i] = a[i] + b[i];
}
''', 'add')

n = 1 << 20
a = cp.ones(n, dtype=cp.float32); b = cp.ones(n, dtype=cp.float32); c = cp.empty_like(a)
add_kernel(((n + 255) // 256,), (256,), (a, b, c, n))

That is the vector-add kernel from earlier lessons, launched from Python. Notice the launch arguments: a grid tuple, a block tuple, then the kernel argument tuple.

Counting hidden copies

Because a GPU array looks like a NumPy array, it is easy to write code that bounces data across the PCIe bus without noticing. The stand-in below wraps a NumPy array in a toy "device array" that counts each implicit transfer, to show how a harmless-looking loop can be costly.

Python How many hidden transfers does a loop cause?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first loop moves only 400 bytes in total, but each of the 100 copies forces the CPU to wait for all queued GPU work, which stalls the pipeline. The second moves 400 MB. The third moves four bytes. On real hardware, the cost of a synchronizing copy is usually more about waiting than about the bytes.

Numba: CUDA kernels written in Python

When your work cannot be expressed as whole-array operations (a loop with data-dependent logic, a stencil, a custom reduction), Numba compiles a Python function to a GPU kernel. You use the same programming model as CUDA C++: a grid of blocks of threads, with cuda.grid, cuda.threadIdx, shared memory and atomics.

Python
from numba import cuda
import numpy as np

@cuda.jit
def saxpy(a, x, y, out):
    i = cuda.grid(1)                  # blockIdx.x * blockDim.x + threadIdx.x
    if i < out.size:                  # bounds check, as in C++
        out[i] = a * x[i] + y[i]

n = 1_000_000
x = np.ones(n, dtype=np.float32)
y = np.full(n, 2.0, dtype=np.float32)

d_x = cuda.to_device(x)               # explicit copies; Numba would also copy NumPy args
d_y = cuda.to_device(y)
d_out = cuda.device_array_like(x)

threads = 256
blocks = (n + threads - 1) // threads
saxpy[blocks, threads](np.float32(3.0), d_x, d_y, d_out)   # asynchronous launch
cuda.synchronize()
result = d_out.copy_to_host()

Rules that differ from ordinary Python:

  • Kernels cannot return a value; they write into array arguments.
  • Inside a kernel only a subset of Python and NumPy is allowed (numbers, tuples, array indexing and arithmetic, math functions). No lists, dicts, strings or classes.
  • The launch syntax is kernel[blocks, threads](args...), a direct counterpart of <<<blocks, threads>>>.
  • Shared memory is cuda.shared.array(shape, dtype) followed by cuda.syncthreads(); atomics are cuda.atomic.add(array, index, value).
  • If you pass plain NumPy arrays Numba copies them for you in both directions on every call, which is convenient and slow; pass device arrays when you launch repeatedly.

The simulator below imitates the decorator-and-launch style closely enough to run a kernel written the same way. It executes threads one after another in Python (so it is slow and exact), not in parallel. It supports cuda.grid(1) and the launch syntax, nothing more.

Python A simulated @cuda.jit kernel

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The kernel body is the same code you would run in real Numba. The grid is rounded up to 1,024 threads for 1,000 elements, and the bounds check discards the extra 24. Real Numba differs mainly in speed, in what the compiler accepts, and in that all threads genuinely run at once.

PyTorch: tensors on the GPU

PyTorch is first of all a deep-learning framework, but its tensor library is a general GPU array library with automatic differentiation:

Python
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

x = torch.randn(4096, 4096, device=device)       # allocated on the GPU
w = torch.randn(4096, 4096, device=device)
y = (x @ w).relu().sum()                         # runs asynchronously on the GPU
print(y.item())                                  # .item() copies one number and waits

model = torch.nn.Linear(4096, 10).to(device)     # parameters move to the GPU
out = model(x)                                   # inputs and parameters must be on the same device

Points to know:

  • tensor.to("cuda") or device="cuda" places data on the GPU; .cpu() brings it back. Mixing a CPU tensor and a GPU tensor in one operation raises an error.
  • Like CuPy, calls are asynchronous; .item(), .cpu(), .numpy() and printing force a wait. Use torch.cuda.synchronize() when timing.
  • Autograd, optimizers, mixed precision (torch.autocast) and torch.compile all operate on GPU tensors; cuDNN and cuBLAS are used automatically.
  • You can write custom GPU kernels for PyTorch with extensions in C++/CUDA or with Triton, a Python-based kernel language. JAX is a similar array library built around compiled, differentiable numerical code.

RAPIDS: data science on the GPU

RAPIDS is a family of libraries that keep the familiar APIs: cuDF for pandas-like data frames, cuML for scikit-learn-style machine learning and cuGraph for graph analytics. Many programs can try the GPU with little or no code change:

Python
# Accelerate existing pandas code without rewriting it
%load_ext cudf.pandas          # in a notebook; or: python -m cudf.pandas script.py
import pandas as pd

df = pd.read_parquet("trips.parquet")
print(df.groupby("zone")["fare"].mean())   # runs on the GPU when supported, else falls back to the CPU

GPU data frames shine when tables are large (millions of rows or more) and operations are joins, group-bys, sorts and aggregations. For small data the transfer and launch overheads dominate.

Choosing one

Situation Reach for
Existing NumPy or SciPy code that needs speed CuPy
Existing pandas code on very large tables RAPIDS (cuDF, or cudf.pandas)
Training or running neural networks PyTorch or JAX
A custom algorithm with loops, stencils, shared memory Numba @cuda.jit, or CuPy RawKernel
Maximum control and the best tooling CUDA C++ (Setting Up CUDA)

General advice: vectorize first. A few whole-array CuPy or PyTorch operations are usually fast enough. Write a custom kernel only after profiling shows that fusing steps or avoiding temporaries would matter. Everything you learned about coalescing, occupancy and transfers applies unchanged, because the kernels underneath are the same kind of kernels. Use the profilers from Profiling with Nsight; they work on Python programs too.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does it mean that CuPy operations are asynchronous?

    Like kernel launches in CUDA C++, CuPy calls queue work and return; timing requires waiting for the device.
  2. 2. In a Numba @cuda.jit kernel, how do you hand a result back to the caller?

    Kernels cannot return values; results go into device arrays passed as arguments.
  3. 3. Which line is likely to cause a hidden synchronizing copy in PyTorch?

    .item() copies a value to the host and waits for all queued GPU work, so calling it constantly stalls the pipeline.
  4. 4. You have a big pandas pipeline of joins and group-bys on a 200-million-row table. What is a good first thing to try?

    cuDF keeps the pandas API, and large joins and aggregations are exactly where GPU data frames pay off.
  5. 5. Why does passing plain NumPy arrays to a Numba kernel on every call hurt performance in a loop?

    Convenient automatic transfers repeat every call; use cuda.to_device once and pass device arrays.

Summary

  • CuPy: NumPy-compatible arrays on the GPU; asynchronous; use cp.asnumpy only at the end; RawKernel and ElementwiseKernel for custom code.
  • Numba @cuda.jit: Python-syntax kernels with cuda.grid, shared memory and atomics; launch with kernel[blocks, threads](...); no return values and a restricted language subset.
  • PyTorch: GPU tensors plus autograd and cuDNN/cuBLAS; place tensors with device= or .to(), and avoid .item() or .cpu() in hot loops.
  • RAPIDS (cuDF, cuML, cuGraph) brings pandas and scikit-learn style APIs to large datasets on the GPU.
  • Vectorize first, write a custom kernel only after profiling, and watch hidden host-device copies in every tool.

Next: learn what happens when things go wrong in Error Handling, Debugging and Pitfalls.