Skip to content
Tutorials Center
My learning

Topics

Lesson 74 of 132 ยท Smart Pointers and Memory

Python Extensions with PyO3

Learn how PyO3 and maturin let you write fast Python extension modules in Rust, with #[pyfunction], #[pyclass], type conversion, error mapping, the GIL and release builds. Read-only, with runnable stand-ins.

Python is wonderful for gluing things together and slow when a loop runs ten million times. The traditional fix is a native extension: a compiled library that Python imports like any module. For decades these were written in C. Today many popular Python packages (including well-known tools for data validation, text tokenizing and linting) are written in Rust, because Rust gives native speed without the memory bugs that make C extensions risky.

The bridge is PyO3, a crate that exposes Rust functions and types to Python (and can also embed Python in a Rust program). maturin is the build tool that compiles the crate and installs it into a Python environment. This lesson shows the shape of a PyO3 project. PyO3 needs Python and the crate registry, so the PyO3 code here is read-only; the runnable examples are standard-library stand-ins for the ideas underneath. Before you start, you should know FFI basics, because PyO3 sits on top of the same C interface.

The workflow

  1. Create a Python virtual environment and install maturin (pip install maturin).
  2. Run maturin new fast_math (or maturin init in an existing folder) and choose the PyO3 bindings.
  3. Write Rust functions and mark them for export.
  4. Run maturin develop to build a debug version and install it into the active environment, or maturin develop --release for a fast one.
  5. In Python: import fast_math.
  6. For distribution, maturin build --release creates a wheel; maturin publish uploads it to PyPI.

The generated Cargo.toml tells Cargo to produce a dynamic library that Python can load:

TOML
[package]
name = "fast_math"
version = "0.1.0"
edition = "2024"

[lib]
name = "fast_math"
crate-type = ["cdylib"]

[dependencies]
pyo3 = "0.2x"   # use the current release; check crates.io

A first function

This is the whole extension module (read-only):

Rust
use pyo3::prelude::*;

/// Sum of squares from 1 to n, computed in Rust.
#[pyfunction]
fn sum_of_squares(n: u64) -> u64 {
    (1..=n).map(|k| k * k).sum()
}

#[pymodule]
fn fast_math(m: &Bound<'_, PyModule>) -> PyResult<()> {
    m.add_function(wrap_pyfunction!(sum_of_squares, m)?)?;
    Ok(())
}

And in Python:

Python
import fast_math
print(fast_math.sum_of_squares(1000))   # 333833500

#[pyfunction] generates the glue: it receives Python objects, converts the argument to u64, calls your function and converts the result back. If Python passes a string or a negative number, PyO3 raises a TypeError or OverflowError for you; you never write that checking by hand. Note that the function body is plain Rust that does not mention Python at all.

Types and conversion

PyO3 converts between Python and Rust types with two traits, FromPyObject (argument in) and IntoPyObject (result out). The built-in mappings cover the common cases:

Python Rust
int i32, u64, i128, ...
float f32, f64
str String, &str
bool bool
list Vec<T>
dict HashMap<K, V>
tuple Rust tuples
None / value Option<T>
bytes Vec<u8>, &[u8]

Converting a Python list to a Vec<f64> copies it. For large arrays, extension authors use the numpy crate to share memory without copying, or accept &[u8]-style buffers.

Errors become exceptions

A Rust function that returns PyResult<T> (a Result whose error is a Python exception) can use ?. A custom error type that implements From<MyError> for PyErr can be returned directly:

Rust
use pyo3::exceptions::PyValueError;
use pyo3::prelude::*;

#[pyfunction]
fn parse_celsius(text: &str) -> PyResult<f64> {
    text.trim()
        .parse::<f64>()
        .map_err(|_| PyValueError::new_err(format!("not a number: {text:?}")))
}

In Python this raises ValueError: not a number: 'abc', which can be caught with an ordinary try/except. A Rust panic! is turned into a special PanicException, so keep panics for bugs.

Classes

#[pyclass] turns a struct into a Python class, #[pymethods] defines its methods, and #[new] marks the constructor:

Rust
use pyo3::prelude::*;

#[pyclass]
struct Counter {
    count: u64,
}

#[pymethods]
impl Counter {
    #[new]
    fn new() -> Self {
        Counter { count: 0 }
    }

    fn increment(&mut self) -> u64 {
        self.count += 1;
        self.count
    }

    #[getter]
    fn count(&self) -> u64 {
        self.count
    }
}
Python
c = fast_math.Counter()
c.increment(); c.increment()
print(c.count)   # 2

Python holds a reference-counted handle to the Rust struct. PyO3 enforces Rust's borrowing rules at run time (like RefCell): calling a &mut self method while another borrow is active raises an error instead of causing undefined behavior.

The GIL, briefly

CPython's global interpreter lock (GIL) lets one thread run Python code at a time (recent Python versions also offer an optional free-threaded build). PyO3 models holding the lock with a token of type Python<'py>; you receive one inside functions that touch Python objects. When your Rust code does long work that does not touch Python objects, you can release the lock so other Python threads run:

Rust
#[pyfunction]
fn heavy(py: Python<'_>, n: u64) -> u64 {
    py.allow_threads(|| (1..=n).map(|k| k * k % 7).sum())
}

Two rules follow: Rust data you hand to another thread must be Send, and while the lock is released you must not touch Python objects. The type system enforces both.

Runnable stand-ins

The core of a binding is a conversion layer: a dynamic value is checked and turned into a typed Rust value, and a failure becomes an error that the caller can handle. Here is that idea with plain enums and TryFrom:

Rust What FromPyObject does: dynamic value to typed value

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

The "GIL" idea also has a standard-library cousin: a lock that guarantees only one thread touches shared data at a time. Releasing it around heavy work is what allow_threads does.

Rust Do heavy work outside the lock

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

Pitfalls

  • Crossing the boundary has a cost. Calling a tiny Rust function a million times from Python can be slower than doing the work in Python. Move the whole loop into Rust.
  • Copies. Converting big lists copies them; use buffers or the numpy crate for large arrays.
  • Versioning. PyO3 changes its API between releases (for example the Bound API). Follow the guide for the version you install.
  • Wheels. To distribute, build wheels for each platform; maturin and CI services such as GitHub Actions can produce them, and the abi3 feature builds one wheel for many Python versions.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does maturin do?

    maturin drives Cargo to compile the cdylib and then installs it (develop) or packages it as a wheel (build).
  2. 2. What does #[pyfunction] generate?

    The macro writes the conversion and error-handling wrapper so your function body can be ordinary Rust.
  3. 3. How does an Err(PyErr) returned from a Rust function show up in Python?

    PyResult errors are raised as Python exceptions that try/except can catch.
  4. 4. Why can calling a tiny Rust function in a Python loop be slow?

    Conversion and call overhead dominate tiny functions. Do the whole loop in Rust instead.

Summary

  • PyO3 exposes Rust functions and classes to Python; maturin builds and packages them as wheels.
  • #[pyfunction], #[pyclass], #[pymethods] and #[pymodule] generate the glue, so function bodies stay ordinary Rust.
  • Arguments convert through FromPyObject; failures and PyResult errors become Python exceptions.
  • Release the GIL with allow_threads for long computations that do not touch Python objects.
  • Keep the boundary coarse: fewer calls, bigger chunks of work, avoid copying large data.

Next: Testing in Rust starts the testing and tooling chapter.