Skip to content
Tutorials Center
My learning

Topics

Lesson 103 of 132 ยท Testing and Tooling

Property Testing and Fuzzing

Test Rust with properties instead of examples: random inputs, shrinking, round-trip and invariant checks, plus fuzzing with cargo-fuzz. Build a mini property runner and fuzzer.

Example-based tests check the cases you thought of: assert_eq!(add(2, 2), 4). The bugs that reach production are usually the cases you did not think of: an empty list, a run of 256 identical bytes, a string that ends in the middle of a multi-byte character. Two techniques attack exactly those blind spots. Property-based testing states a rule that must hold for every input, and a framework tries hundreds of random inputs, then shrinks a failing one to a minimal example. Fuzzing throws huge numbers of mutated, often malformed inputs at your code, usually looking for panics, crashes and hangs.

The real tools are the proptest and quickcheck crates for properties and cargo-fuzz (libFuzzer) for fuzzing. They need Cargo and dependencies, so our editor shows them read-only. To learn the ideas properly, though, this lesson builds tiny runnable stand-ins from the standard library: a random generator, a property runner with shrinking, and a mutation fuzzer. You should already know the basics from Testing and Test Patterns.

Thinking in properties

A property is a statement that is true for all inputs of some kind. Finding good ones is the hard part. These families cover most code:

  • Round trip: decode(encode(x)) == x. Serialization, compression, parsing and printing.
  • Idempotence: doing it twice equals doing it once. sort(sort(x)) == sort(x), normalizing a path.
  • Invariants: something stays true. A sorted list is non-decreasing; a balanced tree stays balanced.
  • Oracle: compare a fast, clever implementation with a slow, obvious one.
  • No crash: for any input, the function returns instead of panicking.
  • Metamorphic: a change to the input has a predictable effect on the output. Adding a constant to every element adds it to the sum times the length.
flowchart LR
    G["Generate random input"] --> P{"Property holds?"}
    P -->|"yes"| N{"Enough cases?"}
    N -->|"no"| G
    N -->|"yes"| OK["Pass"]
    P -->|"no"| S["Shrink: try smaller inputs"]
    S --> S2{"Still fails?"}
    S2 -->|"yes, keep smaller"| S
    S2 -->|"no more reductions"| R["Report minimal counterexample"]

A property framework is a loop of generate, check and, on failure, shrink.

A property runner from scratch

We need three pieces: a source of pseudo-randomness, a way to generate values, and a shrinker. The random generator below is a tiny xorshift with a fixed seed, so every run in this lesson prints the same thing. (Real frameworks print the seed of a failing run so you can replay it.)

Rust A mini property-testing framework

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

The first two properties hold for all 200 cases. The third is false, and notice what the shrinker reports: not the original random list of a dozen numbers, but the smallest list that still breaks the claim. A minimal counterexample is far easier to understand and turn into a regression test than a big random one. That shrinking step is the real value of a property framework over a plain random loop.

Finding a real bug with a property

Here is a function that is supposed to return the distinct values of a list in sorted order. It looks plausible, and a hand-written example test with [3, 1, 3]... may even pass depending on the input. The property "no duplicates in the output" exposes it.

Rust A property finds a bug in sorted_unique

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

The shrunk counterexample is three numbers where the repeated value is separated by a different one: a classic shape that a person writing examples rarely tries. Keeping such cases as ordinary unit tests afterwards is good practice (proptest even saves them in a proptest-regressions file for you).

The same thing with proptest

In a Cargo project you rarely write the runner yourself. With proptest the first property above looks like this (read-only: it needs the crate as a dev-dependency):

Rust
use proptest::prelude::*;

proptest! {
    #[test]
    fn reverse_twice_is_identity(v in prop::collection::vec(any::<i32>(), 0..100)) {
        let mut w = v.clone();
        w.reverse();
        w.reverse();
        prop_assert_eq!(w, v);
    }

    #[test]
    fn parse_never_panics(s in "\\PC*") {
        let _ = my_crate::parse(&s);   // any printable string, must not panic
    }
}

v in ... describes a strategy: how to generate values and how to shrink them. Strategies compose (prop_map, prop_filter, tuples, prop_oneof!), so you can generate valid structs, not just numbers. Useful settings: ProptestConfig::with_cases(1000) to run more cases, and the PROPTEST_CASES environment variable to turn the dial in CI.

When a property fails and you fix the bug, turn the minimal counterexample into a plain #[test] with a descriptive name. Properties explore; example tests document and lock in what you learned.

Fuzzing: mutation and feedback

Property tests usually generate well-formed values. A fuzzer is happy to feed your code garbage. A basic fuzzer keeps a corpus of interesting inputs, mutates them (flip a bit, delete a byte, duplicate a chunk, splice two inputs), runs the target, and watches for a crash. A coverage-guided fuzzer such as libFuzzer or AFL also watches which branches each input reaches, and keeps the mutants that reach new code. That feedback is what lets it climb through layers of checks (a magic header, a length field, a checksum) that pure random bytes never get past.

The typical targets are parsers, decoders, deserializers, regex engines, and anything that takes untrusted bytes. The typical findings are panics (index out of bounds, unwrap on None, arithmetic overflow in debug builds), infinite loops, huge allocations, and, in code with unsafe, memory errors that sanitizers report.

Here is a deliberately tiny mutation fuzzer. The target parses text like (12,34). It contains an easy-to-miss bug: it assumes the closing parenthesis exists and slices up to the last byte.

Rust A mutation fuzzer finds a panic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with io::stdin().read_line()), it gets the next line from this box.

The fuzzer found a crashing input without us guessing it: (1,, where the comma is the last byte, so the slice start moves past its end. Note how it got there: an input is added to the corpus whenever it produces an outcome not seen before, so (1,2 (a new error) became a stepping stone to (1,. That is a crude version of the coverage feedback described above. The fixed parser uses strip_prefix/strip_suffix and returns an Err for every bad input, so no input ever panics. This "never panic on any input" property is the most common fuzzing goal.

cargo-fuzz in a real project

The real tool wraps libFuzzer. Setup and a typical target (read-only, needs a nightly toolchain and a Cargo project):

Terminal
cargo install cargo-fuzz
cargo +nightly fuzz init                 # creates fuzz/ with a sample target
cargo +nightly fuzz add parse_pair       # new target: fuzz/fuzz_targets/parse_pair.rs
cargo +nightly fuzz run parse_pair       # runs until it finds a crash or you stop it
cargo +nightly fuzz run parse_pair -- -max_total_time=60   # stop after a minute
cargo +nightly fuzz cmin parse_pair      # minimize the corpus
Rust
// fuzz/fuzz_targets/parse_pair.rs
#![no_main]
use libfuzzer_sys::fuzz_target;

fuzz_target!(|data: &[u8]| {
    // The only rule: do not panic, hang or corrupt memory.
    let _ = my_crate::parse_pair(data);
});

When it finds a crash it saves the input to fuzz/artifacts/parse_pair/ and prints the path; re-run it with cargo +nightly fuzz run parse_pair <path> under a debugger, shrink it with cargo fuzz tmin, and add it as a regression test. For structured inputs the arbitrary crate derives Arbitrary so the fuzzer produces valid structs and enums instead of raw bytes. Run fuzzing together with the sanitizers and Miri described in Testing Unsafe Code with Miri and Sanitizers.

Practical advice

  • Start small. One round-trip property per serializer and one fuzz target per parser find a surprising number of bugs.
  • Keep generators realistic. Too many invalid inputs and the interesting code never runs; too narrow and you miss edge cases. Bias toward small numbers, empty collections and boundary values (0, 1, MAX).
  • Keep tests deterministic. Record the seed or let the framework persist failures, so a red build can be reproduced.
  • Fuzz in CI on a time budget, and nightly for long runs. Save the corpus between runs.
  • Mind the cost of slow properties. A property that takes a second per case will run 100 times; keep each case fast.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main benefit of shrinking in a property-testing framework?

    A failing random input is usually large and noisy. Shrinking repeatedly tries smaller inputs that still fail, ending with a minimal counterexample.
  2. 2. Which of these is a round-trip property?

    A round trip converts a value to another form and back and expects the original. The others are idempotence, an invariant and a no-crash property.
  3. 3. What makes a coverage-guided fuzzer better than throwing purely random bytes at a parser?

    Feedback about which code each input covers lets the fuzzer build on inputs that got further, instead of starting from nothing each time.
  4. 4. A fuzzer found a panicking input for your parser. What should you do next?

    Fix the cause, then keep the minimized crashing input as a permanent test so the bug can never return unnoticed.

Summary

  • Property-based tests assert rules that hold for all inputs, run them on many generated inputs, and shrink failures to a minimal counterexample.
  • Good property families: round trip, idempotence, invariants, oracle comparison, no-crash and metamorphic relations.
  • proptest (and quickcheck) provide strategies, shrinking and failure persistence; a seeded generator keeps runs reproducible.
  • Fuzzing mutates inputs, ideally with coverage feedback, to find panics, hangs and memory errors in code that handles untrusted data; cargo-fuzz wraps libFuzzer.
  • Turn every found failure into a regression test, and combine fuzzing with sanitizers and Miri for unsafe code.

Next: Benchmarking Rust Code