Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 37 ยท Validation and Contract Testing

Contract Testing with Schemathesis, Dredd and Pact

Learn contract testing for OpenAPI: property-based testing with Schemathesis, example-based tests with Dredd, consumer-driven contracts with Pact, plus a runnable fuzz tester.

A contract test checks that a real API behaves as its OpenAPI document says. Linting checks the document, validation checks single messages, and contract testing sends many requests to a running server and compares every response with the contract. It finds the bugs that appear when a server and its spec drift apart: an undocumented 500, a field that is sometimes null, a status code that is missing from the spec.

The tools in this lesson need a running API and cannot be used on this site. Their commands are read-only; a Python stand-in shows how automatic input generation finds bugs.

Two ways to generate tests from a spec

Approach Tool How tests come about
Property-based / fuzz Schemathesis Generates many valid and boundary inputs from schemas and checks general properties
Example-based Dredd Uses the examples in the spec as concrete request/response pairs
Consumer-driven Pact Each consumer records what it needs; the provider verifies those expectations

Schemathesis

Schemathesis is a Python tool built on the Hypothesis property-testing library. It reads the spec (or a URL), generates requests, sends them and applies checks:

Terminal
pip install schemathesis

# test a running API against its published spec
schemathesis run https://api.example.com/openapi.json

# local file with a base URL
schemathesis run openapi.yaml --url http://127.0.0.1:8000

Typical checks (names may differ by version):

  • Not a server error: no response should be 5xx; an invalid input should produce 4xx.
  • Status code conformance: the status is one that the spec lists.
  • Content type conformance and response schema conformance: the body matches the declared schema.
  • Response headers conformance: declared headers are present.

It shrinks failures to minimal examples and prints a curl command to reproduce them. Because inputs include extremes (empty strings, huge numbers, unicode), it finds crashes that hand-written tests never try. You can also use it as a library inside your pytest suite.

Dredd

Dredd takes a more literal approach: for each operation it builds the request from examples in the spec, calls the server and compares the response (status, headers, body) with the example response.

Terminal
npm install --save-dev dredd
npx dredd openapi.yaml http://127.0.0.1:3000

Because tests rely on examples, you need hooks (small scripts in JavaScript, Python and other languages) to prepare data, for example to create the book that GET /books/b1 expects. Dredd is simple to understand but is less actively developed than the alternatives; check its status before adopting it.

Pact and bi-directional contracts

Pact reverses the direction. A consumer (say a mobile app) writes tests that say, "when I call GET /books/1, I need a response with id and title." Those expectations are saved as a pact file; the provider then replays them against its real code. OpenAPI says what the API offers; Pact says what each consumer actually uses. Pact's bi-directional contract testing can compare a provider's OpenAPI document with consumer pacts, so the two approaches combine.

A property-based tester in miniature

The stand-in creates inputs from a schema (valid values, boundaries and bad values), calls a deliberately buggy fake API, and reports where it breaks the contract. The inputs are fixed lists so that the output is identical on every run.

Python Stand-in: generate inputs and check the contract

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The tester finds both bugs without anyone writing a test for them: one is a crash at a boundary, the other a type mismatch against the schema.

Running contract tests in CI

  1. Start the API with a test database in CI.
  2. Run Schemathesis (or Dredd) against it with the spec from the same commit.
  3. Fail the build on any finding, and save the reproduction commands as artifacts.
  4. Keep a short list of known and accepted differences, each with a ticket, rather than turning checks off.

Start with Schemathesis in "not a server error" mode only. It is almost all signal: any 5xx under valid input is a bug. Then add conformance checks as your spec matures.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. What makes Schemathesis find bugs that hand-written tests miss?

    Property-based generation explores edge cases systematically and shrinks failures to small examples.
  2. 2. Why does Dredd usually need hooks?

    Example-based tests assume that data exists, so hooks create or clean it before and after each transaction.
  3. 3. Which approach lets each consumer state what it needs from the provider?

    Pact records consumer expectations as pact files that the provider verifies.

Summary

  • Contract tests run against a live API and compare real behavior with the spec.
  • Schemathesis generates inputs from schemas and checks status codes, schemas and server errors; Dredd replays spec examples; Pact verifies consumer expectations.
  • Start with "no 5xx" checks, then add conformance checks.
  • Run them in CI from the same commit as the spec.
  • The Python tester is a stand-in showing why generated inputs find boundary bugs.

Next: Design-first versus code-first