Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 37 ยท Projects and Practice

Project: Build a Contract Test Suite

Build a contract test suite from an OpenAPI document: example-based tests, generated boundary cases and response checks against a stand-in API, then learn how to run it for real.

In this project you assemble a complete contract test suite from one OpenAPI document. It has three layers: tests built from the examples in the spec (the Dredd idea), tests built from generated boundary inputs (the Schemathesis idea) and shape checks on every response. You will run all three against a small stand-in API that has a few planted bugs, and see the suite catch them.

Real contract-testing tools need a live server and cannot run on this site. The code here is Python and runs in your browser; the final section shows the real commands to run against a real API.

The plan

flowchart TB
  S["OpenAPI document"] --> E["Layer 1<br/>example tests"]
  S --> B["Layer 2<br/>boundary tests"]
  S --> R["Layer 3<br/>response shape checks"]
  E --> T["API under test"]
  B --> T
  T --> R
  R --> X["Report"]

Layers 1 and 2 create requests; layer 3 judges every response.

The contract and the API under test

The contract describes GET /books/{bookId} and GET /books?limit=. The "server" is an ordinary Python function. Two bugs are planted: the limit upper bound is off by one, and GET /books/{bookId} returns an id as a number for one book instead of a string. Your suite should find both, without tests written for those specific bugs.

Python Stand-in: a three-layer contract test suite

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Layer 1 alone would have missed both bugs: the examples only touch b1 and a missing ID. The boundary layer found the off-by-one and visiting every known resource found the data-dependent type bug. This is why the layers complement each other.

Turn it into a regression pack

A suite is most useful when failures are reproducible and trackable. Add to your stand-in:

  • A short reproduction line (method, URL, query) printed with every failure.
  • An allow-list of known, ticketed failures with expiry dates, so the suite can be green while you fix them.
  • A summary by operation, to see which parts of the API are weakest.
Python Stand-in: allow-list of known failures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The build now fails only for new problems, and the debt is visible with owners and dates.

Running it for real

With a live API and its OpenAPI document you would run, for example:

Terminal
# generated, property-based tests with Schemathesis
pip install schemathesis
schemathesis run openapi.yaml --url http://127.0.0.1:8000

# or run the examples in the spec with Dredd
npm install --save-dev dredd
npx dredd openapi.yaml http://127.0.0.1:8000

In CI, start the API with a seeded test database, run the tool, and keep the reports as artifacts. Combine this with the quality gate so one pipeline checks the document and the running service together.

Stretch goals

  • Add a layer that sends invalid input (wrong types, missing required fields) and expects a 4xx that matches the error schema.
  • Test authentication: call a protected operation with no token, a bad token and a good one.
  • Add a check that every status code declared in the spec is exercised at least once (response coverage).
  • Compare the suite's findings with what request/response validators report.

Check your understanding

Quick check ยท 3 questions

of 3 answered You scored / 3 ยท perfect!

  1. 1. Why did the boundary layer find a bug the example layer missed?

    Typical examples rarely hit limits, so bugs at minimum and maximum values stay hidden.
  2. 2. What is the purpose of an allow-list of known failures?

    With tickets and expiry dates the debt stays visible, and new regressions still fail the build.
  3. 3. Which layer would catch a response whose id is a number instead of a string?

    Shape checks compare each response body with the schema for its status code.

Summary

  • A contract suite combines example-based tests, generated boundary tests and response shape checks.
  • Each layer finds different bugs, so use all three.
  • Reproduction lines and an allow-list with tickets make the suite practical.
  • Real tools (Schemathesis, Dredd) run the same ideas against a live API in CI.
  • The Python suite is a stand-in for those tools.

Next: Practice: review a flawed OpenAPI document