Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 30 ยท Advanced API Testing

Contract testing and OpenAPI

Learn how OpenAPI and JSON Schema describe an API contract, how to validate responses in Postman, spot breaking changes and use consumer-driven contracts.

An API is a promise. The server promises that GET /books/1 returns an object with an id that is a number and a title that is a string, and every client is written on the strength of that promise. A contract is that promise written down in a form a machine can check. Contract testing is the practice of verifying, automatically, that the real API still keeps it.

Ordinary tests check values ("the title is Dune"). Contract tests check shape: which fields exist, what types they have, which are required, which values are allowed. Shape checks are cheap, they work on any data, and they catch the most expensive class of API bugs: a change that looks harmless on the server and silently breaks a client.

This lesson covers the three pieces of that story: OpenAPI as the contract document, JSON Schema as the checking language, and consumer-driven contracts as a way to let clients state what they really need. Postman, OpenAPI tooling and Pact-style brokers cannot run inside this page, so their commands and files are read-only. The runnable examples are stand-ins: a small schema validator and a breaking-change detector written in plain JavaScript.

OpenAPI: the contract as a document

OpenAPI (formerly called Swagger) is a standard format, written in YAML or JSON, for describing an HTTP API: its paths, methods, parameters, request bodies, responses and the schemas of the data. Because it is structured, tools can read it to generate documentation, client code, mock servers and tests.

A tiny OpenAPI 3 description of one endpoint looks like this. It is read-only here:

YAML
openapi: 3.0.3
info:
  title: Bookstore API
  version: 1.2.0
paths:
  /books/{id}:
    get:
      parameters:
        - name: id
          in: path
          required: true
          schema: { type: integer }
      responses:
        "200":
          description: One book
          content:
            application/json:
              schema:
                $ref: "#/components/schemas/Book"
        "404":
          description: No such book
components:
  schemas:
    Book:
      type: object
      required: [id, title]
      properties:
        id: { type: integer }
        title: { type: string }
        year: { type: integer, minimum: 0 }

The Book schema inside components is the heart of it. It is written in a dialect of JSON Schema, the vocabulary used to describe the shape of JSON data. Postman can import an OpenAPI file and turn it into a collection with one request per operation, which is a quick way to start testing an API that already has a description. Exact menu names change between versions, so look for the import button in your workspace.

Three layers of a contract

OpenAPI The whole API in one document.
JSON Schema The shape of each body.
Validation A test that checks reality against it.

The document is only useful if something checks the server against it.

1 / 2

Validating a response against a schema

In a Postman test you can check the whole response body against a schema in one line. The assertion below is real Postman scripting and is read-only here, because it needs a response from a real request:

JavaScript
const schema = {
  type: "object",
  required: ["id", "title"],
  properties: {
    id: { type: "integer" },
    title: { type: "string", minLength: 1 },
    year: { type: "integer", minimum: 0 },
  },
};

pm.test("Body matches the Book schema", () => {
  pm.response.to.have.jsonSchema(schema);
});

When the body does not match, the test fails and lists which rule was broken. Postman checks the body with a real schema validation library, so the full JSON Schema vocabulary is available to you. You can keep a schema in a collection variable, so a dozen requests share one definition of Book.

To see what such a validator does under the hood, here is a small stand-in. It supports a handful of the most common keywords: type, enum, minimum, maximum, minLength, pattern, required, properties, additionalProperties and items. A real library supports many more, but the idea is the same: walk the data and the schema together and collect every mismatch.

JS A miniature JSON Schema validator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Read the extraField line closely. With additionalProperties: false the schema rejects a leaked passwordHash, which makes schema validation a cheap way to catch accidental data exposure. The trade-off is strictness: a strict schema also fails when the server adds a harmless new field, so decide deliberately which endpoints deserve it.

A schema describes shape, not truth. A response with { "id": 1, "title": "" } can satisfy a loose schema and still be wrong. Use schema checks together with value checks from Writing tests with pm.test and Chai, not instead of them.

Breaking versus compatible changes

APIs must change. The question is whether existing clients survive the change. A breaking change forces clients to update; a compatible change does not. Direction matters, because a response is something clients read and a request is something clients send.

Changes to a response

Breaking Removing a field, renaming it, or changing its type.
Compatible Adding a new optional field.

Clients read responses, so they depend on what is there today.

1 / 2

Even "compatible" changes can bite. Adding a new value to an enum in a response is compatible by shape, yet a client with a switch statement may crash on a value it has never seen. That is one reason well-written clients ignore unknown fields and handle unknown enum values gracefully.

A contract test suite can find these changes automatically if you compare the old schema with the new one. The stand-in below does that for the properties of an object schema.

JS Detecting breaking changes between two schema versions

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Real OpenAPI diff tools do the same thing across every path, method, parameter and status code. Running one in your pipeline turns "did we break anyone?" from a hope into a build result.

Consumer-driven contracts

A provider-side schema describes everything the server can return. But a given client usually uses only a few fields. In consumer-driven contract testing, each consumer writes down exactly what it relies on, and the provider's build must prove it still satisfies every consumer. A mobile app that reads only id and title records that fact; the provider may then freely rename year without worrying about that app.

Tools such as Pact implement this with contract files that consumers publish and providers verify, often through a shared broker. Those tools are real and are not runnable here. The idea, though, fits in a few lines:

JS A consumer states the fields it depends on

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The second run breaks only the reporting job, and the third breaks both. With real consumer-driven tooling, the provider team would see which consumer a change breaks before merging it. That is the whole value.

Where each kind of check belongs

Check Question it answers Typical place
Schema validation in Postman Does this response have the promised shape? Collection tests, run in CI
OpenAPI diff Did the contract itself change in a breaking way? Pull request on the API repository
Consumer-driven contract Does the provider still satisfy each real client? Provider's build, per consumer

For a modest team, the best first step is the cheapest: keep an OpenAPI file in the repository, validate key responses against its schemas in your collection, and run the collection on every change. The strategy lesson, API testing strategy and test design, explains how contract checks fit among all other kinds of tests.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a contract test mainly verify?

    Contract tests check structure and rules (required fields, types, allowed values) rather than specific data values.
  2. 2. Which change to a RESPONSE is breaking for existing clients?

    Clients depend on the fields they read. Removing one (or renaming it, or changing its type) breaks them, whereas an extra optional field is ignored.
  3. 3. Which change to a REQUEST is breaking?

    Existing clients do not send the new required field, so their requests start to fail. Relaxing a rule never breaks callers.
  4. 4. What is the idea behind consumer-driven contracts?

    Consumers publish their expectations, and the provider verifies them, so a change that would break one consumer is found before release.

Summary

  • A contract is the API's promise in a machine-checkable form; OpenAPI describes the API and JSON Schema describes each body.
  • In Postman, pm.response.to.have.jsonSchema(schema) checks a whole response body against a schema in one assertion.
  • Strict schemas (additionalProperties: false) also catch leaked fields, at the cost of failing on harmless additions.
  • Schema checks verify shape, not correctness; combine them with value assertions.
  • Removing, renaming or retyping a response field is breaking; adding a required request field is breaking; adding optional response fields is compatible.
  • Consumer-driven contracts let each client declare what it uses, so providers learn who a change would break.
  • The runnable examples here are stand-ins: a hand-written validator and diff tool, not Postman, Ajv or Pact themselves.

Next: Testing GraphQL, WebSocket and gRPC.