Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 30 ยท Advanced API Testing

API testing strategy and test design

Design better API tests: positive, negative and boundary cases, a status-code matrix, validation errors, idempotency, pagination, flaky tests and what to skip.

Knowing how to write a test is one skill. Knowing which tests to write is another, and it is the one that separates a suite that catches real bugs from a suite that is just a large pile of green checkmarks. You cannot test every possible request. Time is limited, and every test you write must also be kept up to date. A strategy is how you spend that effort where bugs are most likely and most expensive.

This lesson is about test design. You will sort cases into categories, build a status-code matrix, learn the boundary-value habit, check validation errors, test idempotency and pagination, tame flaky tests, and decide what is not worth testing. The ideas apply in Postman and in any other tool.

The runnable examples are stand-ins: small JavaScript functions pretending to be an API, so you can practice generating cases and spotting bugs without a server. Where Postman scripts appear, they are read-only snippets.

Start from risk, not from the endpoint list

A common mistake is to write one happy-path test per endpoint and call the job done. A better question is: what could go wrong here, and how bad would it be? An endpoint that moves money deserves many more tests than one that returns the opening hours. Rank your endpoints by impact (money, data loss, security, many users) and by change (code that changes often breaks often), then spend your tests accordingly.

Next, for each important endpoint, think in five categories.

Five kinds of test case

Positive Valid input gives the right result.
Negative Bad input or no access gives a clear error.
Boundary The edges of every rule.
Sequence Create, change, delete, repeat.

A fifth kind, non-functional (speed, security), has its own lessons.

1 / 2

A positive test proves the feature works when used correctly. A negative test proves it fails correctly when used wrongly, with the right status and a helpful message, and without side effects. A boundary test probes the edges of every rule, where programmers most often make off-by-one mistakes. A sequence test follows a resource through its life (see Chaining requests and workflows). Each category finds different bugs.

The status-code matrix

For one endpoint, list the situations and the status code each should produce. This small table is one of the most useful design tools you have. Here it is for POST /orders:

Situation Expected status
Valid body, valid token 201 Created (and a Location header or the new id in the body)
Body is not valid JSON 400 Bad Request
Required field missing or wrong type 400 or 422, depending on the API's documented convention
No token or an invalid token 401 Unauthorized
Valid token, but the user may not do this 403 Forbidden
Referenced item (such as a product id) does not exist 404 Not Found or 422, by convention
Duplicate of something that must be unique 409 Conflict
Wrong Content-Type (for example, plain text) 415 Unsupported Media Type
Too many requests in a short time 429 Too Many Requests

The exact code for a validation failure (400 or 422) differs between APIs, and neither is "wrong". What matters is that your API's documentation or contract says which one, and that the API is consistent. A test pins down the promise. When the promise is unclear, that is a finding: raise it with the team.

Each row of the matrix becomes one or more test requests. In Postman, a small data file (see The Collection Runner and data-driven testing) can drive one request through all the rows, with the expected status in a column.

Boundary values

Bugs cluster at edges. If a quantity must be between 1 and 100, the interesting values are not 50 or 51. They are 0, 1, 2, 99, 100 and 101. This is called boundary value analysis, and it is cheap: for any numeric or length rule, test the minimum, the maximum, one step inside each edge and one step outside each edge.

Add the odd values that real clients send:

  • an empty string and a string of only spaces,
  • null and a missing field,
  • the wrong type (a number as text, text instead of a number, an array instead of an object),
  • a fraction where an integer is required, and a negative number,
  • an extremely long string or a huge number,
  • Unicode characters, such as accents or emoji, in text fields.

You do not need to write these by hand every time. A few lines of code can generate the cases from the rule. The stand-in below does exactly that for two fields (quantity: integer 1 to 100, name: string of 3 to 20 characters). The "API" under test has one deliberate off-by-one bug in its name-length check, and the generated cases find it.

JS Generating boundary cases from rules

Press Run (or Ctrl + Enter) to see the console output here.

No output.

All the quantity cases pass, but the "longest allowed" name (exactly 20 characters) is wrongly rejected with 422. A tester who tried only "lamp" and "x" would never have seen it. Generating the edge cases from the written rule found the bug in a second, and the rule table doubles as documentation.

The API above is a plain function with a planted bug. It stands in for a real service so that you can practice the technique; real boundary rules come from your API's specification.

Check the validation errors, not only the status

When input is rejected, the response body matters as much as the code. A client developer needs to know which field was wrong and why. Good tests check:

  • the status code (400 or 422 as documented),
  • that the body has the agreed error shape (for example errors: [{ field, message }]),
  • that the right field is named, and that several problems are reported together when several fields are wrong,
  • that the message does not leak internals (stack traces, SQL, file paths),
  • that nothing was saved: a follow-up read must not find the rejected item.

In Postman that last check is a second request in a chain. A rejected creation that still wrote half a record is a classic bug that a status check alone never notices.

Idempotency and repeated requests

An operation is idempotent when doing it once or many times leaves the system in the same state. GET, PUT and DELETE are meant to be idempotent; POST normally is not (two posts create two orders). Networks fail and clients retry, so this matters for money and inventory. Useful tests:

  • send the same PUT twice: the second response and the stored data are the same as the first,
  • send the same DELETE twice: typically 204 then 404 (or 204 both times, if documented),
  • send a creation request twice with the same idempotency key (a unique id in a header that many payment-style APIs support): the second call must return the original result and must not create a second record.

Pagination

List endpoints split their results into pages. The typical bugs are not in the first page: they are duplicates and gaps between pages, a broken last page, and a page size that is ignored. Test the invariants:

  • the first page has the requested size,
  • walking all pages returns every item exactly once (no duplicates, no gaps),
  • the last page is shorter or empty, and asking beyond it gives the documented result (an empty list or 404),
  • the sort order is stable between calls, and a limit larger than the allowed maximum is clamped or rejected, as documented.

The next stand-in builds a small in-memory API twice: once with a planted pagination bug and once correct. It then runs the same checks (pagination invariants and idempotency) against both, so you can see a test suite distinguishing a broken API from a good one.

JS Pagination and idempotency checks on a buggy and a fixed API

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The buggy API repeats an item at the start of each later page, so the "no duplicates" and size checks fail while the idempotency checks still pass. The same suite is green against the fixed version. Notice how these tests describe rules that must always hold (every item exactly once) rather than one fixed example; such invariants find bugs that a single hard-coded expectation misses.

Contract checks

Besides values, check the shape of the response: required fields exist, types are right, no unexpected fields appear (a password hash in a user object, for example). A schema check in Postman (tv4 or Ajv-style validation inside pm.test) is stronger than ten separate field checks, and it protects every consumer of the API. The topic has its own lesson: Contract testing and OpenAPI.

Flaky tests

A flaky test sometimes passes and sometimes fails with no change in the code. Flaky tests are poison: people stop trusting red builds, rerun until green, and eventually miss a real failure. The usual causes and cures:

  • Shared data. Two tests (or two people) use the same record. Give each run its own data with unique names and clean up after yourself.
  • Order dependence. A test only passes after another one ran. Make each test (or each chain) create what it needs.
  • Time and randomness. Comparing against the current time or random values. Assert on properties (format, range) instead of exact values.
  • Slow or unreliable dependencies. A third-party service in the loop. Mock it for most tests and check it separately.
  • Rate limits and cold starts. Add a sensible delay or warm-up, but do not hide real problems with blind retries.
  • Too-tight time limits. A 50 ms assertion fails on a busy CI machine. Use percentiles and generous budgets (see Monitors, performance and load basics).

A test that cannot be fixed quickly should be quarantined (clearly marked and excluded) and tracked, never left to fail randomly.

What not to test

More tests are not always better. Skip or reduce:

  • The framework or the library. You do not need to prove that your web framework parses JSON.
  • Third-party APIs in depth. Test that you handle their success and failure responses; do not test their internals.
  • The same thing at every layer. If a unit test already covers a calculation exhaustively, the API test needs one check that the endpoint is wired to it.
  • Exact generated values. Auto-generated ids and timestamps change on every run. Check their format instead.
  • Every combination. Pick representative cases from each equivalence class (all "valid" values behave alike) and the boundaries between classes.
  • Implementation details. Test behavior a client can see, so internal refactoring does not break your suite.

Writing readable tests

  • Name tests as sentences about behavior: "returns 422 and names the field when quantity is zero".
  • Keep each test to one reason to fail, so the report points at the cause.
  • Arrange, act, assert: set up the data, send the request, check the result. In that order.
  • Keep tests independent where you can; where a chain is needed, make it explicit.
  • Review the suite regularly and delete tests that no longer protect anything.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A quantity field accepts integers from 1 to 100. Which set of values is the best boundary-value choice?

    Boundary analysis tests the edges and one step on each side of them, where off-by-one bugs live.
  2. 2. An authenticated user tries to delete another user's resource and the API should refuse. Which status fits best?

    The caller is authenticated but not allowed to do this, which is what 403 Forbidden means. 401 is for missing or invalid credentials.
  3. 3. A test passes locally but fails about one run in ten on CI, and both runs use the same shared test record. What is the most likely class of problem?

    Random failures with shared data usually mean tests interfere with each other. Give each run its own unique data and clean up.
  4. 4. What does it mean that a pagination test checks an invariant?

    An invariant is a property that must be true for any data, which finds duplicates and gaps that a single fixed example would miss.

Summary

  • Spend test effort by risk and impact, not by endpoint count.
  • Cover positive, negative, boundary and sequence cases for each important endpoint.
  • Build a status-code matrix (201, 400/422, 401, 403, 404, 409, 415, 429) and test it against the API's documented conventions.
  • Generate boundary cases from the rules: minimum, maximum, one step inside and outside, empty, null, wrong type.
  • Check validation error bodies and that rejected requests save nothing.
  • Test idempotency (repeat PUT, DELETE and keyed POST) and pagination invariants (no duplicates, no gaps, a correct last page).
  • Fix flaky tests at the cause; skip what is already covered or not yours to test.
  • The runnable examples are stand-ins with planted bugs, not a real service.

Next: Security testing basics for APIs.