Skip to content
Tutorials Center
My learning

Topics

Lesson 16 of 30 ยท Running at Scale

The Collection Runner and data-driven testing

Run a whole Postman collection at once, repeat it with CSV or JSON data files, read the results, and learn data-driven testing with a Python stand-in.

Sending requests by hand, one at a time, is fine for exploring. A test suite needs the opposite: press one button and watch every request run, in order, with every assertion checked. In Postman that button lives in the Collection Runner. It takes a collection (or a folder), sends its requests one after another, runs each request's scripts and tests, and shows you a report.

The runner also does something a human would find boring: it can repeat the whole run many times, each time with a different row of data. That is data-driven testing. You write one request and one set of assertions, then feed in fifty combinations of input and expected output from a spreadsheet. The same test now covers fifty cases.

This lesson explains iterations, data files, data variables, delays, run order and results. Postman cannot run here, so the settings and scripts are described and shown read-only, while the runnable examples are Python stand-ins: a fake API function and a small loop that does what the runner does with a data file. They are honest models of the idea, not the real runner.

Opening the Collection Runner

You can start a run from the collection's context menu or from the Runner in the interface (the exact menu names change between Postman versions, so look for Run). You choose a collection or folder, set the options, and press the run button. The runner then:

  1. sends each request in order,
  2. executes the pre-request script, sends the request, then executes the test script,
  3. records each test as passed or failed,
  4. repeats everything for each iteration, then shows a summary.

The options you will use most are these.

The main run settings

Iterations How many times to run the collection.
Delay Milliseconds to wait between requests.
Data file A CSV or JSON file with one row per iteration.
Request list Choose which requests run and in what order.

Environment and variable values are used as they are at the start of the run.

1 / 2

Iterations is the simplest knob. Setting it to 3 with no data file runs the same collection three times. That is useful for catching flaky behavior (does it fail one time in three?), but with identical input it rarely teaches more than one run does. The real power comes from a data file.

Delay adds a pause between requests. Use it when the server enforces rate limits or when you want the run to look less like an attack. It makes the run longer, so keep it at zero when speed matters.

Data files: CSV and JSON

A data file gives the runner one row of values per iteration. If the file has six rows, the collection runs six times, once per row. Postman accepts two formats.

CSV. The first line holds the column names; each later line is one iteration.

TEXT
email,age,expected_status
[email protected],36,201
[email protected],17,422

JSON. An array of objects; each object is one iteration.

JSON
[
  { "email": "[email protected]", "age": 36, "expected_status": 201 },
  { "email": "[email protected]", "age": 17, "expected_status": 422 }
]

The two formats are similar, with one important difference: in JSON, numbers stay numbers and true stays a boolean. In CSV, everything is text. The value 36 arrives as the string "36". This trips people up when they compare a response status code (a number) with a value read from a CSV file, so convert it in the script, as shown below.

Using data variables

The column names become data variables. You use them like any other variable, with double curly braces, in the URL, headers or body of a request:

JSON
{
  "email": "{{email}}",
  "age": {{age}}
}

In scripts you read them through pm.iterationData (real Postman script, read-only here):

JavaScript
// Tests tab (real Postman script, read-only here)
const expectedStatus = Number(pm.iterationData.get("expected_status"));

pm.test("status matches the row's expectation", function () {
  pm.response.to.have.status(expectedStatus);
});

console.log("Iteration " + (pm.info.iteration + 1) + " of " + pm.info.iterationCount);

Two details deserve attention.

  • pm.info.iteration counts from zero, so add one when you print a human-friendly number. pm.info.iterationCount is the total number of iterations.
  • Data variables have the local scope for that iteration, and local-style values win over environment, collection and global variables with the same name. If your environment has a variable called email and your data file has a column called email, the data file's value is used. This is handy for overriding defaults and a surprise when you forget it.

Put the expected result in the data, next to the input. A row such as [email protected],17,422 documents the case, drives the request and gives the assertion its target, all in one line. Reviewers can read the file like a table of examples.

The first stand-in: a CSV-driven loop

The runner's loop is simple enough to imitate in a few lines. The "API" below is a Python function standing in for a signup endpoint, and the CSV text is parsed with Python's csv module, so the header row becomes the keys, just like Postman's data variables. Each row is one iteration: send, compare with the expected status, print a result. Values stay as text, exactly like a real CSV file, so the script converts the expected status with int(...).

Python A CSV-driven run against a fake signup API

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All six iterations pass, and the last line proves the pitfall: even age is a string. In a real collection you would write Number(pm.iterationData.get("age")) whenever a number is needed.

Notice the boundary rows. Ages 17 and 18 or 120 and 121 sit exactly on the edge of the rule. A data file makes these edge cases cheap to add: one more line, no new request. Choosing good rows is the topic of API testing strategy and test design.

Order, folders and what runs

The runner executes requests in the order they appear in the collection, folder by folder. You can run a single folder instead of the whole collection, and in the runner you can choose which requests take part and change their order for that run. Use this to skip slow or destructive requests, or to run a smoke test of just the important ones.

Two good habits keep runs predictable:

  • Order requests deliberately. If "Create" must run before "Delete", the collection order should say so. The chaining lesson shows how to pass ids between them and how pm.execution.setNextRequest can change the order from a script.
  • Keep each iteration independent. With a data file, the whole sequence of requests repeats per row. If iteration 3 depends on leftovers from iteration 2, one failure will cascade. Create what you need inside the iteration and delete it at the end.

Reading the results

When the run finishes you see the totals and, for each iteration, every request with its tests marked passed or failed. Failed tests show the assertion message (for example, expected 422 to equal 201), and you can open a request to see the exact request and response. You can usually export the results as a JSON file, which is useful to attach to a bug report or to compare with an earlier run.

A few things to look at besides red and green:

  • Which iterations failed? If only the rows with abc in a column fail, the bug lives in that input handling, not in the endpoint as a whole.
  • How long did it take? The summary shows the total time and per-request time, which is a quick performance smell test.
  • Did any request return an unexpected status but have no test? A request without assertions cannot fail. A green run with zero tests proves nothing.

The second stand-in: JSON data, scopes and a delay

JSON data files keep their types, and data values beat environment values with the same name. The following example models both, plus the runner's delay. To keep the output identical on every run, the stand-in does not really sleep: it adds up a simulated clock, with a fixed response time per request and the delay between requests, so you can see how a delay stretches a run.

Python JSON rows, variable precedence and a simulated delay

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first row names a currency, so the data file wins over the environment; the other rows have none and fall back to EUR. With four requests of 90 ms and three delays of 250 ms, the simulated run lasts 1,110 ms. Doubling the delay would add three more pauses of the same size, which is why delays should be used with care. Unlike CSV, the JSON qty stays an integer.

What is real and what is stand-in: the file formats, the data-variable idea, the zero-based iteration number and the precedence of data over environment values describe Postman honestly. The Python loop, the fake API and the simulated clock are written for this lesson. The real runner talks to a real server over HTTP and measures real time.

When to use the runner, and when not to

The Collection Runner is excellent for regression checks on your own machine, for trying a new data set, and for demos. It has limits:

  • It runs on your computer while the Postman window is open. For automatic runs on every code change you want Newman and CI/CD.
  • It is not a load-testing tool. Running many iterations one after another is a slow sequence of requests, not many users at once. See Monitors, performance and load basics.
  • Large data files are fine for dozens or hundreds of rows. If you need to generate thousands of combinations, consider generating the file with a script first.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A CSV data file has a header row and eight data rows, and iterations is left to match the file. How many times does the collection run?

    The header row names the variables; each data row is one iteration, so eight rows mean eight runs.
  2. 2. A CSV column holds 201 and a test compares pm.response.code with pm.iterationData.get('expected_status') directly. Why might it fail?

    Everything read from a CSV file is a string. Convert it with Number(...) before comparing it with a numeric status code.
  3. 3. An environment variable and a data-file column are both called email. Which value does a request use during the run?

    Data variables are local to the iteration and take precedence over environment, collection and global variables.
  4. 4. pm.info.iteration returns 0 during the first iteration. What should you do to show 'Iteration 1'?

    The iteration index is zero-based, so add one when printing a number for people.

Summary

  • The Collection Runner sends a collection's requests in order, runs their scripts and tests, and reports the results.
  • Iterations repeat a run; a data file (CSV or JSON) supplies one row of values per iteration.
  • Column names become data variables, used as {{name}} in requests and read with pm.iterationData.get("name") in scripts.
  • CSV values are always text; JSON keeps numbers and booleans. Convert CSV values before numeric comparisons.
  • Data variables beat environment, collection and global variables of the same name; pm.info.iteration starts at zero.
  • Put expected results in the data file and include boundary rows.
  • A delay slows the run on purpose; the runner is for local regression runs, not for load or unattended automation.
  • The Python examples are stand-ins for the idea, not the real runner.

Next: Newman, the CLI and CI/CD.