Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 30 ยท Projects and Practice

Project: automated regression in CI

Put a Postman collection in a repository, run it with Newman in a GitHub Actions pipeline, publish JUnit reports, fail builds on regressions and schedule nightly runs.

A test suite that someone has to remember to run is a test suite that stops being run. The final step in turning a Postman collection into a real safety net is continuous integration (CI): a server that runs your tests automatically on every change, shows a green or red result, and blocks bad changes before they reach users. A second trigger, a schedule, runs the same tests every night to catch problems that no code change caused, such as an expired certificate or a broken third-party service.

In this project you will plan the repository layout, write the pipeline configuration, choose reports, and design the rules for failing a build. The pipeline files and the Newman commands are read-only, because there is no CI server, no Node toolchain and no network on this page. The runnable examples are stand-ins: a small collection runner that produces a JUnit report and an exit code, and a cron-expression matcher that shows how schedules work.

If Newman is new to you, read Newman, the CLI and CI/CD first. This lesson assembles those ideas into one working pipeline.

Step 1: put the tests in the repository

For CI to run your collection, the collection must be available to the pipeline, which almost always means files in version control next to the code they test. Export the collection (as a JSON file in Postman's collection format) and the environment, and commit them:

TEXT
my-api/
  src/                     (the application itself)
  postman/
    shop-api.collection.json
    ci.environment.json
    data/
      products.csv
  .github/
    workflows/
      api-tests.yml
  reports/                 (created by the pipeline, ignored by git)

Several habits keep this manageable:

  • One source of truth. Decide whether the repository or the Postman workspace is the master. Editing in both leads to conflicts (see Postman Flows, collaboration and AI features).
  • No secrets in the files. The committed environment holds only non-secret values, such as the base URL. Tokens come from the CI system's secret store and are passed in at run time.
  • Reviewable diffs. Collection JSON is long, but it is text, so changes show up in a pull request, where a reviewer can see that a test was removed.

Step 2: run it with Newman

Newman is Postman's command-line collection runner. It needs Node.js. These are real commands, shown read-only:

Terminal
npm install --global newman

newman run postman/shop-api.collection.json \
  --environment postman/ci.environment.json \
  --env-var "token=$API_TOKEN" \
  --iteration-data postman/data/products.csv \
  --reporters cli,junit \
  --reporter-junit-export reports/newman.xml \
  --timeout-request 10000 \
  --bail

What each option does:

Option Purpose
--environment Loads the environment file (base URL and similar)
--env-var "token=..." Sets one variable on the command line, ideal for a secret from the CI store
--iteration-data Runs the collection once per row of a CSV or JSON data file
--reporters Chooses output formats: cli for the log, junit for machine-readable results
--reporter-junit-export Where to write the JUnit XML file
--timeout-request Gives up on a single request after this many milliseconds
--bail Stops at the first failure (optional; useful for quick feedback)

The most important behavior is the exit code. Newman exits with 0 when every test passed and with a non-zero code when any test failed or the run could not complete. CI systems treat a non-zero exit code as a failed step, and that is the entire mechanism by which a broken API turns the build red.

From commit to verdict

Push A change reaches the repository.
Start app The pipeline brings up the API.
Run Newman The collection runs against it.
Verdict Exit code 0 is green, anything else is red.

The exit code is the whole contract between Newman and CI.

1 / 2

Step 3: the pipeline

Here is a complete GitHub Actions workflow (read-only). It runs on every pull request and push to main, and also on a nightly schedule:

YAML
name: API regression

on:
  pull_request:
  push:
    branches: [main]
  schedule:
    - cron: "30 2 * * 1-5"   # 02:30 UTC, Monday to Friday

jobs:
  api-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - name: Install Newman
        run: npm install --global newman

      - name: Run the collection
        env:
          API_TOKEN: ${{ secrets.API_TOKEN }}
        run: |
          mkdir -p reports
          newman run postman/shop-api.collection.json \
            --environment postman/ci.environment.json \
            --env-var "token=$API_TOKEN" \
            --reporters cli,junit \
            --reporter-junit-export reports/newman.xml

      - name: Keep the report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: newman-report
          path: reports/

Walk through the details, because each one solves a real problem:

  • timeout-minutes stops a hung job from running for hours.
  • The secret is injected through env, not written into the file, and it is referenced as a shell variable. Secrets from the store are masked in logs.
  • if: always() on the upload step means the report is kept even when the test step fails, which is exactly when you want to read it.
  • No continue-on-error. If you add it, a failed test no longer fails the build, and the whole point is lost.
  • The cron line uses UTC and five fields: minute, hour, day of month, month, day of week. Scheduled workflows run from the default branch, and their start time can be delayed when the platform is busy, so do not rely on second-level precision.

On GitLab CI, the same idea is a job in .gitlab-ci.yml that uses a Node image, installs Newman and runs the same command; schedules are configured in the project's pipeline schedules page rather than in the file. Other systems (Jenkins, CircleCI, Azure Pipelines) follow the same recipe: get Node, install Newman, run the command, keep the report, honor the exit code.

Your pipeline needs something to test. If the API is deployed to a shared test environment, point base_url at it. If the pipeline must start the application itself, add a step that starts it and waits until a health endpoint answers before Newman begins, or the first requests will fail for the wrong reason.

What the runner does inside: a stand-in

The runner below is a miniature of Newman's job. It takes a collection (a JSON structure of named requests with expectations), sends each request to an in-process API function, evaluates the expectations, prints a log, writes a JUnit XML report and decides on an exit code. The expectations are simple data here, not Postman scripts, and the API is a function rather than a server, so this is a model of the process and not Newman itself.

It runs the collection twice: first against an API with a planted bug (creation answers 200 instead of 201), then against the fixed version.

Node.js A mini Newman: run a collection, write JUnit XML, set an exit code

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Look at the XML. A CI system that understands JUnit reads exactly this shape: one testsuite, one testcase per test, and a failure element with a message when a test failed. That is how the pipeline's web page can show "Create product failed: expected status 201 but got 200" without anyone opening a log file.

Rules for failing builds

Decide up front what makes a build fail, and write it down:

  1. Any failed assertion fails the build. A "known failure" that stays red teaches everyone to ignore red. Fix the test or remove it deliberately.
  2. A run that cannot complete also fails. If the API is down, Newman reports errors and exits non-zero, which is correct.
  3. Flaky tests are bugs. If a test fails one run in twenty, find the cause (shared data, timing, ordering) instead of adding retries. Retries hide real intermittent defects.
  4. Keep the pull-request run fast. A few minutes at most. Put slower, broader checks (large data files, soak tests) in the nightly schedule.
  5. Make failure visible. For scheduled runs, nobody is waiting for the result, so configure a notification (email, chat message) on failure.

Understanding the schedule

A schedule is a cron expression: five fields for minute, hour, day of month, month and day of week. A * means every value, */15 means every 15th, 1-5 is a range and 1,15 is a list. For day of week, 0 is Sunday and 1-5 is Monday to Friday. In this common form, both GitHub Actions and many other systems read the time as UTC, so 30 2 * * 1-5 is 02:30 UTC on weekdays, not your local 02:30.

The stand-in below implements just enough of cron to answer the question people actually ask: "when will this run next?"

JS When does a cron schedule fire?

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The weekday schedule skips the weekend: after Friday it jumps to Monday. Remember that this is a teaching sketch. It handles the common forms but ignores special cases such as names for months or days, and the rule that real cron treats day-of-month and day-of-week specially when both are set.

Everything marked read-only above is real configuration, but it was written without a live pipeline to run it on. Before relying on it, test it in a scratch repository: option names and action versions evolve, so check them against the current Newman and GitHub Actions documentation.

Extending the pipeline

  • Matrix runs. Run the same collection against several environments (staging, a pre-production copy) as separate jobs.
  • Contract checks. Add schema validation and an OpenAPI diff, as in Contract testing and OpenAPI.
  • Better reports. Newman supports additional reporters distributed as separate packages, such as HTML dashboards. Install them in the pipeline and add their name to --reporters.
  • Performance budgets. Assert response times in tests so a slow endpoint fails the build, the idea behind Monitors, performance and load basics.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What makes a CI pipeline fail when a Newman test fails?

    CI systems judge a step by its exit code. Newman exits with 0 on success and non-zero on test failures or errors.
  2. 2. Where should the API token for the pipeline come from?

    Secrets must stay out of version control. The CI secret store injects them as environment variables when the job runs.
  3. 3. Why use 'if: always()' on the step that uploads the report?

    Steps after a failed step are skipped by default. always() makes the upload run anyway, and the report matters most when tests fail.
  4. 4. What does the cron expression '30 2 * * 1-5' mean?

    The fields are minute, hour, day of month, month and day of week: minute 30, hour 2, any day or month, Monday to Friday.
  5. 5. What should you do about a test that fails randomly one run in twenty?

    Retries hide real intermittent problems and teach people to distrust the suite. Finding and fixing the cause keeps the signal reliable.

Summary

  • CI runs your collection automatically on changes and on a schedule, so regressions are caught early.
  • Keep the collection and a non-secret environment in the repository; inject secrets from the CI secret store with --env-var.
  • Newman's exit code is the contract with CI: zero is green, non-zero is red.
  • Publish JUnit XML (and keep it with an always() upload step) so failures are readable in the CI interface.
  • Decide the failure rules in advance: any failed assertion fails the build, flaky tests are bugs, and pull-request runs stay fast.
  • A schedule is a cron expression in UTC; use it for slower, broader or environment-health checks and notify people when a scheduled run fails.
  • The runnable code is a stand-in (a mini runner, JUnit writer and cron matcher), while the pipeline files are read-only examples to verify in your own repository.

Next: Practice exercises with solutions.