Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 44 ยท Testing and Quality Gates

Test Stages in a Pipeline

Decide where unit, integration and end-to-end tests belong in a CI/CD pipeline, get fast feedback, handle flaky tests and publish test reports.

A pipeline is only as useful as the answer it gives you, and the answer you want is "is this change safe?" as quickly as possible. Tests are where that answer comes from. But a project can have thousands of tests that range from a few milliseconds to several minutes each, and where you put them in the pipeline decides whether developers get feedback in one minute or in forty.

This lesson is about placement and trust. You will learn which tests belong in which stage, how to order stages so failures surface early, what to do with flaky tests that fail for no reason, and how to publish test results so people can read them without scrolling through logs. If you have not met jobs and dependencies yet, read Jobs, Steps and Dependencies first.

The real GitHub Actions, Jenkins and GitLab services cannot run on this site. The YAML below is read-only. The TryIt blocks are small stand-ins written in JavaScript that model the ideas (ordering math and retry math), so you can experiment with numbers.

The three layers of tests

Most teams talk about a test pyramid: many fast tests at the bottom, few slow tests at the top.

flowchart TB
    E["End-to-end tests<br/>few, slow, whole system"]
    I["Integration tests<br/>some, medium, real dependencies"]
    U["Unit tests<br/>many, fast, one piece in isolation"]
    U --> I --> E

The pyramid is about count and cost: the lower the layer, the more tests you want and the cheaper each one is.

  • Unit tests exercise one function or class with its collaborators replaced by simple fakes. They need no network, database or browser, so thousands run in seconds. They are your first line of defense.
  • Integration tests run several real parts together: your code against a real database, a real message queue, or your HTTP layer. They are slower and need setup (often a service container, see CI for Java, Go and PHP Projects).
  • End-to-end (E2E) tests drive the whole system the way a user would, often through a browser. They catch wiring problems nothing else catches, but they are the slowest and the most fragile.

A pipeline should mirror the pyramid. Run the cheap layers first and the expensive ones only after the cheap ones pass.

Fast feedback is the design goal

Imagine a pipeline with four stages: lint (1 minute), unit tests (3), integration tests (8) and E2E tests (20). If a typo breaks linting and you run E2E first, you wait twenty minutes to learn about a typo. If you run lint first, you learn in one.

Two ideas follow:

  1. Order sequential stages so the stage most likely to fail per minute of cost goes first. Cheap checks that fail often are perfect first stages.
  2. Run independent stages in parallel. Lint and unit tests do not need each other, so they can start together. Only stages that truly depend on something (a build artifact, a deployed preview) need needs.

Here is the arithmetic made visible. Each stage has a duration and a rough chance of failing on a typical change. For a given order, the expected time until you get an answer (a failure, or a full green run) is the sum over every possible outcome. The code compares several orderings.

JS Expected time to feedback by stage order

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Run it and notice that running the slowest stage first is clearly the worst order, and the ratio rule (failures per minute) gives the best result for sequential stages. Parallelism wins on a green run but costs more runner minutes, so it is a trade-off, not a free lunch. A common compromise is to run cheap checks in parallel as a "fast lane" and gate the expensive jobs behind them with needs.

A staged workflow

Here is a read-only workflow that follows the pyramid. Lint and unit tests run in parallel; integration tests start only if both pass; E2E starts only after integration succeeds.

YAML
name: ci
on:
  pull_request:
  push:
    branches: [main]

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run lint

  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm test -- --reporter=junit --outputFile=reports/unit.xml
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: unit-report
          path: reports/

  integration:
    needs: [lint, unit]
    runs-on: ubuntu-latest
    services:
      db:
        image: postgres:17
        env:
          POSTGRES_PASSWORD: test
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s
          --health-retries 10
        ports:
          - 5432:5432
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run test:integration
        env:
          DATABASE_URL: postgres://postgres:test@localhost:5432/postgres

  e2e:
    needs: integration
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npm run test:e2e

Notice if: ${{ !cancelled() }} on the artifact step. By default a step after a failed step is skipped, which means the report would vanish exactly when you need it. Conditions like this are covered in Expressions and Contexts.

The same shape in other tools: in GitLab you would write stages: [lint, unit, integration, e2e] and put jobs in them (jobs in one stage run in parallel); in a Jenkins declarative pipeline you use stage('Lint') blocks and a parallel { } block inside a stage.

Where each test belongs

A simple placement guide:

Test type Runs on Typical limit
Lint, type check, format every push and pull request seconds to 1 minute
Unit tests every push and pull request a few minutes
Integration tests every pull request, after unit tests 5 to 10 minutes
End-to-end smoke tests every pull request (a small set) under 10 minutes
Full E2E, browser matrix, load tests on merge to main, nightly, or before release as long as needed

The key idea is that not every test has to run on every change. A pull request needs a quick, trustworthy verdict. A nightly job can run the 90-minute suite and file an issue when it fails. You can also split a big suite across several runners by file or timing data; see Making Pipelines Fast.

Flaky tests

A flaky test passes and fails on the same code. Causes are usually timing (waiting a fixed 100 ms for something), shared state between tests, test order dependence, real network calls, or unseeded randomness. Flaky tests are poison, because people learn to hit "re-run" instead of reading failures, and then a real failure gets ignored too.

The math is unforgiving. If one test fails 2% of the time by chance, a suite of 100 such tests passes only about 13% of the time. The next TryIt shows how retries change that, and why retries are a bandage and not a cure.

JS How flaky tests multiply and what retries do

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Two lessons come out of that. First, a few flaky tests are enough to make a large suite red most of the time. Second, retries also make a real intermittent bug look green most of the time. So:

  • Quarantine known flaky tests: move them to a separate non-blocking job so they cannot stop merges, and track them as bugs with an owner.
  • Retry sparingly, ideally only for E2E tests, and make the retry visible in the report ("passed on attempt 2") so you can fix the cause.
  • Never wrap the whole pipeline in an automatic "re-run on failure".
  • Fix the root causes: wait for conditions instead of sleeping, give every test its own data, and seed random generators.

Test frameworks usually have their own retry option (for example a retries setting in Playwright). Prefer that over rerunning the whole job, because it retries only the failed test.

Test reports

Raw logs are a poor interface. Most test runners can write JUnit XML, a simple format that nearly every CI system understands. Jenkins has a junit step, GitLab shows artifacts:reports:junit inside the merge request, and in GitHub Actions you upload the files as artifacts or use a reporting action. Coverage numbers belong in a report too, which is the subject of Linting, Coverage and Quality Gates.

GitLab example (read-only):

YAML
unit:
  stage: test
  script:
    - npm ci
    - npm test -- --reporter=junit --outputFile=report.xml
  artifacts:
    when: always
    reports:
      junit: report.xml

when: always matters for the same reason as !cancelled() above: you want the report even when tests fail.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A pipeline has lint (1 min), unit (3 min) and E2E (20 min) stages that run one after another. Which order usually gives feedback fastest?

    Cheap stages that fail often should go first, so a broken change is reported after a minute instead of after twenty.
  2. 2. What is the main danger of automatically retrying every failed test?

    If a real bug fails 60% of runs, two retries make the job pass about 78% of the time, so the bug is hidden.
  3. 3. Why does the artifact upload step in the example use if: ${{ !cancelled() }}?

    Steps after a failure are skipped by default. The condition keeps the upload running so you can read the failing report.
  4. 4. Which tests are best saved for a nightly or main-branch run instead of every pull request?

    Pull requests need a quick verdict. Very slow suites can run later and report problems without blocking every change.

Summary

  • Follow the test pyramid: many fast unit tests, fewer integration tests, a small set of end-to-end tests.
  • Put cheap, often-failing checks first, run independent jobs in parallel and gate expensive jobs with needs.
  • Not every test has to run on every change; schedule the slowest suites for merge or nightly runs.
  • Flaky tests destroy trust. Quarantine them, track them, fix the cause, and keep retries visible and rare.
  • Publish JUnit-style reports and keep them even when tests fail.

Next, turn test results into rules the whole team has to follow in Linting, Coverage and Quality Gates.