Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 35 ยท CI and Scale

Running Tests in CI

Run browser tests on every push: GitHub Actions workflows for Playwright and Cypress, installing browsers, Docker images, artifacts, caching and secrets.

Tests that only run on your laptop protect nothing. The point of automated browser tests is that every change is checked before it is merged, by a machine, in a clean environment, the same way each time. That machine is a continuous integration (CI) service such as GitHub Actions, GitLab CI or Jenkins.

Running browser tests in CI has a few special needs. The machine has no screen, so browsers run headless. The browsers and their system libraries must be installed. The app under test must be started and be ready before the first test. And when something fails, you were not watching, so the run must save artifacts (reports, traces, screenshots) that let you see what happened. This lesson walks through all of that with GitHub Actions; the ideas transfer to any CI system.

CI services and the real tools cannot run on this site, so the YAML and commands below are read-only. The runnable examples are stand-ins: a small pipeline simulator that mimics how a CI job runs steps, skips them after a failure and still uploads artifacts, and a cache-key calculator. Real CI systems are much more capable.

What a CI job does

A typical browser-test job has the same shape in every system:

flowchart TD
  A["Check out the code"] --> B["Install Node and dependencies"]
  B --> C["Install browsers"]
  C --> D["Build and start the app"]
  D --> E["Run the tests headless"]
  E --> F["Upload report, traces, screenshots"]
  E -->|"fail"| G["Job fails: merge is blocked"]

The artifact upload must run even when the tests fail, because failures are exactly when you need the evidence.

A Playwright workflow

Save this as .github/workflows/e2e.yml:

YAML
# read-only
name: e2e
on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: lts/*
          cache: npm

      - run: npm ci
      - run: npx playwright install --with-deps

      - name: Run Playwright tests
        run: npx playwright test
        env:
          BASE_URL: https://staging.example.com
          TEST_USER_PASSWORD: ${{ secrets.TEST_USER_PASSWORD }}

      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: playwright-report
          path: playwright-report/
          retention-days: 14

Read it line by line:

  • on: decides when the workflow runs: every push to main and every pull request.
  • timeout-minutes stops a hung run from burning minutes for hours.
  • actions/setup-node installs Node, and cache: npm caches the npm download cache, keyed on the lock file, so npm ci is faster.
  • npm ci installs exactly what the lock file says. Use it in CI instead of npm install.
  • npx playwright install --with-deps downloads the browsers and also installs the Linux system libraries they need. Without --with-deps a fresh Ubuntu machine often fails to launch a browser.
  • env: passes configuration. The password comes from secrets, never from the repository.
  • if: ${{ !cancelled() }} makes the upload run after a failing test step too. Without a condition, GitHub skips every later step once one fails.

Playwright's config should know it is on CI. A common pattern uses the CI environment variable, which GitHub Actions sets:

TypeScript
// playwright.config.ts (read-only)
import { defineConfig } from '@playwright/test';

export default defineConfig({
  forbidOnly: !!process.env.CI,          // fail if someone left test.only in the code
  retries: process.env.CI ? 2 : 0,       // retry flaky failures on CI only
  reporter: process.env.CI ? [['html'], ['github']] : 'list',
  use: {
    baseURL: process.env.BASE_URL ?? 'http://localhost:3000',
    trace: 'on-first-retry',             // record a trace only when a test is retried
  },
  webServer: {
    command: 'npm run start',
    url: 'http://localhost:3000',
    reuseExistingServer: !process.env.CI,
  },
});

The webServer option starts your app and waits until the URL answers before the tests begin, so you do not need a separate wait step. Reports and traces are explained in Traces, Videos and Reports, and retries in Finding and Fixing Flaky Tests.

A Cypress workflow

Cypress publishes an official GitHub Action that installs dependencies, caches the Cypress binary, starts the server and waits for it:

YAML
# read-only
name: cypress
on: [push, pull_request]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: cypress-io/github-action@v6
        with:
          build: npm run build
          start: npm start
          wait-on: 'http://localhost:3000'
          browser: chrome
        env:
          CYPRESS_PASSWORD: ${{ secrets.TEST_USER_PASSWORD }}

      - uses: actions/upload-artifact@v4
        if: failure()
        with:
          name: cypress-screenshots
          path: cypress/screenshots
          if-no-files-found: ignore

cypress run is already headless by default. Failing tests write screenshots under cypress/screenshots; videos are only recorded if your config turns them on, and then they go in cypress/videos. Values in environment variables named CYPRESS_... become available in tests through Cypress.env('PASSWORD'), which is covered in Configuration and Environments. Pin the action to a major version (@v6) and check its documentation for current options.

Docker images

Instead of installing browsers on every run, you can run the job inside an image that already contains them:

YAML
# read-only
jobs:
  test:
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.50.0-noble
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright test

The version tag of the Playwright image must match the version of @playwright/test in your package.json, because each Playwright release expects specific browser builds. The (hypothetical) version above is only an example; use the one from your lock file. Cypress has similar images, such as cypress/included, which bundle Cypress and browsers. Images give you the same environment on CI and on a laptop (run the same container locally to reproduce a CI-only failure), at the cost of a bigger download.

Secrets and configuration

Never write passwords, API tokens or cookies in the repository or in test files. In GitHub, add them under the repository's secret settings and read them as ${{ secrets.NAME }}; expose them to the test process as environment variables. GitHub masks secret values in logs, but a test that prints a response body or a trace that records a typed password can still leak one, so use test accounts with limited rights, and in Cypress hide typed values with type(value, { log: false }). Fork pull requests do not receive secrets by default, which is a security feature: design your PR checks to work with mock data or a public test environment.

Caching and speed

  • npm packages: setup-node with cache: npm handles it.
  • Cypress binary: the official action caches it for you; if you set up manually, cache the folder named by CYPRESS_CACHE_FOLDER (by default under your home directory's .cache/Cypress).
  • Playwright browsers: they are stored under ~/.cache/ms-playwright on Linux. Caching them is possible, but the Playwright team does not recommend it as a default, because restoring a large cache is often not faster than downloading, and system dependencies must still be installed. A Docker image avoids the question.
  • Build output: if you build the app in an earlier job, upload it as an artifact and download it in the test job, rather than building twice.

A cache is stored under a key. When the key changes, the old cache is not used. A good key includes everything that should invalidate it, typically the operating system and a hash of the lock file.

A pipeline simulator

This stand-in models the rule that surprises beginners: after a failed step, the following steps are skipped unless they say otherwise. It runs a list of steps, each with an optional condition that works like if: in GitHub Actions.

Node.js A tiny CI pipeline simulator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The first job fails and uploads nothing, which is the classic mistake: the report that explains the failure is never saved. The second job fails the same way but still uploads. Real GitHub Actions evaluates many more expressions, but this skipping rule is exactly how it behaves.

Computing a cache key

The second stand-in builds a cache key from the platform and a hash of the lock file contents. Change one dependency version and the key changes, so the old cache is ignored and a new one is saved.

Node.js A cache key from the lock file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Real caches also support fallback "restore keys" so a near match can warm a new key. The stand-in shows only the exact-match case.

Keeping CI useful

  • Fast feedback first. Run unit tests and linting before browser tests, so cheap failures stop the pipeline early.
  • Run a small smoke set on every pull request and the full suite on main or nightly if the full suite is slow. The tagging approach from Hooks, Fixtures and Custom Commands selects subsets.
  • Fix or quarantine flaky tests quickly. A suite people re-run "until green" has lost its value.
  • Make failures easy to open. Link the report in the job summary, keep traces for failed runs and give artifacts a retention period.
  • Reproduce locally with the same container image and the same environment variables.

When one job gets too slow, split it across machines. That is the subject of the next lesson, Parallel Runs, Sharding and Speed.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the upload-artifact step in the Playwright workflow use if: ${{ !cancelled() }}?

    By default, GitHub Actions skips later steps once a step fails. The condition makes the upload run after a failure too, which is when you need the report.
  2. 2. What does npx playwright install --with-deps do on a fresh CI machine?

    It downloads the browser builds and installs system dependencies, which a clean Ubuntu runner usually lacks.
  3. 3. How should a test password be provided to CI?

    Secrets stay out of the repository. The workflow reads them from the secret store and passes them as environment variables.
  4. 4. What must be true of the Playwright Docker image tag?

    Each Playwright version expects specific browser builds, so the image and the library must be the same release.
  5. 5. Which belongs in a good cache key for npm dependencies?

    A key built from the platform and the lock file hash changes exactly when the dependencies do, so stale caches are not reused.

Summary

  • CI runs your browser tests on every change in a clean, headless environment and blocks merges on failure.
  • Playwright needs npx playwright install --with-deps; the official Cypress action installs, builds, starts and waits for you.
  • Use npm ci, a timeout-minutes limit and CI-aware config such as forbidOnly and retries.
  • Upload reports, traces and screenshots with a condition that also runs after failures.
  • Docker images give identical environments, but the tag must match the library version.
  • Keep secrets in the CI secret store, and build cache keys from the platform and the lock file hash.

Next: Parallel Runs, Sharding and Speed