Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 35 ยท Quality and Reliability

Running in CI

Run Selenium tests in CI: a GitHub Actions workflow, headless Chrome in containers, pytest-xdist and TestNG parallel runs, and uploading reports as artifacts.

A Selenium suite that only runs on your laptop is a suite that gets forgotten. The real value arrives when every pull request triggers the tests on a clean machine, nobody has to remember to run them, and a red build blocks a broken change. That machine is a CI server, and it behaves differently from your desktop: there is no screen, no logged-in user, a fresh file system every time, and a hard time limit.

This lesson shows how to run a suite there. You will see a GitHub Actions workflow, the browser options that make containers behave, how to run tests in parallel with pytest-xdist and TestNG, and how to keep reports and screenshots after the machine disappears. If you have not met workflows before, Your First Workflow covers the basics.

The workflow, Docker and Java snippets below are real configuration, but the site cannot run CI, a browser or a container. They are read-only. The two runnable examples are stand-ins that model the ideas: how tests are divided between workers, and why workers need their own data.

What changes in CI

Your laptop vs a CI runner

No display Run the browser headless.
Clean slate Nothing is installed or cached by default.
Time limit Slow suites get cancelled.
Disposable Logs and screenshots vanish unless uploaded.

Design the suite for a machine nobody is watching.

1 / 2

The test command is the contract. CI does not understand Selenium; it only checks whether your command exited with code 0. pytest and Maven both do that correctly, so you rarely need extra glue.

A GitHub Actions workflow

Here is a complete workflow for a Python suite. It checks out the code, installs dependencies with caching, runs the tests on four workers, and uploads the report whether the tests pass or fail.

YAML
name: ui-tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  ui:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    env:
      BASE_URL: https://staging.example.test
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: pip

      - run: pip install -r requirements.txt

      - name: Run UI tests
        run: pytest -n 4 --headless --html=reports/report.html --self-contained-html
        env:
          TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}

      - name: Keep evidence
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: ui-test-evidence
          path: |
            reports/
            artifacts/
          retention-days: 7

A few points deserve attention:

  • timeout-minutes stops a hung browser from burning the runner for the default six hours.
  • if: always() on the upload step is essential. Without it the step is skipped when tests fail, which is exactly when you want the screenshots.
  • Secrets such as TEST_PASSWORD come from the repository's secret store and are passed through env, never written into the repo. See Secrets, Variables and Environments and, for the testing side, Security, Ethics and Scraping Limits.
  • --headless is not a built-in pytest flag. It is a custom option you add in conftest.py, as shown in Selenium with pytest, which then passes the headless argument to the browser options.

GitHub's Ubuntu runners come with Chrome already installed (check the runner image notes for the current version). Selenium Manager downloads a matching driver the first time a session starts, so no separate driver install step is needed.

Headless Chrome in containers

If you run the suite in your own Docker image, or against a Selenium container, a few options matter more than they do locally. The usual Chrome options for a container are:

Python
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

def make_driver():
    options = Options()
    options.add_argument("--headless=new")
    options.add_argument("--window-size=1366,900")
    options.add_argument("--disable-dev-shm-usage")
    # Only when the container runs as root and cannot create the sandbox:
    options.add_argument("--no-sandbox")
    return webdriver.Chrome(options=options)
  • --window-size matters because a headless browser has a default size that is often smaller than a desktop. A layout that shows a desktop menu on your machine may show a hamburger menu in CI, and your locators will not match. Set the size explicitly.
  • --disable-dev-shm-usage makes Chrome use a temporary folder instead of /dev/shm. Docker gives containers a very small shared-memory area by default, and Chrome tabs can crash with it ("tab crashed" errors). The alternative is to start the container with a larger --shm-size.
  • --no-sandbox turns off Chrome's own sandbox. It is a workaround for containers where the sandbox cannot start. Use it only inside throwaway CI containers, not on machines where you browse the web.

For the full list of options see Headless Mode and Browser Options. If you would rather not install a browser in your own image, the Selenium project publishes ready-made browser images, and the suite connects to them through a remote driver; Selenium Grid explains that setup.

Running tests in parallel

A UI suite is slow because every step waits for a real browser. Ten minutes of serial tests become three minutes when four workers share them. The cost is that tests must be independent.

pytest-xdist

Install pytest-xdist and add -n:

Terminal
pytest -n 4                    # four worker processes
pytest -n auto                 # one per CPU core
pytest -n 4 --dist loadfile    # keep all tests of one file on the same worker

Each worker is a separate process named gw0, gw1 and so on. If a fixture creates a driver per test, every worker gets its own browser automatically. Be careful with a session-scoped fixture: it runs once per worker, not once for the whole run.

TestNG

In Java, TestNG parallelism is configured in testng.xml:

XML
<suite name="ui" parallel="methods" thread-count="4">
  <test name="regression">
    <packages>
      <package name="com.example.ui.tests"/>
    </packages>
  </test>
</suite>

Threads share the same objects, so a single static WebDriver field would be used by all of them at once. The standard fix is a ThreadLocal<WebDriver> that hands each thread its own session. See Selenium with JUnit and TestNG.

How tests get split

The default strategy sends tests to whichever worker is free. If you shard the suite across several CI jobs instead (for example with a matrix of jobs that each run a slice), you must choose the slices yourself. A naive split by position can be badly unbalanced because tests have very different durations. The stand-in below compares two strategies using made-up timings.

Node.js Splitting tests across four workers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

The suite takes 127 seconds serially. Round robin finishes in 46 seconds because one worker draws the 28-second checkout test plus two more. Giving the longest tests out first and always filling the emptiest worker brings the wall time to 33 seconds, close to the ideal of about 32. The run ends when the slowest worker ends, so balance matters as much as the number of workers. Real tools such as pytest-xdist's scheduler and the pytest-split plugin (which can store durations from earlier runs) are far more sophisticated, but the arithmetic is the same. The durations here are invented; the stand-in does not start a browser.

Parallel tests must not share state

Parallelism exposes hidden coupling. If every test logs in as qa-user and edits the same cart, one worker's actions break another's assertions, and the failures look random. The cure is isolation: each worker gets its own account, its own records, or its own sandbox. pytest-xdist exposes the worker name through the PYTEST_XDIST_WORKER environment variable (values like gw0) and through the worker_id fixture, which you can use to derive unique data.

JS Why workers need their own data

Press Run (or Ctrl + Enter) to see the console output here.

No output.

With one shared cart, gw0 reads back 1 item instead of 3, because gw1 overwrote it. With per-worker keys both read what they wrote. In a real suite the unique value might be a user name, an order prefix, or a database schema, and creating it through an API before the test is faster than creating it through the UI.

Reports and artifacts

When a CI job ends, its disk is deleted. Anything you want to look at later must be uploaded. The useful set is:

  • The test report (HTML or JUnit XML, which most CI systems can render as a test summary).
  • Screenshots and page source captured at the moment of failure.
  • Browser or driver logs if you enable them.

The next lesson, Reporting and Logging, shows how to produce these files. Keep retention short (a week is plenty) so artifacts do not fill your storage.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should the artifact upload step use if: always()?

    By default a step is skipped after a failed step. if: always() forces it to run, so screenshots and reports from failing tests are still saved.
  2. 2. A Selenium test passes locally but crashes in a Docker container with 'tab crashed'. Which option is the usual fix?

    Docker's default shared-memory area is tiny and Chrome uses it heavily. Moving that storage to a temp folder, or enlarging the area, prevents the crashes.
  3. 3. In pytest-xdist, what does a session-scoped fixture do?

    Workers are separate processes, so each one sets up its own session-scoped fixtures. Remember this when the fixture creates accounts or data.
  4. 4. Four workers run your suite, but the wall time barely improves. What is the most likely cause?

    The run ends when the slowest worker ends. A single 28-second test or a poor split dominates the total time.

Summary

  • CI has no display, a clean disk and a time limit: run headless, set the window size, and set a job timeout.
  • The test command's exit code is the contract; upload reports and screenshots with if: always().
  • In containers, set --disable-dev-shm-usage or a bigger shared-memory size, and use --no-sandbox only in throwaway containers.
  • pytest-xdist (-n) and TestNG (parallel, thread-count) run tests concurrently; give every thread or worker its own driver.
  • Balanced splits matter, and parallel tests need isolated data (accounts, carts, records).
  • Pass secrets through the CI secret store, never through the repository.

Next, learn how to turn a run into something readable: Reporting and Logging.