Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 37 ยท Cloud Device Farms

Parallel Runs and the Device Matrix

Run Appium tests in parallel: build a device matrix, give each worker its own device and ports, use pytest-xdist or TestNG, and avoid shared-state failures.

A suite of 200 mobile tests at three minutes each takes ten hours on one device. Nobody waits ten hours. The answer is parallel execution: run many sessions at the same time, each on its own device. On a cloud farm that is a plan setting; on your own machines it is configuration you must get right. This lesson shows how to describe the devices you want (the device matrix), how to give every worker its own device, ports and data, and how to split tests so that the workers finish at the same time.

Devices and farms are not available on this site. The runner commands are read-only; the stand-ins schedule tests across fake workers so you can experiment with the arithmetic.

What a device matrix is

A matrix lists every combination of platform, device and OS version you want to cover:

Id Platform Device OS
a1 Android Pixel 8 15
a2 Android Galaxy S23 14
a3 Android Galaxy A15 14
i1 iOS iPhone 15 18
i2 iOS iPhone SE (3rd gen) 17

Keep it in a file (JSON or YAML) instead of code, so CI can read it and a new device is a one-line change:

JSON
[
  {"id": "a1", "platformName": "Android", "deviceName": "Pixel 8", "platformVersion": "15"},
  {"id": "i1", "platformName": "iOS", "deviceName": "iPhone 15", "platformVersion": "18"}
]

Two matrices are worth having: a full one for nightly or release runs, and a small smoke one (one Android, one iOS) for every pull request.

Locally: one device per Appium server or one server, many sessions

With your own devices there are two layouts.

  1. One server, several sessions. A single Appium server can run many sessions, as long as each session targets a different device (appium:udid) and uses its own system ports.
  2. One server per device. Start separate servers on different ports; each is isolated, so a crash affects only one device.

The ports that must be unique per Android session:

Python
options.udid = device["udid"]
options.system_port = 8200 + index          # UiAutomator2 server on the host
# for Chrome inside webviews: options.set_capability("appium:chromedriverPort", 9500 + index)

For iOS, give each device its own appium:wdaLocalPort (for example 8100 + index) and, when sharing a machine, its own appium:derivedDataPath. Two sessions fighting over one port is the most common local parallel failure; the error is often a confusing connection reset.

Stand-in: assign devices and ports

Python Assigning devices and ports to workers (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Running with pytest-xdist

pytest-xdist starts worker processes and distributes tests among them. Each worker has a name (gw0, gw1, ...) available as the worker_id fixture, so the driver fixture can pick a different device per worker:

Python
# conftest.py
import json, pytest

MATRIX = json.load(open("devices.json"))

@pytest.fixture
def device(worker_id):
    index = 0 if worker_id == "master" else int(worker_id.replace("gw", ""))
    return MATRIX[index % len(MATRIX)]
Terminal
pytest -n 3 --dist loadgroup

Here -n 3 runs three workers, one per device. The loadgroup mode keeps tests with the same @pytest.mark.xdist_group name on one worker, which is useful when tests in a group depend on the same logged-in account.

On a cloud farm you rarely map workers to devices by hand. You give the matrix to a CI matrix job instead, so each job runs the whole suite (or a share of it) on one device configuration:

YAML
strategy:
  matrix:
    device: [a1, a2, i1]
steps:
  - run: pytest --device ${{ matrix.device }}

For Java, TestNG's parallel="tests" with a thread-count and one <test> block per device in testng.xml does the same; JUnit 5 uses junit.jupiter.execution.parallel.enabled.

Rules that make parallel runs safe

  • No shared mutable data. Two tests logging in as the same user and changing the same cart will fail randomly. Generate a unique user per test or per worker.
  • No shared device. One session per device at a time.
  • Tests must not depend on order. A test that needs another test to run first breaks as soon as they land on different workers.
  • Thread-safe driver handling. In Java, store the driver in a ThreadLocal; never use a static field.
  • Idempotent setup. App install and reset must work on a device in any state.

Stand-in: how to split tests evenly

Simple round-robin gives uneven finish times when tests differ in length. A greedy scheduler gives each test to the worker that is free earliest.

Python Splitting tests across workers (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Some runners can balance by measured duration from the previous run (pytest-split, TestNG listeners, vendor features). The principle is identical: the run ends when the slowest worker ends, so even loads save time.

When a worker fails

A device can disconnect, a cloud session can be dropped or an OS dialog can block a worker. Design for it: keep the retry policy at the test level (see Flaky Tests, Retries and Quarantine), make sure teardown always runs, and report which device configuration each failure came from, since "fails only on the Galaxy A15" is a valuable clue.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which pair of values must be unique per parallel Android session on one host?

    Each session needs its own device and its own UiAutomator2 system port.
  2. 2. Which iOS capability avoids WebDriverAgent port clashes in parallel runs?

    Give every simulator or device its own WDA local port.
  3. 3. Why is a longest-first scheduler often better than round robin?

    A run ends when the last worker finishes, so balanced loads reduce wall-clock time.
  4. 4. What is the problem with two parallel tests that share one user account?

    Shared mutable data causes order-dependent, flaky failures.

Summary

  • Describe devices in a matrix file; use a small smoke matrix per pull request and the full matrix nightly.
  • Locally, every parallel session needs its own device and unique ports (systemPort, wdaLocalPort).
  • pytest-xdist, TestNG and CI matrix jobs distribute work; use the worker id or matrix value to choose the device.
  • Avoid shared accounts and test ordering; balance tests by duration.
  • Record the device configuration with every failure.

Next: Appium in CI: GitHub Actions and Emulators