Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท CI and Best Practices

Flaky Tests, Retries and Quarantine

Find and fix flaky mobile tests: classify causes, measure flakiness from history, use retries responsibly, quarantine unstable tests and keep the suite trustworthy.

A flaky test passes and fails on the same code. Mobile suites are especially prone to it, because a test depends on a device, an operating system, a network, an animation frame and a backend all behaving on time. The real damage is not the wasted minutes: it is that people stop believing red builds. When "just re-run it" becomes normal, a real regression slips through unnoticed.

This lesson treats flakiness as an engineering problem: classify causes, measure the problem, use retries as a bandage with a time limit, and quarantine tests so they stop blocking the team while they are fixed. The code that talks to devices is read-only; the runnable examples analyze test history and simulate retry logic.

Where flakiness comes from

Cause Typical symptom Fix
Timing "no such element" only sometimes Explicit waits on conditions (see Waits and Synchronization)
Shared state Fails after another test, passes alone Isolate data, reset the app between tests
Test data User already exists, item sold out Generate data per test, use APIs for setup
Environment Fails on one device or one CI node Compare device logs, pin OS and tool versions
Network or backend Timeouts, 500s Mock or stub the backend, or retry the call, not the test
Animations and overlays Tap lands on the wrong thing Disable animations, wait for stability, dismiss pop-ups
Locators Matches different elements on different runs Unique accessibility IDs, avoid index-based XPath
Resource limits Slow or crashing emulator More memory, fewer parallel emulators

A useful first question when a test fails randomly: "what is different between a passing and a failing run?" Compare screenshots, the Appium log and timing, not just the assertion message.

Measure before you fix

Gut feeling about "which tests are flaky" is usually wrong. Record each test result with its commit, device and time, and compute a flake rate over recent runs. The same commit failing and passing is the strongest signal of flakiness.

Python Flake rate from test history (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note the difference between flaky (both outcomes on the same commit) and broken (always fails). A consistently failing test is a real signal, not flakiness; do not retry it away.

Retries: a bandage, not a cure

Retrying a failed test is sometimes justified, particularly for infrastructure noise such as a cloud device that disconnected. In pytest, the pytest-rerunfailures plugin does it:

Terminal
pytest --reruns 2 --reruns-delay 3

or per test: @pytest.mark.flaky(reruns=2). TestNG has IRetryAnalyzer, and JUnit 5 has the @RepeatedTest/retry extensions.

Rules for using retries responsibly:

  1. Report retried passes. A test that passed on attempt 2 is still a problem; your report must show it, so the flake rate stays visible.
  2. Retry the test, never individual steps, and always with a clean app state.
  3. Limit to one or two reruns. If it needs five, it is broken.
  4. Do not retry assertions about business logic. A wrong total that is right the second time is a race condition you want to find.
  5. Set an expiry. Retries added "temporarily" tend to stay forever; track them with a ticket and a date.

Stand-in: how retries change the picture

A retry hides a flaky test: if it fails 20% of the time, with two retries the build fails only 0.8% of the time, even though the test is still unreliable. This calculation shows why you must still track the underlying rate.

Python Effect of retries on build failures (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With a 30% failure rate and two retries the build still goes red about one run in thirty-seven; but the same test now wastes three times the device minutes on every bad run, and masks a real bug.

Quarantine

Sometimes you cannot fix a flaky test today. Quarantine it: move it to a separate group that still runs and reports, but does not block merging.

Python
@pytest.mark.quarantine      # registered in pytest.ini under markers
def test_push_notification_shows_banner(driver): ...
Terminal
pytest -m "not quarantine"     # the blocking run
pytest -m quarantine           # the separate, non-blocking job

Quarantine only works with discipline: every quarantined test has an owner, a ticket and an expiry date. After the date it is fixed or deleted. A quarantine that grows without end is just a graveyard.

A fix workflow

  1. Reproduce: run the test 20 to 50 times in a loop on the same device (pytest --count=30 with pytest-repeat).
  2. Collect evidence on failure: screenshot, page source, video, Appium and device logs (see Test Data, Screenshots and Reports).
  3. Classify the cause using the table above.
  4. Fix the cause. If it is a timing problem, replace sleeps with explicit waits. If it is data, isolate it.
  5. Verify with the same loop, then move the test out of quarantine.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which observation most strongly indicates a flaky test?

    Different outcomes on identical code point to nondeterminism.
  2. 2. Why should retried passes still appear in the report?

    Hiding flakiness hides the problem; the retried pass is still a defect to fix.
  3. 3. What is quarantine?

    Quarantined tests still run and report but do not block merges, and they must be fixed or removed on a deadline.
  4. 4. A test fails every single time since a recent commit. What is it?

    Consistent failure is a signal; investigate rather than retry.

Summary

  • Flakiness has identifiable causes: timing, shared state, data, environment, network, animations, locators and resources.
  • Measure flake rate from history; same commit with different results marks a flaky test.
  • Use retries sparingly, visibly and temporarily; they hide, not fix.
  • Quarantine with an owner, a ticket and an expiry date.
  • Reproduce in a loop, collect evidence, classify, fix and verify.

Next: Debugging and Performance: Logs, Inspector and Faster Sessions