Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 35 ยท Quality and Reliability

Flaky Tests and Debugging

Find and fix flaky Selenium tests: common causes, screenshots and logs on failure, video, why blind retries hide bugs, and how to isolate test state.

A flaky test passes and fails on the same code. Run it again and the result changes, with no edit in between. Browser tests are especially prone to this, because they depend on timing, networks, animations and shared data. The damage goes beyond annoyance: when a red build might be noise, people stop trusting it, they click "re-run" by reflex and a real bug slips through. This lesson covers why flakiness happens, how to gather evidence when a test fails, why retries are a bandage and not a cure, and how to keep tests from affecting each other.

Real browsers cannot run on this site, so Selenium, pytest and TestNG code is read-only. The runnable examples are stand-ins in plain JavaScript: a retry wrapper around a flaky function and a tool that measures a test's pass rate. They use a seeded random generator, so every run prints the same output.

Where flakiness comes from

Timing Script races the page.
Shared state Tests change each other's data.
Environment Slow network, busy machine.
Animation Elements move while you click.

Most flakiness is a test problem, not a browser problem.

1 / 2

The usual suspects

Most flaky browser tests come from a short list of causes:

  • Missing or wrong waits. The script clicks before the page is ready, or reads text before an update arrives. This is the number one cause, and the cure is explicit waits, covered in Why Tests Need Waits and Implicit, Explicit and Fluent Waits. A fixed sleep only moves the problem: it is too short on a slow day and wasteful on a fast one.
  • Stale or moving elements. A framework re-renders a list and your saved element is replaced (a stale element error), or an animation slides a button while the click happens.
  • Shared or leftover state. Test B assumes the cart is empty, but test A left an item. Run order, parallel runs and previous failures change the result.
  • Uncontrolled data. The test depends on today's date, a random product, or content that other people edit on the staging site.
  • Environment load. A busy CI machine, a slow staging server or a cold cache stretches response times beyond a timeout.
  • Real bugs. Sometimes the flakiness is the product: a genuine race condition that only some runs expose. Dismissing it as "test noise" hides a defect your users will meet.

The mindset is simple: every flaky test has a cause, and the cause is either in the test or in the product. Your job is to find out which.

Gather evidence at the moment of failure

You cannot debug what you cannot see, and by the time you read the report the browser is gone. So make the test capture evidence automatically when it fails:

  1. A screenshot of the page. It answers "was the button hidden by a cookie banner?" at a glance.
  2. The page source or HTML of the area, because a screenshot cannot show attributes or hidden elements.
  3. Browser console messages. In Chromium drivers, driver.get_log("browser") returns recent console entries in many versions; otherwise use the event approach from DevTools Protocol and BiDi.
  4. The current URL and the test's own log lines such as "clicked checkout".
  5. A video, when your Grid image or cloud provider records one. It shows the steps before the failure, which a final screenshot cannot.

In pytest, a hook in conftest.py can do this for every test. The test reports its outcome to the hook, which saves artifacts when it sees a failure:

Python
# read-only: conftest.py
import pytest

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if report.when == "call" and report.failed:
        driver = item.funcargs.get("driver")   # the fixture named "driver"
        if driver is not None:
            name = item.name.replace("/", "_")
            driver.save_screenshot(f"artifacts/{name}.png")
            with open(f"artifacts/{name}.html", "w", encoding="utf-8") as f:
                f.write(driver.page_source)

In JUnit and TestNG, a listener or extension plays the same role (see Selenium with JUnit and TestNG), and CI should publish the artifacts folder, as described in Running in CI. Name files after the test so you can match them to failures.

A debugging habit that saves hours: run the failing test headed, not headless, on your own machine, and watch it. If it only fails headless, compare window sizes first (see Headless Mode and Browser Options); then check fonts and timing. Reproduce it in a loop (pytest --count with the pytest-repeat plugin, or a shell loop) to measure how often it fails before and after your fix.

Retries: useful, dangerous

A retry reruns a failed test automatically. In pytest the pytest-rerunfailures plugin offers pytest --reruns 2, or @pytest.mark.flaky(reruns=2) on one test. In TestNG you implement IRetryAnalyzer, whose retry method returns true to rerun. Retries are popular because they turn a red build green quickly. They also have real costs:

  • They hide the problem. A test that passes on the second attempt still has a bug, in the test or the product; you just stopped hearing about it.
  • They hide real defects. A bug that appears one time in three is retried into invisibility, then reaches users at the same rate.
  • They slow the suite and add load that can cause more flakiness.
  • They break assumptions if the test is not safe to repeat (it already created an order).

If you use retries, treat them as a measurement tool, not a cure: always report which tests needed a retry, count them, and fix the worst ones. A good policy is a small fixed limit (once or twice), a visible "flaky" label in the report, and a rule that a test retried too often gets quarantined and fixed, not ignored. The first stand-in builds a retry wrapper that does this.

JS A retry wrapper that reports flakiness

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Look at the report format: a test that passed on a retry is not shown as a plain pass. That label is the whole point of the wrapper.

Measure before you guess

Intuition about flakiness is poor: a test that failed twice last week may fail 1% of the time or 30% of the time. Run it many times and count. A stable test passes every time. A flaky one lands somewhere in between. A broken one fails every time. The second stand-in does that with three simulated tests, including a fixed sleep that races a "page" that loads in a variable time, which is the classic case, next to a proper polling wait.

JS Measure the pass rate: sleep versus polling

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Counting runs turns a vague feeling ("that test is annoying") into a number you can compare before and after a fix. It also separates the three situations: stable, flaky and broken, which need different actions.

Isolate state

Many flaky tests are really order-dependent tests. Make each test own its world:

  • Create what you need inside the test or its fixture, through an API or database setup, not through a long UI journey and not by relying on another test.
  • Use unique data, such as a user name with a timestamp or random suffix, so parallel tests do not collide.
  • Start from a clean browser: a new driver, or at least cleared cookies and storage, for each test (see Selenium with pytest).
  • Clean up what you created, or reset the environment before each run, so a crash does not poison the next one.
  • Control time and randomness when you can, for example with a test mode in the application.
  • Do not share a logged-in session between tests unless you treat it as read-only.

Finally, track flakiness like any other quality metric: how many tests needed a retry this week, which tests fail most, how long a red build takes to fix. A suite that people trust is one where red means a real problem.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is the most common cause of flaky Selenium tests?

    A race between the script and the page, usually from missing or wrong waits, causes most flaky browser tests.
  2. 2. Why is a blind automatic retry risky?

    A test that needs a retry still has a defect. Without a report of retried tests, nobody notices.
  3. 3. What should a failure hook save for later debugging?

    After the run, the browser is gone, so artifacts captured at the moment of failure are the only evidence.
  4. 4. A test passes 35 out of 50 runs. What do you call it?

    A pass rate between 0% and 100% on unchanged code is the definition of flaky.
  5. 5. Which practice best reduces order-dependent failures?

    Independent tests can run in any order, or in parallel, without affecting each other.

Summary

  • A flaky test passes and fails on unchanged code; the cause is in the test or in the product, and both deserve attention.
  • Common causes are bad waits, stale or moving elements, shared state, uncontrolled data and a loaded environment.
  • Capture a screenshot, page source and logs automatically on failure (a pytest hook, or a JUnit or TestNG listener), and publish them from CI.
  • Retries can measure and soften flakiness but hide bugs; always report retried tests and fix them.
  • Measure pass rates over many runs and isolate state so tests are independent.

Next: Running in CI