Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 35 ยท Async, Waiting and Flakiness

Finding and Fixing Flaky Tests

Learn why end-to-end tests become flaky, how to reproduce them with repeat runs, when retries help or hide problems, and how to fix timing and shared-state bugs.

A flaky test is a test that sometimes passes and sometimes fails when nothing in the code changed. Flaky tests are one of the most expensive problems in browser testing. Each random failure makes someone re-run the pipeline, and after a few weeks the team starts ignoring red builds. A suite nobody trusts has no value, even if it contains a thousand tests.

The good news is that flakiness has a small number of causes, and each one has a known fix. In this lesson you will learn to recognize the causes, reproduce a flaky test on purpose, use retries wisely, and quarantine a test you cannot fix yet. The stand-ins use a seeded random generator, so the "random" failures are the same on every run and the numbers you read below match what you will see.

The commands and configuration for Cypress and Playwright are read-only. The runnable examples are stand-ins that simulate flakiness with a seeded random number generator; they measure failure rates the way a repeat run would, but they do not drive a browser.

The usual causes

Almost every flaky test falls into one of these groups.

  1. Timing. The test acts or asserts before the page is ready. A fixed sleep that is long enough on your laptop is too short on a busy CI machine. See Auto-Waiting in Playwright and Assertions That Retry.
  2. Shared state. Tests depend on data or settings left behind by an earlier test, so the result depends on test order, on parallel workers, or on whether the previous run cleaned up.
  3. Unstable data. The test expects something that changes: today's date, a random ID, a list sorted by an unstable key, or a record that another test deletes.
  4. Animations and layout shifts. A button moves while the click is being aimed, or a banner loads late and pushes everything down.
  5. Network and third parties. A real API is slow, rate limited or down, or an ad script interferes. See Intercepting and Mocking Network Requests.
  6. Environment. CI has less CPU and memory, a different screen size, a different time zone, or a browser version that differs from yours.

A quick way to guess the group is to look at how it fails. Failures that depend on speed are timing. Failures that depend on order or which worker ran them are shared state. Failures that appear only at midnight, on Mondays or in another country are data or environment.

Reproduce it before you fix it

You cannot fix a failure you cannot see. The first step is to run the test many times and count the failures.

Terminal
# Playwright (read-only): run the test file 30 times, in parallel workers
npx playwright test tests/checkout.spec.ts --repeat-each=30

# Playwright: reproduce a worker or order problem by forcing one worker
npx playwright test --workers=1 --repeat-each=10

# Cypress (read-only): there is no built-in repeat flag; a shell loop works anywhere
for i in $(seq 1 20); do npx cypress run --spec cypress/e2e/checkout.cy.js || echo "failed on run $i"; done

If the test fails 3 times in 30 runs, you now have a measurable problem and a way to prove the fix: 0 failures in 100 runs. Also try to make the machine behave like CI: run with CPU throttling, a smaller viewport, or many workers at once.

The next stand-in does exactly this on a simulated app. The server answers in a random time between 150 and 1,050 ms. Three tests wait for it with a fixed sleep of different lengths, and one polls until the answer is there. It runs 2,000 times and counts failures.

Node.js Measuring a failure rate with repeat runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Short sleeps fail 58.4% and 25.8% of the time. The 1,100 ms sleep never fails here, but it makes every run wait the full 1,100 ms, while polling averages 587 ms and never fails. In real life the slowest response is not known in advance, so a fixed sleep that works today fails the day the server is slow. Wait for a condition, not for a duration.

Fixing shared state with isolation

The second classic cause is tests that depend on each other. The stand-in below has two tests that use one shared cart object. "Adds one item" leaves a book in the cart. "Starts empty" assumes the cart is empty. Run them in one order and everything passes; reverse the order, or let two workers run them together, and a test breaks.

Node.js Order dependence and test isolation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

With a shared cart, the order A then B fails the "starts empty" test, while the reverse order passes everything. That is the signature of order dependence: the failure depends on what ran before. With a fresh cart for every test, every order passes.

Both tools help you get isolation. Playwright gives every test a new browser context (separate cookies and storage) by default. Cypress clears cookies, local storage and the page before each test (this is the testIsolation setting, on by default in current versions). What neither tool resets for you is data on your server. Create the data each test needs, use unique values (an email with a random part), and avoid assertions about "the whole list" when other tests may add to it. Test Data and Logging In Fast shows how to seed and reset data through an API.

flowchart TD
  F["A test failed once"] --> R{"Fails again with repeat runs?"}
  R -->|"always"| B["Real bug or broken test: fix it"]
  R -->|"sometimes"| W{"Depends on speed or order?"}
  W -->|speed| T["Timing: wait for a condition, stub slow calls"]
  W -->|order or worker| S["Shared state: isolate data"]
  W -->|"date, locale, machine"| E["Data or environment: pin it"]
  R -->|"never"| M["Could be a one-off: watch it"]

Reproduce first; then the way it fails points to the group of causes.

Retries: a safety net, not a cure

Both tools can re-run a failed test automatically.

TypeScript
// Playwright (read-only) in playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  retries: process.env.CI ? 2 : 0,   // retry only on CI
});
JavaScript
// Cypress (read-only) in cypress.config.js
const { defineConfig } = require('cypress');

module.exports = defineConfig({
  retries: { runMode: 2, openMode: 0 },  // `cypress run` retries twice; `cypress open` does not
});

When a test fails and then passes on a retry, Playwright reports it as flaky (not simply passed), and Cypress shows the failed attempts in its logs. Use that information.

Retries work because independent failures multiply: a test that fails 20% of the time fails all three attempts of a run with two retries only 0.8% of the time (0.2 ร— 0.2 ร— 0.2), which the table in the stand-in shows. But this is also the danger. Retries make a bad test look healthy while it still wastes time on every run, and a failure that is real but intermittent (a race condition in your app) gets hidden. So:

  • Keep retries low (1 or 2) and keep them on in CI only, so you notice flakiness on your own machine.
  • Review the list of tests that needed retries every week.
  • Never use retries as a replacement for fixing a test you understand.

Quarantine what you cannot fix today

If a test is flaky and the fix is not obvious, leave it running somewhere it cannot block everyone, instead of deleting it or ignoring red builds.

TypeScript
// Playwright (read-only)
test('applies a coupon @flaky', async ({ page }) => {
  test.fixme(true, 'Flaky since the promo banner was added; ticket SHOP-482');
  // ... the test body stays in the file
});
Terminal
# run the main suite without quarantined tests, and the quarantine separately
npx playwright test --grep-invert @flaky
npx playwright test --grep @flaky --retries=0

test.fixme() skips a test and marks it as needing a repair (test.skip() is for tests that do not apply). Give every quarantined test an owner and a ticket, and a deadline. Cypress has it.skip() for the same purpose, and tags via the community @cypress/grep plugin. A quarantine list that grows forever is just a slower way of deleting tests.

A fixing checklist

  1. Reproduce: repeat runs, one worker, throttled CPU.
  2. Read the failure message and the evidence (screenshot, trace, video; see Traces, Videos and Reports).
  3. Remove sleeps; wait for conditions; use retrying assertions.
  4. Make the test own its data and clean up (or use unique data).
  5. Stub unreliable third-party calls; pin dates and random values when the app lets you.
  6. Prove the fix: 50 to 100 repeat runs with zero failures.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A test passes when run alone but fails when the whole suite runs. Which cause is most likely?

    A result that depends on what ran before points to shared state or order dependence. Give each test its own data.
  2. 2. Why is a fixed sleep (for example 1,000 ms) a weak way to avoid flakiness?

    A fixed duration is a guess about speed. Waiting for a condition acts as soon as the page is ready, however long that takes.
  3. 3. A test fails 20% of the time and is retried twice. Roughly how often does the whole run fail?

    All three attempts must fail: 0.2 cubed is 0.008. The test is still flaky; retries only hide it, which is why flaky results should be reviewed.
  4. 4. What does Playwright's --repeat-each=30 option help you do?

    Repeating a test many times turns a vague 'sometimes fails' into a failure rate you can track and verify after a fix.
  5. 5. Which is a healthy way to handle a flaky test you cannot fix today?

    Quarantine keeps the problem visible without blocking everyone. More retries or giant timeouts just hide it.

Summary

  • Flaky tests come from timing, shared state, unstable data, animations, network dependence and environment differences.
  • Reproduce with repeat runs first; the way the test fails (speed, order, date) points to the cause.
  • Wait for conditions, not durations; isolate data so tests do not depend on each other.
  • Retries lower the chance of a red build but hide flakiness; use few, in CI, and review what needed them.
  • Quarantine unfixed tests with an owner and a ticket.
  • Prove a fix with many repeat runs and zero failures.
  • The stand-ins use seeded random numbers to measure failure rates; they do not drive a real browser.

Next: Intercepting and Mocking Network Requests