Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 37 ยท Patterns and Pitfalls

Flaky and Slow Suites: Diagnosis and Cures

Find and fix flaky Cucumber scenarios and speed up slow suites: causes, a diagnosis routine, quarantine tags, honest retries, shortcuts for login and setup, and test levels.

Two problems kill trust in automated acceptance tests faster than anything else. A flaky scenario passes and fails on the same code, so people learn to rerun the build and ignore red. A slow suite makes feedback arrive after the developer has moved on, so people stop waiting for it. Both are fixable with a habit of diagnosis and a few standard cures.

This lesson explains why scenarios flake, how to prove it and find the cause, how to handle flaky tests while you fix them, and how to make a suite fast. The real Cucumber and CI tools cannot run on this site, so the configuration is read-only; the runnable stand-ins simulate a flake-rate measurement and the time saved by shortcuts.

Why scenarios flake

Cause Symptom Cure
Timing: acting before the page or service is ready element not found, stale element explicit waits in page objects (Waits)
Shared data: scenarios collide passes alone, fails in suite or in parallel unique data, isolation (Test data)
Order dependence fails when run alone or reordered each scenario sets up its own state
Shared state in glue code fails only in parallel per-scenario world (State)
Unreliable environment: shared staging, slow third-party service failures cluster at certain times local or contained test doubles for external systems
Time and randomness: tests depend on "now", time zones, random values fails near midnight or month end inject a clock, seed random generators
Asynchronous behavior: e-mail sent, queue processed assertion runs before the effect poll for the outcome with a timeout
Resource limits: CI machine overloaded random timeouts fewer threads, bigger machine, smaller browser windows

Almost none of these are "Cucumber problems"; they are test design problems that Cucumber makes visible.

A diagnosis routine

  1. Measure. Run the suspect scenario 20 or 50 times in a loop, alone, then with the whole suite, then in parallel. A scenario that fails 3 of 50 times alone has a timing or randomness problem; one that fails only in the suite has a data or order problem.
  2. Capture evidence. Screenshots, page source, browser console logs, API responses and server logs attached to the report (Reports).
  3. Look for the pattern: same step, same time of day, same worker thread, same environment?
  4. Fix the cause, not the symptom. Adding sleep(2000) or retry-on-failure treats the symptom.
  5. Prove the fix by repeating the loop.

This stand-in simulates step 1: a scenario with a hidden failure probability is run many times with a fixed random seed, and the failure rate is reported for the "alone" and "in suite" conditions.

Java Measuring a flake rate (Java stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The fixed sleep fails for a chunk of runs; the explicit wait does not. A single run of each would not have told you.

Handling flaky scenarios while you fix them

  • Quarantine, visibly. Tag the scenario @flaky (with a ticket link) and exclude it from the blocking pipeline with not @flaky (Tags). Run quarantined scenarios in a separate non-blocking job, so you see whether they recover. Set a deadline: quarantine is a hospital, not a graveyard.
  • Retries, honestly. Cucumber-JVM can rerun failures (cucumber.execution.retry-style options exist in some runners, or use the rerun file). A single automatic retry may keep CI usable, but it hides the problem unless you count and report every retried pass. Track flake rate as a metric and treat any retry as a bug to fix.
  • Never delete the evidence. Keep screenshots and logs for the first failure, not only the retry.

Making the suite faster

Speed comes from doing less work per scenario and running scenarios in parallel.

  1. Test at the lowest suitable level. Move rule checks from the UI to the service layer (API and service tests). This single change often cuts a suite by half.
  2. Don't log in through the UI in every scenario. Create a session through an API call or a seeded cookie and test the login page only in its own scenarios.
  3. Build state through the back door. A Given a customer with three past orders step should call the API or a repository, not click through three orders.
  4. Reuse browsers carefully, clearing cookies and storage between scenarios, or run headless.
  5. Run in parallel (Parallel execution) and shard across CI machines.
  6. Start the application once, and keep reference data loaded in @BeforeAll.
  7. Prune. Delete scenarios that duplicate others, or that test trivia nobody would notice breaking. Fewer, better scenarios beat many shallow ones.
  8. Look at the numbers. The timeline and JUnit time attributes show the slowest scenarios; fix the top ten first.

The second stand-in adds up the cost of repeating a UI login against a session shortcut, using fixed simulated timings.

Java What shortcuts save (Java stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Keep it healthy

  • Set a time budget for each pipeline stage (for example 10 minutes for pull requests) and treat exceeding it as a failure to fix.
  • Review the slowest and flakiest scenarios monthly, as a team.
  • Make failing builds somebody's job first, never "everyone's".

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A scenario passes alone but fails when the whole suite runs. What is the most likely cause?

    Passing alone and failing in the suite points to interference from other scenarios through data, ordering or shared state.
  2. 2. Why is adding an automatic retry for failures risky?

    A retry turns visible red into invisible flake, so measure and fix every retried pass.
  3. 3. Which change normally gives the biggest speed-up?

    Lower-level tests avoid browser start-up, rendering and navigation costs.
  4. 4. What is the purpose of a @flaky quarantine tag?

    Quarantine unblocks the team while preserving the scenarios, with a ticket and deadline for fixing them.

Summary

  • Flakiness comes from timing, shared data, order dependence, environments, time and randomness; diagnose by repeating runs alone, in suite and in parallel.
  • Fix causes (waits, unique data, isolation); quarantine visibly and be honest about retries.
  • Speed up by testing at lower levels, skipping UI login, building state through back doors, running in parallel and pruning.
  • Track flake rate and run time as team metrics.

Next: Living documentation and the three amigos