Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 29 ยท Quality and pitfalls

Flaky Tests and Common Mistakes

Find and fix flaky JUnit and TestNG tests: shared state, order dependence, time, randomness, sleeps and singletons, with runnable before-and-after examples.

A flaky test is one that sometimes passes and sometimes fails without any change to the code. It is worse than a failing test, because a failing test tells you something is wrong, while a flaky one teaches the team to shrug and press "re-run". Once people stop trusting red builds, real bugs slip through. Nearly every flaky test comes from the same short list of causes, and nearly every cause has a standard fix.

All Java examples in this lesson run on the real JUnit 6 API on our Java runner. The helper at the bottom of each file plays the part of your build tool (see Your First JUnit Test). The same causes and fixes apply to TestNG; only the annotation names differ.

The usual suspects

Where flakiness comes from

Shared state Static fields, files, databases.
Test order One test needs another to run first.
Time Clocks, dates, time zones.
Randomness Unseeded Random and UUIDs.
1 / 2

The guiding rule behind every fix: a test must control all of its inputs. Anything that varies between runs, such as the current time, a random number, the order of other tests or the speed of a thread, is an input you did not control.

Shared state and order dependence

JUnit and TestNG do not promise an order, and a build tool may run classes in parallel or in a different order tomorrow. The classic trap is a static field that one test changes and another reads. The tests pass when run in the written order and fail otherwise, or pass alone and fail with others (or the reverse).

The example below runs the same two tests three times. FragileInOrder and FragileReversed assert on a shared static list; @Order forces a known sequence so you can see the effect. Robust resets the state before each test. Between runs the helper clears the list, so each class starts equal.

Java Same tests, different order, different result

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The two fragile classes contain identical tests; only the order differs, and one passes while the other fails twice. Robust passes in any order because each test builds its own world.

The fixes, in order of preference:

  1. Don't share. Create objects in the test or in @BeforeEach. JUnit creates a new test instance per method, so instance fields are safe; static fields are not.
  2. If you must share, reset. Clear tables, delete temporary files (JUnit's @TempDir gives each test a fresh folder), restore system properties.
  3. Never write a test that needs another test to run first. If you need a multi-step scenario, make it one test with clear steps. TestNG's dependsOnMethods allows ordering on purpose, but it is better used sparingly (see Dependencies, Priority and Test Ordering).

Static singletons and global state

A class with static Config INSTANCE, a static cache or a call to System.setProperty leaks into every later test in the same JVM. Typical symptoms: a test passes alone and fails in the full suite, or fails only when another class ran first. Prefer passing the dependency in through the constructor. If you cannot change the code, add an @AfterEach that restores the original value, and consider isolating the offending tests in a separate fork (see Running Tests with Maven and Gradle).

Java
// Read-only example: a hidden global makes this test depend on every test before it
class PriceFormatter {
    static String currency = "USD";                // global, mutable

    static String format(int cents) {
        return currency + " " + cents / 100.0;
    }
}

Time

Code that calls LocalDateTime.now(), Instant.now() or System.currentTimeMillis() directly is untestable in a stable way: a test written today passes at noon and fails at midnight, on the last day of the month, or in another time zone. The standard fix is to inject a java.time.Clock and call LocalTime.now(clock) or clock.instant(). In production you pass Clock.systemUTC(); in tests you pass a fixed clock or one you can move.

The example tests a greeting at fixed hours, and a session that expires after thirty minutes. The second test moves the clock forward instead of sleeping for thirty minutes.

Java A fixed clock and a clock you can move

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Notice how the boundary (29 minutes, then exactly 30) is trivial to test now. With real time you would sleep for half an hour, or never test the edge at all. Time zones deserve the same care: build test instants with ZoneOffset.UTC or an explicit zone, never with the machine's default.

Randomness

Unseeded randomness makes a test pass 999 times and fail on the thousandth. You have two good options. If the code takes a java.util.Random (or a RandomGenerator) as a parameter, pass one with a fixed seed so every run produces the same sequence. If you need exact control, pass a tiny subclass or a lambda that returns the numbers you choose. The same goes for UUID.randomUUID(), shuffles and any "pick one at random" logic: put the source of randomness behind a seam you can replace.

When a failure came from a random input, print the seed in the failure message. Then you can reproduce it exactly instead of rerunning and hoping.

Sleeps and concurrency

Thread.sleep(500) in a test is a guess about how fast the machine is. On your laptop it works; on a busy CI box it fails; and it makes every run at least half a second slower. The cure is to wait for the event, not for the time: a CountDownLatch, a CompletableFuture, or a library poll with a timeout. The timeout is a safety net for the failure case, not a delay: a passing test continues the moment the event happens.

Java
// Read-only: the flaky version. Passes only if the worker finishes within 50 ms.
@Test
void flakyDownloadCompletes() throws InterruptedException {
    downloader.startAsync(() -> done.set(true));
    Thread.sleep(50);                    // a guess
    assertTrue(done.get());
}

The runnable version below replaces the guess with a latch and a future, and tests a seeded dice roller as well. Everything is deterministic: the generous 2-second timeouts are only ever reached when something is genuinely broken.

Java Seeds, controlled randomness and waiting for events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

A passing test returns as soon as the latch opens, typically after about 80 ms here. The 2 seconds is the most it will ever wait. For deeper treatment of timeouts and asynchronous code see Testing Exceptions, Timeouts and Async Code.

Races between threads inside the code under test are a different class of problem. Test them with deterministic hooks where possible (a latch that forces two threads into the bad interleaving), and keep stress tests, which repeat an operation thousands of times, separate and tagged so they do not slow every build.

Common mistakes that look like flakiness

  • Ignoring failures. Marking a flaky test @Disabled "for now" or wrapping it in a retry loop hides the problem. TestNG's retry analyzer and Gradle/Maven rerun options are fine to measure flakiness, but a test that needs retries to pass is broken and should be fixed or removed, with a ticket.
  • Depending on the environment. Real network calls, fixed port numbers, the default locale or time zone, file paths with Windows separators, or the order of HashMap iteration. Use fakes, port 0 (let the OS choose), explicit locales and sorted comparisons.
  • Floating-point equality. Use assertEquals(expected, actual, delta); see JUnit Assertions in Depth.
  • Tests that share one expensive object by default. @BeforeAll with mutable fixtures reintroduces shared state. Share read-only data only.
  • Assertions inside other threads. An exception on a background thread never fails the test. Collect the result and assert on the test thread.
  • Over-long tests. A 60-line scenario fails for many reasons. Split by behavior, as in Designing Good Tests.

What to do when you meet a flaky test

  1. Reproduce it. Run it in a loop (@RepeatedTest(200) is handy), alone and with the whole suite, and in a different order. Note the seed and the time.
  2. Quarantine, don't delete. Tag it (for example @Tag("quarantine")) and exclude that tag from the gating build so it stops blocking everyone, but keep it visible and track it.
  3. Find the uncontrolled input: state, order, time, randomness, waiting or environment. It is almost always one of those.
  4. Fix the cause, then remove the quarantine. A flaky test is a bug in the test or in the code, and sometimes the test has found a real race in production code.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Two tests pass individually but one fails when the whole suite runs. What is the most likely cause?

    State that survives between tests couples them. Create fresh objects per test or reset shared state in @BeforeEach/@AfterEach.
  2. 2. How should a class that needs the current time be written so tests are stable?

    Injecting a Clock lets production use the system clock while tests use Clock.fixed or a clock they can advance.
  3. 3. What is wrong with Thread.sleep(50) followed by an assertion about work done on another thread?

    Wait for the event itself (a latch or future) with a generous timeout instead of guessing how long the work takes.
  4. 4. Why pass a seeded Random to code that needs randomness?

    A fixed seed makes the 'random' inputs repeatable. Printing the seed on failure lets you replay exactly what happened.
  5. 5. A test only passes after three automatic retries. What is the right response?

    Retries hide real problems. Quarantine with a tracking note, find the cause, then fix the test or the production code.

Summary

  • A flaky test has an input it does not control: shared state, test order, time, randomness, timing guesses, concurrency or the environment.
  • Create fresh state per test, reset what you must share, and never let one test depend on another. Avoid mutable statics.
  • Inject a Clock, a seeded Random, and other sources of change; move time forward in the test instead of sleeping.
  • Wait for events (latches, futures, polling with timeouts), not for durations.
  • Do not hide flakiness with retries or @Disabled: reproduce it, quarantine it with a tag, find the cause, fix it.
  • All Java examples ran on the real JUnit 6 API; only the runner helper stands in for Maven, Gradle or your IDE.

Next: Project: Test-Driven Bank Account, where you put these habits to work by writing the tests first.