Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 29 ยท TestNG advanced

Listeners, Retry and Parallel Execution

Hook into TestNG runs with ITestListener and reporters, retry failures with IRetryAnalyzer, and run tests in parallel without sharing mutable state.

A test run is more than pass or fail. You may want a screenshot when a test fails, a line in a log when it starts, a custom report for your team, a second chance for a test that depends on a flaky network, or a way to finish a 40-minute suite in 10 minutes. TestNG covers all of this with three features: listeners (hooks into the run), retry analyzers (decide whether to rerun a failure) and parallel execution (run tests on several threads).

This lesson explains each feature, shows how to register it, and, most importantly, warns about the traps: retries that hide real bugs, and parallel runs that break tests that share state. It builds on testng.xml, Suites and Test Configuration.

Real versus stand-in. The Java and XML in code blocks is real TestNG, read-only here because the site cannot run TestNG. The TryIt blocks are Node.js stand-ins: a tiny runner with a listener and a retry rule, and a simulated worker pool that uses virtual time so its output is always the same. Real TestNG uses real threads and its own scheduling.

Three tools for a smarter run

Listeners React to start, pass, fail, skip.
Retry analyzers Decide whether to run a failure again.
Parallel runs Use several threads to save time.

All three are powerful and all three can hide problems.

1 / 2

Listeners

A listener is a class that TestNG calls at certain moments. The most common one is ITestListener, with methods such as onTestStart, onTestSuccess, onTestFailure, onTestSkipped and onFinish. Recent TestNG versions give all of these default (empty) implementations, so you override only what you need:

Java
import org.testng.ITestListener;
import org.testng.ITestResult;

public class LoggingListener implements ITestListener {
    @Override
    public void onTestStart(ITestResult result) {
        System.out.println("START   " + result.getMethod().getMethodName());
    }

    @Override
    public void onTestFailure(ITestResult result) {
        System.out.println("FAILURE " + result.getMethod().getMethodName()
                + ": " + result.getThrowable().getMessage());
        // typical real use: take a screenshot, save logs, attach to a report
    }
}

ITestResult carries the method, its parameters, the status, timing and the exception. A listener runs inside the test run, so keep it fast and make sure it never throws, because an exception in a listener can confuse the results.

There are other listener interfaces for other moments: ISuiteListener (suite start and end), IInvokedMethodListener (before and after every method including configuration methods), IAnnotationTransformer (change annotations at runtime, for example to attach a retry analyzer to every test) and IReporter (build reports after the run).

Registering a listener

There are three ways, and they combine:

Java
@Listeners(LoggingListener.class)           // 1. on a test class
public class CartTest { /* ... */ }
XML
<suite name="Shop">                          <!-- 2. in testng.xml -->
  <listeners>
    <listener class-name="shop.support.LoggingListener"/>
  </listeners>
</suite>

The third way is the Java ServiceLoader: list the class name in META-INF/services/org.testng.ITestNGListener inside the test resources, and every run picks it up with no further setup. A @Listeners annotation applies to the whole suite, not just the class that carries it, which surprises people. Prefer the XML or service-file way for suite-wide behavior.

Retrying failed tests

Some tests fail for reasons outside the code: a network blip, a slow browser start, a shared test environment. TestNG lets you rerun them with an IRetryAnalyzer. After each failure TestNG calls retry(ITestResult); if it returns true, the test runs again:

Java
import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;

public class RetryTwice implements IRetryAnalyzer {
    private int attempts = 0;

    @Override
    public boolean retry(ITestResult result) {
        return attempts++ < 2;      // allow two retries, three runs in total
    }
}
Java
@Test(retryAnalyzer = RetryTwice.class)
public void loadsDashboard() { /* ... */ }

TestNG creates a separate analyzer instance for each test method, so the counter is per test. Intermediate failed attempts are reported as skipped, and only the last attempt shows as passed or failed. To retry all tests, attach the analyzer with an IAnnotationTransformer instead of editing every annotation.

The stand-in below wraps a test runner around the same two rules: a listener that logs, and an analyzer that allows two retries. The first test fails twice and then passes; the second always fails.

Node.js A listener and a retry analyzer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Look at the reporter lines: the flaky test ends as PASSED after three attempts, and the broken one as FAILED after three attempts. That first line is exactly the danger of retries. A green result no longer means "this works"; it can mean "this worked once in three tries". Treat retries as a temporary bandage, track how often they fire, and fix the cause. Flaky tests are discussed in Flaky Tests and Common Mistakes.

Reporters

A reporter builds output after the run. IReporter receives all suites and their results, so you can write a CSV, post to a chat channel, or generate your own HTML:

Java
public class CsvReporter implements IReporter {
    @Override
    public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDir) {
        for (ISuite suite : suites) {
            suite.getResults().forEach((name, result) ->
                System.out.println(name + ", passed=" + result.getTestContext().getPassedTests().size()
                        + ", failed=" + result.getTestContext().getFailedTests().size()));
        }
    }
}

Most teams do not need a custom reporter: the built-in HTML and XML reports, or a library such as Allure or ExtentReports registered as a listener, are enough.

Parallel execution

The parallel attribute in testng.xml selects the unit that gets its own thread, and thread-count sets the pool size:

XML
<suite name="Shop" parallel="methods" thread-count="4">
  <test name="All">
    <classes>
      <class name="shop.CartTest"/>
      <class name="shop.PaymentTest"/>
    </classes>
  </test>
</suite>
  • methods: test methods run in parallel, even from the same class. Most parallelism, most risk.
  • classes: all methods of one class stay on one thread, but different classes run at once. Instance fields are safer because a class object is used by one thread at a time.
  • tests: each <test> bucket gets a thread.
  • instances: methods of one instance stay together, different instances run at once; this pairs well with @Factory.

Dependencies between tests (Dependencies, Priority and Test Ordering) are still honored when running in parallel, but ordering by priority is not meaningful across threads.

The pool, simulated

The first part of the stand-in below schedules four tests onto one worker and then two workers, using fake durations in "virtual time", and prints which worker ran what and when. Because the clock is simulated, the output never changes between runs.

Shared state is the real problem

Parallelism does not break your code; it exposes code that was only correct by luck. Typical culprits are static fields, a shared WebDriver, a shared List used as a log, a shared test database row, and fields on the test class when running with parallel="methods". The second part of the stand-in forces the classic race: three "tests" each read a shared counter, yield, and write back seen + 1. All three read 0, so the final value is 1 instead of 3. The fix is to give each test its own state: a local variable, an instance per thread, or ThreadLocal.

Node.js A worker pool and a shared-state race

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Two workers finish the same four tests in 120 ms instead of 195 ms, but the total is bounded by the longest test: no number of threads beats the 120 ms slowReport. Splitting that slow test is the way to get faster.

Making parallel tests safe

  • Keep tests independent: each test creates the data it needs and does not rely on another test's leftovers.
  • Avoid static mutable state. If you must share a resource, make it thread-safe, or hold one per thread with ThreadLocal<WebDriver> created in @BeforeMethod and removed in @AfterMethod.
  • Use unique names and ids (a random suffix, or the thread id) when tests create files, database rows or accounts.
  • Start with parallel="classes" and a small thread-count, then move to methods when the suite is clean.
  • Listeners run on whichever thread triggered the event, so make them thread-safe too.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What happens when IRetryAnalyzer.retry returns true after a failure?

    TestNG reruns the test. Intermediate failed attempts show as skipped; the final attempt decides passed or failed.
  2. 2. Why can a green result from a retried test be misleading?

    A retry masks instability. Track retries and fix the cause instead of relying on them.
  3. 3. Which parallel mode keeps all methods of one class on the same thread?

    With parallel="classes", different classes run at once but each class's methods stay together, which makes instance fields much safer.
  4. 4. Three parallel tests each read a shared static counter, wait, then write back value + 1. What is the likely final value?

    This is a lost-update race: all tests read the same old value. Give each test its own state or use proper synchronization.

Summary

  • Listeners (ITestListener, ISuiteListener, IInvokedMethodListener, IReporter) hook into the run; register them with @Listeners, testng.xml or a service file.
  • @Listeners on one class affects the whole suite.
  • IRetryAnalyzer.retry decides whether a failed test runs again; use retries sparingly and monitor them.
  • parallel accepts methods, classes, tests and instances; thread-count sets the pool.
  • Parallel runs expose shared mutable state: prefer independent tests, per-test data and ThreadLocal.
  • More threads cannot beat your slowest test.

Next: JUnit vs TestNG: Choosing Between Them.