Skip to content
Tutorials Center
My learning

Topics

Lesson 14 of 29 ยท TestNG advanced

testng.xml, Suites and Test Configuration

Configure TestNG runs with testng.xml: suite, test, classes, packages, groups, include/exclude, parameters, Maven integration and reports.

With JUnit, the build tool and your IDE decide what runs: every class named like a test. TestNG gives you a second option, a configuration file called testng.xml that describes exactly which classes, packages, methods and groups form a run, which parameters they receive, and how many threads to use. The same compiled tests can then be run as a quick smoke suite, a full regression suite, or a browser matrix, just by choosing a different file.

This lesson explains the elements of testng.xml, how settings are inherited, how to run it from Maven and the command line, and what the reports contain. You should know the basics from Your First TestNG Test and groups from TestNG Assertions and Groups.

Real versus stand-in. The XML and Java shown in code blocks is real TestNG configuration, read-only here because the site cannot run TestNG. The TryIt blocks run a JavaScript stand-in: a tiny parser for a suite-like description and a selection routine that mimics include/exclude and parameter scoping. It is not TestNG's parser, and it ignores many real options.

The nesting of a suite

suite The whole run, with a name.
test A named group of classes. Not one test method.
classes / packages What TestNG should load.
methods / groups Optional filters inside.

In testng.xml a "test" is a bucket, not a method.

1 / 2

A first testng.xml

Create the file in the project root (or src/test/resources). The smallest useful suite lists a few classes:

XML
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Shop" verbose="1">
  <test name="Checkout">
    <classes>
      <class name="shop.CartTest"/>
      <class name="shop.PaymentTest"/>
    </classes>
  </test>
</suite>

The DOCTYPE line is optional but lets editors validate the file. Names in <class> are fully qualified Java class names. Run it from the command line with the TestNG jar and your compiled classes on the classpath:

Terminal
java -cp "target/classes:target/test-classes:libs/*" org.testng.TestNG testng.xml

IDEs can also run a testng.xml directly from the context menu.

Suite, test and class

The vocabulary trips up newcomers. A suite is the entire run. A <test> element is a named bucket inside the suite, such as "Checkout" or "Catalog", and it holds the classes to run. It is not one test method. Each <test> gets its own lifecycle for @BeforeTest and @AfterTest methods, and its own entry in the report. A class holds test methods, which are the individual tests you annotate with @Test.

Packages instead of classes

To include every test class in a package, use <packages>. A trailing .* includes subpackages:

XML
<test name="Catalog">
  <packages>
    <package name="shop.catalog.*"/>
  </packages>
</test>

This is easier to maintain than a long class list, but it also pulls in helper classes that contain TestNG annotations, so keep test packages tidy.

Filtering methods and groups

Inside a <class> you can list the methods that should (or should not) run:

XML
<class name="shop.PaymentTest">
  <methods>
    <include name="chargesCard"/>
    <exclude name="refundsCard"/>
  </methods>
</class>

Method names are regular expressions, so <include name="charge.*"/> is allowed. For larger selections, groups are cleaner. Tag methods with @Test(groups = {"fast"}) in Java, then choose in XML:

XML
<test name="Smoke">
  <groups>
    <run>
      <include name="fast"/>
      <exclude name="flaky"/>
    </run>
  </groups>
  <classes>
    <class name="shop.CartTest"/>
    <class name="shop.PaymentTest"/>
  </classes>
</test>

Only methods that belong to fast and not to flaky run. If a group contains <include> entries, methods without any included group are left out. Groups can also be composed into meta-groups with <define name="critical"><include name="payment"/><include name="login"/></define>, so a single name expands to several groups.

Parameters and their scope

Elements named <parameter> supply values to methods using @Parameters, as shown in Data Providers and Factories. They can appear at suite, test and class level. When the same name appears in several places, the most specific wins: class beats test, test beats suite.

XML
<suite name="Shop">
  <parameter name="env" value="staging"/>
  <test name="Checkout">
    <parameter name="env" value="local"/>   <!-- overrides staging here -->
    ...
  </test>
</suite>

Parameters in XML are always strings; TestNG converts them to the method's parameter types.

Seeing the structure

The first stand-in parses a suite description (using a small regular-expression tokenizer, not a real XML parser) and prints the tree. It uses the same shape as the XML above so you can compare them.

JS Parse a tiny suite description

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The printed tree shows two <test> buckets under one suite, with nested classes, methods and packages. The parallel="none" attribute on the suite is ignored by this stand-in but is a real TestNG attribute.

Selecting methods the way TestNG does

The second stand-in holds a registry of methods with their groups and applies the three ideas from this lesson: include/exclude groups, a per-class method list, and parameter scoping where a test-level value overrides the suite-level one.

JS Include, exclude and parameter scope

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Look at "Payments only": the method list asks for both chargesCard and refundsCard, but refundsCard belongs to the excluded group slow, so only one method runs. Excludes win over includes. The Smoke test shows env=local replacing the suite's staging while browser falls through from the suite.

Running with Maven

Maven's Surefire plugin runs TestNG tests automatically when TestNG is on the classpath. To run a particular suite file, point Surefire at it:

XML
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <suiteXmlFiles>
      <suiteXmlFile>src/test/resources/smoke.xml</suiteXmlFile>
    </suiteXmlFiles>
  </configuration>
</plugin>

You can make the file a property (<suiteXmlFile>${suite}</suiteXmlFile>) and choose at the command line with mvn test -Dsuite=src/test/resources/regression.xml. Without a suiteXmlFiles setting, Surefire builds a suite on its own from the classes it finds. You can also filter without XML: mvn test -Dgroups=fast -DexcludedGroups=slow. Details of Maven and Gradle are in Running Tests with Maven and Gradle.

Listeners and parallel settings

testng.xml is also where you attach listeners and turn on parallelism, both covered in Listeners, Retry and Parallel Execution:

XML
<suite name="Shop" parallel="classes" thread-count="4">
  <listeners>
    <listener class-name="shop.support.ScreenshotListener"/>
  </listeners>
  ...
</suite>

Reports

After a run TestNG writes a test-output folder (or target/surefire-reports under Maven). Open index.html for a browsable report, emailable-report.html for a single-file summary you can attach to an email, and testng-results.xml for tools that parse results. Surefire also writes JUnit-style TEST-*.xml files that CI servers display. Because testng.xml says exactly what ran, a failing CI job can be reproduced locally by passing the same file.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In testng.xml, what does a <test> element represent?

    A <test> groups classes (or packages) and gets its own lifecycle and report entry. Individual tests are the @Test methods inside the classes.
  2. 2. The same parameter is defined at suite level (staging) and test level (local). Which value does a method in that test receive?

    Parameters are scoped: class overrides test, test overrides suite.
  3. 3. Which Surefire setting selects a testng.xml file to run?

    suiteXmlFiles lists one or more suite files. excludedGroups filters by group instead.
  4. 4. A method belongs to the groups fast and flaky. The test excludes flaky and includes fast. Does it run?

    A method in any excluded group is removed, even when it also belongs to an included group.

Summary

  • testng.xml describes a run: <suite>, <test> buckets, <classes> or <packages>, optional <methods> and <groups>.
  • A <test> is a named bucket of classes, not a test method.
  • <parameter> values are strings, scoped at suite, test and class level, with the most specific winning.
  • Excluded groups beat included ones; method names in <include> and <exclude> can be regular expressions.
  • Maven runs a suite with suiteXmlFiles, or you can filter with -Dgroups and -DexcludedGroups.
  • Reports appear in test-output or target/surefire-reports.

Next: Listeners, Retry and Parallel Execution.