Skip to content
Tutorials Center
My learning

Topics

Lesson 11 of 29 ยท TestNG core

TestNG Assertions and Groups

Learn TestNG's Assert and SoftAssert classes, why assertEquals argument order differs from JUnit, and how groups, include and exclude rules and meta-groups select tests.

Two things make a TestNG test suite feel different from a JUnit one in daily use: the assertion classes, and groups. The assertions look familiar but hide a trap: the argument order of assertEquals is the opposite of JUnit's, so a failure message can tell you the reverse of the truth. Groups are TestNG's built-in way to label tests (smoke, slow, api) and run any combination of labels, without moving code around.

The real TestNG library cannot run in your browser, so the TestNG code in this lesson is read-only. Beside it you will run small JavaScript stand-ins that implement the same ideas: a hard-assert versus soft-assert comparison, and a group filter with include, exclude and meta-group rules. They show how the mechanism works; they are not TestNG itself.

You should know the basic annotations from Your First TestNG Test and the lifecycle from TestNG Annotations and Lifecycle.

Hard or soft?

Assert Stops the test at the first failure.
SoftAssert Collects failures, reports them at assertAll().

Forget assertAll() and nothing ever fails.

1 / 2

The Assert class

TestNG's org.testng.Assert is a class of static methods, much like JUnit's Assertions. The most used ones:

Java
import org.testng.Assert;

Assert.assertEquals(actual, expected);
Assert.assertEquals(actual, expected, "message shown on failure");
Assert.assertEquals(3.14159, Math.PI, 0.001);          // doubles need a delta
Assert.assertTrue(list.isEmpty(), "list should start empty");
Assert.assertFalse(user.isLocked());
Assert.assertNotNull(order.getId());
Assert.assertSame(first, second);                        // same object (==)
Assert.assertEquals(actualList, expectedList);           // collections: compared element by element
Assert.assertEqualsNoOrder(actualArray, expectedArray);  // same elements, any order
Assert.assertThrows(IllegalStateException.class, () -> service.cancel(order));
IllegalStateException ex =
    Assert.expectThrows(IllegalStateException.class, () -> service.cancel(order));  // returns the exception

The argument order trap

In TestNG the actual value comes first and the expected value second. In JUnit Jupiter it is the other way around: expected first, actual second.

Value order Message position
TestNG Assert.assertEquals (actual, expected, message) last
JUnit Jupiter Assertions.assertEquals (expected, actual, message) last
JUnit 4 Assert.assertEquals (message, expected, actual) first

Because both are Object or int parameters, swapping the arguments compiles fine and the test still passes or fails correctly. Only the failure message is affected: it says expected [X] but found [Y], with X and Y taken from the wrong arguments, which sends you looking at the wrong value. When you migrate tests between the frameworks (see Migrating from JUnit 4 and Between Frameworks), swapping the order is a classic source of misleading failures. Pick one habit per project and stick to it.

Soft assertions

A normal assertion throws as soon as it fails, so the rest of the test never runs. That is usually what you want, because a failed precondition makes later checks meaningless. But when you check many independent properties of one object (every field of a response, every cell of a table), seeing only the first failure forces you to fix and rerun repeatedly.

SoftAssert records failures instead of throwing, and reports all of them when you call assertAll():

Java
import org.testng.asserts.SoftAssert;

@Test
public void profileIsComplete() {
    Profile p = service.load("ada");
    SoftAssert soft = new SoftAssert();
    soft.assertEquals(p.name(), "Ada Lovelace", "name");
    soft.assertEquals(p.country(), "UK", "country");
    soft.assertTrue(p.isVerified(), "verified");
    soft.assertAll();    // MUST be called: throws one error listing every failure
}

Two rules: always end with assertAll(), because without it all recorded failures are silently dropped and the test passes; and create the SoftAssert object per test (a field shared between tests or threads would mix their failures). Use soft assertions for grouped checks, not for steps where a later line depends on an earlier one succeeding.

The stand-in below implements both styles in a few lines, so you can see the difference in the failure output.

JS Stand-in: hard asserts versus SoftAssert

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The third test passes although its only check is wrong. That is exactly the bug that a forgotten assertAll() creates in real TestNG too, so some teams wrap soft assertions in a helper that always calls it, or enable a lint rule.

Groups

A group is just a string label on a test. Put any number of labels on a method (or on a whole class) with the groups attribute:

Java
public class CheckoutTest {

    @Test(groups = {"smoke", "fast"})
    public void emptyCartHasZeroTotal() { /* ... */ }

    @Test(groups = {"api"})
    public void orderIsStoredInTheDatabase() { /* ... */ }

    @Test(groups = {"slow", "api"})
    public void exportsAFullYearOfOrders() { /* ... */ }

    @Test   // no group at all
    public void totalIsRounded() { /* ... */ }
}

You then choose which groups run in testng.xml. include says "only methods with this label"; exclude removes methods with a label. The rule that surprises people: exclude wins over include. A method in both an included and an excluded group does not run.

XML
<suite name="Checkout">
  <test name="fast checks">
    <groups>
      <run>
        <include name="smoke"/>
        <include name="api"/>
        <exclude name="slow"/>
      </run>
    </groups>
    <classes>
      <class name="shop.CheckoutTest"/>
    </classes>
  </test>
</suite>

With this file, emptyCartHasZeroTotal and orderIsStoredInTheDatabase run. exportsAFullYearOfOrders is in api (included) but also slow (excluded), so it is dropped. totalIsRounded has no group, so it is not selected at all. Once any <include> is present, TestNG runs only methods that belong to an included group; methods without a group are ignored. If the <groups> element has no includes, only the excludes apply.

Build tools can pass the same choice without editing XML. With Maven Surefire the property -Dgroups=smoke runs one group and -DexcludedGroups=slow skips another; the exact setup is in Running Tests with Maven and Gradle.

Meta-groups

A meta-group is a named bundle of other groups, defined in testng.xml with <define>:

XML
<groups>
  <define name="regression">
    <include name="smoke"/>
    <include name="api"/>
  </define>
  <run>
    <include name="regression"/>
    <exclude name="slow"/>
  </run>
</groups>

Now regression means "smoke or api", and the rest of the file can refer to it. Nested definitions are allowed, so you can build nightly from regression plus slow. Meta-groups keep the knowledge of "what makes up a regression run" in one place.

Group setup

@BeforeGroups("api") and @AfterGroups("api") run once before the first and after the last method of that group, which suits expensive setup used only by those tests (an HTTP server, a test database). Keep group names few and consistent: a typo such as smok creates a new group silently and the test quietly drops out of your smoke run.

Stand-in: a group filter

The next stand-in implements the selection rules: methods carry labels, a define map expands meta-groups, include narrows the set, and exclude removes methods afterwards. Change the rules and see which methods remain.

JS Stand-in: include, exclude and meta-groups

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice case 3: with only an exclude rule, ungrouped methods such as totalIsRounded still run. In case 5 a typo leaves nothing to run, which TestNG would also treat as an empty selection, a strong hint to check group names when a run suddenly looks too fast.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In TestNG, which call is written the TestNG way for 'the method returned 5 and we expected 7'?

    TestNG takes the actual value first and the expected value second. The reverse order (JUnit Jupiter's) still compiles but gives a failure message with the two values swapped.
  2. 2. What happens if a test creates a SoftAssert, makes three failing soft checks, and never calls assertAll()?

    SoftAssert only records failures. assertAll() is what throws them. Without it, the test passes silently.
  3. 3. A method has groups = {"api", "slow"}. The XML includes api and excludes slow. Does it run?

    When a method belongs to both an included and an excluded group, the exclusion applies and the method is not run.
  4. 4. What is a meta-group?

    <define name="regression"> lists other groups, and including regression includes all of them. It keeps your selection logic in one place.

Summary

  • Assert (hard) stops at the first failure. SoftAssert collects failures and throws them all at assertAll(), which you must not forget.
  • TestNG's assertEquals is (actual, expected); JUnit Jupiter's is (expected, actual). Swapping them only garbles the failure message, so be consistent.
  • Label tests with @Test(groups = ...), choose them with <include> and <exclude> in testng.xml or with build-tool properties.
  • Exclude beats include. Once any include exists, ungrouped methods are not selected.
  • <define> creates meta-groups; @BeforeGroups and @AfterGroups do group-level setup. Watch for typos in group names.

Next: Dependencies, Priority and Test Ordering.