Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 29 ยท Real-world projects

Project: Testing a REST API Suite with TestNG

Design a TestNG suite for an HTTP API: data providers for many inputs, groups for smoke runs, parallel execution, listeners and reports, with a runnable Node stand-in.

API testing is where TestNG's extra features pay off. An HTTP API has many endpoints and many inputs per endpoint, and you want to run the cheap checks on every commit and the slow ones nightly. TestNG answers each of those needs directly: data providers feed many inputs into one test method, groups select subsets, parallel settings use your cores, and listeners turn results into logs, screenshots or dashboards.

In this project you will design that suite for a small user service. The TestNG code and XML are read-only here, because our site cannot run TestNG or a real HTTP service you control. To make the ideas tangible, every section also has a runnable Node stand-in: a real HTTP server on the loopback address, a fetch-based client, and a tiny runner that mimics data providers, groups, parallel workers and listeners. Where the stand-in differs from TestNG, we say so. You should know the basics from TestNG Annotations and Lifecycle, Assertions and Groups and Data Providers and Factories.

The service and the plan

The API under test

GET /users/{id} One user, or 404.
GET /users The whole list.
POST /users Create, or 400 for bad input.
1 / 2

A good API suite has three rules:

  1. Tests must not depend on each other. Parallel runs and group filtering both break order assumptions. Each test creates the data it needs, with a unique name.
  2. Assert on contracts, not accidents. Check status codes, required fields and error messages. Do not assert on auto-generated ids when tests run in parallel, because the order of creation decides them.
  3. The base URL is configuration. The same suite should hit a local server, a test environment and a staging environment.

Part 1: data providers

The classic API test is a table: request in, status and field out. With a data provider you write the table once. This is the read-only TestNG version, using the JDK's own HttpClient to avoid extra dependencies:

Java
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Parameters;
import org.testng.annotations.Test;
import static org.testng.Assert.assertEquals;
import static org.testng.Assert.assertTrue;

public class UserLookupTest {
    private final HttpClient client = HttpClient.newHttpClient();
    private String baseUrl;

    @BeforeClass(alwaysRun = true)
    @Parameters("baseUrl")
    public void configure(String baseUrl) {
        this.baseUrl = baseUrl;
    }

    @DataProvider(name = "lookups")
    public Object[][] lookups() {
        return new Object[][] {
            // id, expected status, expected name (null when not applicable)
            { 1,  200, "Ada" },
            { 2,  200, "Linus" },
            { 99, 404, null },
        };
    }

    @Test(groups = {"smoke", "read"}, dataProvider = "lookups")
    public void lookupReturnsExpectedStatus(int id, int status, String name) throws Exception {
        HttpRequest request = HttpRequest.newBuilder(URI.create(baseUrl + "/users/" + id)).build();
        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());

        assertEquals(response.statusCode(), status);          // TestNG: actual first, expected second
        if (name != null) {
            assertTrue(response.body().contains("\"name\":\"" + name + "\""));
        }
    }
}

Two TestNG details matter. First, Assert.assertEquals(actual, expected) takes the actual value first, the reverse of JUnit. Second, TestNG reports each row as a separate invocation of the same method, so a failing row is identified by its parameters rather than by a method name.

The stand-in below starts a real HTTP server on the loopback address with an OS-assigned port, then runs the same table through a mini data-provider runner. The last row is wrong on purpose (it expects 200 for a user that does not exist), so you can see how a failing row reads.

Node.js Loopback API and a data-driven runner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

What is real here: the HTTP server, the requests and the response codes. What is the stand-in: the table loop, which replaces TestNG's @DataProvider machinery and its report. In real TestNG you can also mark a provider parallel = true, and the suite attribute data-provider-thread-count controls how many rows run at once.

Part 2: groups, parallel runs and listeners

Now add write tests and labels. In TestNG you tag methods with groups, then choose what to run in testng.xml. A common scheme:

  • smoke: a handful of fast checks that prove the service is alive. Run on every commit.
  • read and write: functional tests, split by whether they change data.
  • slow: anything heavy. Run nightly.
Java
public class UserCreateTest {
    @Test(groups = {"write"})
    public void createdUserCanBeReadBack() { /* POST, then GET the Location */ }

    @Test(groups = {"write", "negative"})
    public void blankNameIsRejectedWithBadRequest() { /* POST {"name": ""} expects 400 */ }
}

The suite file selects groups and turns on parallelism. Attribute names below are the standard ones; check them against the TestNG version you use.

XML
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="User API" parallel="methods" thread-count="4" data-provider-thread-count="3">
  <parameter name="baseUrl" value="http://localhost:8080"/>
  <listeners>
    <listener class-name="com.shop.api.ResultLogger"/>
  </listeners>
  <test name="Smoke">
    <groups>
      <run>
        <include name="smoke"/>
      </run>
    </groups>
    <classes>
      <class name="com.shop.api.UserLookupTest"/>
      <class name="com.shop.api.UserCreateTest"/>
    </classes>
  </test>
</suite>

A listener is a class that TestNG calls as results come in. This one writes one line per test and could just as easily post to a chat channel:

Java
import org.testng.ITestListener;
import org.testng.ITestResult;

public class ResultLogger implements ITestListener {
    @Override
    public void onTestSuccess(ITestResult result) {
        System.out.println("PASS " + result.getMethod().getMethodName());
    }

    @Override
    public void onTestFailure(ITestResult result) {
        System.out.println("FAIL " + result.getMethod().getMethodName()
                + " -> " + result.getThrowable().getMessage());
    }
}

When the run ends, TestNG writes reports into a test-output folder, including an HTML index and an XML results file that CI systems can read (see Listeners, Retry and Parallel Execution). Your build tool decides where the XML goes; with Maven, Surefire can run a named suite file through its suiteXmlFiles setting (Running Tests with Maven and Gradle).

The next stand-in is a miniature of all three ideas. It registers tests with groups, filters by the group you ask for, runs the survivors on a pool of three workers, and tells a listener about each result. The listener's lines are sorted by test name at the end, because parallel tests finish in unpredictable order and a stable report is easier to compare.

Node.js Groups, a worker pool and a listener

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the output. The first run picked only tests tagged smoke; the second picked everything tagged write or read. deliberatelyBrokenCheck fails on purpose: it expects 200 while the API correctly answers 201 Created. The first run has only two tests, so its peak is 2. The peak-parallelism figure is what thread-count would limit in TestNG. In the real library, the choice between parallel="methods", "classes" and "tests" decides what is allowed to run side by side, and the shared HttpClient field in the TestNG class above is safe only because HttpClient is thread-safe. Any mutable field that tests share is a data race waiting to happen, which is why the stand-in's tests keep all their state in local variables and compare names, not ids.

Design checklist for a real API suite

  • Fixtures through the API. A test that needs a user creates one, with a unique name (a UUID suffix works), instead of relying on rows left in a database.
  • Environment through parameters. @Parameters("baseUrl") or a system property; never hard-code a host.
  • A tiny smoke group. If smoke takes more than a minute, people stop running it.
  • Retries only for known-flaky infrastructure. IRetryAnalyzer can re-run a failed test, but automatic retries hide real problems; log every retry (see Flaky Tests and Common Mistakes).
  • Contract checks over exact bodies. Compare the fields you care about; ignore timestamps and ordering unless they are part of the promise.
  • Cleanup in @AfterClass(alwaysRun = true), so a failing test does not leave data behind.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. In TestNG's Assert.assertEquals, which argument comes first?

    TestNG uses assertEquals(actual, expected), the reverse of JUnit's (expected, actual). Mixing them only reverses the failure message, but it misleads the reader.
  2. 2. Why should parallel API tests avoid asserting on auto-generated ids?

    With parallel creation the order, and therefore the ids, vary from run to run. Assert on values the test supplied, such as the name.
  3. 3. What does a group filter in testng.xml achieve?

    Including smoke selects only methods whose groups contain smoke, which lets one codebase serve both quick commit checks and full nightly runs.
  4. 4. What is a good use of an ITestListener in an API suite?

    Listeners observe results as they happen. They do not decide pass or fail; the assertions in the tests do.
  5. 5. Which statement about the Node code in this lesson is accurate?

    The server listens on the loopback address and fetch really calls it, but the data-provider loop, group filter and worker pool only imitate TestNG's behavior.

Summary

  • Data providers turn a table of requests and expected results into many reported invocations of one test method.
  • Groups (smoke, read, write) let one codebase serve fast and full runs; testng.xml selects them.
  • Parallel settings (parallel, thread-count, data-provider-thread-count) need tests that share no mutable state.
  • Listeners observe each result and feed logs, chat messages or dashboards; reports land in test-output and an XML file for CI.
  • TestNG's assertEquals(actual, expected) is the reverse of JUnit's argument order.
  • The Node examples use a real loopback HTTP server but imitate the TestNG runner; the Java and XML are read-only.

Next: Practice: Exercises with Solutions, twelve graded exercises to test everything in this track.