Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 29 ยท Tooling and collaborators

Running Tests with Maven and Gradle

Run JUnit and TestNG tests with Maven Surefire, Failsafe and the Gradle test task: filters, tags, parallel forks and how to read the XML reports.

Pressing the green arrow in your IDE is fine while you write code, but the tests that matter run somewhere else: on a build server, on a teammate's machine, before a release. That "somewhere else" is almost always Maven or Gradle. They compile your tests, hand them to the JUnit Platform (or TestNG), collect the results into report files, and decide whether the build passes.

Maven and Gradle are not something our site can run, so every build file in this lesson is read-only reference. To keep the ideas hands-on, the two runnable examples are honest stand-ins: a small parser for the XML report format that both tools write, and a matcher that behaves like the test-name filters you pass on the command line. They are not Maven or Gradle, but the formats and rules they model are real.

You already ran real JUnit tests through the launcher in Your First JUnit Test. Build tools do the same thing, plus discovery by file name, forking and reporting. For a general introduction to the build tools themselves see Maven and Gradle in Advanced Java.

Who runs what

A test run has four players

Build tool Maven or Gradle compiles and orchestrates.
Plugin Surefire (Maven) or the test task (Gradle).
Test engine Jupiter, TestNG or Vintage runs the code.
Reports XML files that CI servers read.
1 / 2

Maven: Surefire for unit tests

Maven does nothing with tests by itself. The Surefire plugin is bound to the test phase, so mvn test (and anything later, such as mvn package) runs it. For JUnit Jupiter you add the dependency with test scope and keep Surefire reasonably current (3.x understands the JUnit Platform without extra configuration):

XML
<dependencies>
  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <version>${junit.version}</version>
    <scope>test</scope>
  </dependency>
</dependencies>

<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>${surefire.version}</version>
    </plugin>
  </plugins>
</build>

Surefire picks classes by name, not by looking for @Test. The defaults are **/Test*.java, **/*Test.java, **/*Tests.java and **/*TestCase.java. A class called CartChecks with perfectly good tests is silently skipped. Either follow the naming convention or override the patterns:

XML
<plugin>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <includes>
      <include>**/*Test.java</include>
      <include>**/*Spec.java</include>
    </includes>
  </configuration>
</plugin>

TestNG with Surefire

Surefire chooses the provider from your dependencies. With org.testng:testng on the test classpath and no JUnit, it runs TestNG. You can point it at a suite file (see testng.xml, Suites and Test Configuration):

XML
<configuration>
  <suiteXmlFiles>
    <suiteXmlFile>src/test/resources/testng.xml</suiteXmlFile>
  </suiteXmlFiles>
</configuration>

If both JUnit and TestNG are present, say which one you mean explicitly, or the choice can surprise you. Our JUnit vs TestNG lesson covers running both.

Maven: Failsafe for integration tests

Integration tests (database, HTTP, containers) are slower and need things to be started first. The Failsafe plugin exists for them. It looks for *IT.java, IT*.java and *ITCase.java, runs in the integration-test phase, and, importantly, does not fail the build immediately. It records the result and lets the post-integration-test phase tear down containers or servers, then the verify goal decides pass or fail.

XML
<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>${failsafe.version}</version>
  <executions>
    <execution>
      <goals>
        <goal>integration-test</goal>
        <goal>verify</goal>
      </goals>
    </execution>
  </executions>
</plugin>

The practical rule: run mvn test while you work (only Surefire, fast), and mvn verify in CI (both). If you use mvn integration-test and your server stays running, you have skipped the teardown phase; verify is the one to remember.

Gradle: the test task

Gradle's built-in test task runs your unit tests, but it defaults to JUnit 4 style discovery unless you ask for the JUnit Platform. The one line people forget is useJUnitPlatform():

Kotlin
dependencies {
    testImplementation(platform("org.junit:junit-bom:${junitVersion}"))
    testImplementation("org.junit.jupiter:junit-jupiter")
    testRuntimeOnly("org.junit.platform:junit-platform-launcher")
}

tasks.test {
    useJUnitPlatform()
    testLogging {
        events("passed", "skipped", "failed")
        showStandardStreams = false
    }
}

For TestNG the call is useTestNG(), optionally with a suite file:

Kotlin
tasks.test {
    useTestNG {
        suites("src/test/resources/testng.xml")
    }
}

Without useJUnitPlatform() (or useTestNG()), a Jupiter project finds zero tests and happily reports "BUILD SUCCESSFUL". When a test count of zero looks suspicious, this is the first thing to check.

Choosing which tests run

Rerunning the whole suite to check one fix is slow. Both tools let you narrow the run from the command line. These are read-only examples:

Terminal
# Maven (Surefire)
mvn test -Dtest=CartTest                     # one class
mvn test -Dtest=CartTest#addsItem            # one method
mvn test -Dtest='Cart*Test,!CartSlowTest'    # patterns, with an exclusion
mvn test -Dgroups=fast                       # JUnit tag expression (or TestNG groups)
mvn test -Dsurefire.failIfNoSpecifiedTests=false

# Gradle
gradle test --tests "com.shop.CartTest"
gradle test --tests "com.shop.CartTest.addsItem"
gradle test --tests "*Cart*"

Tags and groups are better for categories, such as "fast" and "slow", and they work in the build file too. For JUnit, Maven uses <groups> and <excludedGroups> inside the Surefire configuration, and Gradle uses includeTags and excludeTags:

Kotlin
tasks.test {
    useJUnitPlatform {
        includeTags("fast")
        excludeTags("slow")
    }
}

tasks.register<Test>("slowTests") {
    useJUnitPlatform { includeTags("slow") }
}

The next stand-in models the name patterns. It is a simplified matcher, not Surefire's real implementation, but it follows the rules you use daily: comma-separated patterns, * as a wildcard, Class#method, and a leading ! to exclude.

JS A stand-in for -Dtest filters

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check the output against your intuition: *Test matches CartTest, CartSlowTest and CheckoutTest but not CheckoutIT, because the name must end in Test. That is exactly why a naming convention matters so much to Surefire.

Parallel runs and forks

Two different things are called "parallel", and mixing them up causes flaky tests:

  • Forks run test classes in separate JVM processes. Memory is not shared, so tests are isolated from each other's static state. Maven: <forkCount> and <reuseForks>. Gradle: maxParallelForks.
  • Threads in one JVM run tests concurrently within a process. JUnit has its own switch (junit.jupiter.execution.parallel.enabled=true in junit-platform.properties); TestNG has parallel="methods" in its suite. Everything in the same JVM now shares static fields, so tests must be thread-safe (see Listeners, Retry and Parallel Execution).
XML
<!-- Maven: up to 4 JVMs, each reused for several test classes -->
<configuration>
  <forkCount>4</forkCount>
  <reuseForks>true</reuseForks>
</configuration>
Kotlin
// Gradle: use half the cores, and give each fork its own memory budget
tasks.test {
    maxParallelForks = (Runtime.getRuntime().availableProcessors() / 2).coerceAtLeast(1)
    maxHeapSize = "512m"
}

Forks are the safer first step: they speed up suites with many classes and cannot corrupt each other's statics. Thread-level parallelism is faster but is where flaky tests are born.

Reading the reports

Each class gets an XML file: Maven writes target/surefire-reports/TEST-<class>.xml (and Failsafe target/failsafe-reports), Gradle writes build/test-results/test/TEST-<class>.xml and an HTML page under build/reports/tests/test/. The XML follows the long-established JUnit report schema, which is why Jenkins, GitHub Actions summaries, GitLab and IDEs all understand it.

The attributes to know: on <testsuite>, tests, failures (assertion failures), errors (unexpected exceptions) and skipped; on each <testcase>, name, classname, time, and a child <failure>, <error> or <skipped> element when it did not pass. The stand-in below reads a small report and prints a summary that mimics what a CI dashboard shows.

JS Parse a Surefire-style XML report

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Note the split between failure and error, the same distinction you met in the first lesson: a failed assertion versus an unexpected exception. Many dashboards add the two numbers together, but the report format keeps them apart.

Troubleshooting checklist

  • "No tests were executed" / 0 tests run. Maven: the class name does not match the Surefire pattern, or the Jupiter dependency is missing. Gradle: forgot useJUnitPlatform(). Also check that your annotation import is org.junit.jupiter.api.Test, not the JUnit 4 one.
  • Tests pass in the IDE but fail in the build. The IDE may use a different working directory, JDK or classpath. Run mvn test or gradle test from a clean checkout to see what CI sees.
  • Integration tests never ran. mvn test skips Failsafe. Use mvn verify.
  • A flaky test only fails when run with others. Suspect shared static state or a fork reused across classes (reuseForks). Try forkCount with reuseForks=false to confirm.
  • Output is hidden. Both tools show only failing tests' output by default. In Gradle use testLogging; in Maven look in the -output.txt files in the reports directory.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A Maven project has a class CartChecks with working @Test methods, and mvn test runs none of them. What is the most likely cause?

    Surefire selects classes by file name patterns (Test*, *Test, *Tests, *TestCase). A class called CartChecks is skipped unless you add an include pattern.
  2. 2. Which Gradle line makes the test task run JUnit Jupiter tests?

    useJUnitPlatform() tells Gradle to launch tests through the JUnit Platform. Without it Gradle can find zero Jupiter tests and still report success.
  3. 3. Why does Maven provide the Failsafe plugin in addition to Surefire?

    Failsafe records failures during integration-test, lets post-integration-test clean up servers or containers, and verify then fails the build.
  4. 4. What is the main difference between forks and in-JVM parallel threads?

    Separate JVMs cannot corrupt each other's static fields. Threads inside one JVM are faster but require thread-safe tests.
  5. 5. In a report, how does errors="1" differ from failures="1"?

    Failures are assertion mismatches, errors are exceptions the test did not expect. Many dashboards add them together, but the XML keeps them apart.

Summary

  • Maven runs unit tests with Surefire (mvn test) and integration tests with Failsafe (mvn verify); Gradle uses the test task with useJUnitPlatform() or useTestNG().
  • Surefire finds classes by name pattern, so naming conventions are part of the build.
  • Narrow a run with -Dtest=Class#method, Gradle's --tests, or tags and groups for whole categories.
  • Forks isolate JVMs; threads inside one JVM need thread-safe tests.
  • Reports are XML files (TEST-*.xml) that every CI server understands; failures are assertion mismatches and errors are unexpected exceptions.
  • The examples that run on this page are stand-ins for the filter and the report format; the Maven and Gradle files are real, read-only references.

Next: Coverage, Mutation Testing and CI, where the same build runs on a server and measures how much of your code the tests really check.