Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 41 ยท Testing and Quality

Running Tests with Surefire

Run JUnit tests from Maven with the Surefire plugin: dependency setup, naming patterns, -Dtest filters, skipTests versus maven.test.skip, and test reports.

A build that compiles but is not tested only proves that the code is valid Java. Maven runs your unit tests in the test phase of the default lifecycle, so mvn package and mvn install execute them automatically and stop if one fails. The plugin that does the work is Surefire.

This lesson shows how to add JUnit to a project, how Surefire decides which classes are tests, how to run one test or skip them all, and where to read the results. The real mvn commands need a Maven installation, so they are read-only here. The runnable parts are stand-ins in Java: a model of Surefire's name-pattern discovery and a model of how the two skip flags differ. They illustrate the rules; they are not Surefire itself. If you have not met phases yet, read Lifecycles and Phases first.

Add JUnit to the project

Surefire does not contain a testing framework. It finds one on the test classpath and picks the matching "provider". For JUnit 5 and later (the "Jupiter" API), add the junit-jupiter dependency with test scope, so it is on the classpath for tests but never ends up in your packaged application (see Dependency Scopes):

XML
<properties>
  <junit.version>6.0.0</junit.version>
</properties>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.junit</groupId>
      <artifactId>junit-bom</artifactId>
      <version>${junit.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>org.junit.jupiter</groupId>
    <artifactId>junit-jupiter</artifactId>
    <scope>test</scope>
  </dependency>
</dependencies>

The junit-bom import fixes all JUnit module versions together, so you declare the version once (Managing Versions explains the mechanism). Use a current JUnit release that supports your Java version; JUnit 5.x and 6.x both use the same org.junit.jupiter coordinates.

Surefire itself is already bound to the test phase by the jar packaging, so you do not need to declare it. You only declare it to choose a version or change settings. Use a recent 3.x release, because older releases (2.x) do not run the JUnit Platform:

XML
<build>
  <plugins>
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.5.4</version>
    </plugin>
  </plugins>
</build>

Tests live in src/test/java, mirroring the package of the class under test. A first test:

Java
package com.example.shop;

import static org.junit.jupiter.api.Assertions.assertEquals;
import org.junit.jupiter.api.Test;

class PriceTest {
    @Test
    void addsTax() {
        assertEquals(120, Price.withTax(100, 20));
    }
}

Running mvn test compiles main code, compiles test code, and runs every test class.

What Surefire does

Discover Finds test classes by file name.
Run Starts a forked JVM with your test classpath.
Report Writes results to target/surefire-reports.
Fail Stops the build when a test fails.
1 / 2

Which classes are tests?

By default Surefire runs classes whose file names match these patterns:

  • **/Test*.java
  • **/*Test.java
  • **/*Tests.java
  • **/*TestCase.java

** means "any folder depth". So PriceTest, TestPrice and PriceTests are picked up, while PriceSpec or PriceCheck are not, even if they contain @Test methods. Nested classes (names with $) are excluded by default; use JUnit's @Nested if you need them inside a test. The most common surprise for beginners is a test that "never runs": check the class name before anything else.

You can change the patterns:

XML
<configuration>
  <includes>
    <include>**/*Spec.java</include>
  </includes>
  <excludes>
    <exclude>**/Slow*.java</exclude>
  </excludes>
</configuration>

Abstract classes are never run directly, and classes with names like *IT belong to the integration-test plugin, which the next lesson covers.

Running only some tests

While developing you rarely want the whole suite. The -Dtest option filters by class name, and it accepts patterns and methods:

Terminal
mvn test -Dtest=PriceTest
mvn test -Dtest='PriceTest#addsTax'
mvn test -Dtest='Price*Test,TaxTest'
mvn test -Dtest='com.example.shop.*Test'

Quote values that contain #, * or commas so your shell does not interpret them. In a multi-module build, a module that contains no matching test fails with "No tests matching pattern", because you asked for something that did not exist. Add -Dsurefire.failIfNoSpecifiedTests=false to let those modules pass.

JUnit tags give you a second filter. Mark tests with @Tag("fast") or @Tag("slow"), then select them:

Terminal
mvn test -Dgroups=fast
mvn test -DexcludedGroups=slow

Tags are better than file names for categories such as "needs a database", because a test can change category without being renamed.

Skipping tests: two flags, two behaviors

There are two skip switches, and they are often confused:

Flag Compiles test code? Runs tests? Typical use
-DskipTests yes no "I just want the jar quickly"
-Dmaven.test.skip=true no no Test code is broken or you do not want it compiled

With -DskipTests, your test classes still compile, so a compile error in src/test still fails the build. With -Dmaven.test.skip=true, both testCompile and test are skipped, which is faster but has a side effect: test classes and test resources are not built, and a module that publishes a test-jar ends up without one. Prefer -DskipTests as the default.

Skipping tests on your own machine to save time is fine. Skipping them in the pipeline that produces a release is not. If a build only passes with -DskipTests, the build is not passing.

Reports and failures

Surefire writes results to target/surefire-reports/: a plain-text summary per class (*.txt) and a machine-readable XML file per class (TEST-*.xml) that CI systems read to show test results. The console prints a summary like this:

TEXT
[INFO] Tests run: 12, Failures: 1, Errors: 0, Skipped: 0
[ERROR] Failures:
[ERROR]   PriceTest.addsTax:9 expected: <120> but was: <119>
[INFO] BUILD FAILURE

"Failures" are failed assertions; "Errors" are unexpected exceptions. Either stops the build before package. When you want the build to continue and fail at the end (for example to collect every module's results in CI), use -Dmaven.test.failure.ignore=true, and make sure something later in the pipeline looks at the reports. Another useful setting is rerunFailingTestsCount, which retries a failing test and reports it as "flaky" if it passes on retry. Use it to find flaky tests, not to hide them.

Two more settings come up often. argLine passes options to the forked test JVM (a memory limit or -ea, for example). And forkCount and reuseForks control how many JVMs run tests, which matters for speed and for tests that leak state.

Stand-in: test discovery by name pattern

The program below models the discovery rule. It converts the Maven-style patterns into regular expressions and applies includes, excludes and the nested-class rule to a list of file names. The real Surefire works on compiled class files and has more options, but the matching idea is the same.

Java Which classes does Surefire pick up?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

OrderSpec is not selected because its name matches no pattern, even though it might contain perfectly good tests. CartIT is left to the integration-test plugin. PriceTest$Helper is a nested class and is excluded. To run OrderSpec you would add **/*Spec.java to includes.

Stand-in: the two skip flags

Now the difference between -DskipTests and -Dmaven.test.skip=true. The model walks through the test-related phases and reports what each flag does to them.

Java skipTests versus maven.test.skip

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The middle case still compiles the tests, so a typo in a test file would break the build. The last case never looks at src/test. That is also why a library that shares test helpers through a test-jar should avoid maven.test.skip.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A class named OrderSpec contains @Test methods but never runs under the default Surefire setup. Why?

    Surefire selects test classes by file name. Add an include such as **/*Spec.java or rename the class.
  2. 2. What is the difference between -DskipTests and -Dmaven.test.skip=true?

    skipTests only skips execution. maven.test.skip skips test compilation and execution, so no test classes are produced.
  3. 3. Which command runs one test method?

    -Dtest accepts Class#method. Quote it so the shell does not treat # as a comment.
  4. 4. Where are the XML test results that CI systems read written?

    Surefire writes TEST-*.xml and .txt files under target/surefire-reports.
  5. 5. Why give the JUnit dependency the test scope?

    Test scope keeps testing libraries out of the compile classpath and out of the shipped artifact.

Summary

  • Surefire runs unit tests in the test phase and fails the build when a test fails.
  • Add org.junit.jupiter:junit-jupiter with test scope (versions managed by the junit-bom) and use a 3.x Surefire.
  • Default test names: Test*, *Test, *Tests, *TestCase; change them with includes and excludes.
  • -Dtest=Class#method selects tests; -Dgroups and -DexcludedGroups select by JUnit tag.
  • -DskipTests skips running; -Dmaven.test.skip=true also skips compiling tests.
  • Read results in target/surefire-reports; do not skip tests in release pipelines.

Next: Integration Tests with Failsafe