Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 29 ยท Tooling and collaborators

Coverage, Mutation Testing and CI

Measure what your JUnit and TestNG tests execute with JaCoCo, check whether they notice bugs with PIT mutation testing, and gate builds in CI.

"How much of the code do your tests cover?" is the question every team asks, and the number you get back is easy to misread. Code coverage tells you which lines your tests ran. It does not tell you whether the tests would notice if those lines were wrong. A second technique, mutation testing, answers that harder question by deliberately breaking your code and seeing whether a test turns red.

The real tools here (JaCoCo, PIT, GitHub Actions) need a build tool and a server, so their configuration appears as read-only blocks. The runnable examples are stand-ins that implement the ideas in a few dozen lines of JavaScript: one counts executed lines and mutants, the other checks a coverage report against minimum rules the way a build gate does. They show how the measurements work, not how JaCoCo or PIT are written.

What coverage measures

Four things people call coverage

Line Was this line executed at all?
Branch Did both sides of each if run?
Instruction Which bytecode instructions ran?
Complexity Were all paths through a method exercised?
1 / 2

JaCoCo is the standard coverage tool for the JVM. It attaches a Java agent to the test JVM, records which bytecode instructions ran, and later produces HTML, XML and CSV reports. It works with JUnit and TestNG alike because it watches the code, not the framework. Its counters are instruction, branch, line, complexity, method and class.

Line coverage can hide a lot. Consider if (express) cost = cost * 2; written on one line: any test that reaches the line counts it as covered, even if express is never true. Branch coverage is stricter: it asks whether both outcomes of every condition ran. Prefer branch numbers when you want an honest picture.

JaCoCo in Maven and Gradle

These configurations are read-only. In Maven you wire two things: the agent, which prepare-agent sets up before tests run, and the report, which is produced after.

XML
<plugin>
  <groupId>org.jacoco</groupId>
  <artifactId>jacoco-maven-plugin</artifactId>
  <version>${jacoco.version}</version>
  <executions>
    <execution>
      <id>prepare</id>
      <goals><goal>prepare-agent</goal></goals>
    </execution>
    <execution>
      <id>report</id>
      <phase>verify</phase>
      <goals><goal>report</goal></goals>
    </execution>
    <execution>
      <id>gate</id>
      <phase>verify</phase>
      <goals><goal>check</goal></goals>
      <configuration>
        <rules>
          <rule>
            <element>BUNDLE</element>
            <limits>
              <limit>
                <counter>LINE</counter>
                <value>COVEREDRATIO</value>
                <minimum>0.80</minimum>
              </limit>
              <limit>
                <counter>BRANCH</counter>
                <value>COVEREDRATIO</value>
                <minimum>0.70</minimum>
              </limit>
            </limits>
          </rule>
        </rules>
      </configuration>
    </execution>
  </executions>
</plugin>

After mvn verify the HTML report is under target/site/jacoco/index.html, with green, yellow and red highlighting per line. Yellow means a branch is only half covered, which is where bugs like to hide. In Gradle the equivalent is the jacoco plugin with the jacocoTestReport and jacocoTestCoverageVerification tasks:

Kotlin
plugins {
    java
    jacoco
}

tasks.test {
    useJUnitPlatform()
    finalizedBy(tasks.jacocoTestReport)
}

tasks.jacocoTestCoverageVerification {
    violationRules {
        rule {
            limit {
                counter = "LINE"
                value = "COVEREDRATIO"
                minimum = "0.80".toBigDecimal()
            }
        }
    }
}

tasks.check {
    dependsOn(tasks.jacocoTestCoverageVerification)
}

The check rules are what make coverage a gate: when a number drops below the minimum, the build fails. The stand-in below reads a report in JaCoCo's CSV layout (shortened to a few columns) and applies bundle-level and per-class rules. Look at how a good overall number can still hide one weak class.

JS A stand-in for the coverage gate

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The bundle passes at 80.7% while Checkout sits at 40%. A single global number is the weakest rule; a per-class floor catches neglected corners.

Choosing a target that means something

  • Do not chase 100%. The last 10% is usually getters, logging and defensive code. Teams that demand it write tests with no assertions just to turn lines green.
  • Pick a floor, not a goal. Something like 70 to 80% line and 60 to 70% branch is a typical guard against regression. The point is that coverage must not drop, so many teams gate on "no worse than the main branch".
  • Exclude noise on purpose. Generated code, plain data classes and main methods can be excluded in the JaCoCo configuration. Write down why.
  • Read the red lines, not the percentage. The useful output of a report is the list of untested decisions in important code.
  • Never reward the number. If bonuses depend on coverage, tests will be written for the metric. Treat it as a smoke alarm.

Mutation testing: do the tests notice?

Take this thought experiment. Your suite has 100% coverage. Now change > to >= somewhere in the code. If no test fails, the suite never checked that boundary, even though it ran the line. Mutation testing automates the experiment. A tool such as PIT (PITest) creates many small variants of your compiled code, called mutants, by changing a condition, swapping an operator, removing a call or altering a return value. It reruns the relevant tests for each mutant:

  • If a test fails, the mutant is killed. Good: your tests notice that kind of bug.
  • If everything stays green, the mutant survived. Your tests have a blind spot.
  • If no test even reaches the changed line, it has no coverage.

The mutation score is killed mutants divided by all mutants. Some survivors are equivalent mutants, changes that do not alter behavior (for example i < n to i != n in a loop that can never skip), and these are unavoidable noise. PIT is slower than a normal test run, so teams usually run it on changed code or nightly rather than on every commit.

A real setup is read-only here:

XML
<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>${pitest.version}</version>
  <dependencies>
    <dependency>
      <groupId>org.pitest</groupId>
      <artifactId>pitest-junit5-plugin</artifactId>
      <version>${pitest.junit5.version}</version>
    </dependency>
  </dependencies>
  <configuration>
    <targetClasses>
      <param>com.shop.*</param>
    </targetClasses>
    <mutationThreshold>70</mutationThreshold>
  </configuration>
</plugin>

Run it with mvn org.pitest:pitest-maven:mutationCoverage. The report appears in target/pit-reports, listing each surviving mutant next to the line it changed. TestNG users need no JUnit plugin, since PIT supports TestNG out of the box.

The next example is our own miniature of both measurements. It compiles a small shipping function, counts which lines executed, then applies six hand-written mutations. Two suites run: a smoke suite with a single weak check and a precise suite that tests the boundaries. Both reach 100% line coverage. Only one deserves your trust.

JS 100% coverage, 0% mutation score

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Every survivor of the smoke suite points at a concrete missing test: the boundary at 10, the express surcharge, the zero-weight check. That is the real value of mutation testing: not the percentage, but the list of "you never checked this" lines. The precise suite kills all six because each test pins one behavior with an exact expected value.

Running it all in CI

Continuous integration runs your build on every push so that a broken test is found minutes after it is written, not at release time. A minimal GitHub Actions workflow for a Maven project looks like this (read-only; for a deeper treatment see CI for Java, Go and PHP):

YAML
name: build
on:
  push:
    branches: [main]
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: "25"
          cache: maven
      - name: Build, test and check coverage
        run: ./mvnw --batch-mode verify
      - name: Keep the reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-reports
          path: |
            target/surefire-reports
            target/failsafe-reports
            target/site/jacoco

Three details deserve attention:

  • verify, not test. It runs Surefire, Failsafe and the JaCoCo check goal, so the coverage gate can fail the job.
  • if: always() on the upload. Reports are most valuable exactly when the build failed, so they must be saved even then.
  • A non-zero exit code is the contract. Any step that exits non-zero fails the job and blocks the pull request when the branch is protected. That is how a coverage drop or a red test stops a merge.

Run slow checks such as PIT on a schedule (on: schedule) or only for changed files, so pull requests stay fast. The same ideas carry over to other CI systems; the Jenkins testing and reports lesson shows the equivalent in a Jenkinsfile.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A class has 100% line coverage. What can you conclude?

    Coverage only records execution. A test without assertions, or with weak ones, still produces full coverage.
  2. 2. In mutation testing, what does a surviving mutant mean?

    A killed mutant is detected by a failing test. A survivor shows a blind spot, unless it is an equivalent mutant that does not change behavior.
  3. 3. Which JaCoCo counter is best at revealing that if (x) y(); never ran with x true?

    A one-line if counts as a covered line as soon as the condition is evaluated. Branch coverage records whether both outcomes happened.
  4. 4. Why upload test reports with if: always() in a CI workflow?

    By default later steps do not run once a step fails. if: always() makes sure the evidence of the failure is kept.
  5. 5. Which approach to coverage targets is the healthiest?

    Coverage is a smoke alarm. A moderate floor with per-class checks guards against neglect without encouraging assertion-free tests.

Summary

  • Coverage (JaCoCo) shows which code the tests executed; prefer branch numbers over line numbers when you want honesty.
  • A coverage gate (check goal, jacocoTestCoverageVerification) fails the build when a minimum is not met; add per-class floors so one weak class cannot hide in a good average.
  • Mutation testing (PIT) breaks your code on purpose; survivors are the missing assertions. It is slow, so run it selectively.
  • Treat percentages as an alarm, not a goal, and avoid rewarding the number.
  • In CI run verify, keep reports with if: always(), and let a non-zero exit code block merges.
  • The measurement examples on this page are stand-ins; the Maven, Gradle and workflow files are real, read-only references.

Next: Designing Good Tests, because tests that kill mutants are tests that are designed well.