Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 40 ยท Build, Test and Package

Tests, JUnit Reports and Quality Gates

Publish test results in Jenkins with the junit step, read trends, spot flaky tests, track coverage and warnings, and fail builds with quality gates.

A green build that never ran the tests is worse than a red one. Jenkins can run any test tool, but to show which tests failed, how long they took and whether the project is getting better or worse, it needs the results in a format it understands. The most common format is the JUnit XML report, which almost every test framework in every language can produce.

This lesson covers the junit step, how results change the build status, trends and flaky tests, code coverage, static analysis with Warnings Next Generation, and quality gates that fail a build when numbers cross a threshold. The Jenkins parts are read-only because the server cannot run here. The runnable examples are Python stand-ins that parse a real JUnit-style XML document with the standard library.

From test run to decision

Run tests Any framework, any language.
Write XML JUnit-format report files.
Publish The junit step reads the files.
Gate Pass, unstable or fail.

Reports turn a log into data.

1 / 2

Publishing results with junit

Your test command writes XML report files; the junit step reads them and shows the results in Jenkins:

GROOVY
pipeline {
  agent any
  stages {
    stage('Test') {
      steps {
        sh './mvnw -B -ntp test'
      }
      post {
        always {
          junit 'target/surefire-reports/*.xml'
        }
      }
    }
  }
}

Putting junit in post { always { ... } } matters. If the tests fail, the command returns an error and the stage stops, so a junit step placed after it in steps would never run, and you would lose the very report that explains the failure.

Where the files come from depends on the tool. Maven's Surefire plugin writes to target/surefire-reports, Gradle to build/test-results/test, pytest with --junitxml=reports/junit.xml, Jest with a JUnit reporter package, Go with a converter tool, and so on. The junit step takes a path pattern, and it fails the step if no report matches unless you pass allowEmptyResults: true. That default is useful: it catches the case where the tests silently did not run.

Build status: unstable versus failure

The junit step marks the build UNSTABLE (yellow) when some tests failed, and leaves it as SUCCESS (blue or green) when all passed. A hard error, such as a compile failure, gives FAILURE (red). The difference is intended: "the code built and some tests fail" is a different situation from "nothing could be built".

Teams that treat any failing test as a failed build can convert it: set skipMarkingBuildUnstable: true and fail explicitly, or fail the stage with error in a later step, or enforce the result with a quality gate (below). Decide a rule and apply it everywhere, so a yellow ball always means the same thing.

On a job's page, Jenkins draws a test result trend: pass and fail counts per build, and the history of each individual test, including how long it takes. Use it to answer questions such as:

  • Which tests are getting slower?
  • Since which build has this test failed?
  • Is the number of tests growing, or did someone delete a suite to make the build green?

A flaky test passes and fails on the same code. Causes include timing assumptions, shared state between tests, random data without a fixed seed, and dependence on external services. Flakiness is poison: people learn to ignore red builds. Plugins can retry failed tests and report flaky ones, and a retry step can re-run a whole stage, but retrying hides a problem rather than solving it. Use retries to keep the pipeline moving while you track flaky tests in a list and fix them with the same priority as bugs.

Coverage and static analysis

Code coverage tools (JaCoCo for Java, coverage.py, Istanbul/nyc and others) write their own reports. A coverage plugin reads them and shows percentages per package and file, plus a trend. A typical call looks like this (the exact options depend on the plugin version you install, so check its documentation):

GROOVY
post {
  always {
    recordCoverage(tools: [[parser: 'JACOCO']])
  }
}

Static analysis finds problems without running the code: compiler warnings, style violations from Checkstyle, bug patterns from SpotBugs, linters like ESLint or Pylint. The Warnings Next Generation plugin collects findings from many tools into one view, with a baseline so you can see which issues are new in this build:

GROOVY
post {
  always {
    recordIssues(
      tools: [java(), checkStyle(pattern: 'target/checkstyle-result.xml')],
      qualityGates: [[threshold: 1, type: 'NEW', unstable: false]]
    )
  }
}

Here the quality gate says: if there is at least one new issue compared with the reference build, fail the build (unstable: false means failure rather than unstable). Looking only at new findings is the practical way to adopt analysis in an old code base with thousands of existing warnings: you stop the situation getting worse without demanding a big cleanup first.

Quality gates

A quality gate is an automatic yes/no decision based on measurements: no failing tests, coverage at least 80 percent, no new critical findings, no known vulnerabilities in dependencies. Jenkins supports gates in several places:

  • Warnings Next Generation has qualityGates as shown above.
  • The coverage plugin can fail or mark unstable a build whose coverage falls below a threshold.
  • SonarQube integration can pause the pipeline until the server computes its gate result.
  • You can always write your own check in a script and call error('message') or set currentBuild.result = 'UNSTABLE'.

Good gates are simple and explainable. A gate that no one understands gets bypassed. Prefer a few rules on new code (for instance "new code needs 80 percent coverage") over one huge global number, and review thresholds as the project matures. Be careful with gates on a single metric: coverage can be gamed by tests without assertions.

A stand-in report parser

The JUnit XML format is a plain document: testsuite elements containing testcase elements, and a failure, error or skipped child when something went wrong. The Python below parses a made-up report with the standard library, then prints totals, the failures and the slowest tests. This is the same information the junit step turns into the test result page.

Python Parsing a JUnit XML report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One failure, one error and one skipped test give the verdict UNSTABLE, matching the behavior of the junit step: the build ran, but not everything is green.

A stand-in flaky detector and gate

The second example looks at the results of the same commit run several times. A test that both passes and fails on one commit is flaky. It then applies two simple gates, the way a script step might. The data is invented for the example.

Python Finding flaky tests and applying a gate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

search is flaky on commit a1, and upload always fails there, which points at a real bug. The latest build passes its tests but fails the coverage gate, so the final result is FAILURE.

Test reports are data about your code, so treat them like data: keep them, graph them, and look at trends. A suite that grows 30 seconds slower each week will eventually be too slow to run.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why is junit usually placed in post { always { ... } }?

    When tests fail, the test command returns an error and later steps are skipped. The post block still runs, so the report is published.
  2. 2. What build result does the junit step give when some tests failed but the build compiled?

    UNSTABLE means the build ran but has failing tests. FAILURE is for hard errors like a compile failure.
  3. 3. What is a flaky test?

    Flaky tests give inconsistent results without any code change and erode trust in the pipeline.
  4. 4. Why gate on new warnings instead of all warnings in a large old code base?

    A baseline plus a gate on new issues lets a team adopt analysis gradually.

Summary

  • Test tools write JUnit XML; the junit step publishes it, and should usually run in post { always }.
  • Failing tests mark a build UNSTABLE; hard errors mark it FAILURE. Pick one team-wide rule.
  • Trends reveal slowdowns and flaky tests; fix flakiness instead of hiding it with retries.
  • Coverage and Warnings Next Generation add numbers and findings; gates turn numbers into decisions.
  • Prefer simple gates on new code over a single global number.

Next, keep the files your build produces in Artifacts, Stash and Fingerprints.