Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท Reporting and CI

Reports and Plugins

Produce readable Cucumber reports: the pretty, summary, html, json, junit and message plugins, publishing, rerun files, Allure and attachments, plus a runnable reporter stand-in.

A test run that nobody can read is almost as bad as no run. When the build turns red, someone has to find out which scenario failed, at which step, with which message, and ideally with a screenshot. Cucumber separates running scenarios from reporting them: the core engine emits events, and plugins turn those events into console output, files or uploads. This lesson tours the built-in plugins, explains how to switch them on, shows how reports reach a continuous integration (CI) server and covers extras such as the rerun file and attachments.

Cucumber cannot run on this site, so the configuration is shown read-only. The runnable stand-in collects scenario results and produces a console summary, a JUnit-style XML file and a failed-scenario list, so you can see what the plugins do with the same data.

One run, many reports

pretty Colourful console output.
html A self-contained page.
json and message Data for other tools.
junit XML that CI servers read.
1 / 2

Built-in plugins

Plugin Output Typical use
pretty every step with colours in the console local development
progress one dot per step (. pass, F fail) long runs, short console
summary totals and failures at the end quiet CI logs
html:target/report.html a single HTML report humans
json:target/report.json classic Cucumber JSON third-party report generators
junit:target/junit.xml JUnit XML, one test case per scenario CI dashboards, test trend graphs
message:target/messages.ndjson the full Cucumber Messages stream custom tools, cucumber-reports uploads
rerun:target/rerun.txt locations of failed scenarios re-run failures only
timeline:target/timeline when each scenario ran on which thread spotting slow or serialized work
usage:target/usage.json which step definitions are used and how slow finding dead steps

You can combine several; each plugin entry is name or name:path.

Switching them on

With the JUnit Platform engine, use junit-platform.properties (works from the IDE, Maven and Gradle alike):

PROPERTIES
cucumber.plugin=pretty, summary, html:target/cucumber/report.html, \
  json:target/cucumber/report.json, junit:target/cucumber/junit.xml, \
  rerun:target/cucumber/rerun.txt
cucumber.publish.quiet=true

On the command line: mvn test -Dcucumber.plugin="pretty,html:target/report.html". In Cucumber.js: --format progress --format html:report.html --format json:report.json. Ruby: cucumber --format pretty --format html --out report.html.

Note cucumber.publish.quiet=true: by default Cucumber-JVM prints an advertisement for the Cucumber Reports hosted service after each run. Setting cucumber.publish.enabled=true uploads the run as a public (unless you use a token) report; never enable it when scenarios contain confidential data.

What a report should tell you

For each failed scenario you want: the feature and scenario name, the failing step, the assertion message, the stack trace if it is an unexpected exception, and any attachments such as a screenshot or the response body. Attachments come from your code:

Java
@After
public void attachEvidence(Scenario scenario) {
    scenario.log("Order id was " + world.order.id());          // text in the report
    if (scenario.isFailed()) {
        scenario.attach(world.lastResponse.asByteArray(),
                        "application/json", "last-response");   // file in the report
    }
}

log adds text next to the current step; attach stores bytes with a media type. The HTML report shows images inline. See Hooks for where to put this code.

Good failure messages are part of the report. assertEquals(3, items) alone says "expected 3 but was 2". A message such as "items in cart for " + customer turns it into something you can act on without a debugger.

Third-party reports

  • Allure (allure-cucumber7-jvm) adds a rich dashboard with history, categories and timelines.
  • Masterthought cucumber-reporting turns the JSON file into a navigable HTML site with tag and step statistics.
  • ExtentReports and similar tools offer styled HTML with screenshots.

They all consume the same events, so you can adopt them later without changing scenarios. Prefer the built-in HTML plus JUnit XML first; add a dashboard when people actually ask for trends.

The rerun file

The rerun plugin writes the file and line numbers of every failed scenario, one per line:

TEXT
file:src/test/resources/features/checkout.feature:12
file:src/test/resources/features/cart.feature:30

You can pass it back in as @target/cucumber/rerun.txt (Cucumber CLI) or via the cucumber.features property to run only those scenarios. In CI some teams automatically rerun failures once; be careful, because that can hide flaky tests rather than fix them (see Flaky and slow suites).

A reporter stand-in

The program below fakes a finished run as a list of results and writes the three outputs most teams need: a summary, JUnit XML and a rerun list. It uses nothing but string building, which is also how real plugins work: they listen to events and print text.

Java Summary, JUnit XML and rerun list (Java stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Tips for readable reports

  • Give scenarios and features clear names; they become the report's headings. Use real language, not TC_0042.
  • Keep step text meaningful, since each step is a row in the report.
  • Store reports under target/ or build/ and have CI archive them, so they exist after the build machine disappears (CI pipelines).
  • Publish JUnit XML even if you prefer another dashboard; nearly every CI tool understands it.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which plugin produces XML that most CI servers can display as test results?

    JUnit XML is the common interchange format for CI test dashboards.
  2. 2. What does the rerun plugin write?

    The file lists failed scenario locations so you can run only those again.
  3. 3. What is a risk of enabling cucumber.publish.enabled=true?

    Publishing uploads the run to reports.cucumber.io, so only use it when the data is not sensitive.
  4. 4. How do you add a screenshot to the report from a hook?

    attach stores binary data with a media type; log is for text.

Summary

  • Plugins turn the engine's events into console output and files; combine several with cucumber.plugin.
  • Use pretty/summary locally, html for people, junit for CI and rerun to re-execute failures.
  • Add context with scenario.log and scenario.attach, and write assertion messages that say what was compared.
  • Allure or Masterthought can be added later; avoid publishing confidential runs.

Next: Parallel execution