Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 36 ยท Getting Started

Output Files: log.html, report.html and output.xml

Learn what Robot Framework writes after a run: output.xml, log.html and report.html. Read results, set log levels, use rebot and find why a test failed.

Every run of robot produces three files in the output directory. Understanding them is half of working with Robot, because a failed test is only useful if you can quickly see why it failed. This lesson shows what each file contains, how log levels work, and how to turn old results into new reports without running anything again.

The three files

File Audience Contents
output.xml Tools The complete, machine-readable result of the run
log.html Testers and developers Every suite, test and keyword with arguments, messages, timings and errors
report.html Managers and stakeholders Pass/fail statistics, tag and suite tables, elapsed time

output.xml is the source of truth. The HTML files are generated from it. That is why you can regenerate them later with rebot (below).

flowchart LR
    R["robot tests/"] --> X["output.xml"]
    X --> L["log.html"]
    X --> P["report.html"]
    X --> J["xunit.xml (optional)"]
    X --> B["rebot: merge, filter, rename"]
    B --> L2["new log / report"]

The run writes output.xml once; everything else is derived from it.

Reading log.html

Open the log in a browser. At the top is a summary and statistics. Below, the Test Execution Log is a tree you can expand:

  • Click a suite to see its tests, a test to see its keywords, and a keyword to see its own keywords, arguments, messages and times.
  • Failed keywords are red and the error text sits at the bottom of the test.
  • Use the expand and collapse controls above the tree; failed tests and keywords are opened for you.

Messages written with Log appear as lines under the keyword. A message can have a level, and a message can contain HTML if you write html=True:

ROBOTFRAMEWORK
*** Test Cases ***
Level demo
    Log    Informational note
    Log    Detailed debugging value    DEBUG
    Log    Something looks odd         WARN
    Log    <b>Bold text</b>            html=True

WARN messages are also copied to the log's "Test Execution Errors" section and shown in the console, so use them sparingly.

Log levels

Robot only records messages at or above the chosen threshold:

Level Typical use
TRACE Very verbose; keyword arguments and return values
DEBUG Detailed values for debugging
INFO The default; normal progress
WARN Something suspicious
ERROR A serious problem (also copied to the errors section)

Use the command line: robot --loglevel DEBUG tests/ records DEBUG and above. The form --loglevel DEBUG:INFO records everything from DEBUG up but shows only INFO and above by default, so the log can be expanded later. Keyword Set Log Level changes it inside a test.

Increase the level while investigating a flaky test, but keep INFO in daily runs: huge logs are slow to open.

The report

report.html is a one-page summary: a big green or red bar, statistics by tag and by suite, and test details. Tags are the main way to get a meaningful report. For example, the tag statistics table can show how many smoke tests passed. You can add custom rows from the command line (for example --tagstatcombine "smokeANDapi"), and --tagdoc adds explanatory text.

If an output.xml already exists you do not need to rerun your tests to change the report.

Rebot: reports without running tests

rebot is the post-processing tool installed with Robot. It reads one or more output.xml files and writes new logs and reports. It accepts most of the same options as robot.

Terminal
# regenerate with a different title
rebot --name "Release 3.2" --outputdir out results/output.xml

# combine two outputs (for example from parallel runs) into one report
rebot --outputdir merged results/ui/output.xml results/api/output.xml

# first run failed tests are replaced by the results of the rerun
rebot --merge --outputdir final results/output.xml results/rerun/output.xml

--merge is the standard way to combine a full run with a --rerunfailed run, so a flaky failure that passes on rerun counts as passed in the final report.

Stand-in 1: reading a result file

The block below is not Robot Framework. It builds a small XML document shaped like output.xml (greatly simplified) and reads statistics from it with Python's standard xml.etree. This is how CI scripts and dashboards usually read real results.

Python Stand-in: parse an output.xml-like file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in 2: log levels

A log keeps a message only when its level is at least the threshold. This stand-in shows why raising --loglevel makes the log bigger.

Python Stand-in: log level threshold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Finding out why a test failed

  1. Open log.html and look for the red test (failed tests are expanded for you).
  2. Read the error message at the bottom of the failed test. Robot's own keywords give clear messages such as 5 != 6.
  3. Expand the failing keyword. Its child keywords and DEBUG messages show what happened inside.
  4. If a screenshot was taken (web tests), it is embedded in the log as an image.
  5. Rerun just that test with --loglevel DEBUG --test "Name".

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which output file is the complete machine-readable result?

    output.xml holds everything; log.html and report.html are generated from it.
  2. 2. What does rebot do?

    rebot post-processes results: combine, merge, filter, rename and regenerate logs and reports.
  3. 3. With --loglevel INFO, which messages are recorded?

    The level is a minimum threshold: that level and everything more severe is recorded.
  4. 4. How do you replace failed tests of a full run with the results of a rerun?

    rebot --merge takes the newer results of tests that appear in more than one output file.

Summary

  • A run writes output.xml (full data), log.html (every step) and report.html (summary).
  • Messages have levels (TRACE to ERROR); --loglevel is a minimum threshold; INFO is the default.
  • Tags drive the statistics in the report.
  • rebot rebuilds, combines and merges results without rerunning tests.
  • To debug a failure, read the error text, expand the failing keyword and rerun that test with DEBUG logging.

Next: Robot File Syntax: Sections, Separators and Comments