Skip to content
Tutorials Center
My learning

Topics

Lesson 26 of 35 ยท Quality and Reliability

Reporting and Logging

Make Selenium runs readable: pytest-html and Allure reports, structured JSON logs, and screenshots, page source and URLs saved automatically when a test fails.

A test run that ends with "3 failed" tells you almost nothing. Which three? On which page? What did the browser show at that moment? In CI, where nobody was watching, that evidence exists only if your suite collected it. Good reporting turns a failed run from a mystery into a five-minute fix, and good logging lets you reconstruct what the test did without re-running it.

This lesson covers three layers: the report (a summary of every test), the log (what happened step by step), and the failure artifacts (screenshot, page source, URL). You will build a small HTML report and a structured logger yourself in Python, so you understand what the big tools do.

The pytest, Allure and Selenium snippets are real but read-only: this site cannot start a browser or install those plugins. The two runnable examples use Python's standard library only, so they run in the page and show the same ideas in miniature.

Three layers of evidence

Report Which tests passed, failed or were skipped.
Log What the test did, step by step.
Artifacts Screenshot, HTML and URL at the failure.

A failure needs all three to be quick to diagnose.

1 / 2

A report with pytest-html

pytest-html is a plugin that writes a single HTML page listing every test with its outcome, duration and traceback.

Terminal
pip install pytest-html
pytest --html=reports/report.html --self-contained-html

--self-contained-html embeds the CSS and images in the file, so you can attach it to an email or a CI artifact and open it anywhere. The plugin also has an extras mechanism that lets you attach your own content (such as a screenshot) to each test row. For team dashboards and history, pytest can also write JUnit XML (--junitxml=reports/junit.xml), which almost every CI system can read and display as a test summary.

Allure for richer reports

Allure is a separate reporting tool with a polished interface: steps, attachments, categories, history trends and timelines. With pytest you install allure-pytest, run the tests to produce raw result files, then generate or serve the report:

Terminal
pip install allure-pytest
pytest --alluredir=allure-results
allure serve allure-results

The allure command is a separate program that must be installed on the machine, and it needs a Java runtime. The gain is the @allure.step decorator and allure.attach, which make each page-object method show up as a named step in the report:

Python
import allure

class LoginPage:
    @allure.step("Log in as {username}")
    def log_in(self, username, password):
        self.driver.find_element(*self.USERNAME).send_keys(username)
        self.driver.find_element(*self.PASSWORD).send_keys(password)
        self.driver.find_element(*self.SUBMIT).click()

Which tool should you pick? Start with pytest-html or JUnit XML; they cost nothing. Move to Allure when many people read the reports and you want step-level detail and history.

Capturing evidence when a test fails

The most valuable addition to any suite is a hook that saves a screenshot, the page source and the current URL at the moment a test fails. In pytest this lives in conftest.py:

Python
import pathlib
import pytest

ARTIFACTS = pathlib.Path("artifacts")

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    if report.when == "call" and report.failed:
        driver = item.funcargs.get("driver")
        if driver is not None:
            ARTIFACTS.mkdir(exist_ok=True)
            stem = item.name.replace("/", "_")
            driver.save_screenshot(str(ARTIFACTS / f"{stem}.png"))
            (ARTIFACTS / f"{stem}.html").write_text(driver.page_source, encoding="utf-8")
            (ARTIFACTS / f"{stem}.txt").write_text(driver.current_url, encoding="utf-8")

How it works: pytest builds a report object for each phase of a test (setup, call, teardown). A hookwrapper runs around that step, so after yield you can inspect the result. item.funcargs holds the fixtures the test requested; if your test asked for a fixture named driver, you can reach it here. Keep the hook defensive: a failure inside the hook must never hide the real failure, so check for None and consider wrapping the saves in try/except.

Remember that artifacts are only useful if CI keeps them. The previous lesson, Running in CI, shows the upload step. For finding the cause once you have the evidence, see Flaky Tests and Debugging.

Build a tiny report yourself

Seeing how a report is built removes the magic. A report is just data (a list of results) turned into HTML. The program below takes a list of test results, escapes everything (a traceback may contain < and &), and builds a standalone page with a summary line and a table. It prints the HTML so you can see the structure.

Python Build a small HTML test report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the failed row: the actual value '$19.99 <b>' came out as &lt;b&gt;, so a weird string in the application cannot inject markup into your report. Real tools do exactly this escaping. The summary line is computed from the same list, so the numbers can never disagree with the table. In a real framework the list is filled by a hook like pytest_runtest_makereport, and the screenshot path would become another table cell with an <img> or a link.

Structured logging

print() is fine while you write a test, but a CI log with thousands of free-text lines is hard to search. Structured logging writes each event as one JSON object per line, with the same keys every time. Tools can then filter by test, level or step.

In real Python code you would use the standard logging module with a JSON formatter, or simply log dictionaries. A common pattern is to log every page action from your page objects:

Python
import logging

log = logging.getLogger("ui")

class BasePage:
    def click(self, locator):
        log.info("click", extra={"locator": str(locator), "url": self.driver.current_url})
        self.driver.find_element(*locator).click()

Log levels have a clear meaning: DEBUG for noisy details (every locator lookup), INFO for user-level steps ("clicked Checkout"), WARNING for recoverable surprises (a retry was needed), and ERROR for failures. Keep passwords, tokens and personal data out of logs; see Security, Ethics and Scraping Limits.

The example below builds a minimal JSON-line logger with the standard library and then reads the lines back to answer a question: which step took longest?

Python JSON-line logs you can query

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Because every line is valid JSON with the same core keys, finding the slowest step or all warnings takes two lines of code (or a jq command, or a log search box). Free-text logs would need fragile regular expressions. The times here are simulated numbers, not real browser timings.

What a good failure report contains

Put yourself in the shoes of the developer who opens a red build. They want, in this order:

  1. The test name and the assertion message ("expected 3 items, found 2").
  2. A screenshot showing what the user would have seen.
  3. The URL and page source, to check the state of the DOM.
  4. The last few log lines, to see what led there.
  5. Browser console errors, if you enabled log capture (see DevTools Protocol and BiDi).

Write assertion messages that include both values (assert total == expected, f"total {total} != {expected}"). They cost nothing and save a trip into the artifacts.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the report example pass test details through html.escape?

    Failure messages often include HTML from the page under test. Escaping turns it into harmless text, so the report always renders correctly.
  2. 2. In the pytest_runtest_makereport hook, why check report.when == "call"?

    Each test produces setup, call and teardown reports. A failure in the call phase means the test body failed, which is when the page state is worth saving.
  3. 3. What is the main advantage of one-JSON-object-per-line logs?

    A consistent structure lets tools and short scripts filter and aggregate events without parsing free text.
  4. 4. A CI job fails, but the screenshots are not available afterwards. What is the most likely cause?

    Headless Chrome can take screenshots. The usual problem is that the upload step was skipped after the failed tests or did not point at the folder where the files were saved.

Summary

  • Reporting has three layers: a report of all tests, logs of what happened, and artifacts captured at the failure.
  • pytest-html produces one self-contained page; JUnit XML integrates with CI; Allure adds steps, attachments and history but needs its own command-line tool.
  • A pytest_runtest_makereport hook in conftest.py can save a screenshot, page source and URL when the call phase fails.
  • Always escape text that goes into HTML reports.
  • Structured JSON-line logs are searchable; use levels consistently and never log secrets.
  • Include both the expected and actual values in assertion messages.

Next, look at the rules around secrets, test accounts and automation limits in Security, Ethics and Scraping Limits.