Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 37 ยท Frameworks and Page Objects

Test Data, Screenshots and Reports

Manage test data for mobile tests, capture screenshots and page source on failure, record the screen, collect logs and publish readable reports.

A mobile test suite that only says "FAILED" is nearly useless. The person reading the CI log needs to know what the screen looked like, which data was used, what the app logged, and whether the failure is new. This lesson covers the supporting pieces that make a suite maintainable: test data that is predictable, evidence captured automatically when something fails, and reports that humans read.

Appium and devices cannot run on this site. The Appium calls are read-only; the runnable examples are stand-ins that handle data and report formatting, which are plain programming problems.

Test data strategy

Most flaky mobile tests are really data problems: the user already exists, the cart is not empty, or the product sold out. Aim for tests that create their own data or use data you fully control.

  • Static data in a file (users.json) works for read-only accounts that tests never change.
  • Generated data avoids collisions: ana+{timestamp}@example.com creates a unique user for each run.
  • API setup is the fastest: create the user and add items to the cart with HTTP calls, then use the UI only for the flow you are testing.
  • Cleanup should be independent of the test result: delete data in teardown, or use a throwaway environment that is reset.

Never put real customer data or production passwords in test files. Read secrets from environment variables:

Python
import os
USERNAME = os.environ["TEST_USER"]
PASSWORD = os.environ["TEST_PASSWORD"]

Stand-in: unique data and a data table

Python Generating unique test users (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A real run would use a timestamp or UUID instead of a counter; the counter keeps this example deterministic.

Capturing evidence on failure

Appium gives you three pieces of evidence for any moment in a session:

Python
driver.get_screenshot_as_file("artifacts/login-failure.png")
page_source = driver.page_source                     # XML tree of the screen
logs = driver.get_log("logcat")                       # Android; iOS uses "syslog"

To capture them automatically in pytest, use a hook that runs after each test and a fixture that exposes the result:

Python
# conftest.py
import pytest

@pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    setattr(item, f"rep_{report.when}", report)

@pytest.fixture
def driver(request):
    drv = create_driver()
    yield drv
    if request.node.rep_call.failed:
        name = request.node.name.replace("/", "_")
        drv.get_screenshot_as_file(f"artifacts/{name}.png")
        with open(f"artifacts/{name}.xml", "w", encoding="utf-8") as f:
            f.write(drv.page_source)
    drv.quit()

In Java with TestNG you would implement ITestListener.onTestFailure; in JUnit 5 a TestWatcher extension does the same. In all cases, capture before quit(), because the screen is gone afterward.

Screen recording

Appium can record the screen as video during a test, which is the most convincing evidence for flaky failures:

Python
driver.start_recording_screen()
# ... test steps ...
video_base64 = driver.stop_recording_screen()

import base64
with open("artifacts/run.mp4", "wb") as f:
    f.write(base64.b64decode(video_base64))

Recording costs performance and storage, so many teams record only failed tests, or rely on the video their cloud vendor already stores. Limits apply: Android records in chunks of up to three minutes per call, and iOS needs ffmpeg installed for some modes.

Logs

  • Appium server log: the text the server prints, with each command and its result. Save it as a CI artifact, and increase detail with appium --log-level debug when you need it.
  • Device logs: logcat on Android and syslog on iOS contain crash stack traces from the app itself.
  • Client log: the standard Python logging module or Java SLF4J in your tests; log steps at INFO level so the log reads like a story.

Reports

A good report answers: what ran, what failed, what did it look like, and is it getting worse? Common options:

Tool Notes
pytest --junitxml=report.xml Machine readable; every CI system understands it
pytest-html A single HTML file with failure details
Allure Rich report with steps, attachments, history and categories; works with pytest, JUnit and TestNG
Cloud dashboards Video, logs and device details per session

With Allure you annotate steps and attach files:

Python
import allure

@allure.step("Log in as {email}")
def log_in(screen, email, password):
    return screen.log_in(email, password)

allure.attach(driver.get_screenshot_as_png(), name="after login", attachment_type=allure.attachment_type.PNG)

Stand-in: parse a JUnit report

CI systems read JUnit XML. This runnable example parses a small report and prints the summary a dashboard would show.

Python Summarizing a JUnit XML report (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why should evidence such as a screenshot be captured before driver.quit()?

    The session and its screen are gone after quit, so capture first.
  2. 2. What is the fastest and most reliable way to prepare test data for a UI test?

    API setup is quick, deterministic, and keeps the UI test focused on the flow under test.
  3. 3. Where should credentials for test accounts come from?

    Secrets stay out of source control.
  4. 4. Which file format do almost all CI systems understand for test results?

    JUnit XML is the common exchange format; Allure and HTML reports are built on top of results.

Summary

  • Prefer test data that each test creates or fully controls; use APIs for setup and keep secrets in the environment.
  • Capture a screenshot, page source and device logs automatically on failure, before the session closes.
  • Record video selectively; keep the Appium server log as a CI artifact.
  • Produce JUnit XML for CI and a richer report such as Allure for people.

Next: Why Cloud Device Farms?