Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 36 ยท Reporting and CI

Run options, tags and selecting tests

Control a Robot Framework run: --include, --exclude, --test, --suite, tag expressions, variables, argument files, dry runs and re-running failed tests.

A suite of five hundred tests cannot always run in full. You want a five-minute smoke check on every commit, the full regression at night, and a single test when you debug. Robot Framework gives you three tools for that: tags on tests, command-line options to select by tag or name, and argument files to keep long command lines readable.

The robot command is not available on this site, so the commands are read-only. The runnable stand-ins implement tag selection logic in Python so you can try the expressions.

Tagging tests

Tags are free-form labels. Add them to one test with [Tags], or to every test in a file with Test Tags (Robot 6.0 and later; older versions used Force Tags and Default Tags):

TEXT
*** Settings ***
Test Tags    web

*** Test Cases ***
Valid Login
    [Tags]    smoke    login
    Log In As    demo    s3cret

Locked Account Is Rejected
    [Tags]    negative    login
    Log In As    locked    s3cret

The first test has the tags web, smoke and login. Tags are case-insensitive and ignore spaces and underscores, so Smoke and smoke are the same tag: keep to lowercase words. Tags starting with robot: are reserved, such as robot:skip and robot:exclude.

Selecting by tag

Terminal
robot --include smoke tests/
robot --exclude slow tests/
robot --include smokeANDlogin tests/
robot --include loginNOTnegative tests/
robot --include smokeORregression tests/

AND, OR and NOT must be uppercase and written between tag names. Combine --include with --exclude: Robot first picks tests matching the include, then removes tests matching the exclude. Patterns work too: --include browser-* selects every tag starting with browser-.

A tag scheme that works

smoke Fast check of the essentials.
regression Everything that must still work.
slow Excluded from quick runs.
bug-123 Links a test to a ticket.

Keep a short documented list; tags nobody remembers do not help.

1 / 2

Selecting by name and suite

Terminal
robot --test "Valid Login" tests/
robot --test "*Login*" tests/
robot --suite checkout tests/
robot tests/checkout.robot            # a file
robot tests/api/ tests/web/login.robot  # several paths

A suite name comes from its file or directory name: tests/Web Tests.robot becomes Web Tests. Matching is case-insensitive, and * and ? are wildcards.

Variables and settings from the command line

Terminal
robot --variable ENV:staging --variable BROWSER:firefox tests/
robot --variablefile envs/staging.py tests/
robot --outputdir results --name "Nightly" tests/
robot --loglevel DEBUG --timestampoutputs tests/

--variable overrides a *** Variables *** value; that is how one suite runs against several environments. --outputdir chooses where the reports go, and --timestampoutputs keeps old results by adding a timestamp to file names.

Argument files

Long command lines get hard to read, so move them into a text file, one option per line:

TEXT
# nightly.args
--variable ENV:staging
--exclude slow
--outputdir results
--loglevel INFO:DEBUG

Then run robot --argumentfile nightly.args tests/. Check the file into your repository and everyone, including CI, runs the same way.

Useful run controls

Option Effect
--dryrun Checks that keywords exist and arguments fit, without running them
--rerunfailed output.xml Runs only the tests that failed in an earlier run
--exitonfailure Stops the whole run at the first failure
--randomize all Shuffles test order (with a seed) to find hidden dependencies
--prerunmodifier Changes the suite structure before running (see Patterns)
--exitonerror Stops on serious errors such as a broken import

--dryrun is great in code review or a quick CI step: it catches typos in keyword names in seconds, though it cannot find problems that only appear at run time.

Re-running failures is a common CI pattern:

Terminal
robot --output first.xml tests/
robot --rerunfailed first.xml --output rerun.xml tests/
rebot --merge --output output.xml first.xml rerun.xml

A test that fails the first time and passes the second was flaky; track those, do not just hide them.

A tag expression evaluator

The stand-in below implements AND, OR and NOT the way --include expressions combine tags. It is a simplified evaluator, enough to understand the rules.

Python Stand-in: tag expressions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Keeping runs fast with order and filters

A second, tiny stand-in shows why --dryrun and tag selection save time: they act on the list of tests before anything slow happens.

Python Stand-in: dry run finds a typo

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A dry run checks structure only. A passing dry run does not mean the tests would pass.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which command runs only tests tagged smoke and login?

    A tag expression with uppercase AND selects tests that have both tags.
  2. 2. What does --dryrun do?

    It validates the structure of the suite without executing the keywords.
  3. 3. How do you re-run only the tests that failed earlier?

    --rerunfailed reads a previous output.xml and selects its failed tests.
  4. 4. Why keep command options in an argument file?

    An argument file is shareable, versioned and easier to read than a long command line.

Summary

  • Tag tests with [Tags] or Test Tags; use a small, agreed set.
  • Select with --include and --exclude, using AND, OR and NOT.
  • Select by --test or --suite names, with wildcards.
  • --variable and --variablefile change data per run; argument files keep options tidy.
  • --dryrun checks structure; --rerunfailed plus rebot --merge handles flaky tests (track them).

Next: CI with GitHub Actions and parallel runs