Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 36 ยท Getting Started

Running Tests and Command-Line Options

Run a file, a folder or one test with the robot command, then filter by name and tag, set variables, change output paths and rerun only the failed tests.

The robot command is how you run everything: one file, a whole folder, a single test, or a tagged group. It has dozens of options, but about a dozen cover daily work. This lesson groups them by purpose so you can find the right one fast.

All commands are read-only text here; the runnable blocks are Python stand-ins for the filtering logic.

What you can run

Terminal
robot tests/login.robot          # one file = one suite
robot tests/                     # a directory = a suite of suites
robot tests/login.robot tests/api  # several at once (a combined top-level suite)
python -m robot tests/           # same thing without the robot script

A directory becomes a suite whose children are its .robot files and sub-directories. Settings that apply to the whole directory go into an __init__.robot file in it (see Libraries and Resources and the suite setup lesson later in this chapter).

Choosing tests

Option Short Effect
--test "Login*" -t Run tests whose name matches the pattern
--suite "Checkout" -s Run only suites with this name
--include smoke -i Run tests having this tag
--exclude slow -e Skip tests having this tag
--rerunfailed output.xml -R Run only the tests that failed in an earlier run
--skip flaky Mark tests with this tag as SKIP
--dryrun Check syntax and keyword names without running anything

Name patterns are case-insensitive and ignore spaces and underscores, with * matching anything and ? matching one character. You may repeat an option: -i smoke -i regression means tests tagged smoke OR regression.

Tag patterns can also combine tags:

Pattern Meaning
smokeANDwindows Both tags
smokeORapi Either tag
smokeNOTslow smoke but not slow
bug-* Any tag starting with bug-

The operators must be in upper case.

Controlling output

Option Effect
--outputdir results (-d) Put all output files in this folder
--output out.xml (-o) Name of the XML file (NONE disables it)
--log mylog.html, --report myreport.html File names (NONE disables them)
--loglevel DEBUG (-L) Detail level written to the log: TRACE, DEBUG, INFO (default), WARN, ERROR
--console verbose verbose, dotted, quiet or none; dotted prints one character per test, handy in CI
--timestampoutputs (-T) Add a timestamp to file names so runs do not overwrite each other
--xunit xunit.xml Also write a JUnit-style XML for CI tools

Giving the run data

Option Effect
--variable ENV:staging (-v) Set ${ENV} to staging from outside
--variablefile vars.py (-V) Load variables from a file
--pythonpath libs (-P) Add a folder to where Python looks for libraries
--name "Nightly" Override the top-level suite name
--metadata Build:1432 Add metadata shown in the report
--argumentfile args.txt (-A) Read options from a file, one per line
--exitonfailure (-X) Stop at the first failing test
--randomize all Shuffle the test order

An argument file keeps long commands tidy. For example args.txt:

TEXT
--outputdir results
--loglevel DEBUG
--variable ENV:staging
--include smoke

and then robot -A args.txt tests/. Options given later on the command line can override earlier ones.

Stand-in 1: filtering by name and tag

The block below is not Robot Framework. It implements the same selection rules (patterns, include, exclude) on a list of tests, so you can see why a command picks the tests it does. It uses fnmatch for the * and ? wildcards.

Python Stand-in: include and exclude

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Excluding always wins: a test that matches both an include and an exclude is not run.

Stand-in 2: tag patterns with AND, OR and NOT

Robot parses a tag pattern such as smokeANDauthNOTslow. The stand-in below handles a simplified version of the grammar. Real Robot also supports wildcards inside the tag names and several NOT parts.

Python Stand-in: AND / OR / NOT

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A typical daily workflow

Terminal
# run the fast checks while developing
robot --include smoke --outputdir results tests/

# run everything on the build server, quiet console, plus CI XML
robot --console dotted --xunit xunit.xml --outputdir results tests/

# fix the failures, then rerun only those
robot --rerunfailed results/output.xml --outputdir results/rerun tests/

# a quick syntax check with no side effects
robot --dryrun tests/

--dryrun is useful before a long run: it reports missing keywords and wrong argument counts, but never calls the real keywords. It cannot catch everything (a keyword that fails only at run time passes a dry run).

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which option runs only tests that have the tag smoke?

    --include (or -i) selects tests by tag; --test selects by name and --suite by suite name.
  2. 2. What does --rerunfailed output.xml do?

    It reads the old result and selects only the failed tests. Pass it the output.xml of the previous run.
  3. 3. A test is tagged slow and smoke. With -i smoke -e slow, is it run?

    A test matching an exclude pattern is not run, even if it also matches an include.
  4. 4. What does the tag pattern smokeNOTslow mean?

    NOT removes tests having the tag that follows it. AND, OR and NOT must be written in upper case.

Summary

  • robot path runs a file, folder or several paths; python -m robot is the same command.
  • Select with --test, --suite, --include, --exclude, --rerunfailed; combine tags with AND, OR, NOT.
  • Control files with --outputdir, --output, --log, --report, --loglevel, --console.
  • Feed data with --variable, --variablefile, --pythonpath and --argumentfile.
  • --dryrun checks structure without executing keywords.

Next: Output Files: log.html, report.html and output.xml