Skip to content
Tutorials Center
My learning

Topics

Lesson 31 of 60 ยท Defects and Test Execution

Test Execution and Result Tracking

Run tests in organized cycles: record pass, fail and blocked, tell smoke from sanity, retest and regress correctly, and write an exit report with a run summary.

Planning and designing tests is only half of the job. Sooner or later someone has to run them, record what happened, and tell the team whether the software is ready. Test execution looks mechanical, but sloppy execution wastes good test design: results get lost, failures are not retested, and "we tested it" turns out to mean different things to different people.

This lesson covers how test cycles are organized, what each result status means, the difference between smoke, sanity, confirmation and regression testing, and how to summarize a run for the people who decide. The code is a stand-in: a short script that tallies results the way a test management tool would. The real tool stores history and links to defects, but the counting and the judgment are the same.

Test cycles and runs

A test cycle (or test run) is one pass through a chosen set of test cases against one specific build. A release usually has several cycles: a first full pass, then retesting of fixes, then a final regression pass. Each cycle should state:

  • the build or version under test (a result without a build number is hard to trust);
  • the environment (see Test Data and Test Environments);
  • the scope: which test cases are in the run, and why;
  • who runs what, and the start and end dates.

During execution you follow each case exactly as written, compare actual with expected results, and record the outcome immediately. If you notice something odd outside the case, note it for later or explore it as described in Exploratory Testing in Practice instead of silently ignoring it.

Result statuses

The usual statuses

Passed Actual result matched expected.
Failed It did not. A bug report follows.
Blocked Cannot run, something stops it.
Skipped Deliberately not run this cycle.

Not run yet is a status too.

1 / 2

The distinction between failed and blocked matters. If the login page is broken, forty checkout tests cannot run. Marking them failed inflates the failure count and points at the wrong place; marking them blocked (with a link to the login defect) shows that one problem is hiding the state of many tests. Statuses vary by tool, but the principle holds: a status must tell the reader what to do next.

Record the actual result, not only the verdict. When a case fails, attach the evidence (screenshot, log, data used) so the person writing the bug report, as covered in Writing Excellent Bug Reports, does not have to repeat the run.

Smoke, sanity, confirmation and regression

These four words are often mixed up. They answer different questions.

Kind Question it answers Typical scope
Smoke test Is this build stable enough to bother testing? A very small set covering the main flows, run on every new build
Sanity test Does this specific change or fix look reasonable? Narrow and quick, focused on the changed area
Confirmation test (retest) Is the reported defect really fixed? The exact failing case, rerun on the new build
Regression test Did the change break anything that used to work? A broader suite around the changed and risky areas

Definitions of smoke and sanity differ between companies, so agree on them with your team. The important part is the purpose: smoke is a gate, sanity is a quick check, retesting verifies a fix, and regression protects the rest. You can see how these relate to the test types in Test Types.

A fix is not done when the developer says so. The cycle for a defect is: retest the original failing case, then run regression around the area, because fixes introduce new bugs more often than we like to admit. Choose the regression set by risk, using the thinking in Risk-Based Testing, and keep stable, valuable cases in a standing regression suite that grows over time. A suite that is never pruned becomes too slow to run, which is why teams automate the repetitive parts (see When and What to Automate).

Entry and exit criteria

Execution should start and end against agreed rules from the test plan:

  • Entry criteria might be: the build passed the smoke test, the environment is ready, test data is loaded.
  • Exit criteria might be: all high-priority cases executed, no open critical defects, at least 95% of planned cases passed, and no blocked cases remain.

Exit criteria are targets for a conversation, not magic numbers: "95% passed" is meaningless if the failing 5% is the checkout.

Try it: tally a test run

This stand-in takes the results of a cycle (invented) and produces the summary a test lead would put at the top of a report: counts per status, pass rate, progress and a check against exit criteria. Change a status and run it again.

JS Summarize a test cycle

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Notice what the numbers say: the pass rate of executed cases looks mediocre, but the more useful fact is that one defect, BUG-101, touches three tests and stops the checkout area entirely. A summary that only reports a percentage hides that.

Try it: choose a regression set after a fix

When a fix lands in one area, the regression pack should not be everything and should not be only the area itself. This stand-in selects cases by the changed area plus its neighbors and any case marked as critical, then shows what is left out. The mapping of "neighbors" is invented, but in a real project it would come from a traceability matrix or from developers' knowledge of the code.

JS Pick a regression set

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The set always includes critical cases, even outside the changed area, because a cheap safety net on the most important flows is almost always worth it.

The test summary report

At the end of a cycle or release you write a short test summary (exit) report. A useful one is a page, not a book:

  1. Scope and build: what was tested, against which version and environment.
  2. Results: counts per status, and what was not run and why.
  3. Defects: open defects by severity, with the important ones named.
  4. Risks and gaps: untested areas, environment limits, unresolved issues.
  5. Recommendation: ready, ready with known risks, or not ready, with reasons.

The report informs a decision; it does not make it. Write plainly, link to evidence, and state the facts that support the recommendation. You will practise this in Project: Plan and Report a Release.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. The login page is broken, so 30 checkout tests cannot be run. How should those 30 be marked?

    Blocked says the test could not run because something else stands in the way. Marking them failed would inflate failures and misplace the problem.
  2. 2. A new build arrives. Which test do you run first to decide whether it is worth testing further?

    Smoke tests are a small, fast gate. If they fail, the build goes back without spending time on deeper testing.
  3. 3. What is the purpose of a confirmation test (retest)?

    A confirmation test reruns the exact case that failed. Regression testing is the separate step that looks for side effects.
  4. 4. Why is a bare pass rate of 95% not enough to approve a release?

    Percentages hide which tests failed. Exit criteria should also consider priority and severity of open defects.
  5. 5. What should a status of 'failed' always lead to?

    A failure shows a possible defect. It is reported with evidence, fixed, and then confirmed by a retest and regression.

Summary

  • A test cycle is one pass through a set of cases against a specific build and environment; record both.
  • Use statuses precisely: passed, failed, blocked, skipped and not run each tell the reader something different.
  • Smoke checks stability, sanity checks a change, confirmation checks a fix, and regression checks for side effects.
  • Start and stop against entry and exit criteria from the test plan, and look beyond percentages.
  • End with a short summary report: scope, results, defects, risks and a recommendation.

Next: Writing Excellent Bug Reports.