Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 60 ยท Defects and Test Execution

Defect Lifecycle and Quality Metrics

Follow a bug from New to Closed, learn triage and root cause analysis, and compute defect density, detection effectiveness and leakage with a runnable stand-in.

A bug report is not the end of the story. Someone must decide whether it is real, who will fix it, when, and how we know the fix works. Teams describe that journey as the defect lifecycle (or bug lifecycle): a small set of states and the allowed moves between them.

Once many defects flow through the lifecycle, they also become data. Counting them in the right way tells you where quality is weak, how well your testing works, and whether the release is getting healthier or worse. This lesson covers both: the states, and the metrics. It assumes you already know how to write the report itself, which is the topic of Writing Excellent Bug Reports.

A bug's journey

New Reported, not yet looked at.
Assigned An owner will fix it.
Fixed A change is ready to retest.
Verified A tester confirmed the fix.

After Verified, the defect is Closed.

1 / 2

The states in detail

Every tool names the states a little differently, but the shape is the same.

  • New: a tester or user has logged the defect. Nobody has checked it yet.
  • Assigned (or Open): the defect was reviewed, accepted as valid and given to a developer.
  • Fixed (or Resolved): the developer changed the code and believes the problem is gone.
  • Verified (or Retest passed): a tester repeated the steps on the new build and the problem no longer happens. A good verification also checks that nothing nearby broke.
  • Closed: the work is finished. Nothing more will happen unless the problem returns.
  • Reopened: the retest failed, or the problem came back later. The defect goes back to a developer.
  • Rejected: the team decided it is not a defect, for example the behavior matches the requirement or the report cannot be reproduced.
  • Duplicate: the same problem was already reported. The newer report points to the older one.
  • Deferred (or Postponed): it is a valid defect, but the team decided to fix it in a later release.

The important idea is that the tester owns the last step. A developer marking a defect Fixed is a claim, not a fact. Only a tester repeating the scenario can move it to Verified.

stateDiagram-v2
    [*] --> New
    New --> Assigned
    New --> Rejected
    New --> Duplicate
    Assigned --> Fixed
    Assigned --> Deferred
    Fixed --> Verified
    Fixed --> Reopened
    Verified --> Closed
    Reopened --> Assigned
    Deferred --> Assigned
    Closed --> Reopened

One typical lifecycle. Your tracker may add states such as "In progress" or "Ready for test".

A lifecycle you can run

Real trackers such as Jira or Bugzilla enforce a workflow like this. The stand-in below is not a tracker; it is a tiny table of allowed moves, plus a replay function that refuses illegal ones. It shows why a defect cannot jump from Fixed straight to Closed.

JS Replay defect histories against a workflow

Press Run (or Ctrl + Enter) to see the console output here.

No output.

BUG-102 shows a reopen: the fix failed once and came back. A high reopen rate is a warning sign; it can mean weak reproduction steps, rushed fixes or missing unit tests. BUG-104 and BUG-105 are the shortcuts that a workflow is designed to stop. Try adding a state of your own, for example "In review" between Assigned and Fixed, and see which histories break.

Triage: deciding what happens next

New defects do not go straight to developers in most teams. They first pass through triage, a short recurring meeting (often 15 to 30 minutes) where a tester, a developer and a product owner look at every new defect and decide:

  1. Is it valid and reproducible? If not, ask for more information or reject it.
  2. Is it a duplicate? Link it to the existing report.
  3. How severe is it? Severity describes the technical impact, from a cosmetic glitch to data loss.
  4. How urgent is it? Priority says how soon to fix it, which also depends on the business.
  5. Who owns it, and for which release?

Severity and priority are different things, a distinction explained in Writing Excellent Bug Reports. A typo in the company name on the home page is low severity but may be high priority. You will practice a full session in Project: Run a Bug Triage Session.

Keep triage short and decisive. If a defect needs a long discussion, assign one person to investigate and move on. A triage meeting that debates every detail trains people to avoid it.

Root cause analysis

Fixing a defect removes one symptom. Root cause analysis (RCA) asks why the defect happened at all, so the same kind of defect does not return. You do not need an RCA for every bug, only for serious ones and for patterns.

Two easy methods:

  • Five whys. Keep asking "why?" until you reach something the team can change. The discount was wrong. Why? The boundary used > instead of >=. Why was it not caught? There was no boundary test. Why? The requirement gave no exact threshold. Now the action is clear: review requirements for exact thresholds.
  • Categorizing causes. Tag each defect with a cause such as requirements, design, coding, environment or test data. After a month, the tags show where to improve.

The outcome of an RCA should be a preventive action with an owner, such as "add boundary checks to the code review checklist". An RCA that ends with "be more careful" has found nothing.

Quality metrics from defect data

Metrics turn the pile of defects into answers. Here are the ones every tester should know.

Metric Formula What it tells you
Defect density defects found / size (often per thousand lines of code, or per feature or story point) Which parts are most defect-prone
Defect detection effectiveness (DDE) defects found before release / all defects found, as a percentage How good your testing is at catching problems
Defect leakage (escape rate) defects found after release / all defects, as a percentage How much slipped through to users
Defect age days between opened and closed How quickly defects are resolved
Reopen rate reopened defects / fixed defects How reliable the fixes are

Note that leakage is the mirror image of detection effectiveness: if 80% are caught, 20% leak. Some organizations define them over a specific phase instead, for example "defects found in system testing out of defects found in system testing and later". State your definition next to the number, because two teams can compute different values from the same data.

Computing them

The stand-in below holds ten invented defects and computes density, detection effectiveness, leakage and average age. The module sizes are made up too.

JS Defect metrics from a small data set

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Eight of ten defects were caught before release, so detection effectiveness is 80% and leakage 20%. By density, Login is the worst module (1.00 per KLOC) even though Checkout has more defects in total, because Login is small. That is why density, not raw count, is the fair way to compare modules. Low-severity defects stayed open longest (9.0 days on average), which is a normal and often acceptable pattern.

Test metrics and formulas

Defect data is only half of the picture. A test lead also tracks how much testing was done, how it went and how long it took. The formulas below appear in nearly every test summary report and interview. Remember that each needs a stated definition, and none is a goal by itself.

Execution and progress

Metric Formula Reads as
Test execution rate executed / planned x 100 How far through the plan we are
Pass rate passed / executed x 100 How much of what ran worked
Fail rate failed / executed x 100 The share that exposed a defect
Blocked rate blocked / planned x 100 How much is waiting on something else
Requirement coverage requirements with at least one test / total requirements x 100 Whether anything has no test at all
Test case effectiveness defects found by test cases / total defects found x 100 How much testing found, compared with luck and users
Automation coverage automated cases / total cases x 100 The share of the suite that runs without a person
Metric Formula Reads as
Defect density defects / size (KLOC, function points, story points) Where quality is poor
Defect leakage defects found after release / (found before + after) x 100 What testing missed
Defect removal efficiency defects removed before release / all defects x 100 The same idea, seen from the positive side
Defect rejection rate rejected defects / reported defects x 100 Quality of bug reports (too many rejects waste time)
Defect reopen rate reopened / fixed x 100 Quality of fixes
Mean time to detect / repair average time from introduction (or report) to detection (or fix) Feedback speed

Effort and estimation

Metric Formula
Test case productivity test cases written / person-days
Execution productivity test cases executed / person-days
Effort variance (actual effort - planned effort) / planned effort x 100
Schedule variance (actual duration - planned duration) / planned duration x 100
Cost of quality prevention + appraisal + failure costs

Worked calculation

A release has 200 planned tests. After the cycle: 180 executed, 150 passed, 24 failed, 6 blocked, 20 not run. Testing found 40 defects before release; 10 more were reported by users in the first month. The product is 50 KLOC. There are 36 requirements and 33 have tests.

  • Execution rate = 180 / 200 = 90%
  • Pass rate = 150 / 180 = 83.3%
  • Blocked rate = 6 / 200 = 3%
  • Requirement coverage = 33 / 36 = 91.7%
  • Defect leakage = 10 / 50 = 20%; removal efficiency = 40 / 50 = 80%
  • Defect density = 50 / 50 = 1.0 per KLOC
JS Compute the release metrics

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The last check is a habit worth keeping: totals that do not add up reveal data-entry mistakes before they reach a report.

Reading metrics responsibly

  • Always report a metric with its definition, scope and date.
  • Look at trends, not single numbers: a pass rate of 90% that rises each day is healthy; one that falls is not.
  • A high pass rate may mean weak tests; a low one may mean the tests are doing their job.
  • Never turn a metric into a quota (tests per day, bugs per tester): people will optimise the number instead of the quality.
  • Pair numbers with a sentence of context.

Using metrics wisely

Numbers can mislead as easily as they inform.

  • Never rank people by defects found or fixed. Testers will split one bug into five, developers will argue about every report, and the numbers become useless.
  • Compare like with like. Density across a one-person script and a payment platform means little.
  • Look at trends, not single values. A leakage of 20% is a number; leakage rising from 5% to 20% over three releases is a story.
  • Small samples lie. Ten defects are enough to practice the arithmetic, not to draw conclusions about a team.
  • A metric is a question, not a goal. When a metric becomes a target, people optimize it instead of quality.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A developer marks a defect as Fixed. Who should move it to Verified?

    Fixed is a claim by the developer. Verified means a tester retested the scenario and confirmed the problem is gone.
  2. 2. System testing found 45 defects and 5 more were reported by users after release. What is the defect leakage, using all defects as the base?

    Leakage is escaped defects divided by all defects: 5 / 50 = 10%. The detection effectiveness is the other 90%.
  3. 3. Module A has 12 defects in 6 KLOC. Module B has 9 defects in 1.5 KLOC. Which is more defect-prone by density?

    A has 12/6 = 2 defects per KLOC and B has 9/1.5 = 6. Density compares modules of different sizes fairly.
  4. 4. Which outcome shows a useful root cause analysis?

    An RCA should end with a concrete preventive action and an owner. General promises do not change how the team works.
  5. 5. A defect in the tracker moves from Fixed straight to Closed with no retest. What does a good workflow do?

    The lifecycle requires verification between Fixed and Closed, so a workflow should refuse the shortcut.

Summary

  • A defect moves through states such as New, Assigned, Fixed, Verified and Closed, with side paths Reopened, Rejected, Duplicate and Deferred.
  • Testers own verification; Fixed is only a claim until a retest passes.
  • Triage decides validity, duplicates, severity, priority and owner, quickly.
  • Root cause analysis (for example, the five whys) should end in a preventive action.
  • Defect density, detection effectiveness, leakage, age and reopen rate each answer a different question.
  • State your metric definitions, compare like with like, and never use defect counts to judge people.

Next up: Exploratory Testing in Practice.