Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 60 ยท Test Design Techniques

White-Box Testing and Code Coverage

Learn white-box testing and code coverage: statement, decision, condition and path coverage, cyclomatic complexity, and why 100 percent coverage is not enough.

Black-box techniques such as equivalence partitioning design tests from the requirements without looking at the code. White-box testing (also called structural or glass-box testing) does the opposite: you read the code, and you design tests so that the code itself is exercised thoroughly. The question it answers is "which parts of the program has my test suite never run?"

The answer is expressed as code coverage: the percentage of some element of the code (statements, decisions, conditions, paths) that the tests executed. Coverage is a measurement tool. It is excellent at pointing out what you have not tested, and poor at telling you whether what you did test was tested well. This lesson explains the common coverage levels, shows how to measure them with a tiny home-made instrumenter, and finishes with the reason 100 percent coverage never means "done".

Coverage levels, weakest to strongest

Statement Was every line executed?
Decision Was every if taken both ways?
Condition Was every part of a && or || both true and false?
Path Was every route through the code run?

Stronger levels need more tests, and path coverage is usually impossible in full.

1 / 2

Where white-box testing fits

White-box techniques work at any test level, but you meet them most at the unit and integration levels (see test levels), where developers write the tests and know the code. A manual tester usually does not read source, but understanding coverage is still useful: it explains what teams mean by "we have 80 percent coverage" and what that number cannot promise.

We will use one function for all the examples. It computes a shipping cost.

JavaScript
function shippingCost(weight, isMember, country) {
  if (weight <= 0) return "invalid";
  let cost;
  if (weight < 1) cost = 5;
  else if (weight < 5) cost = 8;
  else cost = 12;
  if (isMember && cost > 5) cost -= 2;
  if (country !== "US") cost += 10;
  return cost;
}

Light parcels cost 5, medium ones 8, heavy ones 12. Members get 2 off, but only when the cost is above 5. Anything shipped outside the US costs 10 more.

Statement, decision and condition coverage

Statement coverage is the percentage of executable statements the tests ran. If the function has 7 statements and your tests ran 6, coverage is about 86 percent. It is the weakest measure: it ignores the case where an if without an else is always true in your tests.

Decision coverage (often called branch coverage) requires every decision, such as an if or a loop condition, to evaluate to both true and false at least once. It subsumes statement coverage: if you take every branch, you run every statement. The reverse is false.

Condition coverage looks inside a compound decision. In isMember && cost > 5, there are two atomic conditions. Condition coverage asks that each one has been both true and false. It is possible to reach 100 percent decision coverage without it, because of short-circuit evaluation: when isMember is false, JavaScript never looks at cost > 5.

The stand-in below does what real coverage tools do, only by hand. It wraps each decision in a small recording function (b) and each statement in a marker (s), runs a test suite, and reports what was and was not seen. Tools such as Istanbul, JaCoCo and coverage.py insert similar probes automatically.

JS Measuring coverage with hand-made probes

Press Run (or Ctrl + Enter) to see the console output here.

No output.

One test reaches only 4 of 7 statements. Suite A reaches every statement and takes every decision both ways, yet it never saw cost > 5 false: no test had a member sending a light parcel. Suite B adds exactly that test and completes the condition coverage. A higher level asks for more tests, and finds gaps the lower level hides.

Path coverage and cyclomatic complexity

A path is one complete route from the start of the function to its end. Path coverage asks for every possible route. In our function, the weight decision has three outcomes, then the member discount is applied or not, then the country surcharge is applied or not. Some combinations cannot happen: a light parcel costs 5, so the member discount (needs a cost above 5) can never apply to it. Counting the feasible routes by hand is tedious, so the code below counts them from a grid of sample inputs.

Path coverage grows fast. Each independent if doubles the number of paths, and a loop can make the number infinite. That is why full path coverage is almost never a goal.

Cyclomatic complexity (McCabe) measures how many independent paths the code has. For a single function, it equals the number of decision points plus one. Each if, else if, loop, case, && and || counts as one decision. The complexity tells you the minimum number of tests needed to cover every independent path (called the basis set), and it is a warning sign: functions above about 10 are hard to understand and test.

JS Cyclomatic complexity and path counts

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The complexity is 7, so seven well-chosen tests cover all independent routes, while there are 11 distinct feasible paths. With 20 independent decisions there would be more than a million paths. Cyclomatic complexity is the practical middle ground: it tells you how many tests the structure demands without enumerating every route.

Control-flow and data-flow testing

Coverage criteria such as statement and decision coverage come from the control-flow graph (CFG): a diagram where each node is a block of statements that always run together and each arrow is a possible jump. Data-flow testing adds a second view: it follows variables from where they get a value to where it is used.

Reading a control-flow graph

For shippingCost above, the nodes are: the weight check, the three weight branches, the member discount, the country surcharge and the return. Each if creates a diamond with two outgoing edges.

flowchart TB
  A["weight <= 0 ?"] -- yes --> R1["return invalid"]
  A -- no --> B["weight < 1 ?"]
  B -- yes --> C1["cost = 5"]
  B -- no --> C2["weight < 5 ? cost = 8 : cost = 12"]
  C1 --> D["member and cost > 5 ?"]
  C2 --> D
  D -- yes --> E["cost -= 2"]
  D -- no --> F["country != US ?"]
  E --> F
  F -- yes --> G["cost += 10"]
  F -- no --> H["return cost"]
  G --> H

Each path from the top to a return is one possible execution. Statement coverage visits every box; decision coverage takes every yes and no arrow; path coverage tries every route.

Useful numbers come straight from the graph. Cyclomatic complexity is edges minus nodes plus 2 (for one entry and one exit), and it equals the number of independent paths that a basis path test set must cover. The earlier section on path coverage shows how to count them.

Data-flow testing

Bugs often hide not in the branches but in how data travels. Data-flow testing classifies each occurrence of a variable:

  • Definition (def): the variable receives a value (x = 5, a parameter, read(x)).
  • Computation use (c-use): the value is used in a calculation or output (y = x + 1).
  • Predicate use (p-use): the value is used in a decision (if (x > 3)).
  • Kill / undefine: the variable goes out of scope or is released.

A def-use pair (du-pair) is a definition together with a use that it can reach with no redefinition in between. Data-flow coverage criteria ask you to cover such pairs:

  • All-defs: each definition reaches at least one use.
  • All-uses: each definition reaches every use it can reach.
  • All-du-paths: every path between each definition and use is covered (the strongest and the largest).

Data-flow anomalies

Looking at the sequence of events on one variable reveals suspicious patterns, which static analysers also report:

Pattern Name Meaning
define, define dd The first value is never used: wasteful and often a bug.
define, kill dk A value was computed and thrown away.
use before any define ur Reading an uninitialised variable: usually a real bug.
kill, use ku Using a released variable.

The stand-in below scans a list of events for each variable and flags the anomalies. It models what a data-flow analyser does, without parsing real code.

JS Find data-flow anomalies in a trace

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Which du-pairs does a test set cover?

A du-pair is covered when a test actually executes a path from the definition to the use. In the next function, rate is defined in one of two branches and used afterwards.

JS Cover every definition-use pair

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Test set 1 never takes the bulk or the member path, so two pairs are missing. Test set 2 covers all four pairs with just two tests, and test set 3 adds nothing new from a data-flow view. Data-flow criteria therefore help you choose a small set that still exercises every way a value can travel, which plain statement coverage does not guarantee.

Use data-flow thinking when a function has several assignments to the same variable or when state is passed between steps. For most teams, static analysers and linters report the anomalies automatically, and manual data-flow design is kept for critical logic.

Loop testing

Loops are a classic home for off-by-one mistakes, so white-box testers give them their own technique: loop testing designs inputs around the number of times a loop body runs. It checks termination logic, boundary iterations and the interaction of nested loops.

For a simple loop that can run at most n times, test these cases:

  1. Skip the loop entirely (zero passes).
  2. Exactly one pass.
  3. Exactly two passes.
  4. A typical number of passes, m, where m < n.
  5. n - 1, n and n + 1 passes.

For nested loops, testing every combination explodes quickly. Instead, start with the innermost loop and set the outer loops to their minimum values, then test the inner loop with the cases above. Work outward one loop at a time while holding the inner loops at typical values. Concatenated loops that are independent can be tested as simple loops; if the second loop uses a value computed by the first, treat them as nested. Unstructured loops (built with goto or complicated flags) should be redesigned rather than tested.

The technique finds loops that never terminate, loops that stop one iteration early or late, and code that fails on empty input. Its limit is cost: the cases multiply with nesting depth, and it needs source code, so it suits developers and unit tests more than black-box testers.

JS Loop testing a simple loop

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The seven cases exercise zero, one, two, typical and boundary passes, and the last line checks the empty input. If you change i < n to i <= n, the "max" and "max + 1" cases fail at once.

Why 100 percent is not enough

Coverage answers "was this code executed?" It cannot answer "did the test check the right thing?" Four common ways to reach a high number and still ship bugs:

  1. Tests without assertions. A test that calls a function and checks nothing still raises the coverage figure.
  2. Wrong expectations. The test checks the output, but the author copied the behaviour of the code rather than the requirement.
  3. Missing code. Coverage only measures code that exists. If the requirement says "reject negative quantities" and nobody wrote that check, no coverage report will point it out.
  4. Boundaries and data. A decision taken both ways may still be taken at the wrong place, for instance >= where the requirement says >.

The stand-in below shows two of these. A discount applies to orders of more than 10 items, but the code uses >=. Two tests (20 items and 1 item) take both branches, so decision coverage is 100 percent, and the tests with correct assertions pass, yet the bug lives on. Only a boundary test, which comes from boundary value analysis and not from coverage, exposes it.

JS 100 percent decision coverage, bug still there

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Suites 1 and 2 both report 100 percent decision coverage and nothing fails. Only the boundary check at exactly 10 items fails: the code gives 45, the requirement says 50. Coverage was measuring the wrong thing, and it was never designed to catch this.

How to use coverage well

  • Use it as a map of gaps. Open the report, look at the red lines, and ask "should a test cover this?" Sometimes the answer is that the code is dead and should be deleted.
  • Do not turn a number into a target. If a team is rewarded for 90 percent, it will produce 90 percent, with tests that assert nothing. Set a floor to prevent decline and focus on reviewing the quality of tests.
  • Combine with black-box design. Use equivalence partitioning and boundaries to decide what to test, then use coverage to check what you missed.
  • Pay attention to complexity. A function with a cyclomatic complexity of 25 needs many tests, and is a candidate for splitting.
  • Know the other uses. Mutation testing (changing the code on purpose to see whether tests notice) measures whether tests check anything, and is a good companion to coverage.

Coverage tools exist for every major language. They are real and widely used, but they run on your machine or CI server, not inside this page, so the stand-ins here are hand-written probes that show the same idea.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Which coverage level is the weakest of these?

    Statement coverage only asks whether each statement ran. It ignores which way decisions went.
  2. 2. A function has 4 if statements, 1 loop and 2 uses of && (and no other branching). What is its cyclomatic complexity?

    Decision points are 4 + 1 + 2 = 7, and complexity is decision points plus one, so 8.
  3. 3. In 'if (a && b)', a is false in every test. Which statement is true?

    With short-circuit evaluation, b is skipped when a is false, so b can never be seen true or false. The decision as a whole is false.
  4. 4. Why can a suite with 100 percent branch coverage still miss an off-by-one defect?

    Coverage records that each branch ran, not at which values. Boundary tests are needed to expose a wrong operator such as >= instead of >.
  5. 5. What is the best way to use a coverage report?

    Coverage shows what has not been run. Whether the tests check the right things must be judged by reviewing assertions, boundaries and requirements.

Summary

  • White-box testing designs tests from the code structure, and code coverage measures how much of it the tests ran.
  • Statement coverage is weakest; decision (branch) coverage subsumes it; condition coverage looks inside && and ||; path coverage is usually unrealistic.
  • Cyclomatic complexity (decision points plus one) estimates how many tests the structure demands and flags code that is too complex.
  • 100 percent coverage does not mean correct: assertions, boundaries, missing code and wrong expectations all escape it.
  • Use coverage to find gaps, not as a target, and combine it with black-box design techniques.

Continue with Experience-Based Testing.