Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 60 ยท Test Design Techniques

Pairwise and Combinatorial Testing

Learn why test combinations explode, how pairwise (all-pairs) testing cuts them by 80 percent or more, and build a greedy pairwise generator in JavaScript.

Imagine you test a checkout page that must work in 3 browsers, on 3 operating systems, in 3 languages and on 2 kinds of network. That is 3 x 3 x 3 x 2 = 54 combinations. Add a payment method with 4 options and a device type with 3, and you are at 648. Nobody runs 648 tests for one page, yet skipping combinations at random feels irresponsible.

Combinatorial testing is the family of techniques that picks a smart subset of combinations. The best known is pairwise testing (also called all-pairs testing): choose a small set of tests so that every pair of values from any two parameters appears together in at least one test. It is based on a practical observation, covered below, and it is one of the highest-value techniques for configuration matrices. It follows the seven testing principles: since exhaustive testing is impossible, you need a principled way to choose.

Why pairwise works

Explosion Every new parameter multiplies the test count.
Few factors Most failures involve one or two parameters, not five.
Smart subset Cover every pair of values at least once.

Pairwise trades a small risk for a huge saving in tests.

1 / 2

The combinatorial explosion

The number of full combinations is the product of the number of values for each parameter. It grows exponentially, which means that adding parameters is far more expensive than adding values.

JS How fast combinations grow

Press Run (or Ctrl + Enter) to see the console output here.

No output.

For our four parameters there are 54 full combinations but only 45 distinct pairs of values to cover. Each test row contains 6 pairs at once (four parameters give six pairs of parameters), so a handful of well-chosen rows can cover all 45. As parameters grow, the full count explodes while the number of pairs grows only gently, and that is where the saving comes from.

The idea behind pairwise testing

Studies of real defects, repeated across many kinds of software, point the same way: most failures are triggered by a single parameter value, and a large majority of the rest by the interaction of two. Failures that need three or more specific values at the same time exist but become rare quickly. Pairwise testing bets on this pattern.

An example of a single-value bug: the page crashes in Safari, whatever else is set. An example of a pair bug: the date shows wrongly only for the German locale on Linux. An example of a three-way bug: a script times out only when Chrome, Windows and a slow network meet. A pairwise suite will always find the first two kinds, and may or may not hit the third by luck.

Pairwise is a risk-reduction technique, not a guarantee. When the cost of failure is high, raise the strength: 3-wise (every triple covered) needs more tests but still far fewer than all combinations.

Orthogonal arrays

Before software tools, testers borrowed orthogonal arrays from statistics and experiment design. An orthogonal array is a table in which, for any two columns, every combination of values appears the same number of times. The classic name L4(2^3) means: 4 rows, 3 columns, each column with 2 values.

TEXT
Test   A   B   C
1      1   1   1
2      1   2   2
3      2   1   2
4      2   2   1

Pick any two columns, for example A and C: the pairs are (1,1), (1,2), (2,2), (2,1), so all four combinations appear exactly once. Three two-valued parameters would need 8 full combinations; the array covers all pairs in 4. Standard arrays exist for other sizes (L9(3^4) covers four three-valued parameters in 9 rows). Arrays are neat, but they only fit "balanced" problems where all parameters have the same number of values. Real matrices are uneven, which is why testers use generators instead.

JS Check that L4 is an orthogonal array

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Generating a pairwise suite

Real tools (such as PICT, AllPairs and many online generators) use more elaborate algorithms, but the core idea is a greedy search:

  1. List every pair of values that must be covered (the "uncovered" set).
  2. Look at candidate test rows and pick the one that covers the most uncovered pairs.
  3. Add it to the suite and remove its pairs from the uncovered set.
  4. Repeat until nothing is uncovered.

Greedy does not always find the smallest possible suite, but it gets close and is simple to understand. The generator below is a stand-in for a professional tool: it considers all 54 full rows as candidates, which is fine for a small matrix.

JS A greedy pairwise generator

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Nine tests replace 54. Nine is also the theoretical minimum here, because the two largest parameters have 3 x 3 = 9 value pairs and each test row can cover only one of them. In the output, each browser appears with each OS exactly once, and every value of every parameter shows up in at least three rows.

What pairwise catches, and what it misses

Planted bugs make the trade-off visible. The stand-in "app" below fails in three situations: Safari alone (a single-value bug), Safari with ja-JP (a pair), and Chrome with Windows on a slow network (a triple). The script runs both the full suite and the pairwise suite against it.

JS Planted bugs: pairwise suite versus full suite

Press Run (or Ctrl + Enter) to see the console output here.

No output.

In this run the pairwise suite finds the single-value bug (3 of its 9 tests fail) and the pair bug (1 test fails), but it misses the triple bug: none of its 9 rows happens to contain Chrome, Windows and a slow network together, while 3 of the 54 full combinations do. That is the trade-off in one picture. Pairwise guarantees pairs only, and a different tie-break could have caught the triple by luck. If a triple is important, add a test for it by hand or ask the tool for 3-wise strength.

Using pairwise in practice

Choose parameters and values carefully. Use equivalence partitioning to pick values: one representative per partition, plus boundaries where they matter. A parameter with 12 values multiplies cost, so reduce it to the partitions that behave differently (equivalence partitioning and boundary value analysis show how).

Handle constraints. Some combinations are impossible: Safari does not run on Windows today, a "pay by invoice" option may be offered only in some countries. Good tools let you list forbidden combinations and exclude them. If you remove them by hand after generation you will break the pair coverage.

Add high-risk combinations by hand. Pairwise gives you the base. Add the combinations that your analytics say most customers use, and those that failed in the past.

Keep expected results separate. A generator only creates inputs. You still need to know what the correct outcome is for each row, which sometimes needs an oracle or a rule.

Do not use it for rules. If the outcome depends on a combination of conditions (for example "a discount applies only when the member is gold AND the basket is over 100"), the rules matter in every combination, and a decision table is the right tool.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does a pairwise test suite guarantee?

    Pairwise (all-pairs) covers every combination of two values from two different parameters at least once, not every full combination.
  2. 2. A form has 5 parameters with 4 values each. How many full combinations are there?

    The full count is the product: 4 x 4 x 4 x 4 x 4 = 4^5 = 1,024.
  3. 3. Why does pairwise testing work well in practice?

    Empirical defect studies show that single values and two-parameter interactions cause the large majority of failures.
  4. 4. Which problem is pairwise testing a poor fit for?

    When the result depends on specific rule combinations, every combination of rule conditions matters, so a decision table is better.
  5. 5. A pairwise suite passes, but a bug appears only for Chrome + Windows + slow network. What does this show?

    Pairwise guarantees pairs only. For risky interactions, use higher strength (3-wise) or add hand-picked tests.

Summary

  • Full combinations grow exponentially; pairwise testing covers every pair of values with far fewer tests.
  • It works because most failures involve one or two parameters.
  • Orthogonal arrays are the classic hand-made answer; generators handle uneven real-world matrices and constraints.
  • A greedy algorithm (pick the row covering the most uncovered pairs, repeat) builds a good suite.
  • Pairwise is not a guarantee: add high-risk combinations by hand or use 3-wise strength, and use decision tables for business rules.

Continue with Use Case and Scenario Testing.