Skip to content
Tutorials Center
My learning

Topics

Lesson 23 of 60 ยท Test Design Techniques

Mutation Testing

Learn mutation testing: how small code changes called mutants measure the real strength of your tests, mutation score, equivalent mutants, and tools.

Code coverage tells you which lines your tests ran. It does not tell you whether the tests would notice if those lines were wrong. A test that calls a function and checks nothing gives 100 percent coverage and zero protection. Mutation testing attacks that blind spot. It asks a sharper question: "if I deliberately break the code in a small way, do my tests fail?"

If they do not, you have found a hole in the tests, not in the code. This lesson explains mutants, the mutation score, the common pitfalls and the tools, with a runnable mutation tester in miniature.

The idea

Take a working program and a test suite that passes. Then:

  1. Make a tiny change to the code, such as turning > into >=. The changed version is called a mutant.
  2. Run the test suite against the mutant.
  3. If at least one test fails, the mutant is killed: your tests detected the change.
  4. If all tests still pass, the mutant survived: your tests cannot tell the broken code from the correct code.

Repeat for many mutants. Every survivor points to behavior that the tests do not check.

flowchart LR
  P["Original code + passing tests"] --> M["Create a mutant<br/>(one small change)"]
  M --> R["Run the tests"]
  R --> K{"Any test fails?"}
  K -- yes --> KI["Mutant killed:<br/>tests are sensitive"]
  K -- no --> SU["Mutant survived:<br/>missing or weak test"]

Mutation testing is a test of the tests.

Mutation operators

A mutation operator is a rule that creates a mutant. Common groups:

Group Example change
Arithmetic a + b becomes a - b; * becomes /
Relational > becomes >=, == becomes !=
Logical && becomes ||; remove a !
Boundary i < n becomes i <= n
Constants 0 becomes 1; true becomes false
Statement removal delete a call or an assignment
Return value return null, 0 or an empty list
Conditional force a condition to always true or always false

Each mutant should differ from the original by one small change. That is based on two ideas from research: the competent programmer hypothesis (real bugs are mostly small slips) and the coupling effect (tests that catch simple faults tend to catch complex ones too).

First-order and higher-order mutants

A first-order mutant contains a single change; this is the normal choice. A higher-order mutant combines two or more changes, which are fewer in number to run but harder to analyze. Most tools use first-order mutants only.

The mutation score

The result is summarized as the mutation score:

mutation score = killed mutants / (total mutants - equivalent mutants) x 100

A score of 100 percent means every non-equivalent mutant was caught. Unlike coverage, it measures the strength of the assertions, not only whether code ran.

Equivalent mutants

Some mutants change the code's text but not its behavior. For example, replacing i < 10 with i != 10 in a loop that always stops at 10, or changing the order of two independent statements. No test can ever kill them. They are equivalent mutants, and they are the main practical headache: deciding that a surviving mutant is equivalent usually needs a human. That is why the score is usually reported with a note.

Try it: a tiny mutation tester

The stand-in below mutates a small function by editing its source text with simple operator swaps, then runs a "weak" suite and a "strong" suite against each mutant. It uses new Function, which is fine for a classroom example; do not do this with untrusted code.

JS Mutation testing in miniature

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Both suites would reach full line coverage of the three return statements, but only the strong suite pins down the boundaries at 0 and 18. The surviving mutants of the weak suite are exactly the boundary tests you forgot. Compare this with boundary value analysis.

Try it: improve a suite from the survivors

The loop below does what a developer does with a mutation report: for each survivor, add a test that kills it, then re-measure.

JS Killing survivors one test at a time

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Practical issues

  • Cost. Each mutant means a test run, and a real codebase generates thousands. Tools reduce this by running only the tests that cover the mutated line, running mutants in parallel, and mutating only code changed in a pull request.
  • Equivalent mutants inflate the survivor list; review them, and mark the harmless ones as ignored.
  • Not for everything. Use it on core logic and libraries, not on glue code or UI layout.
  • Do not chase 100 percent. Use survivors as hints. A readable test suite matters too.

Tools

You do not write mutants by hand. Typical tools by language:

  • Java: PIT (Pitest).
  • JavaScript and TypeScript: StrykerJS.
  • C#: Stryker.NET.
  • Python: mutmut, Cosmic Ray.
  • PHP: Infection.

They integrate with build tools and output an HTML report listing each survivor with the changed line. Run them in a nightly job or on changed files, not on every commit.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a surviving mutant tell you?

    If all tests still pass when the code is deliberately broken, the tests do not check that behavior.
  2. 2. A suite has 100% line coverage and a mutation score of 40%. What is the most likely explanation?

    Coverage measures execution, mutation score measures detection. Tests with few or weak assertions run the code without verifying it.
  3. 3. What is an equivalent mutant?

    Equivalent mutants behave exactly like the original, so they survive no matter how good the tests are.
  4. 4. Why is mutation testing usually applied selectively?

    Each mutant needs a test run, so teams limit it to core code or changed files and use tools to speed it up.

Summary

  • Mutation testing changes code in small ways and checks whether the tests fail.
  • Killed mutants show sensitive tests; survivors point to missing tests, especially at boundaries.
  • Mutation score = killed / (total - equivalent). It measures assertion strength, which coverage cannot.
  • Equivalent mutants and run time are the main costs; tools like PIT, Stryker, mutmut and Infection reduce the pain.
  • Use survivors as a to-do list for better tests, not as a number to maximize blindly.

Next up: Positive and Negative Testing, Error Guessing.