Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 29 ยท Real-world projects

Project: Build a Mini Test Framework

Build your own tiny JUnit-style test framework in Java with annotations, reflection discovery, lifecycle methods, assertions and a report, then see how real tools work.

JUnit and TestNG can feel like magic: you put @Test on a method, and somebody calls it. In this project you remove the magic. You will build a small test framework in plain Java, about 150 lines, with annotations, reflection-based discovery, @BeforeEach and @AfterEach hooks, assertions, expected exceptions, skipping and a final report.

Nothing here uses the real JUnit or TestNG, and that is the point. The framework below is a stand-in: it is far simpler than the real tools (no parallel runs, no extension model, no IDE integration), but the core loop is the same one that JUnit's Jupiter engine and TestNG run: find classes, find annotated methods, create objects, call methods, catch what they throw, count the results. Once you have written it yourself, a stack trace from a real framework stops being scary.

Everything runs on our Java runner. Press Run on each block.

The plan

Five parts, one loop

Annotations @Test, @BeforeEach, @AfterEach, @Disabled.
Discovery Reflection finds the marked methods.
Assertions Static methods that throw on a mismatch.
Report Pass, fail, error and skip, with a summary.

Real frameworks add many features, but never drop these five.

1 / 2

Three outcomes matter, and we will keep the words JUnit uses. A test passes when it returns. It fails when it throws our assertion error (a wrong value). It errors when it throws anything else (the code crashed). A @Disabled test is skipped.

Step 1: annotations and discovery

An annotation is metadata attached to code. By itself it does nothing. A framework gives it meaning by reading it with reflection. Two details are essential: @Retention(RetentionPolicy.RUNTIME) keeps the annotation in the compiled class (the default, CLASS, drops it before runtime, so reflection would never see it), and @Target limits where it can be written.

The first block defines the annotations and a scanner that lists what it finds. It runs nothing yet.

Java Discovering annotated methods

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Notice the sort. Reflection does not promise any method order, and neither do the real frameworks: JUnit 5 and later use a deterministic but deliberately non-obvious order, and TestNG sorts by name by default. If your tests only pass in one order, they are broken (see Flaky Tests and Common Mistakes).

Step 2: assertions

Assertions are the smallest part. They compare, and they throw when the comparison fails. We throw our own AssertionFailed so the runner can tell "wrong value" from "crash", exactly like AssertionFailedError in JUnit (JUnit Assertions in Depth).

Java
static class AssertionFailed extends RuntimeException {
    AssertionFailed(String message) { super(message); }
}

static void assertEquals(Object expected, Object actual) {
    if (!Objects.equals(expected, actual)) {
        throw new AssertionFailed("expected: <" + expected + "> but was: <" + actual + ">");
    }
}

Objects.equals handles null on either side, which a bare expected.equals(actual) would not.

Step 3: the runner

The runner is a loop with four responsibilities. For each test class it creates a new instance per test method. That is why instance fields do not leak between tests in JUnit by default. Then it runs @BeforeEach methods, the test, and @AfterEach methods, with the after-hooks in a finally so they run even when the test fails. Last it classifies whatever was thrown.

One reflection detail trips everybody up: Method.invoke wraps any exception from the target method in an InvocationTargetException. You must unwrap it with getCause(), otherwise every test would look like it crashed in the same way.

The complete framework follows, together with a small class under test and a test class that exercises all four outcomes. Read the code top to bottom before pressing Run.

Java A complete mini framework

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Run it and match each line with its cause. wrongExpectedValue is a FAIL (our assertion threw). averageOfEmptyBagCrashes is an ERROR (an ArithmeticException from division by zero reached the runner). negativePriceIsRejected passes because an IllegalArgumentException was exactly what we declared. forgotToThrow fails because the code did not throw what the annotation promised. The six cleanup lines at the top come from @AfterEach, once per executed test and not for the skipped one; they print before the report because the report is built only after every test has run.

The expected = ... attribute is the JUnit 4 and TestNG way of testing exceptions (@Test(expected=...) and expectedExceptions). JUnit Jupiter replaced it with assertThrows, which lets you check the message and limits the check to one statement, so an exception thrown by your set-up code cannot accidentally satisfy the test.

How this maps to the real tools

Mini framework JUnit Jupiter TestNG
@Test @Test @Test
@BeforeEach / @AfterEach @BeforeEach / @AfterEach @BeforeMethod / @AfterMethod
New instance per test Default (PER_METHOD lifecycle) One instance per class
expected = X.class assertThrows(X.class, ...) expectedExceptions = X.class
@Disabled @Disabled enabled = false
runClass + report Launcher, engine, listeners TestNG runner, ITestListener

The last row hides most of the real work. JUnit splits it in two: the Platform (launcher, discovery request, result listeners) and engines such as Jupiter that know how to discover and run one style of test. TestNG is a single library with its own runner and its own XML suite format. The differences come up again in JUnit vs TestNG.

Extend it

Good features to add yourself, each a few lines:

  • @BeforeAll / @AfterAll: collect static methods and call them once around the loop in runClass.
  • @Test(timeoutMillis = ...): run the test on another thread and fail it if it takes too long. Real timeouts need care, since a stuck thread cannot be killed safely.
  • Selecting by name: let main accept a name filter and skip tests that do not contain it, the idea behind -Dtest= and --tests.
  • A non-zero exit code if anything failed, so a build server notices. Real runners always do this.
  • Test ordering by annotation: an @Order(int) annotation and a comparator.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why must a custom test annotation use @Retention(RetentionPolicy.RUNTIME)?

    With the default retention the annotation is dropped before runtime, so isAnnotationPresent would always return false.
  2. 2. A test method throws IllegalStateException when invoked through Method.invoke. What does the runner catch first?

    Reflection wraps exceptions from the invoked method in InvocationTargetException. The runner must call getCause() to learn what really happened.
  3. 3. Why does the framework create a new test-class instance for every test method?

    A fresh object per test isolates tests from each other's state, the same default JUnit Jupiter uses.
  4. 4. Where should after-hooks be called inside the runner?

    Clean-up must happen whether the test passed, failed or crashed, otherwise one failure leaves resources behind for later tests.

Summary

  • A test framework is annotations plus reflection plus a loop: discover, instantiate, run hooks, invoke, classify, report.
  • Annotations need RUNTIME retention to be visible to reflection.
  • Unwrap InvocationTargetException to see the real exception, and call after-hooks in finally.
  • Separate failures (assertion errors) from errors (any other exception); both turn a build red, but they point at different causes.
  • Reflection gives no method order, so a deterministic sort is part of every framework.
  • Real tools add parallel runs, extensions, rich reports and build-tool integration, but the core loop is the one you just wrote.

Next: Project: Testing a REST API Suite with TestNG, where data providers, groups and listeners meet an HTTP API.