Skip to content
Tutorials Center
My learning

Topics

Lesson 39 of 57 ยท Modules, packages and tooling

Testing: HUnit, Hspec and QuickCheck

Test Haskell code with HUnit and Hspec for examples and QuickCheck for properties, and learn how generators, shrinking and doctest fit together.

Pure functions are the easiest code in the world to test: same input, same output, nothing to mock. Haskell makes that advantage bigger with property-based testing. Instead of writing a handful of examples, you state a rule that must hold for all inputs, and the library invents hundreds of inputs to try to break it.

This lesson covers example-based tests with HUnit and Hspec, property tests with QuickCheck, and doctest for examples in documentation. The Haskell code is read-only because GHC is not installed here; the runnable examples are JavaScript stand-ins that implement a small random tester with shrinking, so you can watch the idea work.

Two ways to test

Examples Input 3, expect 9.
Properties For any list, reversing twice gives it back.

Use both. They catch different bugs.

1 / 2

The code under test

Imagine a small module in the library of your project (see Cabal, Stack and Packages):

HASKELL
module Stats (average, clamp, insertSorted) where

average :: [Double] -> Maybe Double
average [] = Nothing
average xs = Just (sum xs / fromIntegral (length xs))

clamp :: Int -> Int -> Int -> Int
clamp lo hi x = max lo (min hi x)

insertSorted :: Int -> [Int] -> [Int]
insertSorted x [] = [x]
insertSorted x (y : ys)
  | x <= y    = x : y : ys
  | otherwise = y : insertSorted x ys

Because every function is pure and returns a Maybe instead of crashing on an empty list (see Maybe and Either), tests are one-liners.

HUnit: assertions

HUnit is the classic xUnit-style library. A test is a value of type Test; ~?= compares an actual value with an expected one.

HASKELL
import Test.HUnit
import Stats

tests :: Test
tests = TestList
  [ "average of empty"   ~: average []           ~?= Nothing
  , "average of three"   ~: average [1, 2, 3]    ~?= Just 2
  , "clamp below"        ~: clamp 0 10 (-5)      ~?= 0
  , "clamp inside"       ~: clamp 0 10 7         ~?= 7
  ]

main :: IO ()
main = do
  _ <- runTestTT tests
  return ()

Running it prints a summary such as the number of cases, tries, errors and failures. It works, but the output of failures is plain and the test names are easy to forget. Most new projects choose Hspec for the day-to-day style.

Hspec: readable specifications

Hspec arranges tests as a description of behaviour. The hspec package gives you describe, it and shouldBe:

HASKELL
import Test.Hspec
import Stats

main :: IO ()
main = hspec $ do
  describe "average" $ do
    it "is Nothing for an empty list" $
      average [] `shouldBe` Nothing
    it "computes the mean" $
      average [2, 4, 6] `shouldBe` Just 4

  describe "clamp" $ do
    it "raises small values to the lower bound" $
      clamp 0 10 (-5) `shouldBe` 0
    it "lowers large values to the upper bound" $
      clamp 0 10 99 `shouldBe` 10

The output reads like a specification: the describe names group the it lines, failures show the expected and actual values, and hspec-discover can find every *Spec.hs file in test/ so you never register tests by hand. In .cabal, the test stanza has type: exitcode-stdio-1.0, so cabal test passes only if every example does.

Other useful matchers: shouldSatisfy, shouldReturn for IO actions, shouldThrow for exceptions.

QuickCheck: properties

A property is a function that returns Bool (or a Property) and takes the data as arguments. QuickCheck generates random arguments, runs the property, and reports any counterexample.

HASKELL
import Test.QuickCheck
import Data.List (sort)
import Stats

prop_reverseTwice :: [Int] -> Bool
prop_reverseTwice xs = reverse (reverse xs) == xs

prop_clampInRange :: Int -> Int -> Int -> Property
prop_clampInRange lo hi x =
  lo <= hi ==> let r = clamp lo hi x in lo <= r && r <= hi

prop_insertKeepsSorted :: Int -> [Int] -> Bool
prop_insertKeepsSorted x xs = insertSorted x (sort xs) == sort (x : xs)

main :: IO ()
main = do
  quickCheck prop_reverseTwice
  quickCheck prop_clampInRange
  quickCheck prop_insertKeepsSorted

By default quickCheck tries 100 random cases per property. Points worth knowing:

  • QuickCheck chooses the generator from the type of the property's arguments, through the Arbitrary type class. [Int] gets random lists of random integers.
  • lo <= hi ==> ... is a precondition. Cases that violate it are discarded. If too many are discarded, QuickCheck gives up, so do not hide a rare condition behind ==> when you can generate valid data directly.
  • Properties are about relationships, not specific outputs. Good sources: round trips (decode (encode x) == x), idempotence (sort (sort xs) == sort xs), comparison with a slow simple version (insertSorted against sort), and invariants (the result is still sorted, the length is preserved).

Shrinking

When a property fails, the first random counterexample is often big and confusing, such as a list of 37 numbers. QuickCheck then shrinks it: it tries smaller variants (shorter lists, numbers nearer zero) and keeps any that still fail, until it reaches a minimal case. A report might look like Falsified (after 14 tests and 5 shrinks): [1,0]. That tiny list tells you much more than the original did.

Custom generators

For your own types, write an Arbitrary instance, or build generators with Gen combinators:

HASKELL
data Shape = Circle Double | Rect Double Double deriving (Show, Eq)

instance Arbitrary Shape where
  arbitrary = oneof
    [ Circle <$> choose (0.1, 100)
    , Rect <$> choose (0.1, 100) <*> choose (0.1, 100)
    ]

oneof picks one generator from the list, choose draws a number in a range, and <$> with <*> combine generators exactly as you combine other applicative values. forAll gen prop lets you use a generator for a single property without defining an instance, and Test.QuickCheck.Gen offers listOf, elements, frequency and more. Use cover or classify to check that your random data really exercises the interesting cases.

Property tests with Hspec

You do not have to choose between the libraries. The Test.Hspec.QuickCheck module provides prop, so properties sit beside the examples:

HASKELL
import Test.Hspec
import Test.Hspec.QuickCheck (prop)

main :: IO ()
main = hspec $
  describe "reverse" $
    prop "twice is the identity" $ \xs ->
      reverse (reverse (xs :: [Int])) == xs

doctest: examples that cannot go stale

doctest runs the examples you write in Haddock comments, using lines that start with >>>:

HASKELL
-- | Keep a value between two bounds.
--
-- >>> clamp 0 10 15
-- 10
-- >>> clamp 0 10 (-3)
-- 0
clamp :: Int -> Int -> Int -> Int
clamp lo hi x = max lo (min hi x)

The tool loads the module in GHCi, evaluates the expression and compares the printed result with the next line. Documentation that is also a test never drifts from the real behaviour.

A stand-in: a tiny QuickCheck

The first stand-in shows the core of QuickCheck: a generator, a loop of random cases and a report. The random numbers come from a seeded generator, so every run prints the same output.

JS Random testing (stand-in)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The third property is false for any list with two different elements, so the tester finds a counterexample within a few tries. The first counterexample is often long. The next stand-in adds shrinking.

JS Shrinking a counterexample (stand-in)

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The seven-element list shrinks to [4,6]: removing either element, or halving one of them, makes the sum drop below 10, so the search stops there. Real QuickCheck uses the same strategy, with shrinking functions that are defined per type in the Arbitrary class.

Practical advice

  • Test the pure core heavily and keep IO thin; pure code needs no setup.
  • Write the properties you can state in a sentence: "decoding what I encoded gives the original".
  • Make failing properties reproducible by printing the seed (quickCheckWith stdArgs { replay = ... } can replay it) and add every real bug as a plain example test too.
  • Use -Wall and total functions to remove whole classes of tests (see Debugging, Profiling and Performance).

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is a QuickCheck property?

    A property takes generated arguments and returns Bool or Property. QuickCheck runs it on many random inputs.
  2. 2. What does shrinking do?

    After a failure QuickCheck tries smaller inputs and keeps those that still fail, to give you a minimal counterexample.
  3. 3. How does QuickCheck know how to generate a [Int]?

    Arbitrary instances provide the generators (and shrinkers); lists of Int are built in.
  4. 4. Which of these is a good property for sort?

    Idempotence holds for every list. The first line is an example, and the other two are false for some inputs (such as the empty list).
  5. 5. What does a doctest line starting with >>> do?

    doctest runs the expression in GHCi and checks that the output matches the lines that follow.

Summary

  • Pure functions are easy to test: no mocks, no setup.
  • HUnit and Hspec check concrete examples; Hspec's describe/it style reads like documentation.
  • QuickCheck checks properties against random inputs chosen by Arbitrary; failures are shrunk to a minimal counterexample.
  • Good properties: round trips, idempotence, invariants, comparison with a simple reference.
  • Write custom generators with oneof, choose and applicative combinators.
  • doctest turns >>> examples in comments into tests.

Next: Debugging, Profiling and Performance.