Test faster, fix more
There are plenty of other implementations of property based testing!
We think Hypothesis is the future, and by far the best in the languages that we’ve done serious implementations for, but some of the others are definitely also worth paying attention to.
So we have. These are articles we’ve written taking a look at other implementations of property based testing and related concepts.
If you look at the original property-based testing library, the Haskell version of QuickCheck, tests are very closely tied to types: The way you typically specify a property is by inferring the data that needs to be generated from the types the test function expects for its arguments.
This is a bad idea.
In my last article about shrinking, I discussed the problems with basing shrinking on the type of the values to be shrunk.
In writing it though I forgot that there was a halfway house which is also somewhat bad (but significantly less so) that you see in a couple of implementations.
This is when the shrinking is not type based, but still follows the classic shrinking API that takes a value and returns a lazy list of shrinks of that value. Examples of libraries that do this are theft and QuickTheories.
This works reasonably well and solves the major problems with type directed shrinking, but it’s still somewhat fragile and importantly does not compose nearly as well as the approaches that Hypothesis or test.check take.
Ideally, as well as not being based on the types of the values being generated, shrinking should not be based on the actual values generated at all.
This may seem counter-intuitive, but it actually works pretty well.
One of the big differences between Hypothesis and Haskell QuickCheck is how shrinking is handled.
Specifically, the way shrinking is handled in Haskell QuickCheck is bad and the way it works in Hypothesis (and also in test.check and EQC) is good. If you’re implementing a property based testing system, you should use the good way. If you’re using a property based testing system and it doesn’t use the good way, you need to know about this failure mode.
Unfortunately many (and possibly most) implementations of property based testing are based on Haskell’s QuickCheck and so make the same mistake.
There are a lot of ports of QuickCheck, the original property based testing library, to a variety of different languages.
Some of them are good. Some of them are very good. Some of them are OK. Many are not.
I thought it would be worth keeping track of which are which, so I've put together a list.