Test faster, fix more

There are a lot of ports of QuickCheck, the original property based testing library, to a variety of different languages.

Some of them are good. Some of them are very good. Some of them are OK. Many are not.

We thought it would be worth keeping track of which are which, so we've put together a list.

In order to make it onto this list, an implementation has to meet the following criteria:

  1. Must support random generation of data to a test function. e.g. testing systems based on smallcheck while interesting and related don't fit on this list.
  2. It must be fairly straightforward to generate your own custom types.
  3. It must support shrinking of falsifying examples.
  4. It must be under active development, in the sense that bugs in it will get fixed.
  5. Must be under an OSI approved license.

In this we've tried to to collect a list of what we think the best ones are for any given language. We haven't used all of these, but we've used some and read and talked to people about the others.

Uncontested Winners by Language

For many languages there is a clear winner that you should just use. Here are the ones we've found and what we think of them.

Language Library Our rating
C theft Good but does not come with a library of data generators.
C++ CppQuickCheck Unsure
Clojure test.check Very good
Coq QuickChick Unsure
F# FsCheck Very Good
Go gopter Unsure but looks promising.
Haskell Quickcheck Very good but remarkably poorly documented.
Java QuickTheories Unsure. Extremely new but looks promising.
JavaScript jsverify Good
PHP Eris Unsure. Looks promising.
Python Hypothesis We may be a bit biased on this one.
Ruby Rantly Unsure. We're not convinced, but the alternatives are definitely worse.
Rust Quickcheck Unsure, but probably very good based on initial appearance and usage level.
Scala ScalaCheck Very Good
Swift Swiftcheck Unsure

Where when we've said "Unsure" we really just mean that we think it looks good but we haven't put in the depth of in time to be sure, not that we have doubts.

Special case: Erlang

Erlang is a special case because they have QuviQ's QuickCheck. Their QuickCheck implementation is by all accounts extremely good, but it is also proprietary and fairly expensive. Nevertheless, if you find yourselves in the right language, circumstance and financial situation to use it, we would strongly recommend doing so.

In particular, QuviQ's QuickCheck is really the only implementation in this article we think is simply better than Hypothesis. We suspect Hypothesis to be significantly more user friendly, especially if the users in question are less than familiar with Erlang, but there are many things QuviQ can do that Hypothesis can't and relatively few in the other direction.

If you're using Erlang but not able to pay for QuickCheck, apparently the one to use is PropEr. If you're also unable to use GPLed software there's triq. We know very little about either.

Special case: OCaml

OCaml seems to be suffering from a problem of being close enough to Haskell that people try to do a straight port of Quickcheck but far enough from Haskell that this doesn't work. The result is that there is a "mechanical port" of Quickcheck which is completely abandoned and a fork of it that uses more idiomatic OCaml. We are insufficiently familiar with OCaml or its community to know if either is used or whether there is another one that is.

What does this have to do with Hypothesis?

In some sense these are all "competitors" to Hypothesis, but we're perfectly happy not to compete.

In the case of Erlang, we wouldn't even try. In the case of Scala, F#, or Clojure, we might at some point work with them to try to bring the best parts of Hypothesis to their existing implementation, but we don't consider them a priority - they're well served by what they have right now, and there are many languages that are not.

For the rest though? We're glad they exist! We care about testing about about high quality software, and they're doing their part to make it possible.

But we feel they're being badly served by their underlying model, and that they feel quite unnatural to use in the context of a more conventional test setting. We think Hypothesis is the way forward, and we'll be doing our best to make it possible for everyone to use it in their language of choice.