There are a lot of ports of QuickCheck, the original property based testing library, to a variety of different languages.
Some of them are good. Some of them are very good. Some of them are OK. Many are not.
I thought it would be worth keeping track of which are which, so I've put together a list.
In order to make it onto this list, an implementation has to meet the following criteria:
- Must support random generation of data to a test function. e.g. testing systems based on smallcheck while interesting and related don't fit on this list.
- It must be fairly straightforward to generate your own custom types.
- It must support shrinking of falsifying examples.
- It must be under active development, in the sense that bugs in it will get fixed.
- Must be under an OSI approved license.
In this I've tried to to collect a list of what I think the best ones are for any given language. I haven't used all of these, but I've used some and read and talked to people about the others.
Uncontested Winners by Language
For many languages there is a clear winner that you should just use. Here are the ones I've found and what I think of them.
| Language | Library | Our rating |
|---|---|---|
| C | theft | Good but does not come with a library of data generators. |
| C++ | CppQuickCheck | Unsure |
| Clojure | test.check | Very good |
| Coq | QuickChick | Unsure |
| F# | FsCheck | Very Good |
| Go | gopter | Unsure but looks promising. |
| Haskell | Hedgehog | Comparatively new, but looks solid. See below. |
| Java | QuickTheories | Unsure. Extremely new but looks promising. |
| JavaScript | jsverify | Good |
| PHP | Eris | Unsure. Looks promising. |
| Python | Hypothesis | I may be a bit biased on this one, but it's also unambiguously true. |
| Ruby | Rantly | Unsure. We're not convinced, but the alternatives are definitely worse. |
| Rust | Quickcheck | Unsure, but probably very good based on initial appearance and usage level. |
| Scala | ScalaCheck | Very Good |
| Swift | Swiftcheck | Unsure |
Where when I've said "Unsure" I really just mean that I think it looks good but I haven't put in the depth of in time to be sure, not that I have doubts.
Special case: Haskell
The original QuickCheck was of course written in Haskell, so it may seem odd that it's not the property based testing library I recommend for Haskell!
The reason is that I feel that the design of classic QuickCheck is fundamentally limiting, and that Hedgehog takes it in the direction that the rest of the property-based testing world is moving (and where most of the implementations for dynamic languages, Hypothesis included, already are). In particular its approach starts from generators rather than type classes, and it has integrated shrinking, and a fairly comprehensive library of generators.
Special case: Erlang
Erlang is a special case because they have QuviQ's QuickCheck. Their QuickCheck implementation is by all accounts extremely good, but it is also proprietary and fairly expensive. Nevertheless, if you find yourselves in the right language, circumstance and financial situation to use it, I would strongly recommend doing so.
In particular, QuviQ's QuickCheck is really the only implementation in this article I think is simply better than Hypothesis. Hypothesis is significantly more user friendly, especially if the users in question are less than familiar with Erlang, but there are things QuviQ can do that Hypothesis can't, and the data generation has had a great deal more engineering effort put into it.
If you're using Erlang but not able to pay for QuickCheck, apparently the one to use is PropEr. If you're also unable to use GPLed software there's triq. I know very little about either.
Special case: OCaml
OCaml seems to be suffering from a problem of being close enough to Haskell that people try to do a straight port of Quickcheck but far enough from Haskell that this doesn't work. The result is that there is a "mechanical port" of Quickcheck which is completely abandoned and a fork of it that uses more idiomatic OCaml. I'm insufficiently familiar with OCaml or its community to know if either is used or whether there is another one that is.
What does this have to do with Hypothesis?
In some sense these are all "competitors" to Hypothesis, but we're perfectly happy not to compete.
In the case of Erlang, I wouldn't even try. In the case of Scala, F#, or Clojure, I might at some point work with them to try to bring the best parts of Hypothesis to their existing implementation, but I don't consider them a priority - they're well served by what they have right now, and there are many languages that are not.
For the rest though? I'm glad they exist! I care about testing about about high quality software, and they're doing their part to make it possible.
But I feel they're being badly served by their underlying model, and that they feel quite unnatural to use in the context of a more conventional test setting. I think Hypothesis is the way forward, and I'll be doing my best to make it possible for everyone to use it in their language of choice.