These are a collection of introductory articles that can help you to learn how to use Hypothesis
effectively. You don’t have to read all of these to get started! Feel free to dip in and out as
the mood takes you.
In the movie Die Hard with a Vengeance
(aka Die Hard 3), there is
this famous scene where
John McClane (Bruce Willis) and Zeus Carver (Samuel L. Jackson)
are forced to solve a problem or be blown up: Given a 3 gallon jug and
5 gallon jug, how do you measure out exactly 4 gallons of water?
The encode/decode invariant
is one of the most important properties to know about for testing your code with Hypothesis
or other property-based testing systems, because it captures a very common pattern and is
very good at finding bugs.
But how do you go beyond it? If encoders are that common, surely there must be other things
to test with them?
Eris is a library for property-based testing of PHP code, inspired by the mature frameworks that other languages provide like QuickCheck, Clojure’s test.check and of course Hypothesis.
Here is a side-by-side comparison of some basic and advanced features that have been implemented in both Hypothesis and Eris, which may help developers coming from either Python or PHP and looking at the other side.
Sometimes you want to generate data which is recursive.
That is, in order to draw some data you may need to draw
some more data from the same strategy. For example we might
want to generate a tree structure, or arbitrary JSON.
Hypothesis has the recursive function in the hypothesis.strategies
module to make this easier to do. This is an article about how to
Hypothesis is a library designed to help you write what are called
The key idea of property based testing is that rather than writing a test
that tests just a single scenario, you write tests that describe a range
of scenarios and then let the computer explore the possibilities for you
rather than having to hand-write every one yourself.
In order to contrast this with the sort of tests you might be used to, when
talking about property-based testing we tend to describe the normal sort of
testing as example-based testing.
Property-based testing can be significantly more powerful than example based
testing, because it automates the most time consuming part of writing tests
coming up with the specific examples - and will usually perform it better
than a human would. This allows you to focus on the parts that humans are
better at - understanding the system, its range of acceptable behaviours,
and how they might break.
You don’t need a library to do property-based testing. If you’ve ever
written a test which generates some random data and uses it for testing,
that’s a property-based test. But having a library can help you a lot,
making your tests easier to write, more robust, and better at finding
bugs. In the rest of this article we’ll see how.
Many people are quite comfortable writing ordinary unit tests, but feel a bit
confused when they start with property-based testing. This post shows how two
ordinary programmers started with normal Python unit tests and nudged them
incrementally toward property-based tests, gaining many advantages on the way.
One thing that often causes people problems is figuring out how to generate
the right data to fit their data
model. You can start with just generating strings and integers, but eventually you want
to be able to generate
objects from your domain model. Hypothesis provides a lot of tools to help you build the
data you want, but sometimes the choice can be a bit overwhelming.
Here’s a worked example to walk you through some of the details and help you get to grips with how to use
Hypothesis’s standard testing mechanisms are very good for testing things that can be
considered direct functions of data. But supposed you have some complex stateful
system or object that you want to test. How can you do that?
In this article we’ll see how to use Hypothesis’s rule based state machines to define
tests that generate not just simple data, but entire programs using some stateful
object. These will give the same level of boost to testing the behaviour of the
object as you get to testing the data it accepts.
One of the simplest types of invariant to find once you move past
just fuzzing your code is asserting that two
different operations should produce the same result, and one of the simplest instances of
that is looking for encode/decode pairs. That is, you have some function that takes a
value and encodes it as another value, and another that is supposed to reverse the process.
This is ripe for testing with Hypothesis because it has a natural completely defined
specification: Encoding and then decoding should be exactly the same as doing nothing.