TL;DR: as of version 6.136.9, Hypothesis supports running the same test simultaneously from multiple threads.
Hypothesis has historically had the following thread-safety policy:
- Running tests in multiple processes: fully supported.
- Running separate tests in multiple threads: not officially supported, but mostly worked.
- Running the same test in …
If you watch the Hypothesis changelog, you'll notice the rate of releases sped up dramatically in 2017. We released over a hundred different versions, sometimes multiple times a day.
This is all thanks to our continuous release process. We've completely automated the process of releasing, so every pull request that …
Happy new year everybody!
In this post I'd like to tell you about one of the nice things that happened in 2017: The Hypothesis work that was funded by Smarkets. Smarkets are an exchange for peer-to-peer trading of bets but, more importantly for us, they are fairly heavy users of …
In my last post I mentioned the problem of bug slippage: When you start with one bug, reduce the test case, and end up with another bug.
I've run into another related problem twice now, and it's not one I've seen talked about previously.
The problem is this: Sometimes shrinking …
When Hypothesis finds an example triggering a bug, it tries to shrink the example down to something simpler that triggers it. This is a pretty common feature, and most property-based testing libraries implement something similar (though there are a number of differences between them). Stand-alone test case reducers are also …
If you look at the original property-based testing library, the Haskell version of QuickCheck, tests are very closely tied to types: The way you typically specify a property is by inferring the data that needs to be generated from the types the test function expects for its arguments.
This is …
This post was originally published on the author's personal site. It is reproduced here with his permission.
In the movie Die Hard with a Vengeance (aka Die Hard 3), there is this famous scene where John McClane (Bruce Willis) and Zeus Carver (Samuel L. Jackson) are forced to solve a …
I'm in the process of trying to turn my work on Hypothesis into a PhD and I realised that I don't have a good self-contained summary as to why researchers should care about it.
So this is that piece. I'll try to give a from scratch introduction to the why …
Hypothesis has a very different underlying implementation to any other property-based testing system. As far as I know, it's an entirely novel design that I invented.
Central to this design is the following feature set which every Hypothesis strategy supports automatically (the only way to break this is by having …
In my last article about shrinking, I discussed the problems with basing shrinking on the type of the values to be shrunk.
In writing it though I forgot that there was a halfway house which is also somewhat bad (but significantly less so) that you see in a couple of …
One of the big differences between Hypothesis and Haskell QuickCheck is how shrinking is handled.
Specifically, the way shrinking is handled in Haskell QuickCheck is bad and the way it works in Hypothesis (and also in test.check and EQC) is good. If you're implementing a property based testing system …
This is a release announcement for the 3.6.0 release of Hypothesis for Python. It's a bit of an emergency release.
Hypothesis 3.5.0 inadvertently added a dependency on GPLed code (see below for how this happened) which this release removes. This means that if you are running …
The encode/decode invariant is one of the most important properties to know about for testing your code with Hypothesis or other property-based testing systems, because it captures a very common pattern and is very good at finding bugs.
But how do you go beyond it? If encoders are that …
Probably the number one complaint I hear from Hypothesis users is that it "doesn't work" with py.test fixtures. This isn't true, but it does have one very specific limitation in how it works that annoys people: It only runs function scoped fixtures once for the entire test, not once …
This is a combined release announcement for two releases. 3.5.0 was released yesterday, and 3.5.1 has been released today after some early bug reports in 3.5.0
Changes
3.5.0 - 2016-09-22
This is a feature release.
- fractions() and decimals() strategies now support min_value and …
Eris is a library for property-based testing of PHP code, inspired by the mature frameworks that other languages provide like QuickCheck, Clojure's test.check and of course Hypothesis.
Here is a side-by-side comparison of some basic and advanced features that have been implemented in both Hypothesis and Eris, which may …
This is one of the most common first questions about Hypothesis.
People generally assume that the number of tests run will depend on the specific strategies used, but that's generally not the case. Instead Hypothesis has a fairly fixed set of heuristics to determine how many times to run, which …
Sometimes you want to generate data which is recursive. That is, in order to draw some data you may need to draw some more data from the same strategy. For example we might want to generate a tree structure, or arbitrary JSON.
Hypothesis has the recursive function in the hypothesis …
pytest is a great test runner, and is the one Hypothesis itself uses for testing (though Hypothesis works fine with other test runners too).
It has a fairly elaborate fixture system, and people are often unsure how that interacts with Hypothesis. In this article we'll go over the details of …
Hypothesis is a library designed to help you write what are called property-based tests.
The key idea of property based testing is that rather than writing a test that tests just a single scenario, you write tests that describe a range of scenarios and then let the computer explore the …
This is a bug fix release, fixing a number of problems with the settings system:
- Test functions defined using @given can now be called from other threads (Issue #337)
- Attempting to delete a settings property would previously have silently done the wrong thing. Now it raises an AttributeError.
- Creating a …
This is a bug fix release for a single bug:
- On Windows when running two Hypothesis processes in parallel (e.g. using pytest-xdist) they could race with each other and one would raise an exception due to the non-atomic nature of file renaming on Windows and the fact that you …
Consider the following problem:
You have a list of floating point numbers. No nasty tricks - these aren't NaN or Infinity, just normal "simple" floating point numbers.
Now: Calculate the mean (average). Can you do it?
It turns out this is a hard problem. It's hard to get it even close …
Sometimes you're lucky enough to have problems where the result is completely specified by a few simple properties.
This doesn't necessarily correspond to them being easy! Many such problems are actually extremely fiddly to implement.
It does mean that they're easy to test though. Lets see how.
Lets look at …
A lot of applications end up growing a complex configuration system, with a large number of different knobs and dials you can turn to change behaviour. Some of these are just for performance tuning, some change operational concerns, some have other functions.
Testing these is tricky. As the number of …
Many people are quite comfortable writing ordinary unit tests, but feel a bit confused when they start with property-based testing. This post shows how two ordinary programmers started with normal Python unit tests and nudged them incrementally toward property-based tests, gaining many advantages on the way.
Background
I used to …
I would like to see more posters on the hypothesis.works blog. I'm particularly interested in experience reports from people who use Hypothesis in the wild. Could that be you?
Details of how to guest post on here:
- This site is a Jekyll site on GitHub. To add a post …
We've previously looked into testing performance optimizations using Hypothesis, but this article is about something quite different: It's about testing code that is designed to optimize a value. That is, you have some function and you want to find arguments to it that maximize (or minimize) its value.
As well …
Hypothesis is, of course, a library for writing tests.
But from an implementation point of view this is hardly noticeable. Really it's a library for constructing and exploring data and using it to prove or disprove hypotheses about it. It then has a small testing library built on top of …
I get asked this a lot, and I write property based testing tools for a living, so you'd think I have a good answer to this, but historically I haven't. This is my attempt to fix that.
Historically the definition of property based testing has been "The thing that QuickCheck …
One thing that often causes people problems is figuring out how to generate the right data to fit their data model. You can start with just generating strings and integers, but eventually you want to be able to generate objects from your domain model. Hypothesis provides a lot of tools …
It's a common belief that in order for property based testing to be useful, your code must be referentially transparent. That is, it must be a pure function with no side effects that just takes input data and produces output data and is solely defined by what input data produces …
Once you've flushed out the basic crashing bugs in your code, you're going to want to look for more interesting things to test.
The next easiest thing to test is code where you know what the right answer is for every input.
Obviously in theory you think you know what …
Hypothesis's standard testing mechanisms are very good for testing things that can be considered direct functions of data. But supposed you have some complex stateful system or object that you want to test. How can you do that?
In this article we'll see how to use Hypothesis's rule based state …
There are a lot of ports of QuickCheck, the original property based testing library, to a variety of different languages.
Some of them are good. Some of them are very good. Some of them are OK. Many are not.
I thought it would be worth keeping track of which are …
What is Hypothesis for?
From the perspective of a user, the purpose of Hypothesis is to make it easier for you to write better tests.
From my perspective as the primary author, that is of course also a purpose of Hypothesis. I write a lot of code, it needs testing …
What happens when you run a test using Hypothesis? This article will help you understand.
The Python version of Hypothesis uses decorators to transform a test function that doesn't use Hypothesis into one that does.
Consider the following example using py.test style testing:
from hypothesis import given
from hypothesis …One of the simplest types of invariant to find once you move past just fuzzing your code is asserting that two different operations should produce the same result, and one of the simplest instances of that is looking for encode/decode pairs. That is, you have some function that takes …
You have probably never written a significant piece of correct software.
That's not a value judgement. It's certainly not a criticism of your competence. I can say with almost complete confidence that every non-trivial piece of software I have written contains at least one bug. You might have written small …
Hypothesis will speed up your testing process and improve your software quality, but when first starting out people often struggle to figure out exactly how to use it.
Until you're used to thinking in this style of testing, it's not always obvious what the invariants of your code actually are …