Hypothesis

This is a release announcement for the 3.6.0 release of Hypothesis for Python. It’s a bit of an emergency release.

Hypothesis 3.5.0 inadvertently added a dependency on GPLed code (see below for how this happened) which this release removes. This means that if you are running Hypothesis 3.5.x then there is a good chance you are in violation of the GPL and you should update immediately.

Apologies for any inconvenience this may have caused.

Probably the number one complaint I hear from Hypothesis users is that it “doesn’t work” with py.test fixtures. This isn’t true, but it does have one very specific limitation in how it works that annoys people: It only runs function scoped fixtures once for the entire test, not once per example. Because of the way people use function scoped fixtures for handling stateful things like databases, this often causes people problems.

I’ve been maintaining for a while that this is impossible to fix without some changes on the pytest end.

The good news is that this turns out not to be the case. After some conversations with pytest developers, some examining of other pytest plugins, and a bunch of prototyping, I’m pretty sure it’s possible. It’s just really annoying and a lot of work.

So that’s the good news. The bad news is that this isn’t going to happen without someone funding the work.

I’ve now spent about a week of fairly solid work on this, and what I’ve got is quite promising: The core objective of running pytest fixtures for every examples works fairly seamlessly.

But it’s now in the long tail of problems that will need to be squashed before this counts as an actual production ready releasable piece of work. A number of things don’t work. For example, currently it’s running some module scoped fixtures once per example too, which it clearly shouldn’t be doing. It also currently has some pretty major performance problems that are bad enough that I would consider them release blocking.

As a result I’d estimate there’s easily another 2-3 weeks of work needed to get this out the door.

Which brings us to the crux of the matter: 2-3 additional weeks of free work on top of the one I’ve already done is 3-4 weeks more free work than I particularly want to do on this feature, so without sponsorship it’s not getting finished.

I typically charge £400/day for work on Hypothesis (this is heavily discounted off my normal rates), so 2-3 weeks comes to £4000 to £6000 (roughly $5000 to $8000) that has to come from somewhere.

I know there are a number of companies out there using pytest and Hypothesis together. I know from the amount of complaining about this integration that this is a real problem you’re experiencing. So, I think this money should come from those companies. Besides helping to support a tool you’ve already got a lot of value out of, this will expand the scope of what you can easily test with Hypothesis a lot, and will be hugely beneficial to your bug finding efforts.

This is a model that has worked well before with the funding of the recent statistics work by Jean-Louis Fuchs and Adfinis-SyGroup, and I’m confident it can work well again.

If you work at such a company and would like to talk about funding some or part of this development, please email me at [email protected].

This is a combined release announcement for two releases. 3.5.0 was released yesterday, and 3.5.1 has been released today after some early bug reports in 3.5.0

Changes

3.5.0 - 2016-09-22

This is a feature release.

• fractions() and decimals() strategies now support min_value and max_value parameters. Thanks go to Anne Mulhern for the development of this feature.
• The Hypothesis pytest plugin now supports a –hypothesis-show-statistics parameter that gives detailed statistics about the tests that were run. Huge thanks to Jean-Louis Fuchs and Adfinis-SyGroup for funding the development of this feature.
• There is a new event() function that can be used to add custom statistics.

Additionally there have been some minor bug fixes:

• In some cases Hypothesis should produce fewer duplicate examples (this will mostly only affect cases with a single parameter).
• py.test command line parameters are now under an option group for Hypothesis (thanks to David Keijser for fixing this)
• Hypothesis would previously error if you used function annotations on your tests under Python 3.4.
• The repr of many strategies using lambdas has been improved to include the lambda body (this was previously supported in many but not all cases).

3.5.1 - 2016-09-23

This is a bug fix release.

• Hypothesis now runs cleanly in -B and -BB modes, avoiding mixing bytes and unicode.
• unittest.TestCase tests would not have shown up in the new statistics mode. Now they do.
• Similarly, stateful tests would not have shown up in statistics and now they do.
• Statistics now print with pytest node IDs (the names you’d get in pytest verbose mode).

Notes

Aside from the above changes, there are a couple big things behind the scenes of this release that make it a big deal.

The first is that the flagship chunk of work, statistics, is a long-standing want to have that has never quite been prioritised. By funding it, Jean-Louis and Adfinis-SyGroup successfully bumped it up to the top of the priority list, making it the first funded feature in Hypothesis for Python!

Another less significant but still important is that this release marks the first real break with an unofficial Hypothesis for Python policy of not having any dependencies other than the standard library and backports. This release adds a dependency on the uncompyle6 package. This may seem like an odd choice, but it was invaluable for fixing the repr behaviour, which in turn was really needed for providing good statistics for filter and recursive strategies.

This is a bug fix release, fixing a number of problems with the settings system:

• Test functions defined using @given can now be called from other threads (Issue #337)
• Attempting to delete a settings property would previously have silently done the wrong thing. Now it raises an AttributeError.
• Creating a settings object with a custom database_file parameter was silently getting ignored and the default was being used instead. Now it’s not.

Notes

For historic reasons, _settings.py had been excluded from the requirement to have 100% branch coverage. Issue #337 would have been caught by a coverage requirement: the code in question simply couldn’t have worked, but it was not covered by any tests, so it slipped through.

As part of the general principle that bugs shouldn’t just be fixed without addressing the reason why the bug slipped through in the first place, I decided to impose the coverage requirements on _settings.py as well, which is how the other two bugs were found. Both of these had code that was never run during tests - in the case of the deletion bug there was a __delete__ descriptor method that was never being run, and in the case of the database_file one there was a check later that could never fire because the internal _database field was always being set in __init__.

I feel like this experiment thoroughly validated that 100% coverage is a useful thing to aim for. Unfortunately it also pointed out that the settings system is much more complicated than it needs to be. I’m unsure what to do about that - some of its functionality is a bit too baked into the public API to lightly change, and I’m don’t think it’s worth breaking that just to simplify the code.

This is a bug fix release for a single bug:

• On Windows when running two Hypothesis processes in parallel (e.g. using pytest-xdist) they could race with each other and one would raise an exception due to the non-atomic nature of file renaming on Windows and the fact that you can’t rename over an existing file. This is now fixed.

Notes

My tendency of doing immediate patch releases for bugs is unusual but generally seems to be appreciated. In this case this was a bug that was blocking a py.test merge.

I suspect this is not the last bug around atomic file creation on Windows. Cross platform atomic file creation seems to be a harder problem than I would have expected.

I would like to see more posters on the hypothesis.works blog. I’m particularly interested in experience reports from people who use Hypothesis in the wild. Could that be you?

I get asked this a lot, and I write property based testing tools for a living, so you’d think I have a good answer to this, but historically I haven’t. This is my attempt to fix that.

Historically the definition of property based testing has been “The thing that QuickCheck does”. As a working definition this has served pretty well, but the problem is that it makes us unable to distinguish what the essential features of property-based testing are and what are just accidental features that appeared in the implementations that we’re used to.

As the author of a property based testing system which diverges quite a lot from QuickCheck, this troubles me more than it would most people, so I thought I’d set out some of my thoughts on what property based testing is and isn’t.

This isn’t intended to be definitive, and it will evolve over time as my thoughts do, but it should provide a useful grounding for further discussion.

It’s a common belief that in order for property based testing to be useful, your code must be referentially transparent. That is, it must be a pure function with no side effects that just takes input data and produces output data and is solely defined by what input data produces what output data.

This is, bluntly, complete and utter nonsense with no basis in reality.

What is Hypothesis for?

From the perspective of a user, the purpose of Hypothesis is to make it easier for you to write better tests.

From my perspective as the primary author, that is of course also a purpose of Hypothesis. I write a lot of code, it needs testing, and the idea of trying to do that without Hypothesis has become nearly unthinkable.

But, on a large scale, the true purpose of Hypothesis is to drag the world kicking and screaming into a new and terrifying age of high quality software.

You have probably never written a significant piece of correct software.

That’s not a value judgement. It’s certainly not a criticism of your competence. I can say with almost complete confidence that every non-trivial piece of software I have written contains at least one bug. You might have written small libraries that are essentially bug free, but the chance that you have written a non-trivial bug free program is tantamount to zero.

I don’t even mean this in some pedantic academic sense. I’m talking about behaviour where if someone spotted it and pointed it out to you you would probably admit that it’s a bug. It might even be a bug that you cared about.

Why is this?