JUnit Theories and Other Beasts, Part 2
8 June 2012, by Rowan Hill
Previously, on ‘and Other Beasts’…
In the first post in this series, we took a quick look at the very basics that JUnit has to offer, followed by ways to alter how your tests are run, using Runner subclasses and TestRule implementations.
In this post, we’ll start to take a look at something JUnit calls theories.
You have a theory, I assume?
Whilst reading around about Parameterized tests (looking for best practices, which seem to be having one test per class, and many classes, in general), I found something a bit more interesting, which is what prompted me to write this series of posts. In particular, I wanted something a bit more like TestNG – a way to easily partition which bits of data are executed by which tests, so the testing of a single object doesn’t get broken up over many (related, but not obviously connected) test classes.
Tucked away in the JUnit 4.4 release notes are the introduction of assumptions and theories. As an executive summary, a theory is a more expressive kind of test. Instead of saying “For this input value, I expect this result,” theories try to get closer to saying “The method performs this kind of operation on its inputs” – for example, instead of saying “methodUnderTest(5) should equal 10,” a theory would be “methodUnderTest takes an int and doubles it.” The gist is that theories allow test code to make more general statements about the code under test. Assumptions work nicely with theories to place restrictions on the input data (and exposing developer intent along the way). Assumptions work in a similar way as JUnit assertions – except they rely heavily on Hamcrest matchers, e.g. assumeThat(testParam, is(not(nullValue()))). If a theory or test expresses an assumption that is violated by the given test data, testing with that data does not continue.
Whilst assumptions are first-class citizens, the theory classes are in an ‘experimental’ package (at least, they still are in JUnit 4.10), but they are bundled with the core jar, and they’ve survived 6 versions, so I don’t think they’re going anywhere soon.
The examples included in the release notes seem to be the entire extent of the official documentation, sadly – the API JavaDocs are completely blank, in particular. As a result, I’ve had a bit of a read around, and I’ll summarise what I’ve learned here.
It’s all academic
As far as I can tell, the idea for theories seems to have started at MIT with David Saff (a JUnit developer), amongst others. A few of his papers on the subject are available, and they’re actually very accessible and a good place to start if you want to understand the motivation behind theories. In particular, I’ve found the following:
- The Practice of Theories: Adding “For-all” Statements to “There-Exists” Tests. This takes a nice easy journey through how a TDD-familiar developer might alter their workflow to use theories, the benefits that can bring, and some best practice tips.
- Theory-infected Or How I Learned to Stop Worrying and Love Universal Quantiﬁcation. This is a slightly more formal (but much shorter) summary.
- Theories in Practice: Easy-to-Write Speciﬁcations that Catch Bugs. This is again slightly more formal in presentation, but is relatively lengthy, and explicitly touches on the modifications they made to JUnit (before theories were folded into JUnit proper, I assume).
Of particular interest are the benefits the researchers think theories have:
- Theories facilitate the development of complete abstract interfaces by allowing the developer to remain at the level of domain abstraction such as numerical systems or currencies.
- Theories enable reuse of the test code, since many traditional speciﬁc tests can be recast as instantiations of a speciﬁc theory.
- Theories afford the use of automated testing tools to amplify the effects of a theory by testing a theory with multiple data points.
The researchers also explain when they think theories are most helpful – when there are behaviours that can be easily and precisely described, but cannot be captured as a simple example (with a normal test), because they pertain to a large, or inﬁnite, set of concrete executions. For example:
- Converting from a decimal representation of any integer to Roman Numerals and back should preserve the integer’s value.
- Persisting and restoring any object of class A should always yield an object equal to the original.
- Adding a dollar to any currency collection should always produce a new collection not equal to the ﬁrst.
- If any two Java objects are equal according to the equals(Object) method, then calling the hashCode() method on each of the two objects must produce the same integer result.
One point the authors repeatedly mention is that theories lend themselves to testing that two inverse operations cancel each other out. For example, instead of testing multiplication, they advocate using theories to test that multiplication followed by division gives the same number you started with, and backing this up with the traditional @Tests to check that a small number of human-verified input/output combinations behave as expected for each individual operation. This allows you to easily provide a large number of data points without having to manage the relationships between the inputs and expected results – they should be equal.
Now we’ve covered the intent and background of theories, in our next post we’ll take a look at how to actually write them. See you then…