Theories

I’ve just released a slightly updated version of my NUnit extension for data-driven unit testing.

There’s been a lot of discussion on the NUnit developer list recently regarding Theories – something new in JUnit and xUnit.Net, and it’s taken a while to discover why they’re so powerful (they’re superficially very similar to data-driven unit tests, and a lot of the differences are semantics).

First, there’s some good background on theories written by David Saff:
http://shareandenjoy.saff.net/tdd-specifications.pdf
http://shareandenjoy.saff.net/2007/04/popper-and-junitfactory.html
http://dspace.mit.edu/bitstream/1721.1/40090/1/MIT-CSAIL-TR-2008-002.pdf

Theories on first glance look like a data-driven unit test, but I think that the most important difference is, is that:

Theories are, in theory (excuse the pun), supposed to pass for ANY POSSIBLE parameters, whereas data-driven tests only express the behaviour examples that the developer has provided (they are nothing new in unit testing – just a way for a developer to more clearly group parameters together, or get the parameterized data from an external data source without recompiling tests).

Theories are a generalized statement of how the program should run, whereas in TDDing, a very explicit statement of intent is made, which can be made to pass by coding that specific case in the implementation, and then the program is made to work by triangulization – expressing the generalization by giving more inputs. However, the theory literature points out that as we haven’t passed in too many data points we can’t be sure whether we’ve actually expressed what we meant.

Theories, by forcing us to write our tests such that they take any inputs, are much more powerful a statement, and allow for the possible inputs to be explored with external tools.

As an aside, one question I posted to the NUnit developer list regarding theories: “One thing that comes to mind, is that theories are written such that all possible inputs should pass. Apart from using a tool such as agitator, is there a way to test that the tests are written in a general way (I mean, if you had a theory that took parameters, but it totally ignored those parameters and worked as a vanilla unit test – i.e. created its own input), then it’s not really a valid theory – is
there a way to detect these cases? Probably not, but I was just idly wondering.” Answers on a postcard to… well, I’d prefer a reply comment 😉

2 thoughts on “Theories”

  1. Well, a theory is only valid within some explicitly specified domain. So part of your theory should express constraints on that domain, if necessary.

    I know ideas like Assume.That have been bounced around the NUnit list before.

    You might also look at Microsoft Pex. Unfortunately there is no public release available yet, but if you ask Peli he might send you one… 😉

  2. Hi Jeff,

    Thanks for your comment (sorry about the delay, I only noticed it today).

    I hadn’t heard about Theories only being valid within a specific domain – I was understanding that a theory is the basically the same as a unit test, with the difference that it’s written to be a more general statement about the code under test, but not necessarily specific?

    I’d read a bit about Pex before, do you know anything about the licencing?

    Cheers,
    Gary

Comments are closed.