This is a blog post to explain exactly what exactly parameterized data-driven unit testing is and why it’s there, and then to explain a bit about how this effected the implementation of the Rakija NUnit add-in.
First, what is data-driven unit testing? It’s running tests with data which is driven from external data (as may be from an xml file or database). NUnit makes this quite difficult, as the tests are attribute driven, they are fixed, and fixed at the time that the tests are created.
Data-driven fixtures.
A common desire is to run fixtures with different set of parameters. An example of this is to test key input filtering under different UI cultures.
This is a very contrived example, and the tests are in no way realistic, but it demonstrates what data-driven tests are about. My example is a simple class that takes various string inputs and performs operations on them. Here it is:
///
/// Parses and operates on various string values.
///
public static class ValueParser
{
///
/// Returns the addition of two strings representing
/// double values.
///
/// The combined value.
public static double GetAdditiveValue
(string value1, string value2)
{
return double.Parse(value1) + double.Parse(value2);
}
}
And of course, the contributing unit test:
[TestFixture]
public class TestValueParser
{
[Test]
public void TestAdditiveValue()
{
Assert.AreEqual(4.5, ValueParser.GetAdditiveValue("2.3", "2.2"));
}
} Of course in real life, we’d have tests to pass in garbage input, checking for any exceptions etc. Our class would also be much larger, containing many methods, along with many (maybe hundreds) of corresponding unit tests in our test fixture. We’ve happily deployed the first release of our software, and have many happy customers, when our customer representative or product manager informs us that we need our software to run on European users computers. We don’t get panicked, we know that our input data is coming from XML, and is formatted in invariant culture, but we need to make sure to change our code such that double.Parse takes in CultureInfo.InvariantCulture. We’d ideally like to ensure that our tests run in both cultures. We could duplicate our fixture with a different setup test, but we all know that duplicate code is bad, right? So we use the following pattern.
// NOTE: TestFixture class has been removed.
public class TestValueParser
{
protected string cultureString = "invalid";
private CultureInfo initialCulture;
[SetUp]
public void Setup()
{
this.initialCulture = Thread.CurrentThread.CurrentCulture;
Thread.CurrentThread.CurrentCulture = CultureInfo.GetCultureInfoByIetfLanguageTag(
cultureString);
}
[TearDown]
public void TearDown()
{
Thread.CurrentThread.CurrentCulture = this.initialCulture;
}
[Test]
public void TestAdditiveValue()
{
Assert.AreEqual(4.5, ValueParser.GetAdditiveValue("2.3", "2.2"));
}
}
[TestFixture]
public class TestValueParserUSCulture : TestValueParser
{
public TestValueParserUSCulture()
{
cultureString = "en-US";
}
}
[TestFixture]
public class TestValueParserFRCulture : TestValueParser
{
public TestValueParserFRCulture()
{
cultureString = "fr-FR";
}
} We leave all of our tests intact, we have to make some minor modifications to our original test fixture (remove the test fixture attribute, and do some setup work), but we’re able to test our fix very quickly (obviously we follow red bar/green bar/refactor, and write the tests before changing our code), and as we have a high test code the new release of the software works flawlessly, we finish early and spend the remaining time playing Crysis.
This pattern is very common in NUnit, but I haven’t found it named anywhere, I call it the Test Fixture Inheritance Pattern. NOTE: This pattern is not supported in Visual Studio Team System 2005 testing, but it’s fixed in the 2008 version.
(As an aside, I’ve read about the upcoming xUnit framework, and it says that tests aren’t grouped into fixtures, I’m not sure how it would cope with this pattern in that case? It’s possible that you could parameterized each test, and use something like a RowTest attribute to pass in the culture string, but by testing a new culture you’d have to add a new attribute to each and every test.)
The first thing to notice about the Test Fixture Inheritance Pattern is that it isn’t very OO. It wouldn’t be too much work though to create an NUnit extension that would allow parameterizing the test fixture constructor, in a similar way to how the RowTest attribute works, i.e.
[TestFixture]
[Parameter("en-US")]
[Parameter("fr-FR")]
public class TestValueParser
{
protected string cultureString = "invalid";
private CultureInfo initialCulture;
public TestValueParser(string cultureString)
{
this.cultureString = cultureString;
}
[SetUp]
public void Setup()
{
this.initialCulture = Thread.CurrentThread.CurrentCulture;
Thread.CurrentThread.CurrentCulture =
CultureInfo.GetCultureInfoByIetfLanguageTag(
cultureString);
}
[TearDown]
public void TearDown()
{
Thread.CurrentThread.CurrentCulture = this.initialCulture;
}
[Test]
public void TestAdditiveValue()
{
Assert.AreEqual(4.5, ValueParser.GetAdditiveValue(
"2.3", "2.2"));
}
} That would be nothing more than syntactic sugar though (it does make the tests look much neater). This could be more powerful though – it could be expanded to read data from a database or an XML file (again, similar to VSTS’s DataSource attribute).
Rakija instead provides an interface IDynamicFixtureSpecifier, which is used on a class to specify which fixtures to create. Each fixture then must implement another interface, and the fixture need to implement IDynamicFixture. Our example now looks like this:
// NOTE: TestFixture attribute has been removed.
public class TestValueParser : IDynamicFixture
{
protected string cultureString = "invalid";
private CultureInfo initialCulture;
public TestValueParser(string cultureString)
{
this.cultureString = cultureString;
}
[SetUp]
public void Setup()
{
this.initialCulture = Thread.CurrentThread.CurrentCulture;
Thread.CurrentThread.CurrentCulture =
CultureInfo.GetCultureInfoByIetfLanguageTag(
cultureString);
}
[TearDown]
public void TearDown()
{
Thread.CurrentThread.CurrentCulture = this.initialCulture;
}
[Test]
public void TestAdditiveValue()
{
Assert.AreEqual(4.5, ValueParser.GetAdditiveValue("2.3", "2.2"));
}
#region IDynamicFixture Members
public string Name
{
get { return "TestValueParser " + cultureString; }
}
#endregion
} The fact that this is a test fixture is detected from the add-in by the fact that it implements IDynamicFixture, I could have instead created a new attribute for the class, and either reflected for the Name property (or created an attribute to return the name), but that’s by-the-by.
To actually specify the instances of our fixture, we use code like the following:
public class SpecifyFixtures : IDynamicFixtureSpecifier
{
#region IDynamicFixtureSpecifier Members
public Type FixtureType
{
get
{
return typeof(TestValueParser);
}
}
public IList GetUserFixtures()
{
List fixtures = new List(2);
fixtures.Add(new TestValueParser("fr-FR"));
fixtures.Add(new TestValueParser("en-US"));
return fixtures;
}
#endregion
} Your first thought may be to wonder why I’ve created such a convoluted way of creating fixtures, when a page or so ago I showed how neatly this could be done by creating a new attribute. Well, this is much more powerful, we’re not limited to querying a database or reading an XML file – we could load up our assembly to test, and reflect over certain types in it, creating a new fixture for each type. The sky’s the limit.
Parameterized test methods.
I’ll quickly go over the motivation over parameterized test methods, before discussing why I’ve been forced to implement these myself due to the current NUnit (as of 2.4.6) extension interface, rather than deferring these to Kelly Anderson’s extension.
Another common requirement when unit testing is to run a given test with different input data. Before the existence of any RowTest attribute, a common thing to do may have been.
[Test]
public void SomeTest()
{
// .. read some data in an xml file
// .. for each piece of data, perform some common testing
// .. { assert on the data; }
} This obviously works, but the problem of it is lack of visibility/feedback. For a file that may contain hundreds of pieces of data, we have a single red or green bar. We’d like to have visibility of each individual case. VSTS has a DataSource attribute to cope with this cases like this (reading from a database or XML File).
Now that I’ve, very briefly, covered the motivations behind parameterized test cases, I’m going to go into the implementation details of Rakija.
Kelly Anderson has created a more powerful interface, similar to Rakija’s, where a method can be decorated with an IterativeTest attribute to specify the test data for the test. (Kelly’s interface is much more user-friendly than Rakija’s).
Unfortunately, I’ve had to duplicate the functionality of this Kelly’s work in Rakija, due to the internals of NUnit’s extensibility mechanism. I’ll explain why below, and propose a change to the mechanism to allow the extensions to co-exist.
Similar to Rakija, Kelly’s mechanism creates a TestCaseBuilder to recognise the parameterized test cases. This works without any problem, but will fail when used with any of the above parameterized test fixtures described above.
Kelly’s extension works by finding the method decorated with his IterativeTestAttribute, creating a new instance of the type that contains that attribute, and calling the method decorated by that attribute to find the parameters to create the test with. He gets away with creating a new instance of the type by the fact that all instances of the type will contain the same data.
The problem comes when you have parameterized test cases – each test case has been created with different parameters, and depending on those constructor parameters we may wish to pass different parameters to our tests (our test fixtures may have been created with a given node of an xml file, and we may wish to pass various sub-nodes of that node to an XML file).
Rakija is forced to deal with this situation by taking control of the parameterized test case situation when parameterized fixtures are also involved.
The proposed fix, to allow dynamic fixtures to exist with any other extension (e.g. Kelly Anderson’s IterativeTest extension, or Andreas Schlapsi’s RowTest extension) is to make a change to the extensibility mechanism.
Currently, NUnitTestFixtureBuilder (or our derived class), has to call
this.testCaseBuilders.BuildFrom(method);
where method is a MethodInfo
And this forces each test case builder either call a static method, or construct a type to create a method on. The return type from this method is a test, which gets added onto the fixture/suite’s list of tests to run.
In the parameterized fixture case, the interface on AbstractTestCaseBuilder would better be:
protected override TestCase MakeTestCase(object instance, MethodInfo method);
or something similar, where instance would be the instance that contains the method. That would allow the method to be invoked on the instance that it lived on, allowing parameterized fixtures to co-exist with the other mentioned extensions.