Another proposed NUnit feature.

This post will clarify the proposed NUnit feature mentioned in the last post, and also propose a new feature (right near the end).

I was thinking of another concrete example of where adding a new feature to NUnit, to allow a more flexible method to create TestFixtures (proposed via a TestFixtureProvider attribute), as mentioned in my last post, may be useful. First, a bit of background on how NUnit was being used:

In the past, I worked on the mapping of financial instruments from an-memory representation of a complex hierarchy of objects into a multi-line string for consumption by another product.

The test fixtures were broadly organised into product types (e.g. bond), with tests for the different conditions encountered by that product (e.g.buy/sell, expired, amortising, etc.)

It was chosen that the tests should be XML driven:

To act as documentation: as the mappings were complex it would provide concrete human-readable examples of the instrument states, and associated mappings.
Tests are more manageable – can generate XML automatically, to give creater coverate quicker.
Allows financial analysts to create additional tests – don’t have to know programming.

The original instrument was represented by COM objects, with the C# mapping code consuming these objects via interop. As COM is interface-driven, it was fairly trivial to create mock objects implementing these interfaces, for testing.

As the instrument was hierarchical, an XML representation was a natural fit. There were XML files for each test fixture, each containing nodes which contained the XML representation, and either the expected string output from the mapping or any expected exception.

e.g.

InstrumentTypeATests.xml


<TestFixture name="InstrumentATests">
  <Test name="TestACertainConfiguration">
    <Instrument>
      This is an XML representation of an instrument.
    </Instrument>
    <ExpectedMapping>
      An expected mapping string in a CData section.
    </ExpectedMapping>
  </Test>
</TestFixture>

This is where we’re getting to the point of this post. We want to run these XML tests in NUnit – there are a number of ways to do this.

One is to have a fixture, with a RunXmlTest method, that would load the XML file (with the filename either hardcoded into the file, or into the fixture constructor – remember we’ve currently got no way to specify constructor arguments, other than via a parameter as passed into the commandline?), then would iterate through the nodes, populating the mock objects, map these mock objects into their string representation, and check that mapping against the expected mapping as specified in the tests. Which is fine, except that we’ll have a single green bar per fixture/test xml file, which isn’t very useful when there are hundreds of test cases.

(forgive any poor syntax/noddy-errors – I can’t remember exactly what this looked like, but this should be enough for you to get the idea)


[TestFixture]
InstrumentATestFixtureOriginal : BaseXmlFixture
{
  [FixtureSetUp]
  public void FixtureSetup()
  {
    // Set the path as used by the base class.
    pathToXml = "InstrumentTypeATests.xml";

    // Base class method to load the XML.
    LoadXml(); // loads into xmlDocument member
  }

  [Test]
  public void RunXmlTest()
  {
    foreach( XmlNode node in xmlDocument )
    {
      RunTest(node.Name);
    }
  }
}

An improvement is to have a TestACertainConfiguration test corresponding with each Test node in the XML – so we get a green bar for each of the tests:


[TestFixture]
InstrumentATestFixtureNew : BaseXmlFixture
{
  [FixtureSetUp]
  public void FixtureSetup()
  {
    // Set the path as used by the base class.
    pathToXml = "InstrumentTypeATests.xml";

    // Base class method to load the XML.
    LoadXml(); // loads into xmlDocument member
  }

  [Test]
  public void RunXmlTest1()
  {
    RunTest("Test1");
  }

  [Test]
  public void RunXmlTest2()
  {
    RunTest("Test2");
  }
}

But only tests as specified in the C# NUnit test are executed.

The solution that I used for the above, as all of the functionality is in the base
classes, is to generate the classes InstrumentATestFixtureNew from using XSLT on the XML files to produce .cs files which are compiled into the build.

This worked nicely, it allowed any changes in the XML files to be taken into account in the tests, but still required a rebuild.

So, the desired behaviour is to allow new test files to be added into a folder somewhere, and the tests to be modified, and have test fixtures generated according to the data in the xml files, without a rebuild of the test project. This is where the new proposed NUnit features would be useful.

Method 1, using the proposed TestFixtureProvider, is to have a constructor on InstrumentATestFixtureOriginal taking the path to the XML file to load, and registering instances of this class with the NUnit framework (see last post for more details on the syntax). This would work OK, but would mean that all of the tests contained in the fixture would only register one red/green bar in the GUI.

Method 2 is a variation on method 1, in that we’d modify InstrumentATestFixtureOriginal to be constructed with the path to the XML, and also the name of the test node – i.e. we’d create a fixture for each test. This would work, but would look cluttered on the GUI. It may be that we could specify in the TestFixtureProviderAttribute how to group these fixtures on the GUI. But this still feels a bit hack-y.

Method 3 is to create the fixtures in memory dependant on the XML, using either CodeDOM or Reflection.Emit. We’d still need to use the proposed TestFixtureProvider property to get these test fixtures to be registered, but it would be the neatest solution. However, it would be a lot of work.

Method 4 is where we get to the second proposed new NUnit functionality. It’s basically to -somehow- implement method 3 but without having to generate the fixtures in code. What we’d like is for every time the base class’s RunTest method is called, it appears on the UI. Maybe a new method called “RegisterTest(string name)” could be added to NUnit to be called from RunTest.

Unit testing frameworks not OO / Suggested new NUnit features.

I was looking at the arguments into one assertion per test (not wanting to open a can of worms here), and how the proposed solution shows how testing frameworks lead to non-OO code. In the original blog entry to be found at:

http://www.artima.com/weblogs/viewpost.jsp?thread=35578
http://blog.daveastels.com/articles/category/programming/page/4

Converting the example provided to C#/NUnit (rather than the original Java/JUnit), the test-driven developed Address being tested is:


public class Address
{
 private string addr1;
 private string cityStatePostalCd;
 private string country;

 public Address(string address)
 {
  Parse(address);
 }

 private void Parse(string address)
 {
  string[] tokens = address.Split('$');
  addr1 = tokens[0];
  cityStatePostalCd = tokens[1];
  country = tokens.Length > 2 ? tokens[2] : "";
 }

 public string Addr1
 {
  get { return addr1; }
 }

 public string CityStatePostalCd
 {
  get { return cityStatePostalCd; }
 }

 public string Country
 {
  get { return country; }
 }
}

And the original non-one assertion per test fixture is:


[TestFixture]
public class OriginalFixture
{
 [Test]
 public void TestAddress1()
 {
  Address a = new Address("ADDR1$CITY IL 60563$COUNTRY");
  Assert.AreEqual(a.Addr1, "ADDR1");
  Assert.AreEqual(a.CityStatePostalCd, "CITY IL 60563");
  Assert.AreEqual(a.Country, "COUNTRY");
 }
}

The suggested solution to this is to have new fixtures for each of the assertions:


[TestFixture]
public class Addr1CspTests
{
 private Address anAddress;

 [SetUp]
 public void setUp()
 {
  anAddress = new Address("ADDR1$CITY IL 60563");
 }

 [Test]
 public void TestAddr1()
 {
  Assert.AreEqual("ADDR1", anAddress.Addr1);
 }

 [Test]
 public void TestCsp()
 {
  Assert.AreEqual("CITY IL 60563", anAddress.CityStatePostalCd);
 }
}

[TestFixture]
public class Addr1CspCountryTest
{
 private Address anAddress;

 [SetUp]
 public void setUp()
 {
  anAddress = new Address("ADDR1$CITY IL 60563$COUNTRY");
 }

 [Test]
 public void TestAddr1()
 {
  Assert.AreEqual("ADDR1", anAddress.Addr1);
 }

 [Test]
 public void TestCsp()
 {
  Assert.AreEqual("CITY IL 60563", anAddress.CityStatePostalCd);
 }

 [TestAttribute]
 public void TestCountry()
 {
  Assert.AreEqual("COUNTRY", anAddress.Country);
 }
}

The author suggest as the Addr and Csp tests are similar in both cases, then they could be refactored into a base class.

However, this got me thinking about the test code I’ve written/worked with – it’s usually quite script like and not very OO, and this example shows why – even though it leads to the definition of more classes, it’s not really OO – each of the classes corresponds with an instance that is instantiated by the test framework. And each of the instances doesn’t differ in their operation, but only in the data used in the construction and the test methods.

It would be tidier if we could instantiate these test fixtures with the data required, and allow NUnit to add them to the list of fixtures and maintain them. I think MBUnit lets you somehow define what data the tests will use via XML, which may do the trick.

So what I’m thinking is is that -somewhere-, I’m not sure where would be best, there would be a method decorated with a TestFixtureProviderAttribute, and this would return a set of TestFixtures when called – so no longer would the TestFixtures only be created using their default constructors.

In our case, the above example would be replaced by something like (excuse any syntax errors):


[TestFixture]
public class AddrTest
{
 private Address anAddress;

 public AddrTest(string address, string expectedAddr1, string expectedCsp, string expectedCounty)
 {
  ...// assign to members
 }

 [SetUp]
 public void setUp()
 {
  anAddress = new Address(address);
 }

 [Test]
 public void TestAddr1()
 {
  Assert.AreEqual(expectedAddress, anAddress.Addr1);
 
}
 [Test]
 public void TestCsp()
 {
  Assert.AreEqual(expectedCsp,anAddress.CityStatePostalCd);
 }

 [TestAttribute]
 public void TestCountry()
 {
  Assert.AreEqual(expectedCountry,anAddress.Country);
 }
}

And this would be created by:


 [TestFixtureProvider]
 ArrayList GetTestFixtures()
 {
  ArrayList fixtures = new ArrayList();
  fixtures.Add(new AddrTest("ADDR1$CITY IL 60563", "ADDR1",
    "CITY IL 60563", "");
  fixtures.Add(new AddrTest("ADDR1$CITY IL 60563$COUNTRY", "ADDR1",
    "CITY IL 60563", "COUNTRY");
  return fixtures;
 }

This would be more OO – the data could be easily populated from an XML file, or created manually, and more importantly, to add a test case wouldn’t require creating a whole new Class definition. This would stop the profileration of classes problem caused by test fixtures leading to non-OO code. If anyone’s reading this, and cares, I can try modifying the NUnit sources to investigate further.