This post will clarify the proposed NUnit feature mentioned in the last post, and also propose a new feature (right near the end).
I was thinking of another concrete example of where adding a new feature to NUnit, to allow a more flexible method to create TestFixtures (proposed via a TestFixtureProvider attribute), as mentioned in my last post, may be useful. First, a bit of background on how NUnit was being used:
In the past, I worked on the mapping of financial instruments from an-memory representation of a complex hierarchy of objects into a multi-line string for consumption by another product.
The test fixtures were broadly organised into product types (e.g. bond), with tests for the different conditions encountered by that product (e.g.buy/sell, expired, amortising, etc.)
It was chosen that the tests should be XML driven:
To act as documentation: as the mappings were complex it would provide concrete human-readable examples of the instrument states, and associated mappings.
Tests are more manageable - can generate XML automatically, to give creater coverate quicker.
Allows financial analysts to create additional tests - don’t have to know programming.
The original instrument was represented by COM objects, with the C# mapping code consuming these objects via interop. As COM is interface-driven, it was fairly trivial to create mock objects implementing these interfaces, for testing.
As the instrument was hierarchical, an XML representation was a natural fit. There were XML files for each test fixture, each containing nodes which contained the XML representation, and either the expected string output from the mapping or any expected exception.
e.g.
InstrumentTypeATests.xml
<TestFixture name="InstrumentATests">
<Test name="TestACertainConfiguration">
<Instrument>This is an XML representation of an instrument.</Instrument>
<ExpectedMapping>
An expected mapping string in a CData section.
</ExpectedMapping>
</Test>
</TestFixture>This is where we’re getting to the point of this post. We want to run these XML tests in NUnit - there are a number of ways to do this.
One is to have a fixture, with a RunXmlTest method, that would load the XML file (with the filename either hardcoded into the file, or into the fixture constructor - remember we’ve currently got no way to specify constructor arguments, other than via a parameter as passed into the commandline?), then would iterate through the nodes, populating the mock objects, map these mock objects into their string representation, and check that mapping against the expected mapping as specified in the tests. Which is fine, except that we’ll have a single green bar per fixture/test xml file, which isn’t very useful when there are hundreds of test cases.
(forgive any poor syntax/noddy-errors - I can’t remember exactly what this looked like, but this should be enough for you to get the idea)
[TestFixture]
InstrumentATestFixtureOriginal : BaseXmlFixture
{
[FixtureSetUp]
public void FixtureSetup()
{
// Set the path as used by the base class.
pathToXml = "InstrumentTypeATests.xml";
// Base class method to load the XML.
LoadXml(); // loads into xmlDocument member
}
[Test]
public void RunXmlTest()
{
foreach( XmlNode node in xmlDocument )
{
RunTest(node.Name);
}
}
}An improvement is to have a TestACertainConfiguration test corresponding with each Test node in the XML - so we get a green bar for each of the tests:
[TestFixture]
InstrumentATestFixtureNew : BaseXmlFixture
{
[FixtureSetUp]
public void FixtureSetup()
{
// Set the path as used by the base class.
pathToXml = "InstrumentTypeATests.xml";
// Base class method to load the XML.
LoadXml(); // loads into xmlDocument member
}
[Test]
public void RunXmlTest1()
{
RunTest("Test1");
}
[Test]
public void RunXmlTest2()
{
RunTest("Test2");
}
}But only tests as specified in the C# NUnit test are executed.
The solution that I used for the above, as all of the functionality is in the base
classes, is to generate the classes InstrumentATestFixtureNew from using XSLT on the XML files to produce .cs files which are compiled into the build.
This worked nicely, it allowed any changes in the XML files to be taken into account in the tests, but still required a rebuild.
So, the desired behaviour is to allow new test files to be added into a folder somewhere, and the tests to be modified, and have test fixtures generated according to the data in the xml files, without a rebuild of the test project. This is where the new proposed NUnit features would be useful.
Method 1, using the proposed TestFixtureProvider, is to have a constructor on InstrumentATestFixtureOriginal taking the path to the XML file to load, and registering instances of this class with the NUnit framework (see last post for more details on the syntax). This would work OK, but would mean that all of the tests contained in the fixture would only register one red/green bar in the GUI.
Method 2 is a variation on method 1, in that we’d modify InstrumentATestFixtureOriginal to be constructed with the path to the XML, and also the name of the test node - i.e. we’d create a fixture for each test. This would work, but would look cluttered on the GUI. It may be that we could specify in the TestFixtureProviderAttribute how to group these fixtures on the GUI. But this still feels a bit hack-y.
Method 3 is to create the fixtures in memory dependant on the XML, using either CodeDOM or Reflection.Emit. We’d still need to use the proposed TestFixtureProvider property to get these test fixtures to be registered, but it would be the neatest solution. However, it would be a lot of work.
Method 4 is where we get to the second proposed new NUnit functionality. It’s basically to -somehow- implement method 3 but without having to generate the fixtures in code. What we’d like is for every time the base class’s RunTest method is called, it appears on the UI. Maybe a new method called “RegisterTest(string name)” could be added to NUnit to be called from RunTest.