Zagreb


We just got back from a long weekend in Zagreb, and the photos as ever are up on the gallery. We stayed at Ivanas (she was a very good host, as always, thank you! We’re returning the favour in England soon though 🙂 )

I went out for a long weekend, but as Mare had accrued more holiday than me she spent a week out there (and got to do stuff like go to the theatre – things that she can’t usually do when I’m with her). She visited a couple of exhibitions, including the painter Mark Chagall (who I’d never heard of!)

We had a great time, the weather was surprisingly warm and sunny (though it was cold for Mare at the start of the week – she was there for the Snow Queen ski races, and saw many politicians and celebrities in the town).

We went to the Zlatni Medo (golden bear) pub – they have their own microbrewery. Last time we were in Zagreb we went, but I overdid it eating delicious sausages, this time I concentrated on the beer 🙂 I sampled all of the beers – Crna Kraljica is very good, but my favourite was the GRIČKA VJEŠTICA, it went down very well! Mare is not a big lager drinker, but she did enjoy the zlatni medo…

We ventured outside of Zagreb; it’s not every day that you get to visit the birthplace of a communist dictator. We stopped off briefly in the town of Klanjec, to visit the museum of renowned Croatian sculptor Antun Augustincic. We visisted the village where Marshal Tito was born, and it’s setup just as it was in the olden days, with examples of rural life and customs. It was very peaceful, fascinating, and in beautiful surroundings. Not far away we visited a castle, again very interesting.

We finished off by eating at a restaurant serving traditional cuisine in the style of Hrvatsko Zagorje (the part of Croatia west of Zagreb, and close to the Slovenian border), very different to the Dalmatian style of food I’ve been used to so far. I gorged myself silly on homemade cheese, bread, followed by exceptional country style mushroom soup, and -then- venison goulash. I did feel guitly afterwards though, just outside the restaurant were the brother and sister deers!!! Marijana was then trying to make me feel guity to point out how cute the bambis were (but my stomach beat my head).

Sadly the time went too quickly, but we managed to squeeze in another trip to town in the morning, where we made it to the antique market in the Britanski Trg (British Square). We then sat around drinking coffees enjoying the last rays of sunshine before we had to return to rainy England.

This is a blog post to explain exactly what exactly parameterized data-driven unit testing is and why it’s there, and then to explain a bit about how this effected the implementation of the Rakija NUnit add-in.

First, what is data-driven unit testing? It’s running tests with data which is driven from external data (as may be from an xml file or database). NUnit makes this quite difficult, as the tests are attribute driven, they are fixed, and fixed at the time that the tests are created.

Data-driven fixtures.
A common desire is to run fixtures with different set of parameters. An example of this is to test key input filtering under different UI cultures.

This is a very contrived example, and the tests are in no way realistic, but it demonstrates what data-driven tests are about. My example is a simple class that takes various string inputs and performs operations on them. Here it is:

///

/// Parses and operates on various string values.
///

public static class ValueParser
{
///

/// Returns the addition of two strings representing
/// double values.
///

/// The combined value.
public static double GetAdditiveValue
(string value1, string value2)
{
return double.Parse(value1) + double.Parse(value2);
}
}

And of course, the contributing unit test:

[TestFixture]
public class TestValueParser
{
[Test]
public void TestAdditiveValue()
{
Assert.AreEqual(4.5, ValueParser.GetAdditiveValue(
“2.3”, “2.2”));
}
}

Of course in real life, we’d have tests to pass in garbage input, checking for any exceptions etc. Our class would also be much larger, containing many methods, along with many (maybe hundreds) of corresponding unit tests in our test fixture. We’ve happily deployed the first release of our software, and have many happy customers, when our customer representative or product manager informs us that we need our software to run on European users computers. We don’t get panicked, we know that our input data is coming from XML, and is formatted in invariant culture, but we need to make sure to change our code such that double.Parse takes in CultureInfo.InvariantCulture. We’d ideally like to ensure that our tests run in both cultures. We could duplicate our fixture with a different setup test, but we all know that duplicate code is bad, right? So we use the following pattern.

// NOTE: TestFixture class has been removed.
public class TestValueParser
{
protected string cultureString = “invalid”;
private CultureInfo initialCulture;

[SetUp]
public void Setup()
{
this.initialCulture
= Thread.CurrentThread.CurrentCulture;
Thread.CurrentThread.CurrentCulture =
CultureInfo.GetCultureInfoByIetfLanguageTag(
cultureString);
}

[TearDown]
public void TearDown()
{
Thread.CurrentThread.CurrentCulture =
this.initialCulture;
}

[Test]
public void TestAdditiveValue()
{
Assert.AreEqual(4.5, ValueParser.GetAdditiveValue(
“2.3”, “2.2”));
}
}

[TestFixture]
public class TestValueParserUSCulture : TestValueParser
{
public TestValueParserUSCulture()
{
cultureString = “en-US”;
}
}

[TestFixture]
public class TestValueParserFRCulture : TestValueParser
{
public TestValueParserFRCulture()
{
cultureString = “fr-FR”;
}
}

We leave all of our tests intact, we have to make some minor modifications to our original test fixture (remove the test fixture attribute, and do some setup work), but we’re able to test our fix very quickly (obviously we follow red bar/green bar/refactor, and write the tests before changing our code), and as we have a high test code the new release of the software works flawlessly, we finish early and spend the remaining time playing Crysis.

This pattern is very common in NUnit, but I haven’t found it named anywhere, I call it the Test Fixture Inheritance Pattern. NOTE: This pattern is not supported in Visual Studio Team System 2005 testing, but it’s fixed in the 2008 version.

(As an aside, I’ve read about the upcoming xUnit framework, and it says that tests aren’t grouped into fixtures, I’m not sure how it would cope with this pattern in that case? It’s possible that you could parameterized each test, and use something like a RowTest attribute to pass in the culture string, but by testing a new culture you’d have to add a new attribute to each and every test.)

The first thing to notice about the Test Fixture Inheritance Pattern is that it isn’t very OO. It wouldn’t be too much work though to create an NUnit extension that would allow parameterizing the test fixture constructor, in a similar way to how the RowTest attribute works, i.e.

[TestFixture]
[Parameter(“en-US”)]
[Parameter(“fr-FR”)]
public class TestValueParser
{
protected string cultureString = “invalid”;
private CultureInfo initialCulture;

public TestValueParser(string cultureString)
{
this.cultureString = cultureString;
}

[SetUp]
public void Setup()
{
this.initialCulture
= Thread.CurrentThread.CurrentCulture;
Thread.CurrentThread.CurrentCulture =
CultureInfo.GetCultureInfoByIetfLanguageTag(
cultureString);
}

[TearDown]
public void TearDown()
{
Thread.CurrentThread.CurrentCulture =
this.initialCulture;
}

[Test]
public void TestAdditiveValue()
{
Assert.AreEqual(4.5, ValueParser.GetAdditiveValue(
“2.3”, “2.2”));
}
}

That would be nothing more than syntactic sugar though (it does make the tests look much neater). This could be more powerful though – it could be expanded to read data from a database or an XML file (again, similar to VSTS’s DataSource attribute).

Rakija instead provides an interface IDynamicFixtureSpecifier, which is used on a class to specify which fixtures to create. Each fixture then must implement another interface, and the fixture need to implement IDynamicFixture. Our example now looks like this:

// NOTE: TestFixture attribute has been removed.
public class TestValueParser : IDynamicFixture
{
protected string cultureString = “invalid”;
private CultureInfo initialCulture;

public TestValueParser(string cultureString)
{
this.cultureString = cultureString;
}

[SetUp]
public void Setup()
{
this.initialCulture
= Thread.CurrentThread.CurrentCulture;
Thread.CurrentThread.CurrentCulture =
CultureInfo.GetCultureInfoByIetfLanguageTag(
cultureString);
}

[TearDown]
public void TearDown()
{
Thread.CurrentThread.CurrentCulture =
this.initialCulture;
}

[Test]
public void TestAdditiveValue()
{
Assert.AreEqual(4.5, ValueParser.GetAdditiveValue(
“2.3”, “2.2”));
}

#region IDynamicFixture Members

public string Name
{
get { return “TestValueParser ” + cultureString; }
}

#endregion
}

The fact that this is a test fixture is detected from the add-in by the fact that it implements IDynamicFixture, I could have instead created a new attribute for the class, and either reflected for the Name property (or created an attribute to return the name), but that’s by-the-by.

To actually specify the instances of our fixture, we use code like the following:

public class SpecifyFixtures : IDynamicFixtureSpecifier
{
#region IDynamicFixtureSpecifier Members

public Type FixtureType
{
get
{
return typeof(TestValueParser);
}
}

public IList GetUserFixtures()
{
List fixtures =
new List(2);
fixtures.Add(new TestValueParser(“fr-FR”));
fixtures.Add(new TestValueParser(“en-US”));
return fixtures;
}

#endregion
}

Your first thought may be to wonder why I’ve created such a convoluted way of creating fixtures, when a page or so ago I showed how neatly this could be done by creating a new attribute. Well, this is much more powerful, we’re not limited to querying a database or reading an XML file – we could load up our assembly to test, and reflect over certain types in it, creating a new fixture for each type. The sky’s the limit.

Parameterized test methods.
I’ll quickly go over the motivation over parameterized test methods, before discussing why I’ve been forced to implement these myself due to the current NUnit (as of 2.4.6) extension interface, rather than deferring these to Kelly Anderson’s extension.

Another common requirement when unit testing is to run a given test with different input data. Before the existence of any RowTest attribute, a common thing to do may have been.

[Test]
public void SomeTest()
{
// .. read some data in an xml file
// .. for each piece of data, perform some common testing
// .. { assert on the data; }
}

This obviously works, but the problem of it is lack of visibility/feedback. For a file that may contain hundreds of pieces of data, we have a single red or green bar. We’d like to have visibility of each individual case. VSTS has a DataSource attribute to cope with this cases like this (reading from a database or XML File).

Now that I’ve, very briefly, covered the motivations behind parameterized test cases, I’m going to go into the implementation details of Rakija.

Kelly Anderson has created a more powerful interface, similar to Rakija’s, where a method can be decorated with an IterativeTest attribute to specify the test data for the test. (Kelly’s interface is much more user-friendly than Rakija’s).

Unfortunately, I’ve had to duplicate the functionality of this Kelly’s work in Rakija, due to the internals of NUnit’s extensibility mechanism. I’ll explain why below, and propose a change to the mechanism to allow the extensions to co-exist.

Similar to Rakija, Kelly’s mechanism creates a TestCaseBuilder to recognise the parameterized test cases. This works without any problem, but will fail when used with any of the above parameterized test fixtures described above.

Kelly’s extension works by finding the method decorated with his IterativeTestAttribute, creating a new instance of the type that contains that attribute, and calling the method decorated by that attribute to find the parameters to create the test with. He gets away with creating a new instance of the type by the fact that all instances of the type will contain the same data.

The problem comes when you have parameterized test cases – each test case has been created with different parameters, and depending on those constructor parameters we may wish to pass different parameters to our tests (our test fixtures may have been created with a given node of an xml file, and we may wish to pass various sub-nodes of that node to an XML file).

Rakija is forced to deal with this situation by taking control of the parameterized test case situation when parameterized fixtures are also involved.

The proposed fix, to allow dynamic fixtures to exist with any other extension (e.g. Kelly Anderson’s IterativeTest extension, or Andreas Schlapsi’s RowTest extension) is to make a change to the extensibility mechanism.

Currently, NUnitTestFixtureBuilder (or our derived class), has to call

this.testCaseBuilders.BuildFrom(method);

where method is a MethodInfo
And this forces each test case builder either call a static method, or construct a type to create a method on. The return type from this method is a test, which gets added onto the fixture/suite’s list of tests to run.

In the parameterized fixture case, the interface on AbstractTestCaseBuilder would better be:
protected override TestCase MakeTestCase(object instance, MethodInfo method);

or something similar, where instance would be the instance that contains the method. That would allow the method to be invoked on the instance that it lived on, allowing parameterized fixtures to co-exist with the other mentioned extensions.

Excuses

For not having another Jabuka post yet… Well, I’ve been really busy at my previous employers getting upto speed on various new technologies, then as you may have read on my other blog, I’ve been busy preparing for a new job (I might blog a bit about that at some point), and work always takes priority. I did have to create an interesting small WPF 3D application as part of my application process (I’ll tart this up and get it online when I get chance).

I haven’t given up on Jabuka, I spent a lot of time over Christmas researching different constraint methods (LCP solvers, PGS solvers and Sequential Impulses), and will playing around with this as soon as I get a chance to breathe (oh, and I finally packaged up my NUnit data-driven testing extension, here).

Otherwise, my new years resolution is to play with F# – I attended a Next Gen Usergroup talk last year at Microsoft’s Cambridge site by Don Syme. Since then I’ve had time to do some reading around, but not to actually create anything in anger. Obviously, some functional ideas have influenced C# (lambda expressions etc.) but I want to see how working in a functional mindset might chance some of my ideas of designing code.

There’s been lots of blogging recently on immutability in C# (see here and here) recently – in functional languages everything is immutable, and obviously not having immutable state gets around guarding state with multiple threads, it’s just the overall program design I can’t picture yet (I’m used to thinking of looking for common interfaces, design patterns etc.) There’s also new parallel/Concurrent resources available from Microsoft which do fit nicely into all this (see F#’s asynchronous workflows).

A change of approach/jabuka 0.6

As mentioned on the Jabuka website, I’m moving away from a tutorial-based approach, mainly because of lack of time. It’s much quicker to implement new code and then blog retrospectively, highlighting new features and interesting code/design decisions. In the tutorial based approach, making a change/bug fix in an earlier tutorial forces all later tutorials to be updated.

That being said, it’s worth discussing changes are in the latest version, 0.6, of Jabuka.
This code introduces a couple of simple steering behaviours, and flocking boids. Given the roadmap, why did I suddenly go off on a total tangent and implement boids? Well, apart from flocking behaviour being especially cool (look on youtube for the Carling Belong commercial, or starling flocks), I wanted to stress the engine (I use that term very loosely given the current functionality of Jabuka) as I’m itching to demonstrate multithreading and performance enhancements. This version of Jabuka has shown that it is still too early to look at performance, however, as the application still has some fundamental flaws to be addressed.

The flocking application implements a flock (based on the algorithms on Chris Reynolds website, and roughly based on the algorithms in the Killer Game Programming in Java book). The changes to Jabuka were that the force for a rigid body is obtained in the Update() method from introducing an IForceProvider interface, implemented by various behaviours.
If you look at the implementation of the steering behaviours, you can see that there is a large number of classes implementing this interface, each performing a simple operation such as aggregation/limitation. This does lead to a proliferation of classes, but I’m happy to live with it for now as it makes unit testing easier, and may lead to more easily changing behaviour dynamically at runtime.

The Update() method calling on the IForceProvider is not perfect in this case – the objects are iterated over, and each object’s Update() method calls on its IForceProvider. For the steering behaviours, the behaviour compares the body’s location to its neighbours to calculate its force and update its position. The problem is that the resulting locations are dependent on the order in which the bodies are updated. This has problems in measuring the flocking quality (more on this later), and it may be desired to update the steering forces independent of Update(), so we don’t do this in every time step (as it’s an expensive operation – O(2)?). Also, having the state of an object dependent on every other object on every timestep will get really messy when we introduce multithreading. Having each body’s Update() method independent of any other means that it’s much easier to parallelise/vectorise/split into multiple threads – it’s likely we’d want to avoid making every IEulerRigidBody thread safe, and would want to implement the thread safety at a higher level (though I won’t jump the gun, we’ll get there soon enough).

The sort of changes above hint that it may be better to have a Physics Manager class, that owns the Collision object, the rigid bodies, and also has knowledge of the IForceProviders and manages the overall sequence of operation.

The larger problem with having the currently flocking behaviour, including collision detection, is that I haven’t yet implemented resting contacts (well, any contact forces) apart from collisions, and as objects are forced towards each other, the simulation can halt. I’ve temporarily fixed this by introducing a scene where the boids don’t have any collision detection. I’ve also introduced a scene which demonstrates the halting, with only two spheres.

As to steering behaviours and flocking itself… I’ve introduced a scene for the seek behaviour, demonstrating that having the steering force directed towards the target is insufficient, the velocity tangential to the targed needs to be taken into account. The idea is to find the optimal trajectory towards the target. Reynolds copes with this by providing a force such that the body’s velocity is adjusted to be the maximum velocity towards the target. This is better, but does mean that the body needs a concept of its maximum velocity (this works OK on Earth, as air friction gives a terminal velocity, but in space this is arbitrary). Also, the maximum velocity may vary – the maximum velocity into the wind is less than having a following wind, and may depend on a body’s shape (or its orientation with respect to its velocity).

Looking at the arrive behaviour, where the speed is arbitrarily slowed down, the first thought that comes to mind is that a PID control towards the target would be desired. However, the problem is that the velocity tangential/perpendicular and distance from the target all need to be taken into account – a more complicated control scheme would need to be used, and that may be too computationally expensive.

The current boid implementation currently uses the same proximity distance for all individual behaviours (separation, alignment, cohesion). It may be that for separation, a smaller proximity should be used. Also, currently, I’m using a simple sphere without taking into account the blind spot behind the boid.

There are lots of tweaks to be made (and a Genetic Algorithm could be used to change these dynamically), but many of them are pointless without a method of measuring the quality of the flock. There is an interesting paper on that here. Some background on the flock theory / the underlying physics of flocking, can be found here, and here.

Croatia Christmas and New Year 2006

We spent Christmas and New Year in Split (http://www.split-online.com/.) partying, eating, drinking. Being on the Mediterranean, the weather was very spring-like, being about 13 deg.C every day but feeing warmer in the sunshine. I must have put on a few kilos by eating an extraordinary amount of food, and enjoyed sitting in the sunshine drinking proper coffees. Photos are up here (and I’m trying to clear up the backlog, so last year’s photos of Zagreb and Ljubljana are also up here

We didn’t really do much sightseeing this time, we were catching up with Mare’s family and friends (and playing a lot of Risk). Oh, thanks to Robi for the home made herbal Rakija and red wine (unfortunately I couldn’t bring it through airport security, but will get hold of it soon!).

I really like being in Split – everybody is always really warm and hospitable, and really makes me feel welcome. I also like the culture there, loads of English people say how they’d miss the local pub and real ales, but I much prefer being in a konoba (like a bistro) eating and drinking with live performers playing traditional music. Everybody’s into singing and dancing – Mare’s got a great voice (her brother was a pop idol finalist and her cousin is in a band), and it’s great when everyone in the bar starts singing along. My dad said that it was quite like that in pubs in Mid Wales maybe 30 years ago – at the end of the night everyone would be singing along together; it’s a pity that that’s died out now (well, not for anyone that would have to suffer hearing my singing!)

Anyway, here’s the list of foods that rocked my boat this time: Salata od hobotnice (octobus salad), black squid risotto, pašticada, Francuska Salata (with homemade mayonnaise), pršut, tuna pašteta, fried white bait, anchovies, olives, home made cheese, sarma (minced meat in cabbage leaves) along with so many cakes I lost count!

mixed mode debugging tips

I’m currently working on a C# component to be consumed by a COM application (and did the same at my last place of work), and thought I’d give a few hints on debugging this (as there doesn’t seem to be much info on this via a web search, and I forgot all the little tips I learnt about this last time after a few months gap).

Firstly, the assembly is registered with regasm /codebase (or RegistrationServices.RegisterAssembly()), so that COM can load it up from
wherever it resides on disc. However, when mixed mode debugging, the debugger loads the assemblies from the path of the unmanaged executable. So make sure that the assemblies and
pdbs are copied to this location, and remember to do this whenever rebuilding the managed component (else you could end up very frustrated from debugging the wrong version).

There’s a bit of a rant coming on, ’cause even if you remember to do all of this, Visual Studio 2005 crashes out inexplicable far far far too regularly, (or hangs, or fails to connect the debugger, sometimes fixed by a restart of a computer, or voodoo magic) and we’re not in a position at work to see whether the upcoming SP1 beta fixes this *but fingers crossed*.

Jabuka Tutorial 3

Tutorial 3 of Jabuka (C# physics and 3D engine) is finished (a lot shorter than tutorial 2, but I didn’t want to be working on it for six months again). The main page is here http://www.taumuon.co.uk/jabuka/, and the pdf file is here http://www.taumuon.co.uk/jabuka/Tutorial3/Tutorial3_v1.pdf.

This tutorial basically just consumes the work done in the last tutorial (the Test-Driven Developed physics component), by our 3D app. Also has a brief discussion of favouring composition over inheritance.

I think next up will be simple sphere-sphere collision detection, and then once this is working then add rotations. Some feedback on handling rotations would be good – I’ve got quaternion rotation working, but as it’s kinda a beginners engine maybe I should just use matrices? Anyone care?

Jabuka Tutorial 2

Tutorial 2 of Jabuka is now finished! It can be found on http://www.taumuon.co.uk/jabuka/ (or a direct link to http://www.taumuon.co.uk/jabuka/Tutorial2/Tutorial2_v1.pdf ).

Tutorial 1 was an introduction to Visual Studio 2005 and the Tao Framework in developing a simple 3D application (as well as being a tutorial on OO design and refactoring).

Tutorial 2 is predominantly a tutorial on Test-Driven Development using NUnit (whilst developing a very simple C# Physics Engine).

This tutorial was far too big, and took far too long to finish, so the next ones will be much shorter (5 to 10 pages) but hopefully more regular!

Data-driven NUnit using 2.4 Extensibility.

(Edit: There is now a build NUnit add-in supporting this post, hosted at rakija)

Following my earlier blog posts, I posted a feature request to the NUnit list on Sourceforge, and NUnit maintainer Charlie Poole replied saying that I should nvestigate whether I could get the functionality I want using the new Extensibility features in NUnit 2.4.

Many thanks to Charlie Poole for giving me an insight into how Extensibility works, into the inner workings of NUnit, and for giving me guidance in creating these plug-ins. Any mistakes in this posting are purely my own. There are two problems that these solve – creating fixtures dynamically
(parameterized fixtures), and creating tests dynamically (parameterized tests). This can be combined to give dynamic fixtures that contain dynamic tests. The dynamic fixtures and tests that are loaded are specified at user assembly load time. This will be clearer below.

The add-in code can either be build into an assembly to live in an addins sub-folder under the NUnit install location, or can be supplied in the actual user assembly that is loaded into NUnit (primarily for quick development of add-in code).

The terminology for what I was originally thought in my head were called test fixtures, as are the classes present in the user assembly decorated with the [TestFixture] attribute, are known as user fixtures, to distinguish them from NUNit’s internal TestFixture classes.

The code below is built against NUnit 2.4 beta 1 (build 2.3.6162).

Dynamic fixtures

I’ll show how the tests make use of the plug-ins first, then show how the plug-ins are implemented.

We want to have a user fixture that takes a constructor, and create many instances of these with different constructor arguments. We expect the test TestConstructorValue1 to pass when we construct it with an int value of 1, and
similar for the other methods.

The string in the constructor will be used to give a name to this instance of the user fixture.


public class MyTestFixture1 : IDynamicFixture
{
private int value = 0;
private bool fixtureSetupCalled = false;
private string name = "default";

public MyTestFixture1(string name, int value)
{
this.name = name;
this.value = value;
}

[Test]
public void TestConstructorValue1()
{
Assert.IsTrue(value == 1);
}

[Test]
public void TestConstructorValue2()
{
Assert.IsTrue(value == 2);
}

[Test]
public void TestConstructorValue3()
{
Assert.IsTrue(value == 3);
}

[Test]
public void TestFixtureSetupCalled()
{
Assert.IsTrue(fixtureSetupCalled);
}

[TestFixtureSetUp]
public void FixtureSetup()
{
fixtureSetupCalled = true;
}

#region IDynamicFixture Members

public string Name
{
get
{
return name;
}
}

#endregion
}

This class isn’t decorated with the TestFixture attribute, to distinguish it from standard NUnit user fixtures. But otherwise it behaves the same. It has Test methods with the same attributes. It implements the IDynamicFixture interface, defined in the plug-in, that has a name property to provide a name for this user fixture in NUnit’s tree view.

Plug-in code


public interface IDynamicFixture
{
string Name
{
get;
}
}

Now, we need a way for instances of this class, our user fixtures, to be created and provided to NUnit. We do this by having a class that implements an interface, IDynamicFixtureBuilder. This class is created by NUnit on loading the assembly, and the GetUserFixtures() method called to register our user fixtures with NUnit:


public class MyDynamicFixtureBuilder : IDynamicFixtureBuilder
{
public List<IDynamicFixture> GetUserFixtures()
{
List<IDynamicFixture> fixtures = new List<IDynamicFixture>(3);
fixtures.Add(new MyTestFixture1("instance1", 1));
fixtures.Add(new MyTestFixture1("instance2", 2));
fixtures.Add(new MyTestFixture1("instance3", 3));
return fixtures;
}

public Type FixtureType
{
get
{
return typeof(MyTestFixture1);
}
}
}

We can also have normal NUnit user fixtures decorated by the TestFixture attribute in the user assembly. The plug-in code defines the interface that will be implemented and used to create the user fixtures:


public interface IDynamicFixtureBuilder
{
List<IDynamicFixture> GetUserFixtures();

Type FixtureType
{
get;
}
}

Now, we need to implement a SuiteBuilder that will create our user fixtures. NUnit loads our assembly, and for each type found in the assembly that it loads, it passes the type to two methods on the ISuiteBuilder interface, BuildFrom, and CanBuildFrom, to see whether that builder wants to create a suite from that type. In the normal case, a NUnit user fixture is decorated by the TestFixture attribute, and the NUNitTestFixtureBuilder informs NUNit that it can build that type, and builds it. We want to have our MyDynamicFixtureBuilder type recognised – we want to create the instances of MyTestFixture1 ourselves.


[SuiteBuilder]
public class MyFixtureBuilder : NUnitTestFixtureBuilder
{
#region ISuiteBuilder Members

public override TestSuite BuildFrom(Type type)
{
if (CanBuildFrom(type))
{
IDynamicFixtureBuilder dynamicFixtureBuilder =
(IDynamicFixtureBuilder)Reflect.Construct(type);

TestSuite suite = new TestSuite(
dynamicFixtureBuilder.FixtureType);

// Iterate over our user fixtures.
foreach (IDynamicFixture userFixture in
dynamicFixtureBuilder.GetUserFixtures())
{
TestFixture fixture =
new ParameterizedTestFixture(userFixture);
fixture.TestName.Name = userFixture.Name;

IList methods = GetCandidateTestMethods(
userFixture.GetType());
foreach (MethodInfo method in methods)
{
TestCase testCase = TestCaseBuilder.Make(method);

if (testCase != null)
{
fixture.Add(testCase);
}
}
suite.Add(fixture);
}

return suite;
}

return null;
}

public override bool CanBuildFrom(Type type)
{
if (Reflect.HasInterface(type,
"Taumuon.DynamicFixture.IDynamicFixtureBuilder"))
{
return true;
}
return false;
}

#endregion
}

Our CanBuildFrom method just checks for types implementing our IDynamicFixtureBuilder interface. The BuildFrom method creates an instance of this type, then it creates a TestSuite passing in the fixture type profided from the IDynamicFixtureBuilder – this is the actual type of the user fixtures. It calls GetUserFixtures(), which is where we construct our parameterized user fixtures.In the loop I create a ParameterizedTestFixture, and use the GetCandidateTestMethods method on the base NUnitTestFixtureBuilder class to get methods of the correct signature, and the TestCaseBuilder.Make() creates standard NUnit tests (as they implement the [Test] attribute).

These tests are added to the fixtures, and the fixtures are added to the TestSuite.

The purpose of the ParameterizedTestFixture is to keep the instance of our
user fixure, as the NUNitTestFixture implementation of DoOneTimeSetup instantiates new instances of the user fixtures from the FixtureType. We want to keep an instance of our non-default-constructor constructed types around so that we can call their test methods.


public class ParameterizedTestFixture : NUnitTestFixture
{
private object userFixture;

public ParameterizedTestFixture(object userFixture)
: base(userFixture.GetType())
{
this.userFixture = userFixture;
}

protected override void DoOneTimeSetUp(TestResult suiteResult)
{
// We assign the user fixture here,
// so the correct instance is invoked on when running the test.
this.Fixture = userFixture;
base.DoOneTimeSetUp(suiteResult);
}
}

Dynamic tests (parameterized test cases)

The idea behind dynamic tests is to have a fixture that is something like the following. The fixture can have normal [Test] decorated NUnit tests, but the plug-in has a new attribute [DynamicTest], to indicate a test that takes parameters. The user fixture implements the IDynamicTestBuilder interface to indicate that this method contains dynamic tests, and this interface contains the SetTestsToRun method, as obviously NUnit wouldn’t know what arguments to call the test method with. The SetTestsToRun() method is called by the plug-in, passing in an object implementing IDynamicTestDetails. Our user fixture calls on this interface, specifing for each call of the DynamicTest method, what name it should appear with in the NUnit tree view, the name of the actual method to call, and the arguments to pass into the method.


public class MyUserFixure : IDynamicTestBuilder
{
[DynamicTest]
public void MyTestMethod(string arg1)
{
Assert.AreEqual(arg1, "arg1");
}

[Test]
public void NormalTestMethod()
{
Assert.IsTrue(true, "This is a normal NUnit test");
}

#region IDynamicTestBuilder Members

public void SetTestsToRun(IDynamicTestDetails setDetails)
{
setDetails.SetTestDetails("Test 1", "MyTestMethod", "arg1");
setDetails.SetTestDetails("Test 2", "MyTestMethod", "arg2");
}

#endregion
}

Plug-in code

The plug-in code contains the definition of our DynamicTest attribute, and our two interfaces:


[AttributeUsage(AttributeTargets.Method, Inherited=false,
AllowMultiple=false)]
public class DynamicTestAttribute : Attribute
{
}

public interface IDynamicTestDetails
{
void SetTestDetails(string testName, string testMethodName,
params object[] parameters);
}

public interface IDynamicTestBuilder
{
void SetTestsToRun(IDynamicTestDetails details);
}

Similar to the Dynamic Fixtures, we want to provide a SuiteBuilder that will handle adding our custom user fixtures containing dynamic tests into the system.


[SuiteBuilder]
public class MyDynamicTestFixtureBuilder :
NUnitTestFixtureBuilder, IDynamicTestDetails
{
private TestSuite fixture = null;

public override TestSuite BuildFrom(Type type)
{
if (CanBuildFrom(type))
{
IDynamicTestBuilder testBuilder =
(IDynamicTestBuilder)Reflect.Construct(type);
fixture = base.BuildFrom(type);

// base.BuildFrom will have added a single instance
// of our test - it will have found the
// DynamicTestMethodBuilder and added the 'vanilla'
// (i.e. no parameter, test name set to
// the method name) version of the test method.
// We only want the user-added version of this method to be present.
for (int i = fixture.Tests.Count - 1; i >= 0; --i)
{
if (fixture.Tests[i] is DynamicTestMethod)
{
fixture.Tests.RemoveAt(i);
}
}

testBuilder.SetTestsToRun(this);

return fixture;
}
return null;
}

public override bool CanBuildFrom(Type type)
{
if (Reflect.HasInterface(type,
"Taumuon.DynamicTests.IDynamicTestBuilder"))
{
return true;
}
return false;
}

... more code below!

}

This just looks for types implementing the IDynamicTestBuilder interface, and calls on the base NUnitTestFixtureBuilder class to build our user fixture. This makes sure that all normal NUnit TestAttribute-decorated methods are included. This will also have detected our DynamicTest decorated method, and added an instance of this to our test, we’ll cover why this is in a bit when we look at our custom TestBuilder, but for now just see that we remove this DynamicTest from the fixture, as we want to add the versions we build ourselves, containing parameters.

We create our type implementing IDynamicTestBuilder from the type passed into the SuiteBuilder, and call SetTestsToRun on it, passing in this, as our SuiteBuilder implements IDynamicTestDetails. Here’s the implementation of this interface:


/// <summary>
/// TODO: exceptions etc.
/// </summary>
/// <param name="testName"></param>
/// <param name="testMethodName"></param>
/// <param name="parameters"></param>
public void SetTestDetails(string testName, string testMethodName,
params object[] parameters)
{
if (null == fixture)
{
throw new ApplicationException("Test fixture is null");
}

// TODO: should cache some of this stuff.
MethodInfo mi = fixture.FixtureType.GetMethod(testMethodName);
ParameterInfo[] methodParameters = mi.GetParameters();
if (methodParameters.Length != parameters.Length)
{
throw new ApplicationException(
"Wrong number of parameters");
}

for (int i = 0; i < parameters.Length; ++i)
{
ParameterInfo pi = methodParameters[i];
if (pi.ParameterType != parameters[i].GetType())
{
throw new ApplicationException(
"Parameter is of the wrong type");
}
}

TestCase testCase = TestCaseBuilder.Make(mi);
testCase.TestName.Name = testName;
if (testCase is DynamicTestMethod)
{
((DynamicTestMethod)testCase).Parameters = parameters;
}
if (testCase != null)
{
fixture.Add(testCase);
}
}

#endregion
}

The user fixture calls back into this method, specifying the testName, testMethodName, and any parameters (there’s not much point of this if there are no parameters). We get the MethodInfo from the user fixture, from the MethodName that they’ve supplied, check that our parameters array matches the parameters on the DynamicMethod.

Now, to build up the test case, we call TestCaseBuilder.Make() with the method info. This will call into our DynamicTestBuilder (code below), to create a DynamicTestMethod, it sets the parameters on it, and adds this to the user fixture.

This is our DynamicTestMethod:


public class DynamicTestMethod : NUnitTestMethod
{
private MethodInfo method;

#region Constructors
public DynamicTestMethod(MethodInfo method) : base(method)
{
this.method = method;
}

public DynamicTestMethod(MethodInfo method,
Type expectedException, string expectedMessage,
string matchType)
: base(method, expectedException, expectedMessage, matchType)
{
this.method = method;
}

public DynamicTestMethod(MethodInfo method,
string expectedExceptionName, string expectedMessage,
string matchType)
: base(method, expectedExceptionName, expectedMessage,
matchType)
{
this.method = method;
}
#endregion

public override void RunTestMethod(TestCaseResult testResult)
{
// The method info is private in the base class,
// so have a member in this class.

// This is the same as Reflect.InvokeMethod,
// except that it invokes with arguments.
if (this.method != null)
{
try
{
this.method.Invoke(this.Fixture, parameters);
}
catch (TargetInvocationException e)
{
Exception inner = e.InnerException;
throw new NunitException("Rethrown", inner);
}
}
}

public object[] Parameters
{
get
{
return parameters;
}
set
{
parameters = value;
}
}

private object[] parameters;
}

It overrides NUnitTestMethod, and the constructors are provided to capture a copy of the method MethodInfo, as this member is private in the base class. There is a Parameters array, where we can store the parameters that the user fixture specified should be used for this test instance. This overrides RunTestMethod – this is called from the the base class TestMethod.Run after the Setup method is called. The base class RunTestMethod isn’t much different from this, it just calls Reflect.InvokeMethod() that invokes with no parameters. we’ve just copied the guts of that method to here, invoking with the parameters set on our DynamicTestMethod.

This is the code for our DynamicTestMethodBuilder (I’ll introduce it gradually)


[TestCaseBuilder]
public class DynamicTestMethodBuilder : NUnitTestCaseBuilder
{
#region ITestCaseBuilder Members

public override bool CanBuildFrom(
System.Reflection.MethodInfo method)
{
if (Reflect.HasAttribute(method,
"Taumuon.DynamicTests.DynamicTestAttribute", false))
return true;
return false;
}

CanBuildFrom just checks for methods decorated with our DynamicTest attribute.


// Duplication of AbstractTestMethodBuilder and
// NUnitTestMethodBuilder code, as HasValidTestCaseSignature
// isn't marked as virtual
public override NUnit.Core.TestCase BuildFrom(MethodInfo method)
{
// NUnit.Core.TestCase testCase = base.BuildFrom(method);

TestCase testCase = null;

#region From AbstractTestMethodBuilder
if (HasValidTestCaseSignature(method))
{
testCase = MakeTestCase(method);

string reason = null;
if (!IsRunnable(method, ref reason))
{
testCase.RunState = RunState.NotRunnable;
testCase.IgnoreReason = reason;
}

testCase.Description = GetTestCaseDescription(method);

SetTestProperties(method);
}
else
{
testCase = new NotRunnableTestCase(method);
}
#endregion From AbstractTestMethodBuilder

#region From NUnitTestCaseBuilder

if (testCase != null)
{
PlatformHelper helper = new PlatformHelper();
if (!helper.IsPlatformSupported(method))
{
testCase.RunState = RunState.Skipped;
testCase.IgnoreReason = helper.Reason;
}

testCase.Categories = CategoryManager.GetCategories(method);
testCase.IsExplicit = Reflect.HasAttribute(method,
"NUnit.Framework.ExplicitAttribute", false);
if (testCase.IsExplicit)
testCase.RunState = RunState.Explicit;

System.Attribute[] attributes =
Reflect.GetAttributes(method,
"NUnit.Framework.PropertyAttribute", false);

foreach (Attribute propertyAttribute in attributes)
{
string name = (string)Reflect.GetPropertyValue
(propertyAttribute,
"Name", BindingFlags.Public |
BindingFlags.Instance);
if (name != null && name != string.Empty)
{
object value =
Reflect.GetPropertyValue(propertyAttribute,
"Value", BindingFlags.Public |
BindingFlags.Instance);
testCase.Properties[name] = value;
}
}
}

#endregion From NUnitTestCaseBuilder

return testCase;
}

BuildFrom is pretty simple. It is only present as the base class’s HasValidTestCaseSignature isn’t marked as virtual (though this should change in a later NUnit build – I’ve got a feature request in for this). We have to copy the code out of both of the base class methods, just so that our own version of HasValidTestCaseSignature is called. Obviously, this will be much neater once that method is marked as abstract.

Here’s our HasValidTestCaseSignature method:


// Change over base class method is to remove requirement
// for number of parameters to be zero.
// TODO: note that AbstractTestCaseBuilder method should
// be marked as virtual.
protected new bool HasValidTestCaseSignature(MethodInfo method)
{
return !method.IsStatic
&& !method.IsAbstract
&& method.IsPublic
&& method.ReturnType.Equals(typeof(void));
}

This method’s only difference over the base class method is to remove a check for the number of parameters to be zero (as obviously we’ve got parameterized tests).

The final method of this class is MakeTestCase.


protected override NUnit.Core.TestCase MakeTestCase(
MethodInfo method)
{
Type expectedException = null;
string expectedExceptionName = null;
string expectedMessage = null;
string matchType = null;

Attribute attribute = Reflect.GetAttribute(method,
"NUnit.Framework.ExpectedExceptionAttribute", false);
if (attribute != null)
{
expectedException = Reflect.GetPropertyValue(
attribute, "ExceptionType",
BindingFlags.Public | BindingFlags.Instance) as Type;
expectedExceptionName = (string)Reflect.GetPropertyValue(
attribute, "ExceptionName",
BindingFlags.Public | BindingFlags.Instance) as String;
expectedMessage = (string)Reflect.GetPropertyValue(
attribute, "ExpectedMessage",
BindingFlags.Public | BindingFlags.Instance) as String;
object matchEnum = Reflect.GetPropertyValue(
attribute, "MatchType",
BindingFlags.Public | BindingFlags.Instance);
if (matchEnum != null)
matchType = matchEnum.ToString();
}

if (expectedException != null)
return new DynamicTestMethod(method, expectedException,
expectedMessage, matchType);
else if (expectedExceptionName != null)
return new DynamicTestMethod(method, expectedExceptionName,
expectedMessage, matchType);
else
return new DynamicTestMethod(method);
}
}

The code for this method is identical to the base class NUnitTestCaseBuilder method, except that at the end it constructs instances of DynamicTestMethod rather than NUnitTestMethod.

That’s it! Well, not quite, I did promise that we could use have dynamic fixtures that make use of dynamic tests. But that involves a few changes; the code as presented above could be shipped as two separate plug-ins, but to get them to play well together means that we have to do a little bit more work, and the dynamic fixture will need some rejiggling. It would be nice if neither plug-in knows about each other (as we would want dynamic fixtures and dynamic tests to work with any other type of plug-in). I’ll do this in the next blog posting, but I’ll whet your appetite by showing the sort of user tests that I want to have:

Desired combined dynamic test and fixtures


using System;
using System.Collections.Generic;
using System.Text;

using NUnit.Framework;

using Taumuon.DynamicFixture;
using Taumuon.DynamicTests;

namespace Taumuon.TestDynamic
{
public class MyDynamicFixtureBuilder : IDynamicFixtureBuilder
{
public List<IDynamicFixture> GetUserFixtures()
{
List<IDynamicFixture> fixtures =
&nbspnew List<IDynamicFixture>(2);
fixtures.Add(new MyTestFixture1("instance1", 1));
fixtures.Add(new MyTestFixture1("instance2", 2));
return fixtures;
}

public Type FixtureType
{
get
{
return typeof(MyTestFixture1);
}
}
}

public class MyTestFixture1 : IDynamicFixture, IDynamicTestBuilder
{
private int numberOfDynamicTests = 0;
private string name = "default";

[DynamicTest]
public void MyTestMethod(string arg1)
{
Assert.AreEqual(arg1, "arg1");
}

public MyTestFixture1(string name, int numberOfDynamicTests)
{
this.name = name;
this.numberOfDynamicTests = numberOfDynamicTests;
}

#region IDynamicFixture Members

public string Name
{
get
{
return name;
}
}

#endregion

#region IDynamicTestBuilder Members

public void SetTestsToRun(IDynamicTestDetails setDetails)
{
for (int i = 0; i < numberOfDynamicTests; ++i)
{
setDetails.SetTestDetails("Test " + i.ToString(),
"MyTestMethod", "arg" + i.ToString());
}
}

#endregion
}
}

Another proposed NUnit feature.

This post will clarify the proposed NUnit feature mentioned in the last post, and also propose a new feature (right near the end).

I was thinking of another concrete example of where adding a new feature to NUnit, to allow a more flexible method to create TestFixtures (proposed via a TestFixtureProvider attribute), as mentioned in my last post, may be useful. First, a bit of background on how NUnit was being used:

In the past, I worked on the mapping of financial instruments from an-memory representation of a complex hierarchy of objects into a multi-line string for consumption by another product.

The test fixtures were broadly organised into product types (e.g. bond), with tests for the different conditions encountered by that product (e.g.buy/sell, expired, amortising, etc.)

It was chosen that the tests should be XML driven:

To act as documentation: as the mappings were complex it would provide concrete human-readable examples of the instrument states, and associated mappings.
Tests are more manageable – can generate XML automatically, to give creater coverate quicker.
Allows financial analysts to create additional tests – don’t have to know programming.

The original instrument was represented by COM objects, with the C# mapping code consuming these objects via interop. As COM is interface-driven, it was fairly trivial to create mock objects implementing these interfaces, for testing.

As the instrument was hierarchical, an XML representation was a natural fit. There were XML files for each test fixture, each containing nodes which contained the XML representation, and either the expected string output from the mapping or any expected exception.

e.g.

InstrumentTypeATests.xml


<TestFixture name="InstrumentATests">
  <Test name="TestACertainConfiguration">
    <Instrument>
      This is an XML representation of an instrument.
    </Instrument>
    <ExpectedMapping>
      An expected mapping string in a CData section.
    </ExpectedMapping>
  </Test>
</TestFixture>

This is where we’re getting to the point of this post. We want to run these XML tests in NUnit – there are a number of ways to do this.

One is to have a fixture, with a RunXmlTest method, that would load the XML file (with the filename either hardcoded into the file, or into the fixture constructor – remember we’ve currently got no way to specify constructor arguments, other than via a parameter as passed into the commandline?), then would iterate through the nodes, populating the mock objects, map these mock objects into their string representation, and check that mapping against the expected mapping as specified in the tests. Which is fine, except that we’ll have a single green bar per fixture/test xml file, which isn’t very useful when there are hundreds of test cases.

(forgive any poor syntax/noddy-errors – I can’t remember exactly what this looked like, but this should be enough for you to get the idea)


[TestFixture]
InstrumentATestFixtureOriginal : BaseXmlFixture
{
  [FixtureSetUp]
  public void FixtureSetup()
  {
    // Set the path as used by the base class.
    pathToXml = "InstrumentTypeATests.xml";

    // Base class method to load the XML.
    LoadXml(); // loads into xmlDocument member
  }

  [Test]
  public void RunXmlTest()
  {
    foreach( XmlNode node in xmlDocument )
    {
      RunTest(node.Name);
    }
  }
}

An improvement is to have a TestACertainConfiguration test corresponding with each Test node in the XML – so we get a green bar for each of the tests:


[TestFixture]
InstrumentATestFixtureNew : BaseXmlFixture
{
  [FixtureSetUp]
  public void FixtureSetup()
  {
    // Set the path as used by the base class.
    pathToXml = "InstrumentTypeATests.xml";

    // Base class method to load the XML.
    LoadXml(); // loads into xmlDocument member
  }

  [Test]
  public void RunXmlTest1()
  {
    RunTest("Test1");
  }

  [Test]
  public void RunXmlTest2()
  {
    RunTest("Test2");
  }
}

But only tests as specified in the C# NUnit test are executed.

The solution that I used for the above, as all of the functionality is in the base
classes, is to generate the classes InstrumentATestFixtureNew from using XSLT on the XML files to produce .cs files which are compiled into the build.

This worked nicely, it allowed any changes in the XML files to be taken into account in the tests, but still required a rebuild.

So, the desired behaviour is to allow new test files to be added into a folder somewhere, and the tests to be modified, and have test fixtures generated according to the data in the xml files, without a rebuild of the test project. This is where the new proposed NUnit features would be useful.

Method 1, using the proposed TestFixtureProvider, is to have a constructor on InstrumentATestFixtureOriginal taking the path to the XML file to load, and registering instances of this class with the NUnit framework (see last post for more details on the syntax). This would work OK, but would mean that all of the tests contained in the fixture would only register one red/green bar in the GUI.

Method 2 is a variation on method 1, in that we’d modify InstrumentATestFixtureOriginal to be constructed with the path to the XML, and also the name of the test node – i.e. we’d create a fixture for each test. This would work, but would look cluttered on the GUI. It may be that we could specify in the TestFixtureProviderAttribute how to group these fixtures on the GUI. But this still feels a bit hack-y.

Method 3 is to create the fixtures in memory dependant on the XML, using either CodeDOM or Reflection.Emit. We’d still need to use the proposed TestFixtureProvider property to get these test fixtures to be registered, but it would be the neatest solution. However, it would be a lot of work.

Method 4 is where we get to the second proposed new NUnit functionality. It’s basically to -somehow- implement method 3 but without having to generate the fixtures in code. What we’d like is for every time the base class’s RunTest method is called, it appears on the UI. Maybe a new method called “RegisterTest(string name)” could be added to NUnit to be called from RunTest.