(Edit: There is now a build NUnit add-in supporting this post, hosted at rakija)
Following my earlier blog posts, I posted a feature request to the NUnit list on Sourceforge, and NUnit maintainer Charlie Poole replied saying that I should investigate whether I could get the functionality I want using the new Extensibility features in NUnit 2.4.
Many thanks to Charlie Poole for giving me an insight into how Extensibility works, into the inner workings of NUnit, and for giving me guidance in creating these plug-ins. Any mistakes in this posting are purely my own. There are two problems that these solve - creating fixtures dynamically
(parameterized fixtures), and creating tests dynamically (parameterized tests). This can be combined to give dynamic fixtures that contain dynamic tests. The dynamic fixtures and tests that are loaded are specified at user assembly load time. This will be clearer below.
The add-in code can either be build into an assembly to live in an addins sub-folder under the NUnit install location, or can be supplied in the actual user assembly that is loaded into NUnit (primarily for quick development of add-in code).
The terminology for what I was originally thought in my head were called test fixtures, as are the classes present in the user assembly decorated with the [TestFixture] attribute, are known as user fixtures, to distinguish them from NUNit’s internal TestFixture classes.
The code below is built against NUnit 2.4 beta 1 (build 2.3.6162).
Dynamic fixtures
I’ll show how the tests make use of the plug-ins first, then show how the plug-ins are implemented.
We want to have a user fixture that takes a constructor, and create many instances of these with different constructor arguments. We expect the test TestConstructorValue1 to pass when we construct it with an int value of 1, and
similar for the other methods.
The string in the constructor will be used to give a name to this instance of the user fixture.
public class MyTestFixture1 : IDynamicFixture
{
private int value = 0;
private bool fixtureSetupCalled = false;
private string name = "default";
public MyTestFixture1(string name, int value)
{
this.name = name;
this.value = value;
}
[Test]
public void TestConstructorValue1()
{
Assert.IsTrue(value == 1);
}
[Test]
public void TestConstructorValue2()
{
Assert.IsTrue(value == 2);
}
[Test]
public void TestConstructorValue3()
{
Assert.IsTrue(value == 3);
}
[Test]
public void TestFixtureSetupCalled()
{
Assert.IsTrue(fixtureSetupCalled);
}
[TestFixtureSetUp]
public void FixtureSetup()
{
fixtureSetupCalled = true;
}
#region IDynamicFixture Members
public string Name
{
get
{
return name;
}
}
#endregion
}This class isn’t decorated with the TestFixture attribute, to distinguish it from standard NUnit user fixtures. But otherwise it behaves the same. It has Test methods with the same attributes. It implements the IDynamicFixture interface, defined in the plug-in, that has a name property to provide a name for this user fixture in NUnit’s tree view.
Plug-in code
public interface IDynamicFixture
{
string Name
{
get;
}
} Now, we need a way for instances of this class, our user fixtures, to be created and provided to NUnit. We do this by having a class that implements an interface, IDynamicFixtureBuilder. This class is created by NUnit on loading the assembly, and the GetUserFixtures() method called to register our user fixtures with NUnit:
public class MyDynamicFixtureBuilder : IDynamicFixtureBuilder
{
public List<IDynamicFixture> GetUserFixtures()
{
List<IDynamicFixture> fixtures = new List<IDynamicFixture>(3);
fixtures.Add(new MyTestFixture1("instance1", 1));
fixtures.Add(new MyTestFixture1("instance2", 2));
fixtures.Add(new MyTestFixture1("instance3", 3));
return fixtures;
}
public Type FixtureType
{
get
{
return typeof(MyTestFixture1);
}
}
} We can also have normal NUnit user fixtures decorated by the TestFixture attribute in the user assembly. The plug-in code defines the interface that will be implemented and used to create the user fixtures:
public interface IDynamicFixtureBuilder
{
List<IDynamicFixture> GetUserFixtures();
Type FixtureType
{
get;
}
} Now, we need to implement a SuiteBuilder that will create our user fixtures. NUnit loads our assembly, and for each type found in the assembly that it loads, it passes the type to two methods on the ISuiteBuilder interface, BuildFrom, and CanBuildFrom, to see whether that builder wants to create a suite from that type. In the normal case, a NUnit user fixture is decorated by the TestFixture attribute, and the NUNitTestFixtureBuilder informs NUNit that it can build that type, and builds it. We want to have our MyDynamicFixtureBuilder type recognised - we want to create the instances of MyTestFixture1 ourselves.
[SuiteBuilder]
public class MyFixtureBuilder : NUnitTestFixtureBuilder
{
#region ISuiteBuilder Members
public override TestSuite BuildFrom(Type type)
{
if (CanBuildFrom(type))
{
IDynamicFixtureBuilder dynamicFixtureBuilder = (IDynamicFixtureBuilder)Reflect.Construct(type);
TestSuite suite = new TestSuite(dynamicFixtureBuilder.FixtureType);
// Iterate over our user fixtures.
foreach (IDynamicFixture userFixture in dynamicFixtureBuilder.GetUserFixtures())
{
TestFixture fixture = new ParameterizedTestFixture(userFixture);
fixture.TestName.Name = userFixture.Name;
IList methods = GetCandidateTestMethods(userFixture.GetType());
foreach (MethodInfo method in methods)
{
TestCase testCase = TestCaseBuilder.Make(method);
if (testCase != null)
{
fixture.Add(testCase);
}
}
suite.Add(fixture);
}
return suite;
}
return null;
}
public override bool CanBuildFrom(Type type)
{
if (Reflect.HasInterface(type, "Taumuon.DynamicFixture.IDynamicFixtureBuilder"))
{
return true;
}
return false;
}
#endregion
} Our CanBuildFrom method just checks for types implementing our IDynamicFixtureBuilder interface. The BuildFrom method creates an instance of this type, then it creates a TestSuite passing in the fixture type profided from the IDynamicFixtureBuilder - this is the actual type of the user fixtures. It calls GetUserFixtures(), which is where we construct our parameterized user fixtures.In the loop I create a ParameterizedTestFixture, and use the GetCandidateTestMethods method on the base NUnitTestFixtureBuilder class to get methods of the correct signature, and the TestCaseBuilder.Make() creates standard NUnit tests (as they implement the [Test] attribute).
These tests are added to the fixtures, and the fixtures are added to the TestSuite.
The purpose of the ParameterizedTestFixture is to keep the instance of our
user fixure, as the NUNitTestFixture implementation of DoOneTimeSetup instantiates new instances of the user fixtures from the FixtureType. We want to keep an instance of our non-default-constructor constructed types around so that we can call their test methods.
public class ParameterizedTestFixture : NUnitTestFixture
{
private object userFixture;
public ParameterizedTestFixture(object userFixture)
: base(userFixture.GetType())
{
this.userFixture = userFixture;
}
protected override void DoOneTimeSetUp(TestResult suiteResult)
{
// We assign the user fixture here,
// so the correct instance is invoked on when running the test.
this.Fixture = userFixture;
base.DoOneTimeSetUp(suiteResult);
}
} Dynamic tests (parameterized test cases)
The idea behind dynamic tests is to have a fixture that is something like the following. The fixture can have normal [Test] decorated NUnit tests, but the plug-in has a new attribute [DynamicTest], to indicate a test that takes parameters. The user fixture implements the IDynamicTestBuilder interface to indicate that this method contains dynamic tests, and this interface contains the SetTestsToRun method, as obviously NUnit wouldn’t know what arguments to call the test method with. The SetTestsToRun() method is called by the plug-in, passing in an object implementing IDynamicTestDetails. Our user fixture calls on this interface, specifing for each call of the DynamicTest method, what name it should appear with in the NUnit tree view, the name of the actual method to call, and the arguments to pass into the method.
public class MyUserFixure : IDynamicTestBuilder
{
[DynamicTest]
public void MyTestMethod(string arg1)
{
Assert.AreEqual(arg1, "arg1");
}
[Test]
public void NormalTestMethod()
{
Assert.IsTrue(true, "This is a normal NUnit test");
}
#region IDynamicTestBuilder Members
public void SetTestsToRun(IDynamicTestDetails setDetails)
{
setDetails.SetTestDetails("Test 1", "MyTestMethod", "arg1");
setDetails.SetTestDetails("Test 2", "MyTestMethod", "arg2");
}
#endregion
} Plug-in code
The plug-in code contains the definition of our DynamicTest attribute, and our two interfaces:
[AttributeUsage(AttributeTargets.Method, Inherited=false, AllowMultiple=false)]
public class DynamicTestAttribute : Attribute
{
}
public interface IDynamicTestDetails
{
void SetTestDetails(string testName, string testMethodName, params object[] parameters);
}
public interface IDynamicTestBuilder
{
void SetTestsToRun(IDynamicTestDetails details);
} Similar to the Dynamic Fixtures, we want to provide a SuiteBuilder that will handle adding our custom user fixtures containing dynamic tests into the system.
[SuiteBuilder]
public class MyDynamicTestFixtureBuilder : NUnitTestFixtureBuilder, IDynamicTestDetails
{
private TestSuite fixture = null;
public override TestSuite BuildFrom(Type type)
{
if (CanBuildFrom(type))
{
IDynamicTestBuilder testBuilder = (IDynamicTestBuilder)Reflect.Construct(type);
fixture = base.BuildFrom(type);
// base.BuildFrom will have added a single instance
// of our test - it will have found the
// DynamicTestMethodBuilder and added the 'vanilla'
// (i.e. no parameter, test name set to
// the method name) version of the test method.
// We only want the user-added version of this method to be present.
for (int i = fixture.Tests.Count - 1; i >= 0; --i)
{
if (fixture.Tests[i] is DynamicTestMethod)
{
fixture.Tests.RemoveAt(i);
}
}
testBuilder.SetTestsToRun(this);
return fixture;
}
return null;
}
public override bool CanBuildFrom(Type type)
{
if (Reflect.HasInterface(type, "Taumuon.DynamicTests.IDynamicTestBuilder"))
{
return true;
}
return false;
}
... more code below!
} This just looks for types implementing the IDynamicTestBuilder interface, and calls on the base NUnitTestFixtureBuilder class to build our user fixture. This makes sure that all normal NUnit TestAttribute-decorated methods are included. This will also have detected our DynamicTest decorated method, and added an instance of this to our test, we’ll cover why this is in a bit when we look at our custom TestBuilder, but for now just see that we remove this DynamicTest from the fixture, as we want to add the versions we build ourselves, containing parameters.
We create our type implementing IDynamicTestBuilder from the type passed into the SuiteBuilder, and call SetTestsToRun on it, passing in this, as our SuiteBuilder implements IDynamicTestDetails. Here’s the implementation of this interface:
/// <summary>
/// TODO: exceptions etc.
/// </summary>
/// <param name="testName"></param>
/// <param name="testMethodName"></param>
/// <param name="parameters"></param>
public void SetTestDetails(string testName, string testMethodName, params object[] parameters)
{
if (null == fixture)
{
throw new ApplicationException("Test fixture is null");
}
// TODO: should cache some of this stuff.
MethodInfo mi = fixture.FixtureType.GetMethod(testMethodName);
ParameterInfo[] methodParameters = mi.GetParameters();
if (methodParameters.Length != parameters.Length)
{
throw new ApplicationException("Wrong number of parameters");
}
for (int i = 0; i < parameters.Length; ++i)
{
ParameterInfo pi = methodParameters[i];
if (pi.ParameterType != parameters[i].GetType())
{
throw new ApplicationException("Parameter is of the wrong type");
}
}
TestCase testCase = TestCaseBuilder.Make(mi);
testCase.TestName.Name = testName;
if (testCase is DynamicTestMethod)
{
((DynamicTestMethod)testCase).Parameters = parameters;
}
if (testCase != null)
{
fixture.Add(testCase);
}
}
#endregion
}The user fixture calls back into this method, specifying the testName, testMethodName, and any parameters (there’s not much point of this if there are no parameters). We get the MethodInfo from the user fixture, from the MethodName that they’ve supplied, check that our parameters array matches the parameters on the DynamicMethod.
Now, to build up the test case, we call TestCaseBuilder.Make() with the method info. This will call into our DynamicTestBuilder (code below), to create a DynamicTestMethod, it sets the parameters on it, and adds this to the user fixture.
This is our DynamicTestMethod:
public class DynamicTestMethod : NUnitTestMethod
{
private MethodInfo method;
#region Constructors
public DynamicTestMethod(MethodInfo method) : base(method)
{
this.method = method;
}
public DynamicTestMethod(MethodInfo method, Type expectedException, string expectedMessage,
string matchType)
: base(method, expectedException, expectedMessage, matchType)
{
this.method = method;
}
public DynamicTestMethod(MethodInfo method, string expectedExceptionName, string expectedMessage,
string matchType)
: base(method, expectedExceptionName, expectedMessage, matchType)
{
this.method = method;
}
#endregion
public override void RunTestMethod(TestCaseResult testResult)
{
// The method info is private in the base class,
// so have a member in this class.
// This is the same as Reflect.InvokeMethod,
// except that it invokes with arguments.
if (this.method != null)
{
try
{
this.method.Invoke(this.Fixture, parameters);
}
catch (TargetInvocationException e)
{
Exception inner = e.InnerException;
throw new NunitException("Rethrown", inner);
}
}
}
public object[] Parameters
{
get
{
return parameters;
}
set
{
parameters = value;
}
}
private object[] parameters;
} It overrides NUnitTestMethod, and the constructors are provided to capture a copy of the method MethodInfo, as this member is private in the base class. There is a Parameters array, where we can store the parameters that the user fixture specified should be used for this test instance. This overrides RunTestMethod - this is called from the the base class TestMethod.Run after the Setup method is called. The base class RunTestMethod isn’t much different from this, it just calls Reflect.InvokeMethod() that invokes with no parameters. we’ve just copied the guts of that method to here, invoking with the parameters set on our DynamicTestMethod.
This is the code for our DynamicTestMethodBuilder (I’ll introduce it gradually)
[TestCaseBuilder]
public class DynamicTestMethodBuilder : NUnitTestCaseBuilder
{
#region ITestCaseBuilder Members
public override bool CanBuildFrom(System.Reflection.MethodInfo method)
{
if (Reflect.HasAttribute(method, "Taumuon.DynamicTests.DynamicTestAttribute", false))
return true;
return false;
} CanBuildFrom just checks for methods decorated with our DynamicTest attribute.
// Duplication of AbstractTestMethodBuilder and
// NUnitTestMethodBuilder code, as HasValidTestCaseSignature
// isn't marked as virtual
public override NUnit.Core.TestCase BuildFrom(MethodInfo method)
{
// NUnit.Core.TestCase testCase = base.BuildFrom(method);
TestCase testCase = null;
#region From AbstractTestMethodBuilder
if (HasValidTestCaseSignature(method))
{
testCase = MakeTestCase(method);
string reason = null;
if (!IsRunnable(method, ref reason))
{
testCase.RunState = RunState.NotRunnable;
testCase.IgnoreReason = reason;
}
testCase.Description = GetTestCaseDescription(method);
SetTestProperties(method);
}
else
{
testCase = new NotRunnableTestCase(method);
}
#endregion From AbstractTestMethodBuilder
#region From NUnitTestCaseBuilder
if (testCase != null)
{
PlatformHelper helper = new PlatformHelper();
if (!helper.IsPlatformSupported(method))
{
testCase.RunState = RunState.Skipped;
testCase.IgnoreReason = helper.Reason;
}
testCase.Categories = CategoryManager.GetCategories(method);
testCase.IsExplicit = Reflect.HasAttribute(method, "NUnit.Framework.ExplicitAttribute", false);
if (testCase.IsExplicit)
testCase.RunState = RunState.Explicit;
System.Attribute[] attributes = Reflect.GetAttributes(method, "NUnit.Framework.PropertyAttribute", false);
foreach (Attribute propertyAttribute in attributes)
{
string name = (string)Reflect.GetPropertyValue(propertyAttribute, "Name",
BindingFlags.Public | BindingFlags.Instance);
if (name != null && name != string.Empty)
{
object value = Reflect.GetPropertyValue(propertyAttribute,
"Value", BindingFlags.Public | BindingFlags.Instance);
testCase.Properties[name] = value;
}
}
}
#endregion From NUnitTestCaseBuilder
return testCase;
} BuildFrom is pretty simple. It is only present as the base class’s HasValidTestCaseSignature isn’t marked as virtual (though this should change in a later NUnit build - I’ve got a feature request in for this). We have to copy the code out of both of the base class methods, just so that our own version of HasValidTestCaseSignature is called. Obviously, this will be much neater once that method is marked as abstract.
Here’s our HasValidTestCaseSignature method:
// Change over base class method is to remove requirement
// for number of parameters to be zero.
// TODO: note that AbstractTestCaseBuilder method should
// be marked as virtual.
protected new bool HasValidTestCaseSignature(MethodInfo method)
{
return !method.IsStatic
&& !method.IsAbstract
&& method.IsPublic
&& method.ReturnType.Equals(typeof(void));
} This method’s only difference over the base class method is to remove a check for the number of parameters to be zero (as obviously we’ve got parameterized tests).
The final method of this class is MakeTestCase.
protected override NUnit.Core.TestCase MakeTestCase(MethodInfo method)
{
Type expectedException = null;
string expectedExceptionName = null;
string expectedMessage = null;
string matchType = null;
Attribute attribute = Reflect.GetAttribute(method,
"NUnit.Framework.ExpectedExceptionAttribute", false);
if (attribute != null)
{
expectedException = Reflect.GetPropertyValue(attribute, "ExceptionType",
BindingFlags.Public | BindingFlags.Instance) as Type;
expectedExceptionName = (string)Reflect.GetPropertyValue(attribute, "ExceptionName",
BindingFlags.Public | BindingFlags.Instance) as String;
expectedMessage = (string)Reflect.GetPropertyValue(attribute, "ExpectedMessage",
BindingFlags.Public | BindingFlags.Instance) as String;
object matchEnum = Reflect.GetPropertyValue(attribute, "MatchType",
BindingFlags.Public | BindingFlags.Instance);
if (matchEnum != null)
matchType = matchEnum.ToString();
}
if (expectedException != null)
return new DynamicTestMethod(method, expectedException, expectedMessage, matchType);
else if (expectedExceptionName != null)
return new DynamicTestMethod(method, expectedExceptionName, expectedMessage, matchType);
else
return new DynamicTestMethod(method);
}
} The code for this method is identical to the base class NUnitTestCaseBuilder method, except that at the end it constructs instances of DynamicTestMethod rather than NUnitTestMethod.
That’s it! Well, not quite, I did promise that we could use have dynamic fixtures that make use of dynamic tests. But that involves a few changes; the code as presented above could be shipped as two separate plug-ins, but to get them to play well together means that we have to do a little bit more work, and the dynamic fixture will need some rejiggling. It would be nice if neither plug-in knows about each other (as we would want dynamic fixtures and dynamic tests to work with any other type of plug-in). I’ll do this in the next blog posting, but I’ll whet your appetite by showing the sort of user tests that I want to have:
Desired combined dynamic test and fixtures
using System;
using System.Collections.Generic;
using System.Text;
using NUnit.Framework;
using Taumuon.DynamicFixture;
using Taumuon.DynamicTests;
namespace Taumuon.TestDynamic
{
public class MyDynamicFixtureBuilder : IDynamicFixtureBuilder
{
public List<IDynamicFixture> GetUserFixtures()
{
List<IDynamicFixture> fixtures = new List<IDynamicFixture>(2);
fixtures.Add(new MyTestFixture1("instance1", 1));
fixtures.Add(new MyTestFixture1("instance2", 2));
return fixtures;
}
public Type FixtureType
{
get
{
return typeof(MyTestFixture1);
}
}
}
public class MyTestFixture1 : IDynamicFixture, IDynamicTestBuilder
{
private int numberOfDynamicTests = 0;
private string name = "default";
[DynamicTest]
public void MyTestMethod(string arg1)
{
Assert.AreEqual(arg1, "arg1");
}
public MyTestFixture1(string name, int numberOfDynamicTests)
{
this.name = name;
this.numberOfDynamicTests = numberOfDynamicTests;
}
#region IDynamicFixture Members
public string Name
{
get
{
return name;
}
}
#endregion
#region IDynamicTestBuilder Members
public void SetTestsToRun(IDynamicTestDetails setDetails)
{
for (int i = 0; i < numberOfDynamicTests; ++i)
{
setDetails.SetTestDetails("Test " + i.ToString(),
"MyTestMethod", "arg" + i.ToString());
}
}
#endregion
}
}