Is It Time for a Unit Test Evolution?

Unit testing is important for code quality.  Most people don’t question this fact today.  There are lots of tools available to help you write unit tests.  But here’s the problem – you have to morph your design to make it testable.  I’m all for making my code better but I get annoyed when I have to modify a design just to fit the tools I’m using.  To me this is a bad sign.  Our tools are there to support our work.  If we’re modifying our code to make the tools work then we have things backwards.  When we’re designing our code we shouldn’t be focused on the tools we will use (IDEs, testing, build, etc).  That would be equivalent to designing our code with the limits of our database or communication infrastructure in mind.  While these will impact the implementation, they should not impact the design.  And yet unit testing, more often than not, requires that we design our code with testing in mind.

Mocking objects is a very common practice in unit testing.  It allows us to focus on what is specifically being tested without having to worry about setting up all the extra stuff.  There are many mocking frameworks available but the majority of them have the same limitations, just with different syntax.  Most mocking frameworks can only mock interfaces or extensible types.  Sealed and static types are out of the question.  Even more frustrating is that often the member(s) to be mocked must be public or internal and virtual (but not always).  Sealed and static types have very specific uses in design.  They identify classes that are either self-contained and/or non-extensible.  It is a design decision.  Unit testing with these types is difficult so the common approach is to either modify the design (bad) or use abstraction. 

A lot of code these days go overboard with abstraction.  It reminds me of the early database days when DBAs wanted to normalize everything.  Abstraction is important but it should be used only when it is needed.  Unit testing is not a need.  This is just one example of modifying design to meet the needs of the tools.  As an example take DateTime.Now.  If you need to be able to test code that uses this member then you either have to get tricky with date management or you have to abstract out the current date from your design.  Keep in mind that your production code would (probably) never need a time other than the current time and yet you abstract it out for testing purposes.  Some folks will argue that you wouldn’t hard code such a value anyway, you’d just pass it as a parameter but that is just moving the problem up the call hierarchy.  Somewhere along the way the time has to be specified.

In state-based testing it is generally necessary to expose property getters (and even setters) internally so the framework can access it.  This allows us to test the state of an object but not expose the members to the production code directly.  (We’re ignoring the whole discussion for and against state-based testing and domain development.)  This is a dirty hack.  We are once again modifying our design (albeit in a hidden manner) to allow for testing.  As an aside MSTest has an interesting approach using accessors to allow access to private members without using this hack.  Unfortunately though it is generally broken, hard to maintain and not recommended.

What is the problem with the current set of tests?  The problem is that they almost always use reflection as they evolved from the existing framework tools.  Reflection unto itself is slow but for testing it is acceptable.  What is a little harder though is working around the security model of .NET to get access to private members.  Even worse is that testing tools can enumerate objects, create types (or derive new types) and invoke members but only if the base type is “designed” properly.  This is hardly the fault of the tools but rather a consequence of their reliance on reflection.  It is my belief that it is time for unit testing frameworks and tools to evolve into the tools we really need.

How can testing tools evolve?  If I knew the answer I would have already shared it.  We can take a look at a few tools available today to get an idea of what could be though.  TypeMock and Moles approach mocking in an interesting way – they rewrite the code under test.  As an aside note that I’ve never used TypeMock as it is commercially available.  I have used Moles in limited scenarios but I intend to use it more going forward. 

Code rewriting is an old technique.  Traditionally it is slow and brings into question the validity of testing but the benefits are tremendous.  Using either of these tools we can stub out almost any code irrelevant of how it was designed.  These tools allow us to design our code the way we want to and still unit test them.  As an example we can use DateTime.Now without the wasteful abstraction and yet we can set the time in our tests.  Need to stub out a member that isn’t virtual?  No problem the rewriter can rewrite the method body.  This, to me, is the approach that has the best hopes of evolving testing but it is still a relative new, and difficult, task. 

There are two big issues with current rewriting tools (for me at any rate).  Firstly is performance.  Rewriting takes a while.  Unit tests should run quickly.  We can’t be rewriting code for every test.  Even with processors as fast as they are this would be too slow.  It might take runtime-level rewriting to get performance where it needs to be but once performance gets better then rewriting will become more feasible.

The other problem is configuration.  Today, at least for Moles, you have to be explicit about what you want stubbed.  For a handful of types this would be OK but as code gets more complex we don’t want to have huge lists to manage.  We need to have the same facilities available to us that mocking frameworks use today.  When the test starts the rewriter takes a look at what needs to be rewritten and does it on the fly.  Today rewriting happens beforehand and this is simply not flexible enough. 

The ideal test tool would allow us to test any code.  We can configure the tool to return stubbed values anywhere in the calling code, we can mock up objects to be returned from methods so we can track expectations, we can control when things should and shouldn’t be called and we can do it all at test time rather than at compile time.  The test framework cannot modify what gets called, only what happens when it gets called and what it returns.  Just like the mocking frameworks of today.

In summary unit testing and the tools that it uses, such as mocking, are critical for properly testing your code.  But today’s tools are using technologies that require too many sacrifices on our design.  Testing tools need to evolve to allow us to design our code the way it needs to be designed and the tools just adapt.  Code rewriting currently looks like a good way to go but it is still too early to be fully usable in reasonable size tests.  This is a challenge to all testing tools – revolution the testing landscape!!!  Create tools that adapt to our needs rather than the other way around.  The testing tool that can do that will become the clear winner of the next generation of tools.


  1. Jonathan Allen

    The answer is, as always, to stop obsessing about unit tests. Learn to use the other forms of testing as well. Unit test your libraries, but look at the big picture when testing your application as a whole.