Unit testing is important for code quality.  Most people don’t question this fact today.  There are lots of tools available to help you write unit tests.  But here’s the problem – you have to morph your design to make it testable.  I’m all for making my code better but I get annoyed when I have to modify a design just to fit the tools I’m using.  To me this is a bad sign.  Our tools are there to support our work.  If we’re modifying our code to make the tools work then we have things backwards.  When we’re designing our code we shouldn’t be focused on the tools we will use (IDEs, testing, build, etc).  That would be equivalent to designing our code with the limits of our database or communication infrastructure in mind.  While these will impact the implementation, they should not impact the design.  And yet unit testing, more often than not, requires that we design our code with testing in mind.

Mocking objects is a very common practice in unit testing.  It allows us to focus on what is specifically being tested without having to worry about setting up all the extra stuff.  There are many mocking frameworks available but the majority of them have the same limitations, just with different syntax.  Most mocking frameworks can only mock interfaces or extensible types.  Sealed and static types are out of the question.  Even more frustrating is that often the member(s) to be mocked must be public or internal and virtual (but not always).  Sealed and static types have very specific uses in design.  They identify classes that are either self-contained and/or non-extensible.  It is a design decision.  Unit testing with these types is difficult so the common approach is to either modify the design (bad) or use abstraction. 

A lot of code these days go overboard with abstraction.  It reminds me of the early database days when DBAs wanted to normalize everything.  Abstraction is important but it should be used only when it is needed.  Unit testing is not a need.  This is just one example of modifying design to meet the needs of the tools.  As an example take DateTime.Now.  If you need to be able to test code that uses this member then you either have to get tricky with date management or you have to abstract out the current date from your design.  Keep in mind that your production code would (probably) never need a time other than the current time and yet you abstract it out for testing purposes.  Some folks will argue that you wouldn’t hard code such a value anyway, you’d just pass it as a parameter but that is just moving the problem up the call hierarchy.  Somewhere along the way the time has to be specified.

In state-based testing it is generally necessary to expose property getters (and even setters) internally so the framework can access it.  This allows us to test the state of an object but not expose the members to the production code directly.  (We’re ignoring the whole discussion for and against state-based testing and domain development.)  This is a dirty hack.  We are once again modifying our design (albeit in a hidden manner) to allow for testing.  As an aside MSTest has an interesting approach using accessors to allow access to private members without using this hack.  Unfortunately though it is generally broken, hard to maintain and not recommended.

What is the problem with the current set of tests?  The problem is that they almost always use reflection as they evolved from the existing framework tools.  Reflection unto itself is slow but for testing it is acceptable.  What is a little harder though is working around the security model of .NET to get access to private members.  Even worse is that testing tools can enumerate objects, create types (or derive new types) and invoke members but only if the base type is “designed” properly.  This is hardly the fault of the tools but rather a consequence of their reliance on reflection.  It is my belief that it is time for unit testing frameworks and tools to evolve into the tools we really need.

How can testing tools evolve?  If I knew the answer I would have already shared it.  We can take a look at a few tools available today to get an idea of what could be though.  TypeMock and Moles approach mocking in an interesting way – they rewrite the code under test.  As an aside note that I’ve never used TypeMock as it is commercially available.  I have used Moles in limited scenarios but I intend to use it more going forward. 

Code rewriting is an old technique.  Traditionally it is slow and brings into question the validity of testing but the benefits are tremendous.  Using either of these tools we can stub out almost any code irrelevant of how it was designed.  These tools allow us to design our code the way we want to and still unit test them.  As an example we can use DateTime.Now without the wasteful abstraction and yet we can set the time in our tests.  Need to stub out a member that isn’t virtual?  No problem the rewriter can rewrite the method body.  This, to me, is the approach that has the best hopes of evolving testing but it is still a relative new, and difficult, task. 

There are two big issues with current rewriting tools (for me at any rate).  Firstly is performance.  Rewriting takes a while.  Unit tests should run quickly.  We can’t be rewriting code for every test.  Even with processors as fast as they are this would be too slow.  It might take runtime-level rewriting to get performance where it needs to be but once performance gets better then rewriting will become more feasible.

The other problem is configuration.  Today, at least for Moles, you have to be explicit about what you want stubbed.  For a handful of types this would be OK but as code gets more complex we don’t want to have huge lists to manage.  We need to have the same facilities available to us that mocking frameworks use today.  When the test starts the rewriter takes a look at what needs to be rewritten and does it on the fly.  Today rewriting happens beforehand and this is simply not flexible enough. 

The ideal test tool would allow us to test any code.  We can configure the tool to return stubbed values anywhere in the calling code, we can mock up objects to be returned from methods so we can track expectations, we can control when things should and shouldn’t be called and we can do it all at test time rather than at compile time.  The test framework cannot modify what gets called, only what happens when it gets called and what it returns.  Just like the mocking frameworks of today.

In summary unit testing and the tools that it uses, such as mocking, are critical for properly testing your code.  But today’s tools are using technologies that require too many sacrifices on our design.  Testing tools need to evolve to allow us to design our code the way it needs to be designed and the tools just adapt.  Code rewriting currently looks like a good way to go but it is still too early to be fully usable in reasonable size tests.  This is a challenge to all testing tools – revolution the testing landscape!!!  Create tools that adapt to our needs rather than the other way around.  The testing tool that can do that will become the clear winner of the next generation of tools.

Visual Studio 2010 is adding the ability to run tests in parallel.  If you have lots of tests to run and a multi-processor machine then this is a great feature.  It is discussed here.  There have been several posts on the forums about this feature not working but it actually does.  Here are the formal requirements for parallel test execution (you can read the gory details in the link).

1. Must be running on a multi-processor machine.
2. Must be running unit tests.  No other test category is supported.
3. Tests must be thread-safe.  Most unit tests are but tests that use shared resources like a database or the file system may have issues.
4. Data collections are not allowed. 
5. Tests must be run locally only.
6. You must enable the option.  There is no user interface for setting it.

Here is a sample test file that I used to test parallel execution.  It consists of 4 test cases where each test case sleeps for 5 seconds.  When running sequentially this test should take approximately 20 seconds but when running in parallel (assuming 2 processors) it should only take 10. 

Here are steps you need to take to get parallel tests to run.  Note that since tests must be run locally you will be updating the local test settings.

1. Open your test project and double click on the local test settings file (Local.testsettings).
2. In the Test Settings dialog go to the Data and Diagnostics section and ensure that all the options are unchecked.  This will disable the data collection.

   

3. Close the Test Settings dialog.  Now you need to enable parallel test execution.  There is no UI for this so do the following.
   1. Right-click the test settings file in Solution Explorer, select Open With and then XML Editor to open the file in the XML editor.
   2. Go to the Execution element and add the parallel attribute.  You can set it to a specific number of processors or to 0 to allow the tests to run on all processors.

      

   3. Save and close the file.
4. Close the solution and reopen it.  The parallel settings are only read when the project loads, at least for Beta 2 and the RC.
5. Open the Test Results window and group by Result so you can see the parallel execution.
6. Start debugging (F5 or via the menu).
7. Open the Test Results window again and you should see the tests running parallel.

   

A couple of caveats about parallel tests.  Firstly you must close and reopen the solution (or at least the project) in order for changes in the parallel settings to take effect.  Secondly the settings file is rewritten whenever you make changes in the Test Settings dialog.  So if you make changes to the test settings through the UI you will need to modify the settings file manually to get the parallel settings back again.

Parallel test execution is a really neat idea for running unit tests quickly.  Hopefully Microsoft will move the option into the UI so we do not have to go through the manual editing, reload process each time.  Whether that happens before the final release of VS 2010 or not is anyone’s guess.