P3.NET

Parallel Test Execution in Visual Studio 2010

Visual Studio 2010 is adding the ability to run tests in parallel.  If you have lots of tests to run and a multi-processor machine then this is a great feature.  It is discussed here.  There have been several posts on the forums about this feature not working but it actually does.  Here are the formal requirements for parallel test execution (you can read the gory details in the link).

  1. Must be running on a multi-processor machine.
  2. Must be running unit tests.  No other test category is supported.
  3. Tests must be thread-safe.  Most unit tests are but tests that use shared resources like a database or the file system may have issues.
  4. Data collections are not allowed. 
  5. Tests must be run locally only.
  6. You must enable the option.  There is no user interface for setting it.

Here is a sample test file that I used to test parallel execution.  It consists of 4 test cases where each test case sleeps for 5 seconds.  When running sequentially this test should take approximately 20 seconds but when running in parallel (assuming 2 processors) it should only take 10. 

[TestMethod] 
public void TestMethod1 () 

   Thread.Sleep(5000); 

[TestMethod] 
public void TestMethod2 () 

   Thread.Sleep(5000); 

[TestMethod] 
public void TestMethod3 () 

   Thread.Sleep(5000); 

[TestMethod] 
public void TestMethod4 () 

   Thread.Sleep(5000); 
}

Here are steps you need to take to get parallel tests to run.  Note that since tests must be run locally you will be updating the local test settings.

  1. Open your test project and double click on the local test settings file (Local.testsettings).
  2. In the Test Settings dialog go to the Data and Diagnostics section and ensure that all the options are unchecked.  This will disable the data collection.

  3. Close the Test Settings dialog.  Now you need to enable parallel test execution.  There is no UI for this so do the following.
    1. Right-click the test settings file in Solution Explorer, select Open With and then XML Editor to open the file in the XML editor.
    2. Go to the Execution element and add the parallel attribute.  You can set it to a specific number of processors or to 0 to allow the tests to run on all processors.

    3. Save and close the file.
  4. Close the solution and reopen it.  The parallel settings are only read when the project loads, at least for Beta 2 and the RC.
  5. Open the Test Results window and group by Result so you can see the parallel execution.
  6. Start debugging (F5 or via the menu).
  7. Open the Test Results window again and you should see the tests running parallel.

A couple of caveats about parallel tests.  Firstly you must close and reopen the solution (or at least the project) in order for changes in the parallel settings to take effect.  Secondly the settings file is rewritten whenever you make changes in the Test Settings dialog.  So if you make changes to the test settings through the UI you will need to modify the settings file manually to get the parallel settings back again.

Parallel test execution is a really neat idea for running unit tests quickly.  Hopefully Microsoft will move the option into the UI so we do not have to go through the manual editing, reload process each time.  Whether that happens before the final release of VS 2010 or not is anyone’s guess.