Covering MSTest in Visual Studio 2012

An earlier NCover blog post described how to make coverage in VS 2010 easier by changing the VS test settings.

With VS 2012, there’s a new coverage wrinkle — VS 2012 starts a new test execution process called “vstest.executionengine”. This takes the form of the “vstest.executionengine.x86.exe” or “vstest.executionengine.exe” process when the IDE launches.

This test engine process takes the place of qtagent in the older versions of MSTest. Still, it presents the same coverage obstacles as previous editions of VS. When a process you’re trying to cover has already started, the NCover profiler can’t attach to it, so coverage can’t be collected.

Unlike VS 2008, there’s no setting in the VS 2012 GUI (that we’ve found so far) to turn off the test execution engine between test runs.

The current best solution, which we honed while working with one of our VS 2012 extension users, is as follows:

1. Create a new project, and start Auto-Configure before launching VS 2012

2. Start VS 2012, and Auto-config will detect the expected modules: the dev.env.exe, any VS plugins that are being loaded, etc.

3. Run any MSTest unit test from VS, and you’ll see the “vstest.executionengine.x86.exe” detected, along with your unit test DLLs.

4. Stop detection. Modify your match rules and filters as desired, and save your NCover project.

5. Re-start VS 2012. NCover is now ready to monitor the VS test engine process.

6. Run your MSTest unit test again, and NCover should start profiling the results.

7. Since NCover will continue profiling the vstest.executionengine unitl it stops, you have to stop the test engine by killing the process, or exiting VS 2012.

8. A good alternative to interrupting VS 2012 when your unit tests are done is the NCover snapshot feature.

When your VS unit tests are complete, simply click on the snapshot icon beside the project name.

Your execution doesn’t stop, but while it continues normally, a snapshot view of the current execution data is created.

While the snapshot is being assembled, the “Last Run” status for the project will change from the most recent time stamp to “Loading…”

Once the snapshot is complete, you can drill down to the Executions view to see the results. Snapshots are labeled as such in the execution caption, and their build ID corresponds to the execution that is still pending.

Snapshots are cumulative. If you take several over the course of coverage collection, the most recent snapshot will contain all the data of its predecessors, plus whatever new data has been aggregated since the last snap.

Our VS 20112 user found this was a simple and flexible solution to the problem of profiling MSTest in VS 2012 new test execution engine.

Let us know what works for you!