NCover Pre-Instrumentation

Introducing NCover Pre-Instrumentation

Version 5 of NCover introduces a pre-instrumentation option for collecting coverage data. Pre-instrumentation is a well-established paradigm used by many types of profilers and involves the insertion of coverage collection code into the assemblies of an application on-disk rather, than dynamically in-memory. Prior to version 5, NCover has always performed dynamic in-memory instrumentation. Now, NCover pre-instrumentation provides the ability to use the on-disk option.

In this blog post, we will provide an brief overview of NCover’s newest approach to collecting coverage and an introduction to the situations where you may benefit from NCover pre-instrumentation. Future blog posts will provide a more thorough and detailed approach of ways in which you can deploy NCover pre-instrumentation.  If you are ready to start today, please contact us and we would be happy to help your team get started with pre-instrumentation.

NCover Live Profiling & The .NET Profiling API

All previous versions of NCover, including the default configuration of version 5 of NCover, have relied upon the .NET profiling API. This API provides a window into the assembly loading, JIT-ing, and execution process. Historically, prior to version 4, NCover required command line invocation of the profiler. Beginning with Version 4, NCover introduced the automatic detection of applications which are to be profiled based on predefined match rules. Both of these methodologies leverage the same profiling API provided by .NET and are subject to the rules of engagement the API provides. In-memory instrumentation is especially convenient for capturing code coverage on strong-named assemblies without re-signing those assemblies. The .NET framework supports emitting code in memory, making this approach more accessible than re-writing the assembly to disk.

Special Situations

Using the API to profile an application is not always possible or practical. Certain scenarios encountered by today’s .NET teams present new obstacles to profiling and data collection. Newer flavors of .NET apps for Azure, Windows Phone, and Windows Store are becoming more mainstream. Their default CLR environment is quite different from traditional .NET architecture. All require specialized SDKs and run in a more limited context on development and testing machines. In fact, many scenarios introduce complete sandboxing of applications.  Sandboxing by definition is an obstacle to third party communication by design and becomes a near show stopper for remote profiler data collection.

Additional scenarios which are less than ideal for collecting coverage via the profiling API include object mocking frameworks, common to complex object model testing and the use of a variety of other profilers that are contending for the same CLR profiling interface. While modern versions of the .NET CLR allow us to chain profilers, profiler vendors must pay special attention and create a well thought-out and well-executed integration for chaining to function.

Process Optimization

Another reason to consider NCover pre-instrumentation relates to process optimization. Any time you attach a dynamic profiler to a process via the .NET profiling API, it is going to add overhead to the total process.  Quite simply, you are requiring more work to be done.  All profilers receive the same messages from the API regarding module, type loading and method JITing.  Depending on what the profiler does with the messages received in turn determines just how noticeable that increase in time becomes. If this profiling overhead is not acceptable in your current process or environment, the pre-instrumented methodology for coverage collection may be attractive.

How Pre-Instrumentation Works

Pre-instrumenting an assembly is as simple as executing the following command:

c:> ncover instrument myassembly.dll

or

c:> ncover instrument myassembly.dll myapp.exe myassembly2.dll

or

c:> ncover instrument *.dll *.exe

While it is worth noting that wildcards are available for this command, it is important to be judicious about instrumentation. Avoid instrumenting third party libraries and system assemblies. There is very limited value in collecting coverage on code you can’t change. Who really wants to keep that data around anyway? So, as an important reminder, always instrument only what is actionable. Selective pre-instrumentation becomes a form of pre-coverage filtering.

Pre-instrumentation is valid for both a single assembly in a solution or multiple assemblies, on a simultaneous basis, within a solution.  Through this approach, selective pre-instrumentation becomes an additional coverage filter allowing you to instrument only the assemblies for which coverage collection is needed. It is not necessary to pre-instrument an .exe file in order to collect coverage on a specific .dll file. Instrument only the .dll file and execute testing on the application as usual.

The execution of a pre-instrumented assembly collects coverage counters into a mapped file on disk while the application testing is underway. When the process exits, the binary coverage file remains on disk and can be imported via NCover Desktop, Code Central or Collector. These files carry the .ncprof extension and are raw coverage data. By making this file a raw binary counter file, the file remains valid for import even if your application crashes before you complete testing.

The coverage file follows a naming convention that includes the process name and a timestamp.  All pre-instrumented assemblies in the same process store their coverage counters in the same data file. This is similar to the same behavior of in-memory profiling, which places all of the coverage data in a single execution. Each .ncprof file represents a single execution of an pre-instrumented application.

Importing a coverage file is accomplished simply with the following command:

c:> ncover import --project=Project1 --file=MyAppName-[timestamp].ncprof

or

c:> ncover import --project=”My Coverage Project” --file=*.ncprof

When To Use NCover Pre-Instrumentation

Covering a Windows Store app is the most common scenario for using pre-instrumentation with NCover.  Pre-instrumentation is a good workaround anytime you encounter a conflict created by using multiple profilers concurrently. You may also consider using NCover pre-instrumentation when you are collecting coverage on machines with limited resources. By eliminating all messaging, which is the normal overhead of profiling in .NET, and by eliminating threads used for process monitoring during live profiling, you may experience a noticeable improvement in the performance of an application or test suite during coverage collection.