Code Coverage Metrics in Regulated Industries

Code Coverage Metrics in Regulated Industries

In a previous post, Code Coverage In The Medical Device Software Industry, we discussed the role a governing organization such as the U.S. Food and Drug Administration plays in providing guidance for software development companies in the medical device software industry. Specifically, we reviewed the implications of software validation and code coverage and the prevalence of software defects when changes were made after the initial introduction of the software product. We also discussed how those defects, and the costs and risks associated with them, can be reduced through a well-structured code coverage and software validation process.

Key Code Coverage Metrics

In this post, we will review section 5.2.5 of the General Principles of Software Validation as issued by the FDA and the specific metrics that can be used to ensure compliance.  This is not a full or exhaustive list, but rather a summary of three key coverage metrics presented by the FDA, how the FDA defines them, and how they can be analyzed when using NCover to collect code coverage on your .NET applications.

Statement Coverage (Sequence Point Coverage)

Statement Coverage – This criteria requires sufficient test cases for each program statement to be executed at least once; however, its achievement is insufficient to provide confidence in a software product’s behavior.

In NCover, Statement Coverage can be tracked for .NET applications with Sequence Point Coverage.  Sequence Point Coverage is a base code coverage metric that tracks each sequence point of code and calculates the percentage of sequence points covered during the testing of the application.  Although it does not take into consideration multiple branches or conditions, it is a particularly useful metric for tracking down specific portions of code that may be impacting overall coverage.

Decision (Branch) Coverage

Decision (Branch) Coverage – This criteria requires sufficient test cases for each program decision or branch to be executed so that each possible outcome occurs at least once. It is considered to be a minimum level of coverage for most software products, but decision coverage alone is insufficient for high-integrity applications.

The definition of Decision Coverage correlates with NCover’s Branch Coverage. Branch Coverage measures the fraction of independent code segments that were executed. Independent code segments are sections of code that have no branches into or out of them. Each independent code segment is a section of code that you would expect to execute in its entirety every time it is run. Branch coverage is one of several key metrics for determining how well the code base for a .NET application has been tested.

Condition Coverage

Condition Coverage – This criteria requires sufficient test cases for each condition in a program decision to take on all possible outcomes at least once. It differs from branch coverage only when multiple conditions must be evaluated to reach a decision.

Condition Coverage is another standard code coverage metric that was first added to NCover in Version 5. NCover added Condition Coverage to highlight scenarios when sets of branches can be followed, but not through all the possible conditions that lead to that branch. This is most helpful when a method is executed under coverage by multiple tests. These tests together may cover the entire method from a branch coverage perspective, but still not meet all conditions.