Code Coverage: Myth vs Reality

Through our work with developers and development teams over the years, we have found that there are certain myths about code coverage that still exist today. In an effort to ensure that your development and QA teams are all working towards the same goal, we wanted to both share and dispel a few of them. Here are some of the most common ones:

Myth #1 – There is a perfect code coverage score

We often get asked “what is the right code coverage number for our team?” That question implies that there is a “perfect” coverage code number, which is our first myth. Although it would be nice, coverage scores can range across the spectrum and are determined through your specific testing strategy, the critical path of our your code and the overall complexity of your code. In fact, the way you structure your modules and write your actual code impacts your coverage results. Although there isn’t a single perfect score, we invite you to watch best practices webinar for suggestions on how to achieve coverage results that work best for your organization.

Myth #2  – All code coverage metrics are the same

Earlier this week we had a prospect share with us, “we already have really high code coverage so we know our code is in good shape.” Although it’s great that they are measuring coverage, this leads to our second myth which is that all code coverage metrics are created equal. In this particular instance, the prospect was measuring line coverage. Line coverage and other metrics like method coverage fall into the category of metrics you can track but may not be particularly useful. In .NET, there may several sequence points in a single line, each with their own implications on the code. Does exercising any part of the line mean that you have full coverage?  In a word, “no.”  Not all code coverage metrics are created equal. We encourage you to understand your metrics and make decisions based on information rich metrics when it comes to the health of your code. You can find a quick refresher on code coverage metrics over here.

Myth #3 – 100% code coverage is 100% bug-free

Don’t get us wrong, if you have 100% branch coverage, you can feel pretty good that you are on the right path. In fact, we salute you!  However, our final myth, is exactly that, a myth. Remember, tests measure the quality of your code and code coverage measures the quality of your tests….as designed. Shakespeare referred to beauty as being in the eye of the beholder. As developers, we may feel that bugs are in the eye of the beholder. Is it performing as designed but not as the customer expects or wants? Are there new environmental or usage scenarios that didn’t exist when you published? Is the latest version of a web browser causing your web app to fail? These are questions that every development team faces sooner or later and although code coverage alone can’t solve these problems, it can help you ensure that as you engineer new solutions, those solutions perform as expected.

Understanding metrics like branch coverage, sequence point coverage, change risk anti-pattern score and how they fit into your testing strategy are all important in busting these myths and increasing both your confidence as developers and the quality of your code.