Test Coverage tools

SD supplies test (or code) coverage tools for arbitrary procedural languages, including C, C++, Java, C#, PHP, and many more. Such tools provide statistics and detail information about which parts of an application program have been executed (usually by a test suite). This information is invaluable to software teams to help determine the readiness of software for actual use. The type of coverage information collected is branch coverage, which subsumes statement coverage.

Test coverage tools may also be used to locate application functionality. One simply exercises the functionality of interest, and the test coverage tool indicates what part of the application code is executed. This a a very effective way to locate functionality in a large, poorly understood system.

SD's test coverage tools operate by inserting language-specific probes for each basic block in the source files of interest before compilation/execution. At execution time, the probes record which blocks get executed ("coverage data"). On completion of execution, the coverage data is typically written to a test coverage vector file. Finally, the test coverage data is used to generate a detailed test coverage report (huge Java example), or can be displayed on top of browsable source text (FORTRAN example below) for the system under test, enabling a test engineer to see what code has (green) or has not (red) been executed.

F90 Test Coverage Display screen shot

Aside: There are tools from other vendors that instrument binaries or class files rather than source. Such tools typically don't give answers that are as precise as those of source code instrumentation and this causes grief for test engineers (see this Stack Overflow complaint about instrumenting Java class files). The basic reason for imprecision from binary/classfile instrumentation tools is that the binary code simply does not have the same shape as the source code, loses information, and often contains additional artifacts introduced by compilation. Not-the-same shape comes from optimizing away code (that is logically executed even if it is optimized away), and splitting/shuffling source code blocks. Most binary files have at best source-code-line resolution; column positions are lost. Yet in source code often multiple conditionals happen within single source lines. Newly introduced code artifacts (e.g., extra jump table entries on case statements) have no corresponding place in the source code, yet contribute (or not) to what has been covered, in ways that are completely opaque to people looking the source code. The differences between binary and source may seem small, until you a well into testing and tricky question about why some modules aren't getting covered well come up. Bottom line: source code instrumentation produces better, less confusing answers.

Typical Features

  • Not dependent on any particular compiler or object formats
  • Works with tens of thousands of files
  • Very low probe overhead (one or two machine instructions per executed probe)
  • Can accumulate coverage data from multiple execution runs
  • Browsable source files overlayed with collected coverage information
  • Produces coverage report by application, subsystem and file
  • Test Coverage delta computation and display, to enable test case minimization
  • Can operate with custom/embedded application execution environments
  • Consistent style and operation across different languages
  • Probe installer and display tool operate on Windows 2000/XP/Vista/7/8
  • Application under test can execute in arbitrary native enviroment including embedded WinCE
  • Test coverage data from multiple subsystems can be combined into single comprehensive display and summary reports
  • Test coverage data from multiple languages can be combined into single comprehensive display and summary reports
  • Incremental test coverage (C#, C, C++, Java), telling which tests need to be re-run when source code base changes

Available for the following languages

Semantic Designs also provides profiling tools.

Download an evaluation copy

Enhancing test coverage by removing dead or redundant code

A key question for most testing organizations is, "how to get the coverage higher?" The usual answer is "write more tests", which is clearly a lot of work. Another way that takes less effort and has other quality payoff is to remove dead or redundant code. SD provides tools to complement test coverage tools to help with that task.

Thumbs up to Test Coverage tool

"LSI has integrated Semantic Designs' Test Coverage tool into its special firmware build to measure effectiveness of its internal testing. Fantastic support from Sematic Designs and the simplicity of the tool ensured we were able to integrate the tool in our process in just couple of days. Great thing about this tool is, IT JUST WORKS as expected and as advertised. We are already seeing measurable benefits of using the tool. Today, we have quantifiable data to assess our testing methodologies and track the improvements."
-- Abhijeet Aphale, Lead Engineer, LSI Corporation

Unusual Requirements?

Your language not listed, runs in an unusual environment, or you have some custom need? SD can configure a test coverage tool for you! These test coverage tools are based on DMS, and inherit DMS's language agility and scalability. A white paper on how test coverage is implemented with DMS is available: TestCoverage.pdf

For more information: info@semanticdesigns.com    Follow us at Twitter: @SemanticDesigns

Test Coverage Tools