Rawad Abou Assi, Wes Masri, and Fadi A. Zaraket, American University of Beirut
The goal of regression testing is to ensure that the behavior of existing code, believed correct by previous testing, is not altered by new program changes. We argue that the primary focus of regression testing should be on code associated with:
Existing coverage criteria do not enable such focus, e.g., 100% branch coverage does not guarantee that a given bug fix is exercised or a given application scenario is tested. Therefore, there is a need for a new and complementary coverage criterion in which the user can define a test requirement characterizing a given behavior to be covered as opposed to choosing from a pool of pre-defined and generic program elements. We propose this new methodology and call it UCov, a user-defined coverage criterion wherein a test requirement is an execution pattern of program elements, and possibly predicates, that a test case must satisfy. Our proposed criterion is not meant to replace existing criteria, but to complement them as it focuses the testing on important code patterns that could go untested otherwise.
UCov supports test case intent verification. For example, following a bug fix, the testing team may augment the regression suite with the test case that revealed the bug. However, this test case might become obsolete due to code modifications not related to the bug. But if a test requirement characterizing the bug was defined by the user, UCov would determine that test case intent verification failed. It is also worth mentioning that our methodology paves the way for test case intent preservation, e.g., a failed verification could be followed by automated test case generation, the subject of future work.
We implemented our methodology for the Java platform and successfully applied it onto nine real life case studies. Our implementation comprises the following:
UCov comes with a suite of supporting tools: TRSpec Eclipse plug-in, Command line TRMatcher, and TRMigrate.
The TRSpec Eclipse plug-in is available here. To install it, extract it into the Eclipse dropins folder.
The rest of the tools (TRCheck and TRMigrate) and the case studies are packed under one compressed folder ( link to tools and case studies).
TRSpec depends on Zest, which you may obtain from (the Eclipse website). From within Eclipse, go to the menu item Help > Install new software…, use http://download.eclipse.org/tools/gef/updates/releases/ (GEF 3.x / Zest 1.x Releases) as a site and install both “Graphical Editing Framework Zest Visualization Toolkit” and “Graphical Editing Framework Zest Visualization Toolkit SDK” toolkits.
We tested our tools with JDK 1.7.
A shell script is provided for each of the case studies reported in the paper. It builds and runs the case study with the given parameters, depending on the case study.
Invoke it as follows.
./run-case-study-name.sh [case-study-parameters]
Note: the input files provided in the “case-studies” subdirectory were generated by the TRSpec plugin.