11:00 - 11:30

• Automated Modularization of GUI Test Cases

  Test cases that drive an application under test via its graphical user interface (GUI) consist of sequences of steps that perform actions on, or verify the state of, the application user interface. Such tests can be hard to maintain, especially if they are not properly modularized—that is, common steps occur in many test cases, which can make test maintenance cumbersome and expensive. Performing modularization manually can take up considerable human effort. To address this, we present an automated approach for modularizing GUI test cases. Our approach consists of multiple phases. In the first phase, it analyzes individual test cases to partition test steps into candidate subroutines, based on how user-interface elements are accessed in the steps. This phase can analyze the test cases only or also leverage execution traces of the tests, which involves a cost-accuracy tradeoff. In the second phase, the technique compares candidate subroutines across test cases, and refines them to compute the final set of subroutines. In the last phase, it creates callable subroutines, with parameterized data and control flow, and refactors the original tests to call the subroutines with context-specific data and control parameters. Our empirical results, collected using open-source applications, illustrate the effectiveness of the approach.

   

  Preprint available:

Rahulkrishna Yandrapally; Giriprasad Sridhara; Saurabh Sinha

IBM Research, India

11:30 - 12:00

• Making System User Interactive Tests Repeatable: When and What Should We Control?

  System testing and invariant detection is usually conducted from the user interface perspective when the goal is to evaluate the behavior of an application as a whole. A large number of tools and techniques have been developed to generate and automate this process, many of which have been evaluated in the literature or internally within companies. Typical metrics for determining effectiveness of these techniques include code coverage and fault detection, however, with the assumption that there is determinism in the resulting outputs. In this paper we examine the extent to which a common set of factors such as the system platform, Java version, application starting state and tool harness configurations impact these metrics. We examine three layers of testing outputs: the code layer, the behavioral (or invariant) layer and the external (or user interaction) layer. In a study using five open source applications across three operating system platforms, manipulating several factors, we observe as many as 184 lines of code coverage difference between runs using the same test cases, and up to 96 percent false positives with respect to fault detection. We also see some a small variation among the invariants inferred. Despite our best efforts, we can reduce, but not completely eliminate all possible variation in the output. We use our findings to provide a set of best practices that should lead to better consistency and smaller differences in test outcomes, allowing more repeatable and reliable testing and experimentation.

   

  Preprint available:

Zebao Gao; Yalan Liang; Myra Cohen; Atif Memon; Zhen Wang

University of Maryland, USA; University of Nebraska-Lincoln, USA 

12:00 - 12:30

• ZoomIn: Discovering Failures by Detecting Wrong Assertions

  Automatic testing, although useful, is still quite ineffective against faults that do not cause crashes or uncaught exceptions. In the majority of the cases automatic tests do not include oracles, and only in some cases they incorporate assertions that encode the observed behavior instead of the intended behavior, that is if the application under test produces a wrong result, the synthesized assertions will encode wrong expectations that match the actual behavior of the application. In this paper we present Zoom In, a technique that extends the fault-revealing capability of test case generation techniques from crash-only faults to faults that require non-trivial oracles to be detected. Zoom In exploits the knowledge encoded in the manual tests written by developers and the similarity between executions to automatically determine an extremely small set of suspicious assertions that are likely wrong and thus worth manual inspection. Early empirical results show that Zoom In has been able to detect 50% of the analyzed non-crashing faults in the Apache Commons Math library requiring the inspection of less than 1.5% of the assertions automatically generated by EvoSuite.

   

  Preprint available:

Fabrizio Pastore; Leonardo Mariani

University of Luxembourg, Luxembourg; University of Milano Bicocca, Italy