04:00 - 04:30

• Performance Analysis Using Subsuming Methods: An Industrial Case Study

  Large-scale object-oriented applications consist of tens of thousands of methods and exhibit highly complex runtime behaviour that is difficult to analyse for performance. Typical performance analysis approaches that aggregate performance measures in a method-centric manner result in thinly distributed costs and few easily identifiable optimisation opportunities. Subsuming methods analysis is a new approach that aggregates performance costs across repeated patterns of method calls that occur in the application's runtime behaviour. This allows automatic identification of patterns that are expensive and represent practical optimisation opportunities. To evaluate the practicality of this analysis with a real world large-scale object-oriented application we completed a case study with the developers of letterboxd.com — a social network website for movie goers. Using the results of the analysis we were able to rapidly implement changes resulting in a 54.8% reduction in CPU load and an 49.6% reduction in average response time.

   

  Preprint available:

David Maplesden; Karl von Randow; Ewan Tempero; John Hosking; John Grundy

University of Auckland, New Zealand; Cactuslab, New Zealand; Swinburne University of Technology, Australia

04:30 - 05:00

• An Industrial Case Study on the Automated Detection of Performance Regressions in Heterogeneous Environments

  A key goal of performance testing is the detection of performance degradations (i.e., Regressions) compared to previous releases. Prior research has proposed the automation of such analysis through the mining of historical performance data (e.g., CPU and memory usage) from prior test runs. Nevertheless, such research has had limited adoption in practice. Working with a large industrial performance testing lab, we noted that a major hurdle in the adoption of prior work (including our own work) is the incorrect assumption that prior tests are always executed in the same environment (i.e., Labs). All too often, tests are performed in heterogenous environments with each test being run in a possibly different lab with different hardware and software configurations. To make automated performance regression analysis techniques work in industry, we propose to model the global expected behaviour of a system as an ensemble (combination) of individual models, one for each successful previous test run (and hence configuration). The ensemble of models of prior test runs are used to flag performance deviations (e.g., CPU counters showing higher usage) in new tests. The deviations are then aggregated using simple voting or more advanced weighting to determine whether the counters really deviate from the expected behaviour or whether it was simply due to an environment-specific variation. Case studies on two open-source systems and a very large scale industrial application show that our weighting approach outperforms a state-of-the-art environment-agnostic approach. Feedback from practitioners who used our approach over a 4 year period (across several major versions) has been very positive.

   

  Preprint available:

King Chun Foo; Zhen Ming (Jack) Jiang; Bram Adams; Ahmed E. Hassan; Ying Zou; Parminder Flora

BlackBerry, Canada; York University, Canada; École Polytechnique de Montréal, Canada; Queen's University, Canada

05:00 - 05:30

• Industry Practices and Event Logging: Assessment of a Critical Software Development Process

  Practitioners widely recognize the importance of event logging for a variety of tasks, such as accounting, system measurements and troubleshooting. Nevertheless, in spite of the importance of the tasks based on the logs collected under real workload conditions, event logging lacks systematic design and implementation practices. The implementation of the logging mechanism strongly relies on the human expertise. This paper proposes a measurement study of event logging practices in a critical industrial domain. We assess a software development process at Selex ES, a leading Finmeccanica company in electronic and information solutions for critical systems. Our study combines source code analysis, inspection of around 2.3 millions log entries, and direct feedback from the development team to gain process-wide insights ranging from programming practices, logging objectives and issues impacting log analysis. The findings of our study were extremely valuable to prioritize event logging reengineering tasks at Selex ES.

Antonio Pecchia; Marcello Cinque; Gabriella Carrozza; Domenico Cotroneo

Critiware, Italy; Federico II University of Naples, Italy; Selex ES, Italy