Most performance engineering approaches focus on understanding the use of runtime resources. However such approaches do not quantify the value being provided in return for the consumption of these resources. Without such a measure it is not possible to compare the efficiency of these components (that is whether the runtime cost is reasonable given the benefit being provided). We have created an empirical approach that measures the value being provided by a code path in terms of the visible data it generates for the rest of the application. Combining this with traditional performance cost data, creates an efficiency measure for every code path in the application. We have evaluated our approach using the DaCapo benchmark suite, demonstrating our analysis allows us to quantify the efficiency of the code in each benchmark and find real optimisation opportunities, providing improvements of up to 36% in our case studies.

Large-scale object-oriented applications consist of tens of thousands of methods and exhibit highly complex runtime behaviour that is difficult to analyse for performance. Typical performance analysis approaches that aggregate performance measures in a method-centric manner result in thinly distributed costs and few easily identifiable optimisation opportunities. Subsuming methods analysis is a new approach that aggregates performance costs across repeated patterns of method calls that occur in the application's runtime behaviour. This allows automatic identification of patterns that are expensive and represent practical optimisation opportunities. To evaluate the practicality of this analysis with a real world large-scale object-oriented application we completed a case study with the developers of letterboxd.com - a social network website for movie goers. Using the results of the analysis we were able to rapidly implement changes resulting in a 54.8% reduction in CPU load and an 49.6% reduction in average response time.

The majority of existing application profiling techniques aggregate and report performance costs by method or calling context. Modern large-scale object-oriented applications consist of thousands of methods with complex calling patterns. Consequently, when profiled, their performance costs tend to be thinly distributed across many thousands of locations with few easily identifiable optimisation opportunities.

Performance is a crucial attribute for most software, making performance analysis an important software engineering task. The difficulty is that modern applications are challenging to analyse for performance. Many profiling techniques used in real-world software development struggle to provide useful results when applied to large-scale object-oriented applications. There is a substantial body of research into software performance generally but currently there exists no survey of this research that would help identify approaches useful for object-oriented software. To provide such a review we performed a systematic mapping study of empirical performance analysis approaches that are applicable to object-oriented software. Using keyword searches against leading software engineering research databases and manual searches of relevant venues we identified over 5,000 related articles published since January 2000. From these we systematically selected 253 applicable articles and categorised them according to ten facets that capture the intent, implementation and evaluation of the approaches. Our mapping study results allow us to highlight the main contributions of the existing literature and identify areas where there are interesting opportunities. We also find that, despite the research including approaches specifically aimed at object-oriented software, there are significant challenges in providing actionable feedback on the performance of large-scale object-oriented applications.