Performance (i.e., response time, throughput, resource consumption) is a key quality metric of today's applications as it heavily affects customer satisfaction. SAP strives to identify and fix performance problems before customers face them. Therefore, performance engineering methods are applied in all stages of the software lifecycle. However, especially in the development phase continuous performance evaluations can introduce a lot of overhead for developers which hinders their broad application in practice. In order to evaluate the performance of a certain software artefact (e.g. comparing two design alternatives), a developer has to run measurements that are tailored to the software artefact under test. The use of standard benchmarks would create less overhead, but the information gain is often not sufficient to answer the specific questions of developers. In this industrial paper, we present an approach that enables exhaustive, tailored performance testing with minimal effort for developers. The approach allows to define benchmark applications through a domain-specific model and realizes the transformation of those models to benchmark applications via a generic Benchmark Framework. The application of the approach in the context of the SAP Netweaver Cloud development environment demonstrated that we can efficiently identify performance problems that would not have been detected by our existing performance test infrastructure.

The performance of a system directly influences business critical metrics like total cost of ownership (TCO) and user satisfaction. However, building responsive, resource efficient and scalable applications is a challenging task. Thus, software engineering approaches are required to support software architects and developers in meeting these challenges. In this PhD research abstract, we propose a novel performance evaluation process applied during the software development phase. The goal is to increase the performance awareness of developers by providing feedback with respect to performance properties that is integrated in the every day development process. The feedback is based on domain-specific prediction functions derived by a generic methodology that executes a series of systematic measurements. We apply and validate the approach in different development scenarios at SAP.

Understanding the dependency between performance metrics (such as response time) and software configuration or usage parameters is crucial in improving software quality. However, the size of most modern systems makes it nearly impossible to provide a complete performance model. Hence, we focus on scenario-specific problems where software engineers require practical and efficient approaches to draw conclusions, and we propose an automated, measurement-based model inference method to derive goal-oriented performance prediction functions. For the practicability of the approach it is essential to derive functional dependencies with the least possible amount of data. In this paper, we present different strategies for automated improvement of the prediction model through an adaptive selection of new measurement points based on the accuracy of the prediction model. In order to derive the prediction models, we apply and compare different statistical methods. Finally, we evaluate the different combinations based on case studies using SAP and SPEC benchmarks.

Software performance engineering for enterprise applications is becoming more and more challenging as the size and complexity of software landscapes increases. Systems are built on powerful middleware platforms, existing software components, and 3rd party services. The internal structure of such a software basis is often unknown especially if business and system boundaries are crossed. Existing model-driven performance engineering approaches realise a pure top down prediction approach. Software architects have to provide a complete model of their system in order to conduct performance analyses. Measurement-based approaches depend on the availability of the complete system under test. In this paper, we propose a concept for the combination of model-driven and measurement-based performance engineering. We integrate software performance curves with the Palladio Component Model (PCM) (an advanced model-based performance prediction approach) in order to enable the evaluation of enterprise applications which depend on a large software basis.

Measurement-based performance evaluations are heavily used in practice to test system behavior under load, identify resource bottlenecks, or size system landscapes. Existing literature provides guidelines on how to conduct performance evaluations correctly. Many tools (e.g. for load generation, monitoring, or statistical analyses)provide basic assets to conduct such evaluations. However, the wide range of knowledge required to conduct performance evaluations and control the available tools restricts the group of users to a small set of performance experts. Additionally, the large effort to set up systems for performance evaluations often limits their application. In this demo paper, we present a framework that encapsulates best practices and allows for separation of concerns regarding the different aspects of a performance evaluation. The Performance Cockpit provides a single point of configuration for performance analysts and orchestrates plug-ins provided by corresponding experts. The resulting flexibility and automation enables new approaches for quality assurance and lowers the hurdles for conducting performance evaluations.