• We introduced a fully automated framework that quantifies the relationship between microservice architecture complexity, derived from multiple types of statically detected dependencies, and performance-related quality attributes. • We analyzed five complex benchmark systems, including four structurally distinct releases of the Train-Ticket microservice benchmark and one instance of the DeathStarBench suite. • Regarding the relationship between microservice-architecture complexity and overall performance, our analysis demonstrated that systems with poor structural complexity, reflected in high propagation cost and high Clique ratios, exhibited reduced scalability and supported fewer user requests. • Regarding the relationship between individual service complexity and their performance, our results show that endpoints with higher coupling scores tend to have longer response times. Using Spearman’s rank correlation to assess the correlation between each endpoint’s coupling score and its performance score, we find a statistically significant positive correlation. A microservice architecture is intended to promote modularity and evolvability. In this paper, we present an automated framework for assessing the relationship between microservice architecture complexity and performance-related quality attributes. In this framework, we use PPTAM, a performance testing tool, to evaluate system response time under varying user loads, and DV8, an architecture analysis tool, to assess architectural complexity and the complexity of individual services using coupling scores, propagation cost, and architectural antipatterns derived from various types of dependency relations. Using this approach, we evaluated five benchmark systems, including four releases of a microservice system that share similar functionalities but differ in structural design. The results show that microservice architectures with poor complexity scores also exhibited degraded performance outcomes. This automated framework, for the first time, enables a comprehensive measurement of microservice architecture complexity, formed through multiple types of statically extracted dependencies, and its correlation with dynamically obtained performance metrics.

Software performance is a critical quality attribute that determines the success of a software system. However, practitioners lack comprehensive and holistic understanding of how real-life performance issues are caused and resolved in practice from the technical, engineering, and economic perspectives. This paper presents a large-scale empirical study of 570 real-life performance issues from 13 open source projects from various problem domains, and implemented in three popular programming languages, Java (192 issues), C/C++ (162 issues), and Python (216 issues). From the technical perspective, we summarizeeightgeneral types of performance issues with corresponding root causes and resolutions that apply for all three languages. We also identify available tools for detecting and resolving different types of issues from the literature. In addition, we found that 27% of the 570 issues are resolved by design-level optimization—coordinated revision of a group of related source files and their design structure. We reveal four typical design-level optimization patterns, includingclassic design patterns,change propagation,optimization clone, andparallel optimizationthat practitioners should be aware of in resolving performance issues. From the engineering perspective, this study analyzes how test code changes in performance optimization. We found that only 15% of the 570 performance issues involve revision of test code. In most cases, the revised test cases focus on the functional logic of the performance optimization, rather than directly evaluate the performance improvement. This finding points to the potential lack of engineering standard for formally verifying performance optimization in regression testing. Finally, from the economic perspective, we analyze the“Return On Investment”of performance optimization. We found that design-level optimization usually requires more investment, but not always yields to higher performance improvement. However, developers tend to use design-level optimization when they concern about other quality attributes, such as maintainability and readability.

Empirical experience regarding how real-life performance issues are caused and resolved can provide valuable insights for practitioners to effectively and efficiently prevent, detect, and fix performance issues. Prior work shows that most performance issues have their roots in poor architectural decisions. This paper contributes a large scale empirical study of 192 real-life performance issues, with an emphasis on software design. First, this paper contributes a holistic view of eight common root causes and typical resolutions that recur in different projects, and surveyed existing literature, in particular, tools, that can detect and fix each type of performance issue. Second, this study is first-of-its-kind to investigate performance issues from a design perspective. In the 192 issues, 33% required design-level optimization, i.e. simultaneously revising a group of related source files for resolving the issues. We reveal four design-level optimization patterns, which have shown different prevalence in resolving different root causes. Finally, this study investigated the Return on Investment for addressing performance issues, to help practitioners choose between localized or design-level optimization resolutions, and to prioritize issues due to different root causes.

Explicitly stated response time, throughput, and other performance requirements of an application implicitly impose other performance requirements on the system components that implement it. We call these derived performance requirements. The explicit performance requirements cannot be met if the derived performance requirements are not met. Explicit performance requirements naturally give rise to corresponding derived performance requirements expressed in terms of the same metrics. Derived performance requirements may also be identified that specify the sizes of object pools or the amount of memory needed to meet explicit and other derived requirements. Moreover, derived requirements may be identified that depend on the implementation of the components. We explore how derived requirements arise and present a methodology for identifying and specifying them.

In many software development projects, attention is only paid to performance concerns after functional testing, when it usually too late to remedy disabling performance problems. Early attention to performance concerns and early planning of performance requirements and performance testing can prevent debacles like the early rollout of healthcare.gov while addressing cross-cutting concerns such as scalability, reliability and, security. Performance engineering methods may be integrated into all phases of the software lifecycle, from the conception of a system to requirements specification, architecture, testing, and finally to production. Performance expectations can be managed by carefully specifying performance requirements. Reviewing the architecture of a system before design and implementation take place reduces the risk of designing a system that contains inherent performance vice. Performance modeling can be used to justify architectural and design decisions and to plan performance tests. The outputs of such performance tests enable us able to identify concurrent programming and other issues that would not be apparent in unit testing. Finally, risk is mitigated by avoiding design antipatterns that undermine scalability and performance.

Performance requirements are one of the main drivers of architectural decisions. Because many performance problems have their roots in architectural decisions, and since poor performance is a principal cause of software project risk, it is essential that performance requirements be developed early in the software lifecycle, and that they be clearly formulated. In this tutorial, we shall look at criteria for high-quality performance requirements, including algebraic consistency, measurability, testability, and linkage to business and engineering needs. While focus of this tutorial is on practice, we shall show how the drafting of performance requirements can be aided by performance modeling. We shall show methods for presenting and managing performance requirements that will improve their chances of being accepted by architects, developers, testers, contract negotiators, and purchasers; and of their being successfully implemented and tested.

We describe our experience of training a remotely located team of developers and testers to prepare and execute performance tests. The team is located in India. The lead performance engineer and the test project manager are based in New Jersey. The team members had little or no prior experience of performance testing. We describe how we overcame cultural differences and a large time difference to develop a performance testing team that is now functioning well with far less supervision than was required at its inception. Cultural differences included contrasting views on adherence to strict laboratory procedures and assumptions about the prior knowledge, experience, and expectations of working habits of the India-based and New Jersey-based teams. We show how these differences and organizational challenges were overcome with intensive on-site training, the use of twice-daily scrum meetings, the careful designation of team leaders and role players at the remote testing site, and, eventually, the development intensive use of automated tools to execute performance tests and track the results.

Summary form only given. Functional requirements refer to how a piece of a system functions or what it does given specific inputs. Nonfunctional requirements (NFRs) refer to such business and engineering needs as performance, reliability, availability, stability, usability, compatibility with interfaces, and security. They are crucial inputs for architects and system designers when performing such tasks as specifying information flows, defining deployment scenarios, choosing software platforms, sizing system configurations, determining required levels of redundancy, and identifying system measurement instrumentation. NFRs that have been clearly and succinctly written contribute to the success of a project by reducing the time required to understand them and to devise and implement procedures for ensuring and verifying that they have been met. By contrast, poorly written NFRs can lead to confusion among developers and architects, resulting in much wasted time, or to the failure and cancellation of a project, especially if the delivered system crashes often or runs too slowly. The focus of this tutorial is quantitative nonfunctional requirements relating to performance, reliability, and availability (PRA). In this tutorial, we shall look at criteria for high-quality PRA NFRs, including algebraic consistency, measurability, testability, and linkage to business and engineering needs. We shall show methods for presenting and managing PRA NFRs that will improve their chances of being accepted by architects, developers, contract negotiators, and purchasers; and of their being successfully implemented and tested. Attendees will be invited to participate in exercises in which they will interpret and draft PRA NFRs and develop procedures for validating them.

We present three algorithms for detecting the need for software rejuvenation by monitoring the changing values of a customer-affecting performance metric, such as response time. Applying these algorithms can improve the values of this customer-affecting metric by triggering rejuvenation before performance degradation becomes severe. The algorithms differ in the way they gather and use sample values to arrive at a rejuvenation decision. Their effectiveness is evaluated for different sets of control parameters, including sample size, using simulation. The results show that applying the algorithms with suitable choices of control parameters can significantly improve system performance as measured by the response time

Approximations are given for the mean response times of each priority level in a multiple-class multiserver M/G/m queue operating under preemptive resume scheduling. The results have been tested against simulations of systems with two and three priority classes and different numbers of servers.