This paper addresses the challenge of accurately detecting the transition from the warmup phase to the steady state in performance metric time series, which is a critical step for effective benchmarking. The goal is to introduce a method that avoids premature or delayed detection, which can lead to inaccurate or inefficient performance analysis. The proposed approach adapts techniques from the chemical reactors domain, detecting steady states online through the combination of kernel-based step detection and statistical methods. By using a window-based approach, it provides detailed information and improves the accuracy of identifying phase transitions, even in noisy or irregular time series. Results show that the new approach reduces total error by 14.5% compared to the best selected state-of-the-art method. It offers more reliable detection of the steady-state onset, delivering greater precision for benchmarking tasks. For users, the new approach enhances the accuracy and stability of performance benchmarking, efficiently handling diverse time series data. Its robustness and adaptability make it a valuable tool for real-world performance evaluation, ensuring consistent and reproducible results.

Modern software applications generate a wide range of runtime metrics, which are vital to many quality assurance activities. These data are often recorded and aggregated as time series to observe patterns and trends of various runtime aspects over time. In this context, Time Series Forecasting (TSF) offers unique opportunities for predicting software runtime behavior and identifying potential anomalies. Although TSF models have been successfully applied in fields such as economics and climatology, their capabilities for forecasting software runtime metrics remain relatively underexplored. In this paper, we conduct a comprehensive empirical evaluation of 8 TSF models on 110 real-world software runtime metrics recorded over the course of about one year. Our evaluation encompasses three classical statistical models, three neural network models, and two time series foundation models. Results show that the foundation models achieve state-of-the-art performance on TSF of software runtime metrics, outperforming other models with strong statistical significance. Our findings indicate that foundation models, despite being trained exclusively on time series data from other domains, can effectively generalize to software runtime metrics in a zero-shot setting. This makes them a convenient plug-and-play solution for practitioners and researchers aiming to integrate TSF into their software quality assurance processes. Yet, their performance is not uniformly superior across all the time series, underscoring the absence of a “ silver bullet ” solution.

Code optimization is the process of enhancing code efficiency, while preserving its intended functionality. This process often requires a deep understanding of the code execution behavior at run-time to identify and address inefficiencies effectively. Recent studies have shown that language models can play a significant role in automating code optimization. However, these models may have insufficient knowledge of how code execute at run-time. To address this limitation, researchers have developed strategies that integrate code execution information into language models. These strategies have shown promise, enhancing the effectiveness of language models in various software engineering tasks. However, despite the close relationship between code execution behavior and efficiency, the specific impact of these strategies on code optimization remains largely unexplored. This study investigates how incorporating code execution information into language models affects their ability to optimize code. Specifically, we apply three different training strategies to incorporate four code execution aspects - line executions, line coverage, branch coverage, and variable states - into CodeT5+, a well-known language model for code. Our results indicate that executionaware models provide limited benefits compared to the standard CodeT5+ model in optimizing code.

Abstract Performance testing aims to ensure the operational efficiency of software systems. However, many factors influencing the efficacy and adoption of performance tests in practice are not yet fully understood. For instance, while code coverage is widely regarded as a key quality metric for evaluating the efficacy of functional testing suites, there is limited knowledge about the types and levels of coverage that performance tests specifically achieve. Another important factor, often perceived as a barrier to the broader adoption of performance tests yet remaining relatively unexplored, is their extended execution time. In this paper, we examine (i) the coverage of performance testing suites, (ii) the characteristics of source code associated with performance-tested components, and (iii) the time cost of executing performance tests. Our analysis on open-source Java systems reveals that performance tests achieve significantly lower code coverage than functional tests, as expected, and it highlights a significant trade-off between coverage and execution time. Our results also indicate a lack of generalizable characteristics in the source code covered by performance tests.

Software architecture optimization aims to enhance non-functional attributes like performance and reliability while meeting functional requirements. Multi-objective optimization employs metaheuristic search techniques, such as genetic algorithms, to explore feasible architectural changes and propose alternatives to designers. However, this resource-intensive process may not always align with practical constraints. This study investigates the impact of designer interactions on multi-objective software architecture optimization. Designers can intervene at intermediate points in the fully automated optimization process, making choices that guide exploration towards more desirable solutions. Through several controlled experiments as well as an initial user study (14 subjects), we compare this interactive approach with a fully automated optimization process, which serves as a baseline. The findings demonstrate that designer interactions lead to a more focused solution space, resulting in improved architectural quality. By directing the search toward regions of interest, the interaction uncovers architectures that remain unexplored in the fully automated process. In the user study, participants found that our interactive approach provides a better trade-off between sufficient exploration of the solution space and the required computation time.

Performance testing aims to ensure the operational efficiency of software systems. However, many factors influencing the efficacy and adoption of performance tests in practice are not yet fully understood. For instance, while code coverage is widely regarded as a key quality metric for evaluating the efficacy of functional testing suites, there is limited knowledge about the types and levels of coverage that performance tests specifically achieve. Another important factor, often perceived as a barrier to the broader adoption of performance tests yet remaining relatively unexplored, is their extended execution time. In this paper, we analyze the performance testing suites of 28 open-source systems to study (i) the magnitude of their code coverage, and (ii) their execution time. Our analysis shows that performance tests achieve significantly lower code coverage than functional tests, as expected, and it highlights a significant trade-off between coverage and execution time. Our results also suggest, in perspective, that automated test generation methods might not ensure affordable performance testing due to the associated time cost. This finding poses new challenges in the field of performance test generation.

As organizations increasingly migrate their applications to the cloud, the optimization of microservices architectures becomes imperative for achieving sustainability goals. Nonetheless, sustainable deployments may increase costs and deteriorate performance, thus the identification of optimal trade-offs among these conflicting requirements is a key objective not easy to achieve. This paper introduces a novel approach to support cloud deployment of microservices architectures by targeting optimal combinations of application performance, deployment costs, and power consumption. By leveraging genetic algorithms, specifically NSGA-II, we automate the generation of alternative architectural deployments. The results demonstrate the potential of our approach through a comprehensive assessment of the Train Ticket case study.

Performance testing aims at uncovering efficiency issues of software systems. In order to be both effective and practical, the design of a performance test must achieve a reasonable trade-off between result quality and testing time. This becomes particularly challenging in Java context, where the software undergoes a warm-up phase of execution, due to just-in-time compilation. During this phase, performance measurements are subject to severe fluctuations, which may adversely affect quality of performance test results. Both practitioners and researchers have proposed approaches to mitigate this issue. Practitioners typically rely on a fixed number of iterated executions that are used to warm-up the software before starting to collect performance measurements (state-of-practice). Researchers have developed techniques that can dynamically stop warm-up iterations at runtime (state-of-the-art). However, these approaches often provide suboptimal estimates of the warm-up phase, resulting in either insufficient or excessive warm-up iterations, which may degrade result quality or increase testing time. There is still a lack of consensus on how to properly address this problem. Here, we propose and study an AI-based framework to dynamically halt warm-up iterations at runtime. Specifically, our framework leverages recent advances in AI for Time Series Classification (TSC) to predict the end of the warm-up phase during test execution. We conduct experiments by training three different TSC models on half a million of measurement segments obtained from JMH microbenchmark executions. We find that our framework significantly improves the accuracy of the warm-up estimates provided by state-of-practice and state-of-the-art methods. This higher estimation accuracy results in a net improvement in either result quality or testing time for up to +35.3% of the microbenchmarks. Our study highlights that integrating AI to dynamically estimate the end of the warm-up phase can enhance the cost-effectiveness of Java performance testing.

The successful operation of a modern company re-lays on the dependability of its software infrastructure. However, ensuring a robust and dependable software infrastructure can be challenging, as software applications are subject to continuous updates that can introduce bugs and performance regressions. To mitigate this challenge, many companies use Application Performance Management (APM) tools to monitor their digital devices and identify potential issues that could affect business operability. However, the large volume and heterogeneity of the data collected by these tools can make it difficult to effectively analyze and exploit the rich source of information available. In this paper, we propose RADig-X, a tool designed to support the identification and analysis of digital experience issues. RADig-X leverages AI algorithms and a ranking heuristic to: (i) detect anomalies in runtime metrics collected by APM tools, (ii) assess the relevance of these anomalies based on their impact on the overall IT infrastructure, and (iii) rank problematic software updates that may be the root cause of relevant anomalies. We report on the adoption of RADig-X by a large company that monitors over 30,000 digital devices around the world. Our results demonstrate that RADig-X is able to improve the effectiveness of the identification process of digital experience issues, by enabling to identify and address potential anomalies that could impact business operations. RADig-X is currently used in production within the case company to support the diagnosis and problem resolution of digital experience issues.

Software applications can produce a wide range of runtime software metrics (e.g., number of crashes, response times), which can be closely monitored to ensure operational efficiency and prevent significant software failures. These metrics are typically recorded as time series data. However, runtime software monitoring has become a high-effort task due to the growing complexity of today's software systems. In this context, time series forecasting (TSF) offers unique opportunities to enhance software monitoring and facilitate proactive issue resolution. While TSF methods have been widely studied in areas like economics and weather forecasting, our understanding of their effectiveness for software runtime metrics remains somewhat limited. In this paper, we investigate the effectiveness of four TSF methods on 25 real-world runtime software metrics recorded over a period of one and a half years. These methods comprise three recurrent neural network (RNN) models and one traditional time series analysis technique (i.e., SARIMA). The metrics are gathered from a large-scale IT infrastructure involving tens of thousands of digital devices. Our results indicate that, in general, RNN models are very effective in the runtime software metrics prediction, although in some scenarios and for certain specific metrics (e.g., waiting times) SARIMA proves to outperform RNN models. Additionally, our findings suggest that the advantages of using RNN models vanish when the prediction horizon becomes too wide, in our case when it exceeds one week.

The continuous software engineering paradigm is gaining popularity in modern development practices, where the interleaving of design and runtime activities is induced by the continuous evolution of software systems. In this context, performance assessment is not easy, but recent studies have shown that architectural models evolving with the software can support this goal. In this paper, we present a mapping study aimed at classifying existing scientific contributions that deal with the architectural support for performance-targeted continuous software engineering. We have applied the systematic mapping methodology to an initial set of 215 potentially relevant papers and selected 66 primary studies that we have analyzed to characterize and classify the current state of research. This classification helps to focus on the main aspects that are being considered in this domain and, mostly, on the emerging findings and implications for future research. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board. (see [https://www.sciencedirect.com/science/article/pii/S0164121221002168] for an example for where to place the statement and how to format it).

Abstract Microbenchmarking is a widely used form of performance testing in Java software. A microbenchmark repeatedly executes a small chunk of code while collecting measurements related to its performance. Due to Java Virtual Machine optimizations, microbenchmarks are usually subject to severe performance fluctuations in the first phase of their execution (also known as warmup). For this reason, software developers typically discard measurements of this phase and focus their analysis when benchmarks reach a steady state of performance. Developers estimate the end of the warmup phase based on their expertise, and configure their benchmarks accordingly. Unfortunately, this approach is based on two strong assumptions: (i) benchmarks always reach a steady state of performance and (ii) developers accurately estimate warmup. In this paper, we show that Java microbenchmarks do not always reach a steady state, and often developers fail to accurately estimate the end of the warmup phase. We found that a considerable portion of studied benchmarks do not hit the steady state, and warmup estimates provided by software developers are often inaccurate (with a large error). This has significant implications both in terms of results quality and time-effort. Furthermore, we found that dynamic reconfiguration significantly improves warmup estimation accuracy, but still it induces suboptimal warmup estimates and relevant side-effects. We envision this paper as a starting point for supporting the introduction of more sophisticated automated techniques that can ensure results quality in a timely fashion.

Software quality estimation is a challenging and time-consuming activity, and models are crucial to face the complexity of such activity on modern software applications. In this context, software refactoring is a crucial activity within development life-cycles where requirements and functionalities rapidly evolve. One main challenge is that the improvement of distinctive quality attributes may require contrasting refactoring actions on software, as for trade-off between performance and reliability (or other non-functional attributes). In such cases, multi-objective optimization can provide the designer with a wider view on these trade-offs and, consequently, can lead to identify suitable refactoring actions that take into account independent or even competing objectives. In this paper, we present an approach that exploits the NSGA-II as the genetic algorithm to search optimal Pareto frontiers for software refactoring while considering many objectives. We consider performance and reliability variations of a model alternative with respect to an initial model, the amount of performance antipatterns detected on the model alternative, and the architectural distance, which quantifies the effort to obtain a model alternative from the initial one. We applied our approach on two case studies: a Train Ticket Booking Service, and CoCoME. We observed that our approach is able to improve performance (by up to 42%) while preserving or even improving the reliability (by up to 32%) of generated model alternatives. We also observed that there exists an order of preference of refactoring actions among model alternatives. Based on our analysis, we can state that performance antipatterns confirmed their ability to improve performance of a subject model in the context of many-objective optimization. In addition, the metric that we adopted for the architectural distance seems to be suitable for estimating the refactoring effort.

Performance debugging in production is a fundamental activity in modern service-based systems. The diagnosis of performance issues is often time-consuming, since it requires thorough inspection of large volumes of traces and performance indices. In this paper we present DeLag, a novel automated search-based approach for diagnosing performance issues in service-based systems. DeLag identifies subsets of requests that show, in the combination of their Remote Procedure Call execution times, symptoms of potentially relevant performance issues. We call such symptomsLatency Degradation Patterns. DeLag simultaneously searches for multiplelatency degradation patternswhile optimizing precision, recall and latency dissimilarity. Experimentation on 700 datasets of requests generated from two microservice-based systems shows that our approach provides better and more stable effectiveness than three state-of-the-art approaches and general purpose machine learning clustering algorithms. DeLag is more effective than all baseline techniques in at least one case study (with$p\leq 0.05$and non-negligible effect size). Moreover, DeLag outperforms in terms of efficiency the second and the third most effective baseline techniques on the largest datasets used in our evaluation (up to 22%).

Microservices are quite widely impacting on the software industry in recent years. Rapid evolution and continuous deployment represent specific benefits of microservice-based systems, but they may have a significant impact on non-functional properties like performance. Despite the obvious relevance of this property, there is still a lack of systematic approaches that explicitly take into account performance issues in the lifecycle of microservice-based systems. In such a context of evolution and re-deployment, Model-Driven Engineering techniques can provide major support to various software engineering activities, and in particular they can allow managing the relationships between a running system and its architectural model. In this paper, we propose a model-driven integrated approach that exploits traceability relationships between the monitored data of a microservice-based running system and its architectural model to derive recommended refactoring actions that lead to performance improvement. The approach has been applied and validated on two microservice-based systems, in the domain of e-commerce and ticket reservation, respectively, whose architectural models have been designed in UML profiled with MARTE.

Refactoring aims at improving the maintainability of source code without modifying its external behavior. Previous works proposed approaches to recommend refactoring solutions to software developers. The generation of the recommended solutions is guided by metrics acting as proxy for maintainability (e.g., number of code smells removed by the recommended solution). These approaches ignore the impact of the recommended refactorings on other non-functional requirements, such as performance, energy consumption, and so forth. Little is known about the impact of refactoring operations on non-functional requirements other than maintainability. We aim to fill this gap by presenting the largest study to date to investigate the impact of refactoring on software performance, in terms of execution time. We mined the change history of 20 systems that defined performance benchmarks in their repositories, with the goal of identifying commits in which developers implemented refactoring operations impacting code components that are exercised by the performance benchmarks. Through a quantitative and qualitative analysis, we show that refactoring operations can significantly impact the execution time. Indeed, none of the investigated refactoring types can be considered “safe” in ensuring no performance regression. Refactoring types aimed at decomposing complex code entities (e.g., Extract Class/Interface, Extract Method) have higher chances of triggering performance degradation, suggesting their careful consideration when refactoring performance-critical code.

Software quality estimation is a challenging and time-consuming activity, and models are crucial to face the complexity of such activity on modern software applications. One main challenge is that the improvement of distinctive quality attributes may require contrasting refactoring actions on an application, as for trade-off between performance and reliability. In such cases, multi-objective optimization can provide the designer with a wider view on these trade-offs and, consequently, can lead to identify suitable actions that take into account independent or even competing objectives. In this paper, we present an approach that exploits the NSGA - II multi-objective evolutionary algorithm to search optimal Pareto solution frontiers for software refactoring while considering as objectives: i) performance variation, ii) reliability, iii) amount of performance antipatterns, and iv) architectural distance. The algorithm combines randomly generated refactoring actions into solutions (i.e., sequences of actions) and compares them according to the objectives. We have applied our approach on a train ticket booking service case study, and we have focused the analysis on the impact of performance antipatterns on the quality of solutions. Indeed, we observe that the approach finds better solutions when antipatterns enter the multi-objective optimization. In particular, performance antipatterns objective leads to solutions improving the performance by up to 15% with respect to the case where antipatterns are not considered, without affecting the solution quality on other objectives.

Software architecture refactoring can be induced by multiple reasons, such as satisfying new functional requirements or improving non-functional properties. Multi-objective optimization approaches have been widely used in the last few years to introduce automation in the refactoring process, and they have revealed their potential especially when quantifiable attributes are targeted. However, the effectiveness of such approaches can be heavily affected by configuration characteristics of the optimization algorithm, such as the composition of solutions. In this paper, we analyze the behavior of EASIER, which is an Evolutionary Approach for Software archItecturE Refactoring, while varying its configuration characteristics, with the objective of studying its potential to find near-optimal solutions under different configurations. In particular, we use two different solution space inspection algorithms (i.e., NSGA−II and SPEA2) while varying the genome length and the solution composition. We have conducted our experiments on a specific case study modeled in Æmilia ADL, on which we have shown the ability of EASIER to identify performance-critical elements in the software architecture where refactoring is worth to be applied. Beside this, from the comparison of multi-objective algorithms, NSGA−II has revealed to outperform SPEA2 in most of cases, although the latter one is able to induce more diversity in the proposed solutions. Our results show that the EASIER thoroughly automated process for software architecture refactoring allows to identify configuration contexts of the evolutionary algorithm in which multi-objective optimization more effectively finds near-optimal Pareto solutions.

Context: Managing Non-Functional Requirements (NFRs) in software projects is challenging, and projects that adopt Model-Driven Development (MDD) are no exception. Although several methods and techniques have been proposed to face this challenge, there is still little evidence on how NFRs are handled in MDD by practitioners. Knowing more about the state of the practice may help researchers to steer their research and practitioners to improve their daily work. Objective: In this paper, we present our findings from an interview-based survey conducted with practitioners working in 18 different companies from 6 European countries. From a practitioner's point of view, the paper shows what barriers and benefits the management of NFRs as part of the MDD process can bring to companies, how NFRs are supported by MDD approaches, and which strategies are followed when (some) types of NFRs are not supported by MDD approaches. Results: Our study shows that practitioners perceive MDD adoption as a complex process with little to no tool support for NFRs, reporting productivity and maintainability as the types of NFRs expected to be supported when MDD is adopted. But in general, companies adapt MDD to deal with NFRs. When NFRs are not supported, the generated code is sometimes changed manually, thus compromising the maintainability of the software developed. However, the interviewed practitioners claim that the benefits of using MDD outweight the extra effort required by these manual adaptations. Conclusion: Overall, the results indicate that it is important for practitioners to handle `NFRs in MDD, but further research is necessary in order to lower the barrier for supporting a broad spectrum of NFRs with MDD. Still, much conceptual and tool implementation work seems to be necessary to lower the barrier of integrating the broad spectrum of NFRs in practice.

Refactoring is often needed to ensure that software systems meet their performance requirements in deployments with different operational profiles, or when these operational profiles are not fully known or change over time. This is a complex activity in which software engineers have to choose from numerous combinations of refactoring actions. Our paper introduces a novel approach that uses performance antipatterns and stochastic modelling to support this activity. The new approach computes the performance antipatterns present across the operational profile space of a software system under development, enabling engineers to identify operational profiles likely to be problematic for the analysed design, and supporting the selection of refactoring actions when performance requirements are violated for an operational profile region of interest. We demonstrate the application of our approach for a software system comprising a combination of internal (i.e., in-house) components and external third-party services.

Performance in heterogeneous service-based systems shows non-determistic trends. Even for the same request type, latency may vary from one request to another. These variations can occur due to several reasons on different levels of the software stack: operating system, network, software libraries, application code or others. Furthermore, a request may involve several Remote Procedure Calls (RPC), where each call can be subject to performance variation. Performance analysts inspect distributed traces and seek for recurrent patterns in trace attributes, such as RPCs execution time, in order to cluster traces in which variations may be induced by the same cause. Clustering "similar" traces is a prerequisite for effective performance debugging. Given the scale of the problem, such activity can be tedious and expensive. In this paper, we present an automated approach that detects relevant RPCs execution time patterns associated to request latency degradation, i.e. latency degradation patterns. The presented approach is based on a genetic search algorithm driven by an information retrieval relevance metric and an optimized fitness evaluation. Each latency degradation pattern identifies a cluster of requests subject to latency degradation with similar patterns in RPCs execution time. We show on a microservice-based application case study that the proposed approach can effectively detect clusters identified by artificially injected latency degradation patterns. Experimental results show that our approach outperforms in terms of F-score a state-of-art approach for latency profile analysis and widely popular machine learning clustering algorithms. We also show how our approach can be easily extended to trace attributes other than RPC execution time (e.g. HTTP headers, execution node, etc.).

With the ever-increasing evolution of software systems, their architecture is subject to frequent changes due to multiple reasons, such as new requirements. Appropriate architectural changes driven by non-functional requirements are particularly challenging to identify because they concern quantitative analyses that are usually carried out with specific languages and tools. A considerable number of approaches have been proposed in the last decades to derive non-functional analysis models from architectural ones. However, there is an evident lack of automation in the backward path that brings the analysis results back to the software architecture. In this paper, we propose a model-driven approach to support designers in improving the availability of their software systems through refactoring actions. The proposed framework makes use of bidirectional model transformations to map UML models onto Generalized Stochastic Petri Nets (GSPN) analysis models and vice versa. In particular, after availability analysis, our approach enables the application of model refactoring, possibly based on well-known fault tolerance patterns, aimed at improving the availability of the architectural model. We validated the effectiveness of our approach on an Environmental Control System. Our results show that the approach can generate: (i) an analyzable availability model from a software architecture description, and (ii) valid software architecture models back from availability models. Finally, our results highlight that the application of fault tolerance patterns significantly improves the availability in each considered scenario. The approach integrates bidirectional model transformation and fault tolerance techniques to support the availability-driven refactoring of architectural models. The results of our experiment showed the effectiveness of the approach in improving the software availability of the system.

Model-Driven Engineering techniques may achieve a major support to the software development when they allow to manage relationships between a running system and its architectural model. These relationships can be exploited for different goals, such as the software evolution due to new functional requirements. In this paper, we define and use relationships that work as support to the performance improvement of a running system. In particular, we combine: (i) a bidirectional model transformation framework tailored to define relationships between performance monitoring data and an architectural model, with (ii) a technique for detecting performance antipatterns and for suggesting architectural changes, aimed at removing performance problems identified on the basis of runtime information. The result is an integrated approach that exploits traceability relationships between the monitoring data and the architectural model to derive recommended refactoring solutions for the system performance improvement. The approach has been applied to an e-commerce application based on microservices that has been designed by means of UML software models profiled with MARTE.

The satisfaction of ever more stringent performance requirements is one of the main reasons for software evolution. However, it is complex to determine the primary causes of performance degradation, because they may depend on the joint combination of multiple factors (e.g., workload, software deployment, hardware utilization). With the increasing complexity of software systems, classical bottleneck analysis shows limitations in capturing complex performance problems. Hence, in the last decade, the detection of performance antipatterns has gained momentum as an effective way to identify performance degradation causes. We introduce PADRE (Performance Antipattern Detection and REfactoring), that is a tool for: (i) detecting performance antipattern in UML models, and (ii) refactoring models with the aim of removing the detected antipatterns. PADRE has been implemented within Epsilon, an open-source platform for model-driven engineering. It is based on a methodology that allows performance antipattern detection and refactoring within the same implementation context.

Context awareness is a first-class attribute of today software systems. Indeed, many applications need to be aware of their context in order to adapt their structure and behavior for offering the best quality of service even in case the software and hardware resources are limited. Modeling the context, its evolution, and its influence on the services provided by (possibly resource constrained) applications are becoming primary activities throughout the whole software life cycle, although it is still difficult to capture the multidimensional nature of context. We propose a framework for modeling and reasoning on the context and its evolution along multiple dimensions. Our approach enables (1) the representation of dependencies among heterogeneous context attributes through a formally defined semantics for attribute composition and (2) the stochastic analysis of context evolution. As a result, context can be part of a model-based software development process, and multidimensional context analysis can be used for different purposes, such as non-functional analysis. We demonstrate how certain types of analysis, not feasible with context-agnostic approaches, are enabled in our framework by explicitly representing the interplay between context evolution and non-functional attributes. Such analyses allow the identification of critical aspects or design errors that may not emerge without jointly taking into account multiple context attributes. The framework is shown at work on a case study in the eHealth domain.

Multi-objective optimization has demonstrated, in the last few years, to be an effective paradigm to tackle different architectural problems, such as service selection, composition and deployment. In particular, multi-objective approaches for searching architectural configurations that optimize quality properties (such as performance, reliability and cost) have been introduced in the last decade. However, a relevant amount of complexity is introduced in this context when performance are considered, often due to expensive iterative generation of performance models and interpretation of results. In this paper we introduce EASIER (Evolutionary Approach for multi-objective Software archItecturE Refactoring), that is an approach for optimizing architecture refactoring based on performance and on the intensity of changes. We focus on the actionable aspects of architectural optimization, instead of merely searching over a set of alternatives. We also start to investigate on the potential influence of performance antipatterns on such process. We have implemented our approach on AEmilia ADL, so to carry out performance analysis and architecture refactoring within the same environment. We demonstrate the effectiveness and applicability of our approach through its experimentation on a case study.

Software architecture is nowadays subject to frequent changes due to multiple reasons, such as evolution induced by new requirements. Architectural changes driven by non-functional requirements are particularly difficult to identify, because they attain quantitative analyses that are usually carried out with specific languages and tools. A considerable number of approaches, based on model transformations, have been proposed in the last decades to derive non-functional models from software architectural descriptions. However, there is a clear lack of automation in the backward path that brings the analysis results back to the software architecture. In this paper we address this problem in the context of software availability. We introduce a bidirectional model transformation between UML State Machines (SM), annotated with availability properties, and Generalized Stochastic Petri Nets (GSPN). Such transformation, implemented in the JTL language, is used both to derive a GSPN-based availability model from a SM-based software architecture and, after the analysis, to propagate back on the SM the changes carried out on the GSPN. We demonstrate the effectiveness of our approach on an Environmental Control System to which we apply well-known fault tolerance patterns aimed at improving its software availability.

Model-Driven Development (MDD) is no longer a novel development paradigm. It has become mature from a research perspective and recent studies show its adoption in industry. Still, some issues remain a challenge. Among them, we are interested in the treatment of non-functional requirements (NFRs) in MDD processes. Very few MDD approaches have been reported to deal with NFRs (and they do it in a limited way). However, it is clear that NFRs need to be considered somehow in the final product of the MDD process. To better understand how NFRs are integrated into the existing MDD approaches, we have initiated the NFR4MDD project, a multi-national empirical study, based on interviews with companies working on MDD projects. Our project aims at surveying the state of the practice for this topic. In this paper, we summarize our research protocol and present the current status of our study. The discussion will focus on the peculiarities of our study's context and organization involving about 20 researchers from 8 European countries.

The problem of interpreting the results of performance analysis is quite critical, mostly because the analysis results (i.e. mean values, variances, and probability distributions) are hard to transform into feedback for software engineers that allows to remove performance problems. Approaches aimed at identifying and removing the causes of poor performance in software systems commonly fall in two categories: (i) bottleneck analysis, aimed at identifying overloaded software components and/or hardware resources that affect the whole system performance, and (ii) performance antipatterns, aimed at detecting and removing common design mistakes that notably induce performance degradation.

The analysis of non-functional properties of Service-Based Systems (SBSs) is a complex task, mostly because it requires models that encompass the composition of service properties into architectural properties. For example, the reliability of a SBS is given by the composition of service and interconnection reliabilities. Although several approaches have been introduced in the last few years to address these issues, the tradeoff analysis among non-functional properties of software services has not yet been studied enough. The goal of this paper is to introduce a set of optimization models that allow quantifying the costs of service failure repair/mitigation actions aimed at keeping the whole SBS reliability over a certain threshold. On the basis of our previous work in this area, we first introduce an optimization model aimed at selecting either in-house built or provided services with the goal of minimizing the SBS cost while guaranteeing a certain level of reliability. Thereafter we strengthen the reliability constraints, and we build two different optimization models that aim to solve the same problem under new constraints, where one model starts from the solution obtained in the original model and tries to improve it, while the other one looks for an optimal solution in the whole search space. Finally, we introduce a fourth model, based on stochastic optimization, with the goal of rather searching for solutions that explicitly take into account the stochastic nature of the problem and search for new repair/mitigation actions cheaper than the ones identified by the other models. Each optimization model has been experimented on about 300 variations of a nominal model. The experimental results show the efficacy of our optimization models to quantify the costs of different failure repairing/mitigation actions in different contexts.

This paper presents a project that has been recently funded by means of the ERC Starting Grant (http://erc.europa.eu/). The main goal of the project is to develop a new generarion of wireless sensor networks infrastructure to support innovative services for ubiquitous sensing and video. The work is in a very preliminary phase so the paper presents a general overview and the main challenges.

In the next future we will be surrounded by a virtually infinite number of software applications that provide computational software resources in the open Globe. This will radically change the way software will be produced and used. Users will be keen on producing their own piece of software, by also reusing existing software, to better satisfy their needs, therefore with a goal oriented, opportunistic use in mind. The produced software will need to be able to evolve, react and adapt to a continuously changing environment, while guaranteeing dependability. The strongest adversary to this view is the lack of knowledge on the software's structure, behavior, and execution context. Despite the possibility to extract observational models from existing software, a producer will always operate with software artifacts that exhibit a degree of uncertainty in terms of their functional and non functional characteristics. We believe that uncertainty can only be controlled by making it explicit and by using it to drive the production process itself. In this paper, we introduce a novel paradigm of software production process that explores available software and assesses its degree of uncertainty in relation to the opportunistic goal G, assists the producer in creating the appropriate integration means towards G, and validates the quality of the integrated system with respect to G and the current context.

The problem of interpreting the results of performance analysis is quite critical in the software performance domain. Mean values, variances, probability distributions are hard to interpret for providing feedback to software architects. Instead, what architects expect are solutions to performance problems, possibly in the form of architectural alternatives (e. g. split a software component in two components and re-deploy one of them). In a software performance engineering approach this path from analysis results to software alternatives still lacks of automation and is based on the skills and experience of analysts. In this paper we propose an automated approach for the performance feedback generation process based on performance antipatterns. To this aim, we model performance antipatterns as logical predicates and we provide a java engine, based on such predicates, that is able to detect performance antipatterns in an XML representation of the software system. Finally, we show the approach at work on a simple case study.

We present in this paper our experience with applying a new architecture review process that uses a globally distributed review team to assess architecture risk of a complex mission critical system. The new architecture review process uses aspects of the checklist-based architecture review process and the operational scenario-based architecture review process. We present the architecture review process approach, a summary of the architecture under review and the detailed analysis of the most important operational scenarios. We conclude by presenting a summary of the lessons we learned using the new process.

In a component-based development process the selection of components is an activity that takes place over multiple lifecycle phases that span from requirement specifications through design to implementation-integration. Automated tool support for component selection would be very helpful in each phase. In this paper we introduce a framework that supports the selection of COTS components in the requirements phase. The framework lays on a tool that builds and solves an optimization model, whose solution provides the optimal COTS component selection. The selection criterion is based on cost minimization of the whole system while assuring a certain degree of satisfaction of the system requirements. The output of the model solution indicates the optimal combination of single COTS components and assemblies of COTS that satisfy the requirements while minimizing costs

Software models are, in most cases, considered as functional abstractions of systems. They represent the backbone of transformational processes aimed at code generation. On the other end, modeling is a traditional activity in the field of non-functional validation of software/hardware systems, although non-functional models found on different notations (such as Petri Nets) and embed additional information (such as the operational profile) with respect to software models. In this paper we widen the scope of model-driven architecture by introducing a Non-Functional-MDA framework that, beside the typical model transformations for code generation, embeds new types of model transformations that allow to generate non-functional models. For an uniform integration of these practices, we define Platform Independent/Specific Models in the non-functional domain.

System performance is a key factor to take into account throughout the software life cycle of modern computer systems, mostly due to their typical characteristics such as distributed deployment, code mobility, and platform heterogeneity. An open challenge in this direction is to integrate the performance validation as a transparent and efficient activity in the system development process. Several methodologies have been proposed to automate the transformation of software/hardware models into performance models. In this paper, we do not take a transformational approach; rather, we present a framework to integrate a software model with a platform model in order to build a performance model. Performance indices are obtained from simulation of the resulting performance model. Our framework provides a library of predefined resource models, model annotation and integration procedures, and simulation support that makes the performance analysis a much easier activity. We present the results obtained from two different industrial case studies that show the maturity and the stability of our approach.

Increased security risk in international travel has resulted in the creation of new programs to determine the admissibility of foreign travelers at official ports of entry within a country. Primary program goals are improving border security and, at the same time, facilitating the flow of legitimate travelers. Major program requirements include the adoption of machine readable travel documents (i.e., passports, visas, etc.), the use of biometric identifiers, and the interoperability among multiple information systems for travelersy identity verification and background checks. Performance analysis of a border inspection system early in its development life-cycle is essential to predict its ability to meet established performance goals, to identify key performance drivers and potential bottlenecks and to suggest possible design improvements. This paper presents our experience with performance evaluation of a hypothetical inspection system. We adopt an analytical modeling technique based on layered queuing networks. Compared with similar studies which use extensive simulations, we observe that our methodology achieves comparably accurate results while being simpler and less costly

It is widely recognized that in order to make performance validation an integrated activity along the software lifecycle it is crucial to be supported from automated approaches. Easiness to annotate software models with performance parameters (e.g. the operational profile) and automated translations of the annotated models into "ready-to-validate" models are the key challenges in this direction. Several methodologies have been introduced in the last few years to address these challenges. The tutorial introduces the attendance to the main methodologies for annotating and transforming software models into performance models.

This paper illustrates an industrial application of a performance validation methodology, which integrates software and resource models in a UML-RT development environment. The methodology is successfully applied on a synchronous digital hierarchy (SDH) telecommunication system modeled and simulated using the Rose Real Time toolset. The case study demonstrates the validity of the methodology, and it is an opportunity to refine the mechanisms of the methodology as well as to extend its scope. One of the key factors that the case study highlights is the concept of "transparency", i.e. the ability to apply a performance validation methodology without deeply modifying the software development process and environment.

Performance is a nonfunctional software attribute that plays a crucial role in wide application domains spreading from safety-critical systems to e-commerce applications. Software risk can be quantified as a combination of the probability that a software system may fail and the severity of the damages caused by the failure. In this paper, we devise a methodology for estimation of performance-based risk factor, which originates from violations, of performance requirements, (namely, performance failures). The methodology elaborates annotated UML diagrams to estimate the performance failure probability and combines it with the failure severity estimate which is obtained using the functional failure analysis. We are thus able to determine risky scenarios as well as risky software components, and the analysis feedback can be used to improve the software design. We illustrate the methodology on an e-commerce case study using step-by step approach, and then provide a brief description of a case study based on large real system.

The performance validation of software systems is becoming a crucial activity of the software development process. This is mostly due to the resource sharing and the remote deployment of software objects that may introduce critical delays in performance indices like the system response time. Hard and soft real-time systems are particularly affected from performance issues, therefore the ability to model and validate this attribute may become an extra value in software development environments. In this paper we introduce a framework to model performance aspects using the real-time object modeling (ROOM) notation. We devise a standard approach to represent hardware resources (such as CPUs and disks), to formulate resource requests of software objects, and to model the delays and the resource contentions that may arise from such requests. The integration of the software model and the resources is made transparent to the software developer by exploiting the integration features of the Rose Real Time (RRT) tool. RRT is based on the real time UML notation that is an implementation of ROOM. We also show an example of application of our framework on a video system case study.

An increasing number of applications operate in heterogeneous computing environments, often with mobile components. Methodologies that help developers assess the ability of such applications to meet their performance requirements throughout the software life-cycle are needed. In particular, early in the design phases, analysis techniques are critical for ensuring the future system's behavior, evaluating and comparing design alternatives. A performability evaluation is the most appropriate means to assess the expected system's ability to perform, including the effects of component failures and repairs. This paper focuses on model-based analysis of performability of mobile software systems. We propose a general methodology that starts from design artifacts expressed in a UML-based notation. Inferred performability models are based on the stochastic activity networks notation. The viability of the proposed approach is demonstrated through its application in a case study.

The importance of early performance assessment grows as software systems increase in terms of size, logical distribution and interaction complexity. Lack of time on the side of software developers, as well as distance between software model notations and performance model representation do not help to build an integrated software process that takes into account, from the early phases of the lifecycle, nonfunctional requirements. We work towards filling this gap by extending the capabilities of a simulative environment developed for the UML notation. Our intent is to introduce new stereotypes representing performance related items, such as resource types and job dispatchers. They allow the software designers to homogeneously represent a software architecture integrated with a running platform as well as parameterized with the resource demand that the components require.

It is generally believed that component-based software development leads to improved application quality, maintainability and reliability. However most software reliability techniques model integrated systems. These models disregard system's internal structure, taking into account only the failure data and interactions with the environment. We propose a novel approach to reliability analysis of component-based systems. Reliability prediction algorithm allows system architects to analyze reliability of the system before it is built, taking into account component reliability estimates and their anticipated usage. Fully integrated with the UML, this step can guide the process of identifying critical components and analyze the effect of replacing them with the more/less reliable ones. Reliability assessment algorithm, applicable in the system test phase, utilizes these reliability predictions as prior probabilities. In the Bayesian estimation. framework, posterior probability of failure is calculated from the priors and test failure data.

Discrete event simulation is a methodology to study the behavior of complex systems. Its drawback is that, in order to get reliable results, simulations usually have to be run over a long stretch of time. This time requirement could decrease through the usage of parallel or distributed computing systems. In this paper, we analyze the Time Warp synchronization protocol for parallel discrete event simulation and present an analytical model evaluating the upper bound on the completion time of a Time Warp simulation. In our analysis, we consider the case of a simulation model with homogeneous logical processes, where "homogeneous" means they have the same average event routine time and the same state saving cost. Then we propose a methodology to determine when it is time-convenient to use a Time Warp synchronized simulation, instead of a sequential one, for a simulation model with features matching those considered in our analysis. We give an answer to this question without the need to preliminary generate the simulation code. Examples of methodology usage are reported for the case of both a synthetic benchmark and a real world model.