Abstract As a software system evolves, its performance can improve or degrade over time. Performance evolution is especially delicate in configurable software systems, where performance degradation may manifest only for specific configurations, making it especially hard to spot and fix. Problem. Prior work concentrated mainly on performance-bug detection and root-cause analysis of a single version of a system. The big picture of how performance co-evolves with a system and what role configurability plays is largely unclear. Approach. In an empirical study, we investigate the relation between configurability and performance evolution. Specifically, we analyze a total of 190 releases of 12 configurable real-world systems and examine the extent to which performance changes are specific to particular configurations and whether few or many configuration options cause performance changes. We triangulate our findings by analyzing change logs and commit messages of the respective projects to pin down causes of performance changes. Results. We found that almost every release of every subject system exhibits performance changes in some of their configurations. Notably, the majority of performance changes affects only a subset of the configuration space, and most performance changes are triggered by multiple options (up to 6). In a deeper analysis, we found that a considerable number of releases mention performance changes in the change log and commits: performance changes are reported in $$45\%$$ 45 % and $$69\%$$ 69 % of the releases in the change log and the commit messages, respectively, but only a fraction report the involved configuration options.

Abstract In scientific computing, researchers often use feature-rich software frameworks to simulate physical, chemical, and biological processes. Commonly, researchers follow a clone-and-own approach: Copying the code of an existing, similar simulation and adapting it to the new simulation scenario. In this process, a user has to select suitable artifacts (e.g., classes) from the given framework and replaces the existing artifacts from the cloned simulation. This manual process incurs substantial effort and cost as scientific frameworks are complex and provide large numbers of artifacts. To support researchers in this area, we propose a lightweight API-based analysis approach, called VORM, that recommends appropriate artifacts as possible alternatives for replacing given artifacts. Such alternative artifacts can speed up performance of the simulation or make it amenable to other use cases, without modifying the overall structure of the simulation. We evaluate the practicality of VORM—especially, as it is very lightweight but possibly imprecise—by means of a case study on the DUNE numerics framework and two simulations from the realm of physical simulations. Specifically, we compare the recommendations by VORM with recommendations by a domain expert (a developer of DUNE). VORM recommended 34 out of the 37 artifacts proposed by the expert. In addition, it recommended 2 artifacts that are applicable but have been missed by the expert and 32 artifacts not recommended by the expert, which however are still applicable in the simulation scenario with slight modifications. Diving deeper into the results, we identified an undiscovered bug and an inconsistency in DUNE, which corroborates the usefulness of VORM.

Configurable software systems provide a multitude of configuration options to adjust and optimize their functional and non-functional properties. For instance, to find the fastest configuration for a given setting, a brute-force strategy measures the performance of all configurations, which is typically intractable. Addressing this challenge, state-of-the-art strategies rely on machine learning, analyzing only a few configurations (i.e., a sample set) to predict the performance of other configurations. However, to obtain accurate performance predictions, a representative sample set of configurations is required. Addressing this task, different sampling strategies have been proposed, which come with different advantages (e.g., covering the configuration space systematically) and disadvantages (e.g., the need to enumerate all configurations). In our experiments, we found that most sampling strategies do not achieve a good coverage of the configuration space with respect to covering relevant performance values. That is, they miss important configurations with distinct performance behavior. Based on this observation, we devise a new sampling strategy, called distance-based sampling, that is based on a distance metric and a probability distribution to spread the configurations of the sample set according to a given probability distribution across the configuration space. This way, we cover different kinds of interactions among configuration options in the sample set. To demonstrate the merits of distance-based sampling, we compare it to state-of-the-art sampling strategies, such as t-wise sampling, on 10 real-world configurable software systems. Our results show that distance-based sampling leads to more accurate performance models for medium to large sample sets.

Summary Many problems formulated in partial differential equations lead to stencil‐type structures after applying an appropriate structured discretization. On one hand, exploiting these stencil structures in simulations can lead to massive performance improvements, compared to forming a sparse matrix. On the other hand, the generality of the simulation is restricted, depending on the exact definition of the stencils. In this article, we discuss the variability of stencils in the domain of porous‐media applications and present a family of models that grows in complexity. To demonstrate the relation between equation and discretization on the resulting stencil used to simulate the equation, we consider 4 models from the porous media domain. This way, we describe the influence of design decisions made during the discretization on the shape of stencils, to give application engineers' information on the variability they have to consider. This leads us to 2 variability models that shall help application engineers to understand the complexity and choices of stencil computations in the porous media domain.

Summary Multigrid methods are among the most efficient algorithms for solving discretized partial differential equations. Typically, a multigrid system offers various configuration options to tune performance for different applications and hardware platforms. However, knowing the best performing configuration in advance is difficult, because measuring all multigrid system variants is costly. Instead of direct measurements, we use machine learning to predict the performance of the variants. Selecting a representative set of configurations for learning is nontrivial, although, but key to prediction accuracy. We investigate different sampling strategies to determine the tradeoff between accuracy and measurement effort. In a nutshell, we learn a performance‐influence model that captures the influences of configuration options and their interactions on the time to perform a multigrid iteration and relate this to existing domain knowledge. In an experiment on a multigrid system working on triangular grids, we found that combining pair‐wise sampling with the D‐Optimal experimental design for selecting a learning set yields the most accurate predictions. After measuring less than 1 % of all variants, we were able to predict the performance of all variants with an accuracy of 95.9 %. Furthermore, we were able to verify almost all knowledge on the performance behavior of multigrid methods provided by 2 experts.

For the analysis of highly configurable systems, analysis approaches need to take the inherent variability of these systems into account. The notion of presence conditions is central to such approaches. A presence condition specifies a subset of system configurations in which a certain artifact or a concern of interest is present (e.g., a defect associated with this subset). In this paper, we introduce and analyze the problem of presence-condition simplification. A key observation is that presence conditions often contain redundant information, which can be safely removed in the interest of simplicity and efficiency. We present a formalization of the problem, discuss application scenarios, compare different algorithms for solving the problem, and empirically evaluate the algorithms by means of a set of substantial case studies.

Almost every complex software system today is configurable. While configurability has many benefits, it challenges performance prediction, optimization, and debugging. Often, the influences of individual configuration options on performance are unknown. Worse, configuration options may interact, giving rise to a configuration space of possibly exponential size. Addressing this challenge, we propose an approach that derives a performance-influence model for a given configurable system, describing all relevant influences of configuration options and their interactions. Our approach combines machine-learning and sampling heuristics in a novel way. It improves over standard techniques in that it (1) represents influences of options and their interactions explicitly (which eases debugging), (2) smoothly integrates binary and numeric configuration options for the first time, (3) incorporates domain knowledge, if available (which eases learning and increases accuracy), (4) considers complex constraints among options, and (5) systematically reduces the solution space to a tractable size. A series of experiments demonstrates the feasibility of our approach in terms of the accuracy of the models learned as well as the accuracy of the performance predictions one can make with them.

In the recent past, the amount of high-dimensional data, such as feature vectors extracted from multimedia data, increased dramatically. A large variety of indexes have been proposed to store and access such data efficiently. However, due to specific requirements of a certain use case, choosing an adequate index structure is a complex and time-consuming task. This may be due to engineering challenges or open research questions. To overcome this limitation, we present QuEval, an open-source framework that can be flexibly extended w.r.t. index structures, distance metrics, and data sets. QuEval provides a unified environment for a sound evaluation of different indexes, for instance, to support tuning of indexes. In an empirical evaluation, we show how to apply our framework, motivate benefits, and demonstrate analysis possibilities.

SUMMARY Software is changed frequently during its life cycle. New requirements come, and bugs must be fixed. To update an application, it usually must be stopped, patched, and restarted. This causes time periods of unavailability, which is always a problem for highly available applications. Even for the development of complex applications, restarts to test new program parts can be time consuming and annoying. Thus, we aim at dynamic software updates to update programs at runtime. There is a large body of research on dynamic software updates, but so far, existing approaches have shortcomings either in terms of flexibility or performance. In addition, some of them depend on specific runtime environments and dictate the program's architecture. We present JavAdaptor, the first runtime update approach based on Java that (a) offers flexible dynamic software updates, (b) is platform independent, (c) introduces only minimal performance overhead, and (d) does not dictate the program architecture. JavAdaptor combines schema changing class replacements by class renaming and caller updates with Java HotSwap using containers and proxies. It runs on top of all major standard Java virtual machines. We evaluate our approach's applicability and performance in non‐trivial case studies and compare it with existing dynamic software update approaches. Copyright © 2012 John Wiley & Sons, Ltd.

Dynamic software updates (DSU) are one of the top-most features requested by developers and users. As a result, DSU is already standard in many dynamic programming languages. But, it is not standard in statically typed languages such as Java. Even if at place number three of Oracle's current request for enhancement (RFE) list, DSU support in Java is very limited. Therefore, over the years many different DSU approaches for Java have been proposed. Nevertheless, DSU for Java is still an active field of research, because most of the existing approaches are too restrictive. Some of the approaches have shortcomings either in terms of flexibility or performance, whereas others are platform dependent or dictate the program's architecture. With JavAdaptor, we present the first DSU approach which comes without those restrictions. We will demonstrate JavAdaptor based on the well-known arcade game Snake which we will update stepwise at runtime.