The SPEC SERT suite is the industry-standard benchmark for evaluating server energy efficiency and is widely adopted in government regulations and certification programs. Current certification rules require every CPU memory channel to be populated with at least one Dual In-line Memory Module (DIMM). In real-world deployments, however, servers are sometimes configured with fewer DIMMs, leaving some channels unpopulated. This discrepancy introduces a significant gap between certified efficiency scores and the actual efficiency of deployed systems. To address this issue, there are ongoing discussions about certifying servers with partially populated memory channels. However, the impact of memory configuration on SPEC SERT results has not been systematically studied. In this paper, we present a comprehensive analysis of how different memory configurations affect performance, power consumption, and energy efficiency on two state-of-the-art server systems using the SPEC SERT 2 suite. We vary both the number of populated channels and the type of DIMMs. Our experimental results show that server efficiency scores can be up to 3.4 times lower with only one channel populated and still up to 1.3 times lower with half the channels, compared to fully populated configurations. Detailed analysis of individual SPEC SERT 2 worklets reveals that some CPU worklets are highly sensitive to memory bandwidth, and that the impact of memory configuration is dependent on both server architecture and workload intensity. These findings underscore the need to reconsider certification criteria and highlight the importance of memory configuration for accurate energy efficiency assessment.

Earth observation (EO) scientists use sophisticated algorithms, large datasets, and high-performance data analytics (HPDA) clusters to develop and execute complex EO workflows. To improve portability and reusability within the domain, the Open Geospatial Consortium (OGC) published a set of best practices for developing EO workflows. Even though EO data products are often shared openly, representative EO workflow implementations are hardly available.In this paper, we contribute a case study on implementing and scaling a complex EO workflow, adhering to the OGC best practices and demonstrating its portability by deploying it both locally and on the HPDA platform "terrabyte" at the Leibniz Supercomputing Centre. Contentwise, the contributed workflow analyzes settlement patterns by first delineating coherent settlements derived from leveraged satellite data that maps builtup areas and subsequently calculating centrality measures to characterize the spatial arrangement and hierarchy of settlements in a given region.We demonstrate the workflow’s scalability and variance in resource demands by analyzing its time-to-result, total CPU time, and resource efficiency under different inputs and configurations. Interestingly, implicit parameters hidden in the input data semantics, like the number and area of settlements in the region of interest, significantly impact the time required to complete the processing. Concurrent processing is restricted to connected components of the settlement graph in the analysis stage, leading to an unbalanced workload distribution when analyzing large urban areas, showcasing the scalability challenges EO scientists face.

Microservice applications in practice often face runtime adaptations, like autoscaling or updates. These adaptations cause these applications to leave their steady-state performance, leading to so-called transient phases. There is a lack of understanding and quantitative data about the transient phases of microservice applications. Several use cases, including microbenchmarking and autoscaling, can profit from empirical data on transient phases. To this end, we present the most comprehensive empirical study of transient phases to date. We investigated 92 microservices from 10 reference applications, analyzing the influence of programming languages, workloads, and container properties on the duration of transient phases.

With the ever increasing importance of cloud computing and a strong focus on reliable data centers, a high amount of research has been done on failure prediction for hard disk drives. The collection of monitoring data, such as SMART statistics (Self-Monitoring, Analysis, and Reporting Technology) from operational HDDs, enables operators to obtain predictions about the expected remaining useful life. Numerous methods for HDD failure prediction have been published in recent years, and their evaluation has shown decent results. However, a naive splitting into training and test sets can lead to data leakage and, thus, over-optimistic results that cannot be achieved on the data of scientific interest. In this paper, we propose a novel data leakage measure for quantifying the amount of data leakage in training and test datasets. Further, we define four splitting techniques and evaluate our measure in terms of the performance optimism of classification models with respect to these different splitting strategies. Our results consistently show that splitting techniques prone to data leakage induce an overestimation of predictive performance. Overall, we were able to show the usefulness of the defined data leakage measure, as well as its connection with different splitting techniques and the performance optimism of prediction models.

Microservice applications are the building blocks of modern cloud applications. As such, their performance aspects have been receiving increasing attention in the software engineering community. However, many microservice performance studies use only a small set of popular microservice test applications for experiments, questioning the applicability of their approaches in practice. Researchers currently lack the opportunity to collect large and diverse datasets containing performance metrics of microservices. This is because popular test applications only represent specific technology stacks and often come with custom benchmark tooling (e.g., load generation and monitoring). In this paper, we present Creo, a framework for generating microservice applications that (1) are fully executable, (2) have configurable properties and resource usage profiles, and (3) have built-in support for standardized monitoring, load generation, and deployment. Our approach enables researchers to run experiments with diverse microservice applications with minimal effort. We demonstrate the value of our approach in the context of two use cases. First, we show that using generated applications when training machine learning models for predicting performance degradation can improve the prediction accuracy. Second, we evaluate a recent approach for performance anomaly classification on a set of generated applications highlighting strengths and weaknesses not discussed in the original work.

The processing of large amounts of data in central high performance data analytics (HPDA) systems is playing an increasingly important role in science and business. However, many HPDA systems exhibit a low utilization of their available resources during normal operation. An important reason for this underutilization is that too many resources are reserved for individual jobs. This is often a consequence of the common practice of reserving a uniform amount of resources such as CPU or memory for the entire execution time of a job. Given that many data intensive (DI) jobs consist of different phases with different resource demands, resources are normally reserved according to the demand of the most resource-intensive phase. This results in more resources being reserved over a long period of time than are actually needed.

Abstract The need for efficient and reliable logistics solutions has increased significantly in the last decade. Traffic forecasts are a promising source of information that can be used to improve the planning of delivery schedules. However, most existing traffic forecasting approaches only support a forecasting horizon of up to an hour, which is insufficient for per-day-based schedule planning. In this paper, we focus on short-term traffic forecasting for up to four hours. We first propose a data collection process integrating traffic speed, incidents, weather, and holiday information. We have used this process to collect real-world traffic data for 115 days. We then define and evaluate twelve models for vehicle traffic forecasting, including well-known time series forecasting approaches and state-of-the-art deep learning models. Our results show that the best model in our comparison improved the accuracy by approximately 30% compared to a naive forecaster that repeats the last known value. The evaluation also shows that LSTM-based approaches are competitive to state-of-the-art models. Overall, the proposed deep-learning-based models perform best while requiring a smaller input timeframe than statistical models.

As data storage and processing increasingly shift to the cloud, the risk of data breaches also rises. One way to address this is using Homomorphic Encryption (HE), which allows for data processing while the data remains encrypted, unlike traditional methods. However, current HE libraries support only addition and multiplication, requiring users to implement other mathematical functions themselves. To this end, we developed and analyzed basic mathematical functions in a previous work. Since polynomial approximations are more common in HE, this paper expands on that by examining and comparing polynomial approximations of these functions with the previously implemented methods. Our findings indicate that while polynomial approximations offer the benefit of low multiplication depth, the previously implemented methods generally outperform them in most scenarios despite their higher computational cost.

Autoscaling is a core capability in cloud computing with significant impact on service quality and cost. Modern applications, like microservices and serverless functions, consist of many containers that enable fine-grained, component-wise scaling. Effective autoscaling across large, heterogeneous service landscapes remains challenging. As cloud adoption increases, workloads have become more diverse, exhibiting highly variable request patterns, payload characteristics, and response time requirements. This limits the effectiveness of conventional autoscalers, whose fixed intervals and cooldown periods restrict responsiveness. At the same time, the growing number of services and frequent updates strain approaches based on predefined models, motivating more adaptive solutions.

In recent years, digital services have experienced significant growth, exemplified by platforms like Netflix achieving unprecedented revenue levels. Some of these services employ subscription models, with certain content requiring additional payments or offering third-party products. To ensure the widespread availability of diverse digital services anytime and anywhere, providers must have control over content accessibility. To address the multifaceted challenges in this domain, one promising solution is the adoption of attribute-based encryption (ABE). Over the years, various approaches have been proposed in the literature, offering a wide range of features. In a prior study [18], we assessed the security of one of these proposed approaches and identified one that did not meet its promised security standards. In this research we focuses on conducting a security analysis for another ABE scheme to pinpoint its shortcomings and emphasize the critical importance of evaluating the safety and effectiveness of newly proposed schemes. Specifically, we uncover an attack vector within this ABE scheme, which enables malicious users to decrypt content without the required permissions or attributes. Furthermore, we propose a solution to rectify this identified vulnerability.

A core selling point of application containers is their fast start times compared to other virtualization approaches like virtual machines. Predictable and fast container start times are crucial for improving and guaranteeing the performance of containerized cloud, serverless, and edge applications. While previous work has investigated container starts, there remains a lack of understanding of how start times may vary across container configurations. We address this shortcoming by presenting and analyzing a dataset of approximately 200,000 open-source Docker Hub images featuring different image configurations (e.g., image size and exposed ports). Leveraging this dataset, we investigate the start times of containers in two environments and identify the most influential features. Our experiments show that container start times can vary between hundreds of milliseconds and tens of seconds in the same environment. Moreover, we conclude that no single dominant configuration feature determines a container's start time, and hardware and software parameters must be considered together for an accurate assessment.

Autoscalers are indispensable parts of modern cloud deployments and determine the service quality and cost of a cloud application in dynamic workloads. The configuration of an autoscaler strongly influences its performance and is also one of the biggest challenges and showstoppers for the practical applicability of many research autoscalers. Many proposed cloud experiment methodologies can only be partially applied in practice, and many autoscaling papers use custom evaluation methods and metrics. This paper presents a practical guideline for obtaining meaningful and interpretable results on autoscaler performance with reasonable overhead. We provide step-by-step instructions for defining realistic usage behaviors and traffic patterns. We divide the analysis of autoscaler performance into a qualitative antipattern-based analysis and a quantitative analysis. To demonstrate the applicability of our guideline, we conduct several experiments with a microservice of our industry partner in a realistic test environment.

Container orchestration frameworks play a critical role in modern cloud computing paradigms such as cloud-native or serverless computing. They significantly impact the quality and cost of service deployment as they manage many performance-critical tasks such as container provisioning, scheduling, scaling, and networking. Consequently, a comprehensive performance assessment of container orchestration frameworks is essential. However, until now, there is no benchmarking approach that covers the many different tasks implemented in such platforms and supports evaluating different technology stacks. In this paper, we present a systematic approach that enables benchmarking of container orchestrators. Based on a definition of container orchestration, we define the core requirements and benchmarking scope for such platforms. Each requirement is then linked to metrics and measurement methods, and a benchmark architecture is proposed. With COFFEE, we introduce a benchmarking tool supporting the definition of complex test campaigns for container orchestration frameworks. We demonstrate the potential of our approach with case studies of the frameworks Kubernetes and Nomad in a self-hosted environment and on the Google Cloud Platform. The presented case studies focus on container startup times, crash recovery, rolling updates, and more.

Abstract In warehouses, order picking is known to be the most labor-intensive and costly task in which the employees account for a large part of the warehouse performance. Hence, many approaches exist, that optimize the order picking process based on diverse economic criteria. However, most of these approaches focus on a single economic objective at once and disregard ergonomic criteria in their optimization. Further, the influence of the placement of the items to be picked is underestimated and accordingly, too little attention is paid to the interdependence of these two problems. In this work, we aim at optimizing the storage assignment and the order picking problem within mezzanine warehouse with regards to their reciprocal influence. We propose a customized version of the Non-dominated Sorting Genetic Algorithm II (NSGA-II) for optimizing the storage assignment problem as well as an Ant Colony Optimization (ACO) algorithm for optimizing the order picking problem. Both algorithms incorporate multiple economic and ergonomic constraints simultaneously. Furthermore, the algorithms incorporate knowledge about the interdependence between both problems, aiming to improve the overall warehouse performance. Our evaluation results show that our proposed algorithms return better storage assignments and order pick routes compared to commonly used techniques for the following quality indicators for comparing Pareto fronts: Coverage, Generational Distance, Euclidian Distance, Pareto Front Size, and Inverted Generational Distance. Additionally, the evaluation regarding the interaction of both algorithms shows a better performance when combining both proposed algorithms.

The rapid growth of connected devices has led to the implementation of Internet of Things (IoT)-centric platforms designed to provide connectivity for machine-to-machine-type communication. Specifically, mobile virtual network operators (MVNOs) provide global service for IoT use cases by leveraging roaming in readily deployed physical networks. This global deployment of connected devices poses a significant challenge regarding the operation of centralized core networks with respect to dimensioning, scaling, as well as system survivability in case of signaling incidents. Overload control mechanisms for IoT mobile networks offer a proactive solution for mitigating excessive signaling traffic from IoT devices in mobile core networks. However, global scaling and a heterogeneous composition of devices present network operators with additional challenges. To address these challenges, this work presents a detailed, protocol-level simulation framework of a real-world IoT MVNO core network. We present both models for the expected IoT signaling load and the processing of signaling messages based on measurements in a live, production environment as well as a dedicated testbed. Finally, we present a case study on various overload mechanisms and identify critical performance characteristics to compare their performance. The results of this study categorize overload control mechanisms and shed light on the necessity for MVNOs to deal with the upcoming IoT traffic by defining appropriate overload control mechanisms in mobile core networks to ensure network survivability under unforeseen conditions.

In today’s world, circumstances, processes, and requirements for software systems are becoming increasingly complex. To operate properly in such dynamic environments, software systems must adapt to these changes, which has led to the research area of Self-Adaptive Systems (SAS). Platooning is one example of adaptive systems in Intelligent Transportation Systems, which is the ability of vehicles to travel with close inter-vehicle distances. This technology leads to an increase in road throughput and safety, which directly addresses the increased infrastructure needs due to increased traffic on the roads. However, the No-Free-Lunch theorem states that the performance of one adaptation planning strategy is not necessarily transferable to other problems. Moreover, especially in the field of SAS, the selection of the most appropriate strategy depends on the current situation of the system. In this article, we address the problem of self-aware optimization of adaptation planning strategies by designing a framework that includes situation detection, strategy selection, and parameter optimization of the selected strategies. We apply our approach on the case study platooning coordination and evaluate the performance of the proposed framework.

Applications on mobile devices face varying network conditions in cellular networks. The connected radio cell is often changing, especially with moving devices. Different access technologies, varying signal strengths, or distance to the connected radio tower influence the Quality of Service (QoS) of mobile applications. Existing technologies like buffering or adaptive video streaming work reactive, i.e., they react to a decreasing download bitrate. In contrast, these technologies and mobile applications in general could benefit from early knowledge of the expected connection quality.This work investigates the predictability of QoS metrics in cellular networks based on the experience of previous measurements. For this, we developed an Android app to measure download bitrates with minimal data consumption. We performed over 90 000 measurements using a single network operator and analyzed how precise QoS indicators like packet round trip times and download bitrates can be predicted. We developed a methodology to predict the expected download bitrate along a route and present our approach of aggregating measurements into hexagons of dynamic size. The core contributions of this work are (i) a methodology and implementation of systematic measurement data collection, (ii) an open data publication of our measurement data set, and (iii) an approach for predicting QoS metrics in cellular networks based on aggregated measurements. Our results show, that our approach is able to predict the downlink bitrate, the packet round trip time (ping), or DNS query duration along a given route.

The critical component in any IoT application is the communication between devices. This must not only function smoothly, but also be secured. An important step in securing IoT communication is its encryption. However, in order for IoT devices to encrypt their communication with each other, they must first agree on appropriate cryptographic keys. In practice, the generation and distribution of such keys is usually managed by a central authority. However, this centralized approach has the disadvantages that (i) a central authority must be trusted, (ii) the central authority represents a single point of failure, and (iii) the central authority may be far away and thus communication with it takes a long time. To overcome these drawbacks, distributed group key agreement approaches have also been proposed. Since these distributed approaches were not originally developed for IoT devices, their performance on such devices is unknown. Therefore, in this work, we investigate the performance of a distributed group encryption scheme on IoT devices. To this end, we have built a measurement environment for distributed group encryption schemes and compare centralized and distributed group encryption schemes for IoT. Our measurements show that under perfect network conditions, the distributed approach performs worse than the centralized approaches in terms of time and memory requirements. However, our measurements also show that the distributed approach allows a group of 5 members to agree on a key in less than a minute. Thus, the distributed method can be used for small IoT groups if agreeing on a key is not time-sensitive.

Benchmarking is a core element in the toolbox of most systems researchers and is used for analyzing, comparing, and validating complex systems. In the quest for reliable benchmark results, a consensus has formed that a significant experiment must be based on multiple runs. To interpret these runs, mean and standard deviation are often used. In case of experiments where each run produces a time series, applying and comparing the mean is not easily applicable and not necessarily statistically sound. Such an approach ignores the possibility of significant differences between runs with a similar average. In order to verify this hypothesis, we conducted a survey of 1,112 publications of selected performance engineering and systems conferences canvassing open data sets from performance experiments. The identified 3 data sets purely rely on average and standard deviation. Therefore, we propose a novel analysis approach based on similarity analysis to enhance the reliability of performance evaluations. Our approach evaluates 12 (dis-)similarity measures with respect to their applicability in analysing performance measurements and identifies four suitable similarity measures. We validate our approach by demonstrating the increase in reliability for the data sets found in the survey.

Autoscaling is a task of major importance in the cloud computing domain as it directly affects both operating costs and customer experience. Although there has been active research in this area for over ten years now, there is still a significant gap between the proposed methods in the literature and the deployed autoscalers in practice. Hence, many research autoscalers do not find their way into production deployments. This paper describes six core challenges that arise in production systems that are still not solved by most research autoscalers. We illustrate these problems through experiments in a realistic cloud environment with a real-world multi-service business application and show that commonly used autoscalers have various shortcomings. In addition, we analyze the behavior of overloaded services and show that these can be problematic for existing autoscalers. Generally, we analyze that these challenges are only insufficiently addressed in the literature and conclude that future scaling approaches should focus on the needs of production systems.

Abstract In the last decades, the classical Vehicle Routing Problem (VRP), i.e., assigning a set of orders to vehicles and planning their routes has been intensively researched. As only the assignment of order to vehicles and their routes is already an NP-complete problem, the application of these algorithms in practice often fails to take into account the constraints and restrictions that apply in real-world applications, the so called rich VRP (rVRP) and are limited to single aspects. In this work, we incorporate the main relevant real-world constraints and requirements. We propose a two-stage strategy and a Timeline algorithm for time windows and pause times, and apply a Genetic Algorithm (GA) and Ant Colony Optimization (ACO) individually to the problem to find optimal solutions. Our evaluation of eight different problem instances against four state-of-the-art algorithms shows that our approach handles all given constraints in a reasonable time.

In the recent past, platooning evolved into an attractive cooperative driving technology, broadly discussed in research and practice. Vehicles in platoons use cooperative adaptive cruise control to drive at close distances to each other. Platooning (i) increases the capacity of the street by a factor of 2; (ii) reduces the fuel consumption and emissions by up to 20%; and (iii) has social implications as it increases driver comfort and safety. As platooning research progresses, platooning coordination becomes a major research focus. The coordination of platoons, including the assignment of vehicles to platoons, the management of inter- and intra-platoon interactions, and the coordination of interactions with other vehicles is an important step towards an effective usage of platooning in practice. Based on a literature review of 1,600 papers, this survey provides an overview of state of the art in platooning coordination research for both cars and trucks. In this paper, we present a novel taxonomy for platooning coordination and classify existing approaches. We use the results of the literature review to discuss challenges and outline avenues for future work such as multi-objectiveness and individualisation.

The growing size of today’s data centers and the expectation of 24/7 availability continuously increase the complexity of hardware administration. To this end, the Self-Monitoring, Analysis, and Reporting Technology has been developed to provide insights into the health state of hard disk drives. Many approaches to predicting hard disk drive failures based on such monitoring data have been proposed in recent years. Nevertheless, most approaches consider this problem only as a static task, i.e., they train a static machine learning model on a given training set and evaluate its performance on a test set. However, due to model aging and changes in failure patterns, previously learned prediction models must be updated during runtime, which requires a time-dependent evaluation. Therefore, we present four machine learning model updating strategies, build multiple models for hard disk drive failure prediction using four machine learning algorithms, and compare the prediction quality of the different model update strategies and machine learning algorithms. Experimental results using a real-world data set of hard disk drives demonstrate the need for model update strategies, with XGBoost using the Hoeffding bound update trigger achieving the overall best prediction performance concerning prediction quality and number of updates required.

Predictive maintenance is an essential aspect of the concept of Industry 4.0. In contrast to previous maintenance strategies, which plan repairs based on periodic schedules or threshold values, predictive maintenance is normally based on estimating the time-to-failure of machines. Thus, predictive maintenance enables a more efficient and effective maintenance approach. Although much research has already been done on time-to-failure prediction, most existing works provide only specialized approaches for specific machines. In most cases, these are either rotary machines (i.e., bearings) or lithium-ion batteries. To bridge the gap to a more general time-to-failure prediction, we propose a generic end-to-end predictive maintenance methodology for the time-to-failure prediction of industrial machines. Our methodology exhibits a number of novel aspects including a universally applicable method for feature extraction based on different types of sensor data, well-known feature transformation and selection techniques, adjustable target class assignment based on fault records with three different labeling strategies, and the training of multiple state-of-the-art machine learning classification models including hyperparameter optimization. We evaluated our time-to-failure prediction methodology in a real-world case study consisting of monitoring data gathered over several years from a large industrial press. The results demonstrated the effectiveness of the proposed methodology for six different time-to-failure pre-diction windows, as well as for the downscaled binary prediction of impending failures. In this case study, the multi-class feed-forward neural network model achieved the overall best results.

In the next few years, both the number of IoT devices and the performance of quantum computers will increase. Both technologies pose a challenge to our current crypto-strategies. Therefore, post-quantum n-to-n communication encryption is a crucial field of research. Here, the development of new schemes and the analysis, and comparison of existing schemes is necessary. However, current work only investigates the performance of post-quantum schemes only for 1-to-1 communication. Therefore, in this paper, we analyze existing post-quantum schemes concerning n-to-n communication and compare them with pre-quantum schemes. Our results show that the pre-quantum schemes perform better regarding computation times than the post-quantum schemes, but the differences are sometimes only marginal. However, these marginal differences in computation times lead to the lower energy efficiency of the post-quantum schemes. In terms of features, there is no difference between both scheme classes. We show that the post-quantum schemes require unicast, whereas some pre-quantum schemes also support broadcast. Deciding whether to use pre- or post-quantum schemes for n-to-n encryption in IoT use cases depends on (i) whether energy efficiency is essential – e.g., in case of limited power supply – and (ii) whether unicast or broadcast is available.

Quantum Computing threatens security of today’s systems. Confidence in today’s security technologies largely relies on Public Key Cryptography (PKC), which depends on computational difficulty of mathematical problems that cannot be solved efficiently using any technology available today. This will, however, change once a sufficiently capable quantum computer will become available. Similarly, security of symmetric crypto algorithms will also be substantially weakened. Current progress in research proves that Quantum Computing is no longer science fiction. Hence, research and development of post-quantum cryptographic algorithms that can withstand attacks in Quantum Computing era are of paramount importance. This paper complements existing research in this domain with a performance analysis of a post-quantum cryptosystem capable of encrypting and decrypting messages either bit-wise or string-wise. Specifically, we describe a workflow for implementing the scheme, design a reproducible hardware performance evaluation testbed for an IoT and an online shopping scenario, define performance metrics, and perform performance evaluation case studies. Our performance analysis shows that bit-wise encryption and decryption and the corresponding key generation fits resource-constrained IoT microcontrollers as well as on average laptops. The encryption and decryption of a bit each take less than 30 ms and the key generation less than 300 ms.

With the rising demand for information technology comes an increase in energy consumption to power it. Namely, cloud computing is constantly growing, in part due to a rising number of devices using the cloud to provide certain functionality. This growth leads to an increase in energy consumption in data centers and is estimated to climb to over 1PWh in 2030. Hardware manufacturers counter the rising demand for energy in cloud data centers by providing techniques to make servers more energy-efficient. However, the advances cannot fully compensate for the growth. To further increase energy efficiency, software needs to be addressed as well but is often neglected by the developers.In this paper, we compare six sorting algorithms, a common task in most programs, against each other in terms of energy efficiency to allow developers to select the best solution to their problem. We selected well-known algorithms in two variants, and two implementation languages, C and Python. We ran each algorithm on two, out of four, state-of-the-art server systems with different CPUs.

Serverless functions are an emerging cloud computing paradigm that is being rapidly adopted by both industry and academia. In this cloud computing model, the provider opaquely handles resource management tasks such as resource provisioning, deployment, and auto-scaling. The only resource management task that developers are still in charge of is selecting how much resources are allocated to each worker instance. However, selecting the optimal size of serverless functions is quite challenging, so developers often neglect it despite its significant cost and performance benefits. Existing approaches aiming to automate serverless functions resource sizing require dedicated performance tests, which are time-consuming to implement and maintain.

In many areas of decision making, forecasting is an essential pillar. Consequently, there are many different forecasting methods. According to the "No-Free-Lunch Theorem", there is no single forecasting method that performs best for all time series. In other words, each method has its advantages and disadvantages depending on the specific use case. Therefore, the choice of the forecasting method remains a mandatory expert task. However, expert knowledge cannot be fully automated. To establish a level playing field for evaluating the performance of time series forecasting methods in a broad setting, we propose Libra, a forecasting benchmark that automatically evaluates and ranks forecasting methods based on their performance in a diverse set of evaluation scenarios. The benchmark comprises four different use cases, each covering 100 heterogeneous time series taken from different domains. The data set was assembled from publicly available time series and was designed to exhibit much higher diversity than existing forecasting competitions. Based on this benchmark, we perform a comprehensive evaluation to compare different existing time series forecasting methods.

Application performance management (APM) tools are useful to observe the performance properties of an application during production. However, APM is normally purely reactive, that is, it can only report about current or past performance degradation. Although some approaches capable of predictive application monitoring have been proposed, they can only report a predicted degradation but cannot explain its root-cause, making it hard to prevent the expected degradation.

The interconnectivity of devices on the Internet of Things (IoT) provides many new and smart applications. However, the integration of many devices - especially by inexperienced users - might introduce several security threats. Further, several often used communication protocols in the IoT domain are not out-of-the-boxsecured. On the other hand, security inherently introduces overhead, resulting in a decrease in performance. The Message QueuingTelemetry Transport (MQTT) protocol is a popular communication protocol for IoT applications - for example, in Industry 4.0, railways, automotive, or smart homes. This paper analyzes the influence on performance when using MQTT with TLS in terms of throughput, connection build-up times, and energy efficiency using a reproducible testbed based on a standard off-the-shelf microcontroller. The results indicate that the impact of TLS on performance across all QoS levels depends on (i) the network situation and (ii) the connection reestablishment frequency. Thus, a negative influence of TLS on the performance is noticeable only in deteriorated networksituations or at a high connection reestablishment frequency.

The number of devices connected to the Internet of Things (IoT) is continuously increasing to several billion nowadays. As those devices often share sensitive data, encryption of those data is an important issue. The pure volume of data and the complexity of communication patterns increases, and, accordingly, the importanceof group encryption is recently gaining more importance. Still, the choice of the best-suited group encryption scheme for a specific application is complicated. Benchmarks can support this choice. However, while literature distinguishes three categories for theschemes (central, decentral, and hybrid), a one-fits-all benchmark seems challenging to achieve. In this paper, we go the first step towards a structured benchmark for group encryption schemes by presenting a benchmark for centralized group encryption schemes in an IoT scenario. To this end, our benchmark includes a descriptionof workloads, a baseline scheme, a measurement setup, and metrics while also considering the requirements and features of centralized group encryption schemes.

Innovation needs a competitive and fair playing field on which products can be compared and informed choices can be made. Standard benchmarks are a necessity to create such a level playing field among competitors in the server market for more energy-efficient servers. That, in turn, motivates their engineers to design more energy-efficient hardware. The SPECpower_ssj 2008 benchmark drove the increase of server energy efficiency by 113 times for single CPU servers, or 19 times on average. Yet, with added functionality and load, they are expected to consume a rising amount of energy. Additionally, server usage in data centers has changed over time with new application types. To continue the effort of increasing server energy efficiency, a new version, SPECpowerNext, is under development. In this work, after a short introduction to SPECpower_ssj 2008, we present the new implementation of SPECpowerNext together with the standardized way to collect server information in heterogeneous data centers. We also give insight, including preliminary measurements, into two of SPECpowerNext new workloads, the Wiki and the APA workload, in addition to an overview of both workloads.

Resource demands are crucial parameters for modeling and predicting the performance of software systems. Currently, resource demand estimators are usually executed once for system analysis. However, the monitored system, as well as the resource demand itself, are subject to constant change in runtime environments. These changes additionally impact the applicability, the required parametrization as well as the resulting accuracy of individual estimation approaches. Over time, this leads to invalid or outdated estimates, which in turn negatively influence the decision-making of adaptive systems. In this article, we present SARDE , a framework for self-adaptive resource demand estimation in continuous environments. SARDE dynamically and continuously tunes, selects, and executes an ensemble of resource demand estimation approaches to adapt to changes in the environment. This creates an autonomous and unsupervised ensemble estimation technique, providing reliable resource demand estimations in dynamic environments. We evaluate SARDE using two realistic datasets. One set of different micro-benchmarks reflecting different possible system states and one dataset consisting of a continuously running application in a changing environment. Our results show that by continuously applying online optimization, selection and estimation, SARDE is able to efficiently adapt to the online trace and reduce the model error using the resulting ensemble technique.

Tor is a widely used anonymization network. Traffic is routed over different relay nodes to conceal the communication partners. However, if a single relay handles too much traffic, de-anonymization attacks are possible. The Tor Load Balancing Mechanism (TLBM) is responsible for balanced and secure load distribution. It must verify that relays cannot attract more traffic than they should by lying about their self-reported bandwidth. This work shows that the current bandwidth measurement method used for bandwidth verification is not suitable to verify the bandwidth of many relays. Most importantly, multiple measurements of high-bandwidth relays are uncorrelated to each other. Furthermore, we analyze the current load distribution in Tor. We show that the current load distribution reduces the resources necessary for several large-scale de-anonymization attacks by more than 80%. Additionally, as Tor favors fast relays during path selection, verifiable relays only handle a small fraction of Tor’s traffic. More precisely, we show that only 7.21% of all paths consist of entry and exit relays verifiable by measurements. We discuss these results’ security implications and argue that future TLBM research should focus at least as much on secure load distribution as on high traffic performance.

The complexity of managing the capacities of large IT infrastructures is constantly increasing as more network devices are connected.This task can no longer be performed manually, so the system must be monitored at runtime and estimations of future conditions must be made automatically.However, since using a single forecasting method typically performs poorly, this paper presents a framework for forecasting univariate network device workload traces using multiple forecasting methods.First, the time series are preprocessed by imputing missing data and removing anomalies.Then, different features are derived from the univariate time series, depending on the type of forecasting method.In addition, a recommendation approach for selecting the most suitable forecasting method from this set of algorithms for each time series based only on its historical values is proposed.For this purpose, the performance of the forecasting methods is approximated using the historical data of the respective time series under consideration.The framework is used in the FedCSIS 2020 Challenge and shows good forecasting quality with an average R 2 score of 0.2575 on the small test data set.

One central problem of machine learning is the inherent limitation to predict only what has been learned -stationarity. Any time series property that eludes stationarity poses a challenge for the proper model building. Furthermore, existing forecasting methods lack reliable forecast accuracy and time-to-result if not applied in their sweet spot. In this paper, we propose a fully automated machine learning-based forecasting approach. Our Telescope approach extracts and transforms features from an input time series and uses them to generate an optimized forecast model. In a broad competition including the latest hybrid forecasters, established statistical, and machine learning-based methods, our Telescope approach shows the best forecast accuracy coupled with a lower and reliable time-to-result.

Correctly configuring a distributed database management system (DBMS) deployed in a cloud environment for maximizing performance poses many challenges to operators. Even if the entire configuration spectrum could be measured directly, which is often infeasible due to the multitude of parameters, single measurements are subject to random variations and need to be repeated multiple times. In this work, we propose Baloo, a framework for systematically measuring and modeling different performance-relevant configurations of distributed DBMS in cloud environments. Baloo dynamically estimates the required number of measurement configurations, as well as the number of required measurement repetitions per configuration based on a desired target accuracy. We evaluate Baloo based on a data set consisting of 900 DBMS configuration measurements conducted in our private cloud setup. Our evaluation shows that the highly configurable framework is able to achieve a prediction error of up to 12 %, while saving over 80 % of the measurement effort. We also publish all code and the acquired data set to foster future research.

New emerging technologies, such as autonomous driving, intelligent buildings, and smart cities, are promising to revolutionize user experience and offer new services. The world has to undergo large scale deployment of billions of things - cost-efficient intelligent sensors that will be interconnected into extensive networks and will collect and supply data to intelligent algorithms - to make it happen. To date, however, it is challenging to secure such an infrastructure for many-fold reasons, such as resource constraints of things, large scale deployment, many-to-many communication patterns, and dynamically changing communication groups. All these factors rule out most of the state-of-the-art encryption and key-management techniques. Group encryption algorithms are well-suitable for many-to-many communication patterns typical for IoT networks, and many of them can deal with dynamic groups. There are, however, very few constructions that could potentially fulfill the computational and storage constraints of IoT devices while providing sufficient scalability for large networks. The promising candidates, such as construction by Nishat et al. [1], were not evaluated using IoT platforms and under constraints typical for IoT networks. In this paper, we aim to fill this gap and present the evaluation of a state-of-the-art group-oriented encryption scheme by Nishat et al. to identify its applicability to IoT systems. In detail, we provide a measurement workflow, a revised version of the approach, and describe a reproducible hardware testbed. Using this evaluation environment, we analyze the performance of the encryption scheme in a typical IoT scenario from a group member perspective. The results show that all calculation times can be assumed to be constant and are always below 2 seconds. The memory requirement for permanent parameters can also be considered to be constant and are below 8.5 kbit in each case. However, the information that has to be stored temporarily for group updates has turned out to be the bottleneck of the scheme, since their memory requirements increase linearly with the group size.

Due to the fast-paced and changing demands of their users, computing systems require autonomic resource management. To enable proactive and accurate decision-making for changes causing a particular overhead, reliable forecasts are needed. In fact, choosing the best performing forecasting method for a given time series scenario is a crucial task. Taking the "No-Free-Lunch Theorem" into account, there exists no forecasting method that performs best on all types of time series. To this end, we propose an automated approach that (i) extracts characteristics from a given time series, (ii) selects the best-suited machine learning method based on recommendation, and finally, (iii) performs the forecast. Our approach offers the benefit of not relying on a single method with its possibly inaccurate forecasts. In an extensive evaluation, our approach achieves the best forecasting accuracy.

Function-as-a-Service (FaaS) platforms enable users to run arbitrary functions without being concerned about operational issues, while only paying for the consumed resources. Individual functions are often composed into workflows for complex tasks. However, the pay-per-use model and nontransparent reporting by cloud providers make it challenging to estimate the expected cost of a workflow, which prevents informed business decisions. Existing cost-estimation approaches assume a static response time for the serverless functions, without taking input parameters into account. In this paper, we propose a methodology for the cost prediction of serverless workflows consisting of input-parameter sensitive function models and a monte-carlo simulation of an abstract workflow model. Our approach enables workflow designers to predict, compare, and optimize the expected costs and performance of a planned workflow, which currently requires time-intensive experimentation. In our evaluation, we show that our approach can predict the response time and output parameters of a function based on its input parameters with an accuracy of 96.1%. In a case study with two audio-processing workflows, our approach predicts the costs of the two workflows with an accuracy of 96.2%.

Modern distributed systems and Internet-of-Things applications are governed by fast living and changing requirements. Moreover, they have to struggle with huge amounts of data that they create or have to process. To improve the self-awareness of such systems and enable proactive and autonomous decisions, reliable time series forecasting methods are required. However, selecting a suitable forecasting method for a given scenario is a challenging task. According to the “No-Free-Lunch Theorem,” there is no general forecasting method that always performs best. Thus, manual feature engineering remains to be a mandatory expert task to avoid trial and error. Furthermore, determining the expected time-to-result of existing forecasting methods is a challenge. In this article, we extensively assess the state-of-the-art in time series forecasting. We compare existing methods and discuss the issues that have to be addressed to enable their use in a self-aware computing context. To address these issues, we present a step-by-step approach to fully automate the feature engineering and forecasting process. Then, following the principles from benchmarking, we establish a level-playing field for evaluating the accuracy and time-to-result of automated forecasting methods for a broad set of application scenarios. We provide results of a benchmarking competition to guide in selecting and appropriately using existing forecasting methods for a given self-aware computing context. Finally, we present a case study in the area of self-aware data-center resource management to exemplify the benefits of fully automated learning and reasoning processes on time series data.

Nowadays, in order to keep track of the fast changing requirements of Internet applications, auto-scaling is used as an essential mechanism for adapting the number of provisioned resources to the resource demand. The straightforward approach is to deploy a set of common and opensource single-service auto-scalers for each service independently. However, this deployment leads to problems such as bottleneck-shifting and increased oscillations. Existing auto-scalers that scale applications consisting of multiple services are kept closed-source. To face these challenges, we first survey existing auto-scalers and highlight current challenges. Then, we introduce Chamulteon, a redesign of our previously introduced mechanism, which can scale applications consisting of multiple services in a coordinated manner. We evaluate Chamulteon against four different well-cited auto-scalers in four sets of measurement-based experiments where we use diverse environments (VM vs. Docker), real-world traces, and vary the scale of the demanded resources. Overall, Chamulteon achieves the best auto-scaling performance based on established user-oriented and endorsed elasticity metrics.

Performance predictions enable software architects to optimize the performance of a software system early in the development cycle. Architectural performance models and statistical response time models are commonly used to derive these performance predictions. However, both methods have significant downsides: Statistical response time models can only predict scenarios for which training data is available, making the prediction of previously unseen system configurations infeasible. In contrast, the time required to simulate an architectural performance model increases exponentially with both system size and level of modeling detail, making the analysis of large, detailed models challenging. Existing approaches use statistical response time models in architectural performance models to avoid modeling subsystems that are difficult or time-consuming to model, yet they do not consider simulation time. In this paper, we propose to model software systems using classical queuing theory and statistical response time models in parallel. This approach allows users to tailor the model for each analysis run, based on the performed adaptations and the requested performance metrics. Our approach enables faster model solution compared to traditional performance models while retaining their ability to predict previously unseen scenarios. In our experiments we observed speedups of up to 94.8%, making the analysis of much larger and more detailed systems feasible.

Architectural performance models are a common approach to predict the performance properties of a software system. Parametric dependencies, which describe the relation between the input parameters of a component and its performance properties, significantly increase the prediction accuracy of architectural performance models. However, manually modeling parametric dependencies is time-intensive and requires expert knowledge. Existing automated extraction approaches require dedicated performance tests, which are often infeasible. In this paper, we introduce an approach to automatically identify parametric dependencies from monitoring data using feature selection techniques from the area of machine learning. We evaluate the applicability of three techniques selected from each of the three groups of feature selection methods: a filter method, an embedded method, and a wrapper method. Our evaluation shows that the filter technique outperforms the other approaches. Based on these results, we apply this technique to a distributed micro-service web-shop, where it correctly identifies 11 performance-relevant dependencies, achieving a precision of 91.7% based on a manually labeled gold-standard.

Today, software operation engineers rely on application key performance indicators (KPIs) for sizing and orchestrating cloud resources dynamically. KPIs are monitored to assess the achievable performance and to configure various cloud-specific parameters such as flavors of instances and autoscaling rules, among others. Usually, keeping KPIs within acceptable levels requires application expertise which is expensive and can slow down the continuous delivery of software. Expertise is required because KPIs are normally based on application-specific quality-of-service metrics, like service response time and processing rate, instead of generic platform metrics, like those typical across various environments (e.g., CPU and memory utilization, I/O rate, etc.)

Attacks on software systems are becoming more and more frequent, aggressive and sophisticated. With the changing threat landscape, in 2018, organizations are looking at when they will be attacked, not if. Intrusion Detection Systems (IDSs) can help in defending against these attacks. The systems that host IDSs require extensive computing resources as IDSs tend to detect attacks under overloaded conditions wrongfully. With the end of Moore's law and the growing adoption of Internet of Things, designers of security systems can no longer expect processing power to keep up the pace with them. This limitation requires ways to increase the performance of these systems without adding additional compute power. In this work, we present two dynamic and a static approach to bypass IDS for traffic deemed benign. We provide its prototype implementation and evaluate our solution. Our evaluation shows promising results. Performance is increased up to the level of a system without an IDS. Attack detection is within the margin of error from the 100% rate. However, our findings show that dynamic approaches perform best when using software switches. The use of a hardware switch reduces the detection rate and performance significantly.

Data center providers and server operators try to reduce the power consumption of their servers. Finding an energy efficient server for a specific target application is a first step in this regard. Estimating the power consumption of an application on an unavailable server is difficult, as nameplate power values are generally overestimations. Offline power models are able to predict the consumption accurately, but are usually intended for system design, requiring very specific and detailed knowledge about the system under consideration.

General Purpose Graphics Processing Units (GPGPUs) are becoming more and more common in current servers and data centers, which in turn consume a significant amount of electrical power. Measuring and benchmarking this power consumption is important as it helps with optimization and selection of these servers. However, benchmarking and comparing the energy efficiency of GPGPU workloads is challenging as standardized workloads are rare and standardized power and efficiency measurement methods and metrics do not exist. In addition, not all GPGPU systems run at maximum load all the time. Systems that are utilized in transactional, request driven workloads, for example, can run at lower utilization levels. Existing benchmarks for GPGPU systems primarily consider performance and are intended only to run at maximum load. They do not measure performance or energy efficiency at other loads. In turn, server energy-efficiency benchmarks that consider multiple load levels do not address GPGPUs.

As the Internet of Things (IoT) continues to gain traction in telecommunication networks, a very large number of devices are expected to be connected and used in the near future. In order to appropriately plan and dimension the network, as well as the back-end cloud systems and the resulting signaling load, traffic models are employed. These models are designed to accurately capture and predict the properties of IoT traffic in a concise manner. To achieve this, Poisson process approximations, based on the Palm–Khintchine theorem, have often been used in the past. Due to the scale (and the difference in scales in various IoT networks) of the modeled systems, the fidelity of this approximation is crucial, as, in practice, it is very challenging to accurately measure or simulate large-scale IoT deployments. The main goal of this paper is to understand the level of accuracy of the Poisson approximation model. To this end, we first survey both common IoT network properties and network scales as well as traffic types. Second, we explain and discuss the Palm–Khintiche theorem, how it is applied to the problem, and which inaccuracies can occur when using it. Based on this, we derive guidelines as to when a Poisson process can be assumed for aggregated periodic IoT traffic. Finally, we evaluate our approach in the context of an IoT cloud scaler use case.

Auto-scalers for clouds promise stable service quality at low costs when facing changing workload intensity. The major public cloud providers provide trigger-based auto-scalers based on thresholds. However, trigger-based auto-scaling has reaction times in the order of minutes. Novel auto-scalers from literature try to overcome the limitations of reactive mechanisms by employing proactive prediction methods. However, the adoption of proactive auto-scalers in production is still very low due to the high risk of relying on a single proactive method. This paper tackles the challenge of reducing this risk by proposing a new hybrid auto-scaling mechanism, called Chameleon, combining multiple different proactive methods coupled with a reactive fallback mechanism. Chameleon employs on-demand, automated time series-based forecasting methods to predict the arriving load intensity in combination with run-time service demand estimation to calculate the required resource consumption per work unit without the need for application instrumentation. We benchmark Chameleon against five different state-of-the-art proactive and reactive auto-scalers one in three different private and public cloud environments. We generate five different representative workloads each taken from different real-world system traces. Overall, Chameleon achieves the best scaling behavior based on user and elasticity performance metrics, analyzing the results from 400 hours aggregated experiment time.

In the Tor anonymity network, the distribution of topology information relies on the correct behavior of five out of the nine trusted directory authority servers. This centralization is concerning since a powerful adversary might compromise these servers and conceal information about honest nodes, leading to the full de-anonymization of all Tor users. Our work aims at distributing the work of these trusted authorities, such increasing resilience against attacks on core infrastructure components of the Tor network. In particular, we leverage several emerging technologies, such as blockchains, smart contracts, and trusted execution environments to design and prototype a system called SmarTor. This system replaces the directory authorities with a smart contract and a distributed network of untrusted entities responsible for bandwidth measurements. We prototyped SmarTor using Ethereum smart contracts and Intel SGX secure hardware. In our evaluation, we show that SmarTor produces significantly more reliable and precise measurements compared to the current measurement system. Overall, our solution improves the decentralization of the Tor network, reduces trust assumptions and increases resilience against powerful adversaries like law enforcement and intelligence services.

Model-based performance analysis can be leveraged to explore performance properties of software systems. To capture the behavior of varying workload mixes, configurations, and deployments of a software system requires formal modeling of the impact of configuration parameters and user input on the system behavior. Such influences are represented as parametric dependencies in software performance models. Existing modeling approaches focus on modeling parametric dependencies at design-time. This paper identifies runtime specific parametric dependency features, which are not supported by existing work. Therefore, this paper proposes a novel modeling methodology for parametric dependencies and a corresponding graph-based resolution algorithm. This algorithm enables the solution of models containing component instance-level dependencies, variables with multiple descriptions in parallel, and correlations modeled as parametric dependencies. We integrate our work into the Descartes Modeling Language (DML), allowing for accurate and efficient modeling and analysis of parametric dependencies. These performance predictions are valuable for various purposes such as capacity planning, bottleneck analysis, configuration optimization and proactive auto-scaling. Our evaluation analyzes a video store application. The prediction for varying language mixes and video sizes shows a mean error below 5% for utilization and below 10% for response time.

Modern distributed applications offer complex performance behavior and many degrees of freedom regarding deployment and configuration. Researchers employ various methods of analysis, modeling, and management that leverage these degrees of freedom to predict or improve non-functional properties of the software under consideration. In order to demonstrate and evaluate their applicability in the real world, methods resulting from such research areas require test and reference applications that offer a range of different behaviors, as well as the necessary degrees of freedom. Existing production software is often inaccessible for researchers or closed off to instrumentation. Existing testing and benchmarking frameworks, on the other hand, are either designed for specific testing scenarios, or they do not offer the necessary degrees of freedom. Further, most test applications are difficult to deploy and run, or are outdated. In this paper, we introduce the TeaStore, a state-of-the-art micro-service-based test and reference application. TeaStore offers services with different performance characteristics and many degrees of freedom regarding deployment and configuration to be used as a benchmarking framework for researchers. The TeaStore allows evaluating performance modeling and resource management techniques; it also offers instrumented variants to enable extensive run-time analysis. We demonstrate TeaStore's use in three contexts: performance modeling, cloud resource management, and energy efficiency analysis. Our experiments show that TeaStore can be used for evaluating novel approaches in these contexts and also motivates further research in the areas of performance modeling and resource management.

Nowadays, to keep track with the fast changing requirements of internet applications, auto-scaling is an essential mechanism for adapting the number of provisioned resources to the resource demand. In the context of public clouds, there exist different natures of cost-models for charging resources. However, the accounted resource units and charged resource units may differ significantly due to the applied cost model. This can lead to a significant increase of charged costs when using an auto-scaler as it tries to match the demand of the application as close as possible. In the literature, several auto-scalers exist that support cost-aware scaling decisions but they introduce inherent drawbacks. In this work, this lack of existing cost-aware mechanisms is addressed by introducing a mediator between an application and the auto-scaler. This cost-aware mechanism is called FOX. It leverages knowledge of the charging model of the public cloud and reviews the scaling decisions found by the auto-scaler to reduce the charged costs to a minimum. More precisely, FOX delays or omits releases of resources to avoid additional charging costs if the resource is required in the future. Hereby, FOX is not restricted to use one specific auto-scaler but offers interfaces to use any auto-scaler. For an evalation under controlled conditions, FOX scales a multi-tier application deployed in a private cloud that is stressed with two real world workloads: BibSonomy and IBM CICS. As FOX provides an interface for auto-scalers, we evaluate the cost-aware mechanism with three state of the art auto-scalers: React, Adapt, and Reg. The experiments show that FOX is able to reduce the charged costs by 34% at maximum for the Amazon EC2 charging model. According to the cost model, FOX provisions more resources than required. This results in a decreased SLO violation rate from 28% to 2% at maximum. The accounted instance time increases at max. by 30%.

The accurate measurement of a server's power consumption when running realistic workloads enables characterization of its energy efficiency and helps to make better provisioning and workload placement decisions. Information on the energy efficiency of a server for a given target workload can greatly influence such decisions and thus the final energy efficiency of a cluster or data center. However, measuring energy efficiency and power consumption of server applications has become challenging as applications are often distributed or require work intensive configuration, setup, and specialized load drivers for reproducible testing. As a result, it may be not feasible to perform tests using the actual workload that is to be deployed. We introduce an approach to create small-scale workloads that emulate the power consumption-relevant behavior of an application by deliberately triggering specific power relevant performance counter events. These workloads can then be easily deployed on a target server for fast and efficient power characterization. We validate the proposed approach by approximating the power consumption behavior of different workloads at multiple load levels. We show that our approach is capable of producing small-scale workloads that reflect the power consumption behavior of their reference applications over multiple load levels with a minimum error of less than 1%.

Simple, threshold-based auto-scaling mechanisms as mainly used in practice bring no features to overcome resource provisioning delays and non-linear scalability of a software service. In this tutorial paper, we guide the reader step-by-step through the design and evaluation of a proactive and application-aware auto-scaling mechanism.

Benchmarking of energy efficiency is important as it helps researchers, customers, and developers to evaluate and compare the energy efficiency of software and hardware solutions. Developing and deploying energy-efficiency benchmarking workloads are challenging tasks, as work must be able to be executed in a power measurement environment using an energy-efficiency measurement methodology.The existing SPEC Chauffeur Worklet Development Kit (WDK) enables the development and use of custom workloads (called worklets) within a standardized power measurement methodology. However, it features no integration in development environments, making building and deployment of workloads challenging. We address this challenge by proposing Autopilot, a plugin for the Eclipse IDE. Autopilot enables fast and easy building and deployment of a workload under development on a system for testing. It also enables benchmark execution directly from the development environment.

High performance workloads with high bandwidth memory utilization are among the most power consuming software applications. When writing such applications, developers can directly influence power consumption of the final software through their choice of data size and traversal method, mostly due to caching characteristics. Explicit knowledge on how choices influence power consumption can thus lead to greater overall energy efficiency. In existing work, power prediction for memory accesses and high bandwidth applications requires either detailed measurement information on the system on which the software is executed or it is too generic, not taking significant aspects, such as caching and data size into account. In this paper, we propose a power model that bridges this gap by modeling power consumption based on concrete software properties, while considering hardware characteristics on a more abstract level, characterizing it primarily using publicly available data. The model is designed to enable developers to compare power consumption of implementation alternatives for high memory bandwidth software components. We validate our model by measuring modified versions of the high bandwidth benchmark stream. We show that our model can predict the relative change of power consumption due to implementation changes and the power consumption of a concrete system under test with an average error of 19 percent.

Today’s system developers and operators face the challenge of creating software systems that make efficient use of dynamically allocated resources under highly variable and dynamic load profiles, while at the same time delivering reliable performance. Autonomic controllers, for example, an advanced autoscaling mechanism in a cloud computing context, can benefit from an abstracted load model as knowledge to reconfigure on time and precisely. Existing workload characterization approaches have limited support to capture variations in the interarrival times of incoming work units over time (i.e., a variable load profile). For example, industrial and scientific benchmarks support constant or stepwise increasing load, or interarrival times defined by statistical distributions or recorded traces. These options show shortcomings either in representative character of load variation patterns or in abstraction and flexibility of their format. In this article, we present the Descartes Load Intensity Model (DLIM) approach addressing these issues. DLIM provides a modeling formalism for describing load intensity variations over time. A DLIM instance is a compact formal description of a load intensity trace. DLIM-based tools provide features for benchmarking, performance, and recorded load intensity trace analysis. As manually obtaining and maintaining DLIM instances becomes time consuming, we contribute three automated extraction methods and devised metrics for comparison and method selection. We discuss how these features are used to enhance system management approaches for adaptations during runtime, and how they are integrated into simulation contexts and enable benchmarking of elastic or adaptive behavior. We show that automatically extracted DLIM instances exhibit an average modeling error of 15.2% over 10 different real-world traces that cover between 2 weeks and 7 months. These results underline DLIM model expressiveness. In terms of accuracy and processing speed, our proposed extraction methods for the descriptive models are comparable to existing time series decomposition methods. Additionally, we illustrate DLIM applicability by outlining approaches of workload modeling in systems engineering that employ or rely on our proposed load intensity modeling formalism.

Modern IT systems have increasingly distributed and dynamic architectures providing flexibility to adapt to changes in the environment and thus enabling higher resource efficiency. However, these benefits come at the cost of higher system complexity and dynamics. Thus, engineering systems that manage their end-to-end application performance and resource efficiency in an autonomic manner is a challenge. In this article, we present a holistic model-based approach for self-aware performance and resource management leveraging the Descartes Modeling Language (DML), an architecture-level modeling language for online performance and resource management. We propose a novel online performance prediction process that dynamically tailors the model solving depending on the requirements regarding accuracy and overhead. Using these prediction capabilities, we implement a generic model-based control loop for proactive system adaptation. We evaluate our model-based approach in the context of two representative case studies showing that with the proposed methods, significant resource efficiency gains can be achieved while maintaining performance requirements. These results represent the first end-to-end validation of our approach, demonstrating its potential for self-aware performance and resource management in the context of modern IT systems and infrastructures.

With the widespread adoption of virtualization, intrusion detection systems (IDSes) are increasingly being deployed in virtualized environments. When securing an environment, IT security officers are often faced with the question of how accurate deployed IDSes are at detecting attacks. To this end, metrics for assessing the attack detection accuracy of IDSes have been developed. However, these metrics are defined with respect to a fixed set of hardware resources available to the tested IDS. Therefore, IDSes deployed in virtualized environments featuring elasticity (i.e., on-demand allocation or deallocation of virtualized hardware resources during system operation) cannot be evaluated in an accurate manner using existing metrics. In this paper, we demonstrate the impact of elasticity on IDS attack detection accuracy. In addition, we propose a novel metric and measurement methodology for accurately quantifying the accuracy of IDSes deployed in virtualized environments featuring elasticity. We demonstrate their practical use through case studies involving commonly used IDSes.

No abstract available.

Experimental analysis of computer systems' power consumption has become an integral part of system performance evaluation, efficiency management, and model-based analysis. As with all measurements, repeatability and reproducibility of power measurements are a major challenge. Nominally identical systems can have different power consumption running the same workload under otherwise identical conditions. This behavior can also be observed for individual system components. Specifically, CPU power consumption can vary amongst different samples of nominally identical CPUs. This in turn has a significant impact on the overall system power, considering that a system's processor is the largest and most dynamic power consumer of the overall system. The concrete impact of CPU sample power variations is unknown, as comprehensive studies about differences in power consumption for nominally identical systems are currently missing. We address this lack of studies by conducting measurements on four different processor types from two different architectures. For each of these types, we compare up to 30 physical processor samples with a total sum of 90 samples over all processor types. We analyze the variations in power consumption for the different samples using six different workloads over five load levels. Additionally, we analyze how these variations change for different processor core counts and architectures. The results of this paper show that selection of a processor sample can have a statistically significant impact on power consumption. With no correlation to performance, power consumption for nominally identical processors can differ as much as 29.6% in idle and 19.5% at full load. We also show that these variations change over different architectures and processor types.

Data centers are increasingly becoming larger and dynamic due to virtualization. In order to leverage the performance modeling and prediction techniques, such as Palladio Component Model or Descartes Modeling Language, in such a dynamic environments, it is necessary to automate the model extraction. Building and maintaining such models manually is not feasible anymore due to their size and the level of details. This paper is focused on traffic models that are an essential part of network infrastructure. Our goal is to decompose real traffic dumps into models suitable for performance prediction using Descartes Network Infrastructure modeling approach. The main challenge was to efficiently encode an arbitrary signal in the form of simple traffic generators while maintaining the shape of the original signal. We show that a typical 15 minute long tcpdump trace can be compressed to 0.4-15% of its original size whereas the relative median of extraction error is close to 0% for the most of the 69 examined traces.

Over the past decades, various methods, techniques, and tools for modeling and evaluating performance properties of software systems have been proposed covering the entire software life cycle. However, the application of performance engineering approaches to solve a given user concern is still rather challenging and requires expert knowledge and experience. There are no recipes on how to select, configure, and execute suitable methods, tools, and techniques allowing to address the user concerns. In this paper, we describe our vision of Declarative Performance Engineering (DPE), which aims to decouple the description of the user concerns to be solved (performance questions and goals) from the task of selecting and applying a specific solution approach. The strict separation of "what" versus "how" enables the development of different techniques and algorithms to automatically select and apply a suitable approach for a given scenario. The goal is to hide complexity from the user by allowing users to express their concerns and goals without requiring any knowledge about performance engineering techniques. Towards realizing the DPE vision, we discuss the different requirements and propose a reference architecture for implementing and integrating respective methods, algorithms, and tooling.

Enterprise applications in virtualized environments are often subject to time-varying workloads with multiple seasonal patterns and trends. In order to ensure quality of service for such applications while avoiding over-provisioning, resources need to be dynamically adapted to accommodate the current workload demands. Many memory-intensive applications are not suitable for the traditional horizontal scaling approach often used for runtime performance management, as it relies on complex and expensive state replication. On the other hand, vertical scaling of memory often requires a restart of the application. In this paper, we propose a proactive approach to memory scaling for virtualized applications. It uses statistical forecasting to predict the future workload and reconfigure the memory size of the virtual machine of an application automatically. To this end, we propose an extended forecasting technique that leverages meta-knowledge, such as calendar information, to improve the forecast accuracy. In addition, we develop an application controller to adjust settings associated with application memory management during memory reconfiguration. Our evaluation using real-world traces shows that the forecast accuracy quantified with the MASE error metric can be improved by 11 - 59%. Furthermore, we demonstrate that the proactive approach can reduce the impact of reconfiguration on application availability by over 80% and significantly improve performance relative to a reactive controller.

Energy efficiency of servers has become a significant issue over the last years. Load distribution plays a crucial role in the improvement of energy efficiency as (un-)balancing strategies can be leveraged to distribute load over one or multiple systems in a way in which resources are utilized at high performance, yet low overall power consumption. This can be achieved on multiple levels, from load distribution on single CPU cores to machine level load balancing on distributed systems. With modern day server architectures providing load balancing opportunities at several layers, answering the question of optimal load distribution has become non-trivial. Work has to be distributed hierarchically in a fashion that enables maximum energy efficiency at each level. Current approaches balance load based on generalized assumptions about the energy efficiency of servers. These assumptions are based either on very machine-specific or highly generalized observations that may or may not hold true over a variety of systems and configurations. In this paper, we use a modified version of the SPEC SERT suite to measure the energy efficiency of a variety of hierarchical load distribution strategies on single and multi-node systems. We introduce a new strategy and evaluate energy efficiency for homogeneous and heterogeneous workloads over different hardware configurations. Our results show that the selection of a load distribution strategy depends heavily on workload, system utilization, as well as hardware. Used in conjunction with existing strategies, our new load distribution strategy can reduce a single system's power consumption by up to 10.7%.

Next generation IT infrastructures are highly driven by virtualization technology. The latter enables flexible and efficient resource sharing allowing to improve system agility and reduce costs for IT services. Due to the sharing of resources and the increasing requirements of modern applications on I/O processing, the performance of storage systems is becoming a crucial factor. In particular, when migrating or consolidating different applications the impact on their performance behavior is often an open question. Performance modeling approaches help to answer such questions, a prerequisite, however, is to find an appropriate workload characterization that is both easy to obtain from applications as well as sufficient to capture the important characteristics of the application. In this paper, we present an automated workload characterization approach that extracts a workload model to represent the main aspects of I/O-intensive applications using relevant workload parameters, e.g., request size, read-write ratio, in virtualized environments. Once extracted, workload models can be used to emulate the workload performance behavior in real-world scenarios like migration and consolidation scenarios. We demonstrate our approach in the context of two case studies of representative system environments. We present an in-depth evaluation of our workload characterization approach showing its effectiveness in workload migration and consolidation scenarios. We use an IBM System z equipped with an IBM DS8700 and a Sun Fire system as state-of-the-art virtualized environments. Overall, the evaluation of our workload characterization approach shows promising results to capture the relevant factors of I/O-intensive applications.

Energy efficiency of servers has become a significant research topic over the last years, as server energy consumption varies depending on multiple factors, such as server utilization and workload type. Server energy analysis and estimation must take all relevant factors into account to ensure reliable estimates and conclusions. Thorough system analysis requires benchmarks capable of testing different system resources at different load levels using multiple workload types. Server energy estimation approaches, on the other hand, require knowledge about the interactions of these factors for the creation of accurate power models. Common approaches to energy-aware workload classification categorize workloads depending on the resource types used by the different workloads. However, they rarely take into account differences in workloads targeting the same resources. Industrial energy-efficiency benchmarks typically do not evaluate the system's energy consumption at different resource load levels, and they only provide data for system analysis at maximum system load.

The ever-increasing I/O resource demands pose significant challenges for today's system environments to meet performance requirements. The resource demand effects are even magnified in modern virtualized environments where workloads are consolidated to save hardware and operating costs. Tool-supported analysis approaches can help to understand I/O performance characteristics and avoid I/O performance and interference issues. In this demo paper, we present the Storage Performance Analyzer (SPA) - a tool for automated I/O performance analysis. SPA is equipped with tailored features for virtualized environments allowing to measure, monitor, and model both I/O performance and interference effects in modern environments. SPA is open-source and available for common operating systems.

During the last decade, researchers have proposed a number of model transformations enabling performance predictions. These transformations map performance-annotated software architecture models into stochastic models solved by analytical means or by simulation. However, so far, a detailed quantitative evaluation of the accuracy and efficiency of different transformations is missing, making it hard to select an adequate transformation for a given context. This paper provides an in-depth comparison and quantitative evaluation of representative model transformations to, e.g., queueing petri nets and layered queueing networks. The semantic gaps between typical source model abstractions and the different analysis techniques are revealed. The accuracy and efficiency of each transformation are evaluated by considering four case studies representing systems of different size and complexity. The presented results and insights gained from the evaluation help software architects and performance engineers to select the appropriate transformation for a given context, thus significantly improving the usability of model transformations for performance prediction.

Software-as-a-Service (SaaS) often shares one single application instance among different tenants to reduce costs. However, sharing potentially leads to undesired influence from one tenant onto the performance observed by the others. This is a significant problem as performance is one of the major obstacles for cloud customers. The application does intentionally not manage hardware resources, and the operating system is not aware of application level entities like tenants which makes the performance control a challenge. In case the SaaS is hosted on a Platform-as-a-Service (PaaS), the SaaS developer usually wants to control performance-related issues according to individual needs, and available information is even more limited. Thus, it is difficult to control the performance of different tenants to keep them isolated. Existing work focuses on concrete methods to provide performance isolation in systems where the whole stack is under control. In this paper we present a concrete PaaS enhancement which enables application developers to realize isolation methods for their hosted SaaS application. In a case study we evaluated the applicability and effectiveness of the enhancement in different environments.

Multi-tenancy is an approach to share one application instance among multiple customers by providing each of them a dedicated view. This approach is commonly used by SaaS providers to reduce the costs for service provisioning. Tenants also expect to be isolated in terms of the performance they observe and the providers inability to offer performance guarantees is a major obstacle for potential cloud customers. To guarantee an isolated performance it is essential to control the resources used by a tenant. This is a challenge, because the layers of the execution environment, responsible for controlling resource usage(e.g., operating system), normally do not have knowledge about entities defined at the application level and thus they cannot distinguish between different tenants. Furthermore, it is hard to predict how tenant requests propagate through the multiple layers of the execution environment down to the physical resource layer. The intended abstraction of the application from the resource controlling layers does not allow to solely solving this problem in the application. In this paper, we propose an approach which applies resource demand estimation techniques in combination with a request based admission control. The resource demand estimation is used to determine resource consumption information for individual requests. The admission control mechanism uses this knowledge to delay requests originating from tenants that exceed their allocated resource share. The proposed method is validated by a widely accepted benchmark showing its applicability in a setup motivated by today's platform environments.

Hypervisors are becoming increasingly ubiquitous with the growing proliferation of virtualized data centers. As a result, attackers are exploring vectors to attack hypervisors, against which an attack may be executed via several attack vectors such as device drivers, virtual machine exit events, or hyper calls. Hyper calls enable intrusions in hypervisors through their hyper call interfaces. Despite the importance, there is very limited publicly available information on vulnerabilities of hyper call handlers and attacks triggering them, which significantly hinders advances towards monitoring and securing these interfaces. In this paper, we characterize the hyper call attack surface based on analyzing a set of vulnerabilities of hyper call handlers. We systematize and discuss the errors that caused the considered vulnerabilities, and activities for executing attacks triggering them. We also demonstrate attacks triggering the considered vulnerabilities and analyze their effects. Finally, we suggest an action plan for improving the security of hyper call interfaces.

Virtualization technology allows to share the physical resources used in IT infrastructures for efficient and flexible system operation. Sharing of physical resources, however, comes usually at the cost of performance and poses significant challenges to respect the Quality-of-Service of consolidated data-intensive applications due to the mutual performance interference among the applications. The non-trivial impact of workload consolidation on the I/O performance can be anticipated using explicit performance analysis techniques. In current practice, however, explicit modeling of I/O performance interference effects in virtualized environments is usually avoided due to their complexity. In this paper, we present an explicit performance modeling approach of I/O performance interference in virtualized environments with queueing Petri nets (QPNs). More specifically, we first highlight major challenges when modeling I/O performance in virtualized environments. Then, we create a single-VM I/O performance model calibrated with response time measurements to capture the complex behavior of a representative, real-world environment based on IBM System z and IBM DS8700 server hardware. Finally, we use the I/O performance model to evaluate the I/O performance when the workload is distributed heterogeneously on colocated virtual machines. Overall, we effectively create an I/O performance interference model capturing the I/O performance effects in a multi-VM environment with less than 10% prediction error on average.

Over the past few decades, many performance modeling formalisms and prediction techniques for software architectures have been developed in the performance engineering community. However, using a performance model to predict the performance of a software system normally requires extensive experience with the respective modeling formalism and involves a number of complex and time consuming manual steps. In this paper, we propose a generic declarative interface to performance prediction techniques to simplify and automate the process of using architecture-level software performance models for performance analysis. The proposed Descartes Query Language (DQL) is a language to express the demanded performance metrics for prediction as well as the goals and constraints of the specific prediction scenario. It reduces the manual effort and learning curve in working with performance models by a unified interface independent of the employed modeling formalism. We evaluate the applicability and benefits of the proposed approach in the context of several representative case studies.

Modern software systems are expected to deliver reliable performance under highly variable load intensities while at the same time making efficient use of dynamically allocated resources. Conventional benchmarking frameworks provide limited support for emulating such highly variable and dynamic load profiles and workload scenarios. Industrial benchmarks typically use workloads with constant or stepwise increasing load intensity, or they simply replay recorded workload traces. In this paper, we present LIMBO - an Eclipse-based tool for modeling variable load intensity profiles based on the Descartes Load Intensity Model as an underlying modeling formalism.

When creating a performance model, it is necessary to quantify the amount of resources consumed by an application serving individual requests. In distributed enterprise systems, these resource demands usually cannot be observed directly, their estimation is a major challenge. Different statistical approaches to resource demand estimation based on monitoring data have been proposed, e.g., using linear regression or Kalman filtering techniques. In this paper, we present LibReDE, a library of ready-to-use implementations of approaches to resource demand estimation that can be used for online and offline analysis. It is the first publicly available tool for this task and aims at supporting performance engineers during performance model construction. The library enables the quick comparison of the estimation accuracy of different approaches in a given context and thus helps to select an optimal one.

SUMMARY As modern enterprise software systems become increasingly dynamic, workload forecasting techniques are gaining an importance as a foundation for online capacity planning and resource management. Time series analysis offers a broad spectrum of methods to calculate workload forecasts based on history monitoring data. Related work in the field of workload forecasting mostly concentrates on evaluating specific methods and their individual optimisation potential or on predicting QoS metrics directly. As a basis, we present a survey on established forecasting methods of the time series analysis concerning their benefits and drawbacks and group them according to their computational overheads. In this paper, we propose a novel self‐adaptive approach that selects suitable forecasting methods for a given context based on a decision tree and direct feedback cycles together with a corresponding implementation. The user needs to provide only his general forecasting objectives. In several experiments and case studies based on real‐world workload traces, we show that our implementation of the approach provides continuous and reliable forecast results at run‐time. The results of this extensive evaluation show that the relative error of the individual forecast points is significantly reduced compared with statically applied forecasting methods, for example, in an exemplary scenario on average by 37%. In a case study, between 55 and 75% of the violations of a given service level objective can be prevented by applying proactive resource provisioning based on the forecast results of our implementation. Copyright © 2014 John Wiley & Sons, Ltd.

Performance management and performance prediction of services deployed in virtualized environments is a challenging task. On the one hand, the virtualization layer makes the estimation of performance model parameters difficult and inaccurate. On the other hand, it is difficult to model the hyper visor scheduler in a representative and practically feasible manner. In this paper, we describe how to obtain relevant parameters, such as the virtualization overhead, depending on the amount and type of available monitoring data. We adapt classical queueing-theory-based modeling techniques to make them usable for different configurations of virtualized environments. We provide answers how to include the virtualization overhead into queueing network models, and how to take the contention between different VMs into account. Finally, we evaluate our approach in representative scenarios based on the SPECjEnterprise2010 standard benchmark and XenServer 5.5, showing significant improvements in the prediction accuracy and discussing further open issues for performance prediction in virtualized environments.

Server virtualization is a key technology to share physical resources efficiently and flexibly. With the increasing popularity of I/O-intensive applications, however, the virtualized storage used in shared environments can easily become a bottleneck and cause performance and scalability issues. Performance modeling and evaluation techniques applied prior to system deployment help to avoid such issues. In current practice, however, virtualized storage and its effects on the overall system performance are often neglected or treated as a black-box. In this paper, we present a systematic I/O performance modeling approach for virtualized storage systems based on queueing theory. We first propose a general performance model building methodology. Then, we demonstrate our methodology creating I/O queueing models of a real-world representative environment based on IBM System z and IBM DS8700 server hardware. Finally, we present an in-depth evaluation of our models considering both interpolation and extrapolation scenarios as well as scenarios with multiple virtual machines. Overall, we effectively create performance models with less than 11% mean prediction error in the worst case and less than 5% prediction error on average.

As modern enterprise software systems become increasingly dynamic, workload forecasting techniques are gaining in importance as a foundation for online capacity planning and resource management. Time series analysis offers a broad spectrum of methods to calculate workload forecasts based on history monitoring data. Related work in the field of workload forecasting mostly concentrates on evaluating specific methods and their individual optimisation potential or on predicting Quality-of-Service (QoS) metrics directly. As a basis, we present a survey on established forecasting methods of the time series analysis concerning their benefits and drawbacks and group them according to their computational overheads. In this paper, we propose a novel self-adaptive approach that selects suitable forecasting methods for a given context based on a decision tree and direct feedback cycles together with a corresponding implementation. The user needs to provide only his general forecasting objectives. In several experiments and case studies based on real-world workload traces, we show that our implementation of the approach provides continuous and reliable forecast results at run-time. The results of this extensive evaluation show that the relative error of the individual forecast points is significantly reduced compared to statically applied forecasting methods, e.g. in an exemplary scenario on average by 37%. In a case study, between 55% and 75% of the violations of a given service level agreement can be prevented by applying proactive resource provisioning based on the forecast results of our implementation.

Modern virtualized environments are key for reducing the operating costs of data centers. By enabling the sharing of physical resources, virtualization promises increased resource efficiency with decreased administration costs. With the increasing popularity of I/O-intensive applications, however, the virtualized storage used in such environments can quickly become a bottleneck and lead to performance and scalability issues. Performance modeling and evaluation techniques applied prior to system deployment help to avoid such issues. In current practice, however, virtualized storage and its performance-influencing factors are often neglected or treated as a black-box. In this paper, we present a measurement-based performance prediction approach for virtualized storage systems based on optimized statistical regression techniques. We first propose a general heuristic search algorithm to optimize the parameters of regression techniques. Then, we apply our optimization approach and create performance models using four regression techniques. Finally, we present an in-depth evaluation of our approach in a real-world representative environment based on IBM System z and IBM DS8700 server hardware. Using our optimized techniques, we effectively create performance models with less than 7% prediction error in the most typical scenario. Furthermore, our optimization approach reduces the prediction error by up to 74%.

Virtualized environments introduce an additional abstraction layer on top of physical resources to enable the collective resource usage by multiple systems. With the rise of I/O-intensive applications, however, the virtualized storage of such shared environments can quickly become a bottleneck and lead to performance and scalability issues. The latter can be avoided through careful design of the application architecture and systematic capacity planning throughout the system life cycle. In current practice, however, virtualized storage and its performance-influencing design decisions are often neglected or treated as a black-box. In this work-in-progress paper, we propose a generic approach for performance modeling and prediction of virtualized storage systems at the software architecture level. More specifically, we propose two performance modeling approaches of virtualized systems. Furthermore, we propose two approaches how the performance models can be combined with architecture-level performance models. The goal is to cope with the increasing complexity of virtualized storage systems with the benefit of intuitive software architecture-level models.

In this work-in-progress paper, we present a new meta-model designed for the performance modeling of dynamic data center network infrastructures. Our approach models characteristic aspects of Cloud data centers which were not crucial in classical data centers. We present our meta-model and demonstrate its use for performance modeling and analysis through an example, including a transformation into OMNeT++ for performance simulation.

Queueing Petri nets are a powerful formalism that can be exploited for modeling distributed systems and evaluating their performance and scalability. By combining the modeling power and expressiveness of queueing networks and stochastic Petri nets, queueing Petri nets provide a number of advantages. This tutorial presents an introduction to queueing Petri nets first introducing the modeling formalism itself and then summarizing the results of several modeling case studies which demonstrate how queueing Petri nets can be used for performance modeling and analysis. As part of the tutorial, we present QPME (Queueing Petri net Modeling Environment), an open-source tool for stochastic modeling and analysis of systems using queueing Petri nets. Finally, we briefly present a model-to-model transformation automatically generating a queueing Petri net model from a higher-level software architecture model annotated with performance relevant information.

Modern enterprise applications have to satisfy increasingly stringent Quality-of-Service requirements. To ensure that a system meets its performance requirements, the ability to predict its performance under different configurations and workloads is essential. Architecture-level performance models describe performance-relevant aspects of software architectures and execution environments allowing to evaluate different usage profiles as well as system deployment and configuration options. However, building performance models manually requires a lot of time and effort. In this paper, we present a novel automated method for the extraction of architecture-level performance models of distributed component-based systems, based on monitoring data collected at run-time. The method is validated in a case study with the industry-standard SPECjEnterprise2010 Enterprise Java benchmark, a representative software system executed in a realistic environment. The obtained performance predictions match the measurements on the real system within an error margin of mostly 10-20 percent.

Event-based communication is used in different domains including telecommunications, transportation, and business information systems to build scalable distributed systems. The loose coupling of components in such systems makes it easy to vary the deployment. At the same time, the complexity to estimate the behavior and performance of the whole system is increased, which complicates capacity planning. In this paper, we present an automated performance prediction method supporting capacity planning for event-based systems. The performance prediction is based on an extended version of the Palladio Component Model - a performance meta-model for component-based systems. We apply this method on a real-world case study of a traffic monitoring system. In addition to the application of our performance prediction techniques for capacity planning, we evaluate the prediction results against measurements in the context of the case study. The results demonstrate the practicality and effectiveness of the proposed approach.

Performance predictions early in the software development process can help to detect problems before resources have been spent on implementation. The Palladio Component Model (PCM) is an example of a mature domain-specific modeling language for component-based systems enabling performance predictions at design time. PCM provides several alternative model solution methods based on analytical and simulation techniques. However, existing solution methods suffer from scalability issues and provide limited flexibility in trading-off between results accuracy and analysis overhead. Queueing Petri Nets (QPNs) are a general-purpose modeling formalism, at a lower level of abstraction, for which efficient and mature simulation-based solution techniques are available. This paper contributes a formal mapping from PCM to QPN models, implemented by means of an automated model-to-model transformation as part of a new PCM solution method based on simulation of QPNs. The limitations of the mapping and the accuracy and overhead of the new solution method compared to existing methods are evaluated in detail in the context of five case studies of different size and complexity. The new solution method proved to provide good accuracy with solution overhead up to 20 times lower compared to PCM's reference solver.

Distributed event-based systems (DEBS) are gaining increasing attention in new application areas such as transport information monitoring, event-driven supply-chain management and ubiquitous sensor-rich environments. However, as DEBS increasingly enter the enterprise and commercial domains, performance and quality of service issues are becoming a major concern. While numerous approaches to performance modeling and evaluation of conventional request/reply-based distributed systems are available in the literature, no general approach exists for DEBS. This paper is the first to provide a comprehensive methodology for workload characterization and performance modeling of DEBS. A workload model of a generic DEBS is developed and operational analysis techniques are used to characterize the system traffic and derive an approximation for the mean event delivery latency. Following this, a modeling technique is presented that can be used for accurate performance prediction. The paper is concluded with a case study of a real life system demonstrating the effectiveness and practicality of the proposed approach.

Performance models are used increasingly throughout the phases of the software engineering lifecycle of distributed component-based systems. However, as systems grow in size and complexity, building models that accurately capture the different aspects of their behavior becomes a more and more challenging task. In this paper, we present a novel case study of a realistic distributed component-based system, showing how queueing Petri net models can be exploited as a powerful performance prediction tool in the software engineering process. A detailed system model is built in a step-by-step fashion, validated, and then used to evaluate the system performance and scalability. Along with the case study, a practical performance modeling methodology is presented which helps to construct models that accurately reflect the system performance and scalability characteristics. Taking advantage of the modeling power and expressiveness of queueing Petri nets, our approach makes it possible to model the system at a higher degree of accuracy, providing a number of important benefits

In this paper we show how Queuing Petri Net (QPN) models can be exploited for performance analysis of distributed e-business systems. We study a real-world application, and demonstrate the benefits, in terms of modelling power and expressiveness, that QPN models provide over conventional modelling paradigms such as Queuing Networks and Petri Nets. As shown, QPNs facilitate the integration of both hardware and software aspects of system behavior in the same model. In addition to hardware contention and scheduling strategies, using QPNs one can easily model simultaneous resource possession, synchronization, blocking and contention for software resources. By validating the models presented through measurements, we show that they are not just powerful as a specification mechanism, but are also very powerful as a performance analysis and prediction tool. However, currently available tools and techniques for QPN analysis are limited. Improved solution methods, which enable larger models to be analyzed, need to be developed. By demonstrating the power of QPNs as a modelling paradigm in realistic scenarios, we hope to motivate further research in this area.