Abstract Microservice‐based applications consist of multiple services that can evolve independently. When a service must be updated, it is first tested with in‐house regression test suites. However, the test suites that are executed are usually designed without the exact knowledge about how the services will be accessed and used in the field; therefore, they may easily miss relevant test scenarios, failing to prevent the deployment of faulty services. To address this problem, we introduce ExVivoMicroTest, an approach that analyzes the execution of deployed services at run‐time in the field, in order to generate test cases for future versions of the same services. ExVivoMicroTest implements lightweight monitoring and tracing capabilities, to inexpensively record executions that can be later turned into regression test cases that capture how services are used in the field. To prevent accumulating an excessive number of test cases, ExVivoMicroTest uses a test coverage model that can discriminate the recorded executions between the ones that are worth to be turned into test cases and the ones that should be discarded. The resulting test cases use a mocked environment that fully isolates the service under test from the rest of the system to faithfully reply interactions. We assessed ExVivoMicroTest with the PiggyMetrics and Train Ticket open source microservice applications and studied how different configurations of the monitoring and tracing logic impact on the capability to generate test cases.

Placement of VMs in the cloud is one of the most fundamental problems in systems research. Traditionally, placement algorithms assume that the schedulers have complete information about the currently available resources at each host. However, this assumption is in many cases unrealistic, as gathering fresh status information from each of the thousands of hosts in a large data center incurs excessive communication overhead, which results in long queueing delays. Efforts to resolve this problem by employing several parallel schedulers typically exhibit collisions when several schedulers are simultaneously trying to place VMs on the same host. Our work analyzes the performance of various placement algorithms and provides empirical evidence that using multiple randomized schedulers obtains high throughput, while significantly decreasing both the communication overhead, and the number of collisions between schedulers. We, therefore, introduce Adaptive Partial State Random (APSR) – an efficient parallel random resource management algorithm that samples only from a small number of hosts and dynamically adjusts the degree of parallelism to provide provable guarantees on the probability of collisions between distinct schedulers. We formally analyze APSR, evaluate it on real workloads, and integrate it into the popular OpenStack cloud management platform. Our evaluation shows that APSR matches the throughput provided by other parallel schedulers, while achieving up to 13x lower decline ratio and a reduction of over 85% in communication overheads.

Process modeling can benefit from automation using knowledge mined from collections of existing processes. One promising technique for such automation is the recommendation of the next elements to be added to the processes under construction. In this paper, we review an autocompletion engine that is based on the semantic similarity of business processes. To assess its efficiency in practical settings, we conduct a user study where domain experts are asked to rate the suggestions made by the engine for a commercial product. Their ratings are then compared to the engine’s accuracy measured by metrics from the natural language processing field. Our study shows a strong correlation between the expert ratings and some of these metrics. We confirm the usefulness of such an autocompletion engine, and enumerate potential improvements to any process autocompletion technique.

Network Function Virtualization (NFV) carries the potential for on-demand deployment of network algorithms in virtual machines (VMs). In large clouds, however, VM resource allocation incurs delays that hinder the dynamic scaling of such NFV deployment. Parallel resource management is a promising direction for boosting performance, but it may significantly increase the communication overhead and the decline ratio of deployment attempts. Our work analyzes the performance of various placement algorithms and provides empirical evidence that state of the art parallel resource management dramatically increases the decline ratio of deterministic algorithms, but hardly affects randomized algorithms. We therefore introduce APSR - an efficient parallel random resource management algorithm that requires information only from a small number of hosts and dynamically adjusts the degree of parallelism to provide provable decline ratio guarantees. We formally analyze APSR, evaluate it on real workloads, and integrate it into the popular OpenStack cloud management platform. Our evaluation shows that APSR matches the throughput provided by other parallel schedulers, while achieving up to 13x lower decline ratio and a reduction of over 85% in communication overheads.

Gradient boosted models are a fundamental machine learning technique. Robustness to small perturbations of the input is an important quality measure for machine learning models, but the literature lacks a method to prove the robustness of gradient boosted models.This work introduces VERIGB, a tool for quantifying the robustness of gradient boosted models. VERIGB encodes the model and the robustness property as an SMT formula, which enables state of the art verification tools to prove the model’s robustness. We extensively evaluate VERIGB on publicly available datasets and demonstrate a capability for verifying large models. Finally, we show that some model configurations tend to be inherently more robust than others.

Network Function Virtualization (NFV) allows operators to deploy network functions in virtual machines (VMs) and benefit from on-demand deployment. VMs are placed on one of the hosts in the cloud, and existing resource management algorithms assume full knowledge of the system's state. For large clusters, attaining the system's state creates bottlenecks and therefore it takes a long time to deploy network functionalities. Intuitively, placement can be accelerated if the resource management algorithm operates on a cached system state which is not entirely up to date, but the placement quality may suffer. Our work introduces a new cache refresh method that achieves an up to a 5.3x reduction in placement time with only a slight degradation of quality compared to having the complete and up to date system's state.

Monitoring systems and ensuring the required service level is an important operation task. However, doing this based on external visible data, such as systems logs, is very difficult since it is very hard to extract from the logged data the exact state and the root cause to the actions taken by the system. Yet, identifying behavioral changes of complex systems can be used for early identification of problems and allow proactive correction measurements. Since it is practically impossible to perform this task manually, there is a critical need for a methodology that can analyze logs, automatically create a behavioral model, and compare the behavior to the expected behavior.In this paper we propose a novel approach for comparison between serviceexecutions as exhibited in their log files. The behavior is captured by FiniteState Automaton models (FSAs), enhanced with performance related data, bothmined from the logs. Our tool then computes the difference between the current model and behavioral models created when the service was known to operate well. A visual framework that graphically presents and emphasizes the changes in the behavior is then used to trace their root cause. We evaluate our approach over real telecommunication logs.

One of the main motivations for the shift to the Cloud (and the more recent shift of telco operators into NFV) is cost reduction due to high utilization of infrastructure resources. However, achieving high utilization in practical scenarios is complex since the term "resources" covers different orthogonal aspects, such as server CPU, storage (or disk) usage and network capacity, and the workload characterization varies over time and over different users.

Software systems tend to suffer from architectural problems as they are being developed. While modern software development methodologies such as Agile and Dev-Ops suggest different ways of assuring code quality, very little attention is paid to maintaining high quality of the architecture of the evolving systems. By detecting and alerting about violations of the intended software architecture, one can often avoid code-level bad smells such as spaghetti code. Typically, if one wants to reason about the software architecture, the burden of first defining the intended architecture falls on the developer's shoulders. This includes definition of valid and invalid dependencies between software components. However, the developers are seldom familiar with the entire software system, which makes this task difficult, time consuming and error-prone. We propose and implement a solution for automatic detection of architectural violations in software artifacts. The solution, which utilizes a number of predefined and user-defined patterns, does not require prior knowledge of the system or its intended architecture. We propose to leverage this solution as part of the nightly build process used by development teams, thus achieving continuous automatic validation of the system's software architecture. As we show in multiple open-source and proprietary cases, a small set of predefined patterns can detect architectural violations as they are introduced over the course of development, and also capture deterioration in existing architectural problems. By evaluating the tool on relatively large open-source projects, we also validate its scalability and practical applicability to large software systems.

One of the challenges in maintaining legacy systems is to be able to locate business logic in the code, and isolate it for different purposes, including implementing requested changes, refactoring, eliminating duplication, unit testing, and extracting business logic into a rule engine. Our new idea is an iterative method to identify the business logic in the code and visualize this information to gain better understanding of the logic distribution in the code, as well as developing a domain-specific business vocabulary. This new method combines and extends several existing technologies, including search, aggregation, and visualization. We evaluated the visualization method on a large-scale application and found that it yields useful results, provided an appropriate vocabulary is available.

A key piece of information in healthcare is a patient's benefit plan. It details which treatments and procedures are covered by the health insurer (or payer), and at which conditions. While the most accurate and complete implementation of the plan resides in the payers claims adjudication systems, the inherent complexity of these systems forces payers to maintain multiple repositories of benefit information for other service and regulatory needs. In this paper we present a technology that deals with this complexity. We show how a large US health payer benefited from using the visualization, search, summarization and other capabilities of the technology. We argue that this technology can be used to improve productivity and reduce error rate in the benefits administration workflow, leading to lower administrative overhead and cost for health payers, which benefits both payers and patients.

Software metrics are easy to define, but not so easy to justify. It is hard to prove that a metric is valid, i.e., that measured numerical values imply anything on the vaguely defined, yet crucial software properties such as complexity and maintainability. This paper employs statistical analysis and tests to check some plausible assumptions on the behavior of software and metrics measured for this software in retrospective on its versions evolution history. Among those are the reliability assumption implicit in the application of any code metric, and the assumption that the magnitude of change, i.e., increase or decrease of its size, in a software artifact is correlated with changes to its version number. Putting a suite of 36 metrics to the trial, we confirm most of the assumptions on a large repository of software artifacts. Surprisingly, we show that a substantial portion of the reliability of some metrics can be observed even in random changes to architecture. Another surprising result is that Boolean-valued metrics tend to flip their values more often in minor software version increments than in major increments.

Component level performance thresholds are widely used as a basic means for performance management. As the complexity of managed applications increases, manual threshold maintenance becomes a difficult task. Complexity arises from having a large number of application components and their operational metrics, dynamically changing workloads, and compound relationships between application components. To alleviate this problem, we advocate that component level thresholds should be computed, managed and optimized automatically and autonomously. To this end, we have designed and implemented a performance threshold management application that automatically and dynamically computes two separate component level thresholds: one for controlling Type I errors and another for controlling Type II errors. Our solution additionally facilitates metric selection thus minimizing management overheads. We present the theoretical foundation for this autonomic threshold management application, describe a specific algorithm and its implementation, and evaluate it using real-life scenarios and production data sets. As our present study shows, with proper parameter tuning, our on-line dynamic solution is capable of nearly optimal performance thresholds calculation.

Many service-oriented systems are not well equipped to guarantee that service time is optimized. We have specifically examined two industrial systems which implement service-oriented architectures in real, field environments. We discovered that both were not engineered to properly address surges in service request rate. In the absence of an integral solution, it is difficult and costly to (re-) engineer such a solution in the field. The challenge faced by this study was to deliver a low cost solution, without re-engineering the target systems. This paper introduces such a generic solution. The solution slows-down some components to deliver improvement in request service time. It was implemented, tested, and successfully applied to two industrial systems with no need to modify their logic or architecture. Experiments with those systems exhibited significant improvement in performance. These results have validated our solution and its industrial applicability across systems and environments.

Reverse engineering is an essential part of the modernization process that enables the evolution of existing software assets. The extraction of state machines out of existing code is an important aspect of the reverse engineering process. However, none of the reverse engineering tools fully support an automatic extraction of state machines. In our work we investigated the process of manual extraction of hierarchical state machines from the source code of an embedded C application and identified the steps of the process that can be automated. We learned that manual creation of state machines out of code is a very complicated task mostly because of the large amount of potential states that can be created by a relatively small amount of global variables. To reduce the complexity of this task we developed a methodology to decompose the code into smaller parts of functionally related elements. We showed how this technique and other system analysis mechanisms provided by the System Grokking technology can automate steps of the state machine extraction process.

Component level performance thresholds are widely used as a basic means for performance management. As the complexity of managed systems increases, manual threshold maintenance becomes a difficult task. This may result from a) a large number of system components and their operational metrics, b) dynamically changing workloads, and c) complex dependencies between system components. To alleviate this problem, we advocate that component level thresholds should be computed, managed and optimized automatically and autonomously. To this end, we have designed and implemented a performance threshold management sub-system that automatically and dynamically computes two separate component level thresholds: one for controlling Type I errors and another for controlling Type II errors. We present the theoretical foundation for this autonomic threshold management system, describe a specific algorithm and its implementation, and evaluate it using real-life scenarios and production data sets. As our present study shows, with proper parameter tuning, our on-line dynamic solution is capable of nearly optimal performance thresholds calculation.

Self-healing capabilities allow software systems to overcome problems occurring during testing and run time, and thus improve overall system behavior. The PANACEA framework introduced in this paper provides a design methodology as well as ready-to-use healing elements aimed at enhancing software systems with self-healing capabilities both at design time and at run time. The PANACEA approach is based on inserting self- healing elements into the system at design and coding time, to be used later for healing at testing and run time. Specifically, the Panacea framework is based on inserting annotations into the system code at design and coding time, to later on serve as an interface for runtime monitoring, managing, configuring and healing of the annotated system components. The current embodiment of PANACEA includes several generic components that provide self-healing capabilities suited for a variety of application types. The PANACEA runtime environment automatically activates and invokes these components in order to optimize and heal the application. The PANACEA framework provides an innovative programming model that enables development of advanced self-healing applications. PANACEA introduces a paradigm shift in which software is made self-healing by design. This paradigm shift, however, is graceful since developers are not required to master neither new programming skills, nor languages. As our initial experiments demonstrate, PANACEA introduces a very small performance overhead, and scales well.