Abstract Microservice‐based applications consist of multiple services that can evolve independently. When a service must be updated, it is first tested with in‐house regression test suites. However, the test suites that are executed are usually designed without the exact knowledge about how the services will be accessed and used in the field; therefore, they may easily miss relevant test scenarios, failing to prevent the deployment of faulty services. To address this problem, we introduce ExVivoMicroTest, an approach that analyzes the execution of deployed services at run‐time in the field, in order to generate test cases for future versions of the same services. ExVivoMicroTest implements lightweight monitoring and tracing capabilities, to inexpensively record executions that can be later turned into regression test cases that capture how services are used in the field. To prevent accumulating an excessive number of test cases, ExVivoMicroTest uses a test coverage model that can discriminate the recorded executions between the ones that are worth to be turned into test cases and the ones that should be discarded. The resulting test cases use a mocked environment that fully isolates the service under test from the rest of the system to faithfully reply interactions. We assessed ExVivoMicroTest with the PiggyMetrics and Train Ticket open source microservice applications and studied how different configurations of the monitoring and tracing logic impact on the capability to generate test cases.

Reactive synthesis is an automated procedure to obtain a correct-by-construction reactive system from its temporal logic specification. GR(1) is an expressive fragment of LTL that enables efficient synthesis and has been recently used in different contexts and application domains. In this paper we investigate the dynamic-update problem for GR(1): updating the behavior of an already running synthesized controller such that it would safely and dynamically, without stopping, start conforming to a modified, up-to-date specification. We formally define the dynamic-update problem and present a sound and complete solution that is based on the computation of a bridge-controller. We implemented the work in the Spectra synthesis and execution environment and evaluated it over benchmark specifications. The evaluation shows the efficiency and effectiveness of using dynamic updates. The work advances the state-of-the-art in reactive synthesis and opens the way to its use in application domains where dynamic updates are a necessary requirement.

This paper describes Resh, a new, statically typed, interpreted programming language and associated runtime for orchestrating multirobot systems. The main features of Resh are: (1) It offloads much of the tedious work of programming such systems away from the programmer and into the language runtime; (2) It is based on a small set of temporal and locational operators; and (3) It is not restricted to specific robot types or tasks. The Resh runtime consists of three engines that collaborate to run a Resh program using the available robots in their current environment. This paper describes both Resh and its runtime and gives examples of its use.

Simulation is an essential tool for developing robotic systems; however it is computationally intensive and does not scale well, especially for multi-robot scenarios. In this paper we introduce daß – a system that facilitates distributable and scalable simulations of robotic applications. It overcomes many of the performance and quality limitations involved in common multi-robot simulation scenarios through horizontal scaling. Through extensive evaluation, we show that daß significantly outperforms the widely used Gazebo robotics simulator, and easily scales well beyond the current limitations with no significant loss in performance and simulation fidelity.

Gradient boosted models are a fundamental machine learning technique. Robustness to small perturbations of the input is an important quality measure for machine learning models, but the literature lacks a method to prove the robustness of gradient boosted models.This work introduces VERIGB, a tool for quantifying the robustness of gradient boosted models. VERIGB encodes the model and the robustness property as an SMT formula, which enables state of the art verification tools to prove the model’s robustness. We extensively evaluate VERIGB on publicly available datasets and demonstrate a capability for verifying large models. Finally, we show that some model configurations tend to be inherently more robust than others.

Enabling functionality in modern network is achieved through the use of middleboxes. Middleboxes suffer from temporal unavailability due to various reasons, such as hardware faults. We design a backup scheme that takes advantage of Network Function Virtualization (NFV), an emerging paradigm of implementing network functions in software, deployed on commodity servers. We utilize the agility of software-based systems, and the gap between the resource utilization of active and standby components, in order to design an optimal limited-resource backup scheme. We focus on the case where a small number of middleboxes fail simultaneously, and study the backup resources required for guaranteeing full recovery from any set of failures, of up to some limited size. Via a novel graph-based presentation, we develop a provably optimal construction of such backup schemes. Since full recovery is guaranteed, our construction does not rely on failure statistics, which are typically hard to obtain. Simulation results show that our proposed approach is applicable even for the case of larger numbers of failures.

Enabling functionality in a modern network is achieved through the use of middleboxes. Middleboxes suffer from temporal unavailability due to various reasons, such as hardware faults. We design a backup scheme that takes advantage of network function virtualization, an emerging paradigm of implementing network functions in software, deployed on commodity servers. We utilize the agility of software-based systems, and the gap between the resource utilization of active and standby components, in order to design an optimal limited-resource backup scheme. We focus on the case where a small number of middleboxes fail simultaneously, and study the backup resources required for guaranteeing full recovery from any set of failures, of up to some limited size. Via a novel graph-based presentation, we develop a provably optimal construction of such backup schemes. Since full recovery is guaranteed, our construction does not rely on failure statistics, which are typically hard to obtain. Simulation results show that our proposed approach is applicable even for the case of larger numbers of failures.

Monitoring systems and ensuring the required service level is an important operation task. However, doing this based on external visible data, such as systems logs, is very difficult since it is very hard to extract from the logged data the exact state and the root cause to the actions taken by the system. Yet, identifying behavioral changes of complex systems can be used for early identification of problems and allow proactive correction measurements. Since it is practically impossible to perform this task manually, there is a critical need for a methodology that can analyze logs, automatically create a behavioral model, and compare the behavior to the expected behavior.In this paper we propose a novel approach for comparison between serviceexecutions as exhibited in their log files. The behavior is captured by FiniteState Automaton models (FSAs), enhanced with performance related data, bothmined from the logs. Our tool then computes the difference between the current model and behavioral models created when the service was known to operate well. A visual framework that graphically presents and emphasizes the changes in the behavior is then used to trace their root cause. We evaluate our approach over real telecommunication logs.

One of the main motivations for the shift to the Cloud (and the more recent shift of telco operators into NFV) is cost reduction due to high utilization of infrastructure resources. However, achieving high utilization in practical scenarios is complex since the term "resources" covers different orthogonal aspects, such as server CPU, storage (or disk) usage and network capacity, and the workload characterization varies over time and over different users.

In enterprise networks, network functions, such as address translation, firewall, and deep packet inspection, are often implemented in middleboxes. Those can suffer from temporary unavailability due to misconfiguration or software and hardware malfunction. Traditionally, middlebox survivability is achieved by an expensive active-standby deployment where each middlebox has a backup instance, which is activated in case of a failure. Network function virtualization (NFV) is a novel networking paradigm allowing flexible, scalable and inexpensive implementation of network services. In this paper, we suggest a novel approach for planning and deploying backup schemes for network functions that guarantee high levels of survivability with significant reduction in resource consumption. In the suggested backup scheme, we take advantage of the flexibility and resource-sharing abilities of the NFV paradigm in order to maintain only a few backup servers, where each can serve one of multiple functions when corresponding middleboxes are unavailable. We describe different goals that network designers can consider when determining which functions to implement in each of the backup servers. We rely on a graph theoretical model to find properties of efficient assignments and to develop algorithms that can find them. Extensive experiments show, for example, that under realistic function failure probabilities, and reasonable capacity limitations, one can obtain 99.9% survival probability with half the number of servers, compared with standard techniques.

In enterprise networks, network functions such as address translation, firewall and deep packet inspection are often implemented in middleboxes. Those can suffer from temporary unavailability due to misconfiguration or software and hardware malfunction. Traditionally, middlebox survivability is achieved by an expensive active-standby deployment where each middlebox has a backup instance, which is activated in case of a failure. Network Function Virtualization (NFV) is a novel networking paradigm allowing flexible, scalable and inexpensive implementation of network services. In this work we suggest a novel approach for planning and deploying backup schemes for network functions that guarantee high levels of survivability with significant reduction in resource consumption. In the suggested backup scheme we take advantage of the flexibility and resource-sharing abilities of the NFV paradigm in order to maintain only a few backup servers, where each can serve one of multiple functions when corresponding middleboxes are unavailable. We describe different goals that network designers can take into account when determining which functions to implement in each of the backup servers. We rely on a graph theoretical model to find properties of efficient assignments and to develop algorithms that can find them. Extensive experiments show, for example, that under realistic function failure probabilities, and reasonable capacity limitations, one can obtain 99.9% survival probability with half the number of servers, compared to standard techniques.

Test design is the process of planning and designing the tests to be performedon a software system. The time scale at which organizations perform and modifytheir test design is typically orders of magnitude larger than that ofdevelopment, especially in modern agile development processes. As a result, often the carefully designed and optimized test suites end up beingrepeatedly executed with no variability between iterations, thus wasting crucialtime and resources. In this work we propose a repeated test planningprocess based on Combinatorial Test Design (CTD), where test iterations areplanned and executed while taking into account the previously executediterations. While each iteration satisfies the same test requirements as before, we leverage the degrees of freedom in test planning in order to reach fullcoverage of higher levels of requirements over the course of iterations. Wesuggest algorithms for doing so efficiently, and evaluate our approach over aset of models collected from different sources in literature. The evaluation demonstrates significant improvement in coverage rate of higher levels, comparedto the naïve approaches.

Software systems tend to suffer from architectural problems as they are being developed. While modern software development methodologies such as Agile and Dev-Ops suggest different ways of assuring code quality, very little attention is paid to maintaining high quality of the architecture of the evolving systems. By detecting and alerting about violations of the intended software architecture, one can often avoid code-level bad smells such as spaghetti code. Typically, if one wants to reason about the software architecture, the burden of first defining the intended architecture falls on the developer's shoulders. This includes definition of valid and invalid dependencies between software components. However, the developers are seldom familiar with the entire software system, which makes this task difficult, time consuming and error-prone. We propose and implement a solution for automatic detection of architectural violations in software artifacts. The solution, which utilizes a number of predefined and user-defined patterns, does not require prior knowledge of the system or its intended architecture. We propose to leverage this solution as part of the nightly build process used by development teams, thus achieving continuous automatic validation of the system's software architecture. As we show in multiple open-source and proprietary cases, a small set of predefined patterns can detect architectural violations as they are introduced over the course of development, and also capture deterioration in existing architectural problems. By evaluating the tool on relatively large open-source projects, we also validate its scalability and practical applicability to large software systems.

Software-intensive systems often consist of many components that interact to fulfill complex functionality. Testing these systems is vital, preferably by a minimal set of tests that covers all relevant cases. The behavior is typically specified by scenarios that describe what the system may, must, or must not do. When designing tests, as in the design of the system itself, the challenge is to consider interactions of scenarios. When doing this manually, critical interactions are easily overlooked. Inspired by Combinatorial Test Design, which exploits that bugs are typically found by regarding the interaction of a small set of parameters, we propose a new test coverage criterion based on scenario interactions. Furthermore, we present a novel technique for automatically synthesizing from Modal Sequence Diagram specifications a minimal set of tests that ensures a maximal coverage of possible t-wise scenario interactions. The technique is evaluated on an example specification from an industrial project.

This work introduces a novel approach for online design and execution of load tests on Cloud applications. Our approach utilizes a Combinatorial Test Design (CTD) engine in order to exercise combinations of levels of resource utilization on the target system's subcomponents. In order to cope with the unpredictability and uncontrollability of Cloud environments, and to align with agile and DevOps paradigms, it designs and executes tests in an iterative online fashion. During test execution, monitoring information is collected from the Cloud, and leveraged for driving and adjusting the subsequent test scenarios. In this work we introduce the overall approach and the algorithms behind it, and demonstrate it on an example setting consisting of three sub-components comprising a typical installation of a web blogging application.

The transition to cloud computing offers a large number of benefits, such as lower capital costs and a highly agile environment. Yet, the development of software engineering practices has not kept pace with this change. Moreover, the design and runtime behavior of cloud based services and the underlying cloud infrastructure are largely decoupled from one another.This paper describes the innovative concepts being developed by CloudWave to utilize the principles of DevOps to create an execution analytics cloud infrastructure where, through the use of programmable monitoring and online data abstraction, much more relevant information for the optimization of the ecosystem is obtained. Required optimizations are subsequently negotiated between the applications and the cloud infrastructure to obtain coordinated adaption of the ecosystem. Additionally, the project is developing the technology for a Feedback Driven Development Standard Development Kit which will utilize the data gathered through execution analytics to supply developers with a powerful mechanism to shorten application development cycles.

Combinatorial Test Design (CTD) is an effective test planning technique that reveals faults resulting from feature interactions in a system. The standard application of CTD requires manual modeling of the test space, including a precise definition of restrictions between the test space parameters, and produces a test suite that corresponds to new test cases to be implemented from scratch. In this work, we propose to use Interaction-based Test-Suite Minimization (ITSM) as a complementary approach to standard CTD. ITSM reduces a given test suite without impacting its coverage of feature interactions. ITSM requires much less modeling effort, and does not require a definition of restrictions. It is appealing where there has been a significant investment in an existing test suite, where creating new tests is expensive, and where restrictions are very complex. We discuss the tradeoffs between standard CTD and ITSM, and suggest an efficient algorithm for solving the latter. We also discuss the challenges and additional requirements that arise when applying ITSM to real-life test suites. We introduce solutions to these challenges and demonstrate them through two real-life case studies.

Combinatorial test design (CTD) is an effective test planning technique that reveals faulty feature interactions in a given system. The test space is modeled by a set of parameters, their respective values, and restrictions on the value combinations. A subset of the test space is then automatically constructed so that it covers all valid value combinations of every t parameters, where t is a user input. When applying CTD to real-life testing problems, it can often occur that the result of CTD cannot be used as is, and manual modifications to the tests are performed. One example is very limited resources that significantly reduce the number of tests that can be used. Another example is complex restrictions that are not captured in the model of the test space. The main concern is that manually modifying the result of CTD might potentially introduce coverage gaps that the user is unaware of. In this paper we present a tool that supports interactive modification of a combinatorial test plan, both manually and with tool assistance. For each modification, the tool displays the new coverage gaps that will be introduced, and enables the user to take educated decisions on what to include in the final set of tests.

Combinatorial test design (CTD) is an effective test planning technique that reveals faults that result from feature interactions in a system. The test space is manually modeled by a set of parameters, their respective values, and restrictions on the value combinations. A subset of the test space is then automatically constructed so that it covers all valid value combinations of every t parameters, where t is usually a user input. In many real-life testing problems, the relationships between the different test parameters are complex. Thus, precisely capturing them by restrictions in the CTD model might be a very challenging and time consuming task. From our experience, this is one of the main obstacles in applying CTD to a wide range of testing problems. In this paper, we introduce two new constructs to the CTD model, counters and value properties, that considerably reduce the complexity of the modeling task, allowing one to easily model testing problems that were practically impossible to model before. We demonstrate the impact of these constructs on two real-life case studies.

Combinatorial test design (CTD) is an effective test planning technique that systematically exercises interactions between parameters of the test space. The test space is manually modeled by a set of parameters, their respective values, and restrictions on the value combinations. A subset of the test space is then automatically constructed so that it covers all valid value combinations of every t parameters, where t is a user input. This paper describes patterns that we have found to be recurring in combinatorial models, i.e., recurring properties of the modeled test spaces. These patterns are often hard to identify and capture correctly in a model, thus are common pitfalls in combinatorial modeling. We describe these patterns, supply methods for identifying them, and suggest simple yet effective solutions for them.

Combinatorial test design (CTD) is an effective test planning technique that reveals faulty feature interaction in a given system. The test space is modeled by a set of parameters, their respective values, and restrictions on the value combinations. A subset of the test space is then automatically constructed so that it covers all valid value combinations of every t parameters, where t is a user input. Various combinatorial testing tools exist, implementing different approaches to finding a set of tests that satisfies t-wise coverage. However, little consideration has been given to the process of defining the test space for CTD, which is usually a manual, labor-intensive, and error-prone effort. Potential errors include missing parameters and their values, wrong identification of parameters and of valid value combinations, and errors in the definition of restrictions that cause them not to capture the intended combinations. From our experience, lack of support for the test space definition process is one of the main obstacles in applying CTD to a wide range of testing domains.

An intrusion detection system (IDS) is usually located and operated at the host, where it captures local suspicious events, or at an appliance that listens to the network activity. Providing an online IDS to the storage controller is essential for dealing with compromised hosts or coordinated attacks by multiple hosts. SAN block storage controllers are connected to the world via block-level protocols, such as iSCSI and Fibre Channel. Usually, block-level storage systems do not maintain information specific to the file-system using them. The range of threats that can be handled at the block level is limited. A file system view at the controller, together with the knowledge of which arriving block belongs to which file or inode, will enable the detection of file-level threats. In this paper, we present IDStor, an IDS for block-based storage. IDStor acts as a listener to storage traffic, out of the controller's I/O path, and is therefore attractive for integration into existing SAN-based storage solutions. IDStor maintains a block-to-file mapping that is updated online. Using this mapping, IDStor infers the semantics of file-level commands from the intercepted block-level operations, thereby detecting file-level intrusions by merely observing the block read and write commands passing between the hosts and the controller.

We consider large scale, distributed storage systems with a redundancy mechanism; cloud storage being a prime example. We investigate how such systems can reduce their power consumption during low-utilization time intervals by operating in a low-power mode. In a low power mode, a subset of the disks or nodes are powered down, yet we ask that each data item remains accessible in the system; this is called full coverage. The objective is to incorporate this option into an existing system rather than redesign the system. When doing so, it is crucial that the low power option should not affect the performance or other important characteristics of the system during full-power (normal) operation. This work is a comprehensive study of what can or cannot be achieved with respect to full coverage low power modes. The paper addresses this question for generic distributed storage systems (where the key component under investigation is the placement function of the system) as well as for specific popular system designs in the realm of storing data in the cloud. Our observations and techniques are instrumental for a wide spectrum of systems, ranging from distributed storage systems for the enterprise to cloud data services. In the cloud environment where low cost is imperative, the effects of such savings are magnified by the large scale.