Abstract We present the OGAN algorithm for automatic requirement falsification of cyber-physical systems. System inputs and outputs are represented as piecewise constant signals over time while requirements are expressed in signal temporal logic. OGAN can find inputs that are counterexamples for the correctness of a system revealing design, software, or hardware defects before the system is taken into operation. The OGAN algorithm works by training a generative machine learning model to produce such counterexamples. It executes tests offline and does not require any previous model of the system under test. We evaluate OGAN using the ARCH-COMP benchmark problems, and the experimental results show that generative models are a viable method for requirement falsification. OGAN can be applied to new systems with little effort, has few requirements for the system under test, and exhibits state-of-the-art CPS falsification efficiency and effectiveness.

Regression testing in software development checks if new software features affect existing ones. Regression testing is a key task in continuous development and integration, where software is built in small increments and new features are integrated as soon as possible. It is therefore important that developers are notified about possible faults quickly. In this article, we propose a test case prioritization schema that combines the use of a static and a dynamic prioritization algorithm. The dynamic prioritization algorithm rearranges the order of execution of tests on the fly, while the tests are being executed. We propose to use a conditional probability dynamic algorithm for this. We evaluate our solution on three industrial datasets and utilize Average Percentage of Fault Detection for that. The main findings are that our dynamic prioritization algorithm can: a) be applied with any static algorithm that assigns a priority score to each test case b) can improve the performance of the static algorithm if there are failure correlations between test cases c) can also reduce the performance of the static algorithm, but only when the static scheduling is performed at a near optimal level.

Context: To effectively defend against ever-evolving cybersecurity threats, software systems should be made as secure as possible. To achieve this, software developers should understand potential vulnerabilities and apply secure coding practices. To prepare these skilled professionals, it is important that cybersecurity concepts are included in programming courses taught at universities. Objective: To present a comprehensive and unbiased literature review on teaching of cybersecurity concepts in programming courses taught at universities. Method: We perform a Systematic Mapping Study. We present six research questions, define our selection criteria, and develop a classification scheme. Results and Conclusions: We select 24 publications. Our results show a wide range of research contributions. We also outline guidelines and identify opportunities for future studies. The guidelines include coverage of security knowledge categories and evaluation of contributions. We suggest that future studies should cover security issues, negative impacts, and countermeasures, as well as apply evaluation techniques that examine students' knowledge. The opportunities for future studies are related to advanced courses, security knowledge frameworks, and programming environments. Furthermore, there is a need of a holistic security framework that covers the security concepts identified in this study and is suitable for education.

We elaborate on a metamorphic approach for testing industrial control systems. The proposed approach consists of two phases: an exploration phase in which we learn about fault patterns of the system under test and an exploitation phase where the observed fault patterns are used for targeted testing. Our method extracts metamorphic relations and input space of the system from its requirements. The seed input used for testing is extracted from the execution logs of the system and used to generate source tests and follow-up tests automatically. The morphed input is constructed based on the seed input and refined using a set of constraints. The approach is exemplified on a position control system and the results show that it is effective in discovering faults with an increased level of automation.

We propose a method for scenario-based testing of maritime collision avoidance systems. The goal is to test an autonomous agent in scenarios that can lead to an unacceptable risk of collision or may clearly not comply with the International Regulations for Preventing Collisions at Sea (COLREGs).Our method is based on the use of a discriminating artificial neural network that is trained online while performing the testing of the agents. Our experimental results show that the proposed algorithm generates test suits composed mostly of challenging scenarios. This allows us to validate quickly if the agent under test can perform the collision avoidance maneuvers safely while abiding the COLREGs.

Performance bottlenecks resulting in high response times and low throughput of software systems can ruin the reputation of the companies that rely on them. Almost two-thirds of performance bottlenecks are triggered on specific input values. However, finding the input values for performance test cases that can identify performance bottlenecks in a large-scale complex system within a reasonable amount of time is a cumbersome, cost-intensive, and time-consuming task. The reason is that there can be numerous combinations of test input values to explore in a limited amount of time. This paper presents PerfXRL, a novel approach for finding those combinations of input values that can reveal performance bottlenecks in the system under test. Our approach uses reinforcement learning to explore a large input space comprising combinations of input values and to learn to focus on those areas of the input space which trigger performance bottlenecks. The experimental results show that PerfxRL can detect 72% more performance bottlenecks than random testing by only exploring the 25% of the input space.

The Foundational Subset for Executable UML Models (fUML) and the Action language for fUML (Alf) can be used for creating executable models in the Eclipse-based UML editing tool called Papyrus. An fUML execution engine in Papyrus, such as Moka, allows to simulate or execute fUML models along with their associated Alf code. However, for exhaustive simulation of such models, one must provide input data required to reach and cover all important elements not only in the graphical fUML models, but also in the textual Alf code. In this paper, we present MATERA2-AlfTester, an Eclipse-plugin for exhaustive simulation and test generation for fUML models. MATERA2-AlfTester integrates with Papyrus and Moka tools and extends their functionally by allowing one to automatically generate test data, test suite with test oracle, and partial Java code at design time. We also present the simulation and testing process of MATERA2-AlfTester with the help of an example and demonstrate how exhaustive simulation and test generation with MATERA2-AlfTester can help designers in assessing and improving the quality of fUML models.

Background: Application Programming Interfaces (APIs) are key to software reuse. Software developers can link functionality and behaviour found in other software with their own software by taking an API into use. However, figuring out how an API works is usually demanding, and may require that the developers spend a notable amount of time familiarizing themselves with the API. Good API documentation is of key importance to simplify this task. Objective: To present a comprehensive, unbiased overview of the state-of-the-art on tools and approaches for API documentation generation. Method: A systematic mapping study on published tools and approaches that can be used for generating API documentation, or for assisting in the API documentation process. Results: 36 studies on API documentation generation tools and approaches analyzed and categorized in a variety of ways. Among other things, the paper presents an overview of what kind of tools have been developed, what kind of documentation they generate, and what sources the documentation approaches require. Conclusion: Out of the identified approaches, many contribute to API documentation in the areas of natural language documentation and code examples and templates. Many of the approaches contribute to ease API users' understanding and learning of the API, but also to the maintenance and generation of API documentation. Most of the approaches are automatic, simplifying the API documentation generation notably, under the assumption that relevant sources for the generation are available. Most of the API documentation approaches are evaluated either by exercise of the approach followed by analysis of the results, or by empirical evaluation methods.

The poor performance of web-based systems can negatively impact the profitability and reputation of the companies that rely on them. Finding those user scenarios which can significantly degrade the performance of a web application is very important in order to take necessary countermeasures, for instance, allocating additional resources. Furthermore, one would like to understand how the system under test performs under increased workload triggered by the worst-case user scenarios. In our previous work, we have formalized the expected behavior of the users of web applications by using probabilistic workload models and we have shown how to use such models to generate load against the system under test. As an extension, in this article, we suggest a performance space exploration approach for inferring the worst-case user scenario in a given workload model which has the potential to create the highest resource utilization on the system under test with respect to a given resource. We propose two alternative methods: one which identifies the exact worst-case user scenario of the given workload model, but it does not scale up for models with a large number of loops, and one which provides an approximate solution which, in turn, is more suitable for models with a large number of loops. We conduct several experiments to show that the identified user scenarios do provide in practice an increased resource utilization on the system under test when compared to the original models.

An Application-Programming Interface or API provides a set of program functions that can be used to build new applications. In this paper, we study how to use the expectation-confirmation theory (ECT) to identify API usability problems, and what obstacles a novice developer faces when learning a new API and its accompanying development tools. We conduct a study over the impact of using a visual editor on API usability and then use the expectation-confirmation theory to study perceptions about the API and the editor. We finally present a list of obstacles found in the study that can be used by others to create more usable APIs and development tools.

Many software engineering organizations around the world are adopting DevOps. One of the goals of DevOps is to foster better collaboration between development and operations personnel, in order to improve organizational efficiency. Since DevOps is lacking a common definition, there are several approaches to adopt it, and organizations largely need to determine how to apply DevOps for themselves. In this paper, we present results from a case study in which a software organization adopts DevOps. The focus of this research is to study the impact of mixing the responsibilities between development and operations engineers. We interviewed 14 employees in the organization during the study, and results indicate several benefits of the chosen approach, such as improved collaboration and trust, and smoother work flow. This comes at the cost of a number of complications, such as new sources for friction among the employees, risk for holistically sub-optimal service configurations, and more.

We present a model and system for deciding on computing versus storage trade-offs in the Cloud using von Neumann-Morgenstern lotteries. We use the decision model in a video-on-demand system providing cost-efficient transcoding and storage of videos. Video transcoding is an expensive computational process that converts a video from one format to another. Video data are large enough to cause concern over rising storage costs. In the general case, our work is of interest when dealing with expensive computations that generate large results that can be cached for future use. Solving the decision problem entails solving two sub-problems: how long to store cached objects and how many requests we can expect for a particular object in that duration. We compare the proposed approach to always storing and to our previous approach over one year using discrete-event simulations. We observe a 72% cost reduction compared to always storing and a 13% reduction compared to our previous approach. This reduction in cost stems from the proposed approach storing fewer unpopular objects when it does not regard it as cost-efficient to do so.

High energy consumption of cloud data centers is a matter of great concern. Dynamic consolidation of Virtual Machines (VMs) presents a significant opportunity to save energy in data centers. A VM consolidation approach uses live migration of VMs so that some of the under-loaded Physical Machines (PMs) can be switched-off or put into a low-power mode. On the other hand, achieving the desired level of Quality of Service (QoS) between cloud providers and their users is critical. Therefore, the main challenge is to reduce energy consumption of data centers while satisfying QoS requirements. In this paper, we present a distributed system architecture to perform dynamic VM consolidation to reduce energy consumption of cloud data centers while maintaining the desired QoS. Since the VM consolidation problem is strictly NP-hard, we use an online optimization metaheuristic algorithm called Ant Colony System (ACS). The proposed ACS-based VM Consolidation (ACS-VMC) approach finds a near-optimal solution based on a specified objective function. Experimental results on real workload traces show that ACS-VMC reduces energy consumption while maintaining the required performance levels in a cloud data center. It outperforms existing VM consolidation approaches in terms of energy consumption, number of VM migrations, and QoS requirements concerning performance.

As the scale of a cloud data center becomes larger and larger, the energy consumption of the data center also grows rapidly. Dynamic consolidation of Virtual Machines (VMs) presents a significant opportunity to save energy by turning off unused Physical Machines (PMs) in data centers. In this paper, we present a distributed controller to perform dynamic VM consolidation to improve the resource utilizations of PMs and to reduce their energy consumption. Moreover, we use the ant colony system to find a near-optimal VM placement solution based on the specified objective function. Experimental results on the real workload traces from more than a thousand PlanetLab VMs show that the proposed approach reduces energy consumption and maintains required performance levels in a large-scale data center.

We present a model and system for cost-efficient and reliable management of sessions in a Cloud, based on the von Neumann-Morgenstern utility theorem. Our model enables a web application provider to maximize profit while maintaining a desired quality of service. The objective is to determine whether, when, where, and how long to store a session, given multiple storage options with various properties, e.g. cost, capacity, and reliability. Reliability is affected by three factors: how often session state is stored, how many stores are used, and how reliable those stores are. To account for these factors, we use a Markovian reliability model and treat the valid storage options for each session as a von Neumann-Morgenstern lottery. We proceed by representing the resulting problem as a knapsack problem, which can be heuristically solved for a good compromise between efficiency and effectiveness. We analyze the results from a discrete-event simulation involving multiple session management policies, including two utility-based policies: a greedy heuristic policy intended to give real-time performance and a reference policy based on solving the linear programming relaxation of the knapsack problem, giving a theoretical upper bound on achievable utility. As the focus of this work is exploratory, rather than performance-based, we do not directly measure the time required for solving the model. Instead, we give the computational complexity of the algorithms. Our results indicate that otherwise unprofitable services become profitable through utility-based session management in a cloud setting. However, if the costs are much lower than the expected revenues, all policies manage to turn a profit. Different policies performed the best under different circumstances.

Infrastructure as a Service (IaaS) clouds provide virtual machines (VMs) under a pay-per-use business model, which can be used to create a dynamically scalable cluster of servers to deploy one or more web applications. In contrast to the traditional dedicated hosting of web applications where each VM is used exclusively for one particular web application, the shared hosting of web applications allows improved VM utilization by sharing VM resources among multiple concurrent web applications. However, in a shared hosting environment, dynamic scaling alone does not minimize over-provisioning of VMs. In this paper, we present a novel approach to consolidate multiple web applications in a cloud-based shared hosting environment. The proposed approach uses Ant Colony Optimization (ACO) to build a web application migration plan, which is then used to minimize over-provisioning of VMs by consolidating web applications on under-utilized VMs. The proposed approach is demonstrated in discrete-event simulations and is evaluated in a series of experiments involving synthetic as well as realistic load patterns.

We present an integrated approach to design and validate RESTful composite web services. We use the Unified Modeling Language (UML) to specify the requirements, behavior and published resources of each web service. In our approach, a service can invoke other services and exhibit complex and timed behavior while still complying with the REST architectural style. We show how to transform service specifications into UPPAAL timed automata for verification and test generation. The service requirements are propagated to the UPPAAL timed automata during the transformation. Their reachability is verified in UPPAAL and they are used for computing coverage level during test generation. We validate our approach with a case study of a holiday booking web service.

This paper presents a novel approach for stream-based admission control and job scheduling for video transcoding called SBACS (Stream-Based Admission Control and Scheduling). SBACS uses queue waiting time of transcoding servers to make admission control decisions for incoming video streams. It implements stream-based admission control with per stream admission. To ensure efficient utilization of the transcoding servers, video streams are segmented at the Group of Pictures level. In addition to the traditional rejection policy, SBACS also provides a stream deferment policy, which exploits cloud elasticity to allow temporary deferment of the incoming video streams. In other words, the admission controller can decide to admit, defer, or reject an incoming stream and hence reduce rejection rate. In order to prevent transcoding jitters in the admitted streams, we introduce a job scheduling mechanism, which drops a small proportion of video frames from a video segment to ensure continued delivery of video contents to the user. The approach is demonstrated in a discrete-event simulation with a series of experiments involving different load patterns and stream arrival rates.

This paper presents prediction-based dynamic resource allocation algorithms to scale video transcoding service on a given Infrastructure as a Service cloud. The proposed algorithms provide mechanisms for allocation and deallocation of virtual machines (VMs) to a cluster of video transcoding servers in a horizontal fashion. We use a two-step load prediction method, which allows proactive resource allocation with high prediction accuracy under real-time constraints. For cost-efficiency, our work supports transcoding of multiple on-demand video streams concurrently on a single VM, resulting in a reduced number of required VMs. We use video segmentation at group of pictures level, which splits video streams into smaller segments that can be transcoded independently of one another. The approach is demonstrated in a discrete-event simulation and an experimental evaluation involving two different load patterns.

In this paper, we present an approach for performance testing of software services. In our approach, we use Probabilistic Timed Automata (PTA) to model the workload of the system, by describing how different user types interact with the system. We use these models to generate load in real-time and we measure different performance indicators.

This paper presents a prediction-based dynamic resource allocation approach for web applications called CRAMP (Cost-efficient Resource Allocation for Multiple web applications with Proactive scaling). The proposed approach provides automatic deployment and proactive scaling of multiple simultaneous web applications on a given Infrastructure as a Service cloud in a shared hosting environment. It monitors and uses resource utilization metrics and does not require a performance model of the applications or the infrastructure dynamics. The shared hosting environment allows us to share virtual machine (VM) resources among deployed applications, reducing the number of required VMs. The approach is demonstrated in a prototype implementation that has been deployed in the Amazon Elastic Compute Cloud.

A UML protocol state machine describes a behavioral interface for a class as a number of states and transitions between states triggered by method calls. In this paper, we present an approach to generate behavioral class interfaces in the form of class contracts from UML protocol state machines. The generated contracts can be used for documentation, test case generation, test case oracle, and as run-time assertions and thus help to test and validate the implementation of a class against its interface. We formalize protocol state machines with its structure and semantics for generating class contracts. The state invariants of the source and target states are considered along with the pre- and post-conditions of the transitions. Different types of transitions like simple, join, fork, high-level, and self transitions are supported, as well as non-deterministic behavior. The approach is supported by a tool to generate automatically the contracts from UML models.

Clinical guidelines systematically assist practitioners with providing appropriate health care for specific clinical circumstances. However, a significant number of guidelines are lacking in quality. In this paper, we use the UML modeling language to capture guidelines and model checking techniques for their verification. We have established a classification of possible properties to be verified in a guideline and we present an automated approach based on a translation from UML to PROMELA, the input language of the SPIN model checker. Our approach is illustrated with a guideline based on a guideline published by the National Guideline Clearing House (NGC).

Statecharts is a design notation to model reactive systems that is part of the Unified Modeling Language (UML) and it is commonly used in the automotive and telecommunication software industry. In this paper we present a study of how the use of some statechart features correlate to the number of pre-release defects in the maintenance of large systems. We discuss possible causes for these correlations and provide some advice to both UML practitioners and to designers of new visual design languages.