Abstract In the last years, the use of microservice architectures is spreading in Cyber-Physical Systems (CPSs) and Internet of Things (IoT) domains. CPSs are systems that integrate digital cyber computations with physical processes. The development of software for CPSs demands a constant maintenance to support new requirements, bug fixes, and deal with hardware obsolescence. The key in this process is code testing and more if the code is fragmented during the development of CPSs. It is important to remark that this process is challenging and time-consuming. In this paper, we report on the experience of instantiating of the microservice-based architecture for DevOps of CPSs to test elevator dispatching algorithms across different test levels (i.e., SiL, HiL and Operation). Such an architecture allows for a continuous deployment, monitoring and validation of CPSs. By integrating the approach with a real industrial case study, we demonstrate that our approach reduces significantly the time needed in the testing process and consequently, reduces the economic cost of the entire process.

While most large web-based software systems (e.g., Amazon, Google) release a new software version every almost a minute, in the context of Cyber-Physical Systems (CPSs), this is still far. However, the software of CPSs needs to evolve while these are in operation to fix bugs, add new functionalities, carry out refactoring activities and deal with unforeseen situations that were discovered while the CPS was operating. In the last three years, the Adeptness project has been developing in a solution to help speedup the software release of CPSs that are in operation while guaranteeing their reliability. In this paper, we summarize the objectives, results and lessons learned from this H2020 project.

The time it takes software systems to be tested is usually long. Search-based test selection has been a widely investigated technique to optimize the testing process. In this article, we propose a set of seeding strategies for the test case selection problem that generates the initial population of Pareto-based multi-objective algorithms, with the goals of (1) helping to find an overall better set of solutions and (2) enhancing the convergence of the algorithms. The seeding strategies were integrated with four state-of-the-art multi-objective search algorithms and applied into two contexts where regression-testing is paramount: (1) Simulation-based testing of Cyber-physical Systems and (2) Continuous Integration. For the first context, we evaluated our approach by using six fitness function combinations and six independent case studies, whereas in the second context, we derived a total of six fitness function combinations and employed four case studies. Our evaluation suggests that some of the proposed seeding strategies are indeed helpful for solving the multi-objective test case selection problem. Specifically, the proposed seeding strategies provided a higher convergence of the algorithms towards optimal solutions in 96% of the studied scenarios and an overall cost-effectiveness with a standard search budget in 85% of the studied scenarios.

Cyber-physical systems(CPSs) are a new generation of systems, which integrate software with physical processes. The increasing complexity of these systems, combined with the uncertainty in their interactions with the physical world, makes the definition of effective test oracles especially challenging, facing the well-knowntest oracle problem. Metamorphic testing has shown great potential to alleviate the test oracle problem by exploiting the relations among the inputs and outputs of different executions of the system, so-calledmetamorphic relations(MRs). In this article, we propose an MR pattern called PV for the identification of performance-driven MRs, and we show its applicability in two CPSs from different domains, which are automated navigation systems and elevator control systems. For the evaluation, we assessed the effectiveness of this approach for detecting failures in an open-source simulation-based autonomous navigation system, as well as in an industrial case study from the elevation domain. We derive concrete MRs based on the PV pattern for both case studies, and we evaluate their effectiveness with seeded faults. Results show that the approach is effective at detecting over 88% of the seeded faults, while keeping the ratio of FPs at 4% or lower.

Orona is a world-renowned elevators developer. During elevators' lives, their software continues to evolve, e.g., due to hardware obsolescence, requirements changes, vulnerabilities, and bug corrections. Such continuous evolution demands the continuous testing of industrial elevators with the minimum manual effort possible. To this end, we present a tool, whose core component is a domain-specific language (DSL) with which a user can specify test oracles at a higher level of abstraction and independent of a testing level. The DSL also supports specifying uncertainty-aware test oracles to test elevators under various uncertainties inherent in them. Finally, the DSL is also equipped with test oracle generation that generates test oracle code automatically at the different DevOps testing levels (i.e., Software and Hardware-in-the-Loop test levels, and in operation) to enable reuse of test oracles across these levels. We evaluated this DSL with an industrial elevators case study at Orona's site to specify and generate test oracles. The evaluation showed that the high expressiveness of the DSL permits the high-level definition of test oracles in our industrial context. Based on the industrial application, we discuss our experiences and lessons learned.

Abstract The software of systems of elevators needs constant maintenance to deal with new functionality, bug fixes, or legislation changes. To automatically validate the software of these systems, a typical approach in industry is to use regression oracles, which execute test inputs both in the software version under test and in a previous software version. However, these practices require a long test execution time and cannot be re‐used at different test phases. To deal with these issues, we propose Dispatching AlgoRIthm Oracle (DARIO), a test oracle that relies on regression machine‐learning algorithms to detect both functional and non‐functional problems of the system. The machine‐learning algorithms of this oracle are trained by using data from previously tested versions to predict reference functional and non‐functional performance values of the new versions. An empirical evaluation with an industrial case study demonstrates the feasibility of using our approach. A total of five regression learning algorithms were validated by using mutation testing techniques. For the context of functional bugs, the accuracy when predicting verdicts by DARIO ranged between 95% and 98%, across the different scenarios proposed. For the context of non‐functional bugs, were competitive too, having an accuracy when predicting verdicts by DARIO ranged between 83% and 87%.

The software of elevators requires maintenance over several years to deal with new functionality, correction of bugs or legislation changes. To automatically validate this software, test oracles are necessary. A typical approach in industry is to use regression oracles. These oracles have to execute the test input both, in the software version under test and in a previous software version. This practice has several issues when using simulation to test elevators dispatching algorithms at system level. These issues include a long test execution time and the impossibility of re-using test oracles both at different test levels and in operation. To deal with these issues, we propose DARIO, a test oracle that relies on regression learning algorithms to predict the Qualify of Service of the system. The regression learning algorithms of this oracle are trained by using data from previously tested versions. An empirical evaluation with an industrial case study demonstrates the feasibility of using our approach in practice. A total of five regression learning algorithms were validated, showing that the regression tree algorithm performed best. For the regression tree algorithm, the accuracy when predicting verdicts by DARIO ranged between 79 to 87%.

One of the major challenges in the verification of complex industrial Cyber-Physical Systems is the difficulty of determining whether a particular system output or behaviour is correct or not, the so-called test oracle problem. Metamorphic testing alleviates the oracle problem by reasoning on the relations that are expected to hold among multiple executions of the system under test, which are known as Metamorphic Relations (MRs). However, the development of effective MRs is often challenging and requires the involvement of domain experts. In this paper, we present a case study aiming at automating this process. To this end, we implemented GAssertMRs, a tool to automatically generate MRs with genetic programming. We assess the cost-effectiveness of this tool in the context of an industrial case study from the elevation domain. Our experimental results show that in most cases GAssertMRs outperforms the other baselines, including manually generated MRs developed with the help of domain experts. We then describe the lessons learned from our experiments and we outline the future work for the adoption of this technique by industrial practitioners.

The time it takes software systems to be tested is usually long. This is often caused by the time it takes the entire test suite to be executed. To optimize this, regression test selection approaches have allowed for improvements to the cost-effectiveness of verification and validation activities in the software industry. In this area, multi-objective algorithms have played a key role in selecting the appropriate subset of test cases from the entire test suite. In this paper, we propose a set of seeding strategies for the test case selection problem that generate the initial population of multi-objective algorithms. We integrated these seeding strategies with an NSGA-II algorithm for solving the test case selection problem in the context of simulation-based testing. We evaluated the strategies with six case studies and a total of 21 fitness combinations for each case study (i.e., a total of 126 problems). Our evaluation suggests that these strategies are indeed helpful for solving the multi-objective test case selection problem. In fact, two of the proposed seeding strategies outperformed the NSGA-II algorithm without seeding population with statistical significance for 92.8 and 96% of the problems.

Elevators are among the oldest and most widespread transportation systems, yet their complexity increases rapidly to satisfy customization demands and to meet quality of service requirements. Verification and validation tasks in this context are costly, since they rely on the manual intervention of domain experts at some points of the process. This is mainly due to the difficulty to assess whether the elevators behave as expected in the different test scenarios, the so-called test oracle problem. Metamorphic testing is a thriving testing technique that alleviates the oracle problem by reasoning on the relations among multiple executions of the system under test, the so-called metamorphic relations. In this practical experience paper, we report on the application of metamorphic testing to verify an industrial elevator dispatcher. Together with domain experts from the elevation sector, we defined multiple metamorphic relations that consider domain-specific quality of service measures. Evaluation results with seeded faults show that the approach is effective at detecting faults automatically.

Software systems that are embedded in autonomous Cyber-Physical Systems (CPSs) usually have a large life-cycle, both during its development and in maintenance. This software evolves during its life-cycle in order to incorporate new requirements, bug fixes, and to deal with hardware obsolescence. The current process for developing and maintaining this software is very fragmented, which makes developing new software versions and deploying them in the CPSs extremely expensive. In other domains, such as web engineering, the phases of development and operation are tightly connected, making it possible to easily perform software updates of the system, and to obtain operational data that can be analyzed by engineers at development time. However, in spite of the rise of new communication technologies (e.g., 5G) providing an opportunity to acquire Design-Operation Continuum Engineering methods in the context of CPSs, there are still many complex issues that need to be addressed, such as the ones related with hardware-software co-design. Therefore, the process of Design-Operation Continuum Engineering for CPSs requires substantial changes with respect to the current fragmented software development process. In this paper, we build a taxonomy for Design-Operation Continuum Engineering of CPSs based on case studies from two different industrial domains involving CPSs (elevation and railway). This taxonomy is later used to elicit requirements from these two case studies in order to present a blueprint on adopting Design-Operation Continuum Engineering in any organization developing CPSs.

Monitoring Industrial Cyber-Physical Systems (ICPSs) is increasingly becoming essential, which requires the development of new approaches to capture data from an ICPS for visualization, automated analyses, decision making, and anomaly detection. Towards this end, first, we present requirements for enabling personalized data visualization for monitoring evolving ICPS during its operation. Such requirements were identified from our experience of designing and developing an industrial Automated Warehouse (AW). Second, the identified requirements were validated with a questionnaire-based survey by soliciting views of industry experts from the Software Monitoring Systems and Data Visualizations domains. Based on the analysis of the survey results, the need of developing personalized ICPS monitoring systems was confirmed as a step forward to enable the efficient detection of anomalies, improvement in productivity, and providing a better overview of the monitored ICPSs. Third, we developed a model-based visualization system (named as PAVS) for monitoring ICPSs, which conforms to the identified requirements, and was validated it with a dataset collected from a real AW developed by our industrial partner: ULMA Handling System, Spain. In the end, we also provide a set of lessons learned when PAVS was transferred to ULMA.

In many domains, engineers build simulation models (e.g., Simulink) before developing code to simulate the behavior of complex systems (e.g., Cyber-Physical Systems). Those models are commonly heavy to simulate which makes it difficult to execute the entire test suite. Furthermore, it is often difficult to measure white-box coverage of test cases when employing such models. In addition, the historical data related to failures might not be available. This paper proposes a cost-effective approach for test case selection that relies on black-box data related to inputs and outputs of the system. The approach defines in total five effectiveness measures and one cost measure followed by deriving in total 15 objective combinations and integrating them within Non-Dominated Sorting Genetic Algorithm-II (NSGA-II). We empirically evaluated our approach with all these 15 combinations using four case studies by employing mutation testing to assess the fault revealing capability. The results demonstrated that our approach managed to improve Random Search by 26% on average in terms of the Hypervolume quality indicator.

The test case generation and prioritization of industrial cyber-physical systems face critical challenges, and simulation-based testing is one of the most commonly used techniques for testing these complex systems. However, simulation models of industrial CPSs are usually very complex, and executing the simulations becomes computationally expensive, which often make it infeasible to execute all the test cases. To address these challenges, this paper proposes a multi-objective test generation and prioritization approach for testing industrial CPSs by defining a fitness function with four objectives and designing different crossover and mutation operators. We empirically evaluated our fitness function and designed operators along with five multi-objective search algorithms [e.g., nondominated sorting genetic algorithm (NSGA-II)] using four case studies. The evaluation results demonstrated that NSGA-II achieved significantly better performance than the other algorithms and managed to improve random search for on average 43.80% for each objective and 49.25% for the quality indicator hypervolume.

The test case generation of Cyber-Physical Systems (CPSs) face critical challenges that traditional methods such as Model-Based Testing cannot deal with. As a result, simulation-based testing is one of the most commonly used techniques for testing CPSs despite sometimes being computationally too expensive. This paper proposes a search-based approach which is implemented on top of Non-dominated Sorting Genetic Algorithm II (NSGA-II), the most commonly applied multi-objective search algorithm for cost-effectively generating executable test cases in order to test CPSs. With the aim of guiding the generation of the optimal set of so-called reactive test cases, the approach formally defines three cost-effectiveness measures: requirements coverage, test case similarity and test execution time. Furthermore, we design one crossover operator and three mutation operators (i.e., mutation at test suite level named Mu TS, mutation at test case level named Mu TC and mutation at both levels named Mu BO) for test case generation. We evaluate our approach by comparing with Random Search (RS) using four case studies (one of them is an industrial system). Moreover, we evaluate the three mutation operators using the four case studies. The results of the experiment (with a rigorous statistical analysis) indicated that our approach in conjunction with the crossover operator operation and three mutation operators significantly outperformed RS. In general, Mu BO achieved the best performance among the three mutation operators and managed to improve on average the test execution time by 14%, the requirements coverage by 34%, and the test similarity by 75% as compared with RS.

With buildings accounting for nearly 40 percent of global energy consumption, the improvement of energy efficiency in buildings and districts is a clear opportunity in the fight against climate change. This has led engineering practitioners as well as researchers to propose solutions for smart energy control in buildings. Simulation-based methods permits the early validation of engineering solutions for these smart energy control systems. However, since these solutions involve different engineering disciplines (e.g., software engineering, electrical engineering, etc.), different simulation tools might be employed. We propose a tool that interconnects EnergyPlus, one of the leading open-source tools for building energy simulation, with Crescendo, a tool for designing and modelling cyber-physical systems using formal methods. A preliminary evaluation suggests that the proposed solution enables the simulation between these two tools in an efficient manner.

The paper presents the action research conducted in the context of System Engineering teaching in Embedded Systems Master to improve the motivation of students. These are the steps carried out: Identification of the problem domain, diagnosis of the problem, action hypothesis definition, design of the action plan, the action and its analysis, reflection and conclusions.

In this paper we present an approach to reduce the verification cost of distributed elevator control systems through embedded code re-use to co-simulate networked devices in a hardware-in-the-loop simulator: the CAN Restbus simulator. The approach is applied to a case study in the field of distributed control system for elevators movement. We discuss the use cases for the CAN Restbus simulator, the functionality of the devices to simulate, the rationale for the development and integration approach and the validation procedures of the implemented Restbus on the test-bench. We describe the adaptations needed to cross-compile the legacy source code for the test platform processor, the shortcomings related to the programming style and tools, the performance and integration issues arising when integrating in the test system. We also discuss the operational issues due to the limits of feasible synchronization. From this experience we conclude that the re-use of pre-validated code is a cost-effective approach to build realistic-behaviour reactive test components, the main saving found at the verification of the test component itself. Finally, we put forward an outlook about forthcoming developments of the overall test system.

Cyber-Physical Systems (CPSs) can be found in many sectors (e.g., automotive and aerospace). These systems are usually configurable to give solutions based on different needs. The variability of these systems is large, which implies they can be set into millions of configurations. As a result, different testing processes are needed to efficiently test these systems: the appropriate configurations must be selected and relevant test cases for each configuration must be chosen as well as prioritized. Prioritizing the order in which the test cases are executed reduces the time for detecting faults in these kinds of systems. However, the test suite size is often large and exploring all the possible test case orders is infeasible. Search algorithms can help find optimal solutions from a large solution space. This paper presents an approach based on weight-based search algorithms for prioritizing the test cases for configurable CPSs. We empirically evaluate the performance of the following algorithms with two case studies: Weight-Based Genetic Algorithms, Random Weighted Genetic Algorithms, Greedy, Alternating Variable Method and Random Search (RS). Our results suggest that all the search algorithms outperform RS, which is taken as a baseline. Local search algorithms have shown better performance than global search algorithms.

Model-based software development promises improvements in terms of quality and cost by raising the abstraction level of the development from code to models, but also requires mature techniques and tools. Although Eclipse Modelling Framework (EMF) introduces a default persistence mechanism for models, namely XMI, its usage is often limited as model size increases. To overcome this limitation, during the last years alternative persistence mechanisms have been proposed in order to store models in RDBMS and NoSQL databases. Under this new paradigm, model operations can be performed at model-level and persistence-level, e.g., mapping EOL model operations into SQL statements. In this paper, we extend our framework (called MQT) to support CRUD (Create, Read, Update and Delete) operations from model- (EOL) to persistence-level (SQL), using a streaming execution of queries at run-time. Through comparable evaluation metrics, we evaluate the performance and memory footprint of the framework usi ng the GraBaTs scenario.

Cyber-Physical Systems (CPSs) are often tested at different test levels following "X-in-the-Loop" configurations: Model-, Software- and Hardware-in-the-loop (MiL, SiL and HiL). While MiL and SiL test levels aim at testing functional requirements at the system level, the HiL test level tests functional as well as non-functional requirements by performing a real-time simulation. As testing CPS product line configurations is costly due to the fact that there are many variants to test, test cases are long, the physical layer has to be simulated and co-simulation is often necessary. It is therefore extremely important to select the appropriate test cases that cover the objectives of each level in an allowable amount of time. We propose an efficient test case selection approach adapted to the "X-in-the-Loop" test levels. Search algorithms are employed to reduce the amount of time required to test configurations of CPS product lines while achieving the test objectives of each level. We empirically evaluate three commonly-used search algorithms, i.e., Genetic Algorithm (GA), Alternating Variable Method (AVM) and Greedy (Random Search (RS) is used as a baseline) by employing two case studies with the aim of integrating the best algorithm into our approach. Results suggest that as compared with RS, our approach can reduce the costs of testing CPS product line configurations by approximately 80% while improving the overall test quality.

The use of Model Driven Development (MDD) is increasing in industry. When a Non Functional Requirement (NFR) not considered in the development must be added metamodels, models and also transformations are affected. Tasks for defining and maintaining model transformation rules can be complex in MDD. Model Transformation By Example (MTBE) approaches have been proposed to ease the development of transformation rules. In this paper an approach based on MTBE to derive the adaptation operations that must be implemented in a legacy model transformation when a NFR appears is presented. The approach derives semi-automatically the model transformations using execution traceability data and models differences. An example where access control property is integrated on a MDD system is introduced to demonstrate the usefulness of the tool to evolve model transformations.

Persisting and querying models larger than a few tens of megabytes using XMI introduces a significant time and memory footprint overhead to MDD workflows. In this paper, we present an approach that attempts to address this issue using an embedded relational database as an alternative persistence layer for EMF models, and runtime translation of OCL-like expressions for efficiently querying such models. We have performed an empirical study of the approach using a set of large-scale reverse engineered models and queries from the Grabats 2009 Reverse Engineering Contest. Main contribution of this paper is the Model Query Translator, an approach that translates (and executes) at runtime queries from model-level (EOL) to persistence-level (SQL).

Cyber-Physical Systems (CPSs) product lines appear in a wide range of applications of different domains (e.g., car's doors' windows, doors of a lift, etc.). The variability of these systems is large and as a result they can be configured into plenty of configurations. Testing each of the configurations can be time consuming as not only software has to be simulated, but also the hardware and the physical layer of the CPS, which is often modelled with complex mathematical models. Choosing the adequate test control strategy is critical when testing CPSs product lines. This paper presents a set of test control algorithms organized in an architecture of three layers (domain, application and simulation) for testing CPSs product lines. An illustrative example of a CPS product line is presented and three experiments are conducted to measure the performance of the proposed test control algorithms. We conclude that test scheduling and test suite minimization significantly help to reduce the overall test costs while preserving the test quality in CPSs product lines. In addition, we conclude that knowing the results of the previously tested configurations permits reducing the time for the detection of anomalous designs.

The number of cyber-physical systems is increasing considerably, being very usual in automation systems. Many cyber-physical systems must deal with variability issues to give response to the current market needs. The use of a plant model that simulates the system controlled by the embedded system might ease verification and validation activities, but building plant models manually for cyber-physical systems with many variants can be time consuming and error prone. This paper proposes a methodology that semi-automatically generates plant models in Simulink, with the main purpose of handling variability issues to validate highly configurable cyber-physical systems.

Cloud service-based applications are to be adapted to serve multiple platforms and stakeholders. Atop of such services, Smart Green Buildings are fostering a plethora of processes within their sustainability life-cycle. This introduces a number of challenges, as how to support multiple perspectives of domain-specific variability and how to deal with large collections of related process variants. To tackle this, there is a need to handle multiperspective variability for processes. This paper introduces an approach to manage multi-perspective process variability by means of a meta-model and a modeling methodology, representing separately people and things variability perspectives in smart environments. Initial experimental results are also described, which indicate encouraging results for managing highly complex variability models.

In dynamic environments, changes are often unpredictable and complex. Process models cannot be fully specified up-front and process flexibility becomes a key issue. Enterprise applications and systems supporting such processes are increasingly being architected in a service-oriented style. In this light, our goal is to analyze service orchestration approaches from a process flexibility perspective. Through a systematic literature review, we evaluate 17 service orchestration approaches and analyze their support for: (i) variability, support for large collections of process variants, (ii) adaptation, need for instance changes during runtime, (iii) evolution, need for schema changes during runtime, and (iv) looseness, need for loosely-specified models. The review findings provide a clearer understanding of process flexibility requirements and service orchestration mechanisms that support them, helping us to understand the limitations and shed light on future research areas.

SUMMARY Nowadays, embedded systems are gaining importance. At the same time, the development of their software is increasing its complexity, having to deal with quality, cost, and time‐to‐market issues among others. With stringent quality requirements such as performance, early verification and validation become critical in these systems. In this regard, advanced development paradigms such as model‐driven engineering and software product line engineering bring considerable benefits to the development and validation of embedded system software. However, these benefits come at the cost of increasing process complexity. This work presents a process based on UML and MARTE for the analysis of embedded model‐driven product lines. It specifies the tasks, the involved roles, and the workproducts that form the process and how it is integrated in the more general development process. Existing tools that support the tasks to be performed in the process are also described. A classification of such tools and a study of traceability among them are provided, allowing engineering teams to choose the most adequate chain of tools to support the process. Copyright © 2012 John Wiley & Sons, Ltd.

Nowadays, embedded system development is increasing its complexity dealing with quality, cost and time-to-market among others. Quality attributes are an important issue to consider in embedded software development where time issues may be critical. Development paradigms such as Model Driven Development and Software Product Lines can be an adequate alternative to traditional software development and validation methods due to the characteristics of embedded systems. But for a proper validation and verification based on MARTE model analysis, all variability issues and critical quality attributes that take part in analysis must be properly modelled and managed. Therefore, a model analysis process for Model Driven Embedded Software Product Lines has been defined as some process lacks have been found.

Variability is a key aspect in software product lines. Functional variability has been largely studied as a way to obtain all the desired products for a line. Quality variability, less understood and more complex, has not received so much attention by researchers. However, different members of the line may require different levels of a quality attribute. The design phase is a good point to assure that quality attributes requirements are met within the product line so this means paying attention to software architecture evaluation during domain engineering. The quality evaluation in software product lines is much more complicated than in single-systems as products can require different quality levels and the product line can have variability on design that in turn affects quality. The evaluation of all the products of a line is very expensive. Thus, ways of reducing the evaluation efforts are necessary. Herein is presented a method for facilitating cost-effective quality evaluation of a product line taking into consideration variability on quality attributes.

Obtaining the appropriate architecture for a software system assures the long-term success of the product. Without a well defined and documented architecture is very hard or impossible to analyze and evaluate the quality of the product and the change impact. Thus it is very expensive to communicate to each stakeholder the information needed. Many existing approaches propose models based on a "closed" set of views to document architectures, such as "4+1" model, Siemens’ model, etcetera. Nevertheless, industrial practices in Small and Medium size Enterprises (SME) works informally, often even not documenting. From a practical experience where the 4+1 model was applied, it was noticed that there were views not proposed that were interesting, therefore a method was defined to select the views that really turn out useful to document in each case. To this end, the approach proposed by Clements et al [1] has been adapted.