Context: Stateflow models are widely used in the industry to model the high-level control logic of Cyber-Physical Systems (CPSs) in Simulink. Many approaches exist to test Simulink models, but once a fault is detected, the process to repair it remains manual. Such a manual process increases the software development cost. Automated Program Repair (APR) techniques can significantly reduce this cost by automatically generating patches that fix bugs. However, current approaches face scalability issues to be applicable in the CPS context. Objectives: The goal of this paper is to propose an APR method which is scalable for Stateflow models. Method: We propose an automated search-based approach called FlowRepair , explicitly designed to repair Stateflow models. The novelty of FlowRepair includes, (1) a new algorithm that combines global and local search for patch generation; (2) a definition of novel repair objectives specifically tailored for repairing CPSs; (3) a set of mutation operators to repair Stateflow models automatically; and (4) an evaluation on a new dataset encompassing 19 faulty stateflow models with real bugs. Results: Our results suggest that (1) FlowRepair can fix bugs in stateflow models; (2) FlowRepair surpasses or performs similarly to a baseline APR technique inspired by a well-known CPS program repair approach. Conclusion: This paper presents the first tool for APR CPSs whose high-level control program is developed in Simulink-Staflow. The results show that the approach is effective and scalable to such complex systems.

Healthcare Internet of Things (IoT) applications often integrate various third-party healthcare applications and medical devices through REST APIs, resulting in complex and interdependent networks of REST APIs. Oslo City’s healthcare department collaborates with various industry partners to develop these applications, enriched with diverse REST APIs that evolve during the DevOps process to accommodate evolving needs such as new features, services, and devices. Oslo City’s primary goal is to utilize automated solutions for continuous testing of REST APIs at each evolution stage to ensure dependability. Although the literature offers various automated REST API testing tools, their effectiveness in regression testing of the evolving REST APIs of healthcare IoT applications within a DevOps context remains undetermined. This article evaluates state-of-the-art and well-established REST API testing tools—specifically, RESTest, EvoMaster, Schemathesis, RESTler, and RestTestGen—for the regression testing of a real-world healthcare IoT application, considering failures, faults, coverage, regressions, and cost. We conducted experiments using all accessible REST APIs (17 APIs with 120 endpoints), and 14 releases evolved during DevOps. Overall, all tools generated tests leading to several failures, 18 potential faults, up to 84% coverage, and 23 regressions. Over 70% of tests generated by all tools fail to detect failures, resulting in significant overhead.

With the rapid advancement of quantum computing, research on quantum machine learning (QML) algorithms has grown significantly. Among these, the Quantum Neural Network (QNN) stands out as one of the promising algorithms that integrates the principles of quantum computing with artificial neural networks to process data. Inspired by applications of QNN across fields, we investigate their use in software testing for the Cancer Registry of Norway (CRN), part of the Norwegian Institute of Public Health (NIPH), responsible for cancer statistics among the Norwegian population. CRN develops a complex socio-technical software system, Cancer Registration Support System ( \(\mathtt{CaReSS}\) ), interacting with many entities (e.g., hospitals, medical laboratories, and other patient registries) to achieve its task. For cost-effective testing of \(\mathtt{CaReSS}\) , CRN has employed \(\mathtt{EvoMaster}\) , an AI-based REST API testing tool combined with an integrated classical machine learning model \(\mathtt{EvoClass}\) . Within this context, we propose \(\mathtt{EvoQlass}\) to investigate the feasibility of using, inside \(\mathtt{EvoMaster}\) , a QNN classifier, instead of the existing classical machine learning model. Results indicate that \(\mathtt{EvoQlass}\) can achieve performance comparable to that of \(\mathtt{EvoClass}\) . We further explore the effects of various QNN configurations on performance and offer recommendations for optimal QNN settings for future QNN developers.

Self-adaptation in maritime autonomous vessels (AVs) enables them to adapt their behaviors to address unexpected situations while maintaining dependability requirements. During the design of such AVs, it is crucial to understand and identify the settings that should trigger adaptations, enabling validation of their implementation. To this end, we focus on the navigation software of AVs, which must adapt their behavior during operation through adaptations. AVs often rely on predefined waypoints to guide them along designated routes, ensuring safe navigation. We propose a multi-objective search-based approach, called WPgen, to generate minor modifications to the predefined set of waypoints, keeping them as close as possible to the original waypoints, while causing the AV to navigate inappropriately when navigating with the generated waypoints. WPgen uses NSGA-II as the multi-objective search algorithm with three seeding strategies for its initial population, resulting in three variations of WPgen. We evaluated these variations on three AVs (one overwater tanker and two underwater). We compared the three variations of WPgen with Random Search as the baseline and with each other. Experimental results showed that the effectiveness of these variations varied depending on the AV. Based on the results, we present the research and practical implications of WPgen.

The Cancer Registry of Norway (CRN) uses an automated cancer registration support system (CaReSS) to support core cancer registry activities, i.e., data capture, data curation, and producing data products and statistics for various stakeholders. GURI is a core component of CaReSS, which is responsible for validating incoming data with medical rules. Such medical rules are manually implemented by medical experts based on medical standards, regulations, and research. Since large language models (LLMs) have been trained on a large amount of public information, including these documents, they can be employed to generate tests for GURI. Thus, we propose an LLM-based test generation and differential testing approach (LLMeDiff) to test GURI. We experimented with four different LLMs, two medical rule engine implementations, and 58 real medical rules to investigate the hallucination, success, time efficiency, and robustness of the LLMs to generate tests, and these tests' ability to find potential issues in GURI. Our results showed that GPT-3.5 hallucinates the least, is the most successful, and is generally the most robust; however, it has the worst time efficiency. Our differential testing revealed 22 medical rules where implementation inconsistencies were discovered (e.g., regarding handling rule versions). Finally, we provide insights for practitioners and researchers based on the results.

Refurbishing laptops extends their lives while contributing to reducing electronic waste, which promotes building a sustainable future. To this end, the Danish Technological Institute (DTI) focuses on the research and development of several robotic applications empowered with software, including laptop refurbishing. Cleaning represents a major step in refurbishing and involves identifying and removing stickers from laptop surfaces. Software plays a crucial role in the cleaning process. For instance, the software integrates various object detection models to identify and remove stickers from laptops automatically. However, given the diversity in types of stickers (e.g., shapes, colors, locations), identification of the stickers is highly uncertain, thereby requiring explicit quantification of uncertainty associated with the identified stickers. Such uncertainty quantification can help reduce risks in removing stickers, which, for example, could otherwise result in software faults damaging laptop surfaces. For uncertainty quantification, we adopted the Monte Carlo Dropout method to evaluate six sticker detection models (SDMs) from DTI using three datasets: the original image dataset from DTI and two datasets generated with vision language models, i.e., DALL-E-3 and Stable Diffusion-3. In addition, we presented novel robustness metrics concerning detection accuracy and uncertainty to assess the robustness of the SDMs based on adversarial datasets generated from the three datasets using a dense adversary method. Our evaluation results show that different SDMs perform differently regarding different metrics. Based on the results, we provide SD$M$selection guidelines and lessons learned from various perspectives.

Self-adaptive robots (SARs) in complex, uncertain environments must proactively detect and address abnormal behaviors, including out-of-distribution (OOD) cases. To this end, digital twins offer a valuable solution for OOD detection. Thus, we present a digital twin-based approach for OOD detection (ODiSAR) in SARs. ODiSAR uses a Transformer-based digital twin to forecast SAR states and employs reconstruction error and Monte Carlo dropout for uncertainty quantification. By combining reconstruction error with predictive variance, the digital twin effectively detects OOD behaviors, even in previously unseen conditions. The digital twin also includes an explainability layer that links potential OOD to specific SAR states, offering insights for self-adaptation. We evaluated ODiSAR by creating digital twins of two industrial robots: one navigating an office environment, and another performing maritime ship navigation. In both cases, ODiSAR forecasts SAR behaviors (i.e., robot trajectories and vessel motion) and proactively detects OOD events. Our results showed that ODiSAR achieved high detection performance—up to 98% AUROC, 96% TNR@TPR95, and 95% F1-score—while providing interpretable insights to support self-adaptation.

Robots are increasingly becoming part of our daily lives, interacting with both the environment and humans to perform their tasks. The software of such robots often undergoes upgrades, for example, to add new functionalities, fix bugs, or delete obsolete functionalities. As a result, regression testing of robot software becomes necessary. However, determining the expected correct behavior of robots (i.e., a test oracle) is challenging due to the potentially unknown environments in which the robots must operate. To address this challenge, machine learning (ML)-based test oracles present a viable solution. This paper reports on the development of a test oracle to support regression testing of autonomous mobile robots built by PAL Robotics (Spain), using quantum machine learning (QML), which enables faster training and the construction of more precise test oracles. Specifically, we propose a hybrid framework, QuReBot, that combines both quantum reservoir computing (QRC) and a simple neural network, inspired by residual connection, to predict the expected behavior of a robot. Results show that QRC alone fails to converge in our case, yielding high prediction error. In contrast, QuReBot converges and achieves 15% reduction of prediction error compared to the classical neural network baseline. Finally, we further examine QuReBot under different configurations and offer practical guidance on optimal settings to support future robot software testing.

Abstract The convergence of Quantum Computing (QC), Quantum Software Engineering (QSE), and Artificial Intelligence (AI) presents transformative opportunities across various domains. However, existing methodologies inadequately address the ethical, security, and governance challenges arising from this technological shift. This paper highlights the urgent need for interdisciplinary collaboration to embed ethical principles into the development of Quantum AI (QAI) and QSE, ensuring transparency, inclusivity, and equitable global access. Without proactive governance, there is a risk of deepening digital inequalities and consolidating power among a select few. We call on the software engineering community to actively shape a future where responsible QSE and QAI are foundational for ethical, accountable, and socially beneficial technological progress.

Abstract As a new research area, quantum software testing lacks systematic testing benchmarks to assess testing techniques’ effectiveness. Recently, some open-source benchmarks and mutation analysis tools have emerged. However, there is insufficient evidence on how various quantum circuit characteristics (e.g., circuit depth, number of quantum gates), algorithms (e.g., Quantum Approximate Optimization Algorithm), and mutation characteristics (e.g., mutation operators) affect the detection of mutants in quantum circuits. Studying such relations is important to systematically design faulty benchmarks with varied attributes (e.g., the difficulty in detecting a seeded fault) to facilitate assessing the cost-effectiveness of quantum software testing techniques efficiently. To this end, we present a large-scale empirical evaluation with more than 700K faulty benchmarks (quantum circuits) generated by mutating 382 real-world quantum circuits. Based on the results, we provide valuable insights for researchers to define systematic quantum mutation analysis techniques. We also provide a tool to recommend mutants to users based on chosen characteristics (e.g., a quantum algorithm type) and the required difficulty of detecting mutants. Finally, we also provide faulty benchmarks that can already be used to assess the cost-effectiveness of quantum software testing techniques.

Abstract Cyber-physical systems (CPSs) blend digital and physical processes. CPS software is the key to realizing their functionalities. This software needs to evolve to deal with different aspects, such as the implementation of new functionalities or bug fixes. Because of this, design–operation methods, colloquially known as “DevOps,” are paramount to be adopted within these systems. During DevOps phases, automating test execution at design time is a key enabler of streamlined software development and software quality improvement. Likewise, monitoring whether a CPS is behaving as expected at operation is similarly important. In DevOps, test oracles play an important role in enabling automated testing, ensuring the reliability of software deployments, providing feedback to developers, etc. However, defining and generating test oracles in the context of DevOps practices in CPSs need to accommodate aspects specific to CPSs, such as their time-continuous behavior and inherent uncertainties. To this end, in this paper, we propose a domain-specific language (DSL) to ease the definition of test oracles and an automated solution for generating a microservice encapsulating the defined test oracles, which is compatible with a DevOps ecosystem for CPSs. We evaluated our DSL with two industrial case study systems and 9 open-source CPSs. Our evaluation results suggest that our DSL can model around 98% of the requirements of these systems through test oracles. Furthermore, it is possible to generate a microservice to be applicable at different test levels within less than 20 min, being fast enough to be adopted in practice.

As quantum computers advance, the complexity of the software they can execute increases as well. To ensure this software is efficient, maintainable, reusable, and cost-effective—key qualities of any industry-grade software—mature software engineering practices must be applied throughout its design, development, and operation. However, the significant differences between classical and quantum software make it challenging to directly apply classical software engineering methods to quantum systems. This challenge has led to the emergence of Quantum Software Engineering (QSE) as a distinct field within the broader software engineering landscape. In this work, a group of active researchers analyze in depth the current state of QSE research. From this analysis, the key areas of QSE are identified and explored in order to determine the most relevant open challenges that should be addressed in the next years. These challenges help identify necessary breakthroughs and future research directions for advancing QSE.

Quantum computing (QC) promises polynomial and exponential speedups in many domains, such as unstructured search and prime number factoring. However, quantum programs yield probabilistic outputs from exponentially growing distributions and are vulnerable to quantum-specific faults. Existing quantum software testing (QST) approaches treat quantum superpositions as classical distributions. This leads to two major limitations when applied to quantum programs: (1) an exponentially growing sample space distribution and (2) failing to detect quantum-specific faults such as phase flips. To overcome these limitations, we introduce a QST approach, which applies a reduction algorithm to a quantum program specification. The reduced specification alleviates the limitations (1) by enabling faster sampling through quantum parallelism and (2) by performing projective measurements in the mixed Hadamard basis. Our evaluation of 143 quantum programs across four categories demonstrates significant improvements in test runtimes and fault detection with our reduction approach. Average test runtimes improved from 169.9 s to 11.8 s, with notable enhancements in programs with large circuit depths (383.1 s to 33.4 s) and large program specifications (464.8 s to 7.7 s). Furthermore, our approach increases mutation scores from \(54.5\%\) to \(74.7\%\) , effectively detecting phase flip faults that non-reduced specifications miss. These results underline our approach's importance to improve QST efficiency and effectiveness.

Testing healthcare Internet of Things (IoT) applications at system and integration levels necessitates integrating numerous medical devices. Challenges of incorporating medical devices are: (i) their continuous evolution, making it infeasible to include all device variants and (ii) rigorous testing at scale requires multiple devices and their variants, which is time-intensive, costly, and impractical. Our collaborator, Oslo City’s health department, faced these challenges in developing automated test infrastructure, which our research aims to address. In this context, we propose a meta-learning-based approach ( MeDeT ) to generate digital twins (DTs) of medical devices and adapt DTs to evolving devices. We evaluate MeDeT in Oslo City’s context using five widely used medical devices integrated with a real-world healthcare IoT application. Our evaluation assesses MeDeT ’s ability to generate and adapt DTs across various devices and versions using different few-shot methods, the fidelity of these DTs, the scalability of operating 1,000 DTs concurrently, and the associated time costs. Results show that MeDeT can generate DTs with over 96% fidelity, adapt DTs to different devices and newer versions with reduced time cost (around one minute), and operate 1,000 DTs in a scalable manner while maintaining the fidelity level, thus serving in place of physical devices for testing.

Quantum annealers are specialized quantum computers for solving combinatorial optimization problems with special quantum computing characteristics, e.g., superposition and entanglement. Theoretically, quantum annealers can outperform classic computers. However, current quantum annealers are constrained by a limited number of qubits and cannot demonstrate quantum advantages. Nonetheless, research is needed to develop novel mechanisms to formulate combinatorial optimization problems for quantum annealing (QA). However, QA applications in software engineering remain unexplored. Thus, we propose BootQA , the very first effort at solving test case minimization (TCM) problems on classical software with QA. We provide a novel TCM formulation for QA and utilize bootstrap sampling to optimize the qubit usage. We also implemented our TCM formulation in three other optimization processes: simulated annealing (SA), QA without problem decomposition, and QA with an existing D-Wave problem decomposition strategy, and conducted an empirical evaluation with three real-world TCM datasets. Results show that BootQA outperforms QA without problem decomposition and QA with the existing decomposition strategy regarding effectiveness. Moreover, BootQA ’s effectiveness is similar to SA. Finally, BootQA has higher efficiency in terms of time when solving large TCM problems than the other three optimization processes.

Healthcare applications using the Internet of Things (IoT) architecture are connected with various medical devices designed to serve patients. Rigorous system testing of health care IoT applications requires integrating multiple medical devices to ensure the dependability of these applications. The integration of numerous physical medical devices with varying versions is a costly and time-consuming process. In this regard, our previous work introduced the concept of employing digital twins (DTs) as substitutes for physical devices for testing purposes. Specifically, we presented a model-based approach to generate DTs of medicine dispensers. The evaluation of our approach with a Karie medicine dispenser demonstrated 92% fidelity of Karie DTs. From our experiences, we observed that the real operating environment of medical devices involves several non-deterministic factors, essential for DTs to reflect devices' behavior precisely. Therefore, we plan to devise a methodology to model and simulate the environment of medical devices DTs, taking into account environmental uncertainties. We intend to empirically evaluate our methodology in the real-world context to analyze the simulation of behavioral models of the environment and uncertain events generated for DTs.

The most promising applications of quantum computing are centered around solving search and optimization tasks, particularly in fields such as physics simulations, quantum chemistry, and finance. However, the current quantum software testing methods face practical limitations when applied in industrial contexts: (i) they do not apply to quantum programs most relevant to the industry, (ii) they require a full program specification, which is usually not available for these programs, and (iii) they are incompatible with error mitigation methods currently adopted by main industry actors like IBM. To address these challenges, we present QOPS, a novel quantum software testing approach. QOPS introduces a new definition of test cases based on Pauli strings to improve compatibility with different quantum programs. QOPS also introduces a new test oracle that can be directly integrated with industrial APIs such as IBM's Estimator API and can utilize error mitigation methods for testing on real noisy quantum computers. We also leverage the commuting property of Pauli strings to relax the requirement of having complete program specifications, making QOPS practical for testing complex quantum programs in industrial settings. We empirically evaluate QOPS on 194,982 real quantum programs, demonstrating effective performance in test assessment compared to the state-of-the-art with a perfect F1-score, precision, and recall. Furthermore, we validate the industrial applicability of QOPS by assessing its performance on IBM's three real quantum computers, incorporating both industrial and open-source error mitigation methods.

Summary Healthcare applications with the Internet of Things (IoT) are often safety‐critical, thus, require extensive testing. Such applications are often connected to smart medical devices from various vendors. System‐level testing of such applications requires test infrastructures physically integrating medical devices, which is time and monetary‐wise expensive. Moreover, applications continuously evolve, for example, introducing new devices and users and updating software. Nevertheless, a test infrastructure enabling testing with a few devices is insufficient for testing healthcare IoT systems, hence compromising their dependability. In this paper, we propose a model‐based approach for the creation and operation of digital twins (DTs) of medicine dispensers as a replacement for physical devices to support the automated testing of IoT applications at scale. We evaluate our approach with an industrial IoT system with medicine dispensers in the context of Oslo City and its industrial partners, providing healthcare services to its residents. We study the fidelity of DTs in terms of their functional similarities with their physical counterparts: medicine dispensers. Results show that the DTs behave more than 92% similar to the physical medicine dispensers, providing a faithful replacement for the dispenser.

Ensuring that software performance does not degrade after a code change is paramount. A solution is to regularly execute software microbenchmarks, a performance testing technique similar to (functional) unit tests, which, however, often becomes infeasible due to extensive runtimes. To address that challenge, research has investigated regression testing techniques, such as test case prioritization (TCP), which reorder the execution within a microbenchmark suite to detect larger performance changes sooner. Such techniques are either designed for unit tests and perform sub-par on microbenchmarks or require complex performance models, drastically reducing their potential application. In this paper, we empirically evaluate single- and multi-objective search-based microbenchmark prioritization techniques to understand whether they are more effective and efficient than greedy, coverage-based techniques. For this, we devise three search objectives, i.e., coverage to maximize, coverage overlap to minimize, and historical performance change detection to maximize. We find that search algorithms (SAs) are only competitive with but do not outperform the best greedy, coverage-based baselines. However, a simple greedy technique utilizing solely the performance change history (without coverage information) is equally or more effective than the best coverage-based techniques while being considerably more efficient, with a runtime overhead of less than 1%. These results show that simple, non-coverage-based techniques are a better fit for microbenchmarks than complex coverage-based techniques.

Testing autonomous vehicles (AVs) under various environmental scenarios that lead the vehicles to unsafe situations is challenging. Given the infinite possible environmental scenarios, it is essential to find critical scenarios efficiently. To this end, we propose a novel testing method, namedEpiTESTER, by taking inspiration from epigenetics, which enables species to adapt to sudden environmental changes. In particular,EpiTESTERadopts gene silencing as its epigenetic mechanism, which regulates gene expression to prevent the expression of a certain gene, and the probability of gene expression is dynamically computed as the environment changes. Given different data modalities (e.g., images, lidar point clouds) in the context of AV,EpiTESTERbenefits from a multi-model fusion transformer to extract high-level feature representations from environmental factors. Next, it calculates probabilities based on these features with the attention mechanism. To assess the cost-effectiveness ofEpiTESTER, we compare it with a probabilistic search algorithm (Simulated Annealing, SA), a classical genetic algorithm (GA) (i.e., without any epigenetic mechanism implemented), andEpiTESTERwith equal probability for each gene. We evaluateEpiTESTERwith six initial environments from CARLA, an open-source simulator for autonomous driving research, and two end-to-end AV controllers, Interfuser and TCP. Our results show thatEpiTESTERachieved a promising performance in identifying critical scenarios compared to the baselines, showing that applying epigenetic mechanisms is a good option for solving practical problems.

Quantum Computing (QC) promises computational speedup over classic computing. However, noise exists in near-term quantum computers. Quantum software testing (for gaining confidence in quantum software's correctness) is inevitably impacted by noise, i.e., it is impossible to know if a test case failed due to noise or real faults. Existing testing techniques test quantum programs without considering noise, i.e., by executing tests on ideal quantum computer simulators. Consequently, they are not directly applicable to test quantum software on real quantum computers or noisy simulators. Thus, we propose a noise-aware approach (named$\mathit{QOIN}$) to alleviate the noise effect on test results of quantum programs.$\mathit{QOIN}$employs machine learning techniques (e.g., transfer learning) to learn the noise effect of a quantum computer and filter it from a program's outputs. Such filtered outputs are then used as the input to perform test case assessments (determining the passing or failing of a test case execution against a test oracle). We evaluated$\mathit{QOIN}$on IBM's 23 noise models, Google's two available noise models, and Rigetti's Quantum Virtual Machine, with six real-world and 800 artificial programs. We also generated faulty versions of these programs to check if a failing test case execution can be determined under noise. Results show that$\mathit{QOIN}$can reduce the noise effect by more than$80\%$on most noise models. We used an existing test oracle to evaluate$\mathit{QOIN}$'s effectiveness in quantum software testing. The results showed that$\mathit{QOIN}$attained scores of$99\%$,$75\%$, and$86\%$for precision, recall, and F1-score, respectively, for the test oracle across six real-world programs. For artificial programs,$\mathit{QOIN}$achieved scores of$93\%$,$79\%$, and$86\%$for precision, recall, and F1-score respectively. This highlights$\mathit{QOIN}$'s effectiveness in learning noise patterns for noise-aware quantum software testing.

Test case optimization (TCO) reduces the software testing cost while preserving its effectiveness. However, to solve TCO problems for large-scale and complex software systems, substantial computational resources are required. Quantum approximate optimization algorithms (QAOAs) are promising combinatorial optimization algorithms that rely on quantum computational resources, with the potential to offer increased efficiency compared to classical approaches. Several proof-of-concept applications of QAOAs for solving combinatorial problems, such as portfolio optimization, energy optimization in power systems, and job scheduling, have been proposed. Given the lack of investigation into QAOA's application for TCO problems, and motivated by the computational challenges of TCO problems and the potential of QAOAs, we present IGDec-QAOA to formulate a TCO problem as a QAOA problem and solve it on both ideal and noisy quantum computer simulators, as well as on a real quantum computer. To solve bigger TCO problems that require many qubits, which are unavailable these days, we integrate a problem decomposition strategy with the QAOA. We performed an empirical evaluation with five TCO problems and four publicly available industrial datasets from ABB, Google, and Orona to compare various configurations of IGDec-QAOA, assess its decomposition strategy of handling large datasets, and compare its performance with classical algorithms (i.e., Genetic Algorithm (GA) and Random Search). Based on the evaluation results achieved on an ideal simulator, we recommend the best configuration of our approach for TCO problems. Also, we demonstrate that our approach can reach the same effectiveness as GA and outperform GA in two out of five test case optimization problems we conducted. In addition, we observe that, on the noisy simulator, IGDec-QAOA achieved similar performance to that from the ideal simulator. Finally, we also demonstrate the feasibility of IGDec-QAOA on a real quantum computer in the presence of noise.

Cyber-physical systems (CPSs), e.g., elevators and autonomous driving systems, are progressively permeating our everyday lives. To ensure their safety, various analyses need to be conducted, such as anomaly detection and time-to-event analysis (the focus of this paper). Recently, it has been widely accepted that digital Twins (DTs) can be an efficient method to aid in developing, maintaining, and safe and secure operation of CPSs. However, CPSs frequently evolve, e.g., with new or updated functionalities, which demand their corresponding DTs be co-evolved, i.e., in synchronization with the CPSs. To that end, we propose a novel method, named PPT, utilizing an uncertainty-aware transfer learning for DT evolution. Specifically, we first pretrain PPT with a pretraining dataset to acquire generic knowledge about the CPSs, followed by adapting it to a specific CPS with the help of prompt tuning. Results highlight that PPT is effective in time-to-event analysis in both elevator and autonomous driving case studies, on average, outperforming a baseline method by 7.31 and 12.58 in terms of Huber loss, respectively. The experiment results also affirm the effectiveness of transfer learning, prompt tuning, and uncertainty quantification in terms of reducing Huber loss by at least 21.32, 3.14, and 4.08, respectively, in both case studies.

While most large web-based software systems (e.g., Amazon, Google) release a new software version every almost a minute, in the context of Cyber-Physical Systems (CPSs), this is still far. However, the software of CPSs needs to evolve while these are in operation to fix bugs, add new functionalities, carry out refactoring activities and deal with unforeseen situations that were discovered while the CPS was operating. In the last three years, the Adeptness project has been developing in a solution to help speedup the software release of CPSs that are in operation while guaranteeing their reliability. In this paper, we summarize the objectives, results and lessons learned from this H2020 project.

The Cancer Registration Support System (CaReSS), built by the Cancer Registry of Norway (CRN), is a complex real-world socio-technical software system that undergoes continuous evolution in its implementation. Consequently, continuous testing of CaReSS with automated testing tools is needed such that its dependability is always ensured. Towards automated testing of a key software subsystem of CaReSS, i.e., GURI, we present a real-world application of an extension to the open-source tool EvoMaster, which automatically generates test cases with evolutionary algorithms. We named the extension EvoClass, which enhances EvoMaster with a machine learning classifier to reduce the overall testing cost. This is imperative since testing with EvoMaster involves sending many requests to GURI deployed in different environments, including the production environment, whose performance and functionality could potentially be affected by many requests. The machine learning classifier of EvoClass can predict whether a request generated by EvoMaster will be executed successfully or not; if not, the classifier filters out such requests, consequently reducing the number of requests to be executed on GURI. We evaluated EvoClass on ten GURI versions over four years in three environments: development, testing, and production. Results showed that EvoClass can significantly reduce the testing cost of evolving GURI without reducing testing effectiveness (measured as rule coverage) across all three environments, as compared to the default EvoMaster. Overall, EvoClass achieved ≈31% of overall cost reduction. Finally, we report our experiences and lessons learned that are equally valuable for researchers and practitioners.

With the increased developments in quantum computing, the availability of systematic and automatic testing approaches for quantum programs is becoming increasingly essential. To this end, we present the quantum software testing tool QuCAT for combinatorial testing of quantum programs. QuCAT provides two functionalities of use. With the first functionality, the tool generates a test suite of a given strength (e.g., pairwise). With the second functionality, it generates test suites with increasing strength until a failure is triggered or a maximum strength is reached. QuCAT uses two test oracles to check the correctness of test outputs. We assess the cost and effectiveness of QuCAT with 3 faulty versions of 5 quantum programs. Results show that combinatorial test suites with a low strength can find faults with limited cost, while a higher strength performs better to trigger some difficult faults with relatively higher cost.

Uncertain, unpredictable, real-time, and lifelong evolution causes operational failures in intelligent software systems, leading to significant damages, safety and security hazards, and tragedies. To fully unleash such systems' potential and facilitate their wider adoption, ensuring the trustworthiness of their decision-making under uncertainty is the prime challenge. To overcome this challenge, an intelligent software system and its operating environment should be continuously monitored, tested, and refined during its lifetime operation. Existing technologies, such as digital twins, can enable continuous synchronisation with such systems to reflect their most up-to-date states. Such representations are often in the form of prior-knowledge-based and machine-learning models, together called ‘model universe’. In this paper, we present our vision of combining techniques from software engineering, evolutionary computation, and machine learning to support the model universe evolution.

With the rapid development of autonomous driving systems (ADSs), testing ADSs under various environmental conditions has become a key method to ensure the successful deployment of ADS in the real world. However, it is impossible to test all the scenarios due to the inherent complexity and uncertainty of ADSs and the driving tasks. Further, testing of ADSs is expensive regarding time and computational resources. Therefore, a large-scale driving scenario dataset consisting of various driving conditions is needed. To this end, we present an open driving scenario dataset DeepScenario, containing over 30K executable driving scenarios, which are collected by 2880 test executions of three driving scenario generation strategies. Each scenario in the dataset is labeled with six attributes characterizing test results. We further show the attribute statistics and distribution of driving scenarios. For example, there are 1050 collision scenarios, in 917 scenarios there were collisions with other vehicles, 105 and 28 with pedestrians and static obstacles, respectively. Target users include ADS developers who need to validate their systems under various environmental conditions.

The Cancer Registry of Norway (CRN) collects, curates, and manages data related to cancer patients in Norway, supported by an interactive, human-in-the-loop, socio-technical decision support software system. Automated software testing of this software system is inevitable; however, currently, it is limited in CRN’s practice. To this end, we present an industrial case study to evaluate an AI-based system-level testing tool, i.e., EvoMaster, in terms of its effectiveness in testing CRN’s software system. In particular, we focus on GURI, CRN’s medical rule engine, which is a key component at the CRN. We test GURI with EvoMaster’s black-box and white-box tools and study their test effectiveness regarding code coverage, errors found, and domain-specific rule coverage. The results show that all EvoMaster tools achieve a similar code coverage; i.e., around 19% line, 13% branch, and 20% method; and find a similar number of errors; i.e., 1 in GURI’s code. Concerning domain-specific coverage, EvoMaster’s black-box tool is the most effective in generating tests that lead to applied rules; i.e., 100% of the aggregation rules and between 12.86% and 25.81% of the validation rules; and to diverse rule execution results; i.e., 86.84% to 89.95% of the aggregation rules and 0.93% to 1.72% of the validation rules pass, and 1.70% to 3.12% of the aggregation rules and 1.58% to 3.74% of the validation rules fail. We further observe that the results are consistent across 10 versions of the rules. Based on these results, we recommend using EvoMaster’s black-box tool to test GURI since it provides good results and advances the current state of practice at the CRN. Nonetheless, EvoMaster needs to be extended to employ domain-specific optimization objectives to improve test effectiveness further. Finally, we conclude with lessons learned and potential research directions, which we believe are applicable in a general context.

The Cancer Registry of Norway (CRN) collects information on cancer patients by receiving cancer messages from different medical entities (e.g., medical labs, hospitals) in Norway. Such messages are validated by an automated cancer registry system: GURI. Its correct operation is crucial since it lays the foundation for cancer research and provides critical cancer-related statistics to its stakeholders. Constructing a cyber-cyber digital twin (CCDT) for GURI can facilitate various experiments and advanced analyses of the operational state of GURI without requiring intensive interactions with the real system. However, GURI constantly evolves due to novel medical diagnostics and treatment, technological advances, etc. Accordingly, CCDT should evolve as well to synchronize with GURI. A key challenge of achieving such synchronization is that evolving CCDT needs abundant data labelled by the new GURI. To tackle this challenge, we propose EvoCLINICAL, which considers the CCDT developed for the previous version of GURI as the pretrained model and fine-tunes it with the dataset labelled by querying a new GURI version. EvoCLINICAL employs a genetic algorithm to select an optimal subset of cancer messages from a candidate dataset and query GURI with it. We evaluate EvoCLINICAL on three evolution processes. The precision, recall, and F1 score are all greater than 91%, demonstrating the effectiveness of EvoCLINICAL. Furthermore, we replace the active learning part of EvoCLINICAL with random selection to study the contribution of transfer learning to the overall performance of EvoCLINICAL. Results show that employing active learning in EvoCLINICAL increases its performances consistently.

Healthcare Internet of Things (IoT) applications require rigorous testing to ensure their dependability. Such applications are typically integrated with various third-party healthcare applications and medical devices through REST APIs. This integrated network of healthcare IoT applications leads to REST APIs with complicated and interdependent structures, thus creating a major challenge for automated system-level testing. We report an industrial evaluation of a state-of-the-art REST APIs testing approach (RESTest) on a real-world healthcare IoT application. We analyze the effectiveness of RESTest’s testing strategies regarding REST APIs failures, faults in the application, and REST API coverage, by experimenting with six REST APIs of 41 API endpoints of the healthcare IoT application. Results show that several failures are discovered in different REST APIs with ≈56% coverage using RESTest. Moreover, nine potential faults are identified. Using the evidence collected from the experiments, we provide our experiences and lessons learned.

Cyber-physical systems (CPSs), like train control and management systems (TCMS), are becoming ubiquitous in critical infrastructures. As safety-critical systems, ensuring their dependability during operation is crucial. Digital twins (DTs) have been increasingly studied for this purpose owing to their capability of runtime monitoring and warning, prediction and detection of anomalies, etc. However, constructing a DT for anomaly detection in TCMS necessitates sufficient training data and extracting both chronological and context features with high quality. Hence, in this paper, we propose a novel method named KDDT for TCMS anomaly detection. KDDT harnesses a language model (LM) and a long short-term memory (LSTM) network to extract contexts and chronological features, respectively. To enrich data volume, KDDT benefits from out-of-domain data with knowledge distillation (KD). We evaluated KDDT with two datasets from our industry partner Alstom and obtained the F1 scores of 0.931 and 0.915, respectively, demonstrating the effectiveness of KDDT. We also explored individual contributions of the DT model, LM, and KD to the overall performance of KDDT, via a comprehensive empirical study, and observed average F1 score improvements of 12.4%, 3%, and 6.05%, respectively.

Industrial elevator systems are commonly used software systems in our daily lives, which operate in uncertain environments such as unpredictable passenger traffic, uncertain passenger attributes and behaviors, and hardware delays. Understanding and assessing the robustness of such systems under various uncertainties enable system designers to reason about uncertainties, especially those leading to low system robustness, and consequently improve their designs and implementations in terms of handling uncertainties. To this end, we present a comprehensive empirical study conducted with industrial elevator systems provided by our industrial partner Orona, which focuses on assessing the robustness of a dispatcher—that is, a software component responsible for elevators’ optimal scheduling. In total, we studied 90 industrial dispatchers in our empirical study. Based on the experience gained from the study, we derived an uncertainty-aware robustness assessment method (named UncerRobua ) comprising a set of guidelines on how to conduct the robustness assessment and a newly proposed ranking algorithm, for supporting the robustness assessment of industrial elevator systems against uncertainties.

Anomaly detection is critical to ensure the security of cyber-physical systems (CPS) . However, due to the increasing complexity of attacks and CPS themselves, anomaly detection in CPS is becoming more and more challenging. In our previous work, we proposed a digital twin-based anomaly detection method, called ATTAIN, which takes advantage of both historical and real-time data of CPS. However, such data vary significantly in terms of difficulty. Therefore, similar to human learning processes, deep learning models (e.g., ATTAIN) can benefit from an easy-to-difficult curriculum. To this end, in this paper, we present a novel approach, named digitaL twin-based Anomaly deTecTion wIth Curriculum lEarning ( LATTICE ) , which extends ATTAIN by introducing curriculum learning to optimize its learning paradigm. LATTICE attributes each sample with a difficulty score, before being fed into a training scheduler. The training scheduler samples batches of training data based on these difficulty scores such that learning from easy to difficult data can be performed. To evaluate LATTICE, we use five publicly available datasets collected from five real-world CPS testbeds. We compare LATTICE with ATTAIN and two other state-of-the-art anomaly detectors. Evaluation results show that LATTICE outperforms the three baselines and ATTAIN by 0.906%-2.367% in terms of the F1 score. LATTICE also, on average, reduces the training time of ATTAIN by 4.2% on the five datasets and is on par with the baselines in terms of detection delay time.

Autonomous vehicles must operate safely in their dynamic and continuously-changing environment. However, the operating environment of an autonomous vehicle is complicated and full of various types of uncertainties. Additionally, the operating environment has many configurations, including static and dynamic obstacles with which an autonomous vehicle must avoid collisions. Though various approaches targeting environment configuration for autonomous vehicles have shown promising results, their effectiveness in dealing with a continuous-changing environment is limited. Thus, it is essential to learn realistic environment configurations of continuously-changing environment, under which an autonomous vehicle should be tested regarding its ability to avoid collisions. Featured with agents dynamically interacting with the environment, Reinforcement Learning (RL) has shown great potential in dealing with complicated problems requiring adapting to the environment. To this end, we present an RL-based environment configuration learning approach, i.e.,DeepCollision, which intelligently learns environment configurations that lead an autonomous vehicle to crash. DeepCollision employs Deep Q-Learning as the RL solution, and selectscollision probabilityas the safety measure, to construct the reward function. We trained four DeepCollision models and conducted an experiment to compare them with two baselines, i.e., random and greedy. Results show that DeepCollision demonstrated significantly better effectiveness in generating collisions compared with the baselines. We also provide recommendations on configuring DeepCollision with the most suitable time interval based on different road structures.

Mutation testing is often used for designing new tests, and involves changing a program in minor ways, which results in mutated versions of the program, i.e., mutants. An effective test suite should find faults (or kill mutants) with a minimum number of test cases, to save resources required for executing test cases. In this paper, in the context of mutation testing for quantum programs, we present a multi-objective and search-based approach (MutTG) to generate the minimum number of test cases killing as many mutants as possible. MutTG tries to estimate the likelihood that a mutant is equivalent, and uses this as a discount factor in the fitness definition to avoid keeping on trying to kill mutants that cannot be killed. We employed NSGA-II as the multi-objective search algorithm. Then, we compared MutTG with another version of the approach that does not use the discount factor in its fitness definition, and with random search (RS), over a set of open-source quantum programs and their mutants of varying complexity. Results show that the discount factor does indeed help in guiding the test generation, as the approach with the discount factor performs better than the one without it.

Industrial elevator systems are complex Cyber-Physical Systems operating in uncertain environments and experiencing uncertain passenger behaviors, hardware delays, and software errors. Identifying, understanding, and classifying such uncertainties are essential to enable system designers to reason about uncertainties and subsequently develop solutions for empowering elevator systems to deal with uncertainties systematically. To this end, we present a method, called RuCynefin, based on the Cynefin framework to classify uncertainties in industrial elevator systems from our industrial partner (Orona, Spain), results of which can then be used for assessing their robustness. RuCynefin is equipped with a novel classification algorithm to identify the Cynefin contexts for a variety of uncertainties in industrial elevator systems, and a novel metric for measuring the robustness using the uncertainty classification. We evaluated RuCynefin with an industrial case study of 90 dispatchers from Orona to assess their robustness against uncertainties. Results show that RuCynefin could effectively identify several situations for which certain dispatchers were not robust. Specifically, 93% of such versions showed some degree of low robustness against uncertainties. We also provide insights on the potential practical usages of RuCynefin, which are useful for practitioners in this field.

Digital twins are increasingly developed to support the development, operation, and maintenance of cyber-physical systems such as industrial elevators. However, industrial elevators continuously evolve due to changes in physical installations, introducing new software features, updating existing ones, and making changes due to regulations (e.g., enforcing restricted elevator capacity due to COVID-19), etc. Thus, digital twin functionalities (often built on neural network-based models) need to evolve themselves constantly to be synchronized with the industrial elevators. Such an evolution is preferred to be automated, as manual evolution is time-consuming and error-prone. Moreover, collecting sufficient data to re-train neural network models of digital twins could be expensive or even infeasible. To this end, we propose unceRtaInty-aware tranSfer lEarning enriched Digital Twins LATTICE, a transfer learning based approach capable of transferring knowledge about the waiting time prediction capability of a digital twin of an industrial elevator across different scenarios. LATTICE also leverages uncertainty quantification to further improve its effectiveness. To evaluate LATTICE, we conducted experiments with 10 versions of an elevator dispatching software from Orona, Spain, which are deployed in a Software in the Loop (SiL) environment. Experiment results show that LATTICE, on average, improves the Mean Squared Error by 13.131% and the utilization of uncertainty quantification further improves it by 2.71%.

Orona is a world-renowned elevators developer. During elevators' lives, their software continues to evolve, e.g., due to hardware obsolescence, requirements changes, vulnerabilities, and bug corrections. Such continuous evolution demands the continuous testing of industrial elevators with the minimum manual effort possible. To this end, we present a tool, whose core component is a domain-specific language (DSL) with which a user can specify test oracles at a higher level of abstraction and independent of a testing level. The DSL also supports specifying uncertainty-aware test oracles to test elevators under various uncertainties inherent in them. Finally, the DSL is also equipped with test oracle generation that generates test oracle code automatically at the different DevOps testing levels (i.e., Software and Hardware-in-the-Loop test levels, and in operation) to enable reuse of test oracles across these levels. We evaluated this DSL with an industrial elevators case study at Orona's site to specify and generate test oracles. The evaluation showed that the high expressiveness of the DSL permits the high-level definition of test oracles in our industrial context. Based on the industrial application, we discuss our experiences and lessons learned.

The use of Deep learning in Cyber-Physical Systems (CPSs) is gaining popularity due to its ability to bring intelligence to CPS behaviors. However, both CPSs and deep learning have inherent uncertainty. Such uncertainty, if not handled adequately, can lead to unsafe CPS behavior. The first step toward addressing such uncertainty in deep learning is to quantify uncertainty. Hence, we propose a novel method called NIRVANA (uNcertaInty pRediction ValidAtor iN Ai) for prediction validation based on uncertainty metrics. To this end, we first employ prediction-time Dropout-based Neural Networks to quantify uncertainty in deep learning models applied to CPS data. Second, such quantified uncertainty is taken as the input to predict wrong labels using a support vector machine, with the aim of building a highly discriminating prediction validator model with uncertainty values. In addition, we investigated the relationship between uncertainty quantification and prediction performance and conducted experiments to obtain optimal dropout ratios. We conducted all the experiments with four real-world CPS datasets. Results show that uncertainty quantification is negatively correlated to prediction performance of a deep learning model of CPS data. Also, our dropout ratio adjustment approach is effective in reducing uncertainty of correct predictions while increasing uncertainty of wrong predictions.

Due to the complexity of designing vehicle products and the inherent uncertainties in their operating environments, ensuring the safety of their Advanced Driver Assistance Systems (ADASs) becomes crucial. Especially, very minor changes to a vehicle design, for instance due to production errors or component degradation, might lead to failures of ADASs and, therefore, catastrophic consequences such as collision occurrences. Motivated by this, we propose a multi-objective search-based approach (employing NSGA-II) to find minimum changes to the configuration of a set of configurable parameters of a vehicle design, such that the collision probability is maximized, consequently leading to a reversal change in its safety. We conducted experiments, in a vehicle driving simulator, to evaluate the effectiveness of our approach. Results show that our approach with NSGA-II significantly outperforms the random search. Moreover, based on the detailed analyses of the results, we identify some parameters for which minor changes to their values lead the vehicle into collisions, and demonstrated the importance of studying the configuration of multiple parameters in a single search and the impact of their interactions on causing collisions.

Quantum programs implement quantum algorithms solving complex computational problems. Testing such programs is challenging due to the inherent characteristics of Quantum Computing (QC), such as the probabilistic nature and computations in superposition. However, automated and systematic testing is needed to ensure the correct behavior of quantum programs. To this end, we present an approach called Quito (QUantum InpuT Output coverage) consisting of three coverage criteria defined on the inputs and outputs of a quantum program, together with their test generation strategies. Moreover, we define two types of test oracles, together with a procedure to determine the passing and failing of test suites with statistical analyses. To evaluate the cost-effectiveness of the three coverage criteria, we conducted experiments with five quantum programs. We used mutation analysis to determine the coverage criteria' effectiveness and cost in terms of the number of test cases. Based on the results of mutation analysis, we also identified equivalent mutants for quantum programs.

Cyber-Physical Systems (CPS) are susceptible to various anomalies during their operations. Thus, it is important to detect such anomalies. Detecting such anomalies is challenging since it is uncertain when and where anomalies can happen. To this end, we present a novel approach called Anomaly deTection with digiTAl twIN (ATTAIN), which continuously and automatically builds a digital twin with live data obtained from a CPS for anomaly detection. ATTAIN builds a Timed Automaton Machine (TAM) as the digital representation of the CPS, and implements a Generative Adversarial Network (GAN) to detect anomalies. GAN uses a GCN-LSTM-based module as a generator, which can capture temporal and spatial characteristics of the input data and learn to produce realistic unlabeled adversarial samples. TAM labels these adversarial samples, which are then fed into a discriminator along with real labeled samples. After training, the discriminator is capable of distinguishing anomalous data from normal data with a high F1 score. To evaluate our approach, we used three publicly available datasets collected from three CPS testbeds. Evaluation results show that ATTAIN improved the performance of two state-of-art anomaly detection methods by 2.413%, 8.487%, and 5.438% on average on the three datasets, respectively. Moreover, ATTAIN achieved on average 8.39% increase in the anomaly detection capability with digital twins as compared with an approach of not using digital twins.

Given that quantum software testing is a new area of research, there is a lack of benchmark programs and bugs repositories to assess the effectiveness of testing techniques. To this end, quantum mutation analysis focuses on systematically generating faulty versions of Quantum Programs (QPs), called mutants, using mutation operators. Such mutants can be used as benchmarks to assess the quality of test cases in a test suite. Thus, we present Muskit - a quantum mutation analysis tool for QPs coded in IBM's Qiskit language. Muskit defines mutation operators on gates of QPs and selection criteria to reduce the number of mutants to generate. Moreover, it allows for the execution of test cases on mutants and generation of results for test analyses. Muskit is provided as command line interface, GUI, and web application. We validated Muskit by using it to generate and execute mutants for four QPs. Muskit code: https://github.com/Simula-COMPLEX/muskitWeb app: https://qiskitmutantcreatorsrl.pythonanywhere.com/YouTube Video: EbPHJOK_AEA Artifact Available: https://doi.org/10.5281/zenodo.5288917

Automation in quantum software testing is essential to support systematic and cost-effective testing. Towards this direction, we present a quantum software testing tool called Quito that can automatically generate test suites covering three coverage criteria defined on inputs and outputs of a quantum program coded in Qiskit, i.e., input coverage, output coverage, and input-output coverage. Quito also implements two types of test oracles based on program specifications, i.e., checking whether a quantum program produced a wrong output or checking a probabilistic test oracle with statistical test. We describe the architecture and methodology of the tool. We also validated the tool with one quantum program and one faulty version of it. Results indicate that Quito can generate test suites and perform test assessments that detect faults, and produce test results with a good time performance.Quito’s code: https://github.com/Simula-COMPLEX/quitoQuito’s video: https://youtu.be/kuI9QaCo8A8Artifact Available: https://doi.org/10.5281/zenodo.5288665

With the aim to reduce car accidents, autonomous driving attracted a lot of attentions these years. However, recently reported crashes indicate that this goal is far from being achieved. Hence, cost-effective testing of autonomous driving systems (ADSs) has become a prominent research topic. The classical model-based testing (MBT), i.e., generating test cases from test models followed by executing the test cases, is ineffective for testing ADSs, mainly because of the constant exposure to ever-changing operating environments, and uncertain internal behaviors due to employed AI techniques. Thus, MBT must be adaptive to guide test case generation based on test execution results in a step-wise manner. To this end, we propose a natural language and model-based approach, named LiveTCM, to automatically execute and generate test case specifications (TCSs) by interacting with an ADS under test and its environment. LiveTCM is evaluated with an open-source ADS and two test generation strategies: Deep Q-Network (DQN)-based and Random. Results show that LiveTCM with DQN can generate TCSs with 56 steps on average in 60 seconds, leading to 6.4 test oracle violations and covering 14 APIs per TCS on average.

The capability of Quantum Computing (QC) in solving complex problems has been increasingly recognized. However, similar to classical computing, to fully exploit QC's potential, it is important to ensure the correctness of quantum programs. Doing so via software testing is, however, very challenging because of QC's inherent properties: superposition and entanglement. Towards the direction of ensuring the correctness of quantum programs, we propose an approach called QuCAT (QUantum CombinAtorial Testing) for systematic and automated testing of quantum programs by benefiting from combinatorial testing, which has been proven to be cost-effective in testing classical programs. QuCAT supports two combinatorial test suite generation scenarios, i.e., generating combinatorial test suites of a given strength, and incrementally generating and executing combinatorial test suites of increasing strength until a fault is found. The approach employs two types of test oracles to assess test results. We performed an empirical study with 18 faulty versions of quantum programs to evaluate QuCAT with strengths of two, three, and four in the two test generation scenarios. We compare the cost-effectiveness of combinatorial testing of various strengths and random testing (taken as baseline approach). Results show that combinatorial testing always performs better than random testing with the same cost and finds faults more quickly (in terms of required number of test cases). In addition, in most cases, combinatorial testing with a higher strength outperforms the lower strength in terms of effectiveness.

Modern systems, such as cyber-physical systems, often consist of multiple products within/across product lines communicating with each other through information networks. Consequently, their runtime behaviors are influenced by product configurations and networks. Such systems play a vital role in our daily life; thus, ensuring their correctness by thorough testing becomes essential. However, testing these systems is particularly challenging due to a large number of possible configurations and limited available resources. Therefore, it is important and practically useful to test these systems with specific configurations under which products will most likely fail to communicate with each other. Motivated by this, we present a search-based configuration recommendation ( SBCR ) approach to recommend faulty configurations for the system under test (SUT) based on cross-product line (CPL) rules. CPL rules are soft constraints, constraining product configurations while indicating the most probable system states with a certain degree of confidence. In SBCR , we defined four search objectives based on CPL rules and combined them with six commonly applied search algorithms. To evaluate SBCR (i.e., SBCR NSGA-II , SBCR IBEA , SBCR MoCell , SBCR SPEA2 , SBCR PAES , and SBCR SMPSO ), we performed two case studies (Cisco and Jitsi) and conducted difference analyses. Results show that for both of the case studies, SBCR significantly outperformed random search-based configuration recommendation ( RBCR ) for 86% of the total comparisons based on six quality indicators, and 100% of the total comparisons based on the percentage of faulty configurations (PFC). Among the six variants of SBCR, SBCR SPEA2 outperformed the others in 85% of the total comparisons based on six quality indicators and 100% of the total comparisons based on PFC.

Requirements review is an effective technique to ensure the quality of requirements in practice, especially in safety-critical domains (e.g., avionics systems, automotive systems). In such contexts, a typical requirements review process often prioritizes requirements, due to limited time and monetary budget, by, for instance, prioritizing requirements with higher implementation cost earlier in the review process. However, such a requirement implementation cost is typically estimated by stakeholders who often lack knowledge about (future) requirements implementation scenarios, which leads to uncertainty in cost overrun. In this article, we explicitly consider such uncertainty (quantified as cost overrun probability) when prioritizing requirements based on the assumption that a requirement with higher importance, a higher number of dependencies to other requirements, and higher implementation cost will be reviewed with the higher priority. Motivated by this, we formulate four objectives for uncertainty-wise requirements prioritization: maximizing the importance of requirements, requirements dependencies, the implementation cost of requirements, and cost overrun probability. These four objectives are integrated as part of our search-based uncertainty-wise requirements prioritization approach with tool support, named as URP. We evaluated six Multi-Objective Search Algorithms (MOSAs) (i.e., NSGA-II, NSGA-III, MOCell, SPEA2, IBEA, and PAES ) together with Random Search ( RS ) using three real-world datasets (i.e., the RALIC, Word, and ReleasePlanner datasets) and 19 synthetic optimization problems. Results show that all the selected MOSAs can solve the requirements prioritization problem with significantly better performance than RS . Among them, IBEA was over 40% better than RS in terms of permutation effectiveness for the first 10% of prioritized requirements in the prioritization sequence of all three datasets. In addition, IBEA achieved the best performance in terms of the convergence of solutions, and NSGA-III performed the best when considering both the convergence and diversity of nondominated solutions.

Many real-world test optimization problems (e.g., test case prioritization) are multi-objective intrinsically and can be tackled using various multi-objective search algorithms (e.g., Non-dominated Sorting Genetic Algorithm (NSGA-II)). However, existing multi-objective search algorithms have certain randomness when selecting parent solutions for producing offspring solutions. In a worse case, suboptimal parent solutions may result in offspring solutions with bad quality, and thus affect the overall quality of the solutions in the next generation. To address such a challenge, we propose CBGA-ES+, a novel cluster-based genetic algorithm with non-dominated elitist selection to reduce the randomness when selecting the parent solutions to support multi-objective test optimization. We empirically compared CBGA-ES+with random search and greedy (as baselines), four commonly used multi-objective search algorithms (i.e., Multi-objective Cellular genetic algorithm (MOCell), NSGA-II, Pareto Archived Evolution Strategy (PAES), and Strength Pareto Evolutionary Algorithm (SPEA2)), and the predecessor of CBGA-ES+(named CBGA-ES) using five multi-objective test optimization problems with eight subjects (two industrial, one real world, and five open source). The results showed that CBGA-ES+managed to significantly outperform the selected search algorithms for a majority of the experiments. Moreover, for the solutions in the same search space, CBGA-ES+managed to perform better than CBGA-ES, MOCell, NSGA-II, PAES, and SPEA2 for 2.2, 13.6, 14.5, 17.4, and 9.9 percent, respectively. Regarding the running time of the algorithm, CBGA-ES+was faster than CBGA-ES for all the experiments.

The oracle problem is a key issue in testing Autonomous Driving Systems (ADS): when a collision is found, it is not always clear whether the ADS is responsible for it. Our recent search-based testing approach offers a solution to this problem by defining a collision as avoidable if a differently configured ADS would have avoided it. This approach searches for both collision scenarios and the ADS configurations capable of avoiding them. However, its main problem is that the ADS configurations generated for avoiding some collisions are not suitable for preventing other ones. Therefore, it does not provide any guidance to automotive engineers for improving the safety of the ADS. To this end, we propose a new search-based approach to generate configurations of the ADS that can avoid as many different types of collisions as possible. We present two versions of the approach, which differ in the way of searching for collisions and alternative configurations. The approaches have been experimented on the path planner component of an ADS provided by our industry partner.

Automated and autonomous driving systems (ADS) are a transformational technology in the mobility sector. Current practice for testing ADS uses virtual tests in computer simulations; search-based approaches are used to find particularly dangerous situations, possibly collisions. However, when a collision is found, it is not always easy to automatically assess whether the ADS should have been able to avoid it, without relying on offline analyses by domain experts. In this paper, we propose a definition of avoidable collision that does not rely on any domain knowledge, but only on the fact that it is possible to reconFigure the ADS (in our case, the path planner component provided by our industry partner) in a way that the collision is avoided. Based on this definition, we propose two search-based approaches for finding avoidable collisions. The first one (named sequential approach), based on current industrial practice, first searches for a collision, and then searches for an alternative configuration of the ADS which avoids it. The second one (named combined approach), instead, searches at the same time for the collision and for the alternative configuration which avoids it. Experiments show that the combined approach finds more avoidable collisions, even when the sequential approach doesn't find any; indeed, the sequential approach, in the first search, may find too severe collisions for which there is no alternative configuration that can avoid them.

Deriving a Cyber-Physical System (CPS) product from a product line requires configuring hundreds to thousands of configurable parameters of components and devices from multiple domains, e.g., computing, control, and communication. A fully automated configuration process for a CPS product line is seldom possible in practice, and a dynamic and interactive process is expected. Therefore, some configurable parameters are to be configured manually, and the rest can be configured either automatically or manually, depending on pre-defined constraints, the order of configuration steps, and previous configuration data in such a dynamic and interactive configuration process. In this article, we propose a pattern-based, interactive configuration derivation methodology (named as Pi-CD) to maximize opportunities of automatically deriving correct configurations of CPSs by benefiting from pre-defined constraints and configuration data of previous configuration steps. Pi-CD requires architectures of CPS product lines modeled with Unified Modeling Language extended with four types of variabilities, along with constraints specified in Object Constraint Language (OCL). Pi-CD is equipped with 324 configuration derivation patterns that we defined by systematically analyzing the OCL constructs and semantics. We evaluated Pi-CD by configuring 20 CPS products of varying complexity from two real-world CPS product lines. Results show that Pi-CD can achieve up to 72% automation degree with a negligible time cost. Moreover, its time performance remains stable with the increase in the number of configuration parameters as well as constraints.

Search-Based Software Engineering (SBSE) researchers who apply multi-objective search algorithms (MOSAs) often assess the quality of solutions produced by MOSAs with one or more quality indicators (QIs). However, SBSE lacks evidence providing insights on commonly used QIs, especially about agreements among them and their relations with SBSE problems and applied MOSAs. Such evidence about QIs agreements is essential to understand relationships among QIs, identify redundant QIs, and consequently devise guidelines for SBSE researchers to select appropriate QIs for their specific contexts. To this end, we conducted an extensive empirical evaluation to provide insights on commonly used QIs in the context of SBSE, by studying agreements among QIs with and without considering differences of SBSE problems and MOSAs. In addition, by defining a systematic process based on three common ways of comparing MOSAs in SBSE, we present additional observations that were automatically produced based on the results of our empirical evaluation. These observations can be used by SBSE researchers to gain a better understanding of the commonly used QIs in SBSE, in particular, regarding their agreements. Finally, based on the results, we also provide a set of guidelines for SBSE researchers to select appropriate QIs for their particular context.

Safety assurance for automotive products is crucial and challenging. It becomes even more difficult when the variability in automotive products is considered. Recently, the notion of automotive multi-product lines (multi-PL) is proposed as a unified framework to accommodate different sources of variability in automotive products. In the context of automotive multi-PL, we propose a stability analysis for safety, motivated by our industrial collaboration, where we observed that under certain operation scenarios, safety varies drastically with small fluctuations in production parameters, environmental conditions, or driving inputs. To characterize instability, we formulate a multi-objective optimization problem, and solve it with a search-based approach. The proposed technique is applied to an industrial automotive multi-PL, and experimental results show its effectiveness to spot instability. Moreover, based on information gathered during the search, we provide some insights on both testing and quality engineering of automotive products.

It is important that a Cyber-Physical System (CPS) with uncertainty in its behavior caused by its unpredictable operating environment, to ensure its reliable operation. One method to ensure that the CPS will handle such uncertainty during its operation is by testing the CPS with model-based testing (MBT) techniques. However, existing MBT techniques do not explicitly capture uncertainty in test ready models, i.e., capturing the uncertain expected behavior of a CPS in the presence of environment uncertainty. To fill this gap, we present an Uncertainty-Wise test-modeling framework, named as UncerTum , to create test ready models to support MBT of CPSs facing uncertainty. UncerTum relies on the definition of a UML profile [the UML Uncertainty Profile ( UUP )] and a set of UML Model Libraries extending the UML profile for Modeling and Analysis of Real-Time and Embedded Systems (MARTE). UncerTum also benefits from the UML Testing Profile V.2 to support standard-based MBT. UncerTum was evaluated with two industrial CPS case studies, one real-world case study, and one open-source CPS case study from the following four perspectives: (1) Completeness and Coverage of the profiles and Model Libraries in terms of concepts defined in their underlying uncertainty conceptual model for CPSs, i.e., U-Model and MARTE, (2) Effort required to model uncertainty with UncerTum , and (3) Correctness of the developed test ready models, which was assessed via model execution. Based on the evaluation, we can conclude that we were successful in modeling all the uncertainties identified in the four case studies, which gives us an indication that UncerTum is sufficiently complete. In terms of modeling effort, we concluded that on average UncerTum requires 18.5% more time to apply stereotypes from UUP on test ready models.

Object Constraint Language (OCL) constraints are typically used to provide precise semantics to models developed with the Unified Modeling Language (UML). When OCL constraints evolve regularly, it is essential that they are easy to understand and maintain. For instance, in cancer registries, to ensure the quality of cancer data, more than one thousand medical rules are defined and evolve regularly. Such rules can be specified with OCL. It is, therefore, important to ensure the understandability and maintainability of medical rules specified with OCL. To tackle such a challenge, we propose an automated search-based OCL constraint refactoring approach (SBORA) by defining and applying four semantics-preserving refactoring operators (i.e., Context Change, Swap, Split and Merge) and three OCL quality metrics (Complexity, Coupling, and Cohesion) to measure the understandability and maintainability of OCL constraints. We evaluate SBORA along with six commonly used multi-objective search algorithms (e.g., Indicator-Based Evolutionary Algorithm (IBEA)) by employing four case studies from different domains: healthcare (i.e., cancer registry system from Cancer Registry of Norway (CRN)), Oil&Gas (i.e., subsea production systems), warehouse (i.e., handling systems), and an open source case study named SEPA. Results show: 1) IBEA achieves the best performance among all the search algorithms and 2) the refactoring approach along with IBEA can manage to reduce on average 29.25 percent Complexity and 39 percent Coupling and improve 47.75 percent Cohesion, as compared to the original OCL constraint set from CRN. To further test the performance of SBORA, we also applied it to refactor an OCL constraint set specified on the UML 2.3 metamodel and we obtained positive results. Furthermore, we conducted a controlled experiment with 96 subjects and results show that the understandability and maintainability of the original constraint set can be improved significantly from the perspectives of the 96 participants of the controlled experiment.

Monitoring Industrial Cyber-Physical Systems (ICPSs) is increasingly becoming essential, which requires the development of new approaches to capture data from an ICPS for visualization, automated analyses, decision making, and anomaly detection. Towards this end, first, we present requirements for enabling personalized data visualization for monitoring evolving ICPS during its operation. Such requirements were identified from our experience of designing and developing an industrial Automated Warehouse (AW). Second, the identified requirements were validated with a questionnaire-based survey by soliciting views of industry experts from the Software Monitoring Systems and Data Visualizations domains. Based on the analysis of the survey results, the need of developing personalized ICPS monitoring systems was confirmed as a step forward to enable the efficient detection of anomalies, improvement in productivity, and providing a better overview of the monitored ICPSs. Third, we developed a model-based visualization system (named as PAVS) for monitoring ICPSs, which conforms to the identified requirements, and was validated it with a dataset collected from a real AW developed by our industrial partner: ULMA Handling System, Spain. In the end, we also provide a set of lessons learned when PAVS was transferred to ULMA.

Evidence has shown that the use of restricted natural languages can reduce ambiguities in textual use case specifications (UCSs). Restricted natural languages often come with specific editors that support particular use case templates and provide enforcement of the language's restrictions. However, whether restriction enforcement facilitates the definition of UCSs as compared to an editor without such support is a fundamental question to answer. To this end, we report results of a controlled experiment in which we compared two approaches for defining restricted UCSs: (i) a specific Restricted Use Case Modeling (RUCM) tool that supports restriction enforcement; and (ii) a general Office Word UCS template without such enforcement. We compared both approaches from multiple perspectives including restriction misuse, understandability, and restrictiveness. Results show that the restriction misuse rates are generally low, which indicates the usefulness of the RUCM, independent of the use of the editors. The results also indicate that the RUCM tool eases the application of more complex restrictions. We also found that the participants profited from extensive training prior to the experiment. The experiment participants further showed their strong willingness to recommend the RUCM tool to others and to use it in the future, which was not the case for the Office Word template.

Object Constraint Language (OCL) constraints are typically used for providing precise semantics to models developed with the Unified Modeling Language (UML). When OCL constraints evolve in a regular basis, it is essential that they are easy to understand and maintain. For instance, in cancer registries, to ensure the quality of cancer data, more than one thousand medical rules are defined and evolve regularly. Such rules can be specified with OCL. It is, therefore, important to ensure the understandability and maintainability of medical rules specified with OCL.

Modern software systems rely on information networks for communication. Such information networks are inherently unpredictable and unreliable. Consequently, software systems behave in an unstipulated manner in uncertain network conditions. Discovering unknown behaviors of these software systems in uncertain network conditions is essential to ensure their correct behaviors. Such discovery requires the development of systematic and automated methods. We propose an online and iterative model-based testing approach to evolve test models with search algorithms. Our ultimate aim is to discover unknown expected behaviors that can only be observed in uncertain network conditions. Also, we have implemented an adaptive search-based test case generation strategy to generate test cases that are executed on the system under test. We evaluated our approach with an open source video conference application-Jitsi with three search algorithms in comparison with random search. Results show that our approach is efficient in discovering unknown system behaviors. In particular, (1+1) Evolutionary Algorithm outperformed the other algorithms.

Test case prioritization (TP) prioritizes test cases into an optimal order for achieving specific criteria (e.g., higher fault detection capability) as early as possible. However, the existing TP techniques usually only produce a static test case order before the execution without taking runtime test case execution results into account. In this paper, we propose an approach for black-box dynamic TP using rule mining and multi-objective search (named as REMAP). REMAP has three key components: 1) Rule Miner, which mines execution relations among test cases from historical execution data; 2) Static Prioritizer, which defines two objectives (i.e., fault detection capability (FDC) and test case reliance score (TRS)) and applies multi-objective search to prioritize test cases statically; and 3) Dynamic Executor and Prioritizer, which executes statically-prioritized test cases and dynamically updates the test case order based on the runtime test case execution results. We empirically evaluated REMAP with random search, greedy based on FDC, greedy based on FDC and TRS, static search-based prioritization, and rule-based prioritization using two industrial and three open source case studies. Results showed that REMAP significantly outperformed the other approaches for 96% of the case studies and managed to achieve on average 18% higher Average Percentage of Faults Detected (APFD).

Driven by the needs of several industrial projects on the applications of multiobjective search algorithms, we observed that user preferences must be properly incorporated into optimization objectives. However, existing algorithms usually treat all the objectives with equal priorities and do not provide a mechanism to reflect user preferences. To address this, we propose an extension-user-preference multiobjective optimization algorithm (UPMOA), to the most commonly applied, nondominated sorting genetic algorithm II by introducing a user preference indicator δ, based on existing weight assignment strategies [e.g., uniformly distributed weights (UDW)]. We empirically evaluated UPMOA using four industrial problems from three diverse domains (i.e., communication, maritime, and subsea oil and gas). We also performed a sensitivity analysis for UPMOA with 625 algorithm parameter settings. To further assess the performance and scalability, 103 500 artificial problems were created and evaluated representing 207 sets of user preferences. Results show that the UDW strategy with UPMOA achieves the best performance and UPMOA significantly outperformed other three multiobjective search algorithms, and has the ability to solve problems with a wide range of complexity. We also observed that different parameter settings led to the varied performance of UPMOA, thus suggesting that configuring proper parameters is highly problem-specific.

A typical way to improve the quality of requirements is to assign them to suitable stakeholders for reviewing. Due to different characteristics of requirements and diverse background of stakeholders, it is needed to find an optimal solution for requirements assignment. Existing search-based requirements assignment solutions focus on maximizing stakeholders' familiarities to assigned requirements and balancing the overall workload of each stakeholder. However, a cost-effective requirements assignment solution should also take into account another two optimization objectives: 1) minimizing required time for reviewing requirements, and 2) minimizing the monetary cost required for performing reviewing tasks. We formulated the requirements assignment problem as a search problem and defined a fitness function considering all the five optimization objectives. We conducted an empirical evaluation to assess the fitness function together with six search algorithms using a real-world case study and 120 artificial problems to assess the scalability of the proposed fitness function. Results show that overall, our optimization problem is complex and further justifies the use for multi-objective search algorithms, and the Speed-constrained Multi-Objective Particle Swarm Optimization (SMPSO) algorithm performed the best among all the search algorithms.

Nowadays, an increasing number of systems are being developed by integrating products (belonging to different product lines) that communicate with each other through information networks. Cost-effectively supporting Product Line Engineering (PLE) and in particular enabling automation of configuration in PLE is a challenge. Capturing rules is the key for enabling automation of configuration. Product configuration has a direct impact on runtime interactions of communicating products. Such products might be within or across product lines and there usually don't exist explicitly specified rules constraining configurable parameter values of such products. Manually specifying such rules is tedious, time-consuming, and requires expert's knowledge of the domain and the product lines. To address this challenge, we propose an approach named as SBRM that combines multi-objective search with machine learning to mine rules. To evaluate the proposed approach, we performed a real case study of two communicating Video Conferencing Systems belonging to two different product lines. Results show that SBRM performed significantly better than Random Search in terms of fitness values, Hyper-Volume, and machine learning quality measurements. When comparing with rules mined with real data, SBRM performed significantly better in terms of Failed Precision (18%), Failed Recall (72%), and Failed F-measure (59%).

To ensure the quality of requirements, a common practice, especially in critical domains, is to review requirements within a limited time and monetary budgets. A requirement with higher importance, larger number of dependencies with other requirements, and higher implementation cost should be reviewed with the highest priority. However, requirements are inherently uncertain in terms of their impact on the requirements implementation cost. Such cost is typically estimated by stakeholders as an interval, though an exact value is often used in the literature for requirements optimization (e.g., prioritization). Such a practice, therefore, ignores uncertainty inherent in the estimation of requirements implementation cost. This paper explicitly takes into account such uncertainty for requirement prioritization and formulates four objectives for uncertainty-wise requirements prioritization with the aim of maximizing 1) the importance of requirements, 2) requirements dependencies, 3) the implementation cost of requirements, and 4) cost over-run probability. We evaluated the multi-objective search algorithm NSGA-II together with Random Search (RS) using the RALIC dataset and 19 artificial problems. Results show that NSGA-II can solve the requirements prioritization problem with a significantly better performance than RS. Moreover, NSGA-II can prioritize requirements with higher priority earlier in the prioritization sequence. For example, in the case of the RALIC dataset, the first 10% of prioritized requirements in the prioritization sequence are on average 50% better than RS in terms of prioritization effectiveness.

Time-related properties are a critical type of extrafunctional requirements for designing real-time systems. Modeling and validating time-related properties at the requirements specification and analysis phases is important for the successful development of real-time systems in terms of cost, quality and productivity. In the literature and practice, timing analyses (e.g., Worst Case Execution Time) are often performed to ensure that the design of a real-time system fully conforms to its time-related constraints. However, such analyses are mostly performed at the design and implementation stages, but not at the requirements level. This paper presents a restricted, natural language based, use case modeling methodology (named as RUCM4RT) to specify functional requirements of real-time systems as use case models, along with associated time-related constraints. RUCM4RT was proposed based on the UML profile for Modeling and Analysis of Real-Time and Embedded Systems (MARTE). In addition, in this paper, we also propose a metamodel-based formalization mechanism named as UCMeta4RT to automatically formalize use case models. We have conducted two real-world case studies to evaluate our solution and 40 use cases were modeled, among which 27 realtime use cases, 118 time-related constraints and 47 other extrafunctional (also commonly called non-functional) constraints were specified. Results show that RUCM4RT was able to handle all the real-time related elements (e.g., time-related constraints) of the use case models.

The Cancer Registry of Norway (CRN) employs a cancer registry system to collect cancer patient data (e.g., diagnosis and treatments) from various medical entities (e.g., clinic hospitals). The collected data are then checked for validity (i.e., validation) and assembled as cancer cases (i.e., aggregation) based on more than 1000 cancer coding rules in the system. However, it is frequent in practice that the collected cancer data changes due to various reasons (e.g., different treatments) and the cancer coding rules can also change/evolve due to new medical knowledge. Thus, such a cancer registry system requires an efficient means to automatically analyze these changes and provide consequent impacts to medical experts for further actions. This paper proposes an automated Rule-based Change Impact Analysis (CIA) approach named RCIA that includes: 1) a change classification to capture the potential changes that can occur at CRN; 2) in total 80 change impact analysis rules including 50 dependency rules and 30 impact rules; and 3) an efficient algorithm to analyze changes and produce consequent impacts. We evaluate RCIA via a case study with 12 real change sets from CRN and a conducted interview. The results showed that RCIA managed to produce 100% actual change impacts and the medical expert at CRN is quite positive to apply RCIA to facilitate their cancer registry system. We also shared a set of lessons learned based on the collaboration with CRN.

Multi-objective search algorithms (e.g., non-dominated sorting genetic algorithm II (NSGA-II)) have been frequently applied to address various testing problems requiring multi-objective optimization such as test case selection. However, existing multi-objective search algorithms have certain randomness when selecting parent solutions for producing offspring solutions. In the worse case, suboptimal parent solutions may result in offspring solutions with bad quality, and thus affect the overall quality of the next generation. To address such a challenge, we propose a cluster-based genetic algorithm with elitist selection (CBGA-ES) with the aim to reduce such randomness for supporting multi-objective test optimization. We empirically compared CBGA-ES with random search, greedy (as baselines) and four commonly used multi-objective search algorithms (e.g., NSGA-II) using two industrial and one real world test optimization problem, i.e., test suite minimization, test case prioritization, and test case selection. The results showed that CBGA-ES significantly outperformed the baseline algorithms (e.g., greedy), and the four selected search algorithms for all the three test optimization problems. CBGA-ES managed to outperform more than 75% of the objectives for all the four algorithms in each test optimization problem. Moreover, CBGA-ES was able to improve the quality of the solutions for an average of 32.5% for each objective as compared to the four algorithms for the three test optimization problems.

Context Cyber-Physical Systems (CPSs), when deployed for operation, are inherently prone to uncertainty. Considering their applications in critical domains (e.g., healthcare), it is important that such CPSs are tested sufficiently, with the explicit consideration of uncertainty. Model-based testing (MBT) involves creating test ready models capturing the expected behavior of a CPS and its operating environment. These test ready models are then used for generating executable test cases. It is, therefore, necessary to develop methods that can continuously evolve, based on real operational data collected during the operation of CPSs, test ready models and uncertainty captured in them, all together termed as Belief Test Ready Models (BMs) Objective Our objective is to propose a model evolution framework that can interactively improve the quality of BMs, based on operational data. Such BMs are developed by one or more test modelers ( belief agents ) with their assumptions about the expected behavior of a CPS, its expected physical environment, and potential future deployments. Thus, these models explicitly contain subjective uncertainty of the test modelers. Method We propose a framework (named as UncerTolve ) for interactively evolving BMs (specified with extended UML notations) of CPSs with subjective uncertainty developed by test modelers. The key inputs of UncerTolve include initial BMs of CPSs with known subjective uncertainty and real data collected from the operation of CPSs. UncerTolve has three key features: 1) Validating the syntactic correctness and conformance of BMs against real operational data via model execution, 2) Evolving objective uncertainty measurements of BMs via model execution, and 3) Evolving state invariants (modeling test oracles) and guards of transitions (modeling constraints for test data generation) of BMs with a machine learning technique. Results As a proof-of-concept, we evaluated UncerTolve with one industrial CPS case study, i.e., GeoSports from the healthcare domain. Using UncerTolve , we managed to evolve 51% of belief elements, 18% of states, and 21% of transitions as compared to the initial BM developed in an industrial setting. Conclusion UncerTolve can successfully evolve model elements of the initial BM, in addition to objective uncertainty measurements using real operational data. The evolved model can be used to generate additional test cases covering evolved model elements and objective uncertainty. These additional test cases can be used to test the current and future deployments of a CPS to ensure that it will handle uncertainty gracefully during its operations.

A key component of model-based testing is the generation of test data from constraints (e.g., specified in the Object Constraint Language (OCL)) associated with models e.g., specified in the Unified Modeling Language (UML). The quality of test data eventually determines the effectiveness of test cases, e.g., fault detection and coverage. A simple way to generate test data from an OCL constraint is to find a set of values for all the variables that satisfies the constraint. One way of improving the quality of test data is by generating test data at the boundaries of each variable in the constraint. In this paper, we extend our search-based test data generation approach to generate test data at the boundaries of each variable involved in a constraint. We present different cases of constraints involving different types of variables and how we rewrite them in order to automatically generate boundary values using our existing OCL Solver called EsOCL based on a set of heuristics. We empirically evaluated our heuristics using three search algorithms, i.e., Genetic Algorithm, (1+1) Evolutionary Algorithm, and Alternating Variable Method, on industrial OCL constraints from our previous work. Results show that AVM on average achieved a 95% of success rate and outperformed the rest of the algorithms in finding all the boundary values of constraints in fewer generations.

Due to limited time and resources available for execution, test case selection always remains crucial for cost-effective testing. It is even more prominent when test cases require manual steps, e.g., operating physical equipment. Thus, test case selection must consider complicated trade-offs between cost (e.g., execution time) and effectiveness (e.g., fault detection capability). Based on our industrial collaboration within the Maritime domain, we identified a real-world and multi-objective test case selection problem in the context of robustness testing, where test case execution requires human involvement in certain steps, such as turning on the power supply to a device. The high-level goal is to select test cases for execution within a given time budget, where test engineers provide weights for a set of objectives, depending on testing requirements, standards, and regulations. To address the identified test case selection problem, we defined a fitness function including one cost measure, i.e., Time Difference (TD) and three effectiveness measures, i.e., Mean Priority (MPR), Mean Probability (MPO) and Mean Consequence (MC) that were identified together with test engineers. We further empirically evaluated eight multi-objective search algorithms, which include three weight-based search algorithms (e.g., Alternating Variable Method) and five Pareto-based search algorithms (e.g., Strength Pareto Evolutionary Algorithm 2 (SPEA2)) using two weight assignment strategies (WASs). Notice that Random Search (RS) was used as a comparison baseline. We conducted two sets of empirical evaluations: 1) Using a real world case study that was developed based on our industrial collaboration; 2) Simulating the real world case study to a larger scale to assess the scalability of the search algorithms. Results show that SPEA2 with either of the WASs performed the best for both the studies. Overall, SPEA2 managed to improve on average 32.7%, 39% and 33% in terms of MPR, MPO and MC respectively as compared to RS.

Many software engineering problems are multi-objective in nature, which has been largely recognized by the Search-based Software Engineering (SBSE) community. In this regard, Pareto-based search algorithms, e.g., Non-dominated Sorting Genetic Algorithm II, have already shown good performance for solving multi-objective optimization problems. These algorithms produce Pareto fronts, where each Pareto front consists of a set of non-dominated solutions. Eventually, a user selects one or more of the solutions from a Pareto front for their specific problems. A key challenge of applying Pareto-based search algorithms is to select appropriate quality indicators, e.g., hypervolume, to assess the quality of Pareto fronts. Based on the results of an extended literature review, we found that the current literature and practice in SBSE lacks a practical guide for selecting quality indicators despite a large number of published SBSE works. In this direction, the paper presents a practical guide for the SBSE community to select quality indicators for assessing Pareto-based search algorithms in different software engineering contexts. The practical guide is derived from the following complementary theoretical and empirical methods: 1) key theoretical foundations of quality indicators; 2) evidence from an extended literature review; and 3) evidence collected from an extensive experiment that was conducted to evaluate eight quality indicators from four different categories with six Pareto-based search algorithms using three real industrial problems from two diverse domains.

In the context of large-scale system product line engineering, manual configuration is often mandatory and therefore inevitably introduces nonconformities: violating pre-defined constraints for conformance checking. Resolving nonconformities without proper tool support is more or less random, as there are usually hundreds and thousands of configurable parameters and conformance constraints, in the context of configuring a large-scale and directly deployable system. Moreover, inter-connections among constraints and configurable parameters worsen the feasibility of manual resolving nonconformities without proper tool support. In this paper, we present an automatic approach (named as Zen-FIX) to optimally recommend solutions to resolve nonconformities using multi-objective search. Solutions recommended by Zen-FIX conform to all pre-defined constraints and are optimized in terms of maximizing the overall efficiency of an interactive product configuration process. We evaluated Zen-FIX with a real-world case study containing 52454 optimization problems, with which we evaluated seven multi-objective search algorithms. Results show that MoCell outperformed all others: CellDE, IBEA, NSGA-II, PESA2, Random, SPEA2, for most of the problems, in terms of Efficiency (a combined metric of finding optimized solutions and time performance).

Industrial Cyber Physical Systems (CPSs) are naturally complex. Manual configuration of CPS product lines is error-prone and inefficient, which warrants the need for automated support of product configuration activities such as decision inference and decision ordering. A fully automated solution is often impossible for CPSs since some decisions must be made manually by configuration engineers and thus requiring an interactive and step-by-step configuration solution. Having an interactive solution with tool support in mind, we propose a search-based solution (named as Zen-DO) to support optimal ordering of configuration steps. The optimization objective has three parts: 1) minimizing overall manual configuration steps, 2) configuring most constraining decisions first, and 3) satisfying ordering dependencies among variabilities. We formulated our optimization objective as a fitness function and investigated it along with four search algorithms: Alternating Variable Method (AVM), (1+1) Evolutionary Algorithm (EA), Genetic Algorithm, and Random Search (a comparison baseline). Their performance is evaluated in terms of finding an optimal solution for two real-world case studies of varying complexity and results show that AVM and (1+1) EA significantly outperformed the others.

Uncertainty is intrinsic in Cyber-Physical Systems (CPSs) due to novel interactions of embedded systems, networking equipment, cloud infrastructures and humans. Our daily life has been increasing dependent on CPS applications in safety/mission critical domains such as healthcare, aerospace, oil/gas and maritime. For example, the National Institute of Standards and Technology (NIST) reported that direct CPS applications account for more than $32.3 trillions and expect to grow $82 trillions by 2025 (about half of the world economy). Expecting enormous dependence of our lives on CPSs in the future, dealing with uncertainty at an acceptable cost is vital to avoid posing undue threats to its users and environment. To ensure correct delivery of their functions at an acceptable cost even in the presence of uncertainty, CPSs must be reliable, robust, efficient, safe, and secure. All these properties are facets of a more general property often known as dependability. Improving system dependability first and foremost relies on the ability to verify and validate CPSs in a cost-effective manner and one way of achieving this is via systematic and automated Model-Based Testing (MBT): automated derivation of test cases from a behavioral model of a system. MBT supports rigorous, systematic, and automated testing, which eventually reduces the number of faults in the delivered systems and thus improves their quality. The goal of the U-Test project (a recently funded project under the EU Horizon2020 program (http://ec.europa.eu/programmes/horizon2020/) is to improve the dependability of CPSs, via cost-effective, model-based and search-based testing of CPSs under unknown risky uncertainty. Unknown uncertainty is the state of a CPS that can only be determined at the runtime as opposed to known uncertainty that is known at the design time and outcome from risky uncertainty is undesirable. To achieve our goal, we will advance the current state-of-art of testing CPSs by developing a novel solution based on sound theoretical foundation for uncertainty testing in the following steps: 1) Developing a light-weight modelling solution with rich formalism to support minimal modelling of known uncertainty with risk information; 2) Intelligently evolving known uncertainty models towards realistic and risky unknown uncertainty models (evolved models) using search algorithms (e.g., genetic algorithms mimicking natural selection); and 3) Automatically generating test cases from the evolved models to test a CPS under unknown uncertainty to ensure that the CPS continues to operate properly and possibly at a reduced quality of operation, rather than failing completely.

Multi-objective search algorithms (e.g., non-dominated sorting genetic algorithm II (NSGA-II)) have been applied extensively to solve various multi-objective optimization problems in software engineering such as problems in testing. However, existing multi-objective algorithms usually treat all the objectives with equivalent priorities and do not provide a mechanism to reflect various user preferences when guiding search. The need to have such a mechanism was observed in one of our industrial projects on applying search algorithms for test optimization of a product line of Videoconferencing Systems (VCSs) called Saturn, where user preferences must be incorporated into optimization objectives, based on domain knowledge of test engineers for VCS testing. To address this, we propose an extension to the most commonly-used multi-objective search algorithm NSGA-II, which has shown promising results with user preferences. We name the extension as User-Preference Multi-Objective Optimization Algorithm (UPMOA), which includes a user preference indicator p and is based on existing weight assignment strategies. We empirically evaluated UPMOA with two industrial problems focusing on optimizing the test execution system for Saturn in Cisco. To assess the performance and scalability of UPMOA, inspired by the two industrial problems, in total we created 64000 artificial problems with 128 different sets of user preferences. The evaluation includes two aspects: 1) Three weight assignment strategies together with UPMOA were empirically evaluated to identify a best weight assignment strategy for p. Results show that the Uniformly Distributed Weights (UDW) strategy can assist UPMOA in achieving the best performance; 2) UPMOA was compared with three representative multi-objective search algorithms (including NSGA-II) and results show that UPMOA significantly outperformed the others and has the ability to solve problems with a wide range of complexity.

Based on our experience of collaborating with industry, we observed that test case generation usually relies on test case specifications (TCSs), commonly written in natural language, specifying test cases of a System Under Test at a high level of abstraction. In practice, TCSs are commonly used by test engineers as reference documents to perform these activities: 1) Manually executing test cases in TCSs; 2) Manually coding test cases in a test scripting language for automated test case execution. In the latter case, the gap between TCSs and executable test cases has to be filled by test engineers, requiring a significant amount of coding effort and domain knowledge. Motivated by the above observations from the industry, we first propose, in this paper, a TCS language, named as Restricted Test Case Modeling (RTCM), which is based on natural language and composed of an easy-to-use template, a set of restriction rules and keywords. Second, we propose a test case generation tool (aToucan4Test), which takes TCSs in RTCM as input and generates either manual test cases or automatically executable test cases, based on various coverage criteria defined on RTCM. To assess the applicability of RTCM, we manually modeled two industrial case studies and examined 30 automatically generated TCSs. To evaluate aToucan4Test, we modeled three subsystems of a Video Conferencing System developed by Cisco Systems, Norway and automatically generated executable test cases. These test cases were successfully executed on two commercial software versions. In the paper, we also discuss our experience of applying RTCM and aToucan4Test in an industrial context and compare our approach with other model-based testing methodologies.

Model-based testing (MBT) provides a systematic and automated way to facilitate rigorous testing of software systems. MBT has been an intense area of research and a large number of MBT techniques have been developed in the literature and in the practice. However, all of the techniques have been developed using their own concepts and terminology of MBT, which are very often different than other techniques and at times have conflicting semantics. Moreover, while working on MBT projects with our industrial partners in the last several years, we were unable to find a unified way of defining MBT techniques based on standard terminology. To precisely define MBT concepts with the aim of providing common understanding of MBT terminology across techniques, we formalize a small subset of the recently released ISO/IEC/IEEE 29119 Software Testing Standard as a conceptual model (UML class diagrams) together with OCL constraints. The conceptual model captures all the necessary concepts based on the standard terminology that are mandatory or optional in the context of MBT techniques and can be used to define new MBT tools and techniques. To validate the conceptual model, we instantiated its concepts for various MBT techniques previously developed in the context of our industrial partners. Such instantiation automatically enforces the specified OCL constraints. This type of validation provided us feedback to further refine the conceptual model. Finally, we also provide our experiences and lessons learnt for such formalization and validation.

Cyber-Physical Systems (CPSs) are the future generation of highly connected embedded systems having applications in diverse domains including Oil and Gas. Employing Product Line Engineering (PLE) is believed to bring potential benefits with respect to reduced cost, higher productivity, higher quality, and faster time-to-market. However, relatively few industrial field studies are reported regarding the application of PLE to develop large-scale systems, and more specifically CPSs. In this paper, we report about our experiences and insights gained from investigating the application of model-based PLE at a large international organization developing subsea production systems (typical CPSs) to manage the exploitation of oil and gas production fields. We report in this paper 1) how two systematic domain analyses (on requirements engineering and product configuration/derivation) were conducted to elicit CPS PLE requirements and challenges, 2) key results of the domain analysis (commonly observed in other domains), and 3) our initial experience of developing and applying two Model Based System Engineering (MBSE) PLE solution to address some of the requirements and challenges elicited during the domain analyses.

Summary Ensuring that avionics software meets safety requirements at each development stage is very important to warrant the safe operation of an avionics system. Many safety requirements are imposed by various standards and industrial regulations that must be met by avionics software. One of such standards is DO‐178B/C, which provides guidelines (e.g., development process and objectives to satisfy in development activities) for meeting safety requirements. This paper presents a modeling methodology including a UML profile for specifying safety requirements on a component‐based architecture model and a set of design guidelines on avionics software. These safety requirements were identified from both standards (mainly DO‐178B/C) and current engineering practices in the domain of avionics systems. The methodology automatically enforces these safety requirements. We have applied the methodology on an industrial autopilot system, and several previously uncaught faults were revealed. Copyright © 2014 John Wiley & Sons, Ltd.

The Object Constraint Language (OCL) is a standard language for specifying constraints on Unified Modeling Language (UML) models. The specified constraints can be used for various purposes including verification, and model-based testing (e.g., test data generation). Efficiently solving OCL constraints is one of the key requirements for the practical use of OCL. In this paper, we propose an improvement in existing heuristics to solve OCL constraints using search algorithms. We evaluate our improved heuristics using two empirical studies with three search algorithms: Alternating Variable Method (AVM), (1+1) Evolutionary Algorithm (EA), and a Genetic Algorithm (GA). We also used Random Search (RS) as a comparison baseline. The first empirical study was conducted using carefully designed artificial problems (constraints) to assess each individual heuristics. The second empirical study is based on an industrial case study provided by Cisco about model-based testing of Video Conferencing Systems. The results of both empirical evaluations reveal that the effectiveness of the search algorithms, measured in terms of time to solve the OCL constraints to generate data, is significantly improved when using the novel heuristics presented in this paper. In particular, our experiments show that (1+1) EA with the novel heuristics has the highest success rate among all the analyzed algorithms, as it requires the least number of iterations to solve constraints.

During the early phase of project development lifecycle of large scale cyber-physical systems, a large number of requirements are needed to be assigned to different stakeholders from different organizations or different departments of the same organization for reviewing, clarifying and checking their conformance to industry standards and government or other regulations. These requirements have different characteristics such as various extents of importance to the organization, complexity, and dependencies between each other, thereby requiring different effort (workload) to review and clarify. While working with our industrial partners in the domain of cyber-physical systems, we discovered an optimization problem, where an optimal solution is required for assigning requirements to different stakeholders by maximizing their familiarities to the assigned requirements while balancing the overall workload of each stakeholder. We propose a fitness function which was investigated with four search algorithms: (1+1) Evolutionary Algorithm (EA), Genetic Algorithm, and Alternating Variable Method, whereas Random Search is used as a comparison base line. We empirically evaluated their performance for finding an optimal solution using a large-scale industrial case study and 120 artificial problems with varying complexity. Results show that (1+1) EA gives the best results together with our proposed fitness function as compared to the other three algorithms.

UML state machines are commonly used to model the state-based behavior of communication and control systems to support various activities such as test cases and code generation. Standard UML state machines are well suited to model functional behavior, however extra-functional behavior such as robustness and security can also be directly modeled on them, but this often results in cluttered models since extra-functional behaviors are often crosscutting. Such modeling crosscutting behavior directly on UML state machines is a common practice. Aspect-Oriented Modeling (AOM) allows systematically modeling of crosscutting behavior and has shown to provide a scalable solution in the recent years. However, due to lack of familiarity of AOM in both academic and industry, extra-functional behavior is often modeled directly on UML state machines and as a result those UML state machines are difficult to read and maintain. To improve the readability of already developed UML state machines and ease maintenance, we propose a set of heuristics, derived from two industrial cases studies, implemented in a tool to automatically identify commonly observed crosscutting behaviors in UML state machines and refactor them as Aspect State Machines. Such refactoring makes the state machines easier to maintain and comprehend. We present the results of applying our proposed heuristics to the existing UML state machines of two industrial case studies developed for model-based testing.

Model-Based Testing (MBT) is a well-established and intense field of research in academia. It has attracted attention of many industries as it can be seen from many industrial experiences of MBT reported in the literature and availability of commercial and open source tools in recent years. The thorough and methodical approach of MBT facilitates automated testing with the intention of improving the quality of software systems. Every industrial application of MBT faces varied challenges depending on the application domain, the current testing practices and tools, and the type of testing. Reporting such challenges, their solutions, and lessons learnt provides a body of knowledge, which can direct practitioners of MBT for their future applications of MBT. With such aim in our mind, we present results from an MBT project that is being carried out for testing embedded video conferencing systems developed by Cisco Systems, Inc. Norway for the last several years. We present challenges faced while conducting MBT, our solutions, some of the key results, and lessons learnt from our experience. Our experience showed that search algorithms provide an efficient solution for test case selection and test data generation. In addition, aspect-oriented modeling provides a scalable modeling solution for non-functional testing. Finally, we learned that model transformation offers an elegant solution for developing a model-based test case generation tool. All of our results are based on a large number of rigorous empirical evaluations.

In the context of product line engineering (PLE), providing immediate feedback on the correctness of a manual configuration step to users has a practical impact on whether a configuration process with tool support can be successfully adopted in practice. Model-based PLE has brought opportunities to enable automated product configuration and derivation for large-scale systems/software, in which models are used as the abstract specification of commonalities and variabilities of products of a product line. In our previous work, we have proposed a UML-based variability modeling methodology and an interactive configuration process. Based on these work, in this paper, we propose an automated and incremental conformance checking approach to ensure that the manual configuration to each variation point conforms to a set of pre-defined conformance rules specified in OCL. The proposed approach, called Zen-CC is implemented as a component of our product configuration and derivation tool, named as Zen-Configurator. The proposed approach is evaluated with two real-world case studies and results showed that the performance of Zen-CC is significantly better than a baseline algorithm checking all the conformance rules at each configuration step. Moreover, the performance of Zen-CC rarely varies during the configuration process, suggesting that our approach is scalable for configuring products with a large number of configuration points.

Test prioritization is crucial for testing products in a product line considering limited budget in terms of available time and resources. In general, it is not practically feasible to execute all the possible test cases and so, ordering test case execution permits test engineers to discover faults earlier in the testing process. An efficient prioritization of test cases for one or more products requires a clear consideration of the tradeoff among various costs (e.g., time, required resources) and effectiveness (e.g., feature coverage) objectives. As an integral part of the future Cisco's test scheduling system for validating video conferencing products, we introduce a search-based multi-objective test prioritization technique, considering multiple cost and effectiveness measures. In particular, our multi-objective optimization setup includes the minimization of execution cost (e.g., time), and the maximization of number of prioritized test cases, feature pairwise coverage and fault detection capability. Based on cost-effectiveness measures, a novel fitness function is defined for such test prioritization problem. The fitness function is empirically evaluated together with three commonly used search algorithms (e.g., (1+1) Evolutionary algorithm (EA)) and Random Search as a comparison baseline based on the Cisco's industrial case study and 500 artificial designed problems. The results show that (1+1) EA achieves the best performance for solving the test prioritization problem and it scales up to solve the problems of varying complexity.

In this paper, we report a comprehensive analysis of Effort required for performing various modeling activities, based on the data we collected from a series of controlled experiments. These experiments were conducted to evaluate the overall applicability of a UML profile for Aspect-oriented Modeling (AOM): AspectSM. AspectSM was developed to model crosscutting behaviors on UML state machines and has been successfully applied for supporting model-based robustness testing at Cisco Systems, Norway. Our analysis results show that Effort required for applying AspectSM does not deviate significantly from Effort required for applying non-AOM modeling approaches for various modeling activities related to crosscutting behaviors. In addition, our previous studies show that AspectSM helps achieving improved readability of state machines, improved modeling quality, and increased understandability, reduced modeling errors, and require lesser number of modeling elements. Therefore, we conclude that modeling crosscutting behaviors using AspectSM can bring significant benefits in practice at no significant additional Effort as compared to using standard UML state machines.

Test minimization techniques aim at identifying and eliminating redundant test cases from test suites in order to reduce the total number of test cases to execute, thereby improving the efficiency of testing. In the context of software product line, we can save effort and cost in the selection and minimization of test cases for testing a specific product by modeling the product line. However, minimizing the test suite for a product requires addressing two potential issues: 1) the minimized test suite may not cover all test requirements compared with the original suite; 2) the minimized test suite may have less fault revealing capability than the original suite. In this paper, we apply weight-based Genetic Algorithms (GAs) to minimize the test suite for testing a product, while preserving fault detection capability and testing coverage of the original test suite. The challenge behind is to define an appropriate fitness function, which is able to preserve the coverage of complex testing criteria (e.g., Combinatorial Interaction Testing criterion). Based on the defined fitness function, we have empirically evaluated three different weight-based GAs on an industrial case study provided by Cisco Systems, Inc. Norway. We also presented our results of applying the three weight-based GAs on five existing case studies from the literature. Based on these case studies, we conclude that among the three weight-based GAs, Random-Weighted GA (RWGA) achieved significantly better performance than the other ones.

Aspect-Oriented Modeling (AOM) has been the subject of intense research over the last decade and aims to provide numerous benefits to modeling, such as enhanced modularization, easier evolution, higher quality as well as reduced modeling effort. However, these benefits can only be obtained at the cost of learning and applying new modeling approaches. Studying their applicability is therefore important to assess whether they are worth using in practice. In this paper, we report a controlled experiment to assess the applicability of AOM, focusing on a recently published UML profile (AspectSM). This profile was originally designed to support model-based robustness testing in an industrial context but is applicable to the behavioral modeling of other crosscutting concerns. This experiment assesses the applicability of AspectSM from two aspects: the quality of derived state machines and the effort required to build them. With AspectSM, a crosscutting behavior is modeled using an “aspect state machine”. The applicability of aspect state machines is evaluated by comparing them with standard UML state machines that directly model the entire system behavior, including crosscutting concerns. The quality of both aspect and standard UML state machines derived by subjects is measured by comparing them against predefined reference state machines. Results show that aspect state machines derived with AspectSM are significantly more complete and correct though AspectSM took significantly more time than the standard approach.

Real Time Operating System for Avionics (RTOS4A) is responsible for providing an operating environment for avionics application software. Avionics software being safety-critical in nature poses several safety and reliability requirements on RTOS4A in addition to the requirements imposed by standards, for instance, DO-178B. Due to this reason, reliability assessment of RTOS4A is very critical to demonstrate confidence about its reliability to its relevant stakeholders. One common way of assessing reliability is by systematic analyses of testing data such as number of tests, number of failures, and coverage using appropriate statistical tests. In this paper, we report our experience of assessing the reliability of an industrial RTOS4A based on testing data collected for 17 months on eight continuous releases. We studied correlation among various measures including: Testing Effort Measures (e.g., complexity of test cases), Testing Effectiveness Measures (e.g., number of failures), and Complexity Measures (e.g., number of functions in a release) and provide in this paper a set of recommendations to assess the reliability of RTOS4A, which serve as guidelines to practitioners in the domain of RTOS4A.

Model-based testing (MBT) aims at automated, scalable, and systematic testing solutions for complex industrial software systems. To increase chances of adoption in industrial contexts, software systems can be modeled using well-established standards such as the Unified Modeling Language (UML) and the Object Constraint Language (OCL). Given that test data generation is one of the major challenges to automate MBT, we focus on test data generation from OCL constraints in this paper. This endeavor is all the more challenging given the numerous OCL constructs and operations that are designed to facilitate the definition of constraints. Though search-based software testing has been applied to test data generation for white-box testing (e.g., branch coverage), its application to the MBT of industrial software systems has been limited. In this paper, we propose a set of search heuristics targeted to OCL constraints to guide test data generation and automate MBT in industrial applications. We evaluate these heuristics for three search algorithms: Genetic Algorithm, (1+1) Evolutionary Algorithm, and Alternating Variable Method. We empirically evaluate our heuristics using complex artificial problems, followed by empirical analyses of the feasibility of our approach on one industrial system in the context of robustness testing. Our approach is also compared with the most widely referenced OCL solver (UMLtoCSP) in the literature and shows to be significantly more efficient.

Aspect-oriented Modeling (AOM) is a relatively recent, very active field of research and is assumed to yield several potential benefits such as enhanced modularization, easier evolution, increased reusability, and improved readability and understandability of models, as well as reduced modeling effort. However, credible, solid empirical evidence of such benefits is very rare. In this paper, we evaluate the understandability of state machines, when modeling crosscutting behavior using AOM and more specifically AspectSM, a UML profile extending the UML state machine notation to provide mechanisms to define aspects using state machines. With AspectSM, crosscutting behavior is modeled using so-called aspect state machines, which are woven into a base state machine (modeling core functionality) to produce a woven state machine. Modeling aspect state machines separately from the base state machine no doubt offers several abovementioned benefits but on the other hand poses extra challenges for a modeler to understand them because of implicit interactions between both aspect and base state machines. This paper reports a study, which was specifically designed to evaluate the understandability of aspect state machines. The understandability of aspect state machines in conjunction with a base state machine is evaluated by comparing woven state machines produced by subjects (subject woven state machines) with the woven state machines automatically generated by our weaver (reference woven state machines). Understandability is measured from the aspects of Completeness and Redundancy of a subject's woven state machine when compared with the corresponding reference woven state machine. Results of the study show that on average, we observed completeness of 71%, whereas we observed approximately 6% of redundancy. We also observed that subjects took significantly more time to weave more complex aspect state machines (with more states, transitions, and pointcuts).

In recent years, Model-Based Testing (MBT) has attracted an increasingly wide interest from industry and academia. MBT allows automatic generation of a large and comprehensive set of test cases from system models (e.g., state machines), which leads to the systematic testing of the system. However, even when using simple test strategies, applying MBT in large industrial systems often leads to generating large sets of test cases that cannot possibly be executed within time and cost constraints. In this situation, test case selection techniques are employed to select a subset from the entire test suite such that the selected subset conforms to available resources while maximizing fault detection. In this paper, we propose a new similarity-based selection technique for state machine-based test case selection, which includes a new similarity function using triggers and guards on transitions of state machines and a genetic algorithm-based selection algorithm. Applying this technique on an industrial case study, we show that our proposed approach is more effective in detecting real faults than existing alternatives. We also assess the overall benefits of model-based test case selection in our case study by comparing the fault detection rate of the selected subset with the maximum possible fault detection rate of the original test suite.

Metaheuristic search techniques have been extensively used to automate the process of generating test cases, and thus providing solutions for a more cost-effective testing process. This approach to test automation, often coined “Search-based Software Testing” (SBST), has been used for a wide variety of test case generation purposes. Since SBST techniques are heuristic by nature, they must be empirically investigated in terms of how costly and effective they are at reaching their test objectives and whether they scale up to realistic development artifacts. However, approaches to empirically study SBST techniques have shown wide variation in the literature. This paper presents the results of a systematic, comprehensive review that aims at characterizing how empirical studies have been designed to investigate SBST cost-effectiveness and what empirical evidence is available in the literature regarding SBST cost-effectiveness and scalability. We also provide a framework that drives the data collection process of this systematic review and can be the starting point of guidelines on how SBST techniques can be empirically assessed. The intent is to aid future researchers doing empirical studies in SBST by providing an unbiased view of the body of empirical evidence and by guiding them in performing well-designed and executed empirical studies.

Abstract Finite‐state machine specifications form the basis for a number of rigorous state‐based testing techniques and can help to understand program behaviour. Unfortunately they are rarely maintained during software development, which means that these benefits can rarely be fully exploited. This paper describes a technique that, given a set of states that are of interest to a developer, uses symbolic execution to reverse‐engineer state transitions from source code. A particularly novel aspect of our approach is that, besides determining whether or not a state transition can take place, it also identifies the paths through the source code that govern a transition. The technique has been implemented as a prototype, enabling its preliminary evaluation with respect to real software systems. Copyright © 2007 John Wiley & Sons, Ltd.