Ensuring the safety and reliability of Cyber-Physical Systems (CPS) is critical, particularly in safety-critical domains such as automotive and aerospace. Fault Injection (FI) is a well-established technique for testing system resilience, but current FI tools often face challenges when applied to Simulink-based CPS models. In this paper, we analyze the shortcomings of existing FI methods, and reflect on the key challenges of FI for Simulink-based CPS models. By offering insights into these challenges and proposing research pathways, we aim to inspire further advances in FI methodologies, enabling more robust testing of CPS in real-world applications.

Design of cyber-physical systems (CPS) typically involves dataflow modeling. The structure of dataflow models differs from the traditional software, making standard coverage metrics not appropriate for measuring the thoroughness of testing. To address this limitation, this article proposes signal feature coverage as a new coverage metric for systematically testing CPS dataflow models. We derive signal feature coverage by leveraging signal features. We developed a testing framework in Simulink, a popular dataflow modeling and simulation environment, that automates the generation and execution of test cases based on the defined coverage metric. We evaluated the effectiveness of our approach by carrying out experiments on five Simulink models tested against ten Signal Temporal Logic specifications. We compared our coverage-based testing approach to adaptive random testing, falsification testing, output diversity-based approaches, and testing using MathWorks’ Simulink Design Verifier. The results demonstrate that our coverage-based testing approach outperforms the conventional techniques regarding fault detection capability.

Cyber-Physical Systems (CPSs) face growing complexity, especially in safety-critical areas. Ensuring their correctness is vital to maintain full operational capacity, as undetected failures can be both costly and life-threatening. Therefore, advanced fault diagnosis procedures are essential for thorough CPS testing, enabling accurate fault detection, explanation, and rectification. This doctoral research contributes to the field by developing novel tools and techniques to enhance fault-based testing and diagnosis of CPSs. Our research focuses on testing of CPS dataflow models created in Simulink, validated against strict formal specifications. Our contributions include (i) an automated tool for systematic fault injection, (ii) a bio-inspired global optimization algorithm, (iii) a robust fault localization method, (iv) a novel approach to mutation testing for evaluating test suites against formal properties, and (v) a new coverage criterion tailored for CPS dataflow models. This comprehensive approach offers significant improvements over existing methods, ensuring thorough testing across various scenarios. We validate the effectiveness of our solutions using publicly available benchmarks from various domains. Our findings open new perspectives on CPS testing, laying the foundation for more robust CPSs

Mutation testing is an established software quality assurance technique for the assessment of test suites. While it is well-suited to estimate the general fault-revealing capability of a test suite, it is not practical and informative when the software under test must be validated against specific requirements. This is often the case for embedded software, where the software is typically validated against rigorously-specified safety properties. In such a scenario (i) a mutant is relevant only if it can impact the satisfaction of the tested properties, and (ii) a mutant is meaningfully-killed with respect to a property only if it causes the violation of that property. To address these limitations of mutation testing, we introduce property-based mutation testing, a method for assessing the capability of a test suite to exercise the software with respect to a given property. We evaluate our property-based mutation testing framework on Simulink models of safety-critical Cyber-Physical Systems (CPS) from the automotive and avionic domains and demonstrate how property-based mutation testing is more informative than regular mutation testing. These results open new perspectives in both mutation testing and test case generation of CPS.

Fault localization plays an important role in the design, verification and debugging of cyber-physical systems (CPS). Finding the exact location of a fault that triggered a failure in a CPS model is however a challenging task, due to the complex structure and data-flow nature of CPS models. In this paper, we propose a method that uses formal specifications and search-based testing to accurately localize faults. Given a CPS Simulink model, a formalized requirement used as a test oracle, and a test case that fails the formalized property, we develop a procedure that uses search-based testing to generate another test case that succeeds on the same formalized property. We then compare our two similar test cases with opposite verdicts to find the accurate location of the fault. We implement our approach and evaluate it on three case studies from automotive and avionic domains. We empirically compare our approach to a state-of-the-art fault localization technique and demonstrate that our procedure (1) is able to considerably narrow down the number of suspicious model variables and blocks compared to the previous work, and (2) remains robust to an increasing number of active faults in the underlying models.

We introduce FIM, an open-source toolkit for automated fault injection and mutant generation in Simulink models. FIM allows the injection of faults into specific parts, supporting common types of faults and mutation operators whose parameters can be customized to control the time of fault actuation and persistence. Additional flags allow the user to activate the individual fault blocks during testing to observe their effects on the overall system reliability. We provide insights into the design and architecture of FIM, and evaluate its performance on a case study from the avionics domain.