Fault Injection (FI) is a well-established method to assess the effect of failures within elements of a system under test. Where FI test cases can be executed automatically, such as in simulation-based or Hardware-in-the-Loop (HiL) FI, numerous test case generators (TCGs) have been proposed that aim to uncover more ‘critical’ test cases or to accomplish this using fewer resources. However, the evaluations of these approaches do not allow for direct comparisons: Experiments are often not reproducible, metrics are commonly specific to use cases, and key properties, such as test case distribution, are often not captured. Further, the authors are not aware of any suitable comparison frameworks. Hence, to support practitioners in selecting the most suitable FI TCG for their use case, test setup, and individual goal, this work introduces a set of use case-independent metrics derived from the ISO 26262 safety standard and identifies how these metrics can be applied and analyzed to capture decisive characteristics such as the distribution, criticality, and coverage of generated test cases. We incorporate these metrics and analyses in a start-to-finish methodology and provide their implementation as an open-source tool to effectively and reproducibly evaluate TCGs for automated Fl. The evaluation methodology is assessed in a case study with an industry-oriented cyber-physical system, demonstrating its ability to support practitioners in making an informed decision about the TCG providing the most ap-propriate balance of coverage and efficiency for their particular use case.

Depending on the autonomy level, safety assessment leads to different functional safety requirements for advanced driver-assistance systems and autonomous driving functions. To provide the necessary guarantees, technical safety requirements are derived that support the safety case by means of appropriate system architectures. These build on safety mechanisms: Technical solutions responsible for maintaining the intended functionality (fail-operational) or transition to a safe state in the presence of hardware and software faults (fail-safe). As the choice and implementation of such safety mechanisms are critical decisions with a high impact on the overall architecture, their early validation is crucial for an efficient engineering process. However, analytical safety analysis techniques applied to date support only coarse time models and do not provide explicit guidance for considering systemic real-time properties of closed-loop systems. Therefore, we propose a simulation-based fault injection framework to identify problematic emerging temporal behaviors such as instability. In contrast to existing solutions, we leverage the Functional Mock-up Interface (FMI) standard for black-box co-simulation to overcome intellectual property concerns in distributed automotive supply chains and to account for heterogeneous tool landscapes. By considering the allocation of software units to processing elements as well as the communication infrastructure, our contribution allows for the injection and propagation of faults affecting a vehicle’s software and its electrical/electronic (E/E) architecture, which is crucial for the assessment of safety mechanisms. Experimental results obtained by applying the approach to an industry-oriented use case indicate its validity and low overhead.