Virtual testing using synthetic data has become a cornerstone of autonomous vehicle (AV) safety assurance. Despite progress in improving visual realism through advanced simulators and generative AI, recent studies reveal that pixel-level fidelity alone does not ensure reliable transfer from simulation to the real world. What truly matters is whether the system-under-test (SUT) bases its decisions on consistent decision evidence in both real and simulated environments, not just whether images "look real" to humans. To this end this paper proposes a behavior-grounded fidelity measure by introducing Decisive Feature Fidelity (DFF), a new SUT-specific metric that extends the existing fidelity spectrum to capture mechanism parity, that is, agreement in the model-specific decisive evidence that drives the SUT's decisions across domains. DFF leverages explainable-AI methods to identify and compare the decisive features driving the SUT's outputs for matched real-synthetic pairs. We further propose estimators based on counterfactual explanations, along with a DFF-guided calibration scheme to enhance simulator fidelity. Experiments on 2126 matched KITTI-VirtualKITTI2 pairs demonstrate that DFF reveals discrepancies overlooked by conventional output-value fidelity. Furthermore, results show that DFF-guided calibration improves decisive-feature and input-level fidelity without sacrificing output value fidelity across diverse SUTs.

Large-scale scenario databases may contain hundreds of thousands of scenarios for the verification and validation (V&V) of autonomous vehicles (AV). Scenarios in the database are often labelled with semantic Operational Design Domain (ODD) tags (e.g., WeatherRainy, RoadTypeHighway and ActorTypeTruck) to be queried via exact tag matching. Such a scenario database design has two major limitations, i.e. combinatorial scenario generation inevitably leads to many redundant scenarios, and each ODD query matches only a small number of scenarios in the database (0.2% in our case study), rendering most of the database wealth wasted. We propose a novel scenario database design and the first ODD-based query-time scenario mutation framework to address the limitations. Our case study results show that the proposed framework has the potential to fully utilize all the database scenarios at query time while eliminating scenario redundancy in the database (in our case study, given the same ODD query, the number of final matched scenarios increased by 36 times, diversity increased by 99 times, and scenario database utilization rate increased from 0.2% to 36%).

This paper presents a novel solution to the demand of a realistic virtual test environment (VTE) for the development and safety assurance of Automated Driving Systems (ADSs). The current VTE offerings suffer limitations when it comes to creating a rich environment (from the Operational Design Domain perspective) based on the ASAM OpenDRIVE files (a formatted description of the scenery), and hence there is no automatic OpenDRIVE scenery generation available that resembles the richness required to thoroughly test an ADS in a virtual environment. Therefore, through the use of Unreal Engine, leveraging the power of esmini’s roadmanager and developing on a primitive OpenDRIVE plugin, a comprehensive scenery with complex lighting, dynamic environmental conditions and photoscanned meshes can be achieved. There is then a demonstration of how this is incorporated into an end-to-end testing framework, using just one user interface to take the user from the creation and retrieval of scenarios through to the execution.

Scenario-based testing for automated driving systems (ADS) is an industry norm for safety assurance. A scenario describes situations that an automated driving systems may encounter during its operation. To ensure accurate representation of real-world situations, including human behavior and system interactions, a formal language is essential. It ensures consistent testing across diverse scenarios and facilitates compatibility with simulation tools. However, while existing scenario languages excel in describing environmental and road structure aspects, they lack the same detail for road users and drivers. We have developed a methodology to identify and incorporate relevant human factors elements into scenario languages. Our methodology focuses on understanding diverse individuals and their interactions with ADS on the road, enabling their representation in scenarios. We offer practical examples to improve language representation of human elements and actions, in WMG-SDL Level-2 for logical scenarios and BSI Flex 1889 for abstract scenario descriptions. This methodology serves as a starting point for language designers to accurately represent all road users and their interactions with ADS.

Scenario-based verification and validation (V&V) has emerged as the predominant approach for the performance evaluation of automated driving systems (ADSs). Many scenario-generation methods have been proposed to search for critical scenarios, i.e. disengagement or traffic rule violations. However, the widely adopted binary (pass/fail) criterion suffers from two main limitations, i.e., the difficulty of locating root causes and the lack of statistical guarantee of testing sufficiency. Recently, new scenario engineering approaches focusing on the intelligence of ADSs enlightened a promising pathway via dynamic driving task decomposition and function atom constraints. However, none of the state-of-the-art scenario description languages support such approaches. To fill this gap and facilitate further research into this promising direction, in this work, we propose a generic architecture to extend the existing scenario description languages for the intelligence testing of ADSs. The case study with WMG SDL demonstrates the capability and flexibility of the proposed extension design in defining intelligence function constraints.

The impact of brake safety is particularly serious in Intelligent vehicles, and braking performance can be improved by introducing Vehicle to Vehicle (V2V) communications. This paper proposes a methodology for an optimisation analysis on vehicle transmitter power and channel delay following the IEEE 802.11p standard to determine the minimum required V2V communications distance for maintaining a safe connected brake in a rural Line-of-sight driving scenario. We have built up a methodology on simulating the maximum delay from Packet Error Rates to analyse under the worst situation. After being optimised for different vehicle initial speeds, the minimum required transmitter power for a smooth and safe brake action is determined.

Automated Driving Systems (ADSs), like human drivers, must be compliant with the rules of the road. However, current rules of the road are not well defined. They use inconsistent and ambiguous language. As a result, they are not sufficiently formal for machine interpretability, a necessity for applications of verification and validation (V&V) of ADSs. Rules must be defined in a way that make them usable to a variety of stakeholders. While first-order and temporal logic forms of rules of the road are needed for monitoring and verification during simulation and testing, a structured natural language for these rules is necessary for consistent definition. They must also adhering to standard vocabulary taxonomies of Operational Design Domain (ODD) and behaviour. This paper contributes a structured natural language based on formal logic, that allows rules of the road to be defined in a natural, yet precise manner, using concepts of ODD and behaviour, making them usable in the V&V of ADSs. We evaluate the effectiveness of the language on a selection of rules from the Vienna Convention on Road Traffic and the UK Highway Code.

This paper presents a scenario-based safety assurance process for Automated Driving Systems (ADSs) combined with the Vehicle-to-Everything (V2X) connectivity aspect, system with connectivity capability is benchmarked with system without such capabilities. In addition, a novel approach to V2X modelling is introduced and implemented to obtain the required configuration of the V2X parameters for individual system, such modelling approach ensures that the V2X is effective within a system during testing by using a distance-based V2X parameter that correlates to the speed of the system. Such parameter is then used as the configuration of the V2X model when carrying out the ADS and V2X combined test. Using a pedestrian crossing scenario, a reduction in failure cases is demonstrated when combining the V2X and the ADS. Therefore, a synchronised approach of vehicle, sensor and communication sub-system can improve the overall safety function of the system.

The move towards connected and autonomous vehicles (CAVs) has gained a strong focus in recent years due to the many benefits they provide. While the autonomous aspect has seen substantial advancement in both the development and testing methodologies, the connected aspect has lagged behind, especially in the verification and validation (V&V) discussions. Integrating connectivity into the development and testing framework for CAVs is a necessity for ensuring the early deployment of cooperative driving systems. A key element within such a framework is a test scenario, which represents a set of scenery, environmental conditions, and dynamic conditions, that a system needs to be tested in. However, the connectivity element is not present in any of the current state of the art scenario description languages (SDLs) that are publicly available. This leaves a gap within the CAV development ecosystem. To accommodate for, and accelerate the development of, connected vehicle systems and their verification and validation methods, this paper proposes a novel V2X extension to the previously published two-level abstraction SDL. The extension enables communications between vehicles, infrastructures, and further additional entities to be specified as part of the scenario and be subsequently tested in virtual testing or real-world testing. Eight new V2X attributes have been added to the SDL. An example set of syntax and semantic definitions are presented in this paper targeting two different abstraction levels – level 1 aims at the abstract scenario level for non-technical end-users such as regulators, and level 2 aims at the logical and concrete scenario level for end-users such as simulation test engineers.

One of the main challenges for the introduction of Automated Driving Systems (ADSs) is their verification and validation (V&V). Simulation based testing has been widely accepted as an essential aspect of the ADS V&V processes. Simulations are especially useful when exposing the ADS to challenging driving scenarios, as they offer a safe and efficient alternative to real world testing. It is thus suggested that evidence for the safety case for an ADS will include results from both simulation and real-world testing. However, for simulation results to be trusted as part of the safety case of an ADS for its safety assurance, it is essential to prove that the simulation results are representative of the real world, thus validating the simulation platform itself. In this paper, we propose a novel methodology for validating the simulation environments focusing on comparing point cloud data from real LiDAR sensor and a simulated LiDAR sensor model. A 3D object dissimilarity metric is proposed to compare between the two maps (real and simulated), to quantify how accurate the simulation is. This metric is tested on collected LiDAR point cloud data and the simulated point cloud generated in the simulated environment.

For Automated Driving Systems (ADSs), vehicle safety and functional correctness are assessed against scenarios the ADS would encounter - within or outside its operational design domain (ODD). A scenario specifies conditions and events that an ADS is expected to respond to when deployed. Scenario specifications underpin the verification and validation (V&V) life-cycle, and are used by a diverse set of stakeholders - from engineers to regulators. Due to the diversity in stakeholder expertise, scenarios must be available at different levels of detail. Further, the chance of writing syntactically or semantically incorrect scenarios is high. Due to the high reliance of V&V on scenarios, their correctness is crucial. Therefore, present-day scenario-description languages (SDLs) need to be supported by technologies to help authors compose scenarios and provide a mechanism for easy translation into executable forms for virtual or real-life testing. This paper addresses these issues by building on the existing two-level abstraction WMG-SDL in the following ways, (1) introducing a human-readable, natural language SDL, replacing the former Level-1 SDL, and complementing the more detailed Level-2 SDL, which is now syntax aligned with the ODD Taxonomy defined in ISO 34503, and (2) providing a tool consisting of a parser and a validator to assist writing syntactically and semantically correct scenarios. Our tool may be used within a graphical scenario editing interface or on the command-line. Further, for developers, an object-oriented interface for parsed scenarios enables further development and integration with off-the-shelf ADS simulation and language tools. The tools and technologies described in this paper are to be made open-source.

Identification of test scenarios for Automated Driving Systems (ADSs) remains a key challenge for the Verification & Validation of ADSs. Various approaches including data based approaches and knowledge based approaches have been proposed for scenario generation. Identifying the conditions that lead to high severity traffic accidents can help us not only identify test scenarios for ADSs, but also implement measures to save lives and infrastructure resources. Taking a data based approach, in this paper, we introduce a novel accident data analysis method for generating test scenarios where we analyze UK's Stats19 accident data to identify trends in high severity accidents for test scenario generation. This paper first focuses on the severity of the accidents with the goal of relating it to static and time-dependent internal and external factors in a comprehensive way taking into account Operational Design Domain (ODD) properties, e.g. road, environmental conditions, and vehicle properties and driver characteristics. For this purpose, the paper utilizes a data grouping strategy (coarse-graining) and builds a logistic regression approach, derived from conventional regression models, in which emerging features become more pronounced, while uninteresting features and noise weaken. The approach makes the relationship between the factors and outcome variable more visible and hence well suited for the severity analysis. The method shows superior performance as compared to ordinary logistic models measured by goodness of fit and accounting for model variance$(R^{2}=0.05$for the ordinary model,$R^{2}=0.85$for the current model). The model is then used to solve the inverse problem of constructing high-risk pre-crash conditions as test scenarios for simulation based testing of ADSs.

The development of Automated Driving Systems (ADSs) is driven by the many benefits they offer. However, the complexities associated with ADSs and their interactions with the environment pose challenges for their safety assurance. A key aspect during its development process is knowing the capabilities, limitations, and being able to convey them in a clear manner for various types of stakeholders. The Operational Design Domain (ODD) concept was introduced to define the operating boundaries where a system can operate safely. It is therefore a key element for the safety assurance of ADSs. Efforts have been made to define the scope and the content an ODD for ADSs should cover, however there remains the need for a common, exchangeable, executable, and human-readable format for the description. This paper presents a language for the description of the ODD of ADSs, in a textual format that leans on natural language influence. Such format is intended to be both human and machine-readable and would be relevant to end users such as regulators and systems designers. The two-level abstraction approach – a structured natural language representation and a formal representation (covered across two papers) -- has been developed to have the ability to describe complex ODD conditionalities and utilize a well-defined domain ontology to achieve rich semantics. It is aimed to support ODD related activities throughout the development cycle of ADSs (specification as well as verification and validation), while covering a diverse range of stakeholders.

A formal representation for the Operational Design Domain (ODD) of Automated Driving Systems (ADSs) is presented in this paper. An ODD specification determines for every situation whether it is included or excluded from the ODD. We focus on methods to provide unambiguous specification in a programmatic format which are simultaneously machine and human readable. We present a logical framework with intuitive and well-defined semantics which directly supports safety engineering process through specification stage to defining uncertainty and acceptable risk. Its rich and diverse feature set include 1) parsimonious permissive and restrictive constraints, 2) the ability to import OWL ontologies, 3) ability to bind to non-uniform complex variable length structures within situation data, and 4) ability for components to control the scope of usage by integrators. The presentation of syntax and formal semantics is illustrated with example demonstrating key concepts and language capabilities. \n

Latest generations of automobiles are gradually being equipped with technologies that have increasing automation, a trend which had led to increase in the system complexity as well as increased human-automation interactions. Failures in such complex human-automation interactions increasingly occur due to the mismatch between what operators know about the system and what the designers expect operators to know. Causes of road accidents also change due to role shift of drivers from controlling the vehicle to monitoring the in-vehicle controllers. Failures in such complex systems involving human-automation interactions increasingly occur due to the emergent behaviours from the interactions, and are less likely due to reliability of individual components. Traditional safety analysis methods fall short in identifying such emergent failures. This paper focuses on using a systems thinking inspired safety analysis method called System Theoretic Process Analysis (STPA) to identify potential failures. The analysis focuses on a SAE Level-4 Vehicle that is in the development phase, and is controlled partially by a safety driver and its built-in Autonomous Driving System (ADS). The analysis yields that while increase in complexity does increase system functionality, it also brings a challenge to evaluate the safety of the system and potentially causes incorrect human-automation interactions, leading to an accident. After the possible inadequate driver-vehicle interactions are identified by STPA, corresponding requirements were then proposed in order to avoid the unsafe behaviour and thus preventing the hazards.

The complexities associated with Automated Driving Systems (ADSs) and their interaction with the environment pose a challenge for their safety evaluation. Number of miles driven has been suggested as one of the metrics to demonstrate technological maturity. However, the experiences or the scenarios encountered by the ADSs is a more meaningful metric, and has led to a shift to scenario-based testing approach in the automotive industry and research community. Variety of scenario generation techniques have been advocated, including real-world data analysis, accident data analysis and via systems hazard analysis. While scenario generation can be done via these methods, there is a need for a scenario description language format which enables the exchange of scenarios between diverse stakeholders (as part of the systems engineering lifecycle) with varied usage requirements. In this paper, we propose a two-level abstraction approach to scenario description language (SDL) - SDL level 1 and SDL level 2. SDL level 1 is a textual description of the scenario at a higher abstraction level to be used by regulators or system engineers. SDL level 2 is a formal machine-readable language which is ingested by testing platform e.g. simulation or test track. One can transform a scenario in SDL level 1 into SDL level 2 by adding more details or from SDL level 2 to SDL level 1 by abstracting.

Recently there has been a growth in the incorporation of autonomous features within vehicles. From being perceived as a comfort feature, autonomous features in vehicles have now become a safety feature which are foreseen to reduce accidents. This has led to a new trend within the automotive industry of focussing on autonomous features for driver safety, which might ultimately lead to fully autonomous vehicles. Considering the fact that most of the accidents on UK roads occur due to driver error, driver-less vehicles would prove to be a benefit. However with automation, an even greater challenge of system validation in all scenarios needs to be addressed. For this, various methods of validation have been developed by different research organizations and manufacturers, but a standardized process still evades the industry. Some of the existing methods have been discussed in this paper to critically compare their quality of results and ease of execution. Subsequently, a new test platform has been proposed using the 3xD driving simulator which encompasses most requirements of a general testing method. A standardized process which would benefit the industry both in terms of reducing costs of having varied processes, and by increasing customer confidence can be developed using a non-invasive platform like the 3xD driving simulator. The novelty of the 3xD simulator is the ability to drive-in any vehicle (production/prototype) and develop testing methodologies in an immersive wireless environment.

This paper gives an overview of the new Drive-in Driver-in-Loop simulator at WMG, University of Warwick, UK, which has been conceptualized to serve as a standard platform for virtual verification and validation of autonomous features. This front loading approach is key to the development of some of the upcoming technologies which have been lagging behind in terms of mass acceptance due to lack of proper simulation environment for testing. One of the key areas for the simulator is the study of driver acceptance of autonomous features in cars. Additionally, the simulator would help in the development of a validation methodology for autonomous systems keeping in mind the industry safety regulations and standards. The drive-in component on this scale adds to the novelty of the simulator, as it's a first of its kind. This enhances the challenge of making the communication interfaces of the simulator general, in order to communicate with any vehicle driving into the simulator. In order to achieve this, emphasis was laid on software architecture to enable modularity and re-configuration. A brief about various applications of the simulator has been provided in this paper.