Reliable detection of road users is critical to the safety of automated driving systems. While object detectors based on deep neural networks are widely used for this purpose, they remain susceptible to errors that could compromise safety. A promising strategy to mitigate these risks involves run-time perception monitoring mechanisms, commonly referred to in the literature as self-assessment or introspection. Current research in this area predominantly addresses anomaly detection, or monitoring camera-based 2D object detection, with insufficient focus on in-distribution errors and 3D object detection. Additionally, existing 2D studies often monitor activation patterns from the final layers of the network backbone, overlooking earlier activations that preserve higher spatial resolution. Yet, high-resolution early-layer activations can be valuable for detecting errors with sparse 3D point clouds. We also argue that not all objects in a scene should equally influence frame-level error detection, a factor often neglected in current methods. To address these gaps, we propose a novel self-assessment mechanism for 3D object detection that leverages activation patterns from multiple network layers. This mechanism employs spatial filtering to focus the model within an area of interest in the close vicinity of the ego vehicle. Additionally, it utilises an object filtering mechanism, which specifically targets the missed objects by excluding the points in those already detected. We evaluate our method using widely recognised object detectors and public datasets. Additionally, we demonstrate its robustness under domain shifts with real-world LiDAR data collected on motorways in diverse weather conditions. Results show the proposed mechanism provides 6% AUROC improvement over last-layer activation methods with spatial filtering on the NuScenes dataset. It also demonstrates a superior ability to transfer knowledge under domain shifts. Code is available at https://github.com/yatbazhakan/multi-layer-introspection .

Future automated vehicles are likely to rely on LiDAR sensors to perform automated functions. These LiDAR sensors have the potential to interfere with each other, with this mutual interference being represented by unwanted points in the point cloud. These points could affect the quality of perception of objects, such as potentially creating false objects or obscuring real objects. Hence, it is necessary to identify interference points and remove them from the point cloud, without removing points that belong to valid objects. This paper presents a novel LiDAR mutual interference mitigation algorithm that successfully mitigates interference and potentially can be performed in real time. The number of interference points when the victim and offending LiDARs were separated by 1m was reduced from over 1000 points to under 40 points. The proposed method performed better than the current state-of-the-art techniques on most metrics and was designed to ensure that consecutive points at the same range, such as those belonging to small objects at far distances, were not permanently removed from consecutive point clouds. These results ensure a removal of interference point without affecting the accuracy of perception tasks based on LiDAR data.

As highly automated vehicles (AVs) are deployed in various countries, mixed-autonomy traffic will become common and persist for the foreseeable future. In such environments, AVs must operate in ways which are both predictable and acceptable to human drivers, particularly in complex intersections where negotiation is crucial. However, how human drivers expect AVs to interact with them, particularly in scenarios where the right-of-way is ambiguous, remains unclear. In this research, we conducted a simulation-based video survey of UK drivers (N = 87), investigating their perception of aggressive and defensive AV driving styles under unclear right-of-way scenarios. The analysis indicates that a defensive driving style is generally preferred by human drivers, while an aggressive style can also be acceptable at lower-speed interaction zones. These findings provide empirical evidence for algorithm engineers seeking to design motion control and negotiation strategies that align with human expectations in mixed traffic.

Accurate velocity estimation of surrounding moving objects and their trajectories are critical elements of perception systems in Automated/Autonomous Vehicles (AVs) with a direct impact on their safety. These are non-trivial problems due to the diverse types and sizes of such objects and their dynamic and random behaviour. Recent point cloud based solutions often use Iterative Closest Point (ICP) techniques, which are known to have certain limitations. For example, their computational costs are high due to their iterative nature, and their estimation error often deteriorates as the relative velocities of the target objects increase ($>$2 m/sec). Motivated by such shortcomings, this paper first proposes a novel Detection and Tracking of Moving Objects (DATMO) for AVs based on an optical flow technique, which is proven to be computationally efficient and highly accurate for such problems. This is achieved by representing the driving scenario as a vector field and applying vector calculus theories to ensure spatiotemporal continuity. We also report the results of a comprehensive performance evaluation of the proposed DATMO technique, carried out in this study using synthetic and real-world data. The results of this study demonstrate the superiority of the proposed technique, compared to the DATMO techniques in the literature, in terms of estimation accuracy and processing time in a wide range of relative velocities of moving objects. Finally, we evaluate and discuss the sensitivity of the estimation error of the proposed DATMO technique to various system and environmental parameters, as well as the relative velocities of the moving objects.

Automated driving systems (ADSs) aim to improve the safety, efficiency and comfort of future vehicles. To achieve this, ADSs use sensors to collect raw data from their environment. This data is then processed by a perception subsystem to create semantic knowledge of the world around the vehicle. State-of-the-art ADSs’ perception systems often use deep neural networks for object detection and classification, thanks to their superior performance compared to classical computer vision techniques. However, deep neural network-based perception systems are susceptible to errors, e.g., failing to correctly detect other road users such as pedestrians. For a safety-critical system such as ADS, these errors can result in accidents leading to injury or even death to occupants and road users. Introspection of perception systems in ADS refers to detecting such perception errors to avoid system failures and accidents. Such safety mechanisms are crucial for ensuring the trustworthiness of ADSs. Motivated by the growing importance of the subject in the field of autonomous and automated vehicles, this paper provides a comprehensive review of the techniques that have been proposed in the literature as potential solutions for the introspection of perception errors in ADSs. We classify such techniques based on their main focus, e.g., on object detection, classification and localisation problems. Furthermore, this paper discusses the pros and cons of existing methods while identifying the research gaps and potential future research directions.

Trust in automation is crucial for the safe and appropriate adoption of automated driving technology. Current research methods to measure trust mainly rely on subjective scales, with several intrinsic limitations. This empirical experiment proposes a novel method to measure trust objectively, using functional near-infrared spectroscopy (fNIRS). Through manipulating participants’ expectations regarding driving automation credibility, we have induced and successfully measured opposing levels of trust in automation. Most notably, our results evidence two separate yet interrelated cortical mechanisms for trust and distrust. Trust is demonstrably linked to decreased monitoring and working memory, whereas distrust is event-related and strongly tied to affective (or emotional) mechanisms. This paper evidence that trust in automation and situation awareness are strongly interrelated during driving automation usage. Our findings are crucial for developing future driver state monitoring technology that mitigates the impact of inappropriate reliance, or over trust, in automated driving systems.

Automated and autonomous vehicles are often required to operate in complex road environments with potential hazards that may lead to hazardous events causing injury or even death. Therefore, a reliable autonomous hazardous event detection system is a key enabler for highly autonomous vehicles (e.g., Level 4 and 5 autonomous vehicles) to operate without human supervision for significant periods of time. One promising solution to the problem is the use of graph-based methods that are powerful tools for relational reasoning. Using graphs to organise heterogeneous knowledge about the operational environment, link scene entities (e.g., road users, static objects, traffic rules) and describe how they affect each other. Due to a growing interest and opportunity presented by graph-based methods for autonomous hazardous event detection, this paper provides a comprehensive review of the state-of-the-art graph-based methods that we categorise as rule-based, probabilistic, and machine learning-driven. Additionally, we present an in-depth overview of the available datasets to facilitate hazardous event training and evaluation metrics to assess model performance. In doing so, we aim to provide a thorough overview and insight into the key research opportunities and open challenges.

Highly automated driving will likely result in drivers being out-of-the-loop during specific scenarios and engaging in a wide range of non-driving related tasks. Manifesting in lower levels of risk perception to emerging events, and thus affect drivers’ availability to take-over manual control in safety-critical scenarios. In this empirical research, we measured drivers’ (N = 20) risk perception with cardiac and skin conductance indicators through a series of high-fidelity, simulated highly automated driving scenarios. By manipulating the presence of surrounding traffic and changing driving conditions as long-term risk modulators, and including a driving hazard event as a short-term risk modulator, we hypothesised that an increase in risk perception would induce greater physiological arousal. Our results demonstrate that heart rate variability features are superior at capturing arousal variations from these long-term, low to moderate risk scenarios. In contrast, skin conductance responses are more sensitive to rapidly evolving situations associated with moderate to high risk. Based on this research, future driver state monitoring systems should adopt multiple physiological measures to capture changes in the long and short term, modulation of risk perception. This will enable enhanced perception of driver readiness and improved availability to safely deal with take-over events when requested by an automated vehicle.

Highly Automated Driving technology will be facing major challenges before being pervasively integrated across production vehicles. One of them will be monitoring drivers’ state and determining whether they are ready to take over control under certain circumstances. Thus, we have explored their physiological responses and the effects on trust of different scenarios with varying traffic complexity in a driving simulator. Using a mixed repeated measures design, twenty-seven participants were divided in two reliability groups with opposite induced automation reliability expectations -low and high-. We hypothesized that expectations would modulate participants’ trust in automation, and consequently, their physiological responses across different scenarios. That is, increasing traffic complexity would also increase participants’ arousal, and this would be accentuated or mitigated by automation reliability expectations. Although reliability group differences could not be observed, our results show an increase of physiological activation within high complexity driving conditions (i.e., a mentally demanding non-driving related task and urban scenarios). In addition, we observed a modulation of trust in automation according to the group expectations delivered. These findings provide a background methodology from which further research in driver monitoring systems can benefit and be used to train machine learning methods to classify drivers’ state in changing scenarios. This would potentially help mitigate inappropriate take-overs, calibrate trust and increase users’ comfort and safety in future Highly Automated Vehicles.

This paper presents results from a series of focus groups, aimed at enhancing technical engineering system requirements, for a public transport system, encompassing a fleet of platooning low-speed autonomous vehicles (LSAV; aka pods) in urban areas. A critical review of the pods was conducted, as part of a series of technical workshops, to examine the key areas of the system that could affect users and other stakeholders, such as businesses and the public. These initial findings were used to inform a series of focus groups, aimed at identifying the public's views of multiple autonomous vehicles being deployed in a pedestrianised area that can join and form platoons. Analysis of findings from the focus groups suggests that while people view platooning public transport vehicles favourably as a passenger, they have some concerns from a pedestrian perspective. Thematic analysis was applied to these findings and a systematic approach was used to identify where subjective outputs could be formalised to inform requirements. Finally, a step-by-step requirements elicitation process is presented that illustrates the method used to convert qualitative user data to objective engineering requirements.