The application of radar sensors for motion estimation has recently been discussed in the research community. However, challenging environments such as garages and tunnels can lead to erroneous motion estimation results. Since the estimate serves as an input for trajectory control in automated driving functions, it can pose a safety hazard, e.g., if undesired acceleration is applied. This work presents a framework to address unsafe controller actions caused by motion estimation failures. A monitoring module continuously observes controller actions and feeds back critically evaluated measures to the motion estimation module. We propose three additional algorithms that adapt both the estimation module and its input data-namely, the filtered radar point cloud. First, a method that incorporates previous motion states into the point cloud filtering process. Second, a ridge regression algorithm that generates alternative estimates and compensates for erroneous conclusions arising from an unfavorably selected point set. Third, an adapted cluster selection approach that increases the number of true positive detection points. Experiments show that critical estimates can be identified in most cases, with a success rate of 97.7%. Additionally, we found that the estimation process becomes significantly more robust. In summary, this work emphasizes the importance of safe motion state measurement for the operation of automated vehicles. It introduces a method for determining the criticality of motion states and presents approaches to mitigate erroneous estimations.

Radar sensors are superior at measuring distances and velocities, even in adverse weather conditions. However, the noisiness of radar point clouds requires a filtering cascade to make these sensors usable for applications in automated driving. This filtering may cause hazardous situations if points on existing objects are removed by a filter. To serve as a perception component in fully automated driving systems, radar’s error rate needs to be reduced significantly. To achieve this goal, we advance the view that points should be filtered adaptively according to their presumable relevance for the current driving task. In this work, we present a real-world study of our radar point criticality estimation algorithm. In addition, we introduce methods for acting on critical points. While the Posterior method recalls points in critical regions, we demonstrate how the creation of clusters and tracks can be facilitated for critical radar points. Our methods are evaluated on a novel dataset including 92 critical scenes with pedestrians in a parking garage. We demonstrate that all methods significantly increase detection rates at both the cluster and track levels and enable earlier detection. However, utilizing every radar point inevitably leads to several false positives. Future work should investigate how to invalidate these points by additional perception sensors.

Adverse weather conditions, particularly rainfall, present substantial challenges to camera-based perception systems in Advanced Driver Assistance Systems (ADAS). Unlike human drivers, camera sensors are more vulnerable to visibility degradation caused by raindrops, which can impair essential functions such as object and lane detection. In this paper, we introduce Rainy-nuScenes, a novel extension of the widely-used nuScenes dataset, specifically designed for benchmarking water droplet detection and segmentation in automotive camera images. The dataset comprises 762 annotated images containing over 1,700 labeled water droplets, enabling a detailed analysis of their spatial distribution and geometric characteristics. We conduct a comparative study between Rainy-nuScenes and related datasets, including WoodScape, emphasizing key differences in droplet coverage, distribution patterns, and annotation strategies. Furthermore, we evaluate various camera defisheye techniques—such as linear and orthographic projections—in conjunction with U-Net [1] based convolutional neural networks (CNNs) trained on both Rainy-nuScenes and fisheye-derived WoodScape images [2]. Our experiments show that the orthographic defisheye approach significantly improves the robustness and generalization capabilities of segmentation models. Rainy-nuScenes serves as a comprehensive benchmark for advancing ADAS algorithms in adverse weather, contributing to the development of safer and more reliable autonomous systems. The data and code are available at: https://github.com/timdietereberhardt/rainynuscenes

In computer vision, image clarity is crucial, particularly when challenges like water droplets on camera lenses can significantly impair the accurate analysis of visual data. While existing methods mainly focus on small droplets, the impact of larger droplets has been largely overlooked. This paper introduces a novel approach that models water droplets using randomly generated points and Bézier curves to simulate their shape on the lens. Based on this geometric framework, we developed a classifier to distinguish between two visual scenarios within the droplet. For larger droplets, we use a heuristic method to simulate various lens block-ages. We evaluate this simulation framework using a real stereo dataset from [19] with clear and soiled images. Our method, evaluated using Mean Squared Error (MSE) and Structural Similarity Index Measure (SSIM), demonstrates superior performance in MSE, while also achieving com-petitive results in SSIM compared to existing techniques for generating realistic water droplets. Additionally, we applied this technique for data augmentation in object detection tasks using the YOLOv7 [25] model. The results show improved robustness, especially in challenging conditions where large droplets obstruct the lens.

For an autonomous vehicle to drive safely and efficiently, it is important to have a good understanding of the surrounding environment. Having early information about other road users can enable the vehicle to anticipate their behavior and take appropriate actions to avoid dangerous situations. Humans often use light reflections caused by oncoming vehicles at night to anticipate their appearance before they are directly visible. This problem of provident vehicle detection at night has already been studied in rural land road scenarios. However, urban scenarios are more complex, as the number of light sources is higher and the light reflections are more complex. In this paper, we therefore present the PVDN-urban dataset, which is the first dataset to study provident vehicle detection at night in urban scenarios. The dataset contains detailed annotations of light reflections caused by oncoming vehicles at night in urban scenarios. Also, it provides bounding box annotations for all vehicles, to make the dataset usable also for conventional vehicle detection tasks under low-light conditions. We provide an in-depth analysis of the dataset, an efficient annotation method for light reflections, as well as baseline results using state-of-the-art semantic segmentation models. With that, we provide the basis to further study provident vehicle detection at night also for complex urban scenarios. Furthermore, we provide a dataset for the development of algorithms for general vehicle detection at night in urban scenarios.

This paper presents a framework for odometry estimation in automotive application using six asynchronously operating millimeter wave radar sensors and a combination of gyroscope and accelerometer. Two different motion models are combined to estimate motion with three degrees of freedom. For this purpose, we propose a novel three-part radar filtering method for outlier detection: By analyzing uncertainties and system limits, sensor-specific outliers are detected and removed in the first filter. We introduce knowledge about the previous motion state by a status-quo-ante filter and hereby identify further false positive raw targets in the current measure which are not accessible from the previous state. Moreover, we suggest employing a downstream, resampling-based algorithm for additional outlier detection. Based on the filtered data, radar motion state estimation is performed by use of curve fitting methods. To fuse the radar odometry estimation with the acceleration and yaw rate measurements handling non-linearities, an Unscented Kalman Filter is used. The developed framework is evaluated with reference data in various scenarios. The results demonstrate that it accurately and robustly determines motion and position states even in radar-challenging scenes, such as environments with few radar targets or with heavy metal structures. Our method keeps up with common approaches such as wheel speed sensor odometry while outperforming it in terms of drift-impairment.

Simultaneous localization and mapping is a prevalent method for localization in automated parking applications. However, it requires to access an area for exploring purposes before the automated parking function can be completely applied. As this is inconvenient for parking functions, where often unknown areas are entered, our approach proposes a radar-based localization method primarily for applications inside of buildings which have a ground plan available. This ground plan contains the walls, pillars and parking lots of the building in a two-dimensional, bird’s eye view perspective. Based on the ground plan, a synthesized point cloud is generated to be matched with the filtered radar point cloud via a Normal Distributions Transform algorithm. The measurements generated hereby are fused with odometry measurements in a factor graph. This architecture is capable of processing independent, asynchronous incoming data in parallel and can easily be extended, e.g., by camera data. We outline our pipeline and show in experiments that it serves as a solid basis which competes with other state-of-the-art localization algorithms. Some drawbacks, e.g., the noisiness of the radar data in slow-speed or standstill situations, are discussed. Future work could incorporate camera data to further improve the robustness of this approach.

Provident detection of other road users at night has the potential for increasing road safety. For this purpose, humans intuitively use visual cues, such as light cones and light reflections emitted by other road users to be able to react to oncoming traffic at an early stage. Computer vision methods can imitate this behavior by predicting the appearance of vehicles based on light reflections caused by the vehicle’s headlights. Since current object detection algorithms are mainly based on detecting directly visible objects annotated via bounding boxes, the detection and annotation of light reflections without sharp boundaries is challenging. For this reason, the extensive open-source PVDN (Provident Vehicle Detection at Night) dataset was published that includes traffic scenarios at night with light reflections annotated via keypoints. In this paper, we explore a generic approach to annotate objects without clear boundaries, such as light reflections, by combining sparse keypoint annotations of humans with the concept of Boolean map saliency. With that, we create context-aware saliency maps that capture unsharp object boundaries, such as of light reflections. We show that this approach allows for an automated derivation of different object representations, such as bounding boxes, so that detection models can be trained and the problem of providently detecting vehicles at night can be tackled from a different perspective. Our approach makes it possible to derive bounding boxes with superior quality compared to previous approaches and to develop better object detection algorithms. With this paper, we provide a powerful method to study the problem of detecting objects with unsharp boundaries and, in particular, to investigate the detection of vehicles at night before they are actually visible.

Detecting oncoming vehicles at night as early as possible is important for highly automated driving and Advanced-Driver-Assistance-Systems (ADAS). The sooner objects are detected, the earlier autonomous systems can take them into consideration to plan more anticipatory and safe actions. Previous work showed that on rural land roads at night, oncoming vehicles can already be detected before they are actually directly visible. This is done based on their emitted light. However, no work exists on covering the problem for more complex scenarios such as urban areas in cities. In this paper, we present a new approach to annotate light reflections in urban scenarios caused by oncoming vehicles at night before they are directly visible. We revisit design decisions in previous work for rural land road scenarios and find several improvements. We propose a pipeline which takes relatively cheap-to-get, yet highly subjective human Bounding-Box (BB) annotations, and automatically turns them into normally expensive-to-get, more objective binary masks. In our annotation experiment, we show that labeling light reflections is far more challenging and complex than conventional objects. Also, we show that our method can improve inter-annotator agreement and filter out annotator subjectivity. We train several State-Of-The-Art (SOTA) neural networks for semantic segmentation to demonstrate that our annotations can be used to detect light reflections from oncoming vehicles in urban scenarios before they are directly visible.

During the use of Advanced Driver Assistance Systems (ADAS), drivers can intervene in the active function and take back control due to various reasons. However, the specific reasons for driver-initiated takeovers in naturalistic driving are still not well understood. In order to get more information on the reasons behind these takeovers, a test group study was conducted. There, 17 participants used a predictive longitudinal driving function for their daily commutes and annotated the reasons for their takeovers during active function use. In this paper, the recorded takeovers are analyzed and the different reasons for them are highlighted. The results show that the reasons can be divided into three main categories. The most common category consists of takeovers which aim to adjust the behavior of the ADAS within its Operational Design Domain (ODD) in order to better match the drivers’ personal preferences. Other reasons include takeovers due to leaving the ADAS’s ODD and corrections of incorrect sensing state information. Using the questionnaire results of the test group study, it was found that the number and frequency of takeovers especially within the ADAS’s ODD have a significant negative impact on driver satisfaction. Therefore, the driver satisfaction with the ADAS could be increased by adapting its behavior to the drivers’ wishes and thereby lowering the number of takeovers within the ODD. The information contained in the takeover behavior of the drivers could be used as feedback for the ADAS. Finally, it is shown that there are considerable differences in the takeover behavior of different drivers, which shows a need for ADAS individualization.

Driving at night is a challenging task for humans due to low-light conditions and often low concentration caused by drowsiness. Here, advanced driver assistance systems can come to the rescue and support the driver to increase comfort as well as safety. For that, the vehicle needs a sound and complete understanding of its environment, especially other road participants. The earlier this information is available, the more proactive decisions can be made by the system. To detect on-coming vehicles as soon as possible even before they are actually directly visible, the light reflections caused by their headlamps can be leveraged. Previous work showed that algorithms can perform this task on rural land roads. However, for more complex urban scenarios no approach exists so far. Yet, before starting the dataset and algorithm development for a computer vision system able to solve the task, a sound understanding of the problem is required. In a recent study we already showed that humans can anticipate oncoming vehicles based on their light reflections also in urban scenarios. Still, it remains unclear whether spatial information alone is sufficient to perform the task, or if the temporal context is required. This understanding is essential when designing labeling pipelines for a dataset, as well as algorithm development. Therefore, in this paper we perform a large experiment to evaluate the importance of temporal context for the human ability to anticipate oncoming vehicles at night in urban scenarios. We present participants scenes with different numbers of frames and measure their anticipation performance. We show that providing temporal context significantly increases the human detection accuracy as well as decision confidence. With this we provide additional insights into the task of anticipatory vehicle detection at night which can be taken into consideration when designing a dataset as well as algorithms.

Automated driving functions make mobility safer and more comfortable. The perception of these functions relies on sensors that capture the vehicle’s environment. A sound configuration of the sensor’s intrinsic and extrinsic parameters is crucial for a high perception performance. Simulation is a method of increasing importance for the evaluation and optimization of sensor sets since it enables fast and cost-effective procedures. This paper presents a method for evaluating and optimizing cameras during traffic scenarios in a simulation based on their raw sensor data quality. We present a framework using the CARLA simulator and evaluate the coverage and visibility of the surrounding objects during virtual traffic scenarios. For this purpose, we propose a general metric to measure the acquired information content of camera systems. We compare our metric to the performance of a state-of the-art object detector to investigate the ability to represent detection performance. The suitability of the metric for the optimization of intrinsic hardware parameters and extrinsic mounting parameters is explored. We show in our experiments that the metric correlates strongly with a benchmark perception function’s performance. Furthermore, we prove that sensor configurations optimized by our metric outperform a baseline sensor set in terms of object detection performance.