The application of radar sensors for motion estimation has recently been discussed in the research community. However, challenging environments such as garages and tunnels can lead to erroneous motion estimation results. Since the estimate serves as an input for trajectory control in automated driving functions, it can pose a safety hazard, e.g., if undesired acceleration is applied. This work presents a framework to address unsafe controller actions caused by motion estimation failures. A monitoring module continuously observes controller actions and feeds back critically evaluated measures to the motion estimation module. We propose three additional algorithms that adapt both the estimation module and its input data-namely, the filtered radar point cloud. First, a method that incorporates previous motion states into the point cloud filtering process. Second, a ridge regression algorithm that generates alternative estimates and compensates for erroneous conclusions arising from an unfavorably selected point set. Third, an adapted cluster selection approach that increases the number of true positive detection points. Experiments show that critical estimates can be identified in most cases, with a success rate of 97.7%. Additionally, we found that the estimation process becomes significantly more robust. In summary, this work emphasizes the importance of safe motion state measurement for the operation of automated vehicles. It introduces a method for determining the criticality of motion states and presents approaches to mitigate erroneous estimations.

Radar sensors are superior at measuring distances and velocities, even in adverse weather conditions. However, the noisiness of radar point clouds requires a filtering cascade to make these sensors usable for applications in automated driving. This filtering may cause hazardous situations if points on existing objects are removed by a filter. To serve as a perception component in fully automated driving systems, radar’s error rate needs to be reduced significantly. To achieve this goal, we advance the view that points should be filtered adaptively according to their presumable relevance for the current driving task. In this work, we present a real-world study of our radar point criticality estimation algorithm. In addition, we introduce methods for acting on critical points. While the Posterior method recalls points in critical regions, we demonstrate how the creation of clusters and tracks can be facilitated for critical radar points. Our methods are evaluated on a novel dataset including 92 critical scenes with pedestrians in a parking garage. We demonstrate that all methods significantly increase detection rates at both the cluster and track levels and enable earlier detection. However, utilizing every radar point inevitably leads to several false positives. Future work should investigate how to invalidate these points by additional perception sensors.

Adverse weather conditions, particularly rainfall, present substantial challenges to camera-based perception systems in Advanced Driver Assistance Systems (ADAS). Unlike human drivers, camera sensors are more vulnerable to visibility degradation caused by raindrops, which can impair essential functions such as object and lane detection. In this paper, we introduce Rainy-nuScenes, a novel extension of the widely-used nuScenes dataset, specifically designed for benchmarking water droplet detection and segmentation in automotive camera images. The dataset comprises 762 annotated images containing over 1,700 labeled water droplets, enabling a detailed analysis of their spatial distribution and geometric characteristics. We conduct a comparative study between Rainy-nuScenes and related datasets, including WoodScape, emphasizing key differences in droplet coverage, distribution patterns, and annotation strategies. Furthermore, we evaluate various camera defisheye techniques—such as linear and orthographic projections—in conjunction with U-Net [1] based convolutional neural networks (CNNs) trained on both Rainy-nuScenes and fisheye-derived WoodScape images [2]. Our experiments show that the orthographic defisheye approach significantly improves the robustness and generalization capabilities of segmentation models. Rainy-nuScenes serves as a comprehensive benchmark for advancing ADAS algorithms in adverse weather, contributing to the development of safer and more reliable autonomous systems. The data and code are available at: https://github.com/timdietereberhardt/rainynuscenes

In computer vision, image clarity is crucial, particularly when challenges like water droplets on camera lenses can significantly impair the accurate analysis of visual data. While existing methods mainly focus on small droplets, the impact of larger droplets has been largely overlooked. This paper introduces a novel approach that models water droplets using randomly generated points and Bézier curves to simulate their shape on the lens. Based on this geometric framework, we developed a classifier to distinguish between two visual scenarios within the droplet. For larger droplets, we use a heuristic method to simulate various lens block-ages. We evaluate this simulation framework using a real stereo dataset from [19] with clear and soiled images. Our method, evaluated using Mean Squared Error (MSE) and Structural Similarity Index Measure (SSIM), demonstrates superior performance in MSE, while also achieving com-petitive results in SSIM compared to existing techniques for generating realistic water droplets. Additionally, we applied this technique for data augmentation in object detection tasks using the YOLOv7 [25] model. The results show improved robustness, especially in challenging conditions where large droplets obstruct the lens.

For an autonomous vehicle to drive safely and efficiently, it is important to have a good understanding of the surrounding environment. Having early information about other road users can enable the vehicle to anticipate their behavior and take appropriate actions to avoid dangerous situations. Humans often use light reflections caused by oncoming vehicles at night to anticipate their appearance before they are directly visible. This problem of provident vehicle detection at night has already been studied in rural land road scenarios. However, urban scenarios are more complex, as the number of light sources is higher and the light reflections are more complex. In this paper, we therefore present the PVDN-urban dataset, which is the first dataset to study provident vehicle detection at night in urban scenarios. The dataset contains detailed annotations of light reflections caused by oncoming vehicles at night in urban scenarios. Also, it provides bounding box annotations for all vehicles, to make the dataset usable also for conventional vehicle detection tasks under low-light conditions. We provide an in-depth analysis of the dataset, an efficient annotation method for light reflections, as well as baseline results using state-of-the-art semantic segmentation models. With that, we provide the basis to further study provident vehicle detection at night also for complex urban scenarios. Furthermore, we provide a dataset for the development of algorithms for general vehicle detection at night in urban scenarios.

This paper presents a framework for odometry estimation in automotive application using six asynchronously operating millimeter wave radar sensors and a combination of gyroscope and accelerometer. Two different motion models are combined to estimate motion with three degrees of freedom. For this purpose, we propose a novel three-part radar filtering method for outlier detection: By analyzing uncertainties and system limits, sensor-specific outliers are detected and removed in the first filter. We introduce knowledge about the previous motion state by a status-quo-ante filter and hereby identify further false positive raw targets in the current measure which are not accessible from the previous state. Moreover, we suggest employing a downstream, resampling-based algorithm for additional outlier detection. Based on the filtered data, radar motion state estimation is performed by use of curve fitting methods. To fuse the radar odometry estimation with the acceleration and yaw rate measurements handling non-linearities, an Unscented Kalman Filter is used. The developed framework is evaluated with reference data in various scenarios. The results demonstrate that it accurately and robustly determines motion and position states even in radar-challenging scenes, such as environments with few radar targets or with heavy metal structures. Our method keeps up with common approaches such as wheel speed sensor odometry while outperforming it in terms of drift-impairment.

Simultaneous localization and mapping is a prevalent method for localization in automated parking applications. However, it requires to access an area for exploring purposes before the automated parking function can be completely applied. As this is inconvenient for parking functions, where often unknown areas are entered, our approach proposes a radar-based localization method primarily for applications inside of buildings which have a ground plan available. This ground plan contains the walls, pillars and parking lots of the building in a two-dimensional, bird’s eye view perspective. Based on the ground plan, a synthesized point cloud is generated to be matched with the filtered radar point cloud via a Normal Distributions Transform algorithm. The measurements generated hereby are fused with odometry measurements in a factor graph. This architecture is capable of processing independent, asynchronous incoming data in parallel and can easily be extended, e.g., by camera data. We outline our pipeline and show in experiments that it serves as a solid basis which competes with other state-of-the-art localization algorithms. Some drawbacks, e.g., the noisiness of the radar data in slow-speed or standstill situations, are discussed. Future work could incorporate camera data to further improve the robustness of this approach.

Detecting oncoming vehicles at night as early as possible is important for highly automated driving and Advanced-Driver-Assistance-Systems (ADAS). The sooner objects are detected, the earlier autonomous systems can take them into consideration to plan more anticipatory and safe actions. Previous work showed that on rural land roads at night, oncoming vehicles can already be detected before they are actually directly visible. This is done based on their emitted light. However, no work exists on covering the problem for more complex scenarios such as urban areas in cities. In this paper, we present a new approach to annotate light reflections in urban scenarios caused by oncoming vehicles at night before they are directly visible. We revisit design decisions in previous work for rural land road scenarios and find several improvements. We propose a pipeline which takes relatively cheap-to-get, yet highly subjective human Bounding-Box (BB) annotations, and automatically turns them into normally expensive-to-get, more objective binary masks. In our annotation experiment, we show that labeling light reflections is far more challenging and complex than conventional objects. Also, we show that our method can improve inter-annotator agreement and filter out annotator subjectivity. We train several State-Of-The-Art (SOTA) neural networks for semantic segmentation to demonstrate that our annotations can be used to detect light reflections from oncoming vehicles in urban scenarios before they are directly visible.

During the use of Advanced Driver Assistance Systems (ADAS), drivers can intervene in the active function and take back control due to various reasons. However, the specific reasons for driver-initiated takeovers in naturalistic driving are still not well understood. In order to get more information on the reasons behind these takeovers, a test group study was conducted. There, 17 participants used a predictive longitudinal driving function for their daily commutes and annotated the reasons for their takeovers during active function use. In this paper, the recorded takeovers are analyzed and the different reasons for them are highlighted. The results show that the reasons can be divided into three main categories. The most common category consists of takeovers which aim to adjust the behavior of the ADAS within its Operational Design Domain (ODD) in order to better match the drivers’ personal preferences. Other reasons include takeovers due to leaving the ADAS’s ODD and corrections of incorrect sensing state information. Using the questionnaire results of the test group study, it was found that the number and frequency of takeovers especially within the ADAS’s ODD have a significant negative impact on driver satisfaction. Therefore, the driver satisfaction with the ADAS could be increased by adapting its behavior to the drivers’ wishes and thereby lowering the number of takeovers within the ODD. The information contained in the takeover behavior of the drivers could be used as feedback for the ADAS. Finally, it is shown that there are considerable differences in the takeover behavior of different drivers, which shows a need for ADAS individualization.