For an autonomous vehicle to drive safely and efficiently, it is important to have a good understanding of the surrounding environment. Having early information about other road users can enable the vehicle to anticipate their behavior and take appropriate actions to avoid dangerous situations. Humans often use light reflections caused by oncoming vehicles at night to anticipate their appearance before they are directly visible. This problem of provident vehicle detection at night has already been studied in rural land road scenarios. However, urban scenarios are more complex, as the number of light sources is higher and the light reflections are more complex. In this paper, we therefore present the PVDN-urban dataset, which is the first dataset to study provident vehicle detection at night in urban scenarios. The dataset contains detailed annotations of light reflections caused by oncoming vehicles at night in urban scenarios. Also, it provides bounding box annotations for all vehicles, to make the dataset usable also for conventional vehicle detection tasks under low-light conditions. We provide an in-depth analysis of the dataset, an efficient annotation method for light reflections, as well as baseline results using state-of-the-art semantic segmentation models. With that, we provide the basis to further study provident vehicle detection at night also for complex urban scenarios. Furthermore, we provide a dataset for the development of algorithms for general vehicle detection at night in urban scenarios.

This paper presents a framework for odometry estimation in automotive application using six asynchronously operating millimeter wave radar sensors and a combination of gyroscope and accelerometer. Two different motion models are combined to estimate motion with three degrees of freedom. For this purpose, we propose a novel three-part radar filtering method for outlier detection: By analyzing uncertainties and system limits, sensor-specific outliers are detected and removed in the first filter. We introduce knowledge about the previous motion state by a status-quo-ante filter and hereby identify further false positive raw targets in the current measure which are not accessible from the previous state. Moreover, we suggest employing a downstream, resampling-based algorithm for additional outlier detection. Based on the filtered data, radar motion state estimation is performed by use of curve fitting methods. To fuse the radar odometry estimation with the acceleration and yaw rate measurements handling non-linearities, an Unscented Kalman Filter is used. The developed framework is evaluated with reference data in various scenarios. The results demonstrate that it accurately and robustly determines motion and position states even in radar-challenging scenes, such as environments with few radar targets or with heavy metal structures. Our method keeps up with common approaches such as wheel speed sensor odometry while outperforming it in terms of drift-impairment.

Simultaneous localization and mapping is a prevalent method for localization in automated parking applications. However, it requires to access an area for exploring purposes before the automated parking function can be completely applied. As this is inconvenient for parking functions, where often unknown areas are entered, our approach proposes a radar-based localization method primarily for applications inside of buildings which have a ground plan available. This ground plan contains the walls, pillars and parking lots of the building in a two-dimensional, bird’s eye view perspective. Based on the ground plan, a synthesized point cloud is generated to be matched with the filtered radar point cloud via a Normal Distributions Transform algorithm. The measurements generated hereby are fused with odometry measurements in a factor graph. This architecture is capable of processing independent, asynchronous incoming data in parallel and can easily be extended, e.g., by camera data. We outline our pipeline and show in experiments that it serves as a solid basis which competes with other state-of-the-art localization algorithms. Some drawbacks, e.g., the noisiness of the radar data in slow-speed or standstill situations, are discussed. Future work could incorporate camera data to further improve the robustness of this approach.

Provident detection of other road users at night has the potential for increasing road safety. For this purpose, humans intuitively use visual cues, such as light cones and light reflections emitted by other road users to be able to react to oncoming traffic at an early stage. Computer vision methods can imitate this behavior by predicting the appearance of vehicles based on light reflections caused by the vehicle’s headlights. Since current object detection algorithms are mainly based on detecting directly visible objects annotated via bounding boxes, the detection and annotation of light reflections without sharp boundaries is challenging. For this reason, the extensive open-source PVDN (Provident Vehicle Detection at Night) dataset was published that includes traffic scenarios at night with light reflections annotated via keypoints. In this paper, we explore a generic approach to annotate objects without clear boundaries, such as light reflections, by combining sparse keypoint annotations of humans with the concept of Boolean map saliency. With that, we create context-aware saliency maps that capture unsharp object boundaries, such as of light reflections. We show that this approach allows for an automated derivation of different object representations, such as bounding boxes, so that detection models can be trained and the problem of providently detecting vehicles at night can be tackled from a different perspective. Our approach makes it possible to derive bounding boxes with superior quality compared to previous approaches and to develop better object detection algorithms. With this paper, we provide a powerful method to study the problem of detecting objects with unsharp boundaries and, in particular, to investigate the detection of vehicles at night before they are actually visible.

Detecting oncoming vehicles at night as early as possible is important for highly automated driving and Advanced-Driver-Assistance-Systems (ADAS). The sooner objects are detected, the earlier autonomous systems can take them into consideration to plan more anticipatory and safe actions. Previous work showed that on rural land roads at night, oncoming vehicles can already be detected before they are actually directly visible. This is done based on their emitted light. However, no work exists on covering the problem for more complex scenarios such as urban areas in cities. In this paper, we present a new approach to annotate light reflections in urban scenarios caused by oncoming vehicles at night before they are directly visible. We revisit design decisions in previous work for rural land road scenarios and find several improvements. We propose a pipeline which takes relatively cheap-to-get, yet highly subjective human Bounding-Box (BB) annotations, and automatically turns them into normally expensive-to-get, more objective binary masks. In our annotation experiment, we show that labeling light reflections is far more challenging and complex than conventional objects. Also, we show that our method can improve inter-annotator agreement and filter out annotator subjectivity. We train several State-Of-The-Art (SOTA) neural networks for semantic segmentation to demonstrate that our annotations can be used to detect light reflections from oncoming vehicles in urban scenarios before they are directly visible.

During the use of Advanced Driver Assistance Systems (ADAS), drivers can intervene in the active function and take back control due to various reasons. However, the specific reasons for driver-initiated takeovers in naturalistic driving are still not well understood. In order to get more information on the reasons behind these takeovers, a test group study was conducted. There, 17 participants used a predictive longitudinal driving function for their daily commutes and annotated the reasons for their takeovers during active function use. In this paper, the recorded takeovers are analyzed and the different reasons for them are highlighted. The results show that the reasons can be divided into three main categories. The most common category consists of takeovers which aim to adjust the behavior of the ADAS within its Operational Design Domain (ODD) in order to better match the drivers’ personal preferences. Other reasons include takeovers due to leaving the ADAS’s ODD and corrections of incorrect sensing state information. Using the questionnaire results of the test group study, it was found that the number and frequency of takeovers especially within the ADAS’s ODD have a significant negative impact on driver satisfaction. Therefore, the driver satisfaction with the ADAS could be increased by adapting its behavior to the drivers’ wishes and thereby lowering the number of takeovers within the ODD. The information contained in the takeover behavior of the drivers could be used as feedback for the ADAS. Finally, it is shown that there are considerable differences in the takeover behavior of different drivers, which shows a need for ADAS individualization.

Driving at night is a challenging task for humans due to low-light conditions and often low concentration caused by drowsiness. Here, advanced driver assistance systems can come to the rescue and support the driver to increase comfort as well as safety. For that, the vehicle needs a sound and complete understanding of its environment, especially other road participants. The earlier this information is available, the more proactive decisions can be made by the system. To detect on-coming vehicles as soon as possible even before they are actually directly visible, the light reflections caused by their headlamps can be leveraged. Previous work showed that algorithms can perform this task on rural land roads. However, for more complex urban scenarios no approach exists so far. Yet, before starting the dataset and algorithm development for a computer vision system able to solve the task, a sound understanding of the problem is required. In a recent study we already showed that humans can anticipate oncoming vehicles based on their light reflections also in urban scenarios. Still, it remains unclear whether spatial information alone is sufficient to perform the task, or if the temporal context is required. This understanding is essential when designing labeling pipelines for a dataset, as well as algorithm development. Therefore, in this paper we perform a large experiment to evaluate the importance of temporal context for the human ability to anticipate oncoming vehicles at night in urban scenarios. We present participants scenes with different numbers of frames and measure their anticipation performance. We show that providing temporal context significantly increases the human detection accuracy as well as decision confidence. With this we provide additional insights into the task of anticipatory vehicle detection at night which can be taken into consideration when designing a dataset as well as algorithms.

When developing autonomous or automated driving functions, having a sound understanding of the environment is critical. The earlier and the more detailed the environment is recognized, the more precise the vehicle can plan future actions. One major component in the field of autonomous driving, therefore, is visual perception. Cameras deliver consecutive images of the world, and computer vision algorithms attempt to extract the relevant information, for example, detecting other road users such as vehicles. However, current automated driving functions share the limitation of relying on the object to be directly visible to be detected. On the other side, humans intuitively use other visual effects to draw assumptions about objects that are not yet directly visible. Such effects can be shadows during the day or light reflections during the night. Recent work has already shown the potential of using this information when driving at night on highways. Highways at night thereby are easy ground: The traffic volume is considerably low, and the surrounding environment is mainly dark (e. g., there are no street lamps causing other light sources or reflections). This paper takes the research on this topic a step further and brings light to the question to which extent this effect of detecting oncoming vehicles at night can also be used in cities. We present a comprehensive analysis of human behavior when detecting other road users at night in urban areas while driving. The data was gathered throughout a test group study in two medium-sized German cities under various weather and traffic conditions. We prove the importance of light reflections when driving through urban areas at night and provide a solid human benchmark to compare future perception algorithms’ performance.

Automated driving functions make mobility safer and more comfortable. The perception of these functions relies on sensors that capture the vehicle’s environment. A sound configuration of the sensor’s intrinsic and extrinsic parameters is crucial for a high perception performance. Simulation is a method of increasing importance for the evaluation and optimization of sensor sets since it enables fast and cost-effective procedures. This paper presents a method for evaluating and optimizing cameras during traffic scenarios in a simulation based on their raw sensor data quality. We present a framework using the CARLA simulator and evaluate the coverage and visibility of the surrounding objects during virtual traffic scenarios. For this purpose, we propose a general metric to measure the acquired information content of camera systems. We compare our metric to the performance of a state-of the-art object detector to investigate the ability to represent detection performance. The suitability of the metric for the optimization of intrinsic hardware parameters and extrinsic mounting parameters is explored. We show in our experiments that the metric correlates strongly with a benchmark perception function’s performance. Furthermore, we prove that sensor configurations optimized by our metric outperform a baseline sensor set in terms of object detection performance.

In current object detection, algorithms require the object to be directly visible in order to be detected. As humans, however, we intuitively use visual cues caused by the respective object to already make assumptions about its appearance. In the context of driving, such cues can be shadows during the day and often light reflections at night. In this paper, we study the problem of how to map this intuitive human behavior to computer vision algorithms to detect oncoming vehicles at night just from the light reflections they cause by their headlights. For that, we present an extensive open-source dataset containing 59 746 annotated grayscale images out of 346 different scenes in a rural environment at night. In these images, all oncoming vehicles, their corresponding light objects (e. g., headlamps), and their respective light reflections (e. g., light reflections on guardrails) are labeled. In this context, we discuss the characteristics of the dataset and the challenges in objectively describing visual cues such as light reflections. We provide different metrics for different ways to approach the task and report the results we achieved using state-of-the-art and custom object detection models as a first benchmark. With that, we want to bring attention to a new and so far neglected field in computer vision research, encourage more researchers to tackle the problem, and thereby further close the gap between human performance and computer vision systems.