Automated vehicles and logistics robots must often position themselves in narrow environments with high precision in front of a specific target, such as a package or their charging station. Often, these docking scenarios are solved in two steps: path following and rough positioning followed by a high-precision motion planning algorithm. This can generate suboptimal trajectories caused by bad positioning in the first phase and, therefore, prolong the time it takes to reach the goal. In this work, we propose a unified approach, which is based on a Model Predictive Control (MPC) that unifies the advantages of Model Predictive Contouring Control (MPCC) with a Cartesian MPC to reach a specific goal pose. The paper's main contributions are the adaption of the dynamic weight allocation method to reach path ends and goal poses inside driving corridors, and the development of the so-called dynamic objective MPC. The latter is an improvement of the dynamic weight allocation method, which can inherently switch state-dependent from an MPCC to a Cartesian MPC to solve the path-following problem and the high-precision positioning tasks independently of the location of the goal pose seamlessly by one algorithm. This leads to foresighted, feasible, and safe motion plans, which can decrease the mission time and result in smoother trajectories.

Monitoring and self-assessment of tracking algorithms are essential in modern automated driving systems. However, the further use of this self-assessment information is another growing and not thoroughly studied area of research. One option is to adapt the parameters configured in the tracking algorithm online to obtain better and more robust tracking results directly. The paper proposes a novel overall concept and framework for adaptive Kalman filtering using subjective logic. Based on a self-assessment method, we present multiple variants of adaptive strategies to adapt the noise assumptions online for Kalman filtering. This paper focuses mainly on adaptation procedures for multi-sensor Kalman filters. The proposed method is evaluated in various experiments and compared with state-of-the-art adaptive Kalman filters.

3D object detection models that exploit both LiDAR and camera sensor features are top performers in large-scale autonomous driving benchmarks. A transformer is a popular network architecture used for this task, in which so-called object queries act as candidate objects. Initializing these object queries based on current sensor inputs is a common practice. For this, existing methods strongly rely on LiDAR data however, and do not fully exploit image features. Besides, they introduce significant latency. To overcome these limitations we propose EfficientQ3M, an efficient, modular, and multimodal solution for object query initialization for transformer-based 3D object detection models. The proposed initialization method is combined with a "modality-balanced" transformer decoder where the queries can access all sensor modalities throughout the decoder. In experiments, we outperform the state of the art in transformer-based LiDAR object detection on the competitive nuScenes benchmark and showcase the benefits of input-dependent multimodal query initialization, while being more efficient than the available alternatives for LiDAR-camera initialization. The proposed method can be applied with any combination of sensor modalities as input, demonstrating its modularity.

In automated driving, the safety and robustness of the overall system are among the most important key challenges today. To tackle these safety and robustness challenges, the monitoring and self-assessment of all modules in the automated system is necessary. Tracking surrounding objects as part of the environmental perception is a key module in automated systems. Thus, this work presents a novel overall concept and framework for self-assessment in multi-object tracking based on the subjective logic theory. The self-assessment concept is comprehensively discussed and evaluated by simulations and real-world data of the KITTI dataset, showing the relevance of this proposed method.

LiDAR-based 3D object detection has become an essential part of automated driving due to its ability to localize and classify objects precisely in 3D. However, object detectors face a critical challenge when dealing with unknown foreground objects, particularly those that were not present in their original training data. These out-of-distribution (OOD) objects can lead to misclassifications, posing a significant risk to the safety and reliability of automated vehicles. Currently, LiDAR-based OOD object detection has not been well studied. We address this problem by generating synthetic training data for OOD objects by perturbing known object categories. Our idea is that these synthetic OOD objects produce different responses in the feature map of an object detector compared to in-distribution (ID) objects. We then extract features using a pre-trained and fixed object detector and train a simple multilayer perceptron (MLP) to classify each detection as either ID or OOD. In addition, we propose a new evaluation protocol that allows the use of existing datasets without modifying the point cloud, ensuring a more authentic evaluation of real-world scenarios. The effectiveness of our method is validated through experiments on the newly proposed nuScenes OOD benchmark. The source code is available at https://github.com/uulm-mrm/mmood3d.

Autonomous vehicles rely on camera, LiDAR, and radar sensors to navigate the environment. Adverse weather conditions like snow, rain, and fog are known to be problematic for both camera and LiDAR-based perception systems. Currently, it is difficult to evaluate the performance of these methods due to the lack of publicly available datasets containing multimodal labeled data. To address this limitation, we propose the SemanticSpray++ dataset, which provides labels for camera, LiDAR, and radar data of highway-like scenarios in wet surface conditions. In particular, we provide 2D bounding boxes for the camera image, 3D bounding boxes for the LiDAR point cloud, and semantic labels for the radar targets. By labeling all three sensor modalities, the SemanticSpray++ dataset offers a comprehensive test bed for analyzing the performance of different perception methods when vehicles travel on wet surface conditions. Together with comprehensive label statistics, we also evaluate multiple baseline methods across different tasks and analyze their performances. The dataset will be available at https://semantic-spray-dataset.github.io

Operation monitoring for automation systems requires self-assessment of all data processing modules. In this work, we extend our new self-assessment method for linear Kalman filters based on subjective logic to nonlinear Kalman filtering. Furthermore, we propose novel approaches within this subjective logic-based framework to assess the overall filter performance in multi-sensor systems online, i.e., in real-time without ground truth data. The results of the proposed self-assessment method for nonlinear Kalman filtering are demonstrated through simulation studies, showing advantages compared to classical consistency measures, like the normalized innovation squared. In addition, the results of the proposed online overall filtering assessment for multi-sensor systems can even compete with consistency measures based on ground truth data, which cannot be applied in online applications.

The multi-sensor Labeled Multi-Bernoulli filter has the challenge of relying on the NP-hard multi-sensor update of the Generalized Labeled Multi-Bernoulli filter. This paper proposes the Fast Product Multi-Sensor Labeled Multi-Bernoulli filter, which is a filter for multi-sensor systems that solves this task by performing computationally simpler single-sensor Labeled Multi-Bernoulli filter updates based on a common prediction for each sensor. These single-sensor updates are then fused using a novel and efficient fusion strategy. Furthermore, the proposed filter is based on the Bayes parallel combination rule and can be seen as an efficient approximation of the multi-sensor Labeled Multi-Bernoulli filter. It enables full parallelization of the update step and benefits from sensor order independence compared to Iterated Corrector implementations. As a result, the robustness is increased, which is important for safety reasons, e.g., in autonomous driving. Our approach is evaluated on simulations, and the results are compared to an Iterated Corrector implementation of the Labeled Multi-Bernoulli filter.

Radar sensors employed for environment perception, e.g. in autonomous vehicles, output a lot of unwanted clutter. These points, for which no corresponding real objects exist, are a major source of errors in following processing steps like object detection or tracking. We therefore present two novel neural network setups for identifying clutter. The input data, network architectures and training configuration are adjusted specifically for this task. Special attention is paid to the downsampling of point clouds composed of multiple sensor scans. In an extensive evaluation, the new setups display substantially better performance than existing approaches. Because there is no suitable public data set in which clutter is annotated, we design a method to automatically generate the respective labels. By applying it to existing data with object annotations and releasing its code, we effectively create the first freely available radar clutter data set representing realworld driving scenarios. Code and instructions are accessible at www.github.com/kopp-j/clutter-ds.

The prediction of surrounding agents' motion is a key for safe autonomous driving. In this paper, we explore navigation maps as an alternative to the predominant High Definition (HD) maps for learning-based motion prediction. Navigation maps provide topological and geometrical information on road-level, HD maps additionally have centimeter-accurate lane-level information. As a result, HD maps are costly and time-consuming to obtain, while navigation maps with near-global coverage are freely available. We describe an approach to integrate navigation maps into learning-based motion prediction models. To exploit locally available HD maps during training, we additionally propose a model-agnostic method for knowledge distillation. In experiments on the publicly available Argoverse dataset with navigation maps obtained from OpenStreetMap, our approach shows a significant improvement over not using a map at all. Combined with our method for knowledge distillation, we achieve results that are close to the original HD map-reliant models. Our publicly available navigation map API for Argoverse enables researchers to develop and evaluate their own approaches using navigation maps4.

Despite the significant research efforts on trajectory prediction for automated driving, limited work exists on assessing the prediction reliability. To address this limitation we propose an approach that covers two sources of error, namely novel situations with out-of-distribution (OOD) detection and the complexity in in-distribution (ID) situations with uncertainty estimation. We introduce two modules next to an encoder-decoder network for trajectory prediction. Firstly, a Gaussian mixture model learns the probability density function of the ID encoder features during training, and then it is used to detect the OOD samples in regions of the feature space with low likelihood. Secondly, an error regression network is applied to the encoder, which learns to estimate the trajectory prediction error in supervised training. During inference, the estimated prediction error is used as the uncertainty. In our experiments, the combination of both modules outperforms the prior work in OOD detection and uncertainty estimation, on the Shifts robust trajectory prediction dataset by 2.8 % and 10.1%, respectively. The code is publicly available44project page: https://github.com/againerju/joodu.

The safety of automated vehicles (AVs) relies on the representation of their environment. Consequently, state-of-the-art AVs employ potent sensor systems to achieve the best possible environment representation at all times. Although these high-performing systems achieve impressive results, they induce significant requirements for the processing capabilities of an AV’s computational hardware components and their energy consumption.To enable a dynamic adaptation of such perception systems based on the situational perception requirements, we introduce a model-agnostic method for the scalable employment of single-frame object detection models using frame-dropping in tracking-by-detection systems. We evaluate our approach on the KITTI 3D Tracking Benchmark, showing that significant energy savings can be achieved at acceptable performance degradation, reaching up to 28% reduction of energy consumption at a performance decline of 6.6% in HOTA score.

We propose a certifiably globally optimal approach for solving the hand-eye robot-world problem supporting multiple sensors and targets at once. Further, we leverage this formulation for estimating a geo-referenced calibration of infrastructure sensors. Since vehicle motion recorded by infrastructure sensors is mostly planar, obtaining a unique solution for the respective hand-eye robot-world problem is unfeasible without incorporating additional knowledge. Hence, we extend our proposed method to include a-priori knowledge, i.e., the translation norm of calibration targets, to yield a unique solution. Our approach achieves state-of-the-art results on simulated and real-world data. Especially on real-world intersection data, our approach utilizing the translation norm is the only method providing accurate results.

Planning trajectories for automated vehicles in urban environments requires methods with high generality, long planning horizons, and fast update rates. Using a path-velocity decomposition, we contribute a novel planning framework, which generates foresighted trajectories and can handle a wide variety of state and control constraints effectively. In contrast to related work, the proposed optimal control problems are formulated over space rather than time. This spatial formulation decouples environmental constraints from the optimization variables, which allows the application of simple, yet efficient shooting methods. To this end, we present a tailored solution strategy based on ILQR, in the Augmented Lagrangian framework, to rapidly minimize the trajectory objective costs, even under infeasible initial solutions. Evaluations in simulation and on a full-sized automated vehicle in real-world urban traffic show the real-time capability and versatility of the proposed approach.

It is desirable to predict the behavior of traffic participants conditioned on different planned trajectories of the autonomous vehicle. This allows the downstream planner to estimate the impact of its decisions. Recent approaches for conditional behavior prediction rely on a regression decoder, meaning that coordinates or polynomial coefficients are regressed. In this work we revisit set-based trajectory prediction, where the probability of each trajectory in a predefined trajectory set is determined by a classification model, and first-time employ it to the task of conditional behavior prediction. We propose RESET, which combines a new metric-driven algorithm for trajectory set generation with a graph-based encoder. For unconditional prediction, RESET achieves comparable performance to a regression-based approach. Due to the nature of set-based approaches, it has the advantageous property of being able to predict a flexible number of trajectories without influencing runtime or complexity. For conditional prediction, RESET achieves reasonable results with late fusion of the planned trajectory, which was not observed for regression-based approaches before. This means that RESET is computationally lightweight to combine with a planner that proposes multiple future plans of the autonomous vehicle, as large parts of the forward pass can be reused.

The development of approaches for trajectory prediction requires metrics to validate and compare their performance. Currently established metrics are based on Euclidean distance, which means that errors are weighted equally in all directions. Euclidean metrics are insufficient for structured environments like roads, since they do not properly capture the agent’s intent relative to the underlying lane. In order to provide a reasonable assessment of trajectory prediction approaches with regard to the downstream planning task, we propose a new metric that is lane distance-based: Lane Miss Rate (LMR). For the calculation of LMR, the ground-truth and predicted endpoints are assigned to lane segments, more precisely their centerlines. Measured by the distance along the lane segments, predictions that are within a certain threshold distance to the ground-truth count as hits, otherwise they count as misses. LMR is then defined as the ratio of sequences that yield a miss. Our results on three state-of-the-art trajectory prediction models show that LMR preserves the order of Euclidean distance-based metrics. In contrast to the Euclidean Miss Rate, qualitative results show that LMR yields misses for sequences where predictions are located on wrong lanes. Hits on the other hand result for sequences where predictions are located on the correct lane. This means that LMR implicitly weights Euclidean error relative to the lane and goes into the direction of capturing intents of traffic agents. The source code of LMR for Argoverse 2 is publicly available1.

Panoptic segmentation is one of the most challenging scene parsing tasks, combining the tasks of semantic segmentation and instance segmentation. While much progress has been made, few works focus on the real-time application of panoptic segmentation methods. In this paper, we revisit the recently introduced K-Net architecture. We propose vital changes to the architecture, training, and inference procedure, which massively decrease latency and improve performance. Our resulting RT-K-Net sets a new state-of-the-art performance for real-time panoptic segmentation methods on the Cityscapes dataset and shows promising results on the challenging Mapillary Vistas dataset. On Cityscapes, RT-K-Net reaches 60.2 % PQ with an average inference time of 32 ms for full resolution 1024×2048 pixel images on a single Titan RTX GPU. On Mapillary Vistas, RT-K-Net reaches 33.2 % PQ with an average inference time of 69 ms. Source code is available at https://github.com/markusschoen/RT-K-Net.

Predicting the motion of surrounding vehicles is essential for autonomous vehicles, as it governs their own motion plan. Current state-of-the-art vehicle prediction models heavily rely on map information. In reality, however, this information is not always available. We therefore propose CRAT-Pred, a multi-modal and non-rasterization-based trajectory prediction model, specifically designed to effectively model social interactions between vehicles, without relying on map information. CRAT-Pred applies a graph convolution method originating from the field of material science to vehicle prediction, allowing to efficiently leverage edge features, and combines it with multi-head self-attention. Compared to other map-free approaches, the model achieves state-of-the-art performance with a significantly lower number of model parameters. In addition to that, we quantitatively show that the self-attention mechanism is able to learn social interactions between vehicles, with the weights representing a measurable interaction score. The source code is publicly available33Source code: https://github.com/schmidt-ju/crat-pred.

In this work, we present MotionMixer, an efficient 3D human body pose forecasting model based solely on multi-layer perceptrons (MLPs). MotionMixer learns the spatial-temporal 3D body pose dependencies by sequentially mixing both modalities. Given a stacked sequence of 3D body poses, a spatial-MLP extracts fine-grained spatial dependencies of the body joints. The interaction of the body joints over time is then modelled by a temporal MLP. The spatial-temporal mixed features are finally aggregated and decoded to obtain the future motion. To calibrate the influence of each time step in the pose sequence, we make use of squeeze-and-excitation (SE) blocks. We evaluate our approach on Human3.6M, AMASS, and 3DPW datasets using the standard evaluation protocols. For all evaluations, we demonstrate state-of-the-art performance, while having a model with a smaller number of parameters. Our code is available at: https://github.com/MotionMLP/MotionMixer.

Advances in the field of environment perception for automated agents have resulted in an ongoing increase in generated sensor data. The available computational resources to process these data are bound to become insufficient for real-time applications. Reducing the amount of data to be processed by identifying the most relevant data based on the agents’ situation, often referred to as situation-awareness, has gained increasing research interest, and the importance of complementary approaches is expected to increase further in the near future. In this work, we extend the applicability range of our recently introduced concept for situation-aware environment perception to the decentralized automation architecture of the UNICARagil project. Considering the specific driving capabilities of the vehicle and using real-world data on target hardware in a post-processing manner, we provide an estimate for the daily reduction in power consumption that accumulates to 36.2%. While achieving these promising results, we additionally show the need to consider scalability in data processing in the design of software modules as well as in the design of functional systems if the benefits of situation-awareness shall be leveraged optimally.

Gesture recognition is essential for the interaction of autonomous vehicles with humans. While the current approaches focus on combining several modalities like image features, keypoints and bone vectors, we present neural network architecture that delivers state-of-the-art results only with body skeleton input data. We propose the spatio-temporal multilayer perceptron for gesture recognition in the context of autonomous vehicles. Given 3D body poses over time, we define temporal and spatial mixing operations to extract features in both domains. Additionally, the importance of each time step is re-weighted with Squeeze-and-Excitation layers. An extensive evaluation of the TCG and Drive& Act datasets is provided to showcase the promising performance of our approach. Furthermore, we deploy our model to our autonomous vehicle to show its real-time capability and stable execution.

Adverse weather conditions can negatively affect LiDAR-based object detectors. In this work, we focus on the phenomenon of vehicle gas exhaust condensation in cold weather conditions. This everyday effect can influence the estimation of object sizes, orientations and introduce ghost object detections, compromising the reliability of the state of the art object detectors. We propose to solve this problem by using data augmentation and a novel training loss term. To effectively train deep neural networks, a large set of labeled data is needed. In case of adverse weather conditions, this process can be extremely laborious and expensive. We address this issue in two steps: First, we present a gas exhaust data generation method based on 3D surface reconstruction and sampling which allows us to generate large sets of gas exhaust clouds from a small pool of labeled data. Second, we introduce a point cloud augmentation process that can be used to add gas exhaust to datasets recorded in good weather conditions. Finally, we formulate a new training loss term that leverages the augmented point cloud to increase object detection robustness by penalizing predictions that include noise. In contrast to other works, our method can be used with both grid-based and point-based detectors. Moreover, since our approach does not require any network architecture changes, inference times remain unchanged. Experimental results on real data show that our proposed method greatly increases robustness to gas exhaust and noisy data.

We present TransMOT, a novel transformer-based end-to-end trainable online tracker and detector for point cloud data. The model utilizes a cross- and a self-attention mechanism and is applicable to lidar data in an automotive context, as well as other data types, such as radar. Both track management and the detection of new tracks are performed by the same transformer decoder module and the tracker state is encoded in feature space. With this approach, we make use of the rich latent space of the detector for tracking rather than relying on low-dimensional bounding boxes. Still, we are able to retain some of the desirable properties of traditional Kalman-filter based approaches, such as an ability to handle sensor input at arbitrary timesteps or to compensate frame skips. This is possible due to a novel module that transforms the track information from one frame to the next on feature-level and thereby fulfills a similar task as the prediction step of a Kalman filter. Results are presented on the challenging real-world dataset nuScenes, where the proposed model outperforms its Kalman filter-based tracking baseline.

Advances in learning-based trajectory prediction are enabled by large-scale datasets. However, in-depth analysis of such datasets is limited. Moreover, the evaluation of prediction models is limited to metrics averaged over all samples in the dataset. We propose an automated methodology that allows to extract maneuvers (e.g., left turn, lane change) from agent trajectories in such datasets. The methodology considers information about the agent dynamics and information about the lane segments the agent traveled along. Although it is possible to use the resulting maneuvers for training classification networks, we exemplary use them for extensive trajectory dataset analysis and maneuver-specific evaluation of multiple state-of-the-art trajectory prediction models. Additionally, an analysis of the datasets and an evaluation of the prediction models based on the agent dynamics is provided.

A common approach for modeling the environment of an autonomous vehicle are dynamic occupancy grid maps, in which the surrounding is divided into cells, each containing the occupancy and velocity state of its location. Despite the advantage of modeling arbitrary shaped objects, the used algorithms rely on hand-designed inverse sensor models and semantic information is missing. Therefore, we introduce a multi-task recurrent neural network to predict grid maps providing occupancies, velocity estimates, semantic information and the driveable area. During training, our network architecture, which is a combination of convolutional and recurrent layers, processes sequences of raw lidar data, that is represented as bird’s eye view images with several height channels. The multi-task network is trained in an end-to-end fashion to predict occupancy grid maps without the usual preprocessing steps consisting of removing ground points and applying an inverse sensor model. In our evaluations, we show that our learned inverse sensor model is able to overcome some limitations of a geometric inverse sensor model in terms of representing object shapes and modeling freespace. Moreover, we report a better runtime performance and more accurate semantic predictions for our end-to-end approach, compared to our network relying on measurement grid maps as input data.

Capturing uncertainty in object detection is indispensable for safe autonomous driving. In recent years, deep learning has become the de-facto approach for object detection, and many probabilistic object detectors have been proposed. However, there is no summary on uncertainty estimation in deep object detection, and existing methods are either built with different network architectures and uncertainty estimation methods, or evaluated on different datasets with a wide range of evaluation metrics. As a result, a comparison among methods remains challenging, as does the selection of a model that best suits a particular application. This paper aims to alleviate this problem by providing a review and comparative study on existing probabilistic object detection methods for autonomous driving applications. First, we provide an overview of practical uncertainty estimation methods in deep learning, and then systematically survey existing methods and evaluation metrics for probabilistic object detection. Next, we present a strict comparative study for probabilistic object detection based on an image detector and three public autonomous driving datasets. Finally, we present a discussion of the remaining challenges and future works. Code has been made available athttps://github.com/asharakeh/pod_compare.git.

The availability of many real-world driving datasets is a key reason behind the recent progress of object detection algorithms in autonomous driving. However, there exist ambiguity or even failures in object labels due to error-prone annotation process or sensor observation noise. Current public object detection datasets only provide deterministic object labels without considering their inherent uncertainty, as does the common training process or evaluation metrics for object detectors. As a result, an in-depth evaluation among different object detection methods remains challenging, and the training process of object detectors is sub-optimal, especially in probabilistic object detection. In this work, we infer the uncertainty in bounding box labels from LiDAR point clouds based on a generative model, and define a new representation of the probabilistic bounding box through a spatial uncertainty distribution. Comprehensive experiments show that the proposed model reflects complex environmental noises in LiDAR perception and the label quality. Furthermore, we propose Jaccard IoU (JIoU) as a new evaluation metric that extends IoU by incorporating label uncertainty. We conduct an in-depth comparison among several LiDAR-based object detectors using the JIoU metric. Finally, we incorporate the proposed label uncertainty in a loss function to train a probabilistic object detector and to improve its detection accuracy. We verify our proposed methods on two public datasets (KITTI, Waymo), as well as on simulation data. Code is released athttps://github.com/ZiningWang/Inferring-Spatial-Uncertainty-in-Object-Detection.

In this work, we present an approach for monocular hand-eye calibration from per-sensor ego-motion based on dual quaternions. Due to non-metrically scaled translations of monocular odometry, a scaling factor has to be estimated in addition to the rotation and translation calibration. For this, we derive a quadratically constrained quadratic program that allows a combined estimation of all extrinsic calibration parameters. Using dual quaternions leads to low run-times due to their compact representation. Our problem formulation further allows to estimate multiple scalings simultaneously for different sequences of the same sensor setup. Based on our problem formulation, we derive both, a fast local and a globally optimal solving approach. Finally, our algorithms are evaluated and compared to state-of-the-art approaches on simulated and real-world data, e.g., the EuRoC MAV dataset.

We introduce MGNet, a multi-task framework for monocular geometric scene understanding. We define monocular geometric scene understanding as the combination of two known tasks: Panoptic segmentation and self-supervised monocular depth estimation. Panoptic segmentation captures the full scene not only semantically, but also on an instance basis. Self-supervised monocular depth estimation uses geometric constraints derived from the camera measurement model in order to measure depth from monocular video sequences only. To the best of our knowledge, we are the first to propose the combination of these two tasks in one single model. Our model is designed with focus on low latency to provide fast inference in real-time on a single consumer-grade GPU. During deployment, our model produces dense 3D point clouds with instance aware semantic labels from single high-resolution camera images. We evaluate our model on two popular autonomous driving benchmarks, i.e., Cityscapes and KITTI, and show competitive performance among other real-time capable methods. Source code is available at https://github.com/markusschoen/MGNet.

Modeling and understanding the environment is an essential task for autonomous driving. In addition to the detection of objects, in complex traffic scenarios the motion of other road participants is of special interest. Therefore, we propose to use a recurrent neural network to predict a dynamic occupancy grid map, which divides the vehicle surrounding in cells, each containing the occupancy probability and a velocity estimate. During training, our network is fed with sequences of measurement grid maps, which encode the lidar measurements of a single time step. Due to the combination of convolutional and recurrent layers, our approach is capable to use spatial and temporal information for the robust detection of static and dynamic environment. In order to apply our approach with measurements from a moving ego-vehicle, we propose a method for ego-motion compensation that is applicable in neural network architectures with recurrent layers working on different resolutions. In our evaluations, we compare our approach with a state-of-the-art particle-based algorithm on a large publicly available dataset to demonstrate the improved accuracy of velocity estimates and the more robust separation of the environment in static and dynamic area. Additionally, we show that our proposed method for ego-motion compensation leads to comparable results in scenarios with stationary and with moving ego-vehicle.

Radar sensors have a long tradition in advanced driver assistance systems (ADAS) and also play a major role in current concepts for autonomous vehicles. Their importance is reasoned by their high robustness against meteorological effects, such as rain, snow, or fog, and the radar’s ability to measure relative radial velocity differences via the Doppler effect. The cause for these advantages, namely the large wavelength, is also one of the drawbacks of radar sensors. Compared to camera or lidar sensor, a lot more surfaces in a typical traffic scenario appear flat relative to the radar’s emitted signal. This results in multi-path reflections or so called ghost detections in the radar signal. Ghost objects pose a major source for potential false positive detections in a vehicle’s perception pipeline. Therefore, it is important to be able to segregate multipath reflections from direct ones. In this article, we present a dataset with detailed manual annotations for different kinds of ghost detections. Moreover, two different approaches for identifying these kinds of objects are evaluated. We hope that our dataset encourages more researchers to engage in the fields of multi-path object suppression or exploitation.

Recent advancements in perception for autonomous driving are driven by deep learning. In order to achieve robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors (e.g. cameras, LiDARs, Radars), and multiple sensing modalities can be fused to exploit their complementary properties. In this context, many methods have been proposed for deep multi-modal perception problems. However, there is no general guideline for network architecture design, and questions of “what to fuse”, “when to fuse”, and “how to fuse” remain open. This review paper attempts to systematically summarize methodologies and discuss challenges for deep multi-modal object detection and semantic segmentation in autonomous driving. To this end, we first provide an overview of on-board sensors on test vehicles, open datasets, and background information for object detection and semantic segmentation in autonomous driving research. We then summarize the fusion methodologies and discuss challenges and open questions. In the appendix, we provide tables that summarize topics and methods. We also provide an interactive online platform to navigate each reference: https://boschresearch.github.io/multimodalperception/.

The fusion of multimodal sensor streams, such as camera, lidar, and radar measurements, plays a critical role in object detection for autonomous vehicles, which base their decision making on these inputs. While existing methods exploit redundant information in good environmental conditions, they fail in adverse weather where the sensory streams can be asymmetrically distorted. These rare ``edge-case'' scenarios are not represented in available datasets, and existing fusion architectures are not designed to handle them. To address this challenge we present a novel multimodal dataset acquired in over 10,000~km of driving in northern Europe. Although this dataset is the first large multimodal dataset in adverse weather, with 100k labels for lidar, camera, radar, and gated NIR sensors, it does not facilitate training as extreme weather is rare. To this end, we present a deep fusion network for robust fusion without a large corpus of labeled training data covering all asymmetric distortions. Departing from proposal-level fusion, we propose a single-shot model that adaptively fuses features, driven by measurement entropy. We validate the proposed method, trained on clean data, on our extensive validation dataset. Code and data are available here https://github.com/princeton-computational-imaging/SeeingThroughFog.

Conventional sensor systems record information about directly visible objects, whereas occluded scene components are considered lost in the measurement process. Non-line-of-sight (NLOS) methods try to recover such hidden objects from their indirect reflections - faint signal components, traditionally treated as measurement noise. Existing NLOS approaches struggle to record these low-signal components outside the lab, and do not scale to large-scale outdoor scenes and high-speed motion, typical in automotive scenarios. In particular, optical NLOS capture is fundamentally limited by the quartic intensity falloff of diffuse indirect reflections. In this work, we depart from visible-wavelength approaches and demonstrate detection, classification, and tracking of hidden objects in large-scale dynamic environments using Doppler radars that can be manufactured at low-cost in series production. To untangle noisy indirect and direct reflections, we learn from temporal sequences of Doppler velocity and position measurements, which we fuse in a joint NLOS detection and tracking network over time. We validate the approach on in-the-wild automotive scenes, including sequences of parked cars or house facades as relay surfaces, and demonstrate low-cost, real-time NLOS in dynamic automotive environments.

A main task for automated vehicles is an accurate and robust environment perception. Especially, an error-free detection and modeling of other traffic participants is of great importance to drive safely in any situation. For this purpose, multi-object tracking algorithms, based on object detections from raw sensor measurements, are commonly used. However, false object hypotheses can occur due to a high density of different traffic participants in complex, arbitrary scenarios. For this reason, the presented approach introduces a probabilistic model to verify the existence of a tracked object. Therefore, an object verification module is introduced, where the influences of multiple digital map elements on a track's existence are evaluated. Finally, a probabilistic model fuses the various influences and estimates an extended existence probability for every track. In addition, a Bayes Net is implemented as directed graphical model to highlight this work's expandability. The presented approach, reduces the number of false positives, while retaining true positives. Real world data is used to evaluate and to highlight the benefits of the presented approach, especially in urban scenarios.

Self-assessment is a key to safety and robustness in automated driving. In order to design safer and more robust automated driving functions, the goal is to self-assess the performance of each module in a whole automated driving system. One crucial component in automated driving systems is the tracking of surrounding objects, where the Kalman filter is the most fundamental tracking algorithm. For Kalman filters, some classical online consistency measures exist for self-assessment, which are based on classical probability theory. However, these classical approaches lack the ability to measure the explicit statistical uncertainty within the self-assessment, which is an important quality measure, particularly, if only a small number of samples is available for the self-assessment. In this work, we propose a novel online self-assessment method using subjective logic, which is a modern extension of probabilistic logic that explicitly models the statistical uncertainty. Thus, by embedding classical Kalman filtering into subjective logic, our method additionally features an explicit measure for statistical uncertainty in the self-assessment.

In this work, we tackle the problem of modeling the vehicle environment as dynamic occupancy grid map in complex urban scenarios using recurrent neural networks. Dynamic occupancy grid maps represent the scene in a bird's eye view, where each grid cell contains the occupancy probability and the two dimensional velocity. As input data, our approach relies on measurement grid maps, which contain occupancy probabilities, generated with lidar measurements. Given this configuration, we propose a recurrent neural network architecture to predict a dynamic occupancy grid map, i.e. filtered occupancy and velocity of each cell, by using a sequence of measurement grid maps. Our network architecture contains convolutional long-short term memories in order to sequentially process the input, makes use of spatial context, and captures motion. In the evaluation, we quantify improvements in estimating the velocity of braking and turning vehicles compared to the state-of-the-art. Additionally, we demonstrate that our approach provides more consistent velocity estimates for dynamic objects, as well as, less erroneous velocity estimates in static area.

The availability of real-world datasets is the prerequisite for developing object detection methods for autonomous driving. While ambiguity exists in object labels due to error-prone annotation process or sensor observation noises, current object detection datasets only provide deterministic annotations without considering their uncertainty. This precludes an in-depth evaluation among different object detection methods, especially for those that explicitly model predictive probability. In this work, we propose a generative model to estimate bounding box label uncertainties from LiDAR point clouds, and define a new representation of the probabilistic bounding box through spatial distribution. Comprehensive experiments show that the proposed model represents uncertainties commonly seen in driving scenarios. Based on the spatial distribution, we further propose an extension of IoU, called the Jaccard IoU (JIoU), as a new evaluation metric that incorporates label uncertainty. Experiments on the KITTI and the Waymo Open Datasets show that JIoU is superior to IoU when evaluating probabilistic object detectors.

This work presents a probabilistic deep neural network that combines LiDAR point clouds and RGB camera images for robust, accurate 3D object detection. We explicitly model uncertainties in the classification and regression tasks, and leverage uncertainties to train the fusion network via a sampling mechanism. We validate our method on three datasets with challenging real-world driving scenarios. Experimental results show that the predicted uncertainties reflect complex environmental uncertainty like difficulties of a human expert to label objects. The results also show that our method consistently improves the Average Precision by up to 7% compared to the baseline method. When sensors are temporally misaligned, the sampling method improves the Average Precision by up to 20%, showing its high robustness against noisy sensor inputs.

Avoiding collisions with vulnerable road users (VRUs) using sensor-based early recognition of critical situations is one of the manifold opportunities provided by the current development in the field of intelligent vehicles. As, especially, pedestrians and cyclists are very agile and have a variety of movement options, modeling their behavior in traffic scenes becomes a challenging task. In this paper, we propose movement models based on machine learning methods, in particular, artificial neural networks, in order to classify the current motion state and to predict the future trajectory of the VRUs. Both model types are also combined to enable the application of specifically trained motion predictors based on a continuously updated pseudo probabilistic state classification. Furthermore, the architecture is used to evaluate motion-specific physical models for starting and stopping and video-based pedestrian motion classification. A comprehensive dataset consisting of a total of 1068 pedestrian and 494 cyclist scenes acquired at an urban intersection is used for optimization, training, and evaluation of the different models. The results show substantially higher classification rates and the ability, through the machine learning approaches, to earlier recognize motion state changes than by the way of interacting multiple model (IMM) Kalman filtering. The trajectory prediction quality has also been improved for all kinds of test scenes, especially when starting and stopping motions are included. Here, 37% and 41% fewer position errors were achieved on average, respectively.

This work introduces an evaluation benchmark for depth estimation and completion using high-resolution depth measurements with angular resolution of up to 25" (arcsecond), akin to a 50 megapixel camera with per-pixel depth available. Existing datasets, such as the KITTI benchmark, provide only sparse reference measurements with an order of magnitude lower angular resolution - these sparse measurements are treated as ground truth by existing depth estimation methods. We propose an evaluation methodology in four characteristic automotive scenarios recorded in varying weather conditions (day, night, fog, rain). As a result, our benchmark allows us to evaluate the robustness of depth sensing methods in adverse weather and different driving conditions. Using the proposed evaluation data, we demonstrate that current stereo approaches provide significantly more stable depth estimates than monocular methods and lidar completion in adverse weather. Data and code are available at https://github.com/gruberto/PixelAccurateDepthBenchmark.git.

We tackle the long-term prediction of scene evolution in a complex downtown scenario for automated driving based on Lidar grid fusion and recurrent neural networks (RNNs). A bird's eye view of the scene, including occupancy and velocity, is fed as a sequence to a RNN which is trained to predict future occupancy. The nature of prediction allows generation of multiple hours of training data without the need of manual labeling. Thus, the training strategy and loss function are designed for long sequences of real-world data (unbalanced, continuously changing situations, false labels, etc.). The deep CNN architecture comprises convolutional long short-term memories (ConvLSTMs) to separate static from dynamic regions and to predict dynamic objects in future frames. Novel recurrent skip connections show the ability to predict small occluded objects, i.e. pedestrians, and occluded static regions. Spatio-temporal correlations between grid cells are exploited to predict multimodal future paths and interactions between objects. Experiments also quantity improvements to our previous network, a Monte Carlo approach, and literature.

We present a robust real-time LiDAR 3D object detector that leverages heteroscedastic aleatoric uncertainties to significantly improve its detection performance. A multi-loss function is designed to incorporate uncertainty estimations predicted by auxiliary output layers. Using our proposed method, the network ignores to train from noisy samples, and focuses more on informative ones. We validate our method on the KITTI object detection benchmark. Our method surpasses the baseline method which does not explicitly estimate uncertainties by up to nearly 9% in terms of Average Precision (AP). It also produces state-of-the-art results compared to other methods, while running with an inference time of only 72ms. In addition, we conduct extensive experiments to understand how aleatoric uncertainties behave. Extracting aleatoric uncertainties brings almost no additional computation cost during the deployment, making our method highly desirable for autonomous driving applications.

Training a deep object detector for autonomous driving requires a huge amount of labeled data. While recording data via on-board sensors such as camera or LiDAR is relatively easy, annotating data is very tedious and time-consuming, especially when dealing with 3D LiDAR points or radar data. Active learning has the potential to minimize human annotation efforts while maximizing the object detector's performance. In this work, we propose an active learning method to train a LiDAR 3D object detector with the least amount of labeled training data necessary. The detector leverages 2D region proposals generated from the RGB images to reduce the search space of objects and speed up the learning process. Experiments show that our proposed method works under different uncertainty estimations and query functions, and can save up to 60% of the labeling efforts while reaching the same network performance.

A main difficulty in autonomous driving is the assurance of maneuver acceptability by other traffic participants. Thus, knowledge about social interaction needs to be incorporated into the motion planning process. In this paper we present a model based framework to verify the acceptance of considered maneuvers and to plan social compliant motions. Therefore, we fuse two powerful approaches, one for decision-making and one for planning and show how the methods benefit from each other. Our method adheres to the classical structure of decision-making with subsequent trajectory planning and is consistent in the sense that both components are subject on the same, identical parametrized driver model. The overall method is real-time capable and the resulting trajectories adhere to kinematic constraints. Thus, the approach is applicable in realworld systems.

Overtaking is a challenging task in the field of autonomous driving, especially on roads with an opposite lane and oncoming vehicles. Since trajectory planning is repeated cyclic it is highly important to trigger the maneuver only if it is guaranteed that collision-free trajectories that satisfy kinematic constraints exist at each planning step. The goal of this paper is to present an algorithm for planning overtaking trajectories on large temporal horizons in real-time. The main idea is as follows: once overtaking is desired by the behavior module an initial trajectory is simulated using a path tracking control algorithm for lane changing combined with a classical PI-controller for approaching the target speed. The controllers are parametrized in a way that the simulated trajectory will satisfy kinematic constraints. If no collisions are detected a corridor containing the simulated trajectory is created to state constraints for a subsequent optimal control problem to relax the trajectory and smooth it to be comfortable to the vehicle passengers.

Most of the semantic segmentation approaches have been developed for single image segmentation, and hence, video sequences are currently segmented by processing each frame of the video sequence separately. The disadvantage of this is that temporal image information is not considered, which improves the performance of the segmentation approach. One possibility to include temporal information is to use recurrent neural networks. However, there are only a few approaches using recurrent networks for video segmentation so far. These approaches extend the encoder-decoder network architecture of well-known segmentation approaches and place convolutional LSTM layers between encoder and decoder. However, in this paper it is shown that this position is not optimal, and that other positions in the network exhibit better performance. Nowadays, state-of-the-art segmentation approaches rarely use the classical encoder-decoder structure, but use multi-branch architectures. These architectures are more complex, and hence, it is more difficult to place the recurrent units at a proper position. In this work, the multi-branch architectures are extended by convolutional LSTM layers at different positions and evaluated on two different datasets in order to find the best one. It turned out that the proposed approach outperforms the pure CNN-based approach for up to 1.6 percent.

High-resolution radar sensors are able to resolve multiple detections per object and, therefore, provide valuable information for vehicle environment perception. For instance, multiple detections allow us to infer the size of an object or to measure the object's motion more precisely. Yet, the increased amount of data raises the demands on tracking modules; measurement models that are able to process multiple detections for an object are necessary and measurement-to-object associations become more complex. This paper presents a new variational radar model for tracking vehicles using radar detections and demonstrates how this model can be incorporated into a random-finite-set-based multi-object filter. The measurement model is learned from actual data using variational Gaussian mixtures and avoids excessive manual engineering. In combination with the multi-object tracker, the entire process chain from raw measurements to the resulting tracks is formulated probabilistically. The presented approach is evaluated on experimental data, and it is demonstrated that the data-driven measurement model outperforms a manually designed model.

A good and robust sensor data fusion in diverse weather conditions is a quite challenging task. There are several fusion architectures in the literature, e.g. the sensor data can be fused right at the beginning (Early Fusion), or they can be first processed separately and then concatenated later (Late Fusion). In this work, different fusion architectures are compared and evaluated by means of object detection tasks, in which the goal is to recognize and localize predefined objects in a stream of data. Usually, state-of-the-art object detectors based on neural networks are highly optimized for good weather conditions, since the well-known benchmarks only consist of sensor data recorded in optimal weather conditions. Therefore, the performance of these approaches decreases enormously or even fails in adverse weather conditions. In this work, different sensor fusion architectures are compared for good and adverse weather conditions for finding the optimal fusion architecture for diverse weather situations. A new training strategy is also introduced such that the performance of the object detector is greatly enhanced in adverse weather scenarios or if a sensor fails. Furthermore, the paper responds to the question if the detection accuracy can be increased further by providing the neural network with a-priori knowledge such as the spatial calibration of the sensors.

Autonomous driving is a topic in computer vision which has captured a great deal of attention in recent years. One key problem is the detection and state analysis of traffic lights. Even over time, very few datasets for research in this topic have been published and they vary widely in quantity and in quality. To address the complexity of traffic light recognition, we introduce the DriveU**driveU is a joint innovation center of the Daimler AG and the University of Ulm Traffic Light Dataset (DTLD), a large-scale dataset consisting of more than 230,000 annotations. All annotations are hand-labeled according to strict rules and show a high quality. Recordings were made in eleven different cities during different weather conditions. Our dataset exceeds previous traffic light datasets in size, variance, annotation quality and amount of additional sensor data. We prove the extent of our dataset by an extensive comparison with existing traffic light datasets. Miscellaneous dataset criteria are compared, illustrated and statistically analyzed. In the process, metrics to express the quality and variance of datasets are developed and verified. The dataset can be downloaded from http://traffic-light-data.de.

Long-term situation prediction plays a crucial role for intelligent vehicles. A major challenge still to overcome is the prediction of complex downtown scenarios with multiple road users, e.g., pedestrians, bikes, and motor vehicles, interacting with each other. This contribution tackles this challenge by combining a Bayesian filtering technique for environment representation, and machine learning as long-term predictor. More specifically, a dynamic occupancy grid map is utilized as input to a deep convolutional neural network. This yields the advantage of using spatially distributed velocity estimates from a single time step for prediction, rather than a raw data sequence, alleviating common problems dealing with input time series of multiple sensors. Furthermore, convolutional neural networks have the inherent characteristic of using context information, enabling the implicit modeling of road user interaction. Pixel-wise balancing is applied in the loss function counteracting the extreme imbalance between static and dynamic cells. One of the major advantages is the unsupervised learning character due to fully automatic label generation. The presented algorithm is trained and evaluated on multiple hours of recorded sensor data and compared to Monte-Carlo simulation. Experiments show the ability to model complex interactions.

Motion trajectory planning is one crucial aspect for automated vehicles, as it governs the own future behavior in a dynamically changing environment. A good utilization of a vehicle's characteristics requires the consideration of the nonlinear system dynamics within the optimization problem to be solved. In particular, real-time feasibility is essential for automated driving, in order to account for the fast changing surrounding, e.g. for moving objects. The key contributions of this paper are the presentation of a fast optimization algorithm for trajectory planning including the nonlinear system model. Further, a new concurrent operation scheme for two optimization algorithms is derived and investigated. The proposed algorithm operates in the submillisecond range on a standard PC. As an exemplary scenario, the task of driving along a challenging reference course is demonstrated.

Sliding window approaches have been widely used for object recognition tasks in recent years [19], [4], [5], [18]. They guarantee an investigation of the entire input image for the object to be detected and allow a localization of that object. Despite the current trend towards deep neural networks, sliding window methods are still used in combination with convolutional neural networks [22]. The risk of overlooking an object is clearly reduced compared to alternative detection approaches which detect objects based on shape, edges or color. Nevertheless, the sliding window technique strongly increases the computational effort as the classifier has to verify a large number of object candidates. This paper proposes a sliding window approach which also uses depth information from a stereo camera. This leads to a greatly decreased number of object candidates without significantly reducing the detection accuracy. A theoretical investigation of the conventional sliding window approach is presented first. Other publications to date only mentioned rough estimations of the computational cost. A mathematical derivation clarifies the number of object candidates with respect to parameters such as image and object size. Subsequently, the proposed disparity sliding window approach is presented in detail. The approach is evaluated on pedestrian detection with annotations and images from the KITTI [10] object detection benchmark. Furthermore, a comparison with two state-of-the-art methods is made. Code is available in C++ and Python https://github.com/julimueller/disparity-sliding-window.

Safety is one of the key aspects of road vehicles. With applications of machine learning and artificial intelligence (AI) technologies, driver assistance and automated driving systems have been rapidly developed. This paper identifies one of the emerging safety issues of automated driving systems: functional deficiencies resulting from limited sensing abilities and algorithmic performance. Safety validation problem and challenges for some methodologies provided by ISO 26262 are addressed. To this end, we provide a methodology framework for identifying functional deficiencies during system development. A novel methodology based on possibility theory and a fuzzy relation model, Causal Scenario Analysis (CSA), is introduced as one essential part in this framework. A traffic light handling case study is presented.

We tackle the problem of object detection and pose estimation in a shared space downtown environment. For perception multiple laser scanners with $360^{o}$ coverage were fused in a dynamic occupancy grid map (DOGMa). A single-stage deep convolutional neural network is trained to provide object hypotheses comprising of shape, position, orientation and an existence score from a single input DOGMa. Furthermore, an algorithm for offline object extraction was developed to automatically label several hours of training data. The algorithm is based on a two-pass trajectory extraction, forward and backward in time. Typical for engineered algorithms, the automatic label generation suffers from misdetections, which makes hard negative mining impractical. Therefore, we propose a loss function counteracting the high imbalance between mostly static background and extremely rare dynamic grid cells. Experiments indicate, that the trained network has good generalization capabilities since it detects objects occasionally lost by the label algorithm. Evaluation reaches an average precision (AP) of 75.9%.

A dynamic occupancy grid map (DOGMa) allows a fast, robust, and complete environment representation for automated vehicles. Dynamic objects in a DOGMa, however, are commonly represented as independent cells while modeled objects with shape and pose are favorable. The evaluation of algorithms for object extraction or the training and validation of learning algorithms rely on labeled ground truth data. Manually annotating objects in a DOGMa to obtain ground truth data is a time consuming and expensive process. Additionally the quality of labeled data depend strongly on the variation of filtered input data. The presented work introduces an automatic labeling process, where a full sequence is used to extract the best possible object pose and shape in terms of temporal consistency. A two direction temporal search is executed to trace single objects over a sequence, where the best estimate of its extent and pose is refined in every time step. Furthermore, the presented algorithm only uses statistical constraints of the cell clusters for the object extraction instead of fixed heuristic parameters. Experimental results show a well-performing automatic labeling algorithm with real sensor data even at challenging scenarios.

The dynamic grid map illustrates the environment of robots with moving and static obstacles. Nuss et al. describe in [1] an implementation of this grid map, in which the state of the grid cells is to be modeled as a random finite set (RFS) based on a stochastic measurement system. For a real-time implementation this approach was approximated with Dempster-Shafer (DS). For this Nuss et al. design the areas without information (unknown areas) so, that no probabilistic calculations are executed. Only in the field of view, hypotheses represent the dynamic behavior of objects. This hypotheses are generated with particles. Therefore, in [1] it was proposed to extend this modeling. In this paper a pure Bayes approach is presented, which calculates all areas of the dynamic grid map probabilistic. Now, the resulting modeling generates hypotheses, which represent the dynamic behavior of unobservable objects. Thus, objects moving out of unknown areas can be detected more quickly. This leads to a more intuitive understanding as well as representation of the environment.

A major challenge in autonomous driving is the prediction of complex downtown scenarios with mutiple road users. This contribution tackles this challenge by combining a Bayesian filtering technique for environment representation and machine learning as long-term predictor. Therefore, a dynamic occupancy grid map representing the static and dynamic environment around the ego-vehicle is utilized as input to a deep convolutional neural network. This yields the advantage of using data from a single timestamp for prediction, rather than an entire time series. Furthermore, convolutional neural networks have the inherent characteristic of using context information, enabling the implicit modeling of road user interaction. One of the major advantages is the unsupervised learning character due to fully automatic label generation. The presented algorithm is trained and evaluated on multiple hours of recorded sensor data containing multiple road users, e.g., pedestrians, bikes and vehicles.

Combining data from heterogeneous sensors allows to enhance tracking systems by increasing the field of view, incorporating redundancy, and improving the performance by exploiting complementary sensor characteristics. This paper proposes a new vehicle tracking approach for vehicle environment perception that fuses radar and laser data. A Random-Finite-Set-based tracking filter, which permits a clear mathematical formulation of the multi-object problem, is used as fusion center. In combination with extended object measurement models that work on the raw sensor data directly, the filter uses all available information without the need for further preprocessing routines, considers object interdependencies, and works in ambiguous situations. The results are evaluated using experimental data from a test vehicle.

The correct handling of complex traffic-light-controlled intersections is still a challenge for automated vehicles. While a number of image-based approaches tackle close-range recognitions, an early traffic light detection at high distances is of great importance in the area of energy-efficient driving. For this reason, a traffic light detection system consisting of multiple on-board cameras is presented in this work, enabling the detection of traffic lights even from a distance of more than 200m. Furthermore, the presented system is based on tracking techniques using a Labeled Multi-Bernoulli filter in combination with the fusion of classifications based on the Dempster-Shafer theory of evidence. The system was tested on a real world data set collected in Germany and an increase in performance was demonstrated by a multi-camera approach.

This paper introduces three methods to improve traffic light recognition by using the stereo camera color image in tandem with the disparity image. The first method is object candidate filtering by analyzing the disparity values inside an object candidate. The second method applies the relative positioning filter. Using the depth measurement obtained from the disparity image as well as the intrinsic and extrinsic calibration, a three dimensional distance from an object to the vehicle can be calculated. Based on known real world traffic light locations, the filter is able to suppress thirty to seventy percent of false positives while only decreasing the detection rate by one percent. This result shows the huge potential for range filtering in traffic light recognition in general. The third method is the hypothesis size enhancement when re-projecting a hypothesis into the image. This process is enabled by a real world traffic light model in conjunction with the depth measurement. It is shown that all true positive hypotheses will benefit from this technique, resulting in a massively better overlap with the ground truth labels. When evaluated frame wise, re-projection can improve detection rate by up to fifteen percent. This work primarily enhances traffic light hypotheses obtained by a baseline detector and thus requires a hypothesis disparity value. As a further contribution this paper presents different methods for determining a hypothesis-wide disparity and evaluates the differences in quality and quantity.

Recent advances in the field of environment perception and cognition enable automated vehicles to safely drive in a growing variety of complex situations. However, in situations where required information cannot be observed directly and thus the consequences of the vehicle's actions cannot be estimated with high certainty, generating a safe behavior is still an unsolved problem. This paper tackles the scenario of a left turn maneuver in an urban environment with the presence of blind corners. We consider pedestrians and vehicles possibly hidden by parking cars, buildings or vegetation. In these cases, our approach allows to safely merge into traffic by using an environment representation based on tracked objects as well as an object-free sensor fusion including the calculation of unobservable regions in a digital map. A free-to-drive section of our desired path is obtained by long-term propagation of observed or possibly unobservable movement. The presented approach allows advancing into the road in a cautious manner, successively increasing the observable area.

Long-term prediction of traffic participants is crucial to enable autonomous driving on public roads. The quality of the prediction directly affects the frequency of trajectory planning. With a poor estimation of the future development, more computational effort has to be put in re-planning, and a safe vehicle state at the end of the planning horizon is not guaranteed. A holistic probabilistic prediction, considering inputs, results and parameters as random variables, highly reduces the problem. A time frame of several seconds requires a probabilistic description of the scene evolution, where uncertainty or accuracy is represented by the trajectory distribution. Following this strategy, a novel evaluation method is needed, coping with the fact, that the future evolution of a scene is also uncertain. We present a method to evaluate the probabilistic prediction of real traffic scenes with varying start conditions. The proposed prediction is based on a particle filter, estimating behavior describing parameters of a microscopic traffic model. Experiments on real traffic data with random leading vehicles show the applicability in terms of convergence, enabling long-term prediction using forward propagation.

This paper proposes a learning-based behavior generation approach for automated vehicles which is adapted sequentially. Instead of engineering behavioral policies for a variety of individual traffic situations by hand, our approach concentrates on a general problem description which is adjusted using a learning algorithm that successively derives safe actions as an outcome. Recent approaches apply Reinforcement Learning techniques for this problem using Markov Decision Processes (MDP). Our approach benefits from a trajectory planning module that uses an optimal control approach and generates realistic trajectories. Further, the trajectory planning module is exploited for the exploration in solving the adaption of the action selection problem. The task of action selection for merging into a roundabout as an exemplary traffic situation is examined. The contributions of this paper are the usage of an underlying optimization-based trajectory generation module and the evaluation of convergence of the adapted behavior, also for real-world data.

In advanced driver assistance systems and autonomous driving, reliable environment perception and object tracking based on radar is fundamental. High-resolution radar sensors often provide multiple measurements per object. Since in this case traditional point tracking algorithms are not applicable any more, novel approaches for extended object tracking emerged in the last few years. However, they are primarily designed for lidar applications or omit the additional Doppler information of radars. Classical radar based tracking methods using the Doppler information are mostly designed for point tracking of parallel traffic. The measurement model presented in this paper is developed to track vehicles of approximately rectangular shape in arbitrary traffic scenarios including parallel and cross traffic. In addition to the kinematic state, it allows to determine and track the geometric state of the object. Using the Doppler information is an important component in the model. Furthermore, it neither requires measurement preprocessing, data clustering, nor explicit data association. For object tracking, a Rao-Blackwellized particle filter (RBPF) adapted to the measurement model is presented.

This paper presents a novel approach to reconstructing the 3D shape of an object from a single image. The approach combines deep neural networks with a silhouette-based 3D reconstruction process. The optimal 3D shape is sought efficiently inside an extremely low-dimensional latent shape space, and the viewpoint and the object shape are jointly optimized based on the result of image segmentation. Evaluation of this approach shows a nearly 20 percent performance gain in viewpoint estimation subsequent to the optimization.

Recent advances in automotive radar technology have led to increasing sensor resolution and hence a more detailed image of the environment with multiple measurements per object. This poses several challenges for tracking systems: new algorithms are necessary to fully exploit the additional information and algorithms need to resolve measurement-to-object association ambiguities in cluttered multi-object scenarios. Also, the information has to be fused if multi-sensor setups are used to obtain redundancy and increased fields of view. In this paper, a Labeled Multi-Bernoulli filter for tracking multiple vehicles using multiple high-resolution radars is presented. This finite-set-statistics-based filter tackles all three challenges in a fully probabilistic fashion and is the first Monte Carlo implementation of its kind. The filter performance is evaluated using radar data from an experimental vehicle.

In an urban environment, one of the key challenges remains to be the reliable estimation of the other traffic participants' motion state. Due to the highly nonlinear motions in city traffic, an instant and precise estimation of heading direction, velocity, and, particularly, yaw rate is required. Radar sensors are well suited for this task due to their robustness to environmental influences and direct measurement of the radial (Doppler) velocity. High-resolution radars receive multiple reflections from an extended object. In comparison to state-of-the-art approaches, not only is the Doppler velocity of a single reference point taken into account, but also is the distribution of the Doppler velocity across the vehicle analyzed. The velocity profile is derived with characteristic features and a corresponding sample covariance. These are fused into an unscented Kalman filter, resulting in a significant accuracy improvement and a reduction in the latency of the filter to almost zero during a change in motion or initialization. This yields a great improvement in determining the trajectories of potential critical objects, increasing the time to avoid collisions. Furthermore, the approach enables simultaneous identification of the rotation center of the object, which is essential for the tracking of highly dynamic maneuvers. All approaches were implemented and evaluated on a large experimental data set using highly precise reference systems as ground truth. The results show an impressive improvement in the accuracy of the yaw rate estimation of a factor of 3-4 compared with state-of-the-art approaches in a dynamic scenario.

An ego-motion estimation method based on the spatial and Doppler information obtained by an automotive radar is proposed. The estimation of the motion state vector is performed in a density-based framework. Compared to standard vehicle odometry the approach is capable to estimate the full two dimensional motion state with three degrees of freedom. The measurement of a Doppler radar sensor is represented as a mixture of Gaussians. This mixture is matched with the mixture of a previous measurement by applying the appropriate egomotion transformation. The parameters of the transformation are found by the optimization of a suitable join metric. Due to the Doppler information the method is very robust against disturbances by moving objects and clutter. It provides excellent results for highly nonlinear movements. Real world results of the proposed method are presented. The measurements are obtained by a 77GHz radar sensor mounted on a test vehicle. A comparison using a high-precision inertial measurement unit with differential GPS support is made. The results show a high accuracy in velocity and yaw-rate estimation.

This paper presents an improved approach to the problem of energy efficient driving of heavy duty vehicles. The proposed model for a map-based Model Predictive Control (MPC) leads to an underlying Quadratic Programming (QP) optimization problem, allowing computationally efficient and robust solutions. A parameter estimation procedure is developed for a vehicle- and optimization-independent parametrization of the tradeoff between saving energy and keeping a desired vehicle velocity. Extensive simulations on a highway scenario for different optimization parameters give further insight to optimization properties, which can be utilized to enhance control performance. Compared to previous literature, we demonstrate a significant improvement of the computation time to under one-fifth of a millisecond, while maintaining (or even increasing) the fuel consumption reduction, which is 8.1 percent with the proposed approach compared to a standard cruise controller, without a decrease in the average cruising speed.

The project “Autonomous Driving” at Ulm University aims at advancing highly-automated driving with close-to-market sensors while ensuring easy exchangeability of the particular components. In this contribution, the experimental vehicle that was realized during the project is presented along with its software modules. To achieve the mentioned goals, a sophisticated fusion approach for robust environment perception is essential. Apart from the necessary motion planning algorithms, this paper thus focuses on the sensor-independent fusion scheme. It allows for an efficient sensor replacement and realizes redundancy by using probabilistic and generic interfaces. Redundancy is ensured by utilizing multiple sensors of different types in crucial modules like grid mapping, localization and tracking. Furthermore, the combination of the module outputs to a consistent environment model is achieved by employing their probabilistic representation. The performance of the vehicle is discussed using the experience from numerous autonomous driving tests on public roads.

Occupancy grid mapping is a well-known environment perception approach. A grid map divides the environment into cells and estimates the occupancy probability of each cell based on sensor measurements. An important extension is the Bayesian occupancy filter (BOF), which additionally estimates the dynamic state of grid cells and allows modeling changing environments. In recent years, the BOF attracted more and more attention, especially sequential Monte Carlo implementations (SMC-BOF), requiring less computational costs. An advantage compared to classical object tracking approaches is the object-free representation of arbitrarily shaped obstacles and free-space areas. Unfortunately, publications about BOF based on laser measurements report that grid cells representing big, contiguous, stationary obstacles are often mistaken as moving with the velocity of the ego vehicle (ghost movements). This paper presents a method to fuse laser and radar measurement data with the SMC-BOF. It shows that the doppler information of radar measurements significantly improves the dynamic estimation of the grid map, reduces ghost movements, and in general leads to a faster convergence of the dynamic estimation.

Grid map registration is an important field in mobile robotics. Applications in which multiple robots are involved benefit from multiple aligned grid maps as they provide an efficient exploration of the environment in parallel. In this paper, a normal distribution transform (NDT)-based approach for grid map registration is presented. For simultaneous mapping and localization approaches on laser data, the NDT is widely used to align new laser scans to reference scans. The original grid quantization-based NDT results in good registration performances but has poor convergence properties due to discontinuities of the optimization function and absolute grid resolution. This paper shows that clustering techniques overcome disadvantages of the original NDT by significantly improving the convergence basin for aligning grid maps. A multi-scale clustering method results in an improved registration performance which is shown on real world experiments on radar data.

This paper introduces a precise self-localization method for road vehicles. The presented approach is based on a single grayscale camera in addition with a conventional estimation of the ego motion and a map of the environment. This map is built in advance and independently from the localization process utilizing the same techniques. The proposed algorithm is based on Maximally Stable Extremal Regions which are robust features that are extracted from grayscale images. These features are matched in consecutive images using moment invariants. Together with an estimation of the ego motion, a 3D reconstruction of corresponding landmarks is obtained by applying multiple view geometry. For the unsupervised mapping process, landmarks are tracked and their corresponding global coordinates are stored in a geospatial database using a high-precision real-time kinematic system. The localization process itself is based on a particle filter to estimate the pose of the vehicle by making use of the previously generated map and currently observed landmarks. A standard GPS receiver is used to initialize the pose estimate. The evaluation with real world data shows that this approach achieves very good results despite the marginal sensor setup.

This contribution proposes a novel algorithm for predicting maneuvers at intersections. With applicability to driver assistance systems and autonomous driving, the presented methodology estimates a maneuver probability for every possible direction at an intersection. For this purpose, a generic intersection-feature, space-based representation is defined which combines static and dynamic intersection information with the dynamic properties of the observed vehicle, provided by a tracking module. A statistical behavior model is learned from previously recorded patterns by approximating the resulting feature space. Because the feature space consists of different types of features (mixed-feature space), a Bernoulli-Gaussian Mixture Model is applied as approximating function. Further, an online learning extension is proposed to adapt the model to the characteristics of different intersections.

Recent trends in human–computer interaction (HCI) show a development towards cognitive technical systems (CTS) to provide natural and efficient operating principles. To do so, a CTS has to rely on data from multiple sensors which must be processed and combined by fusion algorithms. Furthermore, additional sources of knowledge have to be integrated, to put the observations made into the correct context. Research in this field often focuses on optimizing the performance of the individual algorithms, rather than reflecting the requirements of CTS. This paper presents the information fusion principles in CTS architectures we developed for Companion Technologies. Combination of information generally goes along with the level of abstractness, time granularity and robustness, such that large CTS architectures must perform fusion gradually on different levels — starting from sensor-based recognitions to highly abstract logical inferences. In our CTS application we sectioned information fusion approaches into three categories: perception-level fusion, knowledge-based fusion and application-level fusion. For each category, we introduce examples of characteristic algorithms. In addition, we provide a detailed protocol on the implementation performed in order to study the interplay of the developed algorithms.

In this contribution, a new algorithm addressing the simultaneous localization and mapping (SLAM) problem is proposed: a Rao-Blackwellized implementation of the Labeled Multi-Bernoulli SLAM (LMB-SLAM) filter. Further, we establish that the LMB-SLAM does not require the approximations used in Probability Hypothesis Density SLAM (PHD-SLAM). The LMB-SLAM is shown to outperform PHD-SLAM in simulations by providing a more accurate map as well as an improved estimate of the vehicle's trajectory which is an expected result due to the superior performance of the LMB filter in tracking applications.

The estimation of the ego-vehicle's motion is a key capability for advanced driving assistant systems and mobile robot localization. The following paper presents a robust algorithm using radar sensors to instantly determine the complete 2D motion state of the ego-vehicle (longitudinal, lateral velocity and yaw rate). It evaluates the relative motion between at least two Doppler radar sensors and their received stationary reflections (targets). Based on the distribution of their radial velocities across the azimuth angle, non-stationary targets and clutter are excluded. The ego-motion and its corresponding covariance matrix are estimated. The algorithm does not require any preprocessing steps such as clustering or clutter suppression and does not contain any model assumptions. The sensors can be mounted at any position on the vehicle. A common field of view is not required, avoiding target association in space. As an additional benefit, all targets are instantly labeled as stationary or non-stationary.

This contribution presents a precise localization method for advanced driver assistance systems. A Maximally Stable Extremal Region (MSER) detector is used to extract bright areas, i.e. lane markings, from grayscale camera images. Furthermore, this algorithm is also used to extract features from a laser scanner grid map. These regions are automatically stored as landmarks in a geospatial data base during a map creation phase. A particle filter is then employed to perform the pose estimation. For the weight update of the filter the similarity between the set of online MSER detections and the set of mapped landmarks within the field of view is evaluated. Hereby, a two stage sensor fusion is carried out. First, in order to have a large field of view available, not only a forward facing camera but also a rearward facing camera is used and the detections from both sensors are fused. Secondly, the weight update also integrates the features detected from the laser grid map, which is created using measurements of three laser scanners. The performance of the proposed algorithm is evaluated on a 7 km long stretch of a rural road. The evaluation reveals that a relatively good position estimation and a very accurate orientation estimation (0.01 deg ± 0.22 deg) can be achieved using the presented localization method. In addition, an evaluation of the localization performance based only on each of the respective kinds of MSER features is provided in this contribution and compared to the combined approach.

The idea presented in this paper is an online learning approach for behavior prediction of other road participants at an intersection. Learning traffic situations online has the advantage that it is possible to react to changes in driving behavior due to changes in the environment. If visual obstruction occurs because of changes in the environment, e.g. a growing corn field, the behavior of drivers changes. In contrast to pre-trained models an online learning concept is able to react to these changes in driving behavior. In this contribution Case-Based Reasoning, a concept which adapts human reasoning and thinking to a system, is used. The functionality of the concept is shown by predicting the maneuver of an approaching vehicle at an intersection. The presented concept is able to predict if a vehicle turns in front of the ego-vehicle or stops and give the ego-vehicle right of way.

Digital maps are becoming increasingly important for driver assistance systems: providing optimal lighting conditions in night scenarios, presenting the road geometry to the driver, or for usage in autonomous driving tasks. However, recorded digital maps own one drawback: due to road changes and inaccurate recordings, discrepancies between the map and the real world exist. Because these discrepancies can lead to severe application level failures, detection of map errors is essential to ensure overall system integrity. This work proposes a new approach to online verification of digital maps for automotive usage. In contrast to previous work, the described system is able to detect errors in front of the vehicle. On the basis of a large database of map geometry and sensor information, a neural network is trained to classify the digital map integrity by optimally fusing different information sources depending on their strength and reliability. Although generally applicable, it is shown that a combination of orthogonal measurement principles is greatly beneficial for this decision task. A radar sensor, infra-red imagery and road geometry information estimated from visible light images are employed as input for the neural fusion. Experiments on real-world data verify the proposed concepts.

In conjunction with the advancing development of driver assistance systems, driver observation becomes increasingly important. This paper proposes a new approach for driver's head pose estimation. With a stereo camera mounted in a realistic position on top of the center stack the system continuously tracks the orientation of the driver's head in realtime (25fps) using solely 3-D information. The systems processing chain comprises separate modules for head separation, pose estimation and pose tracking. Head separation employes a Bayesian modeling approach for robust head-torso separation. The pose estimation module uses Synchronized Submanifold Embedding (SSE), a nonlinear regression method, which includes a dimensionality reduction, a k-nearest neighbor search and a barycentric coordinate estimation. The tracking module estimates angular velocity of the head, using an Extended Kalman Filter (EKF) in quaternion space. Comprehensive experiments show, that the proposed system achieves high accuracy from a non-central camera position. Since the approach does not rely on facial feature points the system handles large pose variations and is not disturbed by (sun)glasses.

Based on high-resolution radars a new approach for determining the full 2D-motion state (yaw rate, longitudinal and lateral speed) of an extended rigid object in a single measurement is proposed. The system does not rely on any model assumptions and is independent of the exact position, expansion and orientation of the object. In comparison to related methods it is not based on temporal filtering, e.g. a Kalman Filter. These methods are subject to an initialization phase and depend heavily on compliance of the underlying dynamic model. In contrast to temporal filtering, the proposed approach reduces the time to react to critical situations that occur in many safety and advanced driving assistance applications. This paper analyzes the velocity profile (radial velocity over azimuth angles) of the object received by two Doppler radar sensors. The approach can handle white noise and systematic variations (e.g. micro-Doppler of wheels) in the signal. The proposed system is applied to predict the driving path of traffic participants. Measurement results are presented for a set-up with two 77 GHz automotive radar sensors.

Occupancy grid mapping, street topology estimation, and object tracking are basic modules of vehicular environmental perception systems. For the sake of expandability and adaptability, this contribution proposes a hierarchical modular environmental perception architecture (HMEP). It limits the interactivity between individual basic modules. In a combination module, the consistency between occupancy grid cells and object tracks is evaluated. This allows to distinguish between static and dynamic parts of the environment in a postprocessing step. Efficient algorithms for implementation of the combination module are provided and an evaluation based on simulation and real data is presented.

Accurate and robust environment perception is a prerequisite for advanced driver assistance systems such as parking assistance, collision avoidance or night vision systems but also for robot navigation. In this context, occupancy grid mapping is a common way to represent the static car surroundings perceived by radar, laser or camera sensors. This work focuses on the challenges of nighttime grid mapping with a stereo camera — disparity images are perturbed by severe noise caused by wrong correspondences and headlights of oncoming vehicles. Noise removal and headlight detection based on the input images are proposed to make the disparity image suitable for a 2.5D occupancy grid mapping algorithm. In the second part of the work the concrete application of such a stereo grid map is shown and evaluated: The stereo map is applied to a localization task on a digital navigation map. The vehicle's position and orientation to the given navigation map are estimated in a particle filter system where the stereo grid map serves as measurement. Evaluations on real-world sequences and comparisons with an existing radar-based positioning system classify the approach's performance.

Robust tracking of extended objects plays a major role in research on highly automated driving applications and advanced driver assistance systems. This paper proposes a new approach to estimate the extension of dynamic objects based on object-local occupancy grid maps. This enables estimating free-formed object shapes while being robust against errors coming from, e.g., incorrect segmentation or association. Its benefit is shown based on 4-layer laser scanner sensor data and is evaluated against a ground truth based on a D-GPS system fused with a highly accurate IMU.

The Ko-PER (cooperative perception) research project aims at improvements of active traffic safety through cooperative perception systems. Within the project a prototype of a cooperative warning system was realized. This system provides early advisory warnings which are especially useful in critical situations with occluded conflict partners. The development process was accompanied by a series of driving simulator studies to determine both the potential to reduce traffic conflicts and important design characteristics of early advisory warning signals. The most important details of the prototype system's components inter-vehicle information-fusion and situation analysis are described and the achieved warning timings are compared to the results of the driving simulator studies.

This paper presents a novel approach towards highly precise self-localization of a vehicle on a digital map. The proposed approach utilizes a map containing region descriptors extracted from ordinary occupancy grid maps. The Maximally Stable Extremal Regions (MSER) algorithm provides robust feature extraction from grid maps in a completely unsupervised process. This allows for the automatic creation of huge maps. Since only single region descriptor points of grid maps are saved in the map database, the data volume of the produced map is kept low. The approach uses a particle filter to estimate the vehicle position on the digital map. The particle filter associates MSER features extracted from an online generated grid map with features of the digital map. An evaluation with real world sensor data, collected on a German rural road, shows that the approach locates the vehicle very precisely.

Automatic head pose estimation plays an important part in the development of human machine interfaces. This paper proposes a fast and frugal method for accurate and person-independent head pose estimation Based on range images. Head pose estimation is treated as a nonlinear regression problem and addressed with Synchronized Sub manifold Embedding (SSE). The offline training step exploits the local linear structure of label and feature space for a cross-wise synchronization of pose samples from different subjects. Based on this, multiclass Linear Discriminant Analysis (M-LDA) identifies a dimensionality-reducing linear projection, which diminishes non head pose related information. New samples are then projected into this lower dimensional feature space and classified Based on training samples within their local neighborhood. In case of sequential data, the occurrence of outliers can be reduced using a reasonable preselection of neighborhood candidates Based on tracking of pose changes. The experimental results on a publicly available dataBase prove, that the proposed algorithm can handle a large range of pose changes and outperforms existing methods in accuracy.

Advanced Driver Assistance Systems (ADAS) have witnessed a steady increase in complexity during the last few years. Many of these systems could benefit from a reliable long-term prediction of the vehicle's trajectory, for instance the prediction of a turning maneuver at an intersection. The application of probabilistic trajectory prediction provides knowledge of the probability and the uncertainty of the predicted trajectories, allowing a subsequent probabilistic treatment. In this contribution this is achieved by approximating a motion model through a probability density function (pdf) and inferring its parameters with previously observed motion patterns during a training procedure. Predictions can be obtained by calculating statistical parameters of the conditional probability density function (cpdf), for instance the mean and the variance. A common way to obtain the required cpdf is to approximate a joint pdf over the input and output variables and calculate the conditioning. Since the distribution over the input data space is not needed, this can be very wasteful of resources. Therefore in this contribution a novel approach for probabilistic trajectory prediction is proposed which directly approximates the cpdf using Hierarchical Mixture of Experts. Furthermore, the hierarchical structure of the model is exploited to incorporate optional knowledge in terms of categorical information (e.g., turn signal or map information) without the need to directly increase the input parameter space regarding all model components.

Automated driving applications require an environment perception that is reliable and fast. Multi-sensor fusion is a suitable means to combine the advantages of different measurement principles. However, this may lead to out-of-sequence measurements, i.e., asynchronous measurements where the original order of the measurements is lost. High-performance out-of-sequence algorithms are therefore needed that do not depend on the order of the measurements. In addition, existence probabilities can increase the reliability of the fusion system especially in safety critical applications. This paper presents a novel approach to handle out-of-sequence measurements not only in state estimation, but also in existence estimation. The method is shown to result in equal or less computational costs than state-of-the-art methods. The proposed algorithm is evaluated with real world data from crash tests.

This paper presents a self-learning spatial memory approach for advanced driver assistance systems. The storage concept for this memory is based on an object relational data base with support for spatial queries. This memory component is applied to the problem of stationary false alarms of an adaptive cruise control system. Test results which demonstrate the practicality of this approach are also provided. Furthermore, an evaluation for different weather conditions is presented.

Vision-based driver assistance systems have great potential for preventing fatalities. This work addresses the problem of 3D monocular vehicle tracking and turn rate estimation in situations where vehicles need to be tracked along intersections and curves. To estimate the tracked vehicle's turn rate, an approach based on image feature correspondences and a simplified geometric vehicle model is used. The model is robustly and efficiently fitted to the matched image features using an improved RANSAC scheme that automatically enforces physically plausible vehicle motions and speeds up the overall system at the same time. Temporal integration of the computed turn rates is performed by an Extended Kalman Filter with the bicycle motion model. Experiments with real world data show the applicability and robustness of the proposed concepts.

Future driving assistance systems will need an increase ability to handle complex driving situations and to react appropriately according to situation criticality and requirements for risk minimization. Humans, driving on motorways, are able to judge, for example, cut-in situations of vehicles because of their experiences. The idea presented in this paper is to adapt these human abilities to technical systems and learn different situations over time. Case-Based Reasoning is applied to predict the behavior of road participants because it incorporates a learning aspect, based on knowledge acquired from the driving history. This concept facilitates recognition by matching actual driving situations against stored ones. In the first instance, the concept is evaluated on action prediction of vehicles on adjacent lanes on motorways and focuses on the aspect of vehicles cutting into the lane of the host vehicle.

Detecting the road geometry at night time is an essential precondition to provide optimal illumination for the driver and the other traffic participants. In this paper we propose a novel approach to estimate the current road curvature based on three sensors: A far infrared camera, a near infrared camera and an imaging radar sensor. Various Convolutional Neural Networks with different configuration are trained for each input. By fusing the classifier responses of all three sensors, a further performance gain is achieved. To annotate the training and evaluation dataset without costly human interaction a fully automatic curvature annotation algorithm based on inertial navigation system is presented as well.

A major aim of the joint project Ko-PER is the mitigation of fatal accidents at urban intersections. Therefore several test intersections have been equipped with multiple laser range finders to recognize and track road users. Besides a high traffic density the variety of road users is challenging. In this contribution a multiple-model (MM) probability hypothesis density filter with a track representation extended by class probabilities is proposed. The approach enables tracking of road users with appropriate motion models using a single MM filter. Due to the estimation of the class probabilities an adaption of the transition probabilities between the models is possible. The performance of the road user tracking is evaluated using real world data.

Many serious collisions on highways happen while changing lanes. One of the main causes for these accidents is the driver's incorrect assessment of the current rear traffic situation. To support the driver, we propose a framework to intuitively visualize distance, speed and risk potential of approaching vehicles in a rear-view camera application. The proposed visualization techniques are based on color coding, artificial motion blur and depth-of-field rendering, which are motivated by sensory effects of the human eye and interpreted intuitively by the human visual system. The impact on the human assessment of the moving speed of an object rendered with artificial motion enhancement is evaluated in a user study. The required distance and motion estimation of the vehicles are extracted out of monocular video images, by combining lane recognition, vehicle detection and segmentation machine vision algorithms.

Today's night vision driver assistance systems help the driver by displaying an infrared image and detecting and highlighting other road users such as pedestrians or cyclists. To further increase active safety, future night vision systems could also visualize road course information. Especially the road courses at greater distances can help drivers interpret upcoming scenes. However, longer distance road course estimation is a challenging task because on-board sensors have a limited viewing range. This paper proposes a sensor fusion system that employs digital map information in combination with radar and camera sensors to estimate the 3D road course even at longer distances. The positioning task on the digital map is solved by a Bayesian framework that estimates position probability by means of map registration. By fusing road course data from the digital map and an optical lane recognition module, an accurate 3D road course estimation is obtained.

Intersections are challenging locations for drivers. Complex situations are common due to the variety of road users and intersection layouts. This contribution describes a real time method for detecting and tracking vehicles at intersections using images captured by a static camera network. After background subtraction, the foreground segments are projected on a common fusion map. Using this fusion map, the pose, width, and height of the vehicles can be determined. After that, the detected objects are tracked by a Gaussian-Mixture approximation of the Probability Hypothesis Density filter. Results of the intersection perception can further be communicated to equipped vehicles by wireless communication.

As their functionality becomes more and more complex, future driver assistance systems rely on several different sensors in order to combine the advantages of different measurement principles. However, in multi-sensor fusion, measurements may arrive at the fusion unit out-of-sequence, the original order of the measurements may be lost. Whereas out-of-sequence measurement processing in state estimation has been studied extensively, their incorporation in existence estimation has not been solved in the past. This paper presents a new algorithm for state and existence estimation in time-critical applications, where out-of-sequence measurements are handled adequately. The derived algorithm is validated with real-world data from crash tests.

This work addresses the problem of track association in person tracking. We propose a probabilistic model, based on Markov Logic Networks, that aims at associating the individual tracks emerging from a person tracking algorithm to the correct persons. For this purpose the continuous estimates of the object positions acquired by the tracking algorithm are mapped into discrete spatial regions, which are based on a floor plan of the environment. Experiments show that the described model is able to exploit the additional information contained inside the provided floor plan, and deliver good results compared to a state of the art person tracking algorithm despite the lossy discretization step. We discuss the engineered model in detail and give an empirical evaluation using an indoor setting.

The online observation using high-resolution radar of a scene containing extended objects imposes new requirements on a robust and fast clustering algorithm. This paper presents an algorithm based on the most cited and common clustering algorithm: DBSCAN [1]. The algorithm is modified to deal with the non-equidistant sampling density and clutter of radar data while maintaining all its prior advantages. Furthermore, it uses varying sampling resolution to perform an optimized separation of objects at the same time it is robust against clutter. The algorithm is independent of difficult to estimate input parameters such as the number or shape of available objects. The algorithm outperforms DBSCAN in terms of speed by using the knowledge of the sampling density of the sensor (increase of app. 40–70%). The algorithm obtains an even better result than DBSCAN by including the Doppler and amplitude information (unitless distance criteria).

Grid maps are a reliable representation of the environment. Based on a generic grid map definition, this paper presents three formulations: a laser scanner based occupancy grid, a video grid based on the Inverse Perspective Mapping and a novel feature grid, where lane marking features are used. Furthermore, this contribution presents a road course estimation based on such grid maps which yields estimations above 120m. Therefore, a digital road map (comparable to maps of GPS navigation systems) is matched to a grid map. Thus, a global position in the road map is estimated. Finally, a novel evaluation approach is presented to quantize the results of this grid map based map match.

Accident analysis shows that the majority of accidents with body injuries occur in urban areas and more than 50 percent of those urban accidents happen at intersections. Due to that a major aim of the Ko-PER project, which is part of research initiative Ko-FAS, is to improve safety at intersections by infrastructure based perception. To recognize and track the moving objects, a network of laserscanner sensors observes the intersection and provides a 3D profile of the current scene. By means of the 3D measurements a robust and adaptive Gaussian mixture background model is trained to segment the measurements of dynamic objects and static objects. After the segmentation, the foreground points of each sensor are clustered based on the density of the point clouds and finally pedestrians are classified using dimension features. This paper focuses on tracking of pedestrians, which are the most vulnerable road users. In order to be able to integrate dependencies between the states of the pedestrians, a random finite set particle filter is used to track the pedestrians. The performance of the laserscanner based tracking system is shown and evaluated with measurements from the Ko-PER test intersection at Conti-Safety-Park. Therefore, the optimal subpattern assignment (OSPA) metric is used to evaluate the object recognition and tracking system.

In cooperative perception systems, different vehicles share object data obtained by their local environment perception sensors, like radar or lidar, via wireless communication. In this paper, this so-called Car2X-based perception is modeled as a virtual sensor in order to integrate it into a highlevel sensor data fusion architecture. The spatial and temporal alignment of incoming data is a major issue in cooperative perception systems. Temporal alignment is done by predicting the received object data with a model-based approach. In this context, the CTRA (constant turn rate and acceleration) motion model is used for a three-dimensional prediction of the communication partner's motion. Concerning the spatial alignment, two approaches to transform the received data, including the uncertainties, into the receiving vehicle's local coordinate frame are compared. The approach using an unscented transformation is shown to be superior to the approach by linearizing the transformation function. Experimental results prove the accuracy and consistency of the virtual sensor's output.

The camera relative pose is essential information for driver assistance systems in general, and especially so for systems that aim to visualize obstacles or other relevant objects for the driver. In order to display objects that were not detected in the vision frame of the camera (e.g. road information of a digital map), the exact transformation from camera to the world has to be known for every time frame. The horizontal movement of the camera in the real world is well known due to sufficiently accurate inertial sensors and wheel speed sensors available in the car. In contrast, the vertical movement is both unknown and highly dynamic. Relative pose estimation is a widely used technique that determines a vehicle's movement and rotation from frame to frame. These techniques rely primarily on point correspondences computed beforehand. In the nighttime case, these methods fail because of the poor image quality and low number of correspondences. This paper presents a method that fuses vehicle movement information with the features extracted from a mono camera. By restricting the problem to pitch estimation only, this method allows robust and accurate estimation of the vehicle's pitch movements even in nighttime scenarios. A highly accurate inertial navigation system is used to evaluate the results obtained in challenging real world nighttime video sequences. Experiments show the feasibility and robustness of the proposed approach.

This paper addresses the problem of joint state and existence estimation in the presence of temporally asynchronous measurements. In multi-sensor fusion, the problem can occur that measurements from different sensors can arrive at the processing unit out of sequence, i.e., the original temporal order of the measurements is lost. For the first time, the influence of these out-of-sequence measurements on state estimation as well as on existence estimation is examined. The existence probabilities are estimated via the Joint Integrated Probabilistic Data Association Filter (JIPDA) [17]. Two different methods to deal with out-of-sequence measurements in JIPDA are described and compared. It is shown that the handling of out-of-sequence measurements has a considerable influence not only on state, but also on existence estimation.

In the context of driver assistance, an accurate and reliable prediction of the vehicle's trajectory is beneficial. This can be useful either to increase the flexibility of comfort systems or, in the more interesting case, to detect potentially dangerous situations as early as possible. In this contribution, a novel approach for trajectory prediction is proposed which has the capability to predict the vehicle's trajectory several seconds in advance, the so called long-term prediction. To achieve this, previously observed motion patterns are used to infer a joint probability distribution as motion model. Using this distribution, a trajectory can be predicted by calculating the probability for the future motion, conditioned on the current observed history motion pattern. The advantage of the probabilistic modeling is that the result is not only a prediction, but rather a whole distribution over the future trajectories and a specific prediction can be made by the evaluation of the statistical properties, e.g. the mean of this conditioned distribution. Additionally, an evaluation of the variance can be used to examine the reliability of the prediction.

Automotive radar sensors are commonly used for providing environment information required by driver assistance systems. Although they show a good performance in measuring the distance and the speed of other objects even under poor weather conditions, they suffer from the shortcoming of a missing resolution in elevation. Consequently bridges crossing the lane of the ego vehicle may look similar to stationary obstacles. This contribution shows a bridge identification algorithm based on the interference pattern resulting from the multipath propagation of the radar wave. During the approaching to bridges or stationary obstacles, the pattern leads to a variation in the backscattered power from the object. This variation can be used to make statements about the object height position above the road. Results calculated from real scanning radar sensor data show the usability in real traffic scenarios.

This paper presents a new approach to exact timestamping of asynchronous measurements in a multi-sensor setup. In order to improve the performance of a single sensor, driver assistance systems use several different sensors which have different latencies that usually cannot be measured directly. These unknown latencies pose a problem in data association and temporal synchronization. Consequently, a method to estimate or incorporate the latencies is needed in all sensor fusion algorithms in order to derive the real time of a measurement. In this paper a method is described to compensate the sensor latencies even if they cannot be observed directly.

This paper presents a generic grid mapping approach which can be used to map huge areas. A special map definition based on so called grid patches is proposed to limit memory usage and make it real time capable. The grid cells of this map can hold arbitrary data which is calculated based on arbitrary sensors. Here, methods for mapping occupancy likelihoods based on laser scanner data, gray values calculated by an Inverse Perspective Mapping from video images and intensities from an imaging radar sensor are proposed. Furthermore, this contribution presents a road course estimation based on such a grid map which yields estimations above 120 m. Therefore, a digital road map (comparable to maps of GPS navigation systems) is matched to a laser scanner based grid map. In general, this approach is generic as well. Exemplarily, a method based on an occupancy grid map is presented.

In this contribution, we present an overview of modeling techniques for sensory existence evidence using the Dempster Shafer Theory of Evidence (DST). Several modeling aspects are examined. The purpose of this approach is to enhance the detection performance of a sensor fusion system in terms of detection rate versus false alarm rate. An integrated state and existence estimation algorithm is used which directly incorporates the DST-based sensory information. The advantages of this algorithm are evaluated using a sensor fusion and tracking system based on a large database of real-world sensor data.

In cooperative perception systems, different vehicles share object data obtained by their local environment perception sensors, like radar or lidar, via wireless communication. In this paper, a fusion architecture for a cooperative perception system and a concept for a parametrizable offline simulation are proposed. In order to effectively develop and simulate such systems, knowledge about the main parameters of the employed wireless communication solution is crucial. For this reason, an experimental analysis of parameters like transmission latencies and transmission range of a communication solution based on IEEE 802.11p is presented.

Detailed and highly accurate digital maps provide useful information for future driver assistance systems. The information of the positions of infrastructure objects and lane markings can be used to extend the knowledge of the environment obtained by local sensors. To exploit highly accurate maps, the exact position of the vehicle within the map must be known. For that, a rough localization with standard GPS is extended by matching objects detected with a laser scanner and data from the map. The paper focuses on road construction sites, which are a demanding environment for sensorial perception and interpretation. The matching algorithms are based on beacons, which are commonly used infrastructure elements at road works.

In the past, there has been a tremendous advance in the area of simultaneous localization and mapping (SLAM). However, there are relatively few approaches for incorporating prior information or knowledge about structural similarities into the mapping process. Consider, for example, office buildings in which most of the offices have an identical geometric layout. The same typically holds for the individual stories of buildings. In this paper, we propose an approach for generating alignment constraints between different floors of the same building in the context of graph-based SLAM. This is done under the assumption that the individual floors of a building share at least some structural properties. To identify such areas, we apply a particle filter-based localization approach using maps and observations from different floors. We evaluate our system using several real datasets as well as in simulation. The results demonstrate that our approach is able to correctly align multiple floors and allows the robot to generate consistent models of multi-story buildings.

This paper presents a novel approach for road course estimation on rural roads using a mulitlayer laser scanner. The measurements of the sensor are used to build an occupancy grid as a representation of a local map. This mapping step uses a new free space function and a novel method for detecting and eliminating moving objects. Based on this map a feature extraction algorithm yields road border feature points. A Levenberg-Marquardt based optimization fits a flexible two-part road course model to these feature points.

Accident analysis shows that intersections are a focal point for accidents in urban areas. Due to that, traffic monitoring at intersections has attached much attention. A major part of the Ko-PER project, which is part of the research initiative Ko-FAS, promoted by the Federal Ministry of Economics and Technology of Germany, is the infrastructure based perception of all dynamic objects inside an intersection. In this project, a novel system of detecting and tracking objects inside intersections using multiple 4-layer laser scanners is proposed. To reduce occlusions and maximize the observed area the sensors are mounted high over ground-level to achieve a bird's eye view of the scene. One difference between laser scanners and video cameras is the difficulty to identify the area of the street, which can be observed by laser sensors, especially when the monitored area is plain like roads. Therefore, a realistic 3D simulation of the urban intersection and the laser range scanners was implemented. Based on this simulation, a method to calibrate the sensors was developed. The technique is easy to use and due to the 3D model of the intersection we were able to verify the proposed calibration tool.

In an automotive pre-crash application, it is vital to quickly and accurately estimate the position and velocity of objects in the frontal area of the vehicle. To improve such estimations, several radar sensors are fused to detect objects. Due to their different performance characteristics, their measurements can arrive at the pre-crash processing unit out-of-sequence. This work presents several techniques to integrate measurements into a tracking algorithm that arrive with such an out-of-sequence measurement (OOSM) scenario. A comprehensive complexity analysis of the algorithms is also presented. Most importantly, the algorithms are run on a test vehicle during real crash scenarios. The algorithms' performance is evaluated against reference data from a highly accurate laser scanner. It is shown that using advanced OOSM algorithms in pre-crash systems significantly increases performance and reduces computational cost compared to previous approaches.

Modern driver assistance and safety systems need a reliable and precise description of the environment. Fusing the measurement data of two or more sensors can improve the performance of the perception system. A generic fusion system which is independent of the attached sensors could be reused in multiple fusion systems and sensor combinations. This could be very helpful because sensor data fusion is a demanding and complex task. In this contribution, we present the algorithmic basics for a generic fusion system, detailed ways on how to model sensor specific properties and which benefits we can achieve by using these models.

This contribution presents a lane estimation system for night applications which covers distances up to 140 m in rural environment. The high detection range is essential for upcoming warning systems to decide whether a detected object is on the road and thus of immediate importance for the driving task. In order to realize a robust lane detection system we present a fusion system that combines the information provided by an imaging radar system and a digital map. The digital map is used to calculate the shape of the road. Past measurements of the radar sensor are integrated over time into a local map using an egomotion estimator. A particle filter realizes the matching of the digital and local map resulting in an accurate position of the vehicle on the digital map. This positioning algorithm enables an estimation of the position of the lane in front of the vehicle at high distances.

Road construction sites often are the reason for traffic jams and accidents due to the reduced road width. Driver assistance systems for these demanding environments highly benefit from a digital representation of the road layout. This digital map includes all important infrastructure elements, such as barriers, temporary road markings and guiding reflector posts. The paper describes the automatic generation of a detailed and highly accurate “Road Work Map” using video camera and laser scanners.

The autonomous emergency brake (AEB) is an active safety function for vehicles which aims to reduce the severity of a collision. An AEB performs a full brake when an accident becomes unavoidable. Even if this system cannot, in general, avoid the accident, it reduces the energy of the crash impact and is therefore referred to as a collision mitigation system. A new approach for the calculation of the trigger time of an emergency brake will be presented. The algorithm simultaneously considers all physically possible trajectories of the object and host vehicle. It can be applied to all different scenarios including rear-end collisions, collisions at intersections, and collisions with oncoming vehicles. Thus, 63% of possible accidents are addressed. The approach accounts for the object and host vehicles' dimensions. Unlike previous work, the orientation of the vehicles is incorporated into the collision estimation.

In this contribution we apply the probability hypothesis density (PHD) filter algorithm for joint tracking of an unknown varying number of targets to automotive environment sensing systems. We use data from a vision and a lidar sensor as well as the vehicle ESP system. After deriving a method to parametrise the algorithm systematically from detection performance statistics we proof the applicability of the method for automotive tracking based on real sensor data

The growing demand for active safety systems implies also the requirement for new and suitable evaluation and test processes for such systems. However, the evaluation of these systems is in most cases a non trivial, time consuming and expensive task. Especially, for active pedestrian protection systems, which are based on infrared detection of humans. In this paper we present a simulation of Thermopile sensors to evaluate the performance of an infrared based pedestrian detection system. By means of the simulation tool, any test scenario can be generated and the simulation provides the sensor data to the generated scenario. The simulated sensor output depends on the objects placed in the virtual environment and the background radiation. Considering the physical sensor dimensions and Planck's radiation law, an approximate image of the reality can be achieved. Thereby, it is possible to simulate pedestrian crash tests, which saves extensive time of testing and enables a fast optimization and adaption of the system.