High-definition (HD) maps are crucial for autonomous vehicles, but their creation and maintenance is very costly. This motivates the idea of online HD map construction. To provide a continuous large-scale stream of training data, existing HD maps can be used as labels for onboard sensor data from consumer vehicle fleets. However, compared to current, well curated HD map perception datasets, this fleet data suffers from localization errors, resulting in distorted map labels. We introduce three kinds of localization errors, Ramp, Gaussian, and Perlin noise, to examine their influence on generated map labels. We train a variant of MapTRv2, a state-of-the-art on-line HD map construction model, on the Argoverse 2 dataset with various levels of localization errors and assess the degradation of model performance. Since localization errors affect distant labels more severely, but are also less significant to driving performance, we introduce a distance-based map construction metric. Our experiments reveal that localization noise affects the model performance significantly. We demonstrate that errors in heading angle exert a more substantial influence than position errors, as angle errors result in a greater distortion of labels as distance to the vehicle increases. Furthermore, we can demonstrate that the model benefits from nondistorted ground truth (GT) data and that the performance decreases more than linearly with the increase in noisy data. Our study additionally provides a qualitative evaluation of the extent to which localization errors influence the construction of HD maps.

Autonomous vehicles rely on detailed and accurate environmental information to operate safely. High definition (HD) maps offer a promising solution, but their high maintenance cost poses a significant barrier to scalable deployment. This challenge is addressed by online HD map construction methods, which generate local HD maps from live sensor data. However, these methods are inherently limited by the short perception range of onboard sensors. To overcome this limitation and improve general performance, recent approaches have explored the use of standard definition (SD) maps as prior, which are significantly easier to maintain. We propose SDTagNet, the first online HD map construction method that fully utilizes the information of widely available SD maps, like OpenStreetMap, to enhance far range detection accuracy. Our approach introduces two key innovations. First, in contrast to previous work, we incorporate not only polyline SD map data with manually selected classes, but additional semantic information in the form of textual annotations. In this way, we enrich SD vector map tokens with NLP-derived features, eliminating the dependency on predefined specifications or exhaustive class taxonomies. Second, we introduce a point-level SD map encoder together with orthogonal element identifiers to uniformly integrate all types of map elements. Experiments on Argoverse 2 and nuScenes show that this boosts map perception performance by up to +5.9 mAP (+45%) w.r.t. map construction without priors and up to +3.2 mAP (+20%) w.r.t. previous approaches that already use SD map priors.

Classifying intentions of other traffic agents is an essential task for intelligent transportation systems. To simplify this task, vehicles are equipped with various illumination systems, including turn indicators, emergency lights, rear lights, and brake lights. We extend the Waymo open perception dataset with ground truth annotations for different visual intentions to develop methods designed to classify the state of such systems. Furthermore, we propose the VISUAL INTENTION FORMER, a two-step transformer-based architecture to classify visual intentions in image sequences of tracked traffic participants. We use a vision transformer to extract image features, which are passed into a transformer encoder that reasons about temporal dependencies among them. We evaluate against different baseline architectures where our proposed method achieves state-of-the-art results. Additionally, we conduct an in-depth performance analysis of our method regarding different input sequence lengths, vehicle headings, and daytime conditions.

Automated vehicles require carefully designed cost functions, which are challenging to specify due to the complexity of the behavior they need to cover. Inverse reinforcement learning is a principled methodology for deriving cost functions, but it requires high-quality expert demonstrations, which are expensive to obtain. Recently, scenario-based testing has emerged as a promising approach for validation of driving behavior. In this paper, we introduce a novel methodology that circumvents the need for costly expert driving demonstrations by harnessing scenario-based testing. Our Test-Driven Inverse Reinforcement Learning approach leverages Bayesian inference, utilizing the outcomes of scenario tests as observations to infer cost functions. We rigorously evaluate our method on simulated and real-world scenarios and demonstrate its ability to learn cost functions that successfully pass the respective scenario tests. We also show that the learned cost function generalizes well by also passing scenario tests from an unseen validation set and illustrate that few scenario tests are sufficient to learn meaningful cost functions. This innovative framework not only streamlines the cost function specification process but also offers a cost-effective and practical solution for advancing automated driving systems.

Precise object detection is crucial in automated driving. In contrast to lidar and radar sensors, cameras provide high-resolutional measurements at comparatively low cost. A state-of-the-art method for object detection using camera images is panoptic segmentation, which combines semantic and object instance information. Current public datasets use multiple cameras to cover a larger area of the environment. But, the limited field of view occludes objects. As a results, on the one hand, the correct dimensions of objects cannot be captured and, on the other hand, false detections can occur. Objects can also be detected multiple times in the overlapping image area. To track dynamic objects, duplicate detections must be filtered. Rather than directly segmenting all camera images individually, we first stitch them into a horizontal panorama. Using a stitched surround view avoids detection difficulties at the boundaries of the individual images. For this purpose, we leverage the EfficientPS pre-trained network architecture and adapt it for use with panoramic images. In our evaluation, we demonstrate the improvement in panoptic quality of a stitched surround view. In addition, we separately compare the panoptic quality in the overlapping image areas between the panorama and the individual images. Finally, we show further advantages of panoramic images in terms of inference time in runtime analysis.

Accurately predicting human driving behavior, particularly in highly interactive traffic scenarios, poses a significant challenge. In this work, we investigate the application of graph-based observations to Adversarial Imitation Learning (AIL) methods. Unlike conventional feature-based observations, this allows us to flexibly account for different road structures as well as a varying number of surrounding vehicles interacting with each other. We assess the method in a complex roundabout scenario from the INTERACTION dataset, employing several state-of-the-art AIL methods. The results indicate that our proposed approach successfully yields realistic driver models, applicable for accurate predictions of human driving behavior.

There is a gap in risk assessment of trajectories between the trajectory information coming from a traffic motion prediction module and what is actually needed. Closing this gap necessitates advancements in prediction beyond current practices. Existing prediction models yield joint predictions of agents’ future trajectories with uncertainty weights or marginal Gaussian probability density functions (PDFs) for single agents. Although, these methods achieve high accurate trajectory predictions, they only provide little or no information about the dependencies of interacting agents. Since traffic is a process of highly interdependent agents, whose actions directly influence their mutual behavior, the existing methods are not sufficient to reliably assess the risk of future trajectories. This paper addresses that gap by introducing a novel approach to motion prediction, focusing on predicting agent-pair covariance matrices in a "scene-centric" manner, which can then be used to model Gaussian joint PDFs for all agent-pairs in a scene. We propose a model capable of predicting those agent-pair covariance matrices, leveraging an enhanced awareness of interactions. Utilizing the prediction results of our model, this work forms the foundation for comprehensive risk assessment with statistically based methods for analyzing agents’ relations by their joint PDFs.

Accurate intrinsic calibration is essential for camera-based 3D perception, yet, it typically requires targets of well-known geometry. Here, we propose a camera self-calibration approach that infers camera intrinsics during application, from monocular videos in the wild. We propose to explicitly model projection functions and multi-view geometry, while leveraging the capabilities of deep neural networks for feature extraction and matching. To achieve this, we build upon recent research on integrating bundle adjustment into deep learning models, and introduce a self-calibrating bundle adjustment layer. The self-calibrating bundle adjustment layer optimizes camera intrinsics through classical Gauß-Newton steps and can be adapted to different camera models without re-training. As a specific realization, we implemented this layer within the deep visual SLAM system DROID-SLAM, and show that the resulting model, DroidCalib, yields state-of-the-art calibration accuracy across multiple public datasets. Our results suggest that the model generalizes to unseen environments and different camera models, including significant lens distortion. Thereby, the approach enables performing 3D perception tasks without prior knowledge about the camera. Code is available at https://github.com/boschresearch/droidcalib.

Connected automated vehicles (CAV), which incorporate vehicle-to-vehicle (V2V) communication into their motion planning, are expected to provide a wide range of benefits for individual and overall traffic flow. A frequent constraint or required precondition is that compatible CAVs must already be available in traffic at high penetration rates. Achieving such penetration rates incrementally before providing ample benefits for users presents a chicken-and-egg problem that is common in connected driving development. Based on the example of a cooperative driving function for bottleneck traffic flows (e.g. at a roadblock), we illustrate how such an evolutionary, incremental introduction can be achieved under transparent assumptions and objectives. To this end, we analyze the challenge from the perspectives of automation technology, traffic flow, human factors and market, and present a principle that 1) accounts for individual requirements from each domain; 2) provides benefits for any penetration rate of compatible CAVs between 0 % and 100 % as well as upward-compatibility for expected future developments in traffic; 3) can strictly limit the negative effects of cooperation for any participant and 4) can be implemented with close-to-market technology. We discuss the technical implementation as well as the effect on traffic flow over a wide parameter spectrum for human and technical aspects.

As an important part of automated vehicle development and testing, simulation makes heavy use of driver models to reproduce the behavior of traffic participants. Due to their simplicity, most models fail to capture driver behavior in interactive situations like lane changes or merging, where drivers need to consider multiple vehicles simultaneously and smoothly approach gaps. We propose the Gap APproaching Intelligent Driver Model (GAP-IDM), an extension of IDM that takes an arbitrary number of target vehicles into account and produces realistic behavior for approaching traffic gaps, even when the ego vehicle has to overtake or fall behind target vehicles. To this end, we use a target distance rectification to produce smooth behaviors even for small or negative distances, and to enforce time or distance limits on the maneuver. We evaluate the proposed model in an optional and a necessary lane change scenario and demonstrate that it generates realistic driving behavior. Possible applications of our model include simulations of interactive scenarios, development of complex driver models with multiple target vehicles, or the use as a low-level policy in a high-level behavior planning module.

The Large-scale 3D reconstruction, texturing and semantic mapping are nowadays widely used for automated driving vehicles, virtual reality and automatic data generation. However, most approaches are developed for RGB-D cameras with colored dense point clouds and not suitable for large-scale outdoor environments using sparse LiDAR point clouds. Since a 3D surface can be usually observed from multiple camera images with different view poses, an optimal image patch selection for the texturing and an optimal semantic class estimation for the semantic mapping are still challenging.To address these problems, we propose a novel 3D reconstruction, texturing and semantic mapping system using LiDAR and camera sensors. An Adaptive Truncated Signed Distance Function is introduced to describe surfaces implicitly, which can deal with different LiDAR point sparsities and improve model quality. The from this implicit function extracted triangle mesh map is then textured from a series of registered camera images by applying an optimal image patch selection strategy. Besides that, a Markov Random Field-based data fusion approach is proposed to estimate the optimal semantic class for each triangle mesh. Our approach is evaluated on a synthetic dataset, the KITTI dataset and a dataset recorded with our experimental vehicle. The results show that the 3D models generated using our approach are more accurate in comparison to using other state-of-the-art approaches. The texturing and semantic mapping achieve also very promising results.

High Definition (HD) maps are necessary for many applications of automated driving (AD), but their manual creation and maintenance is very costly. Vehicle fleet data from series production vehicles can be used to automatically generate HD maps, but the data is often incomplete and noisy. We propose a system for the generation of HD maps from vehicle fleet data, which is tolerant to missing or misclassified detections and can handle drives with multiple routes, generating a single complete map, model-free and without prior reference lines. Using randomly selected drives as pivot drives, a step-wise lateral sampling of detections is performed. These sampled points are then clustered and aligned using Expectation Maximization (EM), estimating a lateral offset for each drive to compensate localization errors. The clustered points are replaced with the maxima of their probability density function (PDF) and connected to form polylines using a modified rectangular linear assignment algorithm. The data from vehicles on varying routes is then fused into a hierarchical singular map graph. The proposed approach achieves an average accuracy below 0.5 meters compared to a hand annotated ground truth map, as well as correctly resolving lane splits and merges, proving the feasibility of the use of vehicle fleet data for the generation of highway HD maps.

Automated driving systems need to behave as human-like as possible, especially in highly interactive scenarios. In this way, the behavior can be better interpreted and predicted by other traffic participants, in order to prevent misunderstanding, and in the worst case, accidents. With this purpose, more and more human-driven trajectories in real traffic are recorded, making it possible to learn human-like driving styles. In this paper, we extend our previous behavior cloning approach, which has been successfully applied to highway driving, to generate high-level decisions for unsignalized intersections that are challenging during urban driving. Unlike many other approaches that utilize neural networks, either for end-to-end behavior cloning or for approximating Q-functions in reinforcement learning, where their decisions are intractable to understand, the output decisions of our approach are interpretable and easy to track. Meanwhile, the driving decisions are provably safe under reasonable assumptions by generalizing the Responsibility-Sensitive Safety (RSS) concept to complex intersections. Simulation evaluations show that our learned policy produces a more human-like behavior, and meanwhile, balances driving efficiency, comfort, perceived safety, and politeness better.

While complete localization approaches are widely studied in the literature, their data association and data representation subprocesses usually go unnoticed. However, both are a key part of the final pose estimation. In this work, we present DA-LMR (Delta-Angle Lane Marking Representation), a robust data representation in the context of localization approaches. We propose a representation of lane markings that encodes how a curve changes in each point and includes this information in an additional dimension, thus providing a more detailed geometric structure description of the data. We also propose DC-SAC (Distance-Compatible Sample Consensus), a data association method. This is a heuristic version of RANSAC that dramatically reduces the hypothesis space by distance compatibility restrictions. We compare the presented methods with some state-of-the-art data representation and data association approaches in different noisy scenarios. The DA-LMR and DC-SAC produce the most promising combination among those compared, reaching 98.1 % in precision and 99.7% in recall for noisy data with 0.5 m of standard deviation.

Comprehensive and correct state estimation with meaningful uncertainties is the basis of object-based perception for automated mobile platforms. According to fatality statistics, the most endangered group of vulnerable road users are single-track two-wheelers (ST2W), consisting mainly of cyclists, motorcyclists, and scooter riders. Due to counter-steering, they need more time to adjust their driving state to a new situation compared to four-wheelers that can directly steer in the desired direction without loosing balance. Therefore, the roll angle gives valuable information about possible future actions, especially for short, safety-critical prediction horizons. In this work, we present a basic, robust state estimation approach that is tailored to ST2W. Due to the lack of publicly available ST2W datasets with dynamic driving maneuvers and a highly accurate state, we recorded, labeled and published three different ST2W tracks ourselves. According to our results, the roll angle can be estimated bias-free with a standard deviation of between 4.4○to 6.4○, outperforming the chosen baseline.

In this paper, we present TEScalib, a novel extrinsic self-calibration approach of LiDAR and stereo camera using the geometric and photometric information of surrounding environments without any calibration targets for automated driving vehicles. Since LiDAR and stereo camera are widely used for sensor data fusion on automated driving vehicles, their extrinsic calibration is highly important. However, most of the LiDAR and stereo camera calibration approaches are mainly target-based and therefore time consuming. Even the newly developed targetless approaches in last years are either inaccurate or unsuitable for driving platforms. To address those problems, we introduce TEScalib. By applying a 3D mesh reconstruction-based point cloud registration, the geometric information is used to estimate the LiDAR to stereo camera extrinsic parameters accurately and robustly. To calibrate the stereo camera, a photometric error function is builded and the LiDAR depth is involved to transform key points from one camera to another. During driving, these two parts are processed iteratively. Besides that, we also propose an uncertainty analysis for reflecting the reliability of the estimated extrinsic parameters. Our TEScalib approach evaluated on the KITTI dataset achieves very promising results.

While automated research vehicles are already populating the roads, their commercial availability at scale is still to come. Presumably, one of the key challenges is to derive behaviors that are safe and comfortable but at the same time not overcautious, despite considerable uncertainties. These uncertainties stem from imperfect perception, occlusions and limited sensor range, but also from the unknown future behavior of other traffic participants. A holistic uncertainty treatment, for example in a general POMDP formulation, often induces a strong limitation on the action space due to the need for real-time capability. Further, related approaches often do not account for the need for verifiable safety, including traffic rule compliance. The proposed approach is targeted towards scenarios with clear precedence. It is based on an MDP with an action-independent belief (AIB-MDP): We assume that the future belief over the trajectories of other traffic participants is independent of the ego vehicle's behavior. Thus, the future belief can be predicted and simplified in an upstream module, independent of motion planning. This modularization facilitates subsequent ego motion planning in a continuous action space despite the thorough uncertainty consideration. The improved performance compared to state-of-the-art is demonstrated in three example scenarios.

The Connected and Automated Vehicles (CAVs) technology is in continuous growth, especially during the last decade. Accordingly, the development of the V2X protocols grasps the attention of many researchers. However, there is a gap in structuring a holistic architecture that considers the individual agent modules in the transportation system along with their cooperation, especially in the decision-making layer. Thus, the main contribution of this article is to build a Robotics Operating System (ROS)-based architecture, simulating the interaction between the infrastructure and the car agents. The architecture is designed with some essential characteristics: configurable, modular, comprehensive, usable, and generic. In the architecture, the car agent is built to include four main modules to accomplish a high level of autonomy; Localization, Planning, Control, and communication. Meanwhile, the road agent is composed of two modules; mapping and communication. Both agents are considered ROS nodes that can communicate through custom ROS messages through Service/Client architecture using developed V2I protocols. These protocols are responsible for registering the car while joining the road and assigning a unique identifier to each joined car. Moreover, the road assigns the desired speed suiting the road profile and a lane to be kept by the car. The architecture is verified on a designed track map on Webots simulator. A case study of 10 heterogeneous cars is demonstrated to observe the architecture performance. The architecture showed promising results, successfully controlling the cars along with the designed map with acceptable error via the V2I protocols, allowing a human-like driving experience.

One way to increase user acceptance in automated vehicles is to explain their driving decisions, but current methods still involve human interpretations and are thus prone to errors. Therefore, the presented method formulates summaries that clarify the automated vehicle’s driving decision by extracting all necessary information automatically from the planning algorithm. This paper shows the generation of three exemplary statement types and their validation with an online survey that investigated users’ preferences. The results suggest that participants favor statements describing information that affect the driving decision as well as applicable traffic rules. Additionally, individual information needs should be considered when constructing modular explanations. Although this analysis does not consider sophisticated human machine interfaces nor real traffic scenarios, it does show, for the first time, how satisfying statements can be generated using a planning algorithm without any human-induced bias. This is an important step towards self-contained transparency of automated driving functions and can therefore lay the basis for future human machine interfaces.

As automated vehicles are expected to significantly reduce the number of fatalities in road traffic, ensuring their safety is one of the most critical challenges in the industry today. The Responsibility-Sensitive Safety (RSS) concept is a step towards this goal. RSS formalizes reasonable boundaries for the foreseeable worst-case behavior of traffic participants by defining clear mathematically proven rules. All parameters used in RSS have a physical meaning and are thus well understandable, but the values of these parameters have a crucial effect on the applicability of this approach. Choosing too conservative parameter values may impact traffic flow, while the opposite could lead to uncomfortable or potentially unsafe driving behavior. While a majority of work concentrated on the longitudinal use case, in this work we focus on finding reasonable parameter values for lateral safety. We propose scopes and parameter sets for the RSS minimum lateral distance extracted from human driving behavior that allow for a comfortable driving behavior while not hindering traffic flow.

The automation of a truck-trailer offers enormous potential for safe and efficient transportation. The optimal control approaches, e.g., MPC, have significantly improved the tracking accuracy and the smoothness of the lateral control of vehicles. MPC application for a truck-trailer is complex compared to a car as the system behavior is different for forwarding and reversing. In this paper, we propose a lateral MPC algorithm for a truck-trailer, where we linearize the system dynamics around a nominal trajectory computed using a control law. The control law formulated as cascade control computes the nominal trajectory, an initial guess to the optimization process. The nominal trajectory lies in the vicinity of the optimal trajectory. We linearize the system dynamics around the computed nominal trajectory, reducing the linearization errors. The region of validity of the linearized system dynamics is narrow due to the system’s instability during reverse driving. A quadratic optimization problem subjective to the linear dynamics of the truck-trailer and state and input constraints defines the optimal control problem. The nominal trajectory is stable over the prediction horizon while reversing, so is the linear prediction model, improving optimization feasibility. Further, the discretization errors are also reduced using a small discrete step and integrating the model multiple times between two prediction steps. We tested the developed MPC approach on a prototypical full-scale truck-trailer system and discussed results. The developed MPC is considerably fast and accurate for real-time application.

We present a path optimization approach that ensures driveability while considering a vehicle’s lateral dynamics. The lateral dynamics are non-holonomic; therefore, a vehicle cannot follow a path with abrupt changes even with infinitely fast steering. The curvature and sharpness, i.e., the rate change of curvature with respect to the traveled distance, must be continuous to track a defined reference path efficiently. Existing path optimization techniques typically include sharpness limitations but not sharpness continuity. The sharpness discontinuity is especially problematic for heavy-duty vehicles because their actuator dynamics are even slower than cars. We propose an algorithm that constructs a sparsified sharpness continuous path for a given reference path considering the limits on sharpness and its derivative, which subsequently addresses the torque restrictions of the actuator. The sharpness continuous path needs less steering effort and reduces mechanical stress and fatigue in the steering unit. We compare and present the outcomes for each of the three different types of optimized paths. Simulation results demonstrate that computed sharpness continuous path profiles reduce lateral jerks, enhancing comfort and driveability.

Currently, there is a gap in motion planning approaches. On the one hand, there are optimization-based motion planning techniques which can guarantee safety and feasibility, but are either slow and cooperative or fast and uncooperative. On the other hand, there are learned approaches that are fast and cooperative, but cannot give these desirable guarantees.We propose to combine model predictive contouring control (MPCC) with sophisticated collision avoidance formulations to bridge this gap. By optimizing the total utility of all traffic participants, a cooperative, safe, and feasible trajectory can be planned in real time.Examination of various collision avoidance constraints allows to obtain considerate trajectories while preserving real-time capabilities. A novel inter-stage constraint formulation allows to introduce time-based distance measures in time-discretized MPC formulations.We evaluate the resulting motion planner in various scenarios, comparing two state-of-the-art solvers.

When detecting objects, depth sensors are not always available, requiring 3D object detection from monocular images. However, for many object classes, datasets with 3D annotations are missing. Recent monocular 3D object detection methods lack the semantic diversity needed for autonomous systems, because of missing 3D ground truth data for static classes such as poles and traffic lights. To overcome this gap we combine a large scale dataset for 2D object detection, with an unlabeled dataset containing depth measurements. We lift 2D object detections of the depth dataset into the 3D domain, associating detections with corresponding depth values. This leverages 2D annotated datasets to enable semantically rich 3D object detection, without extra labelling effort. We train an object detection model with mixed batches and evaluate it comparing the predicted depth with the projected centerpoint depth of cars manually annotated in 3D space. The result is a monocular object detector that can predict 3D positions of up to 37 static and dynamic object classes from camera only.

Most approaches to camera calibration rely on calibration targets of well-known geometry. During data acquisition, calibration target and camera system are typically moved w.r.t. each other, to allow image coverage and perspective versatility. We show that moving the target can lead to small temporary deformations of the target, which can introduce significant errors into the calibration result. While static inaccuracies of calibration targets have been addressed in previous works, to our knowledge, none of the existing approaches can capture time-varying, dynamic deformations. To achieve high-accuracy calibrations despite moving the target, we propose a way to explicitly model dynamic target deformations in camera calibration. This is achieved by using a low-dimensional deformation model with only few parameters per image, which can be optimized jointly with target poses and intrinsics. We demonstrate the effectiveness of modeling dynamic deformations using different calibration targets and show its significance in a structure-from-motion application.

At the heart of all automated driving systems is the ability to sense the surroundings,e.g.,through semantic segmentation of LiDAR sequences, which experienced a remarkable progress due to the release of large datasets such as SemanticKITTI and nuScenes-LidarSeg. While most previous works focus onsparsesegmentation of the LiDAR input,denseoutput masks provide self-driving cars with almost complete environment information. In this paper, we introduce MASS - a Multi-Attentional Semantic Segmentation model specifically built for dense top-view understanding of the driving scenes. Our framework operates on pillar- and occupancy features and comprises three attention-based building blocks: (1) a keypoint-driven graph attention, (2) an LSTM-based attention computed from a vector embedding of the spatial input, and (3) a pillar-based attention, resulting in a dense 360° segmentation mask. With extensive experiments on both, SemanticKITTI and nuScenes-LidarSeg, we quantitatively demonstrate the effectiveness of our model, outperforming the state of the art by 19.0% on SemanticKITTI and reaching 30.4% in mIoU on nuScenes-LidarSeg, where MASS is the first work addressing the dense segmentation task. Furthermore, our multi-attention model is shown to be very effective for 3D object detection validated on the KITTI-3D dataset, showcasing its high generalizability to other tasks related to 3D vision.

While the automatic creation of maps for localization is a widely tackled problem, the automatic inference of higher layers of HD maps is not. Additionally, approaches that learn from maps require richer and more precise landmarks than currently available.In this work, we fuse semantic detections from a monocular camera with depth and orientation estimation from lidar to automatically detect, track and map parametric, semantic map elements. We propose the use of tailored representations that are minimal in the number of parameters, making the map compact and the estimation robust and precise enough to enable map inference even from single frame detections. As examples, we map traffic signs, traffic lights and poles using upright rectangles and cylinders.After robust multi-view optimization, traffic lights and signs have a mean absolute position error of below 10 cm, extent estimates are below 5 cm and orientation MAE is below 6◦. This proves the suitability as automatically generated, pixel-accurate ground truth, reducing the task of ground truth generation from tedious 3D annotation to a post-processing of misdetections.

Despite recent advances in reinforcement learning (RL), its application in safety critical domains like autonomous vehicles is still challenging. Although penalizing RL agents for risky situations can help to learn safe policies, it may also lead to highly conservative behavior. In this paper, we propose a distributional RL framework in order to learn adaptive policies which allow to tune their level of conservativity at run-time based on the desired comfort and utility. Using a proactive safety verification approach, the proposed framework can guarantee that actions generated from RL are failsafe according to the worst-case assumptions. Concurrently, the policy is encouraged to minimize safety interference and generate more comfortable behavior. We trained and evaluated the proposed approach and baseline policies using a high level simulator with a variety of randomized scenarios including several corner cases which rarely happen in reality but are very crucial. In light of our experiments, the behavior of policies learned using distributional RL is adaptive at run-time and robust to the environment uncertainty. Quantitatively, the learned distributional RL agent reduces the average driving time more than 50% compared to the normal DQN policy. It also requires 83% less safety interference compared to the rule-based policy while only slightly increasing the average driving time. We also study sensitivity of the learned policy in environments with higher perception noise and show that our algorithm learns policies that can still drive reliable when the perception noise is two times higher than in the training configuration in automated merging and crossing at occluded intersections.

Semantic understanding of the surrounding environment is essential for automated vehicles. The recent publication of the SemanticKITTI dataset stimulates the research on semantic segmentation of LiDAR point clouds in urban scenarios. While most existing approaches predict sparse pointwise semantic classes for the sparse input LiDAR scan, we propose PillarSegNet to be able to output a dense semantic grid map. In contrast to a previously proposed grid map method, PillarSegNet uses PointNet to learn features directly from the 3D point cloud and then conducts 2D semantic segmentation in the top view. To train and evaluate our approach, we use both sparse and dense ground truth, where the dense ground truth is obtained from multiple superimposed scans. Experimental results on the SemanticKITTI dataset show that PillarSegNet achieves a performance gain of about 10% mIoU over the state-of-the-art grid map method.

Systems and functions that rely on machine learning (ML) are the basis of highly automated driving. An essential task of such ML models is to reliably detect and interpret unusual, new, and potentially dangerous situations. The detection of those situations, which we refer to as corner cases, is highly relevant for successfully developing, applying, and validating automotive perception functions in future vehicles where multiple sensor modalities will be used. A complication for the development of corner case detectors is the lack of consistent definitions, terms, and corner case descriptions, especially when taking into account various automotive sensors. In this work, we provide an application-driven view of corner cases in highly automated driving. To achieve this goal, we first consider existing definitions of the general outlier, novelty, anomaly, and out-of-distribution detection to show relations and differences to corner cases. Moreover, we extend an existing camera-focused systematization of corner cases by adding RADAR (radio detection and ranging) and LiDAR (light detection and ranging) sensors. For this, we describe an exemplary toolchain for data acquisition and processing, highlighting the interfaces of corner case detection. We also define a novel level of corner cases, the method layer corner cases, which appear due to uncertainty inherent in the methodology.

The need for safety in automated driving is undisputed. Since automated vehicles are expected to reduce the number of fatalities in road traffic significantly, hundreds of millions of test kilometers would be required for statistical safety validation [1]. Physics-based safety verification approaches are promising in order to reduce this validation effort. Towards this goal, Mobileye introduced the concept of Responsibility-Sensitive Safety (RSS). In RSS, bounds for the reasonable worst-case behavior of traffic participants are assumed to be given, such as the reaction time or the maximum deceleration. These parameters have a crucial effect on the applicability of the approach: choosing conservative parameters likely hinders traffic flow, while the opposite could lead to collisions, as the assumptions are violated. Thus, in this work, we focus on finding reasonable parameters of RSS. Based on the physical limits, legal requirements and human driving behavior, we propose scopes and parameter sets that allow for a sound safety verification while not hindering traffic flow. Furthermore, we present an approach that explains seemingly frequent human drivers' RSS violations on highways and may lead to a useful extension of RSS.

High definition (HD) maps can fail by becoming outdated. To still use them safely for automated driving, they need to be verified or updated, both requiring methods for change detection. We propose two significant improvements for HD map change detection that do not require a highly accurate localization prior as localization quickly fails or cannot be trusted in an outdated map. Given a very coarse localization prior, we group stored or measured map features in spatially and semantically separable feature groups. These feature groups are not only intuitive, like the sequence of leftmost dashed lane markings, but changes are also highly correlated within them. The first contribution improves the way internal consistency of each feature group is assured by using boosted classification trees. Additionally, a mutual evaluation scheme is added for all seemingly unchanged feature groups. Always one feature group is used for localization by feature alignment while each other group's alignment is checked for compatibility. Two voting schemes are presented that allow a more or less sensitive change detection on the level of proposed groups. In contrast to almost all other approaches, our approach allows to use still valid parts of the map for automated driving and to update the changed parts. We evaluate our approach on a previously published map verification dataset [1], showing that the number of undetected map changes can be reduced by up to 31 % compared to state of the art using boosted classification trees, at the same time reducing false positive rates by up to 50 %. The additional mutual evaluation step is able to uncover a whole category of previously undetectable changes and reduces undetected changes by an extra 15 %.

Image-based 3D object detection is an inevitable part of autonomous driving because cheap onboard cameras are already available in most modern cars. Because of the accurate depth information, currently most state-of-the-art 3D object detectors heavily rely on LiDAR data. In this paper, we propose a pipeline which lifts the results of existing vision-based 2D algorithms to 3D detections using only cameras as a cost-effective alternative to LiDAR. In contrast to existing approaches, we focus not only on cars but on all types of road users. To the best of our knowledge, we are the first using a 2D CNN to process the point cloud for each 2D detection to keep the computational effort as low as possible. Our evaluation on the challenging KITTI 3D object detection benchmark shows results comparable to state-of-the-art image-based approaches while having a runtime of only a third.

Easy, yet robust long-term localization is still an open topic in research. Existing approaches require either dense maps, expensive sensors, specialized map features or proprietary detectors.We propose using semantic segmentation on a monocular camera to localize directly in a HD map as used for automated driving. This combines lightweight, yet powerful HD maps with the simplicity of monocular vision and the flexibility of neural networks.The major challenges arising from this combination are data association and robustness against misdetections. Association is solved efficiently by applying distance transform on binary per-class images. This provides not only a fast lookup table for a smooth gradient as needed for pose-graph optimization, but also dynamic association by default.A sliding-window pose graph optimization combines single image detections with vehicle odometry, smoothing results and helping overcome even misclassifications in consecutive frames.Evaluation against a highly accurate 6D visual localization shows that our approach can achieve accuracy levels as required for automated driving, being one of the most lightweight and flexible methods to do so.

In this paper, we consider the transformation of laser range measurements into a top-view grid map representation to approach the task of LiDAR-only semantic segmentation. Since the recent publication of the SemanticKITTI data set, researchers are now able to study semantic segmentation of urban LiDAR sequences based on a reasonable amount of data. While other approaches propose to directly learn on the 3D point clouds, we are exploiting a grid map framework to extract relevant information and represent them by using multi-layer grid maps. This representation allows us to use well-studied deep learning architectures from the image domain to predict a dense semantic grid map using only the sparse input data of a single LiDAR scan. We compare single-layer and multi-layer approaches and demonstrate the benefit of a multi-layer grid map input. Since the grid map representation allows us to predict a dense, 360° semantic environment representation, we further develop a method to combine the semantic information from multiple scans and create dense ground truth grids. This method allows us to evaluate and compare the performance of our models not only based on grid cells with a detection, but on the full visible measurement range.

Recent trajectory or maneuver planning approaches in automated driving show the tendency to get more complex or even become a black box. Thus, an algorithm's decisions become less transparent, especially when uncertain input parameters have a large influence. To identify those input parameters whose uncertainty is more relevant than others', Morris' method of elementary effects is used here [1]. It is a quantitative sensitivity analysis that classifies the inputs into relevant and irrelevant, depending on how sensitive the algorithm reacts to changes in the input. The method is adapted to analyze the behavior of a car-following and lane-changing model during an overtaking maneuver with two vehicles. The results show that Morris' method is capable of determining important parameters for each situation. It is even possible to identify boundaries for the necessary accuracy of each input. With this, we are able to determine input parameter ranges for which the planning algorithm is able to produce reliable output.

Reinforcement learning is nowadays a popular framework for solving different decision making problems in automated driving. However, there are still some remaining crucial challenges that need to be addressed for providing more reliable policies. In this paper, we propose a generic risk-aware DQN approach in order to learn high level actions for driving through unsignalized occluded intersections. The proposed state representation provides lane based information which allows to be used for multi-lane scenarios. Moreover, we propose a risk based reward function which punishes risky situations instead of only collision failures. Such rewarding approach helps to incorporate risk prediction into our deep Q network and learn more reliable policies which are safer in challenging situations. The efficiency of the proposed approach is compared with a DQN learned with conventional collision based rewarding scheme and also with a rule-based intersection navigation policy. Evaluation results show that the proposed approach outperforms both of these methods. It provides safer actions than collision-aware DQN approach and is less overcautious than the rule-based policy.

Vehicles nowadays are equipped with several assistance functions for e.g. Cruise Control and Lane Keeping Assist. Classical lateral control approaches such as pure pursuit, Stanley used for lane keeping assist provide good path tracking precision. However the operational domain for these approaches is limited i.e. highway driving. An adaptation of control parameters can increase the operational domain, but its difficult to tune classical approaches for the whole range of automated driving maneuvers. Apart from these classical approaches, optimization approaches are also used for lateral trajectory stabilization. The optimization based approaches have a wider operational domain, but the feasibility and real time execution remain some open issues. In this paper, we combine a classical approach and an optimization method for lateral trajectory stabilization. We present a method to optimize the control input calculated using a classical control approach i.e. pure pursuit based on a performance criteria. The performance criteria weighs the precision and comfort requirements. A non- linear optimization based on Differential Dynamic Programming (DDP) is used to solve the optimization problem. The calculated optimal trajectory is finally evaluated using a line search method to ensure the convergence and to verify the optimization policy. The approach is demonstrated on a full scale automated truck prototype and the experimental results are discussed.

The stabilization of articulated truck i.e. truck-semitrailer is a complex problem due to the instable dynamics. In this paper, a lateral trajectory stabilization algorithm for reverse driving of truck-semitrailer is proposed. A cascade control is presented for lateral trajectory stabilization of truck-semitrailer. A high-level path tracking algorithm handles the path tracking of a virtual vehicle which is an equivalent model for the semitrailer. A low-level Linear Quadratic Regulator (LQR) stabilizes the hitch angle. The path tracking problem is formulated as a linear time-varying differential dynamic programming problem subjective to virtual vehicle dynamics in a receding horizon fashion. The virtual vehicle dynamics are defined by the single track kinematic vehicle model and the hitch angle dynamics is used for formulating the Linear Quadratic Regulator problem. The approach is demonstrated with replays of real-world path tracking scenarios on a full-scale truck-semitrailer prototype.

Pedestrian crossing prediction is a crucial task for autonomous driving. Numerous studies show that an early estimation of the pedestrian's intention can decrease or even avoid a high percentage of accidents. In this paper, different variations of a deep learning system are proposed to attempt to solve this problem. The proposed models are composed of two parts: a CNN-based feature extractor and an RNN module. All the models were trained and tested on the JAAD dataset. The results obtained indicate that the choice of the features extraction method, the inclusion of additional variables such as pedestrian gaze direction and discrete orientation, and the chosen RNN type have a significant impact on the final performance.

High definition (HD) maps have proven to be a necessary component for safe and comfortable automated driving (AD) [1]. Naïvely verifying HD maps requires an accurate localization prior in order to correctly associate measurements with map data. In periodic environments, such as highways, localization results are often ambiguous - in particular in longitudinal direction. To still be able to verify an HD map, we propose the use of quasi-continuous 1D signals that can be computed without pointwise association. These signals can be chosen to change significantly when the map has changed while they only change rarely or slowly along the road, making them robust against localization errors. A spatio-semantic clustering yields intuitive groups of map features. These groups are then ordered using a robust projection approach, yielding quasi-continuous 1D signals. Such signals can be computed for map and measurement data and their comparison allows detecting road changes. The purposeful design of the signals and their computation only requires lane-level lateral localization and a coarse longitudinal prior, vastly relaxing the requirements on prior localization results compared to the current state of the art. With four example signals, we demonstrate the effectiveness of our approach on a map verification dataset [2], detecting between 49 % and 98 % of all changed features at false alarm rates usually below 15 %. Detecting changes per feature allows to still use unchanged features for AD functions. When omitting this ability and aggregating all features, 98 % of all changed road sections can be detected successfully.successfully.

Navigation in congested environments is a challenge for autonomous vehicles and they should consider collision risk metric into their driving behavior. In this paper, we propose a novel two-fold indicator: On the one hand, single-shot risk works in space domain, considering geometries, locations and the velocities of the obstacles in the current scene. On the other hand, long-term risk considers the evolution of the current scene and provides risk values in time domain. The map information and different prediction models (e.g. reachable sets, probabilistic) are considered in the long-term risk, which can then be used in trajectory planning or decision making approaches. Our method can be applied to scenarios with arbitrary road topologies (intersections, roundabouts, highway, etc.) and it is suitable regardless of the scene prediction method. We formulate the single-shot (or short-term) risk with one single function fitted using Monte Carlo (MC) Simulations. The results are evaluated in real scenarios using HighD dataset and compared with other risk indicators such as THW and TTC. In addition, it is applied to a simple trajectory planner in order to demonstrate that the proposed approach imitates human driving style.

We present our approach to unsupervised domain adaptation for single-stage object detectors on top-view grid maps in automated driving scenarios. Our goal is to train a robust object detector on grid maps generated from custom sensor data and setups. We first introduce a single-stage object detector for grid maps based on RetinaNet. We then extend our model by image- and instance-level domain classifiers at different feature pyramid levels which are trained in an adversarial manner. This allows us to train robust object detectors for unlabeled domains. We evaluate our approach quantitatively on the nuScenes and KITTI benchmarks and present qualitative domain adaptation results for unlabeled measurements recorded by our experimental vehicle. Our results demonstrate that object detection accuracy for unlabeled domains can be improved by applying our domain adaptation strategy.

The breakthrough of intelligent vehicles will also be determined by the safety gain they provide. In order to perform accident avoiding reactions at the earliest point in time possible, predictions about the future behavior of other traffic participants are needed. The most exposed share of traffic participants regarding this issue are single-track two-wheelers (1T2W): they share the road with cars and trucks but are not as agile due to their kinematics. Furthermore, they are faster than pedestrians but comparably vulnerable as those. In order to guarantee their safety, we make use of their movement restricting kinematics. We simulate three typical classes of 1T2W under conservative assumptions about their agility in order to generate a spatial region in which they have to be due to physics after a fixed prediction horizon of up to 1.5 seconds. The proposed approach was verified in experiments with real high-dynamic driving maneuvers.

To increase the spatial and temporal resolution of weather forecasts and thereby ensure safe autonomous driving functions, a denser network of measurements is necessary. This paper presents the project Fleet Weather Map, which investigates the potential of using floating car data as a source for meteorological data. Furthermore, the need for bias corrections and quality control of the raw signals is shown. The potential of the approach used gives first promising results and aims for increasing the forecast step width to 5 minutes.

Highly accurate localization with very limited amount of memory and computational power is one of the big challenges for next generation series cars. We propose localization based on geometric primitives which are compact in representation and further valuable for other tasks like planning and behavior generation. The primitives lack distinctive signature which makes association between detections and map elements highly ambiguous. We resolve ambiguities early in the pipeline by online building up a local map which is key to runtime efficiency. Further, we introduce a new framework to fuse association and odometry measurements based on robust pose graph optimization.We evaluate our localization framework on over 30 min of data recorded in urban scenarios. Our map is memory efficient with less than 8 kB/km and we achieve high localization accuracy with a mean position error of less than 10 cm and a mean yaw angle error of less than 0. 25° at a localization update rate of 50Hz.

In this paper, we present a novel geo-referencing approach to align trajectories to an aerial imagery using pole and road marking features. Currently, digital maps are indispensable for automated driving. However, due to the low precision and reliability of Global navigation satellite systems (GNSS) particularly in urban areas, fusing trajectories of independent recording sessions and different regions is a challenging task. To bypass the flaws from direct incorporation of GNSS measurements for geo-referencing, the usage of an aerial imagery seems promising. Furthermore, an accurate geo-referencing improves the global map accuracy and allows to estimate the sensor calibration error. To match extracted features from sensor observations to landmarks extracted from an aerial imagery robustly, a matching approach using RANSAC is applied in a sliding window. For that, we assume that the trajectories are roughly referenced to the imagery which can be achieved by rough GNSS measurements from a low-cost GNSS receiver. Finally, we align the initial trajectories precisely to the aerial imagery by minimizing a geometric cost function comprising all determined matches. Evaluations show that our algorithm yields trajectories which are accurately referenced to the used aerial imagery.

Behavior planning in urban environments must consider the various existing uncertainties in an explicit way. This work proposes a behavior planner, based on a POMDP formulation, that explicitly considers possibly occluded vehicles. The future field of view of the autonomous car is predicted over the whole planning horizon. Both, occlusions which are generated by static as well as generated by dynamic objects are hereby considered. We use Monte Carlo sampling to generate possible future episodes that are used to derive an optimized policy. The sampled episodes consider the uncertain behavior of the known traffic participants as well as the existence probability of so-called phantom vehicles in occluded areas. By representing all possible, occluded vehicle configurations by its reachable set instead of single particles, a very efficient representation is found. Therefore, we ensure to consider all possible configurations which may drive out of the occluded area in our optimized policy. We propose a generic formulation of the POMDP problem that can be applied to various scenarios for urban driving. Its performance is demonstrated by using simulation scenarios at intersections including multiple vehicles and occlusions caused by static and dynamic objects. It is shown, that the autonomous vehicle approaches occluded areas by far less conservative than a baseline strategy which considers only the current field of view (fov). This is because various, future scenarios are already considered in the policy. In fact, we show that our planner is able to drive nearly the same trajectories as an omniscient planner would.

Emergency scenarios may require trajectory planning at actuator and friction limits within tight obstacle constraints. This paper presents an approach to handle such scenarios using Linear Time Varying Model Predictive Control, in the presence of both static and dynamic obstacles. With the proposed approach, the controller transitions between a kinematic vehicle model, which is stable at low velocities, and a dynamic vehicle model, which is more accurate at high velocities, based on a threshold velocity. This ensures the functionality of the controller in emergency braking scenarios and enables manoeuvre initialization with a full braking trajectory. An adjustable safety distance concept is introduced that considers the criticality of the scenario and tightens or loosens the obstacle avoidance and friction constraints accordingly. Simulation results show that the controller is capable of handling various critical situations, including those when the vehicle comes to a stop at the end of a high speed evasive manoeuvre. Furthermore, it is more effective at keeping a safety distance from obstacles in such manoeuvres than by using fixed safety distance concept.

Reliable and precise object tracking is an essential requirement for automated driving. The majority of LiDAR-based tracking algorithms resort to raw range measurements only. In contrast, we propose a novel method to extract compact and salient features from LiDAR intensities. Using the example of an evasive steering maneuver of a leading vehicle, we show that leveraging these intensity features allows for a more accurate estimation of object states. The resulting early detection of target object rotation allows an automated driving system additional time for deriving an appropriate driving policy.

Recent accidents on roads involving cyclists and autonomous vehicles have raised an alarm in the industry to focus more on cyclist safety. Although there are plenty of datasets publicly available in the industry, they don't include enough instances of cyclists in different road conditions to run comprehensive tests. Therefore, in this work, we present a new publicly available Specialized Cyclist Dataset, which focuses solely on cyclist detection. Our dataset was recorded using a monocular RGB camera in various scenarios experienced by cyclists on roads in Autumn and Winter (with snow) for enabling researchers to run rigorous tests in various conditions. There are 62297 total images, about 18200 cyclists instances, and 30 different cyclists. Additionally in the dataset, we present the Specialized cyclist jersey with a diamond pattern designed specifically for improving detection accuracy compared to street clothes. For convenience, we utilized the popular KITTI labeling format and resolution in addition to Full HD resolution.

The recent precision increase in image-based depth estimation encourages to use this type of data for mapping. Recent work proposes different approaches to deal with the problem of occlusion generated by different scene perspectives of stereo cameras. However, there is less attention to depth estimation and inpainting for object removal and object occlusion. In this paper, we study recent inpainting approaches for RGB images and apply these methods on depth maps. We propose a Generative Adversarial Network (GAN) for depth feature extraction to estimate the depth inside a masked area, in order to remove objects on disparity images. Our results show that using depth features on the loss function and on the network architecture, increase the result precision and give to the generated image a depth distribution close to the real data. Our main contribution is a GAN, which estimates depth information in a masked area inside a disparity image.

For a successful introduction of fully automated vehicles, they must behave both provably safe but also convenient, i.e. comfortable and not overcautious. Given the limited sensing capabilities, especially in urban scenarios where buildings and parking vehicles impose occlusions, this is a challenging task. While recent approaches gave first ideas for boundary conditions of safe behavior, an approach for convenient motion planning that fulfills these constraints is still an open issue. Therefore, we utilize and enhance safety approaches for occlusion handling in order to facilitate comfortable and safe motion planning. We consider worst case assumptions, arising from potential objects at critical sensing field edges, along with their probability. With this information, we can ensure to not act overcautiously while still moving provably safe. The potential of our approach is shown in a modified CommonROAD scenario.

We propose a deep convolutional object detector for automated driving applications that also estimates classification, pose and shape uncertainty of each detected object. The input consists of a multi-layer grid map which is well-suited for sensor fusion, free-space estimation and machine learning. Based on the estimated pose and shape uncertainty we approximate object hulls with bounded collision probability which we find helpful for subsequent trajectory planning tasks. We train our models based on the KITTI object detection data set. In a quantitative and qualitative evaluation some models show a similar performance and superior robustness compared to previously developed object detectors. However, our evaluation also points to undesired data set properties which should be addressed when training data-driven models or creating new data sets.

Generating annotations which can be used to train new models has become an independent field of research within machine learning. Its goal is producing highly accurate annotations as cost efficient as possible. 3D point clouds are the common sensor output when recording 3D data from a mobile platform. The latest ways of annotating 3D point clouds include their visualization on a 2D screen. This method contradicts the goal of time-efficient annotating since it is unintuitive and therefore unnecessarily time consuming. We present a novel labeling technique in Virtual Reality. Using our tool, we accelerate the process of data annotation significantly compared to existing approaches. Furthermore, we will give the machine learning community access to our tool and create a new community-labeled dataset for autonomous driving. Furthermore we plan to set up an annotation benchmark in which primarily commercial annotation companies but also researchers active in annotation can take part in. We present results from an experimental plattform based on Oculus Rift indicating a huge potential for VR annotations.

An essential task for Intelligent Transportation System is to obtain precise knowledge of local environments as well as the local (within structured environment) and global vehicle pose. Market entry and large-scale production of autonomous driving functions postulate two elementary constraints. First, the utilized sensor setup has to be both cost-efficient and space-saving. Second, the system has to be fail-safe according to Automotive Safety Integrity Level1D. This paper presents an approach to robustly estimate the vehicles pose both within the current lane and a digital map via pose graph optimization. Outliers and ambiguities are rejected by a suitable loss function and therefore remove the necessity of refined statistical tests. Fall-back solutions, when single sensors are permanently corrupted, are provided by solving various graphs simultaneously. Experimentally, the applicability and performance of the presented approach is demonstrated using an Opel Insignia and its 2D dynamic sensors, with an additional gray-scale camera mounted at the front and at the rear window, a low-cost GNSS receiver and a previously recorded digital map. The graph-based approach has a mean solver time of 14.48 ms and a maximal lateral error below 27.03 cm with a Standard Deviation of 10.05 cm and outperforms the previously presented Extended Kalman Filter and Particle Filter approaches [1].

Accurate traffic participant prediction is the prerequisite for collision avoidance of autonomous vehicles. In this work, we propose to predict pedestrians using goal-directed planning. For this, we infer a mixture density function for possible destinations. We use these destinations as the goal states of a planning stage that performs motion prediction based on common behavior patterns. The patterns are learned by a fully convolutional network operating on maps of the environment. We show that this entire system can be modeled as one monolithic neural network and trained via inverse reinforcement learning. Experimental validation on real world data shows the system's ability to predict both, destinations and trajectories accurately.

Reliable object perception is a vital requirement for automated driving. Despite the availability of precise contour measurements, most state-of-the-art tracking systems still represent object geometry as bounding boxes. However, there are objects operating in public traffic for which the box assumption is highly inappropriate. We therefore propose to represent object contours using 2D polylines. Taking into account the mutual dependence of object poses and shape, our tracking framework targets at a simultaneous estimation of both states. Moreover, we propose to augment scan segments with free-space information at their boundaries and show how this knowledge can be incorporated into the tracking framework and beyond. Evaluation with real scan data shows that our method produces accurate dynamic estimates and consistent shape reconstructions.

In autonomous driving systems a strong relation to highly accurate maps is taken to be inevitable, although street scenes change frequently. However, a preferable system would be to equip the automated cars with a sensor system that is able to navigate urban scenarios without an accurate map. We present a novel pipeline using a deep neural network to detect lane semantics and topology given RGB images. On the basis of this classification, the information about the road scene can be extracted just from the sensor setup supporting mapless autonomous driving. In addition to superseding the huge effort of creating and maintaining highly accurate maps, our system reduces the need for precise localization. Using an extended Cityscapes dataset, we show accurate ego lane detection including lane semantics on challenging scenarios for autonomous driving.

The future of automated driving in urban areas will most probably rely on highly accurate road maps. However, the necessary precision of a localization in such maps has so far only been reached using extra, sensor specific feature layers for localization. In this paper we want to show that it is possible to achieve sufficient accuracy without a separate localization layer. Instead, elements are used that are already contained in high-resolution road maps, such as markings and road borders. For this, we introduce a modular approach in which detections from different detection algorithms are associated with elements in the map and then fused to an absolute pose using an Unscented Kalman Filter. We evaluate our approach using a sensor setup that employs a stereo camera, vehicle odometry and a low-cost GNSS module on a 5km test route covering both narrow urban roads and multi-lane main roads under varying weather conditions. The results show that this approach is capable to be used for highly automated driving, showing an accuracy of 0.08m in typical road scenarios and a is available 98% of the time.

Previously proposed camera calibration methods either use a local camera model in a complex, cumbersome, time consuming and often manual calibration process or a lens specific global camera model, which can be automatically calibrated by simply recording images of chessboards. The drawback of using a global hand crafted camera model is its limited capability of modeling distortions caused by the mounted lens or optical devices in front of the lens like windshields. Therefore, we propose a local camera model based on B-splines which can handle various distortions. Moreover, it will be shown how such a model can be calibrated in an easy-to-use calibration process which were up to now only applicable to global camera models. We demonstrate the benefit of using the proposed local camera model by an extensive evaluation using single and multi-camera setups with different types of lenses, some mounted behind a windshield.

While motion planning techniques for automated vehicles in a reactive and anticipatory manner have already been widely presented, cooperative motion planning has only been addressed recently. For the latter, interaction between traffic participants is crucial. Consequently, simulations where other traffic participants follow simple behavioral rules can no longer beused for development and evaluation. To close this gap, we present a multi vehicle simulation framework. Conventional simulation agents, using a simple, rule-based behavior, are replaced by multiple instances of sophisticated behavior generation algorithms. Thus, development, test and simulative evaluation of cooperative planning approaches is facilitated. The framework is implemented using the Robot Operating System (ROS) and its code will be released open source.

Adverse weather conditions and occlusions in urban environments result in impaired perception. The un-certainties are handled in different modules of an automated vehicle, ranging from sensor level over situation prediction until motion planning. This paper focuses on motion planning given an uncertain environment model with occlusions. We present a method to remain collision free for the worst-case evolution of the given scene. We define criteria that measure the available margins to a collision while considering visibility and interactions and consequently integrate conditions that apply these criteria into an optimization-based motion planner. We show the generality of our method by validating it in several distinct urban scenarios.

A detailed environment representation is a crucial component of automated vehicles. Using single range sensor scans, data is often too sparse and subject to occlusions. Therefore, we present a method to augment occupancy grid maps from single views to be similar to evidential occupancy maps acquired from different views using Deep Learning. To accomplish this, we estimate motion between subsequent range sensor measurements and create an evidential 3D voxel map in an extensive post-processing step. Within this voxel map, we explicitly model uncertainty using evidence theory and create a 2D projection using combination rules. As input for our neural networks, we use a multi-layer grid map consisting of the three features detections, transmissions and intensity, each for ground and non-ground measurements. Finally, we perform a quantitative and qualitative evaluation which shows that different network architectures accurately infer evidential measures in real-time.

This paper presents the concepts and methods utilized by Team AnnieWAY for the 2016 Grand Cooperative Driving Challenge. The paper introduces the automated vehicle BerthaOne. The vehicle, even though being based on the Bertha platform, distinguishes itself from its siblings by its software modules and algorithms. We, therefore, describe its system architecture and algorithms for perception, cooperation and motion planning. In Particular, we present a motion planner that plans different maneuvers flexibly by augmenting the cost function with situation specific cost terms. We subsequently describe the requirements of the 2016 GCDC and evaluate our performance during the competition.

Autonomous driving requires decision making in dynamic and uncertain environments. The uncertainty from the prediction originates from the noisy sensor data and from the fact that the intention of human drivers cannot be directly measured. This problem is formulated as a partially observable Markov decision process (POMDP) with the intention of the other vehicles as hidden variables. The solution of the POMDP is a policy determining the optimal acceleration of the ego vehicle along a preplanned path. Therefore, the policy is optimized for the most likely future scenarios resulting from an interactive, probabilistic motion model for the other vehicles. Considering possible future measurements of the surroundings allows the autonomous car to incorporate the estimated change in future prediction accuracy in the optimal policy. A compact representation allows a low-dimensional state-space so that the problem can be solved online for varying road layouts and number of other vehicles. This is done with a point-based solver in an anytime fashion on a continuous state-space. We show the results with simulations for the crossing of complex (unsignalized) intersections. Our approach performs nearly as good as with full prior information about the intentions of the other vehicles and clearly outperforms reactive approaches.

We present an ego-lane estimation algorithm for downtown lane-level navigation. It is capable of determining the currently used lane reliably, using sensors available in a modern production vehicle, such as odometry, GPS, visual lane-marking detection, and radar-based object detection. The method employs a particle filter with a novel step that combines the importance weight update and sampling. This step avoids performance deterioration in case of sparse particle sets even when the likelihood is very tight compared to the predicted particle set. Preprocessed odometry data allow for a further performance increase. In an extensive test in downtown scenarios on real roads with up to seven lanes, it achieves error probabilities below 1% in the 95th percentile at availabilities above 95%.

In this paper, we present RegNet, the first deep convolutional neural network (CNN) to infer a 6 degrees of freedom (DOF) extrinsic calibration between multimodal sensors, exemplified using a scanning LiDAR and a monocular camera. Compared to existing approaches, RegNet casts all three conventional calibration steps (feature extraction, feature matching and global regression) into a single real-time capable CNN. Our method does not require any human interaction and bridges the gap between classical offline and target-less online calibration approaches as it provides both a stable initial estimation as well as a continuous online correction of the extrinsic parameters. During training we randomly decalibrate our system in order to train RegNet to infer the correspondence between projected depth measurements and RGB image and finally regress the extrinsic calibration. Additionally, with an iterative execution of multiple CNNs, that are trained on different magnitudes of decalibration, our approach compares favorably to state-of-the-art methods in terms of a mean calibration error of 0.28° for the rotational and 6 cm for the translation components even for large decalibrations up to 1.5 m and 20°.

Intelligent vehicles heavily rely on robust and accurate self-localization. Global navigation satellite systems (GNSS) are not reliable in urban environments due to multipath and shadowing effects. Vision-based localization offers a promising alternative. We present a high-precision six degrees of freedom self-localization method using multiple cameras covering the surrounding environment. First, a point feature map is created using images from a previous pass of the area to map. Thereafter, the map is used for high-precision localization in real-time. While localization, a rough prior estimate of the current pose is used to shrink the search space for feature matching by projecting mapped landmarks into current images. Then, stored observations of the projected landmarks are matched to actual observations and the egopose is estimated by back-projection error minimization. Thereby, our map structure provides mapped landmarks efficiently towards localization with multiple cameras. In real-world experiments we show that our approach provides reliable localization results while passing the mapped area in arbitrary orientation.

Driver assistance systems and automated driving are known to strongly benefit from digital maps. Keeping map attributes up to date is a challenge, particularly for the current manual measuring approach. In this paper, we present methods to extract information about intersections and traffic lights through a crowdsourcing approach. We use position and dynamic data from a fleet of test vehicles with close-to-market sensors. A statistical hypothesis test is proposed to identify groups of driving directions at an entry of an intersection, which have synchronous traffic light signaling. This information is used to improve the detection of the relevant traffic light signal in case there is a different signaling for the driving directions. Based on a test data set, we classified whether the signaling is synchronous or not with an accuracy of 93.8%. To assess the usefulness of our mapping scheme, we have investigated its contribution to a camera-based traffic light recognition system. An evaluation of the use of additional map information for the traffic light detection was performed on a set of 344 logged intersection crossings from this vehicle. We showed that there is an improvement in the accuracy up to 5.2%, dependent on the test conditions.

Future lane-precise navigation systems will recommend lane changes to drivers if needed. To achieve this, robust lane-level localization on a navigable map is essential. We propose an ego-lane estimation algorithm to robustly determine the ego-lane in urban scenarios based on a particle filter approach. The method only requires sensors available in a current production car, i.e. visual lane-marking detection, radar, and GPS, and a digital map describing road geometry and topology. Extensive experimental validation has shown an error rate of less than 0.75% with an availability of 95% of the total time and below 0.4% at 96% availability in situations most relevant for navigation. The influence of the used sensors has been evaluated.

Accurate localization is a key task in map based autonomous driving. While in many cases high precision differential GPS is used, more and more vision based methods gain popularity to improve positioning in GNSS denied environments and to avoid high costs in high quality GNSS receivers. However, to generate a globally referenced map, satellite based methods are still important, even in vision based mapping algorithms. In this paper we present a method for integrating locally accurate visual odometry obtained from an onboard stereo camera system with satellite observations of a low cost GNSS receiver. To account for a low number of visible satellites we directly incorporate pseudorange measurements for sensor data fusion. Hence, we present a low cost satellite and camera based positioning system and evaluate it for the usage as part of an inner city mapping system.

The functional system architecture of an automated vehicle plays a crucial role in the performance of the vehicle. When considered as a backbone, it does not only transmit information between distinct layers, but rather serves as a feedback mechanism coordinating the degradation between them and thereby regulates the behavior of the system against failures. Hence, the design of robust functional architectures is essential to cope with the uncertainties of the world. This paper summarizes existing system architectures and investigates them regarding their robustness against measurement inaccuracies, failures, and unexpected evolution of traffic situations. After illustrating their strengths and deficiencies, we derive the requirements and propose a structure for future, robust system architectures.

Collision avoidance maneuvers using braking and steering provide opportunities to avoid a collision at higher velocities compared to braking or steering only. This work investigates control concepts with integrated trajectory planning for combined braking and steering maneuvers using model predictive control (MPC) approaches. A major challenge here is the computational efficiency of the optimization process accounting for nonlinear constraints or nonlinear dynamic models. The main contribution of this work is the introduction of several simplifications which reduces the nonlinear optimization problem to a quadratic program and thus enables application in a vehicle demonstrator. Comparison of the quadratic MPC with the nonlinear MPC shows not only similar performance in a simulation environment, but demonstrates strongly reduced computation time by a factor of approximately 400.

Motion planning plays a key role in autonomous driving. In this work, we introduce the combinatorial aspect of motion planning which tackles the fact that there are usually many possible and locally optimal solutions to accomplish a given task. Those options we call maneuver variants. We argue that by partitioning the trajectory space into discrete solution classes, such that local optimization methods yield an optimum within each discrete class, we can improve the chance of finding the global optimum as the optimum trajectory among the manuever variants. This work provides methods to enumerate the maneuver variants as well as constraints to enforce them. The return of the effort put into the problem modification as suggested is gaining assuredness in the convergency behaviour of the optimization algorithm. We show an experiment where we identify three local optima that would not have been found with local optimization methods.

This paper addresses the problem of curb detection for ADAS or autonomous navigation in urban scenarios. The algorithm is based on clouds of 3D points. It is evaluated using 3D information from a pair of stereo cameras and a LIDAR. Curbs are detected based on road surface curvature. The curvature estimation requires a dense point cloud, therefore the density of the LIDAR cloud has been augmented using Iterative Closest Point (ICP) based on the previous scans. The proposed algorithm can deal with curbs of different curvature and heights, from as low as 3 cm, in a range up to 20 m (whenever that curbs are connected in the curvature image). The curb parameters are modeled using straight lines and compared to the ground-truth using the lateral error as the key parameter indicator. The ground-truth sequences were manually labeled on urban images from the KITTI dataset and made publicly available for the scientific community.

In this paper, we present an approach targeting the automated road map generation for autonomously driving vehicles using low-cost GPS sensor data in a multi-drive setup. Multiple drives with deployed commodity smartphone and stereo camera system are recorded as input data. To overcome the high position uncertainties of the GPS sensor, the GPS trajectory is fused with ego-motion estimates of the vehicle computed by visual odometry. Landmarks are extracted from the recorded imagery data and fused over all recorded drives. The resulting road map consists of a simple, parametric representation of globally referenced lane markings with low storage impact. The challenging aspect in this work is the feature association between multiple drives. Different characteristics of dashed center lines are exploited for this purpose to handle the low precision of the sensor data. The resulting association information is the building block for graph-based SLAM to optimize vehicle poses and landmarks simultaneously. The approach is finally evaluated on real world data comparing the low-precision sensor data with high-precision sensor data as ground-truth.

State of the art highly automated and self-driving vehicles heavily depend on detailed maps since they free the system from many otherwise complex onboard processing tasks. However, depending on the environment and the fineness of the map, the validity span of maps is often short and a periodic remapping of large areas with sensor-packed mapping vehicles is beyond any feasibility. Crowd based mapping approaches using low cost sensors appear more practicable. Herein we propose a general method to align several trajectories of the same area which is fundamental for any life-long mapping. Our algorithm requires previously acquired pose differences as input. These differences induce a pose graph which is aligned yielding a minimum least-squares residual. Therefore, our method is independent from the underlying sensor technology. For evaluation purposes, we align pose graphs from simulated pose differences and compare it against the ground truth. Furthermore, stereo cameras are used to obtain pose difference estimates by common visual odometry methods. We present quantitative results of the robustness and accuracy of our method based on these pose differences. The results are compared against a high precision GPS receiver. Our approach clearly outperforms this costly reference sensor.

In this paper, we present a novel compositional hierarchical framework for road scene understanding that allows for reliable estimation of scene topologies, such as the number, location, and width of lanes and the lane topology, i.e., parallel, splitting, or merging. In our approach, lanes and roads are represented in a hierarchical compositional model in which nodes represent parts of roads and edges represent probabilistic constraints between pairs of parts. A key benefit of our approach is the representation of lanes and roads as a set of common parts. This makes our approach applicable to scenes with rich topological diversity, while bringing along the much desired computational efficiency. To cope with the high-dimensional and continuous parameter space of our model and the non-Gaussian image evidence, we perform inference using nonparametric belief propagation. Based on this approximate inference algorithm, we introduce depth-first message passing for lane detection, which performs inference in several sweeps. Empirical results show that depth-first message passing requires significantly lower computation for performance comparable with classical belief propagation.

Lane estimation of the ego vehicle plays a key role in navigating a car through unknown areas. In fact, solving this problem is a prerequisite for any vehicle driving autonomously in previously unmapped areas. Most of the proposed methods for lane detection are tuned for freeways and rural environments. In urban scenarios, however, they are unable to reliably detect the ego lane in many situations. Often, these methods simply work on the principle of fitting a parametric model to lane markers. Since a large variety of lane shapes are found in urban environments, it is obvious that these models are too restrictive. Moreover, the complex structure of intersection-like situations further hampers the success of the aforementioned methods. Therefore we propose a non-parametric lane model which can handle a wide range of different features such as grass verge, free space, lane markers etc. The ego lane estimation is formulated as a shortest path problem. A directed acyclic graph is constructed from the feature pool rendering it efficiently solvable. The proposed approach is easily extendable as it is able to cope with pixel-wise low level features as well as highlevel ones jointly. We demonstrate the potential of our method in urban and rural areas and present experimental findings on difficult real world data sets.

In this paper we propose a highly detailed map for the field of autonomous driving. We introduce the notion of lanelets to represent the drivable environment under both geometrical and topological aspects. Lanelets are atomic, interconnected drivable road segments which may carry additional data to describe the static environment. We describe the map specification, an example creation process as well as the access library libLanelet which is available for download. Based on the map, we briefly describe our behavioural layer (which we call behaviour generation) which is heavily exploiting the proposed map structure. Both contributions have been used throughout the autonomous journey of the Mercedes Benz S 500 Intelligent Drive following the Bertha Benz Memorial Route in summer 2013.

Estimating surface normals is an important task in computer vision, e.g. in surface reconstruction, registration and object detection. In stereo vision, the error of depth reconstruction increases quadratically with distance. This makes estimation of surface normals an especially demanding task. In this paper, we analyze how error propagates from noisy disparity data to the orientation of the estimated surface normal. Firstly, we derive a transformation for normals between disparity space and world coordinates. Afterwards, the propagation of disparity noise is analyzed by means of a Monte Carlo method. Normal reconstruction at a pixel position requires to consider a certain neighborhood of the pixel. The extent of this neighborhood affects the reconstruction error. Our method allows to determine the optimal neighborhood size required to achieve a pre specified deviation of the angular reconstruction error, defined by a confidence interval. We show that the reconstruction error only depends on the distance of the surface point to the camera, the pixel distance to the principal point in the image plane and the angle at which the viewing ray intersects the surface.

It is well known that the robustness of many computer vision algorithms can be improved by employing large field of view cameras, such as omnidirectional cameras. To avoid obstructions in the field of view, such cameras need to be mounted in an exposed position. Alternatively, a multicamera setup can be used. However, this requires the extrinsic calibration to be known. In the present work, we propose a method to calibrate a fisheye multi-camera rig, mounted on a mobile platform. The method only relies on feature correspondences from pairwise overlapping fields of view of adjacent cameras. In contrast to existing approaches, motion estimation or specific motion patterns are not required. To compensate for the large extent of multi-camera setups and corresponding viewpoint variations, as well as geometrical distortions caused by fisheye lenses, captured images are mapped into virtual camera views such that corresponding image regions coincide. To this end, the scene geometry is approximated by the ground plane in close proximity and by infinitely far away objects elsewhere. As a result, low complexity feature detectors and matchers can be employed. The approach is evaluated using a setup of four rigidly coupled and synchronized wide angle fisheye cameras that were attached to four sides of a mobile platform. The cameras have pairwise overlapping fields of view and baselines between 2.25 and 3 meters.

Knowledge of motion with respect to the ground plane is required in many computer vision applications such as obstacle avoidance, egomotion estimation, and online calibration. The homography matrix comprises motion as well as ground plane information. Estimation of the homography matrix is challenging, as measurements are often not only corrupted by sparse gross outliers, but might also contain other structures, which are inconsistent with the ground plane such as curbstones and sidewalks. Several well studied algorithms regarding the identification of sparse gross outliers already exist. However, identifying structural outliers remains a challenging problem due the outliers' inner coherence. In homography and plane estimation structural outliers often cause plane fits that do not correspond to any physical plane in the scene. We make use of the large field of view of fisheye cameras by exploiting that outlier identification can be performed more robustly in the near field where motion parallax vectors are large. More sensitive data can then be tested subsequently based on the preceding results. The main contribution of this paper is twofold. First, we present a statistical analysis of parallax amplitudes that are to be expected due to the distance of a point from the ground plane and measurement noise. This leads to a statistical test for outliers with local adaptive thresholds. Second, we embed this concept into an extended Kalman filter for efficient processing. Furthermore, we emphasize the importance of warping captured images into a common frame previous to feature detection and matching to avoid distortion effects and to equalize search regions. We demonstrate the robustness of our approach and the effects of prewarping on the estimation using real data.

Many robotics applications nowadays use cameras for various task such as place recognition, localization, mapping etc. These methods heavily depend on image descriptors. A plethora of descriptors have recently been introduced but hardly any address the problem of illumination robustness. Herein we introduce an illumination robust image descriptor which we dub DIRD (Dird is an Illumination Robust Descriptor). First a set of Haar features are computed and individual pixel responses are normalized to L2 unit length. Thereafter features are pooled over a predefined neighborhood region. The concatenation of several such features form the basis DIRD vector. These features are then quantized to maximize entropy allowing (among others) a binary version of DIRD consisting of only ones and zeros for very fast matching. We evaluate DIRD on three test sets and compare its performance with (extended) USURF, BRIEF and a baseline gray level descriptor. All proposed DIRD variants substantially outperform these methods by times more than doubling the performance of USURF and BRIEF.

Digital maps within cars are not only the basis for navigation but also for advanced driver assistance systems. Therefore more and more up-to-date details about the environment of the vehicle are required which means that they have to be enriched with further attributes such as detailed representations of intersections. In the future we will be able to extract details of the environment out of the sensory data of connected cars. We present a generic approach for extracting multiple intersection parameters with the same method by analyzing logged data from a test fleet. Based on that a method for a feature based estimation of the confidence is introduced. The proposed approaches are applied in a completely automated process to estimate stop line positions and traffic flows at intersections with traffic lights. Altogether 203.701 traces of the test fleet were used for developing and testing. The performance of the method and the confidence estimation were analyzed using a ground truth, consisting of 108 stop line positions, which was derived from satellite images. The results show that the approach is fast and predictions with an absolute accuracy of 3.5m can be achieved. Hence the method is able to deliver valuable inputs for driver assistance systems.

Obstacle detection is a fundamental task for Advanced Driver Assistance Systems (ADAS) and Self-driving cars. Several commercial systems like Adaptive Cruise Controls and Collision Warning Systems depend on them to notify the driver about a risky situation. Several approaches have been presented in the literature in the last years. However, most of them are limited to specific scenarios and restricted conditions. In this paper we propose a robust sensor fusion-based method capable of detecting obstacles in a wide variety of scenarios using a minimum number of parameters. Our approach is based on the spatial-relationship on perspective images provided by a single camera and a 3D LIDAR. Experimental tests have been carried out in different conditions using the standard ROAD-KITTI benchmark, obtaining positive results.

In this paper, we present the strategy for trajectory planning that was used on-board the vehicle that completed the 103 km of the Bertha-Benz-Memorial-Route fully autonomously. We suggest a local, continuous method that is derived from a variational formulation. The solution trajectory is the constrained extremum of an objective function that is designed to express dynamic feasibility and comfort. Static and dynamic obstacle constraints are incorporated in the form of polygons. The constraints are carefully designed to ensure that the solution converges to a single, global optimum.

In August 2013, the modified Mercedes-Benz SClass S500 Intelligent Drive (“Bertha”) completed the historic Bertha-Benz-Memorial-Route fully autonomously. The self-driving 103 km journey passed through urban and rural areas. The system used detailed geometric maps to supplement its online perception systems. A map based approach is only feasible if a precise, map relative localization is provided. The purpose of this paper is to give a survey on this corner stone of the system architecture. Two supplementary vision based localization methods have been developed. One of them is based on the detection of lane markings and similar road elements, the other exploits descriptors for point shaped features. A final filter step combines both estimates while handling out-of-sequence measurements correctly.

In this paper, we present a novel probabilistic generative model for multi-object traffic scene understanding from movable platforms which reasons jointly about the 3D scene layout as well as the location and orientation of objects in the scene. In particular, the scene topology, geometry, and traffic activities are inferred from short video sequences. Inspired by the impressive driving capabilities of humans, our model does not rely on GPS, lidar, or map knowledge. Instead, it takes advantage of a diverse set of visual cues in the form of vehicle tracklets, vanishing points, semantic scene labels, scene flow, and occupancy grids. For each of these cues, we propose likelihood functions that are integrated into a probabilistic generative model. We learn all model parameters from training data using contrastive divergence. Experiments conducted on videos of 113 representative intersections show that our approach successfully infers the correct layout in a variety of very challenging scenarios. To evaluate the importance of each feature cue, experiments using different feature combinations are conducted. Furthermore, we show how by employing context derived from the proposed method we are able to improve over the state-of-the-art in terms of object detection and object orientation estimation in challenging and cluttered urban environments.

Autonomous and intelligent vehicles will undoubtedly depend on an accurate ego localization solution. Global navigation satellite systems suffer from multipath propagation rendering this solution insufficient. Herein, we present a real-time system for six-degrees-of-freedom ego localization that uses only a single monocular camera. The camera image is harnessed to yield an ego pose relative to a previously computed visual map. We describe a process to automatically extract the ingredients of this map from stereoscopic image sequences. These include a mapping trajectory relative to the first pose, global scene signatures and local landmark descriptors. The localization algorithm then consists of a topological localization step that completely obviates the need for any global positioning sensors such as GNSS. A metric refinement step that recovers an accurate metric pose is subsequently applied. Metric localization recovers the ego pose in a factor graph optimization process based on local landmarks. We demonstrate centimeter-level accuracy by a set of experiments in an urban environment. To this end, two localization estimates are computed for two independent cameras mounted on the same vehicle. These two independent trajectories are thereafter compared for consistency. Finally, we present qualitative experiments of an augmented reality (AR) system that depends on the aforementioned localization solution. Several screen shots of the AR system are shown confirming centimeter-level accuracy and subdegree angular precision.

Both, the estimation of the trajectory of a sensor and the detection and tracking of moving objects are essential tasks for autonomous robots. This work proposes a new algorithm that treats both problems jointly. The sole input is a sequence of dense 3D measurements as returned by multi-layer laser scanners or time-of-flight cameras. A major characteristic of the proposed approach is its applicability to any type of environment since specific object models are not used at any algorithm stage. More specifically, precise localization in non-flat environments is possible as well as the detection and tracking of e.g. trams or recumbent bicycles. Moreover, 3D shape estimation of moving objects is inherent to the proposed method. Thorough evaluation is conducted on a vehicular platform with a mounted Velodyne HDL-64E laser scanner.

This paper presents an approach for online estimation of the extrinsic calibration parameters of a multi-camera rig. Given a coarse initial estimate of the parameters, the relative poses between cameras are refined through recursive filtering. The approach is purely vision based and relies on plane induced homographies between successive frames. Overlapping fields of view are not required. Instead, the ground plane serves as a natural reference object. In contrast to other approaches, motion, relative camera poses, and the ground plane are estimated simultaneously using a single iterated extended Kalman filter. This reduces not only the number of parameters but also the computational complexity. Furthermore, an arbitrary number of cameras can be incorporated. Several experiments on synthetic as well as real data were conducted using a setup of four synchronized wide angle fisheye cameras, mounted on a moving platform. Results were obtained, using both, a planar and a general motion model with full six degrees of freedom. Additionally, the effects of uncertain intrinsic parameters and nonplanar ground were evaluated experimentally.

Place recognition for loop closure detection lies at the heart of every Simultaneous Localization and Mapping (SLAM) method. Recently methods that use cameras and describe the entire image by one holistic feature vector have experienced a resurgence. Despite the success of these methods, it remains unclear how a descriptor should be constructed for this particular purpose. The problem of choosing the right descriptor becomes even more pronounced in the context of life long mapping. The appearance of a place may vary considerably under different illumination conditions and over the course of a day. None of the handcrafted descriptors published in literature are particularly designed for this purpose. Herein, we propose to use a set of elementary building blocks from which millions of different descriptors can be constructed automatically. Moreover, we present an evaluation function which evaluates the performance of a given image descriptor for place recognition under severe lighting changes. Finally we present an algorithm to efficiently search the space of descriptors to find the best suited one. Evaluating the trained descriptor on a test set shows a clear superiority over its hand crafted counter parts like BRIEF and U-SURF. Finally we show how loop closures can be reliably detected using the automatically learned descriptor. Two overlapping image sequences from two different days and times are merged into one pose graph. The resulting merged pose graph is optimized and does not contain a single false link while at the same time all true loop closures were detected correctly. The descriptor and the place recognizer source code is published with datasets on http://www.mrt.kit.edu/libDird.php.

Next generation driver assistance systems require precise self localization. Common approaches using global navigation satellite systems (GNSSs) suffer from multipath and shadowing effects often rendering this solution insufficient. In urban environments this problem becomes even more pronounced. Herein we present a system for six degrees of freedom (DOF) ego localization using a mono camera and an inertial measurement unit (IMU). The camera image is processed to yield a rough position estimate using a previously computed landmark map. Thereafter IMU measurements are fused with the position estimate for a refined localization update. Moreover, we present the mapping pipeline required for the creation of landmark maps. Finally, we present experiments on real world data. The accuracy of the system is evaluated by computing two independent ego positions of the same trajectory from two distinct cameras and investigating these estimates for consistency. A mean localization accuracy of 10 cm is achieved on a 10 km sequence in an inner city scenario.

Smartphones have long become an omnipresent part of our life. Equipped with both a broadband internet connection and advanced GPS onboard sensors, the idea is to use them as mobile sensors for active safety systems that aim at protecting vulnerable road users such as pedestrians or cyclists. This paper gives a comprehensive analysis of today's smartphones GPS accuracy on an inner-city bicycle track. In addition, the transmission latencies of a prototypical bicycle warning system are evaluated. The results show that while the lateral deviations are still too high to allow for lane-level localization, the longitudinal accuracy as well as the transmission latencies are good enough for many active safety applications already.

Advanced driver-assistance systems (ADASs) employ single-object information to provide safety, comfort, or infotainment features. While today's systems use common sensors, such as radars or cameras, to recognize and predict the future states of relevant traffic participants, next-generation ADASs will also use data from additional sources such as Car-to-X (C2X) communication networks. We present a method that uses information on other traffic participants and furthermore recognizes and considers their interactions in terms of traffic maneuvers. For this purpose, a probabilistic approach is presented, which identifies object interactions and different road characteristics. This method may find a particular application in the C2X domain for evaluating the mobility of neighboring vehicles based on received messages. In this paper, we present M aneuver Assessment For R eliable Verification of C2X mobility data (MARV-X), which is a tool embodying a two-stage process for reliable C2X mobility data verification. The first stage consists of a dedicated mobility estimator realized by a Kalman filter (KF). In the second stage, a plausibility check for highly dynamic traffic situations is applied using the advocated probabilistic traffic maneuver recognition. MARV-X is fully integrated into the vehicle's C2X architecture. Its effectiveness is demonstrated by means of extensive real-world experiments.

Advanced Driver Assistance Systems (ADAS) employ single object information to provide safety, comfort, or infotainment features. The required data is mainly extracted from external sensors to recognize and predict the future states of relevant traffic participants. Next generation ADAS will also use data from additional sources like, e.g., Car-to-X communication networks, to avoid some typical restrictions of common sensor setups. In this work, we present a method, which uses information on other traffic participants, and furthermore recognizes and considers their interactions in terms of traffic maneuvers to better predict their states. For this purpose, a probabilistic framework is presented, which recognizes object interactions as well as different road characteristics by introducing local, adaptive occupancy grids. The resulting maneuver recognition is shown to considerably improve received mobility data in terms of position, speed, and heading. These concepts have been fully implemented and evaluated by means of real world experiments.

State of the art visual odometry systems use bundle adjustment (BA) like methods to jointly optimize motion and scene structure. Fusing measurements from multiple time steps and optimizing an error criterion in a batch fashion seems to deliver the most accurate results. However, often the scene structure is of no interest and is a mere auxiliary quantity although it contributes heavily to the complexity of the problem. Herein we propose to use a recently developed incremental motion estimator which delivers relative pose displacements between each two frames within a sliding window inducing a pose graph. Moreover, we introduce a method to learn the uncertainty associated with each of the pose displacements. The pose graph is adjusted by non-linear least squares optimization while incorporating a motion model. Thereby we fuse measurements from multiple time steps much in the same sense as BA does. However, we obviate the need to estimate the scene structure yielding a very efficient estimator: Solving the nonlinear least squares problem by a Gauss-Newton method takes approximately 1ms. We show the effectiveness of our method on simulated and real world data and demonstrate substantial improvements over incremental methods.

Predicting turn and stop maneuvers of potentially errant drivers is a basic requirement for advanced driver assistance systems for urban intersections. Previous work has shown that an early estimate of the driver's intent can be inferred by evaluating the vehicle's speed during the intersection approach. In the presence of a preceding vehicle, however, the velocity profile might be dictated by car-following behaviour rather than by the need to slow down before doing a left or right turn. To infer the driver's intent under such circumstances, a simple, real-time capable approach using an explicit model to represent both car-following and turning behaviour is proposed. Models for typical turning behavior are extracted from real world data. Preliminary results based on a Bayes net classification are presented.

In this paper, we present the concepts and methods developed for the autonomous vehicle known as AnnieWAY, which is our winning entry to the 2011 Grand Cooperative Driving Challenge. We describe algorithms for sensor fusion, vehicle-to-vehicle communication, and cooperative control. Furthermore, we analyze the performance of the proposed methods and compare them with those of competing teams. We close with our results from the competition and lessons learned.

Accurate localization is a fundamental component of driver-assistance systems and autonomous vehicles. For path-constrained motion, a map offers significant information and assists localization with valuable information about the evolution of the kinematic vehicle states. We propose natural parameterized cubic spline curves to approximate true motion constraints, particularly the centerline of individual road lanes or rail tracks. Vehicle kinematics is modeled in 1-D curve coordinates. Since map information is subject to uncertainties, a probabilistic treatment is a prerequisite to obtaining consistent localization results. The proposed probabilistic curvemap (PCM) and the close map-to-vehicle relation enable a straightforward derivation of measurement update equations without additional map-matching steps and offer themselves to classical filter techniques. Incoming sensor measurements are used for simultaneous vehicle localization and local PCM update around the current vehicle position. Thus, every revisit of a location reduces uncertainty in the local PCM. Moreover, when no prior information is provided in the PCM, extrapolation is carried out to handle these situations with incomplete maps. The proposed filter is validated through simulations and real-world railway experiments.

Accurate 3d perception from video sequences is a core subject in computer vision and robotics, since it forms the basis of subsequent scene analysis. In practice however, online requirements often severely limit the utilizable camera resolution and hence also reconstruction accuracy. Furthermore, real-time systems often rely on heavy parallelism which can prevent applications in mobile devices or driver assistance systems, especially in cases where FPGAs cannot be employed. This paper proposes a novel approach to build 3d maps from high-resolution stereo sequences in real-time. Inspired by recent progress in stereo matching, we propose a sparse feature matcher in conjunction with an efficient and robust visual odometry algorithm. Our reconstruction pipeline combines both techniques with efficient stereo matching and a multi-view linking scheme for generating consistent 3d point clouds. In our experiments we show that the proposed odometry method achieves state-of-the-art accuracy. Including feature matching, the visual odometry part of our algorithm runs at 25 frames per second, while - at the same time - we obtain new depth maps at 3-4 fps, sufficient for online 3d reconstructions.

Estimating a vehicles' own trajectory and generating precise maps of the environment are both important tasks for intelligent vehicles. Especially for the second task laser scanners are the sensor of choice as they provide precise range measurements. This work proposes an approach for simultaneous localization and mapping (SLAM) specifically designed for the Velodyne HDL-64E laser scanner which exhibits characteristics not present in most other systems. This comprises the continuous, spinning data acquisition and the relative high sensor noise. Together, these make standard SLAM approaches generate noisy maps and inaccurate trajectories. We show that it is possible to generate precise maps and localize therein in spite of not using wheel speed sensors or other information. The presented approach is evaluated on a novel, challenging 3D data set being made publicly available.

Precise localization of rail vehicles is a key element toward the development and deployment of novel train control systems that offer enhanced security and efficiency. Typically, research on train navigation systems approaches this task either by data fusion of an increasing number of onboard sensors or by additional infrastructure installations that are combined with the localization of a global navigation satellite system (GNSS). The former approach is cost intensive and only gradually improves reliability and availability of localization information, whereas the latter approach suffers from the absence of satellite signals in places that are important for railroad applications such as tunnels or railway stations. In contrast, this paper employs a novel single eddy current sensor (ECS) mounted on the rail vehicle that directly pursues observations of the rail on a topological map. The localization task is formulated in a model-based probabilistic framework that enables us to derive signal processing techniques for board-autonomous speed estimation and recognition of particular events such as railroad switches by pattern recognition. In particular, turnouts are detected by Bayesian inference based on hidden Markov models (HMMs). In the final step, position on a topological map is estimated by sequential Monte Carlo sampling that combines speed and event information acquired from the ECS signal. Experiments with simulated and real-world data from an experimental rail vehicle indicate that the proposed system yields position and speed information of high reliability in real time.

Generating accurate environment representations can significantly improve the autonomy of mobile robots. In this article we present a novel probabilistic technique for solving the full SLAM problem by jointly solving the data registration problem and the accurate reconstruction of the underlying geometry. The key idea of this paper is to incorporate spatial correlation models as prior knowledge on the map we seek to construct. We formulate the mapping problem as a maximum a-posteriori estimation comprising common probabilistic motion and sensor models as well as two spatial correlation models to guide the optimization. Instead of discarding data at an early stage, our algorithm makes use of all data available in the optimization process. When applied to SLAM, our method generates maps that closely resemble the real environment. We compare our approach to state-of-the-art algorithms, using both real and synthetic data sets.

Visually estimating a robot's own motion has been an active field of research within the last years. Though impressive results have been reported, some application areas still exhibit huge challenges. Especially for car-like robots in urban environments even the most robust estimation techniques fail due to a vast portion of independently moving objects. Hence, we move one step further and propose a method that combines ego-motion estimation with low-level object detection. We specifically design the method to be general and applicable in real-time. Pre-classifying interest points is a key step, which rejects matches on possibly moving objects and reduces the computational load of further steps. Employing an Iterated Sigma Point Kalman Filter in combination with a RANSAC based outlier rejection scheme yields a robust frame-to-frame motion estimation even in the case when many independently moving objects cover the image. Extensive experiments show the robustness of the proposed approach in highly dynamic environments with speeds up to 20m/s.

Numerous applications require precise determination of the motion of a textured surface. Image based sensors are attractive for this purpose, since they are contactless and do not suffer from slip effects. Moreover, they do not only measure scalar speed, but can determine velocity as a 2D vectorial quantity, and, if two sensors are combined, can even observe yaw rate of the surface. If this kind of sensor is mounted inversely on a vehicle, it can determine its motion by measuring the relative displacement of the road surface. In this paper we will use a commercial motion sensor that has been originally designed for application in an optical computer mouse. We will show that, by well-considered dimensioning of the optical part of the system, this sensor can measure velocities in a range that is typical for automotive application. The result is a highly integrated, low-cost, angle sensitive motion sensor that does not exhibit slip effects. We evaluate this sensor through testing on a vehicle that is equipped with reference sensors, with special consideration on the advantages of this measurement principle over conventional wheel speed sensors.

We present a method for fast collision checking that is suitable for application in motion planning for intelligent vehicles. One of the difficulties that arises in this domain is the fact that typical, car-like autonomous vehicles cannot easily be approximated by a rotationally invariant disk shape. Instead, the orientation of the vehicle must be accounted for explicitly. Our proposal is to decompose the vehicle shape into several disk shaped primitives, so that the task of collision checking can be broken down into few very simple collision tests. We also propose a highly optimised method to perform these primitive collision tests that requires a minimum of arithmetic operations. We show by experiments that our method bears significant performance benefits over conventional methods.

We present a method for motion planning in the presence of moving obstacles that is aimed at dynamic on-road driving scenarios. Planning is performed within a geometric graph that is established by sampling deterministically from a manifold that is obtained by combining configuration space and time. We show that these graphs are acyclic and shortest path algorithms with linear runtime can be employed. By reparametrising the configuration space to match the course of the road, it can be sampled very economically with few vertices, and this reduces absolute runtime further. The trajectories generated are quintic splines. They are second order continuous, obey nonholonomic constraints and are optimised for minimum square of jerk. Planning time remains below 20 ms on general purpose hardware.

This paper presents a consistent framework for continuous stereo self-calibration. Based on a practical analysis of the sensitivity of stereo reconstruction to camera calibration uncertainties, we identify important parameters for self-calibration. We evaluate different geometric constraints for estimation and tracking of these parameters: bundle adjustment with reduced structure representation relating corresponding points in image sequences, the epipolar constraint between stereo image pairs, and trilinear constraints between image triplets. Continuous, recursive calibration refinement is obtained with a robust, adapted iterated extended Kalman filter. To achieve high accuracy, physically relevant geometric optimization criteria are formulated in a Gauss-Helmert type model. The self-calibration framework is tested on an active stereo system. Experiments with synthetic data as well as on natural indoor and outdoor imagery indicate that the different constraints are complementing each other and thus a method combining two of the above constraints is proposed: While reduced order bundle adjustment gives by far the most accurate results (and might suffice on its own in some environments), the epipolar constraint yields instantaneous calibration that is not affected by independently moving objects in the scene. Hence, it expedites and stabilizes the calibration process.

In this paper, we review technologies for autonomous ground vehicles and their present capabilities in research and in the automotive market. We outline technology requirements for enhanced functions and for infrastructure development. Since the recent Grand Challenge competition is a major force to advance technology in this field, we specifically refer to our experiences in developing a participating vehicle. We present a multisensor platform that has been proven in an off-road environment. It combines different sensing modalities that inherently yield uncertain information. Finite-state machines are formulated to generate rule-based autonomous behavior that enables fully autonomous off-road driving. Overall, the intent of the paper is to evaluate approaches and technologies used in the two Grand Challenges as they contribute to the needs of autonomous cars on the road

This paper presents a novel monocular detection and tracking algorithm of vehicles applied to electronic vehicle-coupling at close distances. For the sake of safety and reliability the vehicle to track is marked with a pattern. Such an application demands for measuring reliably and with high precision the position and orientation of the vehicle to follow relative to the pursuing vehicle. Additional requirements are real time capability with respect to standard PC hardware and robustness against strong vibrations, high illumination dynamics and other effects that an automotive image processing application faces typically.

Motion estimation belongs to key techniques in image sequence processing. Segmentation of the motion fields such that, ideally, each independently moving object uniquely corresponds to one region, is one of the essential elements in object-based image processing. This paper is concerned with unsupervised simultaneous estimation of dense motion fields and their segmentations. It is based on a stochastic model relating image intensities to motion information. Based on the analysis of natural images, a region-based model of motion-compensated prediction error is proposed. In each region the error is modeled by a white stationary generalized Gaussian random process. The motion field and its segmentation are themselves modeled by a compound Gibbs/Markov random field accounting for statistical bindings in spatial direction and along the direction of motion trajectories. The a posteriori distribution of the motion field for a given image sequence is formulated as an objective function, such that its maximization results in the MAP estimate. A deterministic multiscale relaxation technique with regular structure is employed for optimization of the objective function. Simulation results are in a good agreement with human perception for both the motion fields and their segmentations.

This contribution addresses the simultaneous estimation of dense motion fields and their segmentation from image sequences. Weak constraints incorporated by a stochastic model relate the image sequence to the motion field and its segmentation. Following the analysis of the error signal of motion compensated prediction, a segment-wise stationary generalized Gaussian model is introduced. The motion field and its segmentation are themselves modeled by a compound Gibbs random field accounting for spatio-temporal statistical bindings where the temporal bindings are directed along the motion trajectories. A Bayesian objective function is expressed according to the model. The estimates are calculated simultaneously by multiscale optimization of this objective function and ML-estimation of model parameters. Simulation results demonstrate the performance of the proposed scheme for motion as well as for disparity estimation.

This paper is concerned with linear transforms for arbitrarily-shaped image segments. In contrast to other techniques described in the literature, the proposed transform is based upon a stochastic model of image covariance within the considered region. Emerging from a separable stationary Markov model proposed for rectangular regions, we derive a non-stationary Markov model with natural boundary conditions. We compute its eigentransform, which is the optimum linear transform under a broad variety of performance measures. For the special case of a rectangular region, the method yields the DCT basis functions. Simulation results for natural imagery are provided.

This paper focuses on object-oriented motion estimation and motion field encoding for video compression. A dense motion field and its segmentation are estimated jointly by a MAP estimator. The a priori distribution of the estimates is modelled by a coupled Markov random field accounting for both spatial smoothness and temporal continuity along motion trajectories of the estimates. A lossy contour/content code is proposed for motion field transmission. For each object, a region-oriented linear transform is applied to the motion vectors. It minimizes a distortion measure which approximates the square error of motion compensated prediction. Simulation results demonstrate the efficiency of the scheme for video encoding with high compression ratio.>

A new criterion for displacement-estimation in image sequence coding is proposed. Its derivation is based on a model of the a posteriori probability for the vector field to correspond to the true physical motion of the scene. The model considers statistics of the prediction error image as well as statistical interdependencies between vectors within one vector field. The resulting criterion can be interpreted as a gain/cost criterion since it trades small prediction error against short codestring for the vector field. Experimental results confirm that the presented criterion supplies vector fields close to the physical motion of the scene. Strong statistical bindings between vectors exist within those vector fields. This allows transmitting vector fields with increased spatial resolution and decreased displaced frame difference (DFD) when compared with a full search estimator which minimizes the DFD for each vector at the same data rate.>

A new approach to motion-estimation for hybrid image sequence coding is presented. Instead of minimizing the displaced frame difference (DFD), the estimator introduced in this paper maximizes the probability to determine the ’true’ physical motion of the scene. The probability expression is derived from two models, one for the statistics of the prediction error image and one for the interdependency of vectors in a vectorfield. The physical vectorfield is smoother than the vectorfield of a DFD-estimator and stronger statistical bindings between vectors exist. Therefore a coding algorithm for the vectorfield combining contour coding of regions of similar displacement with predictive coding of the vectors inside each region proves efficient. This allows the estimator to work with decreased blocksize and (even in DFD sense) to supply a distinctly improved displacement-compensation without spending more datarate for displacement-compensation than the DFD- estimator.