Teleoperation enables remote human support of automated vehicles in scenarios where the automation is not able to find an appropriate solution. Remote assistance concepts, where operators provide discrete inputs to aid specific automation modules like planning, is gaining interest due to its reduced workload on the human remote operator and improved safety. However, these concepts are challenging to implement and maintain due to their deep integration and interaction with the automated driving system. In this paper, we propose a solution to facilitate the implementation of remote assistance concepts that intervene on planning level and extend the operational design domain of the vehicle at runtime. Using arbitration graphs, a modular decision-making framework, we integrate remote assistance into an existing automated driving system without modifying the original software components. Our simulative implementation demonstrates this approach in two use cases, allowing operators to adjust planner constraints and enable trajectory generation beyond nominal operational design domains.

Camera-based object detection encounters challenges in adverse weather, which can compromise the robustness of the perception module within autonomous driving systems. Cutting-edge domain adaptive object detection methods use the teacher-student framework and domain adversarial learning to generate domain-invariant pseudo-labels for self-training. However, the pseudo-labels generated by the teacher model often exhibit a bias toward the majority class, incorporating overconfident false positives and underconfident false negatives. We reveal that pseudo-labels vulnerable to adversarial attacks are more likely to be of low quality. To address this issue, we propose a simple yet effective framework named Adversarial Attacked Teacher (AAT) to improve pseudo-label quality. Specifically, we apply adversarial attacks on the teacher model, prompting it to generate adversarial pseudo-labels to correct bias, suppress overconfidence, and encourage underconfident proposals. We introduce an adaptive pseudo-label regularization to emphasize the influence of pseudo-labels with high certainty and reduce the negative impacts of uncertain predictions. Moreover, reliable minority pseudo-labels, verified by pseudo-label regularization, are oversampled to minimize dataset imbalance without introducing false positives. AAT establishes a new state-of-the-art, achieving 53.0 mAP on the Cityscapes to Foggy Cityscapes benchmark. The code is publicly available at https://github.com/KIT-MRT/AAT/.

This paper introduces an extension to the arbitration graph framework designed to enhance the safety and robustness of autonomous systems in complex, dynamic environments. Building on the flexibility and scalability of arbitration graphs, the proposed method incorporates a verification step and structured fallback layers in the decision-making process. This ensures that only verified and safe commands are executed while enabling graceful degradation in the presence of unexpected faults or bugs. The approach is demonstrated using a Pac-Man simulation and further validated in the context of autonomous driving, where it shows significant reductions in accident risk and improvements in overall system safety. The bottom-up design of arbitration graphs allows for an incremental integration of new behavior components. The extension presented in this work enables the integration of experimental or immature behavior components while maintaining system safety by clearly and precisely defining the conditions under which behaviors are considered safe. The proposed method is implemented as a ready to use header-only C++ library, published under the MIT License. Together with the Pac-Man demo, it is available at github.com/KIT-MRT/arbitration_graphs.

Classifying intentions of other traffic agents is an essential task for intelligent transportation systems. To simplify this task, vehicles are equipped with various illumination systems, including turn indicators, emergency lights, rear lights, and brake lights. We extend the Waymo open perception dataset with ground truth annotations for different visual intentions to develop methods designed to classify the state of such systems. Furthermore, we propose the VISUAL INTENTION FORMER, a two-step transformer-based architecture to classify visual intentions in image sequences of tracked traffic participants. We use a vision transformer to extract image features, which are passed into a transformer encoder that reasons about temporal dependencies among them. We evaluate against different baseline architectures where our proposed method achieves state-of-the-art results. Additionally, we conduct an in-depth performance analysis of our method regarding different input sequence lengths, vehicle headings, and daytime conditions.

Automated vehicles require carefully designed cost functions, which are challenging to specify due to the complexity of the behavior they need to cover. Inverse reinforcement learning is a principled methodology for deriving cost functions, but it requires high-quality expert demonstrations, which are expensive to obtain. Recently, scenario-based testing has emerged as a promising approach for validation of driving behavior. In this paper, we introduce a novel methodology that circumvents the need for costly expert driving demonstrations by harnessing scenario-based testing. Our Test-Driven Inverse Reinforcement Learning approach leverages Bayesian inference, utilizing the outcomes of scenario tests as observations to infer cost functions. We rigorously evaluate our method on simulated and real-world scenarios and demonstrate its ability to learn cost functions that successfully pass the respective scenario tests. We also show that the learned cost function generalizes well by also passing scenario tests from an unseen validation set and illustrate that few scenario tests are sufficient to learn meaningful cost functions. This innovative framework not only streamlines the cost function specification process but also offers a cost-effective and practical solution for advancing automated driving systems.

Precise object detection is crucial in automated driving. In contrast to lidar and radar sensors, cameras provide high-resolutional measurements at comparatively low cost. A state-of-the-art method for object detection using camera images is panoptic segmentation, which combines semantic and object instance information. Current public datasets use multiple cameras to cover a larger area of the environment. But, the limited field of view occludes objects. As a results, on the one hand, the correct dimensions of objects cannot be captured and, on the other hand, false detections can occur. Objects can also be detected multiple times in the overlapping image area. To track dynamic objects, duplicate detections must be filtered. Rather than directly segmenting all camera images individually, we first stitch them into a horizontal panorama. Using a stitched surround view avoids detection difficulties at the boundaries of the individual images. For this purpose, we leverage the EfficientPS pre-trained network architecture and adapt it for use with panoramic images. In our evaluation, we demonstrate the improvement in panoptic quality of a stitched surround view. In addition, we separately compare the panoptic quality in the overlapping image areas between the panorama and the individual images. Finally, we show further advantages of panoramic images in terms of inference time in runtime analysis.

Connected automated vehicles (CAV), which incorporate vehicle-to-vehicle (V2V) communication into their motion planning, are expected to provide a wide range of benefits for individual and overall traffic flow. A frequent constraint or required precondition is that compatible CAVs must already be available in traffic at high penetration rates. Achieving such penetration rates incrementally before providing ample benefits for users presents a chicken-and-egg problem that is common in connected driving development. Based on the example of a cooperative driving function for bottleneck traffic flows (e.g. at a roadblock), we illustrate how such an evolutionary, incremental introduction can be achieved under transparent assumptions and objectives. To this end, we analyze the challenge from the perspectives of automation technology, traffic flow, human factors and market, and present a principle that 1) accounts for individual requirements from each domain; 2) provides benefits for any penetration rate of compatible CAVs between 0 % and 100 % as well as upward-compatibility for expected future developments in traffic; 3) can strictly limit the negative effects of cooperation for any participant and 4) can be implemented with close-to-market technology. We discuss the technical implementation as well as the effect on traffic flow over a wide parameter spectrum for human and technical aspects.

When detecting objects, depth sensors are not always available, requiring 3D object detection from monocular images. However, for many object classes, datasets with 3D annotations are missing. Recent monocular 3D object detection methods lack the semantic diversity needed for autonomous systems, because of missing 3D ground truth data for static classes such as poles and traffic lights. To overcome this gap we combine a large scale dataset for 2D object detection, with an unlabeled dataset containing depth measurements. We lift 2D object detections of the depth dataset into the 3D domain, associating detections with corresponding depth values. This leverages 2D annotated datasets to enable semantically rich 3D object detection, without extra labelling effort. We train an object detection model with mixed batches and evaluate it comparing the predicted depth with the projected centerpoint depth of cars manually annotated in 3D space. The result is a monocular object detector that can predict 3D positions of up to 37 static and dynamic object classes from camera only.

Planning for robotic systems is frequently formulated as an optimization problem. Instead of manually tweaking the parameters of the cost function, they can be learned from human demonstrations by Inverse Reinforcement Learning (IRL). Common IRL approaches employ a maximum entropy trajectory distribution that can be learned with soft reinforcement learning, where the reward maximization is regularized with an entropy objective. The consideration of safety constraints is of paramount importance for human-robot collaboration. For this reason, our work addresses maximum entropy IRL in constrained environments. Our contribution to this research area is threefold: (1) We propose Constrained Soft Reinforcement Learning (CSRL), an extension of soft reinforcement learning to Constrained Markov Decision Processes (CMDPs). (2) We transfer maximum entropy IRL to CMDPs based on CSRL. (3) We show that using importance sampling in maximum entropy IRL in constrained environments introduces a bias and fails to achieve feature matching. In our evaluation we consider the tactical lane change decision of an autonomous vehicle in a highway scenario modeled in the SUMO traffic simulation.

Visual Simultaneous Localization and Mapping is one of the main challenges for robotics and automated vehicles. In the state-of-the-art approaches, pixel-level correspondences are mostly used. In this paper, we address the problem of finding stable and repeatable pixel-level correspondences under challenging conditions. As a network basis, we use the D2-Net, which can select keypoints at positions that can be matched easily. Several techniques are implemented in this work to improve the keypoint detection and description performance. We first feed the network with rotated images to achieve a higher rotation invariance of point detections. Furthermore, we analyze the impact of a ranking of score values, adopting a cosine similarity and enforcing more dominant detections by defining a peakiness. The evaluation shows that combining a ranking with a peakiness can achieve the best result, especially to illumination changes. By using this combination, we achieve a mean matching accuracy increase of 12.5% on illumination scenes (9% overall) and a 9% higher repeatability rate at extremely low costs by only modifying the loss function.

Decision making under uncertainty can be framed as a partially observable Markov decision process (POMDP). Finding exact solutions of POMDPs is generally computationally intractable, but the solution can be approximated by sampling-based approaches. These sampling-based POMDP solvers rely on multi-armed bandit (MAB) heuristics, which assume the outcomes of different actions to be uncorrelated. In some applications, like motion planning in continuous spaces, similar actions yield similar outcomes. In this paper, we utilize variants of MAB heuristics that make Lipschitz continuity assumptions on the outcomes of actions to improve the efficiency of sampling-based planning approaches. We demonstrate the effectiveness of this approach in the context of motion planning for automated driving.

Reinforcement learning is nowadays a popular framework for solving different decision making problems in automated driving. However, there are still some remaining crucial challenges that need to be addressed for providing more reliable policies. In this paper, we propose a generic risk-aware DQN approach in order to learn high level actions for driving through unsignalized occluded intersections. The proposed state representation provides lane based information which allows to be used for multi-lane scenarios. Moreover, we propose a risk based reward function which punishes risky situations instead of only collision failures. Such rewarding approach helps to incorporate risk prediction into our deep Q network and learn more reliable policies which are safer in challenging situations. The efficiency of the proposed approach is compared with a DQN learned with conventional collision based rewarding scheme and also with a rule-based intersection navigation policy. Evaluation results show that the proposed approach outperforms both of these methods. It provides safer actions than collision-aware DQN approach and is less overcautious than the rule-based policy.

Estimating the current scene and understanding the potential maneuvers are essential capabilities of automated vehicles. Most approaches rely heavily on the correctness of maps, but neglect the possibility of outdated information. We present an approach that is able to estimate lanes without relying on any map prior. The estimation is based solely on the trajectories of other traffic participants and is thereby able to incorporate complex environments. In particular, we are able to estimate the scene in the presence of heavy traffic and occlusions. The algorithm first estimates a coarse lane-level intersection model by Markov chain Monte Carlo sampling and refines it later by aligning the lane course with the measurements using a non-linear least squares formulation. We model the lanes as 1D cubic B-splines and can achieve error rates of less than 10cm within real-time.

Behavior planning and decision-making are some of the biggest challenges for highly automated systems. A fully automated vehicle (AV) is faced with numerous tactical and strategical choices. Most state-of-the-art AV platforms are implementing tactical and strategical behavior generation using finite state machines. However, these usually result in poor explainability, maintainability and scalability. Research in robotics has raised many architectures to mitigate these problems, most interestingly behavior-based systems and hybrid derivatives. Inspired by these approaches, we propose a hierarchical behavior-based architecture for tactical and strategical behavior generation in automated driving. It is a generalizing and scalable decision-making framework, utilizing modular behavior blocks to compose more complex behaviors in a bottom-up approach. The system is capable of combining a variety of scenario- and methodology-specific solutions, like POMDPs, RRT* or learning-based behavior, into one understandable and traceable architecture. We extend the hierarchical behavior-based arbitration concept to address scenarios where multiple behavior options are applicable, but have no clear priority among each other. Then, we formulate the behavior generation stack for automated driving in urban and highway environments, incorporating parking and emergency behaviors as well. Finally, we illustrate our design in an explanatory evaluation.

High definition (HD) maps have proven to be a necessary component for safe and comfortable automated driving (AD) [1]. Naïvely verifying HD maps requires an accurate localization prior in order to correctly associate measurements with map data. In periodic environments, such as highways, localization results are often ambiguous - in particular in longitudinal direction. To still be able to verify an HD map, we propose the use of quasi-continuous 1D signals that can be computed without pointwise association. These signals can be chosen to change significantly when the map has changed while they only change rarely or slowly along the road, making them robust against localization errors. A spatio-semantic clustering yields intuitive groups of map features. These groups are then ordered using a robust projection approach, yielding quasi-continuous 1D signals. Such signals can be computed for map and measurement data and their comparison allows detecting road changes. The purposeful design of the signals and their computation only requires lane-level lateral localization and a coarse longitudinal prior, vastly relaxing the requirements on prior localization results compared to the current state of the art. With four example signals, we demonstrate the effectiveness of our approach on a map verification dataset [2], detecting between 49 % and 98 % of all changed features at false alarm rates usually below 15 %. Detecting changes per feature allows to still use unchanged features for AD functions. When omitting this ability and aggregating all features, 98 % of all changed road sections can be detected successfully.successfully.

Visual tracking of multiple objects is an essential component for a perception system in autonomous driving vehicles. One of the favorable approaches is the tracking-by-detection paradigm, which links current detection hypotheses to previously estimated object trajectories (also known as tracks) by searching appearance or motion similarities between them. As this search operation is usually based on a very limited spatial or temporal locality, the association can fail in cases of motion noise or long-term occlusion. In this paper, we propose a novel tracking method that solves this problem by putting together information from both enlarged structural and temporal domain. For efficiency without loss of optimality, this approach is decomposed in to three stages, with each dealing with only one constrained association task, and thus, it follows the alternating optimization fashion. In our approach, detections are first assembled into small tracklets based on meta-measurements of object affinity. The association task for tracklets-to-tracks is solved by structural information based on a motion pattern between them. Here, we propose new rules to decouple the processing time from the tracklet length. Furthermore, constraints from temporal domain are introduced to recover objects, which are long-time disappearing due to failed detection or long-term occlusion. By putting together the heterogeneous domain information, our approach exhibits an improved state-of-the-art performance on standard benchmarks. With relatively little processing time, an online and real-time tracking is also permitted in our approach.

Visual object tracking has achieved remarkable progress in recent years and has been broadly applied in intelligent transportation systems such as autonomous vehicles and drones to monitor and analyze the behavior of specific targets. One typical tracking approach is the discriminative tracker, which branches into two main categories: the correlation filter (CF) and the convolutional neural network (CNN). However, most of the current researches consider both categories as two separate techniques and only rely on one of them. Thus, a dense cooperation between the CF and the CNN still remains less discovered and the question of how to effectively join both techniques to further boost the tracking performance is still open. To address this issue, in this paper, we propose a collaborative architecture which incorporates models constructed with both techniques and dynamically aggregates their response maps for target inference. By an alternating optimization, both models are learned on each other's errors to persistently improve the classification power of the whole tracker. For further efficiency, we present a faster solver for our utilized CF and an analytical solution for dynamic model weighting. Through experiments on standard benchmarks, we reveal the influence of key factors on the joint learning architecture and show that it outperforms the state-of-the-art approaches.

Highly accurate localization with very limited amount of memory and computational power is one of the big challenges for next generation series cars. We propose localization based on geometric primitives which are compact in representation and further valuable for other tasks like planning and behavior generation. The primitives lack distinctive signature which makes association between detections and map elements highly ambiguous. We resolve ambiguities early in the pipeline by online building up a local map which is key to runtime efficiency. Further, we introduce a new framework to fuse association and odometry measurements based on robust pose graph optimization.We evaluate our localization framework on over 30 min of data recorded in urban scenarios. Our map is memory efficient with less than 8 kB/km and we achieve high localization accuracy with a mean position error of less than 10 cm and a mean yaw angle error of less than 0. 25° at a localization update rate of 50Hz.

For a successful introduction of fully automated vehicles, they must behave both provably safe but also convenient, i.e. comfortable and not overcautious. Given the limited sensing capabilities, especially in urban scenarios where buildings and parking vehicles impose occlusions, this is a challenging task. While recent approaches gave first ideas for boundary conditions of safe behavior, an approach for convenient motion planning that fulfills these constraints is still an open issue. Therefore, we utilize and enhance safety approaches for occlusion handling in order to facilitate comfortable and safe motion planning. We consider worst case assumptions, arising from potential objects at critical sensing field edges, along with their probability. With this information, we can ensure to not act overcautiously while still moving provably safe. The potential of our approach is shown in a modified CommonROAD scenario.

We propose a deep convolutional object detector for automated driving applications that also estimates classification, pose and shape uncertainty of each detected object. The input consists of a multi-layer grid map which is well-suited for sensor fusion, free-space estimation and machine learning. Based on the estimated pose and shape uncertainty we approximate object hulls with bounded collision probability which we find helpful for subsequent trajectory planning tasks. We train our models based on the KITTI object detection data set. In a quantitative and qualitative evaluation some models show a similar performance and superior robustness compared to previously developed object detectors. However, our evaluation also points to undesired data set properties which should be addressed when training data-driven models or creating new data sets.

Accurate traffic participant prediction is the prerequisite for collision avoidance of autonomous vehicles. In this work, we propose to predict pedestrians using goal-directed planning. For this, we infer a mixture density function for possible destinations. We use these destinations as the goal states of a planning stage that performs motion prediction based on common behavior patterns. The patterns are learned by a fully convolutional network operating on maps of the environment. We show that this entire system can be modeled as one monolithic neural network and trained via inverse reinforcement learning. Experimental validation on real world data shows the system's ability to predict both, destinations and trajectories accurately.

Higher level functionality in autonomous driving depends strongly on a precise motion estimate of the vehicle. Powerful algorithms have been developed. However, their great majority focuses on either binocular imagery or pure LIDAR measurements. The promising combination of camera and LIDAR for visual localization has mostly been unattended. In this work we fill this gap, by proposing a depth extraction algorithm from LIDAR measurements for camera feature tracks and estimating motion by robustified keyframe based Bundle Adjustment. Semantic labeling is used for outlier rejection and weighting of vegetation landmarks. The capability of this sensor combination is demonstrated on the competitive KITTI dataset, achieving a placement among the top 15. The code is released to the community.

In this work we present a novel approach for multi-sensor calibration that significantly outperforms current state-of-the-art. We introduce a new spherical calibration target which has major benefits over existing targets. Those are subresolution detection accuracy in both camera and depth sensor, view invariance and applicability to a wider range of sensor setups than current approaches. With our method a single person achieves high quality calibration in less than a minute. No preparations for setting up the environment for calibration is needed. Our method is fast, easy to use and fully automatic. We evaluate our method in simulation and show high accuracy with an error of less than 3mm in translation and 0.1 0 in rotation on real data.

While motion planning techniques for automated vehicles in a reactive and anticipatory manner have already been widely presented, cooperative motion planning has only been addressed recently. For the latter, interaction between traffic participants is crucial. Consequently, simulations where other traffic participants follow simple behavioral rules can no longer beused for development and evaluation. To close this gap, we present a multi vehicle simulation framework. Conventional simulation agents, using a simple, rule-based behavior, are replaced by multiple instances of sophisticated behavior generation algorithms. Thus, development, test and simulative evaluation of cooperative planning approaches is facilitated. The framework is implemented using the Robot Operating System (ROS) and its code will be released open source.

Trajectory and intention prediction of traffic participants is an important task in automated driving and crucial for safe interaction with the environment. In this paper, we present a new approach to vehicle trajectory prediction based on automatically generated maps containing statistical information about the behavior of traffic participants in a given area. These maps are generated based on trajectory observations using image processing and map matching techniques. The generated maps contain all typical vehicle movements and probabilities in the considered area. Our prediction approach matches an observed trajectory to a behavior contained in the map and uses this information to generate a prediction. We evaluated our approach on a dataset containing over 14000 trajectories and found that it produces significantly more precise mid-term predictions compared to motion model-based prediction approaches.

Despite the fact that in recent years, vision-based tracking approaches have made significant progress, the task of tracking vehicles at night still remains challenging. Visual information is strongly deteriorated or at least degraded due to poor illumination conditions. This reduces the perceptive ability of vision systems significantly and can even lead to target loss, resulting in false estimation and/or false prediction of object behavior. In this paper, we propose a novel online-learning method to track vehicles at night. Our method is based on the kernelized correlation filter and assembles different feature channels to kernelized experts. By estimating their reliabilities, we force the appearance model to focus on the most discriminative visual features to accomplish the classification. In addition, a temporal optimization step in conjunction with a memory model is used to remove outliers and keep the most reliable samples to train the tracker models. Experiments over various daytime and weather conditions show that our approach outperforms existing trackers at night and in case of bad weather while offering state-of-the-art performance in more favorable situations. As our tracker has only little computational cost, it is appropriate for use cases with real-time requirements like in automotive or industrial applications.

The rapid advances of transportation infrastructure have led to a dramatic increase in the demand for smart systems capable of monitoring traffic and street safety. Fundamental to these applications are a community-based evaluation platform and benchmark for object detection and multi-object tracking. To this end, we organize the AVSS2017 Challenge on Advanced Traffic Monitoring, in conjunction with the International Workshop on Traffic and Street Surveillance for Safety and Security (IWT4S), to evaluate the state-of-the-art object detection and multi-object tracking algorithms in the relevance of traffic surveillance. Submitted algorithms are evaluated using the large-scale UA-DETRAC benchmark and evaluation protocol. The benchmark, the evaluation toolkit and the algorithm performance are publicly available from the website http://detrac-db.rit.albany.edu.

Vision systems become more and more popular to be applied in monitoring tasks such as controlling traffic flows or for security issues. The analysis of target behavior is always based on its observed trajectory, which can be acquired by tracking approaches. Although the fashion of tracking-by-detection is favored by the research community, it still faces challenges like unexpected occlusion caused by background objects or other tracked targets, which can interfere the matching operation and result in tracking errors. In this paper, we propose a novel approach by aggregating prediction events within target groups and integrating a graph-modeling based stitching procedure to handle the above mentioned problems. The evaluation results on the UA-DETRAC benchmark demonstrated the state-of-the-art performance of our tracking approach.

Human detection is an important task for many autonomous robots as well as automated driving systems. The Regionlets detector was one of the best-performing approaches for pedestrian detection on the KITTI dataset during 2015. We analysed the Regionlets detector and its performance. This paper discusses the improvements in accuracy that were achieved by the different ideas of the Regionlets detector. It also analyses what the boosting algorithm learns and how this relates to the expectations. We found that the random generation of regionlet configurations can be replaced by a regular grid of regionlets. Doing so reduces the dimensionality of the feature space drastically but does not decrease detection performance. This translates into a decrease in memory consumption and computing time during training.

Stereo cameras are among the most promising sensors for automated driving. For their deployment, however, calibration should be automated and possible in-situ. We propose a restructuring of bundle adjustment into an incremental online calibration system. It allows us to estimate all observable camera parameters on the fly. Both simulations and experiments with real world cameras show its capability to calibrate stereo rigs in real time while driving. With this method, cameras can be employed with almost no calibration overhead. Only the non-observable parameter of scale has to be defined in advance.

Intelligent vehicles heavily rely on robust and accurate self-localization. Global navigation satellite systems (GNSS) are not reliable in urban environments due to multipath and shadowing effects. Vision-based localization offers a promising alternative. We present a high-precision six degrees of freedom self-localization method using multiple cameras covering the surrounding environment. First, a point feature map is created using images from a previous pass of the area to map. Thereafter, the map is used for high-precision localization in real-time. While localization, a rough prior estimate of the current pose is used to shrink the search space for feature matching by projecting mapped landmarks into current images. Then, stored observations of the projected landmarks are matched to actual observations and the egopose is estimated by back-projection error minimization. Thereby, our map structure provides mapped landmarks efficiently towards localization with multiple cameras. In real-world experiments we show that our approach provides reliable localization results while passing the mapped area in arbitrary orientation.

We present an improved model for MRF-based depth upsampling, guided by image-as well as 3D surface normal features. By exploiting the underlying camera model we define a novel regularization term that implicitly evaluates the planarity of arbitrary oriented surfaces. Our method improves upsampling quality in scenes composed of predominantly planar surfaces, such as urban areas. We use a synthetic dataset to demonstrate that our approach outperforms recent methods that implement distance-based regularization terms. Finally, we validate our approach for mapping applications on our experimental vehicle.

Similar to autonomous vehicles, future train applications require an accurate on-board self-localization for railway vehicles. Therefore, a reliable and real-time capable environment perception is required. In particular, the knowledge of the track taken at a turnout overcomes ambiguities in self-localization. As the most important groundwork for this, the paper introduces a new approach for the detection of rails and tracks solely from 2d lidar measurements. The technique is based on a new feature point method for lidar data, a template matching approach, and a spatial clustering technique to extract rails and tracks from the detected rail elements. The new approach is evaluated on six different datasets taken outdoors at a demanding test ground. It provides reliable and accurate detection results with centimeter accuracy, a recall of about 90 %, and a precision of about 95 %. The approach is able to detect rails even in complex real-world topologies such as at turnouts and even on tracks with more than two rails.

Estimating the ground plane is often one of the first steps in geometric reasoning processes as it offers easily accessible context knowledge. Especially unconstrained platforms that capture video from egocentric viewpoints can benefit from such knowledge in various ways. A key requirement here is keeping orientation, which can be greatly achieved by keeping track of the ground. We present an approach to keep track of the ground plane in cluttered inner-urban environments using stereo vision in real-time. We fuse a planar model fit in low-resolution disparity data with the direction of the vertical vanishing point. Our experiments show how this effectively decreases the error of plane attitude estimation compared to classic least-squares fitting and allows to track the plane with camera configurations in which the ground is not visible. We evaluate the approach using ground-truth from an inertial measurement unit and demonstrate long-term stability on a dataset of challenging inner city scenes.

Environment perception is an important task in computer vision for many applications in robotics. Especially for robots navigating through different levels of a building, stair detection constitutes an important perception task. In this paper, we propose a stair detection algorithm using range data. Firstly, we introduce a parameter, which describes local surface orientations w.r.t. a global reference. Secondly, a matched filter is used to detect relevant edges in the orientation data. Afterwards, line segments are determined using these edge data which are further used to estimate stairs. The proposed method is invariant against rotations of the sensor. We show that the system can handle typical outdoor stair types and outperforms the accuracy of state-of-the-art stair detection methods. Moreover, the method is used in real time to assist visually impaired people who wear the camera system on a helmet.

While monocular visual odometry has been widely investigated, one of its key issues restrains its broad appliance: the scale drift. To tackle it, we leverage scene inherent information about the ground plane to estimate the scale for usage on Advanced Driver Assistance Systems. The algorithm is conceived so that it is independent of the unscaled ego-motion estimation, augmenting its adaptability to other frameworks. A ground plane estimation using Structure From Motion techniques is complemented by a vanishing point estimation to render our algorithm robust in urban scenarios. The method is evaluated on the KITTI dataset, outperforming state of the art algorithms in areas where urban scenery is dominant.

This paper describes an algorithm that enables a railway vehicle to determine its position in a track network. The system is based solely on onboard sensors such as a velocity sensor and a Global Navigation Satellite System (GNSS) sensor and does not require trackside infrastructure such as axle counters or balises. The paper derives a probabilistic modeling of the localization task and develops a sensor fusion approach to fuse the inputs of the GNSS sensor and the velocity sensor with the digital track map. We describe how we can treat ambiguities and stochastic uncertainty adequately. Moreover, we introduce the concept of virtual balises that can be used to replace balises on the track and evaluate the approach experimentally. This paper focuses on an accurate modeling of sensor and estimation uncertainties, which is relevant for safety critical applications.

In this paper, we present a novel probabilistic generative model for multi-object traffic scene understanding from movable platforms which reasons jointly about the 3D scene layout as well as the location and orientation of objects in the scene. In particular, the scene topology, geometry, and traffic activities are inferred from short video sequences. Inspired by the impressive driving capabilities of humans, our model does not rely on GPS, lidar, or map knowledge. Instead, it takes advantage of a diverse set of visual cues in the form of vehicle tracklets, vanishing points, semantic scene labels, scene flow, and occupancy grids. For each of these cues, we propose likelihood functions that are integrated into a probabilistic generative model. We learn all model parameters from training data using contrastive divergence. Experiments conducted on videos of 113 representative intersections show that our approach successfully infers the correct layout in a variety of very challenging scenarios. To evaluate the importance of each feature cue, experiments using different feature combinations are conducted. Furthermore, we show how by employing context derived from the proposed method we are able to improve over the state-of-the-art in terms of object detection and object orientation estimation in challenging and cluttered urban environments.

Abstract: Geometric context has been recognised as important high-level knowledge towards the goal of scene under-standing. In this work we present two approaches to estimate the local geometric structure of urban street canyons captured from a head-mounted stereo camera. A dense disparity estimation is the only input for both approaches. First, we show how the left and right building facade can be obtained by planar segmentation based on random sampling. In a second approach we transform the disparity into an elevation map from which we extract the main building orientation. We evaluate both approaches on a set of challenging inner city scenes and demonstrate how visual odometry can be incorporated to keep track of the estimated geometry. 1

In this paper, we present the concepts and methods developed for the autonomous vehicle known as AnnieWAY, which is our winning entry to the 2011 Grand Cooperative Driving Challenge. We describe algorithms for sensor fusion, vehicle-to-vehicle communication, and cooperative control. Furthermore, we analyze the performance of the proposed methods and compare them with those of competing teams. We close with our results from the competition and lessons learned.

3D scene understanding is key for the success of applications such as autonomous driving and robot navigation. However, existing approaches either produce a mild level of understanding, e.g., segmentation, object detection, or are not accurate enough for these applications, e.g., 3D pop-ups. In this paper we propose a principled generative model of 3D urban scenes that takes into account dependencies between static and dynamic features. We derive a reversible jump MCMC scheme that is able to infer the geometric (e.g., street orientation) and topological (e.g., number of intersecting streets) properties of the scene layout, as well as the semantic activities occurring in the scene, e.g., traffic situations at an intersection. Furthermore, we show that this global level of understanding provides the context necessary to disambiguate current state-of-the-art detectors. We demonstrate the effectiveness of our approach on a dataset composed of short stereo video sequences of 113 different scenes captured by a car driving around a mid-size city.

In this case study we show how reinforcement learning can be applied successfully for low level control tasks in autonomous driving like steering control as an alternative to controllers from classical control theory. We describe the learning procedure and compare the resulting control policies with a classical controller. The experiments are made both in simulation and on a real car and we discuss the case of driving forwards as well as of driving backwards.

Learning directly on real world systems such as autonomous robots is a challenging task, especially if the training signal is given only in terms of success or failure (Reinforcement Learning). However, if successful, the controller has the advantage of being tailored exactly to the system it eventually has to control. Here we describe, how a neural network based RL controller learns the challenging task of ball dribbling directly on our Middle-Size robot. The learned behaviour was actively used throughout the RoboCup world championship tournament 2007 in Atlanta, where we won the first place. This contistutes another important step within our Brainstormers project. The goal of this project is to develop an intelligent control architecture for a soccer playing robot, that is able to learn more and more complex behaviours from scratch.