The potential for deploying autonomous systems can be significantly increased by improving the perception and interpretation of the environment. However, the development of deep learning-based techniques for autonomous systems in unstructured outdoor environments poses challenges due to limited data availability for training and testing. To address this gap, we present the German Outdoor and Offroad Dataset (GOOSE), a comprehensive dataset specifically designed for unstructured outdoor environments. The GOOSE dataset incorporates 10000 labeled pairs of images and point clouds, which are utilized to train a range of state-of-the-art segmentation models on both image and point cloud data. We open source the dataset, along with an ontology for unstructured terrain, as well as dataset standards and guidelines. This initiative aims to establish a common framework, enabling the seamless inclusion of existing datasets and a fast way to enhance the perception capabilities of various robots operating in unstructured environments. This framework also makes it possible to query data for specific weather conditions or sensor setups from a database in future. The dataset, pre-trained models for offroad perception, and additional documentation can be found at https://goose-dataset.de/.

Stereo matching can be used to estimate dense but inaccurate depth information for each pixel of a camera image. A LiDAR can provide accurate but sparse depth measurements. The fusion of both can combine their advantages. We propose an efficient method for fusing stereo and LiDAR at the cost level of Semi-Global Matching. It significantly improves density and accuracy of the estimated disparities while remaining real-time capable. Based on a LiDAR point cloud projected into the camera image costs are calculated for each possible disparity. These costs are added to the costs from stereo matching. Our LiDAR-SGM outperforms other real-time capable fusion approaches evaluated on the KITTI Stereo 2015 dataset. In addition to this real data, synthetic datasets are created (and made available) for a detailed analysis of the benefit of stereo LiDAR fusion as well as the evaluation of different sensors.

In this paper, we present a model predictive control (MPC) approach for the control of an autonomous vehicle following a lead vehicle tracked by onboard sensors. Based on the lead vehicle’s movement and position, the MPC calculates a front steering angle and an acceleration in order to follow the path driven by the lead vehicle as accurately as possible, while also maintaining a safety distance between both vehicles. The performance of the MPC was evaluated during practical experiments with an actual autonomous car. The tests were conducted in both on- and off-road environments, with speeds ranging from 1 m/s to 20 m/s.

This work presents a method for object classification in LiDAR point clouds based on the Dempster and Shafer theory of belief functions and on its extension, the transferable belief model for the field of vehicle automation. We use a combination of geometric, motion and context features, to model various classes of objects commonly found in a driving scenario according to their expected behaviour in different contexts. Using the models we derive evidence to support a hypotheses about the class of the objects or to identify new types of objects that are not included in the set of modeled classes. We show that the use of contextual information has a positive influence in the results of the classification.

Predominantly, complex optimization techniques are used for path reconstruction given noisy measurements. However, optimization techniques often require the selection of suitable models, tedious parameter tuning and typically fail to generalize to higher-level tasks. In this paper, we present a model-free path filtering method based on the popular moving average method, namely the Curvature Corrected Moving Average (CCMA), which convinces by its simplicity and broad applicability. The moving average is characterized by its unique noise suppression property, albeit curves are bent inwards, which adversely affects its accuracy. By utilizing the relation between both curvatures, the original curvature can be inferred based on the curvature of filtered points. Extending the symmetric filtering not only succeeds in minimizing noise but retains the original shape of the path, making it a suitable algorithm for a variety of robotic applications. We demonstrate the practicality of the approach in a real-world convoy scenario: The accumulated estimates of the leader vehicle’s position, originating from an Extended Kalman filter, are smoothed using our novel approach to generate proper inputs for the Model Predictive Controller (MPC). This cascade structure of filtering provides both responsiveness and smoothness. Furthermore, we successfully applied this method in the off-road convoy scenario for the ELROB 2022, where we won first place. The source code is publicly available.

To navigate safely, it is essential for a robot to detect all kinds of moving objects that could possibly interfere with the own trajectory. For common object classes, like cars, regular pedestrians, and trucks, there are large scale datasets as well as corresponding machine learning techniques, which provide remarkable results in commonly available detection benchmarks. A big challenge that remains, are less frequent classes, which are not part of a dataset in a sufficient number and variation. Dynamic occupancy grids are a promising approach for detection of moving objects in point clouds since they impose only a few assumptions about the objects' appearance and shape. Typically, they use particle filters to detect motion of occupancy in the grid. Existing approaches, however, often generate false positives at long obstacles because particles move along them. Therefore we propose a highly efficient approach, which performs the classification in a more structured and conservative way by making extensive use of available free space information. As a result, much less false positives are generated while the number of false negatives remains low. Our approach can be used to complement CNN-based object detections in order to detect both, frequent and uncommon object classes reliably. By using polar data structures that match the polar measurement principle, we are able to process even large point clouds of modern LiDARs with 128 lasers efficiently.

For autonomous driving on rural or dirt roads-neither urban nor off-road - a large terrain area needs to be estimated at high spatial resolution. However, available computing time is very limited. Since different areas of the ground surface require different minimum resolution, we propose a dynamic resolution terrain estimation.Based on support points, accumulated measurements are spatially smoothed to a continuous terrain model using maximum a posteriori estimation. Splitting the terrain into tiles, we dynamically adjust the support point resolution of single tiles, depending on their accuracy in areas of interest. Areas of interest are determined by fusing information on probable road areas from LiDAR and vision preprocessing steps.As demonstrated in real-world examples, our approach can model the terrain almost as accurately as if all tiles had the highest resolution, but with much less computational effort.

For autonomous driving in urban environments it is usually assumed that the road is flat. To drive off-road, however, we need a more sophisticated model of the ground surface. While previous work is mapping the terrain along with static obstacles, we propose to separate the tasks and introduce a new approach to probabilistic terrain estimation. It combines recursive Gaussian state estimation with a subsequent maximum a posteriori estimation. This allows us to efficiently accumulate obtained measurements and at the same time get a probabilistic terrain estimate based on a geometric terrain model. This way, also (measurement) uncertainties as well as inter- and extrapolation to unobserved areas are handled stochastically correct. We demonstrate the effectiveness and real-time capability of our approach using real-world data.

We propose a modular framework for 3D tracking not only of paved roads but also of dirt roads. It is based on recursive state estimation of lane boundary points connected by clothoid pieces. While our tracking is flexible to integrate every kind of measurement, we specifically propose two image-based measurements. They combine traditional with modern computer vision: On the one hand, we show how to use directed edge detection to robustly measure road and lane boundaries. On the other hand, we introduce a innovative CNN-based measurement utilizing the self-similarity of (dirt) road areas. We demonstrate the performance of our approach in challenging scenarios. On a marked road, we achieve a median error of 0.13 m for the ego lane's boundaries in 25 m look-ahead. A difficult dirt road can also be tracked reliably with a lookahead length of 25 m, resulting in a median error of 0.3 m. The tracking, as well as both measurements, are real-time capable.

The random hypersurface model is well-suited to describe extended target contours. Its applicability is limited only by the mild assumption that the target contour has to be star convex. Gaussian processes provide a sound way to estimate the contour functions, and the ability to model the contour uncertainty in a detailed way at different contour points. However, the association of measurements to target contour points is not optimal in current implementations using Gaussian Processes and the random hypersurface model. In this work, we provide an improved approach compared to the standard approach. The standard approach projects measurements radially onto the predicted contour. Our approach provides expected measurements matching the physical reality of the measurement process more closely. In addition, we perform the association of the whole batch of measurements to the expected contour measurements at once. Compared to a sequential association of individual measurements, this leads to a better association decision.

Research in autonomous driving for unstructured environments suffers from a lack of semantically labeled datasets compared to its urban counterpart. Urban and unstructured outdoor environments are challenging due to the varying lighting and weather conditions during a day and across seasons. In this paper, we introduce TAS500, a novel semantic segmentation dataset for autonomous driving in unstructured environments. TAS500 offers fine-grained vegetation and terrain classes to learn drivable surfaces and natural obstacles in outdoor scenes effectively. We evaluate the performance of modern semantic segmentation models with an additional focus on their efficiency. Our experiments demonstrate the advantages of fine-grained semantic classes to improve the overall prediction accuracy, especially along the class boundaries. The dataset and pretrained model are available at mucar3.de/icpr2020-tas500.

Fiducial markers like AprilTags play an important role in robotics, e.g., for the calibration of cameras or the localization of robots. One of the most important properties of an algorithm for detecting such tags is its localization accuracy.In this paper, we present the results of an extensive comparison of four freely available libraries capable of detecting AprilTags, namely AprilTag 3, AprilTags C++, ArUco as standalone libraries, and the OpenCV algorithm based on ArUco. The focus of the comparison is on localization accuracy, but the processing time is also examined. Besides working with pure tags, their extension to checkerboard corners is investigated.In addition, we present two new post-processing techniques. Firstly, a method that can filter out very inaccurate detections resulting from partial border occlusion, and secondly a new highly accurate method for edge refinement. With this we achieve a median pixel error of 0.017 px, compared to 0.17 px for standard OpenCV corner refinement.The dataset used for the evaluation, as well as the developed post-processing techniques, are made publicly available to encourage further comparison and improvement of the detection libraries.

Central low-level sensor data fusion approaches are getting more popular in advanced driver assistant systems. They allow for the resolution of ambiguities in the retrieval of environmental information on the basis of a large, raw data pool. Hereby, one emerging challenge is the unification of sensor data of different formats and sensor types. A popular intermediate layer of data is given by spatial occupancy grids. The conversion of a discrete list of radar detections, which is a commonly utilized measurement format, is problematic due to the sparse spatial resolution. This work addresses this conversion by interpolating the data spatially using generic sensor model knowledge. Traditional approaches derive occupancy evidence in the vicinity of a detection. In addition, we analyze spatial and kinematic properties derived from Doppler measurements, compute likelihoods that multiple detections are caused by the same object and deduce the space between them accordingly. The incorporation of sensor parameters allows full-and short-range radars to be used generically. In addition, we outline the deduction of free space evidence. The elaborated models and algorithms are evaluated on realworld datasets and discussed w.r.t. their applicability in a subsequent Dempster-Shafer-based sensor data fusion approach.

In this work, we propose an extension of the Semi-Global Matching framework for three images from a triplet-stereo rig consisting of a horizontal and vertical camera pair. After calculating the matching costs separately for both image pairs, these are merged at cost level using cubic spline interpolation. For cost values near the left/bottom image boundaries, we propose an advanced weighting strategy. Subsequently, the fused matching can be used directly for the cost aggregation and disparity estimation.The benefits of the proposed fusion strategy are demonstrated by an evaluation based on synthetic and real-world data. To encourage further comparisons on triple stereo algorithms, the dataset used for evaluation is made publicly available.

Localization is a key capability for autonomous vehicles. High-Definition maps are a popular method to represent the environment and to enable precise localization. However, the creation is very demanding and it is not always guaranteed to receive accurate map information, especially for unstructured areas. In this paper, we introduce a novel probabilistic localization and mapping framework that brings together the advantages of sparse feature maps, multi-target tracking for landmark detection, probabilistic global vehicle localization and a graph-based formulation to achieve a consistent map. The front-end of our Simultaneous Localization and Mapping framework is based on Monte Carlo Localization. Our novel measurement model integrates a virtual topological Path-Map with sparse map features to obtain global localization. The graph-based back-end optimizes online the vehicle trajectory and the landmarks' configuration to create a globally aligned map. Furthermore, our method allows weaker requirements in terms of accuracy of the sparse feature map as we represent the degree of uncertainty by means of probabilistic distribution. Additionally, the sparse feature map representation needs substantially less memory than other approaches, which is an advantage for autonomous vehicles. The framework has been tested and evaluated in real experiments for several autonomous runs. The results demonstrate the robustness of our system.

The radar sensor is an important component in autonomous driving applications. Compared with other sensor types like LiDAR or camera, the radar sensor comes with best weather robustness and ease of integration. Its exclusive capability to measure electromagnetic reflectivity and Doppler-derived radial speeds plays a major role in environment perception applications. However, the processing chain of the radar is more complex and often results in unintuitive measurement effects. In this paper, we explain the technical background of high-resolution radar detections. We give a probabilistic architecture modeling Doppler and Micro-Doppler measurements and their influence on the Monopulse-based azimuth angle, the peak detection and the resulting measurement data. Examples accompany the description of the modeling steps.

For the purpose of extended object tracking in multiple hypothesis tracking algorithms such as the Gaussian mixture probability hypothesis density filter (GMPHD), we develop an approach for the combination of different contour estimates. The developed approach works for tracking algorithms that represent target shapes using contour functions to describe the target shape as the distance of the contour to a reference point over the angle. In a heterogeneous multiple sensor setup, the individual sensors' measurements lead to different extent estimates due to their individual measurement principles. Thus a straight forward use of the extended object state in the traditional merging algorithm either results in unexpected shapes, or tracks cannot be merged due to the differing shape of the objects. Our merging procedure explicitly takes the extent estimates into account by using a merging function. The choice of the merging function provides the means to reach objectives such as a conservative or a generous extent estimate. We evaluate the approach using simulated multisensor data in a GMPHD filter. Compared to the traditional merging method, our approach results in better shape estimates.

In this paper, we describe how collision checking for car-like robots can be sped up utilizing system knowledge. Their non-holonomic motion, while being a challenge for motion planning, is utilized here to place discs which are used as an approximation of the robot's shape in a predictive manner. For ease of comparison, we assume the robot to be rectangular, i. e., we use bounding boxes. Our algorithm is compared to a widely-used baseline and shows similar performance in terms of under- and oversampling while being approximately 20-40 % faster. Another feature of the algorithm is its predictive nature: with the frontal disc, we already check for collisions that would occur with the rear disc in the next sample, assuming near-constant curvature. While this might be conservative in some cases where large steering rates are necessary, in our evaluation even tight corridors could be navigated without negative effects.

Point-cloud segmentation of 3D LiDAR scans is an important preprocessing task for autonomous vehicles in on-road and especially in off-road scenarios. Clustering point measurements with the same properties into multiple homogeneous regions is a challenging task due to an uneven sampling density and lack of explicit structural information. This paper presents a novel technique to achieve a robust and fast point-cloud segmentation using the characteristic intrinsic sensor pattern. This pattern is characterized by the mounting position of each laser diode. A structured mesh graph is created by taking the beam calibration and the chronology of incoming data packets into account. The proposed graph-based, multi-pass point segmentation algorithm compares this pattern with a flat-world model to detect discontinuities and to set label attributes such as obstacle or free space for each vertex. Furthermore, we directly detect missing measurements and therefore generate artificial vertices considering the laser beam intrinsics. Finally, a region-growing algorithm is applied in order to obtain cohesive objects. Experimental results show that we achieve a reliable overall performance and a good trade-off between segmentation quality and runtime of 15ms in rough terrain as well as suburban areas.

This paper presents an algorithm for continuous online estimation of the twelve degrees-of-freedom (12-DoF) extrinsic calibration of a stereo camera system for an autonomous car. An Extended Kalman Filter (EKF) recursively estimates the stereo camera calibration by tracking salient points in 3D space that are visible in both cameras. All extrinsic parameters of the stereo camera calibration are only observable under translation and two independent rotations of the vehicle. However, when driving on urban, paved roads the vehicle performs only limited pitch or roll movement. An analysis of the Fisher information matrix reveals that in these situations especially the installation height is difficult to estimate. For these scenarios a further measurement model is added to the algorithm that utilizes the homography of salient points. This homography is induced by the planar road surface in consecutive camera images. Recognition of the road surface, when the position and orientation of the cameras is unknown, can be difficult. Therefore, a convolutional neural network (CNN) is developed that segments camera images pixel-wisely into three classes: `road', `static (other)' and `potentially dynamic'. Only salient points that are segmented as `road' are considered for the homography measurement model. Points that are segmented as `potentially dynamic' are not taken into account for the calibration algorithm. The structure of the CNN has been chosen carefully to enable segmentation of camera images on a mid-range GPU on-board of our autonomous vehicle. An evaluation of the extended algorithm, based on a recorded dataset, shows a considerably faster estimation of the installation height.

An important requirement for safe autonomous driving is the perception of dynamic and static objects. In urban scenarios, there exist hundreds of potential obstacles. Therefore, it is crucial to have a fast and accurate fitting method which is a key step for many tracking algorithms. In this paper, we demonstrate an orientation corrected bounding box fit based on the convex hull and a line creation heuristic. Our method is capable of fitting hundreds of objects in less than 10 ms and involves only few tuning parameters. Furthermore, orientation estimated through the dynamics of the object can be used to improve the fitting result. Real-world experiments have proven the robustness and effectiveness of our method.

In this paper, we propose to complement a horizontally aligned stereo rig with a vertically aligned one. Based on artificial and real camera images we show that certain image structures can be detected by only one of the setups. Actually, the depth of structures with significant gradients in just one direction can only be measured using cameras with an offset in the same direction. Thus, the joint usage of both setups allows for significantly increased robustness and reliability. Moreover, we propose an approach for the fusion of two disparity images gained from two stereo camera pairs with one common camera. The algorithm uses an approximation of the actually measured matching cost function. Thus it takes into account the present image structures and the capabilities of each particular stereo setup. Due to its mathematical simplicity, it can be computed within 1 ms for images with a size of 1164 × 335 pixel. The performance of the fusion is demonstrated using different real-world scenarios.

We present a novel approach to trajectory generation that enables an autonomous vehicle to accurately follow a lead vehicle tracked by on-board sensors. In contrast to other approaches, we ignore the structure of the environment (e.g., lane markings), focusing purely on following the path driven by the vehicle ahead. Based on the leader's velocity, its distance to the ego vehicle and its recorded path, a continuous-curvature trajectory is generated using Sequential Quadratic Programming. As the optimization process takes the ego vehicle's kinematic and dynamic constraints into account, the resulting trajectory is guaranteed to be feasible and safe. The algorithm was tested extensively during practical experiments with an actual autonomous car. Our tests were conducted in both on- and off-road environments, with speeds ranging from 1 m/s to 15 m/s. Ground truth data shows that the system achieves a high degree of accuracy even in difficult scenarios.

In this paper, we propose a system for autonomous vehicle following without a line of sight. From monocular camera images, the leading vehicle extracts scene descriptors which it transmits to the following vehicle by means of vehicle-to-vehicle (V2V) communication. The follower is able to recognize the scenes using its own camera and follow autonomously. A particle filter framework is employed for jump-free localization on the driven path of the leading vehicle. We compare the performance of different place features for accurate localization on a custom application-oriented dataset and evaluate methods to reduce the feature size for low-bandwidth V2V communication, while maintaining and even improving the recognition performance. Real-world results demonstrate the applicability of our system.

The Munich Cognitive Autonomous Robot Car 3rdGeneration (MuCAR-3) has won several international achievements in the past. Recently, the system's control architecture (meaning the interplay between perception, planning and control) was overhauled. Our goals were to simplify the interaction between modules as well as to meet higher requirements for both smoothness and precision. The decoupling of modules helps with tackling more challenging scenarios and facilitates the development of each module. Since state machines struggle with scalability, its interactions with other modules were minimized. We now use a generalized planning layer rather than so-called maneuvers. This paper aims at showcasing the difference between our previous and current architecture. We focus on the improvements that were achieved even for very simple scenarios - in this case off-road platooning. Using the same control algorithms, we achieve both improvements in smoothness and precision, two classically orthogonal goals. Tests were conducted in simulation and verified with MuCAR-3 on our test site.

In this paper, we present a method to build precise local terrain maps for an autonomous vehicle from vision and LiDAR that surpass most existing maps used in the field, both in the details they represent and in efficiency of construction. The high level of detail is obtained by spatio-temporal fusion of data from multiple, complementary sensors in a grid map. The map not only consists of obstacle probabilities, but contains different features of the environment: elevation, color, infrared reflectivity, terrain slopes and surface roughness. Still, an efficient way to manage the map's memory allows us to build the maps online on-board our autonomous vehicle. As we demonstrate by describing some of its applications, the maps can serve as a unified representation to solve further perception and navigation problems without having to resort to individual sensor data again. The proposed terrain maps have proven their robustness and precision in many real-world scenarios, leading to award-winning performances at international robotics competitions.

In this paper, we present a motion planning algorithm for autonomous navigation in highly constrained urban environments. Since common approaches to on-road trajectory planning turned out to be unsuitable for this task, we instead extended an A*-based planner originally designed for navigation in unstructured environments. Two novel node expansion methods were added to obtain smooth and accurate trajectories that consider the structure of the environment. The first one attempts to find a trajectory connecting the current node directly to the goal by solving a boundary value problem using numerical optimization. The second method leverages a simulated pure-pursuit controller to generate edges (i.e. short motion primitives) that guide the vehicle toward or along the global reference path. As a result, the planner is able to produce smooth paths while retaining the explorative power of A* that is needed to deal with challenging situations in urban driving (e.g., reversing in order to pass a vehicle that stopped unexpectedly). Its practical usefulness was demonstrated during extensive tests on an electric vehicle navigating a mock urban environment as well as on our own autonomous vehicle MuCAR-3.

In this paper, we introduce a new geometric 3D feature combined with a clustering approach. Besides 3D data provided by a LiDAR point cloud, reflectivity information is used to further enhance the descriptivity of the feature. The proposed feature can be extracted and compared in real-time. Similar parts of an object, such as features belonging to an automobile headlight, are automatically clustered in an object model without explicit specification. Additionally, we provide a method for autonomous vehicles to automatically learn the shapes of observed moving objects and use them for real-time classification. The resulting object models consisting of the extracted feature clusters are interpretable by humans.

Accurate stereo camera calibration is crucial for 3D reconstruction from stereo images. In this paper, we propose an algorithm for continuous online recalibration of all extrinsic parameters of a stereo camera, which is rigidly mounted on an autonomous vehicle. The algorithm estimates the six degrees-of-freedom (6-DoF) of the transformation from the vehicle coordinate system to the coordinate system of the stereo camera and at the same time the relative 6-DoF transformation between the two camera sensors. Salient points in the environment that are observed by both cameras are tracked over time in 3D space. An Unscented Kalman Filter (UKF) is applied to recursively estimate the extrinsic stereo camera calibration and the 3D position of all observed points. The projections of the points and the measured vehicle motion, which is estimated using an inertial measurement unit (IMU), are given as input. The observability of the stereo camera calibration states is analyzed to identify critical vehicle motion sequences. Results with real world data show that the algorithm is capable of continuously estimating the stereo camera calibrations in spite of large initial errors and varying extrinsic parameters.

This contribution describes a system for accurate, robust and fast six degrees-of-freedom object pose estimation based on multi-feature models and a recursive filtering approach in the context of autonomous vehicle recharging. Feature measurements are integrated sequentially to allow full control over the feature detection algorithms and the influence on the estimate. This makes the system able to cope with partial and short-term full object occlusions, Gaussian measurement noise as well as systematic model errors. For highly precise pose estimates, high resolution cameras are employed. Nevertheless, the proposed system achieves real-time performance while at the same time outperforming other algorithms in terms of accuracy and robustness.

We present a new mapping and navigation system based on human-recognizable landmarks with highly compact representations. Road segments, intersections and salient structures such as houses and trees are detected using vision and LiDAR data. The landmarks are entered in a sparse metric-topological map that is used for navigation. In contrast to traditional SLAM approaches, however, we only store the information required to navigate along the path of the robot that built the map. Due to the sparseness of the data, it can easily be transmitted to other robots via a low-bandwidth radio connection (9600 bit/s), allowing the receivers to reconstruct the map, localize themselves in it and follow the path recorded by the sender. All of this is done without the help of global navigation satellite systems such as GPS. The algorithms were tested and evaluated in practical experiments with our autonomous cars MuCAR-3 and MuCAR-4.

We describe a variable-velocity trajectory planning algorithm for navigating car-like robots through unknown, unstructured environments along a series of possibly corrupted GPS waypoints. The trajectories are guaranteed to be kine-matically feasible, i.e., they respect the robot's acceleration and deceleration capabilities as well as its maximum steering angle and steering rate. Their costs are computed using LiDAR and camera data and depend on factors such as proximity to obstacles, curvature, changes of curvature, and slope. In a second step, velocities for the least-cost trajectory are adjusted based on the dynamics of the vehicle. When the robot is faced with an obstacle on its trajectory, the planner is restarted to compute an alternative trajectory. Our algorithm is robust against GPS error and waypoints placed in obstacle-filled areas. It was successfully used at euRathlon 20131, where our autonomous vehicle MuCAR-3 took first place in the “Autonomous Navigation” scenario.

This paper presents a vision-based solution for detection and tracking of convoy vehicles. Our approach is able to estimate the 3D vehicle pose, velocity and steering angle of the leader vehicles and needs template images of each vehicle. The improved template-based algorithm refers to a previously publication. First, we present an extension of our dynamic region growing algorithm, which is used to remove unnecessary image information. Thanks to a new preprocessing step, the segmentation algorithm is more stable while finding the vehicle silhouette. Second, we achieve an improved pose estimation by using rotated image features. Third, the extensive comparison of algorithms to train cascade classifiers leads to the best one for vehicle detection. The algorithm was evaluated while driving autonomously in urban and non-urban environments. Experimental validation shows that our approach can detect and track poorly visible vehicles under different weather conditions in real-time.

Obstacle avoidance is a key feature for automotive navigation that requires an accurate representation of the environment. In the field of visual perception this task has often been addressed with stereo algorithms that try to obtain a depth map of the environment via disparity calculations on a single pair of images. These algorithms do not exploit that especially in automotive scenarios the fields of view between two consecutive frames have large overlapping regions. Instead, the disparity map is computed from scratch for each stereo frame and no information is propagated from one frame to the next. Since monocular image processing has long benefited from recursive estimation techniques, such as the 4D Approach, this paper presents a novel recursive automotive stereo algorithm, called RAS. RAS internally maintains a list of recursively estimated 3D points that are continuously updated based on the vehicle's movement and measurements in the current stereo frame. We show that RAS not only preserves the knowledge of the environment across frames, but also accounts for measurement modalities and is robust against faulty or even missing measurements.

In [1] Lienhart and Maydt introduced the calculation of rotated Haar-Wavelets by 45°. They showed that the extended set of possible wavelets improves the object recognition method of Viola et al. [2]. In this paper, we introduce a novel integral image structure which holds the information of a standard and a 45° rotated integral image. We use this image structure to improve a simple and efficient corner detector by Schweitzer and Wuensche [3] based on Haar wavelets in terms of rotational invariance and define an orientation function. Additionally, we focus on parallelization and remove the recursive character of [1] to make the method suitable for GPUs.

In recent years vehicles have been equipped with more and more sensors for environment perception. Among these sensors are cameras, RADAR, single-layer and multi-layer LiDAR. One key challenge for the fusion of these sensors is sensor calibration. In this paper we present a novel extrinsic calibration algorithm based on sensor odometry. Given the time-synchronized delta poses of two sensors our technique recursively estimates the relative pose between these sensors. The method is generic in that it can be used to estimate complete 6DOF poses, given the sensors provide a 6DOF odometry, as well as 3DOF poses (planar offset and yaw angle) for sensors providing a 3DOF odometry, like a single-beam LiDAR. We show that the proposed method is robust against motion degeneracy and present results on both simulated and real world data using an inertial navigation system (INS) and a stereo camera system.

This paper presents a robust method for vehicle tracking with a monocular camera. A previously published model-based tracking method uses a particle filter which needs an initial vehicle hypothesis both at system start and in case of a tracking loss. We present a template-based solution using different features to estimate a 3D vehicle pose roughly but fast. Combining model- and template-based object tracking keeps the advantages of each algorithm: Precise estimation of the 3D vehicle pose and velocity combined with a fast (re-) initialization approach. The improved tracking system was evaluated while driving autonomously in urban and unstructured environments. The results show that poorly visible vehicles can be tracked during different weather conditions in real-time.

This paper deals with selective attention for the detection and tracking of road-networks for autonomous driving while utilizing a limited field of view sensor mounted on a fast camera platform with limited dynamics. While a previous paper derived an uncertainty cost function to determine where to look when, this paper introduces dynamic sensor constraints and examines the trade-off between a wish to perform frequent saccades on one hand and limiting factors like information loss due to saccadic motion blurr and time required at the new view direction to gain information on the other hand. A variety of those effects is examined and a new cost function is proposed to dynamically select platform orientations promising to minimize information theoretic uncertainty related to objects and road elements of interest required for autonomous driving. The method works within the 100ms cycle time aboard our autonomous vehicle MuCAR-3.

Precise perception of a vehicle's surrounding is crucial for safe autonomous driving. It requires a high sensor resolution and a large field of view. Active perception, i.e. the redirection of a sensor's focus of attention, is an approach to provide both. With active perception, however, the selection of an appropriate sensor orientation becomes necessary. This paper presents a method for determining the sensor orientation in urban traffic scenarios based on three criteria: the importance of traffic participants w.r.t. the current situation, the available information about traffic participants while considering alternative sensor orientations as well as sensor coverage of the vehicle's relevant surrounding area.

Recently, the introduction of dense, long-range 3D sensors has facilitated tracking of arbitrary objects. Especially in the context of autonomous driving, other traffic participants driving the streets usually stay well-segmented from each other. In contrast, pedestrians or bicyclists do not always stay on the road and they often get close to static structure of the environment, e.g. traffic lights or signs, bushes, parking cars etc. These objects are not as easy to segment, often resulting in an under-segmentation of the scene and wrong tracking results. This paper addresses the problem of tracking moving objects that are hard to segment from their static surroundings by utilizing top-down knowledge about the geometry of existing tracks during segmentation. This includes methods for discerning static from moving objects to reduce the rate of false positive tracks as well as a classification of tracks into pedestrian, bicyclist, motor bike, passenger car, van and truck classes by considering an objects appearance and motion history. The proposed tracking system is experimentally validated in challenging real-world inner-city traffic scenes.

Motivated by cognitive considerations about human knowledge representation we introduce a new approach for robot motion planning and control. Instead of reasoning about positions in a global cartesian coordinate frame we utilize relative orientation and distance information between the robot and perceived objects in the environment. A representation built upon these considerations enables on the one hand a tight coupling between perception, planning, and robot control, while on the other hand a means for precise robot control is established. With our approach we try to encourage for a new view in robotic planning problems by illustrating how it allows for plan generation and execution in a human comprehensible fashion by incorporating plans like: ”Overtake vehicle X on the right side, then follow lane Y”.

Tracking of extended targets in high definition 360 degree 3D-LIDAR (Light Detection and Ranging) measurements is a challenging task. It is a key component in robotic applications and is relevant to collision avoidance and autonomous driving. This paper presents a robust method to determine the 3D outline contour of vehicles in disordered 3D-LIDAR measurements while using several geometrical vehicle-specific constraints. In addition, the 3D outline contour contains information on the local reliability of the contour. A weighted registration approach allows calculating the velocity of consecutive 3D outline contours directly. The approach is tested with real sensor data. A robot car equipped with an inertial measurement unit serves as ground truth.

Autonomous vehicles promise numerous improvements to vehicular traffic: an increase in both highway capacity and traffic flow because of faster response times, less fuel consumption and pollution thanks to more foresighted driving, and hopefully fewer accidents thanks to collision avoidance systems. In addition, drivers can save time for more useful activities. In order for these vehicles to safely operate in everyday traffic or in harsh off-road environments, a multitude of problems in perception, navigation, and control have to be solved. This paper gives an overview of the most current trends in autonomous vehicles, highlighting the concepts common to most successful systems as well as their differences. It concludes with an outlook into the promising future of autonomous vehicles.

In this paper we describe a novel approach to model-based monocular vehicle tracking out of a moving vehicle using active vision. The designed algorithm can cope with cluttered color images, complex lighting conditions as well as partial occlusion of the leading vehicle and is able to detect and track a vehicle even within unstructured offroad environments. Thanks to the used 3D model which describes the characteristic vehicle geometry and appearance in terms of vertexes, edges and colored surfaces, no special visual markers are required. The knowledge of vehicle's geometry and appearance gained from the model are used within a particle filter to estimate the 6DoF position relative to the ego vehicle, thereby fusing edge as well as color information. We successfully use the proposed algorithm for pure vision based autonomous offroad convoy driving.

Three-dimensional ultrasound can be an effective imaging modality for image-guided interventions since it enables visualization of both the instruments and the tissue. For robotic applications, its realtime frame rates create the potential for image-based instrument tracking and servoing. These capabilities can enable improved instrument visualization, compensation for tissue motion as well as surgical task automation. Continuum robots, whose shape comprises a smooth curve along their length, are well suited for minimally invasive procedures. Existing techniques for ultrasound tracking, however, are limited to straight, laparoscopic-type instruments and thus are not applicable to continuum robot tracking. Toward the goal of developing tracking algorithms for continuum robots, this paper presents a method for detecting a robot comprised of a single constant curvature in a 3D ultrasound volume. Computational efficiency is achieved by decomposing the six-dimensional circle estimation problem into two sequential three-dimensional estimation problems. Simulation and experiment are used to evaluate the proposed method.

An automatic parking system relies on precise estimation of parking space geometry. This paper proposes the use of a hierarchical three dimensional occupancy grid for the detection of parking spaces. The occupancy grid covers the environment representation of the static world. A hierarchical design allows dynamic selection of the level of detail. Applying a three-dimensional grid provides the additional benefit of supporting a variety of other functions including height estimation using a single environment representation type. The presented approach derives the distance to obstacles and walls and thus is able to represent the free space that forms parking spaces. In a second step, the dimensions of the parking space are calculated. For evaluation, real parking spaces are detected and estimated using short range radar sensors. The calculated dimensions are compared to the ground truth.

This paper presents a novel and robust method to determine the outline contour of vehicles in 3D-LIDAR (Light Detection and Ranging) measurements. To calculate the outline contour, a vehicle is described by its geometrical properties. These properties are used as constraints to fit a surface to unordered, scattered and error-contaminated 3D measurements. The surface can be used to calculate a corresponding 2D outline contour. The algorithm is tested with two different laser scanners. One scanner has 64, the other only 4 layers.

In this paper we describe a novel approach to autonomous dirt road following. The algorithm is able to recognize highly curved roads in cluttered color images quite often appearing in offroad scenarios. To cope with large curvatures we apply gaze control and model the road using two different clothoid segments. A Particle Filter incorporating edge and color intensity information is used to simultaneously detect and track the road farther away from the ego vehicle. In addition the particles are used to generate static road segment estimations in a given look ahead distance. These estimations are predicted with respect to ego motion and fused utilizing Kalman filter techniques to generate a smooth local clothoid segment for lateral control of the vehicle.

This paper describes a method for an object related reactive offset maneuver increasing the number of action alternatives for an autonomous ground vehicle. The method improves a variety of time-critical maneuvers as for example merging into moving traffic, changing lanes, passing other traffic participants as well as emergency obstacle avoidance without having to replan a path using high-level planning methods. It is especially targeted at autonomous driving on country roads. As oncoming traffic increases the relative speed between participants, a quick response to changes of the traffic situation are critically important. Moreover the method does not require constant vehicle velocities throughout a lane change maneuver as most existing approaches do. Simulations show improvements of the behaviour capabilities of an autonomous vehicle using the presented lateral reactive offset maneuvers.

This paper describes a fast method for segmentation of large-size long-range 3D point clouds that especially lends itself for later classification of objects. Our approach is targeted at high-speed autonomous ground robot mobility, so real-time performance of the segmentation method plays a critical role. This is especially true as segmentation is considered only a necessary preliminary for the more important task of object classification that is itself computationally very demanding. Efficiency is achieved in our approach by splitting the segmentation problem into two simpler subproblems of lower complexity: local ground plane estimation followed by fast 2D connected components labeling. The method's performance is evaluated on real data acquired in different outdoor scenes, and the results are compared to those of existing methods. We show that our method requires less runtime while at the same time yielding segmentation results that are better suited for later classification of the identified objects.

In this paper, a generic approach for three-dimensional environment representation is presented. Scans from range finders are accumulated into a three-dimensional occupancy grid. A probabilistic measurement model is used to represent measurement uncertainties. Free regions are modelled as well and contribute to a precise representation of the environment. In order to acquire a feasible three-dimensional grid, a hierarchical data structure is proposed. As a positive result, the level of detail and respectively the grid resolution is controllable by the application or by the content. As an example application, it will be shown how height information can be derived from a sensor with one horizontal scan plane only.

Autonomous navigation in unstructured environments like forest or country roads with dynamic objects remains a challenging task, particularly with respect to the perception of the environment using multiple different sensors. The problem has been addressed from both, the computer vision community as well as from researchers working with laser range finding technology, like the Velodyne HDL-64. Since cameras and LIDAR sensors complement one another in terms of color and depth perception, the fusion of both sensors is reasonable in order to provide color images with depth and reflectance information as well as 3D LIDAR point clouds with color information. In this paper we propose a method for sensor synchronization, especially designed for dynamic scenes, a low-level fusion of the data of both sensors and we provide a solution for the occlusion problem that arises in conjunction with different viewpoints of the fusioned sensors.

Active gaze stabilization is of vital importance for the use of high resolution tele-cameras in autonomous vehicles. Small aperture angles together with large focal lengths cause high sensitivity to rotational vehicle motion induced e.g. by bumps or braking. Due to large latencies in image processing only gaze stabilization based on inertial sensors is fast enough to ensure stable images. As this is a feed-forward control, imperfections of the sensor or the stabilizing actuator may result in undesireable image motion. To further enhance image stabilization we propose a novel vision-based online-calibration of the inertial angular rate sensor of our camera platform. Thus we are able to incorporate visual feedback into the gaze stabilization, while keeping the high bandwith of the inertial sensor.

Active multi-focal vision systems are designed to perform visual tasks such as saccades and smooth pursuit movements. Saccades are fast movements to a given position, which is a time-optimal control problem. Smooth pursuit movement is used to follow moving objects and is a tracking control problem. Therefore we use a hybrid approach for the control of our vision system. For saccades we use sliding-mode control with a switching line, designed such that nearly time-optimal control performance is achieved. For smooth pursuit movement we use state-space control to attain adequate tracking performance. To widen the applicability of our multi-focal vision system, it is designed such that cameras are easily exchangeable. Thus system dynamics is subject to change, which implies the need for adaptive control. This paper presents the design of the resulting hybrid adaptive controller together with experimental results.

This paper describes a LIDAR-based perception system for ground robot mobility, consisting of 3D object detection, classification and tracking. The presented system was demonstrated on-board our autonomous ground vehicle MuCAR-3, enabling it to safely navigate in urban traffic-like scenarios as well as in off-road convoy scenarios. The efficiency of our approach stems from the unique combination of 2D and 3D data processing techniques. Whereas fast segmentation of point clouds into objects is done in a 2¿D occupancy grid, classifying the objects is done on raw 3D point clouds. For fast object feature extraction, we advocate the use of statistics of local point cloud properties, captured by histograms over point features. In contrast to most existing work on 3D point cloud classification, where real-time operation is often impossible, this combination allows our system to perform in real-time at 0.1s frame-rate.