The detection of small road hazards, such as lost cargo, is a vital capability for self-driving cars. We tackle this challenging and rarely addressed problem with a vision system that leverages appearance, contextual as well as geometric cues. To utilize the appearance and contextual cues, we propose a new deep learning-based obstacle detection framework. Here a variant of a fully convolutional network is proposed to predict a pixel-wise semantic labeling of (i) free-space, (ii) on-road unexpected obstacles, and (iii) background. The geometric cues are exploited using a state-of-the-art detection approach that predicts obstacles from stereo input images via model-based statistical hypothesis tests. We present a principled Bayesian framework to fuse the semantic and stereo-based detection results. The mid-level Stixel representation is used to describe obstacles in a flexible, compact and robust manner. We evaluate our new obstacle detection system on the Lost and Found dataset, which includes very challenging scenes with obstacles of only 5 cm height. Overall, we report a major improvement over the state-of-the-art, with a performance gain of 27.4%. In particular, we achieve a detection rate of over 90% for distances of up to 50 m. Our system operates at 22 Hz on our self-driving platform.

Detecting small obstacles on the road ahead is a critical part of the driving task which has to be mastered by fully autonomous cars. In this paper, we present a method based on stereo vision to reliably detect such obstacles from a moving vehicle.

Applications based on stereo vision are becoming increasingly common, ranging from gaming over robotics to driver assistance. While stereo algorithms have been investigated heavily both on the pixel and the application level, far less attention has been dedicated to the use of stereo confidence cues. Mostly, a threshold is applied to the confidence values for further processing, which is essentially a sparsified disparity map. This is straightforward but it does not take full advantage of the available information. In this paper, we make full use of the stereo confidence cues by propagating all confidence values along with the measured disparities in a Bayesian manner. Before using this information, a mapping from confidence values to disparity outlier probability rate is performed based on gathered disparity statistics from labeled video data. We present an extension of the so called Stixel World, a generic 3D intermediate representation that can serve as input for many of the applications mentioned above. This scheme is modified to directly exploit stereo confidence cues in the underlying sensor model during a maximum a posteriori estimation process. The effectiveness of this step is verified in an in-depth evaluation on a large real-world traffic data base of which parts are made publicly available. We show that using stereo confidence cues allows both reducing the number of false object detections by a factor of six while keeping the detection rate at a near constant level.

In this contribution we present an approach to compute object confidences for stereo-vision-based object tracking schemes. Meaningful object confidences help to reduce false alarm rates of safety systems and improve the downstream system performance for modules such as sensor fusion and situation analysis. Several cues from stereo vision and from the tracking process are fused in a Bayesian manner. An evaluation on a 38,000 frames urban drive shows the effectiveness of the approach compared to the same object tracking scheme with simple heuristics for the object confidence. Within the evaluation, also the relevance of occurring phantoms is considered by computing the collision risk. The proposed confidence measures reduce the number of predicted imminent collisions from 86 to 0 maintaining almost the same system availability.

We examine high accuracy stereo estimation for binocular sequences that where obtained from a mobile platform. The ultimate goal is to improve the range of stereo systems without altering the setup. Based on a well-known variational optical flow model, we introduce a novel stereo model that features a second-order regularization, which both allows sub-pixel accurate solutions and piecewise planar disparity maps. The model incorporates a robust fidelity term to account for adverse illumination conditions that frequently arise in real-world scenes. Using several sequences that were taken from a mobile platform we show the robustness and accuracy of the proposed model.

Stereo matching commonly requires rectified images that are computed from calibrated cameras. Since all underlying parametric camera models are only approximations, calibration and rectification will never be perfect. Additionally, it is very hard to keep the calibration perfectly stable in application scenarios with large temperature changes and vibrations. We show that even small calibration errors of a quarter of a pixel are severely amplified on certain structures. We discuss a robotics and a driver assistance example where sub-pixel calibration errors cause severe problems. We propose a filter solution based on signal theory that removes critical structures and makes stereo algorithms less sensitive to calibration errors. Our approach does not aim to correct decalibration, but rather to avoid amplifications and mismatches. Experiments on ten stereo pairs with ground truth and simulated decalibrations as well as images from robotics and driver assistance scenarios demonstrate the success and limitations of our solution that can be combined with any stereo method.

Dense stereo algorithms are able to estimate disparities at all pixels including untextured regions. Typically these disparities are evaluated at integer disparity steps. A subsequent sub-pixel interpolation often fails to propagate smoothness constraints on a sub-pixel level. The determination of sub-pixel accurate disparities is an active field of research, however, most sub-pixel estimation algorithms focus on textured image areas in order to show their precision. We propose to increase the sub-pixel accuracy in low- textured regions in three possible ways: First, we present an analysis that shows the benefit of evaluating the disparity space at fractional disparities. Second, we introduce a new disparity smoothing algorithm that preserves depth discontinuities and enforces smoothness on a sub-pixel level. Third, we present a novel stereo constraint (gravitational constraint) that assumes sorted disparity values in vertical direction and guides global algorithms to reduce false matches, especially in low-textured regions. Our goal in this work is to obtain an accurate 3D reconstruction. Large- scale 3D reconstruction will benefit heavily from these sub- pixel refinements, especially with a multi-baseline extension. Results based on semi-global matching , obtained with the above mentioned algorithmic extensions are shown for the Middlebury stereo ground truth data sets. The presented improvements, called ImproveSubPix, turn out to be one of the top-performing algorithms when evaluating the set on a sub-pixel level while being computationally efficient. Additional results are presented for urban scenes. The three improvements are independent of the underlying type of stereo algorithm and can also be applied to sparse stereo algorithms.

The vehicle-following concept has been widely used in several intelligent-vehicle applications. Adaptive cruise control systems, platooning systems, and systems for stop-and-go traffic employ this concept: The ego vehicle follows a leader vehicle at a certain distance. The vehicle-following concept comes to its limitations when obstacles interfere with the path between the ego vehicle and the leader vehicle. We call such situations dynamic driving situations. This paper introduces a planning and decision component to generalize vehicle following to situations with nonautomated interfering vehicles in mixed traffic. As a demonstrator, we employ a car that is able to navigate autonomously through regular traffic that is longitudinally and laterally guided by actuators controlled by a computer. This paper focuses on and limits itself to lateral control for collision avoidance. Previously, this autonomous-driving capability was purely based on the vehicle-following concept using vision. The path of the leader vehicle was tracked. To extend this capability to dynamic driving situations, a dynamic path-planning component is introduced. Several driving situations are identified that necessitate responses to more than the leader vehicle. We borrow an idea from robotics to solve the problem. Treat the path of the leader vehicle as an elastic band that is subjected to repelling forces of obstacles in the surroundings. This elastic-band framework offers the necessary features to cover dynamic driving situations. Simulation results show the power of this approach. Real-world results obtained with our demonstrator validate the simulation results

We propose a novel model-free pose estimation algorithm to estimate the relative pose of a rigid object. In most pose estimation algorithms, the object of interest covers a large portion of the image. We focus on pose estimation of small objects covering a field of view of less than 5° by 5° using stereo vision. With this new algorithm suitable for small objects, we investigate the effect of the object size on the pose accuracy. In addition, we introduce an object tracking technique that is insensitive to partial occlusion. The main application for this method is the analysis of crash video sequences. In this context, high-speed cameras with high resolution are used. However, the method easily extends to real-time robotics applications with VGA-size images, where relative pose estimation is needed, e.g. for manipulator control.

Our main objective in this paper is to perform a cartography of a road scene into a reference frame at rest, where 3D measurements delivered by on-board sensors serve as input. The main sensors of our autonomous vehicle are two CCD cameras. Their pictures are combined using stereopsis to generate 3D data. We need dead reckoning to properly associate 3D data among the frames. This necessitates us to obtain a precise ego-motion estimation. Dead reckoning using only standard vehicle odometry can cause nonnegligible errors. We use stationary points in the scene to support the determination of our ego-motion. Two types of stationary objects are used: vertical landmarks such as traffic signs and lane markings are used to compensate positioning errors. Preliminary results show that cartography as proposed in this paper is beneficial to detect stationary objects but needs further work for fast moving objects.

Our main objective in the paper is to perform a cartography of a road scene into a reference frame at rest, where 3D measurements delivered by on-board sensors serve as input. The main sensors of our autonomous vehicle are two CCD cameras. Their pictures are combined using stereopsis to generate 3D data. We need dead reckoning to properly associate 3D data among the frames. This necessitates us to obtain a precise ego-motion estimation. Dead reckoning using only standard vehicle odometry (velocity and steering angle) can cause non-negligible errors, especially in situations where side slip or skidding occurs. We use stationary points in the scene to support the determination of our ego-motion. Two types of stationary objects are used: Firstly, stationary vertical landmarks such as traffic signs are used to compensate errors in our localization prediction. Secondly, lane markings measured in consecutive frames are used to compensate orientation errors. Preliminary results show that dead reckoning using stationary objects can vastly improve self-localization.

A crucial task for steering an autonomous vehicle along a safe path in a car following scenario is the lateral control. The sensory input of such a lateral control are the position coordinates of the leader vehicle. The following problem occurs: due to the distance between the leader vehicle and the autonomous ego-vehicle, the lateral control has to interpolate a trajectory between the two vehicles. Using as a trajectory either a straight line or a curve of constant curvature causes the ego-vehicle to deviate from the leader vehicle's trajectory. Given a system delivering 3D points of the leader vehicle with time lags, one has a handle to reconstruct the leader vehicle's trajectory. In addition, one has to compensate the motion of the ego-vehicle by using its motion parameters. Once this transformation is performed, the position coordinates of the leader vehicle are available in a coordinate system at rest. Knowing the position of the ego-vehicle in that coordinate system, one can select the trajectory point of the leader vehicle that is closest to the ego-vehicle as input to the lateral controller. This simple approach increases significantly the precision of car following systems. The algorithm is applied successfully to an autonomous car for platooning at small velocities.