We tackle the long-term prediction of scene evolution in a complex downtown scenario for automated driving based on Lidar grid fusion and recurrent neural networks (RNNs). A bird's eye view of the scene, including occupancy and velocity, is fed as a sequence to a RNN which is trained to predict future occupancy. The nature of prediction allows generation of multiple hours of training data without the need of manual labeling. Thus, the training strategy and loss function are designed for long sequences of real-world data (unbalanced, continuously changing situations, false labels, etc.). The deep CNN architecture comprises convolutional long short-term memories (ConvLSTMs) to separate static from dynamic regions and to predict dynamic objects in future frames. Novel recurrent skip connections show the ability to predict small occluded objects, i.e. pedestrians, and occluded static regions. Spatio-temporal correlations between grid cells are exploited to predict multimodal future paths and interactions between objects. Experiments also quantity improvements to our previous network, a Monte Carlo approach, and literature.

Long-term situation prediction plays a crucial role for intelligent vehicles. A major challenge still to overcome is the prediction of complex downtown scenarios with multiple road users, e.g., pedestrians, bikes, and motor vehicles, interacting with each other. This contribution tackles this challenge by combining a Bayesian filtering technique for environment representation, and machine learning as long-term predictor. More specifically, a dynamic occupancy grid map is utilized as input to a deep convolutional neural network. This yields the advantage of using spatially distributed velocity estimates from a single time step for prediction, rather than a raw data sequence, alleviating common problems dealing with input time series of multiple sensors. Furthermore, convolutional neural networks have the inherent characteristic of using context information, enabling the implicit modeling of road user interaction. Pixel-wise balancing is applied in the loss function counteracting the extreme imbalance between static and dynamic cells. One of the major advantages is the unsupervised learning character due to fully automatic label generation. The presented algorithm is trained and evaluated on multiple hours of recorded sensor data and compared to Monte-Carlo simulation. Experiments show the ability to model complex interactions.

We tackle the problem of object detection and pose estimation in a shared space downtown environment. For perception multiple laser scanners with $360^{o}$ coverage were fused in a dynamic occupancy grid map (DOGMa). A single-stage deep convolutional neural network is trained to provide object hypotheses comprising of shape, position, orientation and an existence score from a single input DOGMa. Furthermore, an algorithm for offline object extraction was developed to automatically label several hours of training data. The algorithm is based on a two-pass trajectory extraction, forward and backward in time. Typical for engineered algorithms, the automatic label generation suffers from misdetections, which makes hard negative mining impractical. Therefore, we propose a loss function counteracting the high imbalance between mostly static background and extremely rare dynamic grid cells. Experiments indicate, that the trained network has good generalization capabilities since it detects objects occasionally lost by the label algorithm. Evaluation reaches an average precision (AP) of 75.9%.

A dynamic occupancy grid map (DOGMa) allows a fast, robust, and complete environment representation for automated vehicles. Dynamic objects in a DOGMa, however, are commonly represented as independent cells while modeled objects with shape and pose are favorable. The evaluation of algorithms for object extraction or the training and validation of learning algorithms rely on labeled ground truth data. Manually annotating objects in a DOGMa to obtain ground truth data is a time consuming and expensive process. Additionally the quality of labeled data depend strongly on the variation of filtered input data. The presented work introduces an automatic labeling process, where a full sequence is used to extract the best possible object pose and shape in terms of temporal consistency. A two direction temporal search is executed to trace single objects over a sequence, where the best estimate of its extent and pose is refined in every time step. Furthermore, the presented algorithm only uses statistical constraints of the cell clusters for the object extraction instead of fixed heuristic parameters. Experimental results show a well-performing automatic labeling algorithm with real sensor data even at challenging scenarios.

A major challenge in autonomous driving is the prediction of complex downtown scenarios with mutiple road users. This contribution tackles this challenge by combining a Bayesian filtering technique for environment representation and machine learning as long-term predictor. Therefore, a dynamic occupancy grid map representing the static and dynamic environment around the ego-vehicle is utilized as input to a deep convolutional neural network. This yields the advantage of using data from a single timestamp for prediction, rather than an entire time series. Furthermore, convolutional neural networks have the inherent characteristic of using context information, enabling the implicit modeling of road user interaction. One of the major advantages is the unsupervised learning character due to fully automatic label generation. The presented algorithm is trained and evaluated on multiple hours of recorded sensor data containing multiple road users, e.g., pedestrians, bikes and vehicles.

Recent advances in the field of environment perception and cognition enable automated vehicles to safely drive in a growing variety of complex situations. However, in situations where required information cannot be observed directly and thus the consequences of the vehicle's actions cannot be estimated with high certainty, generating a safe behavior is still an unsolved problem. This paper tackles the scenario of a left turn maneuver in an urban environment with the presence of blind corners. We consider pedestrians and vehicles possibly hidden by parking cars, buildings or vegetation. In these cases, our approach allows to safely merge into traffic by using an environment representation based on tracked objects as well as an object-free sensor fusion including the calculation of unobservable regions in a digital map. A free-to-drive section of our desired path is obtained by long-term propagation of observed or possibly unobservable movement. The presented approach allows advancing into the road in a cautious manner, successively increasing the observable area.

Long-term prediction of traffic participants is crucial to enable autonomous driving on public roads. The quality of the prediction directly affects the frequency of trajectory planning. With a poor estimation of the future development, more computational effort has to be put in re-planning, and a safe vehicle state at the end of the planning horizon is not guaranteed. A holistic probabilistic prediction, considering inputs, results and parameters as random variables, highly reduces the problem. A time frame of several seconds requires a probabilistic description of the scene evolution, where uncertainty or accuracy is represented by the trajectory distribution. Following this strategy, a novel evaluation method is needed, coping with the fact, that the future evolution of a scene is also uncertain. We present a method to evaluate the probabilistic prediction of real traffic scenes with varying start conditions. The proposed prediction is based on a particle filter, estimating behavior describing parameters of a microscopic traffic model. Experiments on real traffic data with random leading vehicles show the applicability in terms of convergence, enabling long-term prediction using forward propagation.

Vehicle localization is one of the most common tasks in robotics. Using vision as a sensor, most methods perform localization from cameras mounted on board the vehicle. In contrast, we propose a method based on an off-board camera. The system uses a similarity measure between a camera image and a synthetic image generated using a 3-D vehicle model, ideally converging to the true pose of the vehicle. The similarity measure is based on a model of shading appearance, which depends on the surface curvature of the 3-D model. Considering that the area observed by the camera is fairly planar, a rough initial estimation for the position of the object can be obtained using 2-D blob tracking. The 3-D model of the vehicle can be rendered near the vehicle in the real image, and using the proposed similarity measure, it converges to the correct pose. A classification function to discriminate between different vehicles is also proposed, making it possible for the system to identify and track multiple vehicles of interest. A number of experiments are performed with different vehicles outdoors, in a real industrial environment, considering different illumination conditions. The low average error rate compared with a laser-based ground truth illustrates the applicability of the method.