446

This paper presents a novel contingency planning framework that integrates learning-based multi-modal predictions of traffic participants into Branch Model Predictive Control (MPC). Leveraging reachability analysis, we address the computational challenges associated with Branch MPC by organizing the multitude of predictions into driving corridors. Analyzing the overlap between these corridors, their number can be reduced through pruning and clustering while ensuring safety since all prediction modes are preserved. These processed corridors directly correspond to the distinct branches of the scenario tree and provide an efficient constraint representation for the Branch MPC. We further utilize the reachability for determining maximum feasible decision postponing times, ensuring that branching decisions remain executable. Qualitative and quantitative evaluations demonstrate significantly reduced computational complexity and enhanced safety and comfort.

Model Predictive Control lacks the ability to escape local minima in nonconvex problems. Furthermore, in fast-changing, uncertain environments, the conventional warmstart, using the optimal trajectory from the last timestep, often falls short of providing an adequately close initial guess for the current optimal trajectory. This can potentially result in convergence failures and safety issues. Therefore, this paper proposes a framework for learning-aided warmstarts of Model Predictive Control algorithms. Our method leverages a neural network based multimodal predictor to generate multiple trajectory proposals for the autonomous vehicle, which are further refined by a sampling-based technique. This combined approach enables us to identify multiple distinct local minima and provide an improved initial guess. We validate our approach with Monte Carlo simulations of traffic scenarios.

LiDAR Semantic Segmentation is a fundamental task in autonomous driving perception consisting of associating each LiDAR point to a semantic label. Fully-supervised models have widely tackled this task, but they require labels for each scan, which either limits their domain or requires impractical amounts of expensive annotations.Camera images, which are generally recorded alongside LiDAR pointclouds, can be processed by the widely available 2D foundation models, which are generic and dataset-agnostic. However, distilling knowledge from 2D data to improve LiDAR perception raises domain adaptation challenges. For example, the classical perspective projection suffers from the parallax effect produced by the position shift between both sensors at their respective capture times.We propose a Semi-Supervised Learning setup to leverage unlabeled LiDAR pointclouds alongside distilled knowledge from the camera images. To self-supervise our model on the unlabeled scans, we add an auxiliary NeRF head and cast rays from the camera viewpoint over the unlabeled voxel features. The NeRF head predicts densities and semantic logits at each sampled ray location which are used for rendering pixel semantics. Concurrently, we query the Segment-Anything (SAM) foundation model with the camera image to generate a set of unlabeled generic masks. We fuse the masks with the rendered pixel semantics from LiDAR to produce pseudo-labels that supervise the pixel predictions. During inference, we drop the NeRF head and run our model with only LiDAR.We show the effectiveness of our approach in three public LiDAR Semantic Segmentation benchmarks: nuScenes, SemanticKITTI and ScribbleKITTI.

Robustness against Out-of-Distribution (OoD) samples is a key performance indicator of a trajectory prediction model. However, the development and ranking of state-of-the-art (SotA) models are driven by their In-Distribution (ID) performance on individual competition datasets. We present an OoD testing protocol that homogenizes datasets and prediction tasks across two large-scale motion datasets. We introduce a novel prediction algorithm based on polynomial representations for agent trajectory and road geometry on both the input and output sides of the model. With a much smaller model size, training effort, and inference time, we reach near SotA performance for ID testing and significantly improve robustness in OoD testing. Within our OoD testing protocol, we further study two augmentation strategies of SotA models and their effects on model generalization. Highlighting the contrast between ID and OoD performance, we suggest adding OoD testing to the evaluation criteria of trajectory prediction models.

Cooperative Simultaneous Localization and Mapping (C-SLAM) is an active research topic in mobile robotics. However, its application in the field of autonomous driving is rare. While the advent of Vehicle-to-Everything (V2X) communication has empowered Connected Autonomous Vehicles (CAV) to exchange data with each other, recent research on CAV cooperation tasks has primarily focused on cooperative perception and global positioning improvement. Techniques for organizing multiple CAV to work together to achieve localization and mapping in unknown environments have not been actively explored. We propose a C-SLAM system for CAVs that employs sparse LiDAR feature representations to enable vehicles to exchange data using standard V2X messages. The system was tested in real environments using two connected vehicles. The results show that the proposed V2X-based C-SLAM system can operate in both centralized and decentralized manners and output accurate pose estimates and global maps, showing promising application possibilities.

Simultaneous localization and mapping (SLAM) is an essential technique for autonomous driving. Recently, combining image recognition technology to generate semantically meaningful maps has become a new trend in visual SLAM research. However, in the field of LiDAR SLAM, this potential has not been fully explored. We propose a novel object-level SLAM system using 3D LiDARs for autonomous vehicles. We detect and track poles, walls, and parked cars, which are common along urban roads. This paper presents how we process the measurement data of three different shapes of objects to build a graph-based optimization system and facilitate the geometric distribution of poles to detect loops. Experiments were carried out on datasets collected with a test vehicle in city traffic. The results show that our object-level SLAM system can build precise and semantically meaningful maps and produce more accurate pose estimations compared to the state-of-the-art systems on our datasets.

Collision checking and avoidance is an import part of the perception and planning system for autonomous driving. We present a new analytic approach to calculate the probability of a future collision and extend another already known solution to be suitable for ground vehicle navigation. Our new concept of the collision octagon facilitates in both cases the derivation of an analytic solution. Both approaches are compared to each other using simulated and real world scenarios. By comparing the results of the analytic solutions to the corresponding Monte Carlo simulations, their accuracy and real-time capability is demonstrated. The suitability of the analytic solutions for real world autonomous systems is further proven by integrating them into the trajectory prediction and planning system of the self-driving car of the Freie Universität Berlin.

In response to the rise in logistics costs, the degree of automation in the logistics process chain is to be significantly increased in the coming years. To create a basic understanding for the proposed work, the scope of logistics, the existing robot hardware and the first version of its perception modules are covered. The considered perception algorithm, consists of three modules: object detection, object selection and object localization. Subsequently, the performance of state of the art deep neural networks used in this system is analyzed in more detail using a specially created mobile application. The error clusters resulting from this analysis - the strong temporal variance and the false detections - are then countered by extensions to the perception algorithm. While the temporal uncertainties can be eliminated by an aggregation module, a validation module makes it possible to find missing or incorrect detections by including domain-specific context knowledge. The latter module reduces the error rate of the object detection from 10 % to 2 %. Although the former has only a minimal influence on object detection, it improves the performance in the object localization by 10 %. The combination of both modules with the existing perception algorithm allows a faultless use of the robot under the industrial conditions of the logistics environment in the automotive production without risking process stops, damaged parts or even human injuries.

Due to dynamics, flexibility and diversity in logistics, perception-controlled, intelligent robots are required to automate logistical handling steps. Due to the additional optical influences of the industrial environment, such as labeling or damage, these applications seem predestined for the use of generalizing deep neural networks (DNN). These showed continuous improvements over the last few years based on publicly available data sets. If these DNNs are re-trained based on training data from the industrial environment, a lower performance can be observed. The additional extension of the experiments to international locations of the vehicle plants also showed that a drop in performance can be observed in the implementation of a network trained in Germany, for example, when it is used in America. However, in order to be able to use such robots in the logistic processes in the future, further measures such as a revised composition of training data or their extension by data augmentation are proposed.

In this paper, a method based on vector fields for the navigation of autonomous cars is developed. Vector fields-used to generate the desired heading angle of a vehicle toward a specified road lane—attract the car to the desired path and prevent the car from colliding with obstacles. Also, a control law is developed to define the velocity direction and the desired steering angle based on the angle between the car and the vector field. The efficacy of the proposed approach is investigated through several simulations and lab experimental tests.

Today's car traffic is dominated by human drivers. Autonomous cars must comprehend the behavior of human drivers in order to fit in current daily traffic scenarios. To achieve this goal, analysis of the behavior of other traffic participants is necessary. In this paper we present a system to record, store, and analyze the movements of other traffic participants with an autonomous car and evaluate traffic maps, which are obtained from real world experiments. The evaluation shows that the maps cover more than 80% of the driveable area with a precision of 80 to 90%. Additionally, we present the results of a traffic behavior change detection heuristic, which can detect anomalous traffic conditions.

We propose a trajectory planning approach for autonomous vehicles in highly dynamic traffic scenarios, using elastic bands to follow the observed trajectories of other vehicles. The focus of this paper is on the initialization of the elastic band. The proposed method does not rely on a map. We tested our method using recorded urban traffic data. The results show that the presented approach is valid and the proposed initialization process is clearly superior to naive initialization.

Environment sensing is an essential property for autonomous cars. With the help of sensors, nearby objects can be detected and localized. Furthermore, the creation of an accurate model of the surroundings is crucial for high-level planning. In this paper, we focus on vehicle detection based on stereo camera images. While stereoscopic computer vision is applied to localize objects in the environment, the objects are then identified by image classifiers. We implemented and evaluated several algorithms from image based pattern recognition in our autonomous car framework, using HOG-, LBP-, and Haar-like features. We will present experimental results using real traffic data with focus on classification accuracy and execution times.

In this paper we propose a trajectory planning approach for autonomous vehicles in highly dynamic traffic scenarios, capable of exploiting the observed trajectories of other vehicles. For this purpose, we introduce a novel variant of the timed elastic bands (TEB) approach by using fixed time intervals and a flexible goal position. We tested our method with a simulated merge-into-traffic scenario and compare it to a reference TEB implementation with focus on the impact of important parameters and the stability of planned trajectories. The results show that our method is an improvement over TEB in terms trajectory smoothness and stability.

This paper presents a system which constantly monitors the level of attention of a driver in traffic. The vehicle is instrumented and can identify the state of traffic-lights, as well as obstacles on the road. If the driver is inattentive and fails to recognize a threat, the assistance system produces a warning. Therefore, the system helps the driver to focus on crucial traffic situations. Our system consists of three components: computer vision detection of traffic-lights and other traffic participants, an eye tracking device used also for head localization, and finally, a human machine interface consisting of a head-up display and an acoustic module used to provide warnings to the driver. The orientation of the driver's head is detected using fiducial markers visible in video frames. We describe how the system was integrated using an autonomous car as experimental ADAS platform.

Localization is a key capability for autonomous vehicles especially in urban scenarios. We propose the use of pole-like landmarks as primary features in these environments, as they are distinct, long-term stable and can be detected reliably with a stereo camera system. Furthermore, the resulting map representation is memory efficient, allowing for easy storage and on-line updates. The localization is performed in real-time by a stereo camera system as a main sensor, using vehicle odometry and an off-the-shelf GPS as secondary information sources. Localization is performed by a particle filter approach, coupled with an Kalman filter for robustness and sensor fusion. This leads to a lateral accuracy below 20 cm in various urban test areas. The system has been included in our autonomous test vehicle and successfully demonstrated the full loop from mapping to autonomous driving.

In this paper we present a vehicle detection system using convolutional neural networks on 2d image data. Since realtime capabilities are crucial for object detection systems running in real-traffic situations, we will show how the calculation time of our algorithm can be significantly reduced by taking advantage of depth information from lidar sensors. One part of this work focusses on useful network topologies and network parameters to increase the classification precision. We will test the presented algorithm on an autonomous car in different real-traffic scenarios with regards to detection accuracy and calculation time and show experimental results.

In this paper we present an approach for building a graph of drivable paths from the reconstructed trajectories of vehicles detected by lidar and radar sensors mounted in an autonomous car. The perceived objects are tracked, and their trajectories are merged, clustered and labeled with meta information. A graph of the underlying road infrastructure can be generated with this information. We report on the results of testing the validity and accuracy of the method. The generated path graph can be used either to update high precision maps or for generating local temporary maps, both of them useful for autonomous driving.

Appearance-based estimation of grasp affordances is desirable when 3-D scans become unreliable due to clutter or material properties. We develop a general framework for estimating grasp affordances from 2-D sources, including local texture-like measures as well as object-category measures that capture previously learned grasp strategies. Local approaches to estimating grasp positions have been shown to be effective in real-world scenarios, but are unable to impart object-level biases and can be prone to false positives. We describe how global cues can be used to compute continuous pose estimates and corresponding grasp point locations, using a max-margin optimization for category-level continuous pose regression. We provide a novel dataset to evaluate visual grasp affordance estimation; on this dataset we show that a fused method outperforms either local or global methods alone, and that continuous pose estimation improves over discrete output models. Finally, we demonstrate our autonomous object detection and grasping system on the Willow Garage PR2 robot.

In the following paper, we present a framework for quickly training 2D object detectors for robotic perception. Our method can be used by robotics practitioners to quickly (under 30 seconds per object) build a large-scale real-time perception system. In particular, we show how to create new detectors on the fly using large-scale internet image databases, thus allowing a user to choose among thousands of available categories to build a detection system suitable for the particular robotic application. Furthermore, we show how to adapt these models to the current environment with just a few in-situ images. Experiments on existing 2D benchmarks evaluate the speed, accuracy, and flexibility of our system.

We propose a system for human-robot interaction that learns both models for spatial prepositions and for object recognition. Our system grounds the meaning of an input sentence in terms of visual percepts coming from the robot's sensors in order to send an appropriate command to the PR2 or respond to spatial queries. To perform this grounding, the system recognizes the objects in the scene, determines which spatial relations hold between those objects, and semantically parses the input sentence. The proposed system uses the visual and spatial information in conjunction with the semantic parse to interpret statements that refer to objects (nouns), their spatial relationships (prepositions), and to execute commands (actions). The semantic parse is inherently compositional, allowing the robot to understand complex commands that refer to multiple objects and relations such as: “Move the cup close to the robot to the area in front of the plate and behind the tea box”. Our system correctly parses 94% of the 210 online test sentences, correctly interprets 91% of the correctly parsed sentences, and correctly executes 89% of the correctly interpreted sentences.

In this paper we present a novel approach using constraint based techniques for world modeling, i.e. self localization and object modeling. Within the last years, we have seen a reduction of landmarks such as beacons or colored goals within the RoboCup domain. Using other features as line information becomes more important. Using such sensor data is tricky, especially when the resulting position belief is stretched over a larger area. Constraints can overcome this limitations, as they have several advantages: they can represent large distributions and are easy to store and to communicate to other robots. Propagation of several constraints can be computationally cheap. Even high dimensional belief functions can be used. We will describe a sample implementation and show experimental results.

This paper deals with how the absence of an expected sensor reading can be used to improve Markov localization. Negative information has not been used for robot localization for various reasons like sensor imperfections, and occlusions that make it hard to determine if a missing sensor reading is really caused by the absence of a feature. We address these difficulties by carefully modeling the robot's main sensor, its camera. Taking into account the viewing frustum and detected obstacles, the absence of a sensor reading can be associated with the absence of that particular feature. This information can then be integrated into the localization process. We show the positive effect on robot localization in various experiments. (a) In a specific setup, the robot is able to localize using negative information where without it, it is unable to localize. (b) We demonstrate the importance of modeling occlusions and the impact of false negatives on localization. (c) We show the positive impact in a typical run

In this paper we present a novel approach to estimating the position of objects tracked by a team of mobile robots and to use these objects for a better self localization. Modeling of moving objects is commonly done in a robo-centric coordinate frame because this information is sufficient for most low level robot control and it is independent of the quality of the current robot localization. For multiple robots to cooperate and share information, though, they need to agree on a global, allocentric frame of reference. When transforming the egocentric object model into a global one, it inherits the localization error of the robot in addition to the error associated with the egocentric model. We propose using the relation of objects detected in camera images to other objects in the same camera image as a basis for estimating the position of the object in a global coordinate system. The spatial relation of objects with respect to stationary objects (e.g., landmarks) offers several advantages: a) Errors in feature detection are correlated and not assumed independent. Furthermore, the error of relative positions of objects within a single camera frame is comparably small, b) The information is independent of robot localization and odometry. c) As a consequence of the above, it provides a highly efficient method for communicating information about a tracked object and communication can be asynchronous, d) As the modeled object is independent from robo-centric coordinates, its position can be used for self localization of the observing robot. We present experimental evidence that shows how two robots are able to infer the position of an object within a global frame of reference, even though they are not localized themselves and then use this object information for self- localization

This paper explores how the absence of an expected sensor reading can be used to improve Markov localization. This negative information usually is not being used in localization, because it yields less information than positive information (i.e. sensing a landmark), and a sensor often fails to detect a landmark, even if it falls within its sensing range. We address these difficulties by carefully modeling the sensor to avoid false negatives. This can also be thought of as adding an additional sensor that detects the absence of an expected landmark. We show how such modeling is done and how it is integrated into Markov localization. In real world experiments, we demonstrate that a robot is able to localize in positions where otherwise it could not and quantify our findings using the entropy of the particle distribution. Exploiting negative information leads to a greatly improved localization performance and reactivity.