Self-driving cars relying solely on ego-centric perception face limitations in sensing, often failing to detect occluded, faraway objects. Collaborative autonomous driving (CAV) seems like a promising direction, but collecting data for development is non-trivial. It requires placing multiple sensor-equipped agents in a real-world driving scene, simultaneously! As such, existing datasets are limited in locations and agents. We introduce a novel surrogate to the rescue, which is to generate realistic perception from different viewpoints in a driving scene, conditioned on a real-world sample—the ego-car’s sensory data. This surrogate has huge potential: it could potentially turn any ego-car dataset into a collaborative driving one to scale up the development of CAV. We present the very first solution, using a combination of simulated collaborative data and real ego-car data. Our method Transfer Your Perspective (TYP) learns a conditioned diffusion model whose output samples are not only realistic but also consistent in both semantics and layouts with the given ego-car data. Empirical results demonstrate TYP’s effectiveness in aiding in a CAV setting. In particular, TYP enables us to (pre-)train collaborative perception algorithms like early and late fusion with little or no real-world collaborative data, greatly facilitating downstream CAV applications.

Vehicle-to-everything (V2X) collaborative perception has emerged as a promising solution to address the limitations of single-vehicle perception systems. However, existing V2X datasets are limited in scope, diversity, and quality. To address these gaps, we present Mixed Signals, a comprehensive V2X dataset featuring 45.1k point clouds and 240.6k bounding boxes collected from three connected autonomous vehicles (CAVs) equipped with two different configurations of LiDAR sensors, plus a roadside unit with dual LiDARs. Our dataset provides point clouds and bounding box annotations across 10 classes, ensuring reliable data for perception training. We provide detailed statistical analysis on the quality of our dataset and extensively benchmark existing V2X methods on it. The Mixed Signals dataset is ready-to-use, with precise alignment and consistent annotations across time and viewpoints. Dataset website is available at https://mixedsignalsdataset.cs.cornell.edu/.

Accurate 3D object detection in real-world environments requires a huge amount of annotated data with high quality. Acquiring such data is tedious and expensive, and often needs repeated effort when a new sensor is adopted or when the detector is deployed in a new environment. We investigate a new scenario to construct 3D object detectors: *learning from the predictions of a nearby unit that is equipped with an accurate detector.* For example, when a self-driving car enters a new area, it may learn from other traffic participants whose detectors have been optimized for that area. This setting is label-efficient, sensor-agnostic, and communication-efficient: nearby units only need to share the predictions with the ego agent (e.g., car). Naively using the received predictions as ground-truths to train the detector for the ego car, however, leads to inferior performance. We systematically study the problem and identify viewpoint mismatches and mislocalization (due to synchronization and GPS errors) as the main causes, which unavoidably result in false positives, false negatives, and inaccurate pseudo labels. We propose a distance-based curriculum, first learning from closer units with similar viewpoints and subsequently improving the quality of other units' predictions via self-training. We further demonstrate that an effective pseudo label refinement module can be trained with a handful of annotated data, largely reducing the data quantity necessary to train an object detector. We validate our approach on the recently released real-world collaborative driving dataset, using reference cars' predictions as pseudo labels for the ego car. Extensive experiments including several scenarios (e.g., different sensors, detectors, and domains) demonstrate the effectiveness of our approach toward label-efficient learning of 3D perception from other units' predictions.

A novel framework for training a robotic fish to learn how to swim, even in the presence of degradations or failures in actuators is developed. Robotic underwater robots, particularly soft fish-inspired designs have gained significant attention due to their distinct benefits, including superior maneuverability, energy efficiency, versatile applications, and seamless integration with marine environments. However, their material properties and actuators can degrade, leading to pre-mature system failures. In this paper, we introduce the concept of actuator drop-out during training, to enable the robot to learn how to swim even when one or more actuators are degraded or non-functional. A Soft Actor-Critic Deep Reinforcement Learning architecture is used to learn a policy, with actuator degradations/failures introduced during training. A four actuator koi fish is modeled and simulated using the FishGym environment. Navigation-based validation tests show little degradation with one actuator failure, and much more robust swimming behaviors and performance compared to training with no failures, even when two or three actuators fail. These results will improve long-term operational reliability, ensuring robot fish functionality even in challenging underwater conditions.

Accurate 3D object detection and understanding for self-driving cars heavily relies on LiDAR point clouds, necessitating large amounts of labeled data to train. In this work, we introduce an innovative pre-training approach, Grounded Point Colorization (GPC), to bridge the gap between data and labels by teaching the model to colorize LiDAR point clouds, equipping it with valuable semantic cues. To tackle challenges arising from color variations and selection bias, we incorporate color as "context" by providing ground-truth colors as hints during colorization. Experimental results on the KITTI and Waymo datasets demonstrate GPC's remarkable effectiveness. Even with limited labeled data, GPC significantly improves fine-tuning performance; notably, on just 20% of the KITTI dataset, GPC outperforms training from scratch with the entire dataset. In sum, we introduce a fresh perspective on pre-training for 3D object detection, aligning the objective with the model's intended role and ultimately advancing the accuracy and efficiency of 3D object detection for autonomous vehicles.

Accurate 3D object detection is crucial to autonomous driving. Though LiDAR-based detectors have achieved impressive performance, the high cost of LiDAR sensors precludes their widespread adoption in affordable vehicles. Camera-based detectors are cheaper alternatives but often suffer inferior performance compared to their LiDAR-based counterparts due to inherent depth ambiguities in images. In this work, we seek to improve monocular 3D detectors by leveraging unlabeled historical LiDAR data. Specifically, at inference time, we assume that the camera-based detectors have access to multiple unlabeled LiDAR scans from past traversals at locations of interest (potentially from other high-end vehicles equipped with LiDAR sensors). Under this setup, we proposed a novel, simple, and end-to-end trainable framework, termed AsyncDepth, to effectively extract relevant features from asynchronous LiDAR traversals of the same location for monocular 3D detectors. We show consistent and significant performance gain (up to 9 AP) across multiple state-of-the-art models and datasets with a negligible additional latency of 9.66 ms and a small storage cost. Our code can be found at https://github.com/YurongYou/AsyncDepth.

Predicting agents’ trajectories in complex environments is critical for achieving safe autonomous robot navigation. Empirically, agents’ decisions and preferences are susceptible to changes in environmental factors (e.g., interactions with other agents, weather conditions, traffic rules). State-of-the-art methods rely on High-Definition (HD) or semantic maps to model the environment, but do not take into account unpredictable factors such as complex weather conditions. In addition, since HD maps are nontrivial to obtain, those methods are limited in the scope of environments they can be applied in. We propose a more flexible graph based trajectory prediction model that uses only images to model the environment, without requiring expensive map information. We experimentally validate our proposed model, demonstrating robust performances in trajectory prediction compared to state-of-the-art methods, and outperform in complex environments that cannot be modeled with purely map based methods, such as diverse weather conditions.

Ensuring robust 3D object detection and localization is crucial for many applications in robotics and autonomous driving. Recent models, however, face difficulties in maintaining high performance when applied to domains with differing sensor setups or geographic locations, often resulting in poor localization accuracy due to domain shift. To overcome this challenge, we introduce a novel diffusion-based box refinement approach. This method employs a domain-agnostic diffusion model, conditioned on the LiDAR points surrounding a coarse bounding box, to simultaneously refine the box's location, size, and orientation. We evaluate this approach under various domain adaptation settings, and our results reveal significant improvements across different datasets, object classes and detectors. Our PyTorch implementation is available at https://github.com/cxy1997/DiffuBox.

The uncertainty quantification of prediction models (e.g., neural networks) is crucial for their adoption in many robotics applications. This is arguably as important as making accurate predictions, especially for safety-critical applications such as self-driving cars. This paper proposes our approach to uncertainty quantification in the context of visual localization for autonomous driving, where we predict locations from images. Our proposed framework estimates probabilistic uncertainty by creating a sensor error model that maps an internal output of the prediction model to the uncertainty. The sensor error model is created using multiple image databases of visual localization, each with ground-truth location. We demonstrate the accuracy of our uncertainty prediction framework using the Ithaca365 dataset, which includes variations in lighting, weather (sunny, snowy, night), and alignment errors between databases. We analyze both the predicted uncertainty and its incorporation into a Kalman-based localization filter. Our results show that prediction error variations increase with poor weather and lighting condition, leading to greater uncertainty and outliers, which can be predicted by our proposed uncertainty model. Additionally, our probabilistic error model enables the filter to remove ad hoc sensor gating, as the uncertainty automatically adjusts the model to the input data.

A self-driving car must be able to reliably handle adverse weather conditions (e.g., snowy) to operate safely. In this paper, we investigate the idea of turning sensor inputs (i.e., images) captured in an adverse condition into a benign one (i.e., sunny), upon which the downstream tasks (e.g., semantic segmentation) can attain high accuracy. Prior work primarily formulates this as an unpaired image-to-image translation problem due to the lack of paired images captured under the exact same camera poses and semantic layouts. While perfectly-aligned images are not available, one can easily obtain coarsely-paired images. For instance, many people drive the same routes daily in both good and adverse weather; thus, images captured at close-by GPS locations can form a pair. Though data from repeated traversals are unlikely to capture the same foreground objects, we posit that they provide rich contextual information to supervise the image translation model. To this end, we propose a novel training objective leveraging coarsely-aligned image pairs. We show that our coarsely-aligned training scheme leads to a better image translation quality and improved downstream tasks, such as semantic segmentation, monocular depth estimation, and visual localization.

Recent advances in machine learning have shown that Reinforcement Learning from Human Feedback (RLHF) can improve machine learning models and align them with human preferences. Although very successful for Large Language Models (LLMs), these advancements have not had a comparable impact in research for autonomous vehicles—where alignment with human expectations can be imperative. In this paper, we propose to adapt similar RL-based methods to unsupervised object discovery, i.e. learning to detect objects from LiDAR points without any training labels. Instead of labels, we use simple heuristics to mimic human feedback. More explicitly, we combine multiple heuristics into a simple reward function that positively correlates its score with bounding box accuracy, i.e., boxes containing objects are scored higher than those without. We start from the detector’s own predictions to explore the space and reinforce boxes with high rewards through gradient updates. Empirically, we demonstrate that our approach is not only more accurate, but also orders of magnitudes faster to train compared to prior works on object discovery. Code is available at https://github.com/katieluo88/DRIFT.

Advances in perception for self-driving cars have accelerated in recent years due to the availability of large-scale datasets, typically collected at specific locations and under nice weather conditions. Yet, to achieve the high safety requirement, these perceptual systems must operate robustly under a wide variety of weather conditions including snow and rain. In this paper, we present a new dataset to enable robust autonomous driving via a novel data collection process - data is repeatedly recorded along a 15 km route under diverse scene (urban, highway, rural, campus), weather (snow, rain, sun), time (day/night), and traffic conditions (pedestrians, cyclists and cars). The dataset includes images and point clouds from cameras and LiDAR sensors, along with high-precision GPS/INS to establish correspondence across routes. The dataset includes road and object annotations using amodal masks to capture partial occlusions and 3D bounding boxes. We demonstrate the uniqueness of this dataset by analyzing the performance of baselines in amodal segmentation of road and objects, depth estimation, and 3D object detection. The repeated routes opens new research directions in object discovery, continual learning, and anomaly detection. Link to Ithaca365: https://ithaca365.mae.cornell.edu/

Current 3D object detectors for autonomous driving are almost entirely trained on human-annotated data. Although of high quality, the generation of such data is laborious and costly, restricting them to a few specific locations and object types. This paper proposes an alternative approach entirely based on unlabeled data, which can be collected cheaply and in abundance almost everywhere on earth. Our approach leverages several simple common sense heuristics to create an initial set of approximate seed labels. For example, relevant traffic participants are generally not persistent across multiple traversals of the same route, do not fly, and are never under ground. We demonstrate that these seed labels are highly effective to bootstrap a surprisingly accurate detector through repeated self-training without a single human annotated label. Code is available at https://github.com/YurongYou/MODEST.

This paper investigates the usefulness of reasoning about the uncertain presence of obstacles during path planning, which typically stems from the usage of probabilistic occupancy grid maps for representing the environment when mapping via a noisy sensor like a stereo camera. The traditional planning paradigm prescribes using a hard threshold on the occupancy probability to declare that a cell is an obstacle, and to plan a single path accordingly while treating unknown space as free. We compare this approach against a new uncertainty-aware planner, which plans two different path hypotheses and then merges their initial trajectory segments into a single one ending in a “next-best view” pose. After this informative view is taken, the planner commits to one of the hypotheses, or to a completely new one if a collision is imminent. Simulations were conducted comparing the proposed and traditional planner. Results show the existence of planning scenarios -like when the environment contains a dead-end, or when the goal is placed close to an obstacle- in which reasoning about uncertainty can significantly decrease the robot's traveled distance and increase the chances of reaching the goal. The new planner was also validated on a real Clearpath Jackal robot equipped with a ZED 2 stereo camera.

Shape and pose estimation is a critical perception problem for a self-driving car to fully understand its surrounding environment. One fundamental challenge in solving this problem is the incomplete sensor signal (e.g., LiDAR scans), especially for faraway or occluded objects. In this paper, we propose a novel algorithm to address this challenge, which explicitly leverages the sensor signal captured over consecutive time: the consecutive signals can provide more information about an object, including different viewpoints and its motion. By encoding the consecutive signals via a recurrent neural network, not only our algorithm improves the shape and pose estimates, but also produces a labeling tool that can benefit other tasks in autonomous driving research. Specifically, building upon our algorithm, we propose a novel pipeline to automatically annotate high-quality labels for amodal segmentation on images, which are hard and laborious to annotate manually. Our code and data will be made publicly available.

Self-driving cars must detect other traffic participants like vehicles and pedestrians in 3D in order to plan safe routes and avoid collisions. State-of-the-art 3D object detectors, based on deep learning, have shown promising accuracy but are prone to over-fit domain idiosyncrasies, making them fail in new environments-a serious problem for the robustness of self-driving cars. In this paper, we propose a novel learning approach that reduces this gap by fine-tuning the detector on high-quality pseudo-labels in the target domain - pseudo-labels that are automatically generated after driving based on replays of previously recorded driving sequences. In these replays, object tracks are smoothed forward and backward in time, and detections are interpolated and extrapolated-crucially, leveraging future information to catch hard cases such as missed detections due to occlusions or far ranges. We show, across five autonomous driving datasets, that fine-tuning the object detector on these pseudo-labels substantially reduces the domain gap to new driving environments, yielding strong improvements detection reliability and accuracy.

There has been a plethora of work towards im-proving robot perception and navigation, yet their application in hazardous environments, like during a fire or an earthquake, is still at a nascent stage. We hypothesize two key challenges here: first, it is difficult to replicate such scenarios in the real world, which is necessary for training and testing purposes. Second, current systems are not fully able to take advantage of the rich multi-modal data available in such hazardous environments. To address the first challenge, we propose to harness the enormous amount of visual content available in the form of movies and TV shows, and develop a dataset that can represent hazardous environments encountered in the real world. The data is annotated with high-level danger ratings for realistic disaster images, and corresponding keywords are provided that summarize the content of the scene. In response to the second challenge, we propose a multi-modal danger estimation pipeline for collaborative human-robot escape scenarios. Our Bayesian framework improves danger estimation by fusing information from robot's camera sensor and language inputs from the human. Furthermore, we augment the estimation module with a risk-aware planner that helps in identifying safer paths out of the dangerous environment. Through extensive simulations, we exhibit the advantages of our multi-modal perception framework that gets translated into tangible benefits such as higher success rate in a collaborative human-robot mission.

For a self-driving car to operate reliably, its perceptual system must generalize to the end-user's environment --- ideally without additional annotation efforts. One potential solution is to leverage unlabeled data (e.g., unlabeled LiDAR point clouds) collected from the end-users' environments (i.e. target domain) to adapt the system to the difference between training and testing environments. While extensive research has been done on such an unsupervised domain adaptation problem, one fundamental problem lingers: there is no reliable signal in the target domain to supervise the adaptation process. To overcome this issue we observe that it is easy to collect unsupervised data from multiple traversals of repeated routes. While different from conventional unsupervised domain adaptation, this assumption is extremely realistic since many drivers share the same roads. We show that this simple additional assumption is sufficient to obtain a potent signal that allows us to perform iterative self-training of 3D object detectors on the target domain. Concretely, we generate pseudo-labels with the out-of-domain detector but reduce false positives by removing detections of supposedly mobile objects that are persistent across traversals. Further, we reduce false negatives by encouraging predictions in regions that are not persistent. We experiment with our approach on two large-scale driving datasets and show remarkable improvement in 3D object detection of cars, pedestrians, and cyclists, bringing us a step closer to generalizable autonomous driving.

We present a method for detecting and mapping trees in noisy stereo camera point clouds, using a learned 3D object detector. Inspired by recent advancements in 3-D object detection using a pseudo-lidar representation for stereo data, we train a PointRCNN detector to recognize trees in forest-like environments. We generate detector training data with a novel automatic labeling process that clusters a fused global point cloud. This process annotates large stereo point cloud training data sets with minimal user supervision, and unlike previous pseudo-lidar detection pipelines, requires no 3D ground truth from other sensors such as lidar. Our mapping system additionally uses a Kalman filter to associate detections and consistently estimate the positions and sizes of trees. We collect a data set for tree detection consisting of 8680 stereo point clouds, and validate our method on an outdoors test sequence. Our results demonstrate robust tree recognition in noisy stereo data at ranges of up to 7 meters, on 720p resolution images from a Stereolabs ZED 2 camera. Code and data are available at https://github.com/brian-h-wang/pseudolidar-tree-detection.

Reliable and accurate 3D object detection is a necessity for safe autonomous driving. Although LiDAR sensors can provide accurate 3D point cloud estimates of the environment, they are also prohibitively expensive for many settings. Recently, the introduction of pseudo-LiDAR (PL) has led to a drastic reduction in the accuracy gap between methods based on LiDAR sensors and those based on cheap stereo cameras. PL combines state-of-the-art deep neural networks for 3D depth estimation with those for 3D object detection by converting 2D depth map outputs to 3D point cloud inputs. However, so far these two networks have to be trained separately. In this paper, we introduce a new framework based on differentiable Change of Representation (CoR) modules that allow the entire PL pipeline to be trained end-to-end. The resulting framework is compatible with most state-of-the-art networks for both tasks and in combination with PointRCNN improves over PL consistently across all benchmarks --- yielding the highest entry on the KITTI image-based 3D object detection leaderboard at the time of submission. Our code will be made available at https://github.com/mileyan/pseudo-LiDAR_e2e.

In the domain of autonomous driving, deep learning has substantially improved the 3D object detection accuracy for LiDAR and stereo camera data alike. While deep networks are great at generalization, they are also notorious to overfit to all kinds of spurious artifacts, such as brightness, car sizes and models, that may appear consistently throughout the data. In fact, most datasets for autonomous driving are collected within a narrow subset of cities within one country, typically under similar weather conditions. In this paper we consider the task of adapting 3D object detectors from one dataset to another. We observe that naively, this appears to be a very challenging task, resulting in drastic drops in accuracy levels. We provide extensive experiments to investigate the true adaptation challenges and arrive at a surprising conclusion: the primary adaptation hurdle to overcome are differences in car sizes across geographic areas. A simple correction based on the average car size yields a strong correction of the adaptation gap. Our proposed method is simple and easily incorporated into most 3D object detection frameworks. It provides a first baseline for 3D object detection adaptation across countries, and gives hope that the underlying problem may be more within grasp than one may have hoped to believe. Our code is available at https://github. com/cxy1997/3D_adapt_auto_driving.

Planning in unstructured environments is challenging - it relies on sensing, perception, scene reconstruction, and reasoning about various uncertainties. We propose DeepSemanticHPPC, a novel uncertainty-aware hypothesis-based planner for unstructured environments. Our algorithmic pipeline consists of: a deep Bayesian neural network which segments surfaces with uncertainty estimates; a flexible point cloud scene representation; a next-best-view planner which minimizes the uncertainty of scene semantics using sparse visual measurements; and a hypothesis-based path planner that proposes multiple kinematically feasible paths with evolving safety confidences given next-best-view measurements. Our pipeline iteratively decreases semantic uncertainty along planned paths, filtering out unsafe paths with high confidence. We show that our framework plans safe paths in real-world environments where existing path planners typically fail.

We present a probabilistic framework for detailed 3-D shape estimation and tracking using only vision measurements. Vision detections are processed via a bird's eye view representation, creating accurate detections at far ranges. A probabilistic model of the vision based point cloud measurements is learned and used in the framework. A 3-D shape model is developed by fusing a set of point cloud detections via a recursive Best Linear Unbiased Estimator (BLUE). The point cloud fusion accounts for noisy and inaccurate measurements, as well as minimizing growth of points in the 3-D shape. The use of a tracking algorithm and sensor pose enables 3-D shape estimation of dynamic objects from a moving car. Results are analyzed on experimental data, demonstrating the ability of our approach to produce more accurate and cleaner shape estimates.

Existing approaches to depth or disparity estimation output a distribution over a set of pre-defined discrete values. This leads to inaccurate results when the true depth or disparity does not match any of these values. The fact that this distribution is usually learned indirectly through a regression loss causes further problems in ambiguous regions around object boundaries. We address these issues using a new neural network architecture that is capable of outputting arbitrary depth values, and a new loss function that is derived from the Wasserstein distance between the true and the predicted distributions. We validate our approach on a variety of tasks, including stereo disparity and depth estimation, and the downstream 3D object detection. Our approach drastically reduces the error in ambiguous regions, especially around object boundaries that greatly affect the localization of objects in 3D, achieving the state-of-the-art in 3D object detection for autonomous driving.

This paper introduces and studies a graph-based variant of the path planning problem arising in hostile environments. We consider a setting where an agent (e.g. a robot) must reach a given destination while avoiding being intercepted by probabilistic entities which exist in the graph with a given probability and move according to a probabilistic motion pattern known a priori. Given a goal vertex and a deadline to reach it, the agent must compute the path to the goal that maximizes its chances of survival. We study the computational complexity of the problem, and present two algorithms for computing high quality solutions in the general case: an exact algorithm based on Mixed-Integer Nonlinear Programming, working well in instances of moderate size, and a pseudo-polynomial time heuristic algorithm allowing to solve large scale problems in reasonable time. We also consider the two limit cases where the agent can survive with probability 0 or 1, and provide specialized algorithms to detect these kinds of situations more efficiently.

3D object detection is an essential task in autonomous driving. Recent techniques excel with highly accurate detection rates, provided the 3D input data is obtained from precise but expensive LiDAR technology. Approaches based on cheaper monocular or stereo imagery data have, until now, resulted in drastically lower accuracies --- a gap that is commonly attributed to poor image-based depth estimation. However, in this paper we argue that it is not the quality of the data but its representation that accounts for the majority of the difference. Taking the inner workings of convolutional neural networks into consideration, we propose to convert image-based depth maps to pseudo-LiDAR representations --- essentially mimicking the LiDAR signal. With this representation we can apply different existing LiDAR-based detection algorithms. On the popular KITTI benchmark, our approach achieves impressive improvements over the existing state-of-the-art in image-based performance --- raising the detection accuracy of objects within the 30m range from the previous state-of-the-art of 22% to an unprecedented 74%. At the time of submission our algorithm holds the highest entry on the KITTI 3D object detection leaderboard for stereo-image-based approaches.

Many applications of stereo depth estimation in robotics require the generation of accurate disparity maps in real time under significant computational constraints. Current state-of-the-art algorithms force a choice between either generating accurate mappings at a slow pace, or quickly generating inaccurate ones, and additionally these methods typically require far too many parameters to be usable on power- or memory-constrained devices. Motivated by these shortcomings, we propose a novel approach for disparity prediction in the anytime setting. In contrast to prior work, our end-to-end learned approach can trade off computation and accuracy at inference time. Depth estimation is performed in stages, during which the model can be queried at any time to output its current best estimate. Our final model can process 1242×375 resolution images within a range of 10-35 FPS on an NVIDIA Jetson TX2 module with only marginal increases in error - using two orders of magnitude fewer parameters than the most competitive baseline. The source code is available at https://github.com/mileyan/AnyNet.

Person re-identification aims to identify a person from an image collection, given one image of that person as the query. There is, however, a plethora of real-life scenarios where we may not have a priori library of query images and therefore must rely on information from other modalities. In this paper, an attribute-based approach is proposed where the person of interest (POI) is described by a set of visual attributes, which are used to perform the search. We compare multiple algorithms and analyze how the quality of attributes impacts the performance. While prior work mostly relies on high precision attributes annotated by experts, we conduct a human-subject study and reveal that certain visual attributes could not be consistently described by human observers, making them less reliable in real applications. A key conclusion is that the performance achieved by non-expert attributes, instead of expert-annotated ones, is a more faithful indicator of the status quo of attribute-based approaches for person re-identification.

We present a system enabling a modular robot to autonomously build structures in order to accomplish high-level tasks. Building structures allows the robot to surmount large obstacles, expanding the set of tasks it can perform. This addresses a common weakness of modular robot systems, which often struggle to traverse large obstacles. This paper presents the hardware, perception, and planning tools that comprise our system. An environment characterization algorithm identifies features in the environment that can be augmented to create a path between two disconnected regions of the environment. Specially-designed building blocks enable the robot to create structures that can augment the environment to make obstacles traversable. A high-level planner reasons about the task, robot locomotion capabilities, and environment to decide if and where to augment the environment in order to perform the desired task. We validate our system in hardware experiments.

Urban environments offer a challenging scenario for autonomous driving. Globally localizing information, such as a GPS signal, can be unreliable due to signal shadowing and multipath errors. Detailed a priori maps of the environment with sufficient information for autonomous navigation typically require driving the area multiple times to collect large amounts of data, substantial post-processing on that data to obtain the map, and then maintaining updates on the map as the environment changes. This paper addresses the issue of autonomous driving in an urban environment by investigating algorithms and an architecture to enable fully functional autonomous driving with limited information. An algorithm to autonomously navigate urban roadways with little to no reliance on an a priori map or GPS is developed. Localization is performed with an extended Kalman filter with odometry, compass, and sparse landmark measurement updates. Navigation is accomplished by a compass-based navigation control law. Key results from Monte Carlo studies show success rates of urban navigation under different environmental conditions. Experiments validate the simulated results and demonstrate that, for given test conditions, an expected range can be found for a given success rate.

This paper presents a natural framework for information sharing in cooperative tasks involving humans and robots. In this framework, all information gathered over time by a human-robot team is exchanged and summarized in the form of a fused probability density function (pdf). An approach for an intelligent system to describe its belief pdfs in English expressions is presented. This belief expression generation is achieved through two goodness measures: semantic correctness and information preservation. In order to describe complex, multimodal belief pdfs, a Mixture of Statements (MoS) model is proposed such that optimal expressions can be generated through compositions of multiple statements. The model is further extended to a nonparametric Dirichlet process MoS generation, such that the optimal number of statements required for describing a given pdf is automatically determined. Results based on information loss, human collaborative task performances, and correctness rating scores suggest that the proposed method for generating belief expressions is an effective approach for communicating probabilistic information between robots and humans.

A novel approach to estimating the detailed shape of arbitrary extended objects jointly with their kinematics in the absence of a priori information is presented. The proposed shape model represents the tightest enclosing bound of the object projected into the ego motion plane as a polygon with an unknown number of vertices. Probabilistic inference techniques are employed to overcome various sources of uncertainty by rigorously estimating the joint distribution over the object shape and kinematic states, rather than estimating these variables directly. Simulation and experimental results are presented for objects with complex shapes tracked from an autonomous vehicle research platform. In addition to providing a richer set of information for higher level reasoning about extended objects (e.g. about object type, or occupied space), the results demonstrate that detailed shape estimates enable efficient use of sensor information by way of explicit surface-based sensor models; this efficient use of sensor information improves observability of latent object states, thereby improving tracking precision.

Efficient robotic exploration of an unknown, sensor limited, global-information-deficient environment poses a unique challenge to path planning algorithms because no deterministic guarantees on path completion and mission success can be made. Integrated Exploration (IE), which strives to combine localization and exploration, must be solved in order to create an autonomous robotic system capable of long term operation in new and challenging environments. This paper formulates a probabilistic framework which allows the creation of exploration algorithms providing probabilistic guarantees of success. A novel connection is made between the Hamiltonian Path Problem and exploration. The Guaranteed Probabilistic Information Explorer (G-PIE) is developed for the IE problem, providing a probabilistic guarantee on path completion, and asymptotic optimality of exploration.

An efficient surface normal estimation method is presented. The new algorithm estimates surface normal direction for each cell in a grid based on the occupancy information (both occupied and empty) of the neighboring cells. This grid representation allows user-defined sizes and scaling with the environment, not the number of measurements. Recursive and batch formulations to obtain the posterior estimate are presented, and compared. A computationally efficient implementation is derived which provides consistent and accurate estimates as measurements become available. Both simulation and experimental results are shown, demonstrating comparable estimation performance to that of using Point Cloud Library, but with significantly reduced computation time.

A probabilistic qualitative relational mapping (PQRM) algorithm is developed to enable robots to robustly map environments using noisy sensor measurements. Qualitative state representations provide soft, relative map information which is robust to metrical errors. In this paper, probabilistic distributions over qualitative states are derived and an algorithm to update the map recursively is developed. Maps are evaluated using Monte Carlo simulations for convergence and correctness. Validation tests are conducted on the New College dataset to evaluate map performance in realistic environments.

This paper presents a probabilistically rigorous method for jointly estimating the shape and kinematic states of arbitrary extended objects. A non-parametric shape model is defined as a set of points sampled from the object surface, and the joint probability density function over the surface samples is estimated recursively over time from lidar data. The presented work is demonstrated for a single maneuvering, non-convex object, highlighting key advantages over existing methods and motivating further development.

This paper presents a framework for information sharing and fusion in cooperative tasks involving humans and robots. In this context, all information regarding the state of interest is recursively fused and maintained by each agent in a form of belief. For a robot agent, its belief is commonly and practically represented as a probability density function (pdf), formed by traditional sensor fusion and state estimation algorithms. In cooperative tasks with non-expert humans, a robot needs to effectively communicate its belief so that the gathered information can be easily processed and interpreted by the humans. The goal of this research is to provide two-way information exchange and fusion between robots and humans, the former operating on pdfs, while the latter on English sentences. This is achieved by considering two goodness measures: semantic correctness and information preservation. Based on the goodness measures studied, results show that the proposed framework is able to generate optimal statements describing the given belief pdfs and successfully recover the initial inputs used to generate them. Additionally, in order to describe complex belief pdfs, a Mixture of Statements (MoS) model is proposed such that the optimal expression can be generated through a composition of more than one statements. With a nonparametric Dirichlet Process MoS generation, it is found that the robot can determine correctly the number of statements as well as the corresponding reference parameters needed to describe all hypotheses underlying its belief.

A terrain mapping model is proposed using a generalized Markov random field (MRF) representation. Unlike previous work, the proposed MRF can fully represent uncertainties due to sensor pose and measurement errors, as well as data association errors in a single model. Additionally, neither homoscedasticity nor a predefined shape of the likelihood distribution is assumed. The flexibility of an MRF model allows spatial height correlations to be incorporated. The ability to include spatial correlations not only improves the accuracy through the benefits of Bayesian prior modeling, but also serves as a basis for terrain property characterization. Maximum likelihood solutions of terrain roughness are derived. Benefits of the proposed model are demonstrated experimentally on indoor and outdoor datasets. Results show that the MRF model leads to lower height estimation errors. In addition, the capability of estimating non-Gaussian height distributions allows the information about individual terrain features to be preserved. Finally, the model is able to accurately estimate the roughness of the terrain, which is beneficial for edge detection of obstacles and nontraversible terrain regions.

A novel approach to utilize negative information to improve the precision and accuracy of extended multiobject tracking is presented. The parameterized probability density of object tracks undetected in sensor data is updated via inferences about the conditions necessary to result in occlusion of the undetected object. Negative information is also leveraged to inform track existence and data association, both of which contribute to a more sensible belief of the local dynamic scene. Simulation and experimental results are presented from autonomous driving scenarios, demonstrating that the use of negative information leads to a more complete, accurate, precise, and intuitive belief of the local scene, enabling high-level tasks that would otherwise be impractical.

This paper develops a probabilistic anticipation algorithm for dynamic objects observed by an autonomous robot in an urban environment. Predictive Gaussian mixture models are used due to their ability to probabilistically capture continuous and discrete obstacle decisions and behaviors; the predictive system uses the probabilistic output (state estimate and covariance) of a tracking system and map of the environment to compute the probability distribution over future obstacle states for a specified anticipation horizon. A Gaussian splitting method is proposed based on the sigma-point transform and the nonlinear dynamics function, which enables increased accuracy as the number of mixands grows. An approach to caching elements of this optimal splitting method is proposed, in order to enable real-time implementation. Simulation results and evaluations on data from the research community demonstrate that the proposed algorithm can accurately anticipate the probability distributions over future states of nonlinear systems.

A novel approach to utilizing negative information to improve the accuracy of extended multi-object tracking is presented. The parameterized probability density of object tracks unresolved in sensor data is updated via inferences about the sensor-to-object geometries necessary to result in occlusion of the unresolved object. Negative information is also leveraged to improve data association and to enable a novel death model, all of which contribute to a more accurate and precise belief of the local scene. Simulation and experimental results are presented from a common autonomous driving scenario.

This paper considers Bayesian data fusion of conventional robot sensor information with ambiguous human-generated categorical information about continuous world states of interest. First, it is shown that such soft information can be generally modeled via hybrid continuous-to-discrete likelihoods that are based on the softmax function. A new hybrid fusion procedure, called variational Bayesian importance sampling (VBIS), is then introduced to combine the strengths of variational Bayes approximations and fast Monte Carlo methods to produce reliable posterior estimates for Gaussian priors and softmax likelihoods. VBIS is then extended to more general fusion problems that involve complex Gaussian mixture (GM) priors and multimodal softmax likelihoods, leading to accurate GM approximations of highly non-Gaussian fusion posteriors for a wide range of robot sensor data and soft human data. Experiments for hardware-based multitarget search missions with a cooperative human-autonomous robot team show that humans can serve as highly informative sensors through proper data modeling and fusion, and that VBIS provides reliable and scalable Bayesian fusion estimates via GMs.

This paper presents a novel optimization-based path planner that is capable of planning multiple contingency paths to directly account for uncertainties in the future trajectories of dynamic obstacles. This planner addresses the particular problem of probabilistic collision avoidance for autonomous road vehicles that are required to safely interact, in close proximity, with other vehicles with unknown intentions. The presented path planner utilizes an efficient spline-based trajectory representation and fast but accurate collision probability bounds to simultaneously optimize multiple continuous contingency paths in real time. These collision probability bounds are efficient enough for real-time evaluation, yet accurate enough to allow for practical close-proximity driving behaviors such as passing an obstacle vehicle in an adjacent lane. An obstacle trajectory clustering algorithm is also presented to enable the path planner to scale to multiple-obstacle scenarios. Simulation results show that the contingency planner allows for a more aggressive driving style than planning a single path without compromising the overall safety of the robot.

This paper presents a methodology for determining whether human operators anticipate future control needs in order to compensate for time delays when controlling remote vehicles. The approach utilizes techniques drawn from the machine learning community in order to learn statistical models of human decision making. Models are fit to an experimental data set generated by remote operations of a robot subjected to time delays between 0 and 2.5 s, using the least angle regression (LARS) and sparse multinomial logistic regression (SMLR) algorithms. These algorithms make use of regularization to reduce the effects of overparameterization due to redundant or noisy environmental features. Models learned by LARS achieve an average prediction rate between 81% and 98%, depending on time delay, while those learned by SMLR achieve average rates between 68% and 86%. A novel metric of feature “importance” is used to evaluate the relative contributions of environmental features to model performance, motivated by the structure of the LARS algorithm. The degree to which human operators rely on anticipation is determined by examining how “importance” scores for features representing different prediction horizons vary with increasing time delay.

This paper uniquely embeds high-level robot controllers with sensor data obtained from abstracting probabilistic anticipation of the behavior of dynamic obstacles. An example problem of an autonomous vehicle operating in an urban environment, in the presence of other vehicles and pedestrians, is used as motivation. The correct-by-construction controller is automatically synthesized from a set of high-level tasks, specified as temporal logic formulas. The anticipated behavior of other vehicles is abstracted to a set of propositions describing the safety of road segments at intersections, and used as the output of high-level sensors for the controller. Such an input to the controller is inherently probabilistic, and this paper investigates the types of probabilistic guarantees that can be made about the system using both formal and statistical analysis.

An iterative smoothing algorithm is developed using Gaussian mixture models in order to tackle challenging nonlinear estimation problems. Gaussian mixture models naturally capture nonlinear and non-Gaussian systems, while smoothing algorithms provide ability to update using measurements obtained in the past. A tree structure and Gaussian distribution splitting method are proposed to mitigate nonlinearity effects and complexities. Two methods, Children Collapsing and Parent Splitting, are developed to utilize sigma-points smoother for Gaussian mixture model. An indoor localization problem is used to explore and validate the approach. Performance of these new methods is compared to a baseline sigma-points smoother, in both simulation and experiment, and shows much improvement in overall error compared to the truth.

In this paper, a novel approach for commanding mobile robots using a probabilistic multistroke sketch interface is presented. Drawing from prior work in handwriting recognition, sketches are modeled as a variable duration hidden Markov model, where the distributions on the states and transitions are learned from training data. A forward search algorithm is used to find the most likely sketch given the observations on the strokes, interstrokes, and gestures. A heuristic is implemented to discourage breadth-first search behavior, and is shown to greatly reduce computation time while sacrificing little accuracy. To avoid recognition errors, the recognized sketch is displayed to the user for confirmation; a rejection prompts the algorithm to search for and display the next most likely sketch. Upon confirmation of the recognized sketch, the robot executes the appropriate behaviors. A set of experiments was conducted in which operators controlled a single mobile robot in an indoor search-and-identify mission. Operators performed two missions using the proposed sketch interface and two missions using a more conventional point-and-click interface. On average, missions conducted using sketch control were performed as well as those using the point-and-click interface, and results from user surveys indicate that more operators preferred using sketch control.

A novel method for controlling a mobile robot using qualitative inputs in the context of an approximate map, such as one sketched by a human, is presented. By defining a desired trajectory with respect to observable landmarks, human operators can send semi-autonomous robots into areas for which a truth map is not available. Waypoint planning is formulated as a quadratic optimization problem, resulting in robot trajectories in the true environment that are qualitatively similar to those provided by the human. The algorithm is implemented both in simulation and on a mobile robot platform in several different environments. A sensitivity analysis is per formed, illustrating how the method is robust to uncertainties, even large sketch distortions, and allows the robot to adapt and re-plan according to its most current perception of the world.

A hierarchical trajectory clustering algorithm is presented with the goal of clustering a set of mutually exclusive obstacle trajectory predictions for use in a contingency based path planner for an autonomous road vehicle. This clustering algorithm improves the computational scaling of the contingency planner by limiting the total number of required contingency paths while preserving the performance advantages of exhaustive contingency planning. This algorithm seeks to maximize dissimilarity between trajectory clusters with regard to their potential effect on a robot's future path. Simulation results show that the clustering algorithm allows a robot to maintain many of the benefits of contingency planning while requiring fewer contingency paths.

A novel-tracking algorithm is presented as a computationally feasible, real-time solution to the joint estimation problem of data assignment and dynamic obstacle tracking from a potentially moving robotic platform. The algorithm implements a Rao-Blackwellized particle filter (RBPF) to factorize the joint estimation problem into 1) a data assignment problem solved via particle filter and 2) a multiple dynamic obstacle-tracking problem solved with efficient parametric filters. The parametric filters make use of a new target representation and stable features developed specifically for tracking full-size vehicles in a dense traffic environment. The algorithm is validated in real time, both in controlled experiments with full-size robotic vehicles and on data collected at the 2007 Defense Advanced Research Projects Agency (DARPA) Urban Challenge.

A new method is presented for fusing conventional continuous sensor observations with discrete multi-categorical state-dependent information, which can be furnished by humans in many cooperative human-robot interaction problems. The hybrid likelihood function for mapping between continuous hidden states and categorical observations are specified via softmax models. Although softmax models avoid discretization of continuous states, they are challenging to implement for real-time data fusion since they are not analytically integrable. An approximation based on variational Bayesian (VB) methods is presented here to obtain fast closed-form Gaussian solutions to the desired posteriors in cases where the hidden continuous states have Gaussian pdfs. A joint human-robot target localization example illustrates the properties and utility of the VB hybrid fusion strategy, which also applies more generally to inference in hybrid Bayesian networks and mixture models.

This paper presents a novel optimization based path planner that can simultaneously plan multiple contingency paths to account for the uncertain actions of dynamic obstacles. This planner addresses the particular problem of collision avoidance for autonomous road vehicles which are required to safely interact with other vehicles with unknown intentions. The presented path planner utilizes an efficient spline based trajectory representation and fast but accurate collision probability approximations to enable the simultaneous optimization of multiple contingency paths.

In this paper, an algorithm to segment 3D points in dense range maps generated from the fusion of a single optical camera and a multiple emitter/detector laser range finder is presented. The camera image and laser range data are fused using a Markov Random Field to estimate a 3D point corresponding to each image pixel. The textured 3D dense point cloud is segmented based on evidence of a boundary between regions of the textured point cloud. Clusters are discriminated based on Euclidean distance, pixel intensity and estimated surface normal using a fast, deterministic and near linear time segmentation algorithm. The algorithm is demonstrated on data collected with the Cornell University DARPA Urban Challenge vehicle. Performance of the proposed dense segmentation routine is evaluated in a complex urban environment and compared to segmentation of the sparse point cloud. Results demonstrate the effectiveness of the dense segmentation algorithm to avoid over-segmentation better than incorporating color and surface normal data in the sparse point cloud.

A fully probabilistic command interface for controlling robots using multi-stroke sketch commands is presented. Drawing from prior work in handwriting recognition, sketches are modeled as a variable duration hidden Markov model, where the distributions on the states and transitions are learned from training data. A forward search algorithm on the gesture, stroke, and stroke transition observations is used to find the most likely sketch, which is displayed to the user for confirmation. In cases where the most likely sketch is incorrect, the user can reject it, prompting the next most likely sketch to be displayed. Upon confirmation from the user, the robot executes the desired behaviors. A prototype sketch interface was implemented using a pen tablet; two sets of search-and-identify experiments were conducted using a single robot in an indoor environment to test the usability of the proposed framework. Even novice users were able to successfully complete the missions, including those on whom the algorithm was not trained. User surveys indicate that operators generally found the interface to be natural and easy to use.

Recent research has shown that robots can model their world with Multi-Level (ML) surface maps, which utilize dasiapatchespsila in a 2D grid space to represent various environment elevations within a given grid cell. Though these maps are able to produce 3D models of the environment while exploiting the computational feasibility of single elevation maps, they do not take into account in-plane uncertainty when matching measurements to grid cells or when grouping those measurements into dasiapatches.psila To respond to these drawbacks, this paper proposes to extend these ML surface maps into Probabilistic Multi-Level (PML) surface maps, which uses formal probability theory to incorporate estimation and modeling errors due to uncertainty. Measurements are probabilistically associated to cells near the nominal location, and are categorized through hypothesis testing into dasiapatchespsila via classification methods that incorporate uncertainty. Experimental results comparing the performances of the PML and ML surface mapping algorithms on representative objects found in both indoor and outdoor environments show that the PML algorithm outperforms the ML algorithm in most cases including in the presence of noisy and sparse measurements. The experimental results support the claim that the PML algorithm produces more densely populated, conservative representations of its environment with fewer measurements than the ML algorithm.

A Gaussian Sum Filter (GSF) with component extended Kalman filters (EKF) is proposed as an approach to localize an autonomous vehicle in an urban environment with limited GPS availability. The GSF uses vehicle relative vision-based measurements of known map features coupled with inertial navigation solutions to accomplish localization in the absence of GPS. The vision-based measurements are shown to have multi-modal measurement likelihood functions that are well represented as a weighted sum of Gaussian densities and the GSF is ideally suited to accomplish recursive Bayesian state estimation for this problem. A sequential merging technique is used for Gaussian mixture condensation in the posterior density approximation after fusing multi-modal measurements in the GSF to maintain mixture size over time. The representation of the posterior density with the GSF is compared over a common dataset against a benchmark particle filter solution. The Expectation-Maximization (EM) algorithm is used offline to determine the representational efficiency of the particle filter in terms of an effective number of Gaussian densities. The GSF with vision-based vehicle relative measurements is shown to remain converged using 37 minutes of recorded data from the Cornell University DARPA Urban Challenge (DUC) autonomous vehicle in an urban environment that includes a 32 minute GPS blackout.

This study presents the PosteriorPose algorithm, a Bayesian particle filtering approach for augmenting GPS and inertial navigation solutions with vision-based measurements of nearby lanes and stoplines referenced against a known map of environmental features. These relative measurements are shown to improve the quality of the navigation solution when GPS is available, and they are shown to keep the navigation solution converged in extended GPS blackouts. Measurements are incorporated with careful hypothesis testing and error modeling to account for non-Gaussian errors committed by vision-based detection algorithms. The PosteriorPose algorithm is implemented and validated in real-time on Cornell University's 2007 DARPA Urban Challenge entry; experimental data is presented showing the algorithm outperforming a tightly- coupled GPS/inertial navigation solution both in full GPS coverage and in an extended GPS blackout.

In this paper, scalable collaborative human-robot systems for information gathering applications are approached as a decentralized Bayesian sensor network problem. Human-computer augmented nodes and autonomous mobile sensor platforms are collaborating on a peer-to-peer basis by sharing information via wireless communication network. For each node, a computer (onboard the platform or carried by the human) implements both a decentralized Bayesian data fusion algorithm and a decentralized Bayesian control negotiation algorithm. The individual node controllers iteratively negotiate anonymously with each other in the information space to find cooperative search plans based on both observed and predicted information that explicitly consider the platforms (humans and robots) motion models, their sensors detection functions, as well as the target arbitrary motion model. The results of a collaborative multi-target search experiment conducted with a team of four autonomous mobile sensor platforms and five humans carrying small portable computers with wireless communication are presented to demonstrate the efficiency of the approach.

Coupled operator-multiple vehicle systems are modelled in a unified framework using probabilistic graphs to yield a methodology for analyzing semi-autonomous systems. The framework uses conditional probabilistic dependencies between all elements, leading to a Bayesian network (BN) with probabilistic evaluation capability. Vehicle attitude/navigation states and target/classification states can be evaluated using nonlinear estimators such as the EKF, multiple model filter, information filter, or other approaches. Discrete operator decisions are being modeled as Bayesian network blocks, with conditional dependencies on the vehicle and tracking estimators. Initial decision models use combinations of softmax and discrete probability distributions.

A general method for mapping dynamic environments using a Rao-Blackwellized particle filter is presented. The algorithm rigorously addresses both data association and target tracking in a single unified estimator. The algorithm relies on a Bayesian factorization to separate the posterior into: 1) a data association problem solved via particle filter; and 2) a tracking problem with known data associations solved by Kalman filters developed specifically for the ground robot environment. The algorithm is demonstrated in simulation and validated in the real world with laser range data, showing its practical applicability in simultaneously resolving data association ambiguities and tracking moving objects.

A state-dependent model reduction approach to evaluating human-robotic system performance for the purpose of implementing Bayesian Networks is introduced. A procedure for reducing state dependencies on a specific class of operator failure events for probabilistic graph models is developed and verified with user data. In contrast to evaluating performance with respect to objective "workload" measurements, discrete failure "tag" events are classified and modeled as Bayesian network blocks with conditional dependencies on a subset of the total system states. Initial extraction of performance results are shown using data from the RoboFlag experiments.

In this paper, we consider the problem of building a probability map for an unknown hostile environment by utilizing a team of UAVs. Specifically, we first present a centralized map building scheme for the Boeing open experimental platform (OEP) environment, the strategy is then modified into a decentralized map building algorithm to increase the robustness of the system. Some simulation results are provided to demonstrate the validity of the proposed algorithms

This paper presents a decision making simulation and model for a simplified intelligence, surveillance and reconnaissance (ISR) type mission. The simulation presented operators with a binary decision choice for 25 possible scenarios. Data was collected experimentally with a total of 600 sample points, and then analyzed in order to develop an analytically tractable model of operator choice. The distribution of operator decisions was modeled with a binomial distribution as a function of environmental variables. An optimal decision making policy was also prescribed for all scenarios and compared to the operator data. Results showed good agreement between data and the optimal decision making policy in most scenarios, indicating good operator decision making. For scenarios with disagreement, the model gives more detail as to the environmental variables that caused the most confusion among operators.