For robotic interaction in environments shared with other agents, access to volumetric and semantic maps of the scene is crucial. However, such environments are inevitably subject to long-term changes, which the map needs to account for. We thus propose panoptic multi-TSDFs as a novel representation for multi-resolution volumetric mapping in changing environments. By leveraging high-level information for 3D reconstruction, our proposed system allocates high resolution only where needed. Through reasoning on the object level, semantic consistency over time is achieved. This enables our method to maintain up-to-date reconstructions with high accuracy while improving coverage by incorporating previous data. We show in thorough experimental evaluation that our map can be efficiently constructed, maintained, and queried during online operation, and that the presented approach can operate robustly on real depth sensors using non-optimized panoptic segmentation as input.

Unmanned aerial vehicles (UAVs) are seeing increasing adoption to automated remote and in situ inspection of industrial assets, removing the need for hazardous manned access. Aerial manipulator architectures supporting pose-decoupled exertion of force and torque would further enable UAV deployment of contact-based transducers for sub-surface structural health assessment. Herein, for the first time, we introduce an over-actuated multirotor deploying a dry-coupled ultrasonic wheel probe as a novel means of wall thickness mapping. Using bi-axial tilting propellers in a unique tricopter layout, this system performs direct thrust vectoring for efficient omnidirectional flight and application of interaction forces. In laboratory testing, we demonstrate stable and repeatable probe deployment in a variety of representative asset inspection operations. We obtain a mean absolute error (MAE) in measured thickness of under 0.10 mm when measuring an aluminum sample with varying wall thickness. This is maintained over repeated exit and reentry of surface contact and when the sample is mounted vertically or on the underside of a 45° overhang. Furthermore, when rolling the probe dynamically across the sample surface in an area scanning modality, an MAE in wall thickness below 0.28 mm is recorded. Multi-modal operational confidence bounds of the system are thereby quantitatively defined. Note to Practitioners—Motivation for this article stems from the desire to enhance the speed and level of insight into structural health currently offered through remote aerial inspection processes. We approach this by integration of a thrust vectoring multirotor platform with a dry-coupling wheel probe for aerial ultrasonic thickness measurement. This system reliably presses the probe into the target surface and obtains point measurements across various surface orientations without a stabilizing frame. This broadens applicability and permits novel inspections where couplant gel would otherwise contaminate the surface and require manual cleaning. We profile thickness along a scanned linear section, a mode suited to corrosion mapping of large surface areas such as petrochemical storage tanks, pipework, or similar assets. We also make detailed consideration toward measurement accuracy, repeatability, and localization, an aspect commonly overlooked in literature. Future work to characterize variable friction effects currently limiting rolling scan speed and measurement coverage density may be beneficial. Quantitative study of the effectiveness of dry-couplant in the presence of any uncommon surface contaminants specific to a desired use-case is also advised.

In this paper, we propose a robust end-to-end multi-modal pipeline for place recognition where the sensor systems can differ from the map building to the query. Our approach operates directly on images and LiDAR scans without requiring any local feature extraction modules. By projecting the sensor data onto the unit sphere, we learn a multi-modal descriptor of partially overlapping scenes using a spherical convolutional neural network. The employed spherical projection model enables the support of arbitrary LiDAR and camera systems readily without losing information. Loop closure candidates are found using a nearest-neighbor lookup in the embedding space. We tackle the problem of correctly identifying the closest place by correlating the candidates’ power spectra, obtaining a confidence value per prospect. Our estimate for the correct place corresponds then to the candidate with the highest confidence. We evaluate our proposal w.r.t. state-of-the-art approaches in place recognition using real-world data acquired using different sensors. Our approach can achieve a recall that is up to 10% and 5% higher than for a LiDAR- and vision-based system, respectively, when the sensor setup differs between model training and deployment. Additionally, our place selection can correctly identify up to 95% matches from the candidate set.

Robot navigation is a task where reinforcement learning approaches are still unable to compete with traditional path planning. State-of-the-art methods differ in small ways, and do not all provide reproducible, openly available implementations. This makes comparing methods a challenge. Recent research has shown that unsupervised learning methods can scale impressively, and be leveraged to solve difficult problems. In this work, we design ways in which unsupervised learning can be used to assist reinforcement learning for robot navigation. We train two end-to-end, and 18 unsupervised-learning-based architectures, and compare them, along with existing approaches, in unseen test cases. We demonstrate our approach working on a real life robot. Our results show that unsupervised learning methods are competitive with end-to-end methods. We also highlight the importance of various components such as input representation, predictive unsupervised learning, and latent features. We make all our models publicly available, as well as training and testing environments, and tools1. This release also includes OpenAI-gym-compatible environments designed to emulate the training conditions described by other papers, with as much fidelity as possible. Our hope is that this helps in bringing together the field of RL for robot navigation, and allows meaningful comparisons across state-of-the-art methods.

The ability to simultaneously track and reconstruct multiple objects moving in the scene is of the utmost importance for robotic tasks such as autonomous navigation and interaction. Virtually all of the previous attempts to map multiple dynamic objects have evolved to store individual objects in separate reconstruction volumes and track the relative pose between them. While simple and intuitive, such formulation does not scale well with respect to the number of objects in the scene and introduces the need for an explicit occlusion handling strategy. In contrast, we propose a map representation that allows maintaining a single volume for the entire scene and all the objects therein. To this end, we introduce a novel multi-object TSDF formulation that can encode multiple object surfaces at any given location in the map. In a multiple dynamic object tracking and reconstruction scenario, our representation allows maintaining accurate reconstruction of surfaces even while they become temporarily occluded by other objects moving in their proximity. We evaluate the proposed TSDF++ formulation on a public synthetic dataset and demonstrate its ability to preserve reconstructions of occluded surfaces when compared to the standard TSDF map representation. Code is available at https://github.com/ethz-asl/tsdf-plusplus.

The evaluation of robot capabilities to navigate human crowds is essential to conceive new robots intended to operate in public spaces. This paper initiates the development of a benchmark tool to evaluate such capabilities; our long term vision is to provide the community with a simulation tool that generates virtual crowded environment to test robots, to establish standard scenarios and metrics to evaluate navigation techniques in terms of safety and efficiency, and thus, to install new methods to benchmarking robots’ crowd navigation capabilities. This paper presents the architecture of the simulation tools, introduces first scenarios and evaluation metrics, as well as early results to demonstrate that our solution is relevant to be used as a benchmark tool.

We present a novel hierarchical POMDP framework to solve an object search and delivery task where the agent is given a prior belief about the possible item locations. Solving POMDPs is computationally demanding and, as such, applications have typically been limited to small environments. The proposed hierarchical POMDP framework performs reasoning on multiple spatial scales in order to reduce computation time. The problem is first solved in the top layer of the hierarchy with a coarsely discretized state space. Its solution is refined in the lower layers with increasing resolution. Three different methods for propagating information down the spatial hierarchy are discussed and validated in simulation. We show that a two-layer multi-scale POMDP decreases computation time by an order of magnitude allowing for real-time applications while maintaining high solution quality. For large problems that require three layers to reach the desired resolution, computation time speedups by two orders of magnitude are achieved.

Model-based controllers on real robots require accurate knowledge of the system dynamics to perform optimally. For complex dynamics, first-principles modeling is not sufficiently precise, and data-driven approaches can be leveraged to learn a statistical model from real experiments. However, the efficient and effective data collection for such a data-driven system on real robots is still an open challenge. This paper introduces an optimization problem formulation to find an informative trajectory that allows for efficient data collection and model learning. We present a sampling-based method that computes an approximation of the trajectory that minimizes the prediction uncertainty of the dynamics model. This trajectory is then executed, collecting the data to update the learned model. We experimentally demonstrate the capabilities of our proposed framework when applied to a complex omnidirectional flying vehicle with tiltable rotors. Using our informative trajectories results in models which outperform models obtained from non-informative trajectory by 13.3% with the same amount of training data. Furthermore, we show that the model learned from informative trajectories generalizes better than the one learned from non-informative trajectories, achieving better tracking performance on different tasks.

Localization is an essential task for mobile autonomous robotic systems that want to use pre-existing maps or create new ones in the context of SLAM. Today, many robotic platforms are equipped with high-accuracy 3D LiDAR sensors, which allow a geometric mapping, and cameras able to provide semantic cues of the environment. Segment-based mapping and localization have been applied with great success to 3D point-cloud data, while semantic understanding has been shown to improve localization performance in vision based systems. In this paper we combine both modalities in SemSegMap, extending SegMap into a segment based mapping framework able to also leverage color and semantic data from the environment to improve localization accuracy and robustness. In particular, we present new segmentation and descriptor extraction processes. The segmentation process benefits from additional distance information from color and semantic class consistency resulting in more repeatable segments and more overlap after re-visiting a place. For the descriptor, a tight fusion approach in a deep-learned descriptor extraction network is performed leading to a higher descriptiveness for landmark matching. We demonstrate the advantages of this fusion on multiple simulated and real-world datasets and compare its performance to various baselines. We show that we are able to find 50.9 % more high-accuracy prior-less global localizations compared to SegMap on challenging datasets using very compact maps while also providing accurate full 6 DoF pose estimates in real-time.

Abstract Deep learning has enabled impressive progress in the accuracy of semantic segmentation. Yet, the ability to estimate uncertainty and detect failure is key for safety-critical applications like autonomous driving. Existing uncertainty estimates have mostly been evaluated on simple tasks, and it is unclear whether these methods generalize to more complex scenarios. We present Fishyscapes, the first public benchmark for anomaly detection in a real-world task of semantic segmentation for urban driving. It evaluates pixel-wise uncertainty estimates towards the detection of anomalous objects. We adapt state-of-the-art methods to recent semantic segmentation models and compare uncertainty estimation approaches based on softmax confidence, Bayesian learning, density estimation, image resynthesis, as well as supervised anomaly detection methods. Our results show that anomaly detection is far from solved even for ordinary situations, while our benchmark allows measuring advancements beyond the state-of-the-art. Results, data and submission information can be found at https://fishyscapes.com/ .

This article presents and validates active interaction force control and planning for fully actuated and omnidirectional aerial manipulation platforms, with the goal of aerial contact inspection in unstructured environments. We present a variable axis-selective impedance control which integrates direct force control for intentional interaction, using feedback from an on-board force sensor. The control approach aims to reject disturbances in free flight, while handling unintentional interaction and actively controlling desired interaction forces. A fully actuated and omnidirectional tilt-rotor aerial system is used to show capabilities of the control and planning methods. Experiments demonstrate disturbance rejection, push-and-slide interaction, and force-controlled interaction in different flight orientations. The system is validated as a tool for nondestructive testing of concrete infrastructure, and statistical results of interaction control performance are presented and discussed.

This work presents a method to control omnidirectional micro aerial vehicles (OMAVs) for the tracking of 6-DoF trajectories in free space. A rigid body model based approach is applied in a receding horizon fashion to generate optimal wrench commands that can be constrained to meet limits given by the mechanical design and actuators of the platform. Allocation of optimal actuator commands is performed in a separate step. A disturbance observer estimates forces and torques that may arise from unmodeled dynamics or external disturbances and fuses them into the optimization to achieve offset-free tracking. Experiments on a fully overactuated MAV show the tracking performance and compare it against a classical PD-based controller.

Robots require a detailed understanding of the 3D structure of the environment for autonomous navigation and path planning. A popular approach is to represent the environment using metric, dense 3D maps such as 3D occupancy grids. However, in large environments the computational power required for most state-of-the-art 3D dense mapping systems is compromising precision and real-time capability. In this work, we propose a novel mapping method that is able to build and maintain 3D dense representations for large indoor environments using standard CPUs. Topological global representations and 3D dense submaps are maintained as hybrid global map. Submaps are generated for every new visited place. A place (room) is identified as an isolated part of the environment connected to other parts through transit areas (doors). This semantic partitioning of the environment allows for a more efficient mapping and path-planning. We also propose a method for autonomous exploration that directly builds the hybrid representation in real time.We validate the real-time performance of our hybrid system on simulated and real environments regarding mapping and path-planning. The improvement in execution time and memory requirements upholds the contribution of the proposed work.

Object finding in clutter is a skill that requires perception of the environment and in many cases physical interaction. In robotics, interactive perception defines a set of algorithms that leverage actions to improve the perception of the environment, and vice versa use perception to guide the next action. Scene interactions are difficult to model, therefore, most of the current systems use predefined heuristics. This limits their ability to efficiently search for the target object in a complex environment. In order to remove heuristics and the need for explicit models of the interactions, in this work we propose a reinforcement learning based active and interactive perception system for scene exploration and object search. We evaluate our work both in simulated and in real-world experiments using a robotic manipulator equipped with an RGB and a depth camera, and compare our system to two baselines. The results indicate that our approach, trained in simulation only, transfers smoothly to reality and can solve the object finding task efficiently and with more than 88% success rate.

Information gathering algorithms play a key role in unlocking the potential of robots for efficient data collection in a wide range of applications. However, most existing strategies neglect the fundamental problem of the robot pose uncertainty, which is an implicit requirement for creating robust, high-quality maps. To address this issue, we introduce an informative planning framework for active mapping that explicitly accounts for the pose uncertainty in both the mapping and planning tasks. Our strategy exploits a Gaussian Process (GP) model to capture a target environmental field given the uncertainty on its inputs. For planning, we formulate a new utility function that couples the localization and field mapping objectives in GP-based mapping scenarios in a principled way, without relying on manually-tuned parameters. Extensive simulations show that our approach outperforms existing strategies, reducing mean pose uncertainty and map error. We present a proof of concept in an indoor temperature mapping scenario.

Painting textures on 3D surfaces requires an understanding of the surface geometry, paint flow and paint mixing. This work formulates automated painting as a planning problem and proposes a solution based on a self-supervised learning framework that enables a robot to paint monochromatic non-uniform textures on 3D surfaces. We developed a method that iteratively decides the actions to take based on constant feedback of the painting process. Inspired by recent results, we formulate our solution using a recurrent neural network (RNN) to decide where and what to paint on the surface at each time instant. Specifically, the paint delivery tool's flow rate, orientation and position relative to the surface at each time instant are evaluated. This data can then be processed by a robot's planner of choice for generating a painting mission that can achieve the desired end result. We evaluate the proposed approach by providing qualitative and quantitative results of the different components. Furthermore, we validate the effectiveness of the approach for the application by providing renderings from a paint simulation environment and show how a robot executes the planned painting mission on a generic 3D surface.

State-of-the-art approaches for robot navigation among humans are typically restricted to planar movement actions. This work addresses the question of whether it can be beneficial to use interaction actions, such as saying, touching, and gesturing, for the sake of allowing robots to navigate in unstructured, crowded environments. To do so, we first identify challenging scenarios to traditional motion planning methods. Based on the hypothesis that the variation in modality for these scenarios calls for significantly different planning policies, we design specific navigation behaviors as interaction planners for actuated, mobile robots. We further propose a high level planning algorithm for multi-behavior navigation, named Interaction Actions for Navigation (IAN). Through both real-world and simulated experiments, we validate the selected behaviors and the high-level planning algorithm, and discuss the impact of our obtained results on our stated assumptions.

In this paper, we introduce IDOL, an optimization-based framework for IMU-DVS Odometry using Lines. Event cameras, also called Dynamic Vision Sensors (DVSs), generate highly asynchronous streams of events triggered upon illumination changes for each individual pixel. This novel paradigm presents advantages in low illumination conditions and high-speed motions. Nonetheless, this unconventional sensing modality brings new challenges to perform scene reconstruction or motion estimation. The proposed method offers to leverage a continuous-time representation of the inertial readings to associate each event with timely accurate inertial data. The method's front-end extracts event clusters that belong to line segments in the environment whereas the back-end estimates the system's trajectory alongside the lines' 3D position by minimizing point-to-line distances between individual events and the lines' projection in the image space. A novel attraction/repulsion mechanism is presented to accurately estimate the lines' extremities, avoiding their explicit detection in the event data. The proposed method is benchmarked against a state-of-the-art frame-based visual-inertial odometry framework using public datasets. The results show that IDOL performs at the same order of magnitude on most datasets and even shows better orientation estimates. These findings can have a great impact on new algorithms for DVS.

Visually poor scenarios are one of the main sources of failure in visual localization systems in outdoor environments. To address this challenge, we present MOZARD, a multi-modal localization system for urban outdoor environments using vision and LiDAR. By fusing key point based visual multi-session information with semantic data, an improved localization recall can be achieved across vastly different appearance conditions. In particular we focus on the use of curbstone information because of their broad distribution and reliability within urban environments. We present thorough experimental evaluations on several driving kilometers in challenging urban outdoor environments, analyze the recall and accuracy of our localization system and demonstrate in a case study possible failure cases of each subsystem. We demonstrate that MOZARD is able to bridge scenarios where our previous key point based visual approach, VIZARD, fails, hence yielding an increased recall performance, while a similar localization accuracy of 0.2m is achieved.

LiDAR sensors are crucial in automotive perception for accurate object detection. However, LiDAR data is hard to interpret for humans and consequently time-consuming to label. Whereas camera data is easy interpretable and thus, comparably simpler to label. Within this work we present a transductive transfer learning approach to transfer the knowledge for the object detection task from images to point cloud data. We propose a multi-modal adversarial Auto Encoder architecture which disentangles uni-modal features into two groups: common (transferable) features, and complementary (modality-specific) features. This disentanglement is based on the hypothesis that a set of common features exist. An important point of our framework is that the disentanglement is learned in an unsupervised manner. Furthermore, the results show that only a small amount of multi-modal data is needed to learn the disentanglement, and thus to transfer the knowledge between modalities. As a result we our experiments show that training with 75% less data of the KITTI objects, the classification accuracy achieved is of 71.75%, only 3.12% less than when using the full data set. The implications of these findings can have great impact in perception pipelines based on LIDAR data.

This paper presents a pipeline for semantic segmentation of trees into their components. Given a single RGB-D image of a tree, we employ a deep network to predict labels to classify each pixel of the tree into trunk, branches, twigs and leaves. Multiple convolutional neural network architectures to combine the complementary modalities of depth and colour data are investigated. An asynchronous training approach where two networks trained separately on RGB and depth encoded as a 3-channel HHA image are combined using a late fusion architecture with different learning rates performs the best. Training and evaluation are performed on a synthetic dataset of 6 species of broadleaf trees. We further demonstrate the network's generalization capabilities, across various tree species on the synthetic dataset, achieving an accuracy of upto 92.5%. Furthermore, we present a qualitative evaluation of our approach on real-world data.

For autonomous driving applications it is critical to know which type of road users and road side infrastructure are present to plan driving manoeuvres accordingly. Therefore autonomous cars are equipped with different sensor modalities to robustly perceive its environment. However, for classification modules based on machine learning techniques it is challenging to overcome unseen sensor noise. This work presents an object classification module operating on unsupervised learned multi-modal features with the ability to overcome gradual or total sensor failure. A two stage approach composed of an unsupervised feature training and a uni-modal and multimodal classifiers training is presented. We propose a simple but effective decision module switching between uni-modal and multi-modal classifiers based on the closeness in the feature space to the training data. Evaluations on the ModelNet 40 data set show that the proposed approach has a 14% accuracy gain compared to a late fusion approach operating on a noisy point cloud data and a 6% accuracy gain when operating on noisy image data.

In this paper, we propose a visual-inertial framework able to efficiently estimate the camera poses of a non-rigid trinocular baseline for long-range depth estimation on-board a fast moving aerial platform. The estimation of the time-varying baseline is based on relative inertial measurements, a photometric relative pose optimizer, and a probabilistic wing model fused in an efficient Extended Kalman Filter (EKF) formulation. The estimated depth measurements can be integrated into a geo-referenced global map to render a reconstruction of the environment useful for local replanning algorithms. Based on extensive real-world experiments we describe the challenges and solutions for obtaining the probabilistic wing model, reliable relative inertial measurements, and vision-based relative pose updates and demonstrate the computational efficiency and robustness of the overall system under challenging conditions.

In many applications, maintaining a consistent map of the environment is key to enabling robotic platforms to perform higher-level decision making. Detection of already visited locations is one of the primary ways in which map consistency is maintained, especially in situations where external positioning systems are unavailable or unreliable. Mapping in 2D is an important field in robotics, largely due to the fact that man-made environments such as warehouses and homes, where robots are expected to play an increasing role, can often be approximated as planar. Place recognition in this context remains challenging: 2D lidar scans contain scant information with which to characterize, and therefore recognize, a location. This paper introduces a novel approach aimed at addressing this problem. At its core, the system relies on the use of the distance function for representation of geometry. This representation allows extraction of features which describe the geometry of both surfaces and free-space in the environment. We propose a feature for this purpose. Through evaluations on public datasets, we demonstrate the utility of free-space in the description of places, and show an increase in localization performance over a state-of-the-art descriptor extracted from surface geometry.

Changes in appearance is one of the main sources of failure in visual localization systems in outdoor environments. To address this challenge, we present VIZARD, a visual localization system for urban outdoor environments. By combining a local localization algorithm with the use of multi-session maps, a high localization recall can be achieved across vastly different appearance conditions. The fusion of the visual localization constraints with wheel-odometry in a state estimation framework further guarantees smooth and accurate pose estimates. In an extensive experimental evaluation on several hundreds of driving kilometers in challenging urban outdoor environments, we analyze the recall and accuracy of our localization system, investigate its key parameters and boundary conditions, and compare different types of feature descriptors. Our results show that VIZARD is able to achieve nearly 100% recall with a localization accuracy below 0.5m under varying outdoor appearance conditions, including at night-time.

Robust prediction of pedestrian behavior is one of the most challenging problems for autonomous driving. Particularly, predicting pedestrian crossings at crosswalks is of considerable importance for avoiding accidents on the one hand and not unnecessarily slowing down traffic on the other hand. Traditional model-based motion tracking and prediction approaches have difficulties in capturing abrupt changes in motions, as humans can perform them. In this paper, an approach for predicting pedestrian motions that combines established motion tracking algorithms with data-driven methods is presented. The approach is built upon a hierarchical structure, where first, the intent of each pedestrian is classified. Then, the approach computes several qualitative metrics, such as time-to-cross, for the pedestrians classified as crossing. The approach is evaluated on a challenging urban data set collected for different types of crosswalks such as roundabouts and straight roads. The evaluation also provides a thorough analysis of the generalization performance of the proposed approach.

This paper reports on a data-driven, interaction-aware motion prediction approach for pedestrians in environments cluttered with static obstacles. When navigating in such workspaces shared with humans, robots need accurate motion predictions of the surrounding pedestrians. Human navigation behavior is mostly influenced by their surrounding pedestrians and by the static obstacles in their vicinity. In this paper we introduce a new model based on Long-Short Term Memory (LSTM) neural networks, which is able to learn human motion behavior from demonstrated data. To the best of our knowledge, this is the first approach using LSTMs, that incorporates both static obstacles and surrounding pedestrians for trajectory forecasting. As part of the model, we introduce a new way of encoding surrounding pedestrians based on a 1d-grid in polar angle space. We evaluate the benefit of interaction-aware motion prediction and the added value of incorporating static obstacles on both simulation and real-world datasets by comparing with state-of-the-art approaches. The results show, that our new approach outperforms the other approaches while being very computationally efficient and that taking into account static obstacles for motion predictions significantly improves the prediction accuracy, especially in cluttered environments.

The growing popularity of autonomous systems creates a need for reliable and efficient metric pose retrieval algorithms. Currently used approaches tend to rely on nearest neighbor search of binary descriptors to perform the 2D-3D matching and guarantee realtime capabilities on mobile platforms. These methods struggle, however, with the growing size of the map, changes in viewpoint or appearance, and visual aliasing present in the environment. The rigidly defined descriptor patterns only capture a limited neighborhood of the keypoint and completely ignore the overall visual context. We propose LandmarkBoost - an approach that, in contrast to the conventional 2D-3D matching methods, casts the search problem as a landmark classification task. We use a boosted classifier to classify landmark observations and directly obtain correspondences as classifier scores. We also introduce a formulation of visual context that is flexible, efficient to compute, and can capture relationships in the entire image plane. The original binary descriptors are augmented with contextual information and informative features are selected by the boosting framework. Through detailed experiments, we evaluate the retrieval quality and performance of Landmark-Boost, demonstrating that it outperforms common state-of-the-art descriptor matching methods.

Collecting 3D object data sets involves a large amount of manual work and is time consuming. Getting complete models of objects either requires a 3D scanner that covers all the surfaces of an object or one needs to rotate it to completely observe it. We present a system that incrementally builds a database of objects as a mobile agent traverses a scene. Our approach requires no prior knowledge of the shapes present in the scene. Object-like segments are extracted from a global segmentation map, which is built online using the input of segmented RGB-D images. These segments are stored in a database, matched among each other, and merged with other previously observed instances. This allows us to create and improve object models on the fly and to use these merged models to reconstruct also unobserved parts of the scene. The database contains each (potentially merged) object model only once, together with a set of poses where it was observed. We evaluate our pipeline with one public dataset, and on a newly created Google Tango dataset containing four indoor scenes with some of the objects appearing multiple times, both within and across scenes.

In many applications, maintaining a consistent dense map of the environment is key to enabling robotic platforms to perform higher level decision making. Several works have addressed the challenge of creating precise dense 3D maps from visual sensors providing depth information. However, during operation over longer missions, reconstructions can easily become inconsistent due to accumulated camera tracking error and delayed loop closure. Without explicitly addressing the problem of map consistency, recovery from such distortions tends to be difficult. We present a novel system for dense 3D mapping which addresses the challenge of building consistent maps while dealing with scalability. Central to our approach is the representation of the environment as a collection of overlapping Truncated Signed Distance Field (TSDF) subvolumes. These subvolumes are localized through feature-based camera tracking and bundle adjustment. Our main contribution is a pipeline for identifying stable regions in the map, and to fuse the contributing subvolumes. This approach allows us to reduce map growth while still maintaining consistency. We demonstrate the proposed system on a publicly available dataset and simulation engine, and demonstrate the efficacy of the proposed approach for building consistent and scalable maps. Finally we demonstrate our approach running in real-time onboard a lightweight Micro Aerial Vehicle (MAV).

Micro-Aerial Vehicles (MAVs) have the advantage of moving freely in 3D space. However, creating compact and sparse map representations that can be efficiently used for planning for such robots is still an open problem. In this paper, we take maps built from noisy sensor data and construct a sparse graph containing topological information that can be used for 3D planning. We use a Euclidean Signed Distance Field, extract a 3D Generalized Voronoi Diagram (GVD), and obtain a thin skeleton diagram representing the topological structure of the environment. We then convert this skeleton diagram into a sparse graph, which we show is resistant to noise and changes in resolution. We demonstrate global planning over this graph, and the orders of magnitude speed-up it offers over other common planning methods. We validate our planning algorithm in real maps built onboard an MAV, using RGB-D sensing.

We present a complete map management process for a visual localization system designed for multi-vehicle long-term operations in resource constrained outdoor environments. Outdoor visual localization generates large amounts of data that need to be incorporated into a lifelong visual map in order to allow localization at all times and under all appearance conditions. Processing these large quantities of data is non-trivial, as it is subject to limited computational and storage capabilities both on the vehicle and on the mapping backend. We address this problem with a two-fold map update paradigm capable of, either, adding new visual cues to the map, or updating co-observation statistics. The former, in combination with offline map summarization techniques, allows enhancing the appearance coverage of the lifelong map while keeping the map size limited. On the other hand, the latter is able to significantly boost the appearance-based landmark selection for efficient online localization without incurring any additional computational or storage burden. Our evaluation in challenging outdoor conditions shows that our proposed map management process allows building and maintaining maps for precise visual localization over long time spans in a tractable and scalable fashion.

This paper reports on an algorithm for planning trajectories that allow a multirotor micro aerial vehicle (MAV) to quickly identify a set of unknown parameters. In many problems like self calibration or model parameter identification some states are only observable under a specific motion. These motions are often hard to find, especially for inexperienced users. Therefore, we consider system model identification in an active setting, where the vehicle autonomously decides what actions to take in order to quickly identify the model. Our algorithm approximates the belief dynamics of the system around a candidate trajectory using an extended Kalman filter (EKF). It uses sampling-based motion planning to explore the space of possible beliefs and find a maximally informative trajectory within a user-defined budget. We validate our method in simulation and on a real system showing the feasibility and repeatability of the proposed approach. Our planner creates trajectories which reduce model parameter convergence time and uncertainty by a factor of four.

Place recognition in 3D data is a challenging task that has been commonly approached by adapting image-based solutions. Methods based on local features suffer from ambiguity and from robustness to environment changes while methods based on global features are viewpoint dependent. We propose SegMatch, a reliable place recognition algorithm based on the matching of 3D segments. Segments provide a good compromise between local and global descriptions, incorporating their strengths while reducing their individual drawbacks. SegMatch does not rely on assumptions of `perfect segmentation', or on the existence of `objects' in the environment, which allows for reliable execution on large scale, unstructured environments. We quantitatively demonstrate that SegMatch can achieve accurate localization at a frequency of 1Hz on the largest sequence of the KITTI odometry dataset. We furthermore show how this algorithm can reliably detect and close loops in real-time, during online operation. In addition, the source code for the SegMatch algorithm is made publicly available.

Autonomous delivery of goods using a Micro Air Vehicle (MAV) is a difficult problem, as it poses high demand on the MAV's control, perception and manipulation capabilities. This problem is especially challenging if the exact shape, location and configuration of the objects are unknown. In this paper, we report our findings during the development and evaluation of a fully integrated system that is energy efficient and enables MAVs to pick up and deliver objects with partly ferrous surface of varying shapes and weights. This is achieved by using a novel combination of an electro-permanent magnetic gripper with a passively compliant structure and integration with detection, control and servo positioning algorithms. The system's ability to grasp stationary and moving objects was tested, as well as its ability to cope with different shapes of the object and external disturbances. We show that such a system can be successfully deployed in scenarios where an object with partly ferrous parts needs to be gripped and placed in a predetermined location.

Perception systems for outdoor robotics have to deal with varying environmental conditions. Variations in illumination in particular, are currently the biggest challenge for vision-based perception. In this paper we present an approach for radiometric characterization of multispectral cameras. To enable spatio-temporal mapping we also present a procedure for in-situ illumination estimation, resulting in radiometric calibration of the collected images. In contrast to current approaches, we present a purely data driven, parameter free approach, based on maximum likelihood estimation which can be performed entirely on the field, without requiring specialised laboratory equipment. Our routine requires three simple datasets which are easily acquired using most modern multispectral cameras. We evaluate the framework with a cost-effective snapshot multispectral camera. The results show that our method enables the creation of quatitatively accurate relative reflectance images with challenging on field calibration datasets under a variety of ambient conditions.

Learning from demonstration for motion planning is an ongoing research topic. In this paper we present a model that is able to learn the complex mapping from raw 2D-laser range findings and a target position to the required steering commands for the robot. To our best knowledge, this work presents the first approach that learns a target-oriented end-to-end navigation model for a robotic platform. The supervised model training is based on expert demonstrations generated in simulation with an existing motion planner. We demonstrate that the learned navigation model is directly transferable to previously unseen virtual and, more interestingly, real-world environments. It can safely navigate the robot through obstacle-cluttered environments to reach the provided targets. We present an extensive qualitative and quantitative evaluation of the neural network-based motion planner, and compare it to a grid-based global approach, both in simulation and in real-world experiments.

In this paper, we introduce an informative path planning (IPP) framework for active classification using unmanned aerial vehicles (UAVs). Our algorithm uses a combination of global viewpoint selection and evolutionary optimization to refine the planned trajectory in continuous 3D space while satisfying dynamic constraints. Our approach is evaluated on the application of weed detection for precision agriculture. We model the presence of weeds on farmland using an occupancy grid and generate adaptive plans according to information-theoretic objectives, enabling the UAV to gather data efficiently. We validate our approach in simulation by comparing against existing methods, and study the effects of different planning strategies. Our results show that the proposed algorithm builds maps with over 50% lower entropy compared to traditional “lawnmower” coverage in the same amount of time. We demonstrate the planning scheme on a multirotor platform with different artificial farmland set-ups.

Environmental conditions and external effects, such as shocks, have a significant impact on the calibration parameters of visual-inertial sensor systems. Thus long-term operation of these systems cannot fully rely on factory calibration. Since the observability of certain parameters is highly dependent on the motion of the device, using short data segments at device initialization may yield poor results. When such systems are additionally subject to energy constraints, it is also infeasible to use full-batch approaches on a big dataset and careful selection of the data is of high importance. In this paper, we present a novel approach for resource efficient self-calibration of visual-inertial sensor systems. This is achieved by casting the calibration as a segment-based optimization problem that can be run on a small subset of informative segments. Consequently, the computational burden is limited as only a predefined number of segments is used. We also propose an efficient information-theoretic selection to identify such informative motion segments. In evaluations on a challenging dataset, we show our approach to significantly outperform state-of-the-art in terms of computational burden while maintaining a comparable accuracy.

This paper shows a strategy based on passive force control for collaborative object transportation using Micro Aerial Vehicles (MAVs), focusing on the transportation of a bulky object by two hexacopters. The goal is to develop a robust approach which does not rely on: (a) communication links between the MAVs, (b) the knowledge of the payload shape and (c) the position of grasping point. The proposed approach is based on the master-slave paradigm, in which the slave agent guarantees compliance to the external force applied by the master to the payload via an admittance controller. The external force acting on the slave is estimated using a non-linear estimator based on the Unscented Kalman Filter (UKF) from the information provided by a Visual-Inertial (VI) navigation system. Experimental results (online video [1]) demonstrate the performance of the force estimator and show the collaborative transportation of a 1.2 m long object.

Using multiple cooperative robots is advantageous for time critical Search and Rescue (SaR) missions as they permit rapid exploration of the environment and provide higher redundancy than using a single robot. A considerable number of applications such as autonomous driving and disaster response could benefit from merging mapping data from several agents. Online multi-robot localization and mapping has mainly been addressed for robots equipped with cameras or 2D LiDARs. However, in unstructured and ill-lighted real-life scenarios, a mapping system can potentially benefit from a rich 3D geometric solution. In this work, we present an online localization and mapping system for multiple robots equipped with 3D LiDARs. This system is based on incremental sparse pose-graph optimization using sequential and place recognition constraints, the latter being identified using a 3D segment matching approach. The result is a unified representation of the world and relative robot trajectories. The complete system runs in real-time and is evaluated with two experiments in different environments: one urban and one disaster scenario. The system is available open source and easy-to-run demonstrations are publicly available.

When several Multirotor Micro Aerial Vehicles (MAVs) share the same airspace, reliable and robust collision avoidance is required. In this paper we address the problem of multi-MAV reactive collision avoidance. We employ a model-based controller to simultaneously track a reference trajectory and avoid collisions. Moreover, to achieve a higher degree of robustness, our method also accounts for the uncertainty of the state estimator and of the position and velocity of the other agents. The proposed approach is decentralized, does not require a collision-free reference trajectory and accounts for the full MAV dynamics. We validated our approach in simulation and experimentally with two MAV.

Micro Aerial Vehicles (MAVs) that operate in unstructured, unexplored environments require fast and flexible local planning, which can replan when new parts of the map are explored. Trajectory optimization methods fulfill these needs, but require obstacle distance information, which can be given by Euclidean Signed Distance Fields (ESDFs). We propose a method to incrementally build ESDFs from Truncated Signed Distance Fields (TSDFs), a common implicit surface representation used in computer graphics and vision. TSDFs are fast to build and smooth out sensor noise over many observations, and are designed to produce surface meshes. We show that we can build TSDFs faster than Octomaps, and that it is more accurate to build ESDFs out of TSDFs than occupancy maps. Our complete system, called voxblox, is available as open source and runs in real-time on a single CPU core. We validate our approach on-board an MAV, by using our system with a trajectory optimization local planner, entirely on-board and in real-time.

Unmanned aerial vehicles (UAVs) can offer timely and cost-effective delivery of high-quality sensing data. However, deciding when and where to take measurements in complex environments remains an open challenge. To address this issue, we introduce a new multiresolution mapping approach for informative path planning in terrain monitoring using UAVs. Our strategy exploits the spatial correlation encoded in a Gaussian Process model as a prior for Bayesian data fusion with probabilistic sensors. This allows us to incorporate altitude-dependent sensor models for aerial imaging and perform constant-time measurement updates. The resulting maps are used to plan information-rich trajectories in continuous 3-D space through a combination of grid search and evolutionary optimization. We evaluate our framework on the application of agricultural biomass monitoring. Extensive simulations show that our planner performs better than existing methods, with mean error reductions of up to 45% compared to traditional “lawnmower” coverage. We demonstrate proof of concept using a multirotor to map color in different environments.

Deployment of camera and laser based motion estimation systems for controlling platforms operating at high speeds, such as cars or trains, is posing increasingly challenging precision requirements on the temporal calibration of these sensors. In this work, we demonstrate a simple, low-cost system for calibrating any combination of cameras and time of flight LIDARs with respect to the CPU clock (and therefore, also to each other). The newly proposed device is based on widely available off-the-shelf components, such as the Raspberry Pi 3, which is synchronized using the Precision Time Protocol (PTP) with respect to the CPU of the sensor carrying system. The obtained accuracy can be shown to be below 0.1 ms per measurement for LIDARs and below minimal exposure time per image for cameras. It outperforms state-of-the-art approaches also not relying on hardware synchronization by more than a factor of 10 in precision. Moreover, the entire process can be carried out at a high rate allowing the study of how offsets evolve over time. In our analysis, we demonstrate how each building block of the system contributes to this accuracy and validate the obtained results using real-world data.

In this paper, we propose a GPU parallelized SLAM system capable of using photometric and inertial data together with depth data from an active RGB-D sensor to build accurate dense 3D maps of indoor environments. We describe several extensions to existing dense SLAM techniques that allow us to operate in real-time onboard memory constrained robotic platforms. Our primary contribution is a memory management algorithm that scales to large scenes without being limited by GPU memory resources. Moreover, by integrating a visual-inertial odometry system, we robustly track the camera pose even on an agile platform such as a quadrotor UAV. Our robust camera tracking framework can deal with fast camera motions and varying environments by relying on depth, color and inertial motion cues. Global consistency is achieved via regular checking for loop closures in conjunction with a pose graph, as a basis for corrective deformation of the 3D map. Our efficient SLAM system is capable of producing highly dense meshes up to 5mm resolution at rates close to 60Hz fully onboard a UAV. Experimental validations both in simulation and on a real-world platform, show that our approach is fast, more robust and more memory efficient than state-of-the-art techniques, while obtaining better or comparable accuracy.

An increasing number of simultaneous localization and mapping (SLAM) systems are using appearance-based localization to improve the quality of pose estimates. However, with the growing time-spans and size of the areas we want to cover, appearance-based maps are often becoming too large to handle and are consisting of features that are not always reliable for localization purposes. This paper presents a method for selecting map features that are persistent over time and thus suited for long-term localization. Our methodology relies on a CNN classifier based on image patches and depth maps for recognizing which features are suitable for life-long matchability. Thus, the classifier not only considers the appearance of a feature but also takes into account its expected lifetime. As a result, our feature selection approach produces more compact maps with a high fraction of temporally-stable features compared to the current state-of-the-art, while rejecting unstable features that typically harm localization. Our approach is validated on indoor and outdoor datasets, that span over a period of several months.

A novel method for visual place recognition is introduced and evaluated, demonstrating robustness to perceptual aliasing and observation noise. This is achieved by increasing discrimination through a more structured representation of visual observations. Estimation of observation likelihoods are based on graph kernel formulations, utilizing both the structural and visual information encoded in covisibility graphs. The proposed probabilistic model is able to circumvent the typically difficult and expensive posterior normalization procedure by exploiting the information available in visual observations. Furthermore, the place recognition complexity is independent of the size of the map. Results show improvements over the state-of-theart on a diverse set of both public datasets and novel experiments, highlighting the benefit of the approach.

The illumination conditions of a scene create intra-class variability in outdoor visual data, degrading the performance of high-level algorithms. Using only the image, and with hyper-spectral data as a case study, this paper proposes a deep learning approach to learn illumination invariant features from the data in an unsupervised manner. The proposed approach incorporates a similarity measure, the Spectral Angle, that is relatively insensitive to brightness into the cost function of a Stacked Auto-Encoder so that an illumination invariant mapping is learned from the input data to the hidden layer. Experiments using synthetic and real imagery show that this novel feature learning approach produces a more illumination invariant representation of the data, improving the results of a high-level algorithm (clustering) under such conditions.

In this paper, we present an online landmark selection method for distributed long-term visual localization systems in bandwidth-constrained environments. Sharing a common map for online localization provides a fleet of autonomous vehicles with the possibility to maintain and access a consistent map source, and therefore reduce redundancy while increasing efficiency. However, connectivity over a mobile network imposes strict bandwidth constraints and thus the need to minimize the amount of exchanged data. The wide range of varying appearance conditions encountered during long-term visual localization offers the potential to reduce data usage by extracting only those visual cues which are relevant at the given time. Motivated by this, we propose an unsupervised method of adaptively selecting landmarks according to how likely these landmarks are to be observable under the prevailing appearance condition. The ranking function this selection is based upon exploits landmark co-observability statistics collected in past traversals through the mapped area. Evaluation is performed over different outdoor environments, large time-scales and varying appearance conditions, including the extreme transition from day-time to night-time, demonstrating that with our appearance-dependent selection method, we can significantly reduce the amount of landmarks used for localization while maintaining or even improving the localization performance.

Persistent merging of maps created by different sensor modalities is an insufficiently addressed problem. Current approaches either rely on appearance-based features which may suffer from lighting and viewpoint changes or require pre-registration between all sensor modalities used. This work presents a framework using structural descriptors for matching LIDAR point-cloud maps and sparse vision keypoint maps. The matching algorithm works independently of the sensors' viewpoint and varying lighting and does not require pre-registration between the sensors used. Furthermore, we employ the approach in a novel vision-laser map-merging algorithm. We analyse a range of structural descriptors and present results of the method integrated within a full mapping framework. Despite the fact that we match between the visual and laser domains, we can successfully perform map-merging using structural descriptors. The effectiveness of the presented structure-based vision-laser matching is evaluated on the public KITTI dataset and furthermore demonstrated on a map merging problem in an industrial site.

Multirotor unmanned aerial vehicles (UAVs) are rapidly gaining popularity for many applications. However, safe operation in partially unknown, unstructured environments remains an open question. In this paper, we present a continuous-time trajectory optimization method for real-time collision avoidance on multirotor UAVs. We then propose a system where this motion planning method is used as a local replanner, that runs at a high rate to continuously recompute safe trajectories as the robot gains information about its environment. We validate our approach by comparing against existing methods and demonstrate the complete system avoiding obstacles on a multirotor UAV platform.

Future automated driving systems will require a comprehensive scene understanding. Considering these systems in an urban environment it becomes immediately clear that reasoning about the future behavior and trajectories of pedestrians represents one major challenge. In this paper we focus on predicting the pedestrians' time-to-cross when approaching a crosswalk. Due to the complexity of the underlying model, we propose a data-driven approach that by means of regression models learns the target variable. Instead of utilizing a standard mean regression, we propose the use of Quantile Regression. We show that this special type of regression is more suited to describe the variability of real world pedestrian trajectories. We examine and compare two approaches: Linear Quantile Regression and Quantile Regression Forest, which is an extended version of Random Forests. We present evaluations with real data and a detailed analysis emphasizing strengths and weaknesses of quantile regression for the target application.

This paper presents a system for calibrating the extrinsic parameters and timing offsets of an array of cameras, 3-D lidars, and global positioning system/inertial navigation system sensors, without the requirement of any markers or other calibration aids. The aim of the approach is to achieve calibration accuracies comparable with state-of-the-art methods, while requiring less initial information about the system being calibrated and thus being more suitable for use by end users. The method operates by utilizing the motion of the system being calibrated. By estimating the motion each individual sensor observes, an estimate of the extrinsic calibration of the sensors is obtained. Our approach extends standard techniques for motion-based calibration by incorporating estimates of the accuracy of each sensor's readings. This yields a probabilistic approach that calibrates all sensors simultaneously and facilitates the estimation of the uncertainty in the final calibration. In addition, we combine this motion-based approach with appearance information. This gives an approach that requires no initial calibration estimate and takes advantage of all available alignment information to provide an accurate and robust calibration for the system. The new framework is validated with datasets collected with different platforms and different sensors' configurations, and compared with state-of-the-art approaches.

Outdoor robotic systems rely on perception modules that must be robust to variations in environmental conditions. In particular, vision-based perception systems are affected by illumination variations caused by occlusions. We propose an approach to calculate the lighting distribution of outdoor scenes. The new approach enables us to compensate for shadows and therefore obtain images which are invariant to the sun position and scene geometry, while also retaining the dimensionality of the original data. The method combines images with geometric information provided by range sensors to infer shadows. We select a pair of points on a shadow boundary from a single material and estimate a terrestrial sunlight-skylight ratio. Individual scaling factors are then calculated for all points based on their orientation and incident illumination sources. The result is a coloured point cloud that is independent of illumination variation due to occlusions and geometry. To demonstrate the effectiveness and generalisation of the approach, we present evaluations using two datasets with different cameras. The first uses a hyperspectral sensor that allows us to analyse the results for a large number of wavelengths, while the second dataset uses a standard RGB camera. The approach is shown to consistently provide good illumination compensation in both scenarios.

This paper formulates a new pipeline for automated extrinsic calibration of multi-sensor mobile platforms. The new method can operate on any combination of cameras, navigation sensors and 3D lidars. Current methods for extrinsic calibration are either based on special markers and/or chequerboards, or they require a precise parameters initialisation for the calibration to converge. These two limitations prevent them from being fully automatic. The method presented in this paper removes these restrictions. By combining information extracted from both, platform's motion estimates and external observations, our approach eliminates the need for special markers and also removes the need for manual initialisation. A third advantage is that the motion-based automatic initialisation does not require overlapping field of view between sensors. The paper also provides a method to estimate the accuracy of the resulting calibration. We illustrate the generalisation of our approach and validate its performance by showing results with two contrasting datasets. The first dataset was collected in a city with a car platform, and the second one was collected in a tree-crop farm with a Segway platform.

This paper presents an approach to automatically register hyperspectral images with lidar point clouds using a combination of SIFT and SURF feature descriptors. The aim is to generate 3D terrain maps of the environment combining spectral and geometrical information. The datasets are acquired from field-based platforms which, due to the lack of georeferencing, cannot be simply fused and require a registration processing step. In addition, some applications, such as in mining, cannot rely on reliable GPS signal. The proposed method is validated using experimental data acquired from vertical mine walls.

This paper presents a multi-class image segmentation approach to automate fruit segmentation. A feature learning algorithm combined with a conditional random field is applied to multi-spectral image data. Current classification methods used in agriculture scenarios tend to use hand crafted application-based features. In contrast, our approach uses unsupervised feature learning to automatically capture most relevant features from the data. This property makes our approach robust against variance in canopy trees and therefore has the potential to be applied to different domains. The proposed algorithm is applied to a fruit segmentation problem for a robotic agricultural surveillance mission, aiming to provide yield estimation with high accuracy and robustness against fruit variance. Experimental results with data collected in an almond farm are shown. The segmentation is performed with features extracted from multi-spectral (colour and infrared) data. We achieve a global classification accuracy of 88%.

A novel technique for calibrating a multi-modal sensor system has been developed. Our calibration method is based on the comparative alignment of output gradients from two candidate sensors. The algorithm is applied to the calibration of the extrinsic parameters of several camera-lidar systems. In this calibration the lidar scan is projected onto the camera's image using a camera model. Particle swarm optimization is used to find the optimal parameters for this model. This method requires no markers to be placed in the scene. While the system can use a set of scans, unlike many existing techniques it can also automatically calibrate the system reliably using a single scan. The method presented is successfully validated on a variety of cameras, lidars and locations. It is also compared to three existing techniques and shown to give comparable or superior results on the datasets tested.

This paper presents a motion detection approach based on a combination of dense optical flow and 3D stereo reconstruction. Our motion detection is not based on predefined templates, providing a generic framework suitable for a broad range of applications such as situation awareness. The approach estimates the likelihood of pixels motion from the fusion of dense optical flow and dense depth information estimated from a stereo camera. Temporal consistency is incorporated by tracking moving objects across consecutive images. The proposed algorithm is validated with publicly available datasets. The consistent results across different scenarios demonstrate the robustness of our framework, presenting an average detection rate of 92%.

Learning motion patterns in dynamic environments is a key component of any context-aware robotic system, and probabilistic mixture models provide a sound framework for mining these patterns. This paper presents an approach for learning motion models from trajectories provided by the tracking system of a moving platform. We present a learning approach in which a Linear Dynamical System (LDS) is augmented with a discrete hidden variable that has a number of states equal to the number of behaviours in the environment. As a result, a mixture of linear dynamical systems (MLDSs) capable of explaining several motion behaviours is developed. The model is learned by means of the Expectation Maximization (EM) algorithm.

Human drivers are endowed with an inborn ability to put themselves in the position of other drivers and reason about their behavior and intended actions. State-of-the-art driving-assistance systems, on the other hand, are generally limited to physical models and ad hoc safety rules. In order to drive safely amongst humans, autonomous vehicles need to develop an understanding of the situation in the form of a high-level description of the state of traffic participants. This paper presents a probabilistic model to estimate the state of vehicles by considering interactions between drivers immersed in traffic. The model is defined within a probabilistic filtering framework; estimation and prediction are carried out with statistical inference techniques. Memory requirements increase linearly with the number of vehicles, and thus, it is possible to scale the model to complex scenarios involving many participants. The approach is validated using real-world data collected by a group of interacting ground vehicles.

We introduce a novel approach for processing sequential data in the presence of outliers. The outlier-robust Kalman filter we propose is a discrete-time model for sequential data corrupted with non-Gaussian and heavy-tailed noise. We present efficient filtering and smoothing algorithms which are straightforward modifications of the standard Kalman filter Rauch-Tung-Striebel recursions and yet are much more robust to outliers and anomalous observations. Additionally, we present an algorithm for learning all of the parameters of our outlier-robust Kalman filter in a completely unsupervised manner. The potential of our approach is borne out in experiments with synthetic and real data.

This paper presents a probabilistic framework for road geometry estimation using a millimetre wave radar. It aims at estimating the geometry of roads without assuming any particular infrastructure such as lane marks. It provides also the vehicle location with respect to the edges of the road. This system employs a radar sensor in view of its robustness to weather conditions such as fog, dust, rain and snow. The proposed approach is robust to noisy measurements since the radar target locations are modelled as Gaussian distributions. These observations are integrated into a Kalman Particle filter to estimate the posterior distribution of the parameters that best describe the geometry of the road. Experimental results using data acquired on a highway road are presented. The effectiveness of the proposed approach is demonstrated by a qualitative analysis of the results.

One of the most important problems in robot localisation is the detection of previously visited places (loops). When a robot closes a loop, the association between observed features and present ones can be used to update its position. The computational cost involved in the association process makes exhaustive loop search intractable. Most of the current techniques use observations of the environment as their main features to produce loop hypotheses. In this paper, we investigate the feasibility of producing loop candidates from features of the robot trajectory. We propose a new method for selecting loop-closure candidates based on an alignment likelihood function, which measures similarity between trajectory sequences. The algorithm is validated with data gathered in the city with our experimental platform. Positive results show that the trajectory has, indeed, features that can be extracted and applied to robot localisation. The resulting loop hypotheses may be regarded, for example, as a initialisation step to aid current methods.

Human drivers are endowed with an inborn ability to put themselves in the position of other drivers and reason about their behaviors and intended actions. State-of-the-art driving assistance systems, on the other hand, are generally limited to physical models and ad-hoc safety rules. In order to drive safely amongst humans, autonomous vehicles require a high-level description of the state of traffic participants. This paper presents a probabilistic model for estimating and predicting the behavior of drivers immersed in traffic. The model is defined within a stochastic filtering framework and estimation and prediction are carried out with statistical inference techniques. The approach is validated with real data from a fleet of mining vehicles.

Over the last few years, electronic vehicle guidance systems have become increasingly more popular. However, despite their ubiquity, performance will always be subject to availability of detailed digital road maps. Most current digital maps are still inadequate for advanced applications in unstructured environments. Lack of up-to-date information and insufficient refinement of the road geometry are among the most important shortcomings. The massive use of inexpensive Global Positioning System (GPS) receivers, combined with the rapidly increasing availability of wireless communication infrastructure, suggests that large amounts of data combining both modalities will be available in the near future. The approach presented here draws on machine-learning techniques and processes logs of position traces to consistently build a detailed and fine-grained representation of the road network by extracting the principal paths followed by the vehicles. Although this work addresses the road-building problem in dynamic environments such as open-pit mines, it is also applicable to urban environments. New contributions include a fully unsupervised segmentation method for sampling roads and inferring the network topology, which is a general technique for extracting detailed information about road splits, merges, and intersections, as well as a robust algorithm that articulates these two. Experimental results with data from large mining operations are presented to validate the new algorithm.

Over the last ten years, electronic vehicle guidance systems have become increasingly popular. However, their performance is subject to the availability and accuracy of digital road maps. Most current digital maps are still inadequate for advanced applications in unstructured environments. Lack of detailed up-to-date information and insufficient accuracy and refinement of the road geometry are among the most important shortcomings. The massive use of inexpensive GPS receivers, combined with the rapidly increasing availability of wireless communication infrastructure, suggests that large volumes of data combining both modalities will be available in a near future. The approach presented here draws on machine learning techniques to process logs of position traces to consistently build a detailed and accurate representation of the road network and, more importantly, extract the actual paths followed by vehicles. Experimental results with data from large mining operations are presented to validate the algorithm.

This paper presents a ground based system for mapping the geology and the geometry of the environment remotely. The main objective of this work is to develop a framework for a mobile robotic platform that can build 3D geological maps. We investigate classification and registration algorithms that can work without any manual intervention. The system capabilities are demonstrated with data acquired from a working mine environment. Geological maps are built by applying classification techniques to hyperspectral images of the rocks' surface. The result from the classification is then fused with laser images to form the 3D geological models of the environment.

This paper addresses the problem of safety in mining applications. It presents new metrics that can be used to determine dangerous situations during mine operation in real time. It also presents a fast and robust algorithm for extracting significant places from information logged by a state-of-the-art collision avoidance system. Determining significant places provides valuable context information in a variety of applications such as map building, vehicle tracking and user assistance. In our case, we are interested in obtaining context information as a preliminary step towards improving mining safety. The algorithm presented here is validated with experimental data obtained from a fleet of haulage vehicles operating in various open pit mines.

Despite significant developments in the simultaneous localisation and mapping (SLAM) problem, loop closure detection is still challenging in large scale unstructured environments. Current solutions rely on heuristics that lack generalisation properties, in particular when range sensors are the only source of information about the robot's surrounding environment. This paper presents a machine learning approach for the loop closure detection problem using range sensors. A binary classifier based on boosting is used to detect loop closures. The algorithm performs robustly, even under potential occlusions and significant changes in rotation and translation. We developed a number of features, extracted from range data, that are invariant to rotation. Additionally, we present a general framework for scan-matching SLAM in outdoor environments. Experimental results in large scale urban environments show the robustness of the approach, with a detection rate of 85% and a false alarm rate of only 1%. The proposed algorithm can be computed in real-time and achieves competitive performance with no manual specification of thresholds given the features.

This work introduces a self-supervised, multi-sensor architecture that performs automatic moving obstacles classification. Our approach presents a hierarchical scheme that relies on the ldquostabilityrdquo of a subset of features given by a sensor to perform an initial robust classification based on unsupervised techniques. The obtained results are used as labels to train a set of supervised classifiers, which can be then combined to improve the final classification accuracy. The proposed architecture is general and can be instantiated in a variety of ways, using different sensors and classifiers. The applicability and validity of the proposed architecture is evaluated for a particular realization based on range and visual information that achieves 83% accuracy without using manually labeled data. Experimental results also demonstrate how accuracy can be maintained through self-training capabilities when working conditions change.

This paper presents a motion detection scheme using laser scanners mounted on a mobile vehicle. We propose a stable, yet simple motion detection scheme that can be used and improved with tracking and classification procedures. The salient contribution of the developed architecture is twofold. It proposes a spatio-temporal correspondence procedure based on a scan registration algorithm. The detection is cast as a probability decision problem that accounts for sensor noise and achieves robust classification. Probabilistic occlusion checking is finally performed to improve robustness. Experimental results show the performance of the proposed architecture under different settings in urban environments.

In this paper, simultaneous localisation and mapping (SLAM) is combined with landmark recognition to close large loops in unstructured, outdoor environments. Camera and laser information are fused to recognise and create appearance models for landmarks. The representation is obtained through a non-linear probabilistic regression model encoding a neighbourhood preserving dimensionality reduction. A new data association algorithm is proposed where landmarks are associated based on both position and appearance. The resulting system is more robust and able to recover from possible misassociations. Experiments demonstrate the benefits of this approach in challenging problems involving mapping with large loop closings in irregular terrain, and with dynamic objects.

This paper presents an analysis of FastSLAM - a Rao-Blackwellised particle filter formulation of simultaneous localisation and mapping. It shows that the algorithm degenerates with time, regardless of the number of particles used or the density of landmarks within the environment, and would always produce optimistic estimates of uncertainty in the long-term. In essence, FastSLAM behaves like a non-optimal local search algorithm; in the short-term it may produce consistent uncertainty estimates but, in the long-term, it is unable to adequately explore the state-space to be a reasonable Bayesian estimator. However, the number of particles and landmarks does affect the accuracy of the estimated mean and, given sufficient particles, FastSLAM can produce good non-stochastic estimates in practice. FastSLAM also has several practical advantages, particularly with regard to data association, and would probably work well in combination with other versions of stochastic SLAM, such as EKF-based SLAM

This paper presents an analysis of the extended Kalman filter formulation of simultaneous localisation and mapping (EKF-SLAM). We show that the algorithm produces very optimistic estimates once the "true" uncertainty in vehicle heading exceeds a limit. This failure is subtle and cannot, in general, be detected without ground-truth, although a very inconsistent filter may exhibit observable symptoms, such as disproportionately large jumps in the vehicle pose update. Conventional solutions - adding stabilising noise, using an iterated EKF or unscented filter, etc., - do not improve the situation. However, if "small" heading uncertainty is maintained, EKF-SLAM exhibits consistent behaviour over an extended time-period. Although the uncertainty estimate slowly becomes optimistic, inconsistency can be mitigated indefinitely by applying tactics such as batch updates or stabilising noise. The manageable degradation of small heading variance SLAM indicates the efficacy of submap methods for large-scale maps

Although full 3D navigation and mapping is recognized as one of the most important challenges for autonomous navigation, the lack of robust sensors, providing 3D information in real time, has burdened the progress in this direction. This paper presents our ongoing work towards the deployment of an integrated sensing system for 3D mapping in outdoor environments. We first describe a 3D data acquisition architecture based on a standard 2D laser. Techniques for registering scans using a scan matching procedure and for estimating the errors are then introduced. We finally present results showing the performance of the proposed architecture in real outdoor environments by means of the integration of the 3D scans with dead reckoning and inertial measurement unit (IMU) information

This work presents a new hybrid metric map representation (HYMM) that combines feature maps with other dense metric sensory information. The global feature map is partitioned into a set of connected local triangular regions (LTRs), which provide a reference for a detailed multi-dimensional description of the environment. The HYMM framework permits the combination of efficient feature-based SLAM algorithms for localisation with, for example, occupancy grid (OG) maps. This fusion of feature and grid maps has several complementary properties; for example, grid maps can assist data association and can facilitate the extraction and incorporation of new landmarks as they become identified from multiple vantage points. The representation presented here will allow the robot to perform DenseSLAM. DenseSLAM is the process of performing SLAM whilst obtaining a dense environment representation.

This paper proposes a new algorithm named "SIGMA" to address the problem of simultaneous information and global motion analysis for a car working in unstructured outdoor environments. The map of the environment is made by a Simultaneous Localization and Mapping (SLAM) algorithm that uses an Hybrid Metric Map (HYMM) structure for mapping. The path planning approach presents a global solution maximizing overall information gain of the map. The cost function used considers the present and future uncertainty in the map and vehicle and is based on the variation of the covariance matrix trace. Eigenvalue concepts are utilized to determine overall information change from the information matrix properties. An information graph is constructed followed by a search to find the optimal information-based rough path. Results are presented to demonstrate performance of the algorithm.

The ability to simultaneously localise a robot and accurately map its surroundings is considered by many to be a key prerequisite of truly autonomous robots. This paper presents a real-world implementation of FastSLAM, an algorithm that recursively estimates the full posterior distribution of both robot pose and landmark locations. In particular, we present an extension to FastSLAM that addresses the data association problem using a nearest neighbor technique. Building on this, we also present a novel multiple hypotheses tracking implementation (MHT) to handle uncertainty in the data association. Finally an extension to the multi-robot case is introduced. Our algorithm has been run successfully using a number of data sets obtained in outdoor environments. Experimental results are presented that demonstrate the performance of the algorithms when compared with standard Kalman filter-based approaches.

This paper presents two approaches for the problem of multiple target tracking (MTT) and specifically people tracking. Both filters are based on sequential Monte Carlo methods (SMCM) and joint probability data association (JPDA). The filters have been implemented and tested on real data from a laser measurement system. Experiments show that both approaches are able to track multiple moving persons. A comparison of both filters is given and the advantages and disadvantages of the two approaches are presented.