Surface normal integration is a fundamental problem in computer vision, dealing with the objective of reconstructing a surface from its corresponding normal map. Existing approaches require an iterative global optimization to jointly estimate the depth of each pixel, which scales poorly to larger normal maps. In this paper, we address this problem by recasting normal integration as the estimation of relative scales of continuous components. By constraining pixels belonging to the same component to jointly vary their scale, we drastically reduce the number of optimization variables. Our framework includes a heuristic to accurately estimate continuous components from the start, a strategy to rebalance optimization terms, and a technique to iteratively merge components to further reduce the size of the problem. Our method achieves state-of-the-art results on the standard normal integration benchmark in as little as a few seconds and achieves one-order-of-magnitude speedup over pixel-level approaches on large-resolution normal maps.

Tilt-rotor aerial robots are more dynamic and versatile than fixed-rotor platforms, since the thrust vector and body orientation are decoupled. However, the coordination of servos and propellers (the allocation problem) is not trivial, especially accounting for overactuation and actuator dynamics. We incrementally build and present three novel allocation methods for tilt-rotor aerial robots, comparing them to state-of-the-art methods on a real system performing dynamic maneuvers. We extend the state-of-the-art geometric allocation into a differential allocation, which uses the platform's redundancy and does not suffer from singularities. We expand it by incorporating actuator dynamics and propeller power dynamics. These allow us to model dynamic propeller acceleration limits, bringing two main advantages: balancing propeller speed without the need for nullspace goals and allowing the platform to selectively turn off propellers during flight, opening the door to new manipulation possibilities. We also use actuator dynamics and limits to normalize the allocation problem, making it easier to tune and allowing it to track 70% faster trajectories than a geometric allocation.

Recovering a 3D surface from its surface normal map, a problem known as normal integration, is a key component for photometric shape reconstruction techniques such as shape-from-shading and photometric stereo. The vast majority of existing approaches for normal integration handle only implicitly the presence of depth discontinuities and are limited to orthographic or ideal pinhole cameras. In this paper, we propose a novel formulation that allows modeling discontinuities explicitly and handling generic central cameras. Our key idea is based on a local planarity assumption, that we model through constraints between surface normals and ray directions. Compared to existing methods, our approach more accurately approximates the relation between depth and surface normals, achieves state-of-the-art results on the standard normal integration benchmark, and is the first to directly handle generic central camera models.

Personal mobile robotic assistants are expected to find wide applications in industry and healthcare. For example, people with limited mobility can benefit from robots helping with daily tasks, or construction workers can have robots perform precision monitoring tasks on-site. However, manually steering a robot while in motion requires significant concentration from the operator, especially in tight or crowded spaces. This reduces walking speed, and the constant need for vigilance increases fatigue and, thus, the risk of accidents. This work presents a virtual leash with which a robot can naturally follow an operator. We use a sensor fusion based on a custom-built RF transponder, RGB cameras, and a LiDAR. In addition, we customize a local avoidance planner for legged platforms, which enables us to navigate dynamic and narrow environments. We successfully validate on the ANYmal platform [1] the robustness and performance of our entire pipeline in real-world experiments. The video is available at: obstacle-avoidant-leader-following.

Human guidance has emerged as a powerful tool for enhancing reinforcement learning (RL). However, conventional forms of guidance such as demonstrations or binary scalar feedback can be challenging to collect or have low information content, motivating the exploration of other forms of human input. Among these, relative feedback (i.e., feedback on how to improve an action, such as "more to the left") offers a good balance between usability and information richness. Previous research has shown that relative feedback can be used to enhance policy search methods. However, these efforts have been limited to specific policy classes and use feedback inefficiently. In this work, we introduce a novel method to learn from relative feedback and combine it with off-policy reinforcement learning. Through evaluations on two sparse-reward tasks, we demonstrate our method can be used to improve the sample efficiency of reinforcement learning by guiding its exploration process. Additionally, we show it can adapt a policy to changes in the environment or the user’s preferences. Finally, we demonstrate real-world applicability by employing our approach to learn a navigation policy in a sparse reward setting.

Behavior Trees (BTs) were first conceived in the computer games industry as a tool to model agent behavior, but they received interest also in the robotics community as an alternative policy design to Finite State Machines (FSMs). The advantages of BTs over FSMs had been highlighted in many works, but there is no thorough practical comparison of the two designs. Such a comparison is particularly relevant in the robotic industry, where FSMs have been the state-of-the-art policy representation for robot control for many years. In this work we shed light on this matter by comparing how BTs and FSMs behave when controlling a robot in a mobile manipulation task. The comparison is made in terms of reactivity, modularity, readability, and design. We propose metrics for each of these properties, being aware that while some are tangible and objective, others are more subjective and implementation dependent. The practical comparison is performed in a simulation environment with validation on a real robot. We find that although the robot’s behavior during task solving is independent on the policy representation, maintaining a BT rather than an FSM becomes easier as the task increases in complexity.

Recent years have seen a steady rise in the abilities of wheeled-legged balancing robots. Yet, their use is still severely restricted by the lack of efficient control algorithms for overcoming obstacles such as stairs. We take a considerable step towards closing this gap by presenting a fast trajectory optimizer for generating trajectories over a large class of challenging terrains. By limiting the underlying modeling to the planar, nonlinear rigid-body dynamics and subdividing the terrain into contact-phases, a tractable nonlinear programming problem is obtained. The model explicitly accounts for contact switches and impacts, traction limits, and actuation bounds. By introducing an arc-length-related parametrization, the trajectories are rendered inherently contact constraint-consistent. We apply our method to the specific case of the wheeled bipedal robot Ascento, for which we derive closed-form expressions of the dynamics equations, including the kinematic loops. To track the trajectories, we propose a simple LQR-based controller. The approach is validated in real-world experiments where we show the execution of trajectories for traversing steps, driving up ramps, jumping, standing up, and driving up entire stairways. To the authors' best knowledge, enabling the latter by means of trajectory optimization is a novelty for wheeled-legged robots.

By combining rotary- and fixed-wing flight, hybrid uncrewed aerial vehicles (H-UAVs) can uniquely address missions combining long-range aerial transport and precise ground-relative tasks, such as the placement or retrieval of payloads. However, to leverage their full maneuverability, first, the fundamentally different operating modes of rotary- and fixed-wing vehicles need to be unified and second, the system be controlled precisely despite complex aerodynamic effects. This work presents a general and lightweight, cascaded control formulation for such versatile and accurate operation of H-UAVs. First, a novel guidance law unifies ground- and air-relative position control modes typical for the individual flight regimes. Second, we formulate a jerk-level feedback-linearization to accurately track the guidance outputs despite model errors and disturbances. In extensive real flight tests with a tiltwing H-UAV, we demonstrate the versatile allocation of (hybrid) flight states and the overall accuracy enabled by the control system. Position errors remain below 0.5 m (one quarter of the wingspan) in the full flight envelope, including accelerated maneuvers up to 10 ms2and gusting wind reaching 12 m/s. Finally, the control system demonstrates exploiting hybrid flight for transport-related missions with a precise, in-flight pickup of a payload.

LiDAR Upsampling is a challenging task for the perception systems of robots and autonomous vehicles, due to the sparse and irregular structure of large-scale scene contexts. Recent works propose to solve this problem by converting LiDAR data from 3D Euclidean space into an image super-resolution problem in 2D image space. Although their methods can generate high-resolution range images with fine-grained details, the resulting 3D point clouds often blur out details and predict invalid points. In this paper, we propose TULIP, a new method to reconstruct high-resolution LiDAR point clouds from low-resolution LiDAR input. We also follow a range image-based approach but specifically modify the patch and window geometries of a Swin- Transformer-based network to better fit the characteristics of range images. We conducted several experiments on three public real-world and simulated datasets. TULIP outperforms state-of-the-art methods in all relevant metrics and generates robust and more realistic point clouds than prior works. The code is available at https://github.com/ethz-asl/TULIP.git.

Detecting air flows caused by phenomena such as heat convection is valuable in multiple scenarios, including leak identification and locating thermal updrafts for extending UAV flight duration. Unfortunately, the heat signature of these flows is often too subtle to be seen by a thermal camera. While convection also leads to fluctuations in air density and hence causes so-called schlieren – intensity and color variations in images – existing techniques such as Background-oriented schlieren (BOS) allow detecting them only against a known background and from a static camera, making these approaches unsuitable for moving vehicles. In this work we demonstrate the feasibility of visualizing air movement by predicting the corresponding schlieren-induced optical flow from a single greyscale image captured by a moving camera against an unfamiliar background. We first record and label a set of optical flows in an indoor setup using standard BOS techniques. We then train a convolutional neural network (CNN) by applying the previously collected optical flow distortions to a dataset containing a mixture of real and synthetically generated images to predict the two-dimensional optical flow from a single image. Finally, we evaluate our approach on the task of extracting the optical flow caused by schlieren from both a static and moving camera on previously unseen flow patterns and background images.

In recent years, legged and wheeled-legged robots have gained prominence for tasks in environments predominantly created for humans across various domains. One significant challenge faced by many of these robots is their limited capability to navigate stairs, which hampers their functionality in multi-story environments. This study proposes a method aimed at addressing this limitation, employing reinforcement learning to develop a versatile controller applicable to a wide range of robots. In contrast to the conventional velocity-based controllers, our approach builds upon a position-based formulation of the RL task, which we show to be vital for stair climbing. Furthermore, the methodology leverages an asymmetric actor-critic structure, enabling the utilization of privileged information from simulated environments during training while eliminating the reliance on exteroceptive sensors during real-world deployment. Another key feature of the proposed approach is the incorporation of a boolean observation within the controller, enabling the activation or deactivation of a stair-climbing mode. We present our results on different quadrupeds and bipedal robots in simulation and showcase how our method allows the balancing robot Ascento to climb 15cm stairs in the real world, a task that was previously impossible for this robot.

Recently, the utilization of aerial manipulators for performing pushing tasks in non-destructive testing (NDT) applications has seen significant growth. Such operations entail physical interactions between the aerial robotic system and the environment. End-effectors with multiple contact points are often used for placing NDT sensors in contact with a surface to be inspected. Aligning the NDT sensor and the work surface while preserving contact, requires that all available contact points at the end-effector tip are in contact with the work surface. With a standard full-pose controller, attitude errors often occur due to perturbations caused by modeling uncertainties, sensor noise, and environmental uncertainties. Even small attitude errors can cause a loss of contact points between the end-effector tip and the work surface. To preserve full alignment amidst these uncertainties, we propose a control strategy which selectively deactivates angular motion control and enables direct force control in specific directions. In particular, we derive two essential conditions to be met, such that the robot can passively align with flat work surfaces achieving full alignment through the rotation along non-actively controlled axes. Additionally, these conditions serve as hardware design and control guidelines for effectively integrating the proposed control method for practical usage. Real world experiments are conducted to validate both the control design and the guidelines.

Dense, volumetric maps are essential to enable robot navigation and interaction with the environment. To achieve low latency, dense maps are typically computed onboard the robot, often on computationally constrained hardware. Previous works leave a gap between CPU-based systems for robotic mapping which, due to computation constraints, limit map resolution or scale, and GPU-based reconstruction systems which omit features that are critical to robotic path planning, such as computation of the Euclidean Signed Distance Field (ESDF). We introduce a library, nvblox, that aims to fill this gap, by GPU-accelerating robotic volumetric mapping. Nvblox delivers a significant performance improvement over the state of the art, achieving up to a 177× speed-up in surface reconstruction, and up to a 31× improvement in distance field computation, and is available open-source1.

We present COIN-LIO, a LiDAR Inertial Odometry pipeline that tightly couples information from LiDAR intensity with geometry-based point cloud registration. The focus of our work is to improve the robustness of LiDAR-inertial odometry in geometrically degenerate scenarios, like tunnels or flat fields. We project LiDAR intensity returns into an image, and present a novel image processing pipeline that produces filtered images with improved brightness consistency within the image as well as across different scenes. We effectively leverage intensity as an additional modality, using our new feature selection scheme that detects uninformative directions in the point cloud registration and explicitly selects patches with complementary image information. Photometric error minimization in the image patches is then fused with inertial measurements and point-to-plane registration in an iterated Extended Kalman Filter. The proposed approach improves accuracy and robustness on a public dataset. We additionally publish a new dataset, that captures five real-world environments in challenging, geometrically degenerate scenes. By using the additional photometric information, our approach shows drastically improved robustness against geometric degeneracy in environments where all compared baseline approaches fail.

Fast and reliable obstacle avoidance is an important task for mobile robots. In this work, we propose an efficient reactive system that provides high-quality obstacle avoidance while running at hundreds of hertz with minimal resource usage. Our approach combines wavemap, a hierarchical volumetric map representation, with a novel hierarchical and parallelizable obstacle avoidance algorithm formulated through Riemannian Motion Policies (RMP). Leveraging multi-resolution obstacle avoidance policies, the proposed navigation system facilitates precise, low-latency (36ms), and extremely efficient obstacle avoidance with a very large perceptive radius (30m). We perform extensive statistical evaluations on indoor and outdoor maps, verifying that the proposed system compares favorably to fixed-resolution RMP variants and CHOMP. Finally, the RMP formulation allows the seamless fusion of obstacle avoidance with additional objectives, such as goal-seeking, to obtain a fully-fledged navigation system that is versatile and robust. We deploy the system on a Micro Aerial Vehicle and show how it navigates through an indoor obstacle course. Our complete implementation, called waverider, is made available as open source1.

We propose an interpretable framework for reading analog gauges that is deployable on real world robotic systems. Our framework splits the reading task into distinct steps, such that we can detect potential failures at each step. Our system needs no prior knowledge of the type of gauge or the range of the scale and is able to extract the units used. We show that our gauge reading algorithm is able to extract readings with a relative reading error of less than 2%.

State-of-the-art approaches for 6D object pose estimation assume the availability of CAD models and require the user to manually set up physically-based rendering (PBR) pipelines for synthetic training data generation. Both factors limit the application of these methods in real-world scenarios. In this work, we present a pipeline that does not require CAD models and allows training a state-of-the-art pose estimator requiring only a small set of real images as input. Our method is based on a NeuS2 [1] object representation, that we learn through a semi-automated procedure based on Structure-from-Motion (SfM) and object-agnostic segmentation. We exploit the novel-view synthesis ability of NeuS2 and simple cut-and-paste augmentation to automatically generate photorealistic object renderings, which we use to train the correspondence-based SurfEmb [2] pose estimator. We evaluate our method on the LINEMOD-Occlusion dataset, extensively studying the impact of its individual components and showing competitive performance with respect to approaches based on CAD models and PBR data. We additionally demonstrate the ease of use and effectiveness of our pipeline on self-collected real-world objects, showing that our method outperforms state-of-the-art CAD-model-free approaches, with better accuracy and robustness to mild occlusions. To allow the robotics community to benefit from this system, we will publicly release it at https://www.github.com/ethz-asl/neusurfemb.

Fixed-wing small uncrewed aerial vehicles (sUAVs) possess the capability to remain airborne for extended durations and traverse vast distances. However, their operation is susceptible to wind conditions, particularly in regions of complex terrain where high wind speeds may push the aircraft beyond its operational limits, potentially raising safety concerns. Moreover, wind impacts the energy required to follow a path, especially in locations where the wind direction and speed are not favorable. Incorporating wind information into mission planning is essential to ensure both safety and energy efficiency. In this paper, we propose a sampling-based planner using the kinematic Dubins aircraft paths with respect to the ground, to plan energy-efficient paths in non-uniform wind fields. We study the characteristics of the planner with synthetic and real-world wind data and compare its performance against baseline cost and path formulations. We demonstrate that the energy-optimized planner effectively utilizes updrafts to minimize energy consumption, albeit at the expense of increased travel time. The ground-relative path formulation facilitates the generation of safe trajectories onboard sUAVs within reasonable computational timeframes.

Estimating the pose of objects through vision is essential to make robotic platforms interact with the environment. Yet, it presents many challenges, often related to the lack of flexibility and generalizability of state-of-the-art solutions. Diffusion models are a cutting-edge neural architecture transforming 2D and 3D computer vision, outlining remarkable performances in zero-shot novel-view synthesis. Such a use case is particularly intriguing for reconstructing 3D objects. However, localizing objects in unstructured environments is rather unexplored. To this end, this work presents Zero123-6D, the first work to demonstrate the utility of Diffusion Model-based novel-view-synthesizers in enhancing RGB 6D pose estimation at category-level, by integrating them with feature extraction techniques. Novel View Synthesis allows to obtain a coarse pose that is refined through an online optimization method introduced in this work to deal with intra-category geometric differences. In such a way, the outlined method shows reduction in data requirements, removal of the necessity of depth information in zero-shot category-level 6D pose estimation task, and increased performance, quantitatively demonstrated through experiments on the CO3D dataset.

Task and motion planning (TAMP) is a promising approach for efficient long-horizon manipulation planning, which is a prerequisite for being able to deploy manipulation systems in human-centered environments at scale. TAMP systems often rely on so-called predicates to abstractly describe the world. Today, predicates and their groundings are often hand-engineered. Furthermore, robot action parameterizations required to fulfill desired predicates are typically discovered by sampling naively or using oracles (again hand-engineered). We aim to automate predicate discovery and grounding with a system that learns to classify the state of predicates in a set of scenes while concurrently learning to generate scene configurations that fulfill the desired predicates. Our results show that high classification accuracies and generation success rates can be achieved with architectures based on multi-layer perceptrons (MLPs) and graph neural networks (GNNs) that are trained on bounding box as well as point cloud-based features in a Generative Adversarial Network (GAN)-inspired fashion, decisively outperforming both decision tree and uniform sampler baselines. The integration of our framework into a TAMP system demonstrates its positive impact on solving mobile manipulation tasks. A reference implementation of our method and data are available at https://github.com/ethzasl/predicate_learning.

Autonomous mobile robots are an increasingly integral part of modern factory and warehouse operations. Obstacle detection, avoidance and path planning are critical safety-relevant tasks, which are often solved using expensive LiDAR sensors and depth cameras. We propose to use cost-effective low-resolution ranging sensors, such as ultrasonic and infrared time-of-flight sensors by developing VIRUS-NeRF - Vision, InfraRed, and UltraSonic based Neural Radiance Fields.Building upon Instant Neural Graphics Primitives with a Multiresolution Hash Encoding (Instant-NGP), VIRUS-NeRF incorporates depth measurements from ultrasonic and infrared sensors and utilizes them to update the occupancy grid used for ray marching. Experimental evaluation in 2D demonstrates that VIRUS-NeRF achieves comparable mapping performance to LiDAR point clouds regarding coverage. Notably, in small environments, its accuracy aligns with that of LiDAR measurements, while in larger ones, it is bounded by the utilized ultrasonic sensors. An in-depth ablation study reveals that adding ultrasonic and infrared sensors is highly effective when dealing with sparse data and low view variation. Further, the proposed occupancy grid of VIRUS-NeRF improves the mapping capabilities and increases the training speed by 46% compared to Instant-NGP. Overall, VIRUS-NeRF presents a promising approach for cost-effective local mapping in mobile robotics, with potential applications in safety and navigation tasks. The code can be found at https://github.com/ethz-asl/virus_nerf.

Most object-level mapping systems in use today make use of an upstream learned object instance segmentation model. If we want to teach them about a new object or segmentation class, we need to build a large dataset and retrain the system. To build spatial AI systems that can quickly be taught about new objects, we need to effectively solve the problem of single-shot object detection, instance segmentation and re-identification. So far there is neither a method fulfilling all of these requirements in unison nor a benchmark that could be used to test such a method. Addressing this, we propose ISAR, a benchmark and baseline method for single- and few-shot object Instance Segmentation And Re-identification, in an effort to accelerate the development of algorithms that can robustly detect, segment, and reidentify objects from a single or a few sparse training examples. We provide a semi-synthetic dataset of video sequences with ground-truth semantic annotations, a standardized evaluation pipeline, and a baseline method. Our benchmark aligns with the emerging research trend of unifying Multi-Object Tracking, Video Object Segmentation, and Re-identification.

This article surveys recent progress and discusses future opportunities for simultaneous localization and mapping (SLAM) in extreme underground environments. SLAM in subterranean environments, from tunnels, caves, and man-made underground structures on Earth, to lava tubes on Mars, is a key enabler for a range of applications, such as planetary exploration, search and rescue, disaster response, and automated mining, among others. SLAM in underground environments has recently received substantial attention, thanks to theDARPA Subterranean (SubT) Challenge, a global robotics competition aimed at assessing and pushing the state of the art in autonomous robotic exploration and mapping in complex underground environments. This article reports on the state of the art in underground SLAM by discussing different SLAM strategies and results across six teams that participated in the three-year-long SubT competition. In particular, the article has four main goals. First, we review the algorithms, architectures, and systems adopted by the teams; particular emphasis is put on light detection and ranging (LIDAR)-centric SLAM solutions (the go-to approach for virtually all teams in the competition), heterogeneous multirobot operation (including both aerial and ground robots), and real-world underground operation (from the presence of obscurants to the need to handle tight computational constraints). We do not shy away from discussing the “dirty details” behind the different SubT SLAM systems, which are often omitted from technical papers. Second, we discuss the maturity of the field by highlighting what is possible with the current SLAM systems and what we believe is within reach with some good systems engineering. Third, we outline what we believe are fundamental open problems, which are likely to require further research to break through. Finally, we provide a list of open-source SLAM implementations and datasets that have been produced during the SubT challenge and related efforts and constitute a useful resource for researchers and practitioners.

An increasing amount of applications rely on data-driven models that are deployed for perception tasks across a sequence of scenes. Due to the mismatch between training and deployment data, adapting the model on the new scenes is often crucial to obtain good performance. In this work, we study continual multi-scene adaptation for the task of semantic segmentation, assuming that no ground-truth labels are available during deployment and that performance on the previous scenes should be maintained. We propose training a Semantic-NeRF network for each scene by fusing the predictions of a segmentation model and then using the view-consistent rendered semantic labels as pseudo-labels to adapt the model. Through joint training with the segmentation model, the Semantic-NeRF model effectively enables 2D-3D knowledge transfer. Furthermore, due to its compact size, it can be stored in a long-term memory and subsequently used to render data from arbitrary viewpoints to reduce forgetting. We evaluate our approach on Scan-Net, where we outperform both a voxel-based baseline and a state-of-the-art unsupervised domain adaptation method.

Semantic segmentation for robotic systems can enable a wide range of applications, from self-driving cars and augmented reality systems to domestic robots. We argue that a spherical representation is a natural one for egocentric pointclouds. Thus, in this work, we present a novel framework exploiting such a representation of LiDAR pointclouds for the task of semantic segmentation. Our approach is based on a spherical convolutional neural network that can seamlessly handle observations from various sensor systems (e.g., different LiDAR systems) and provides an accurate segmentation of the environment. We operate in two distinct stages: First, we encode the projected input pointclouds to spherical features. Second, we decode and back-project the spherical features to achieve an accurate semantic segmentation of the pointcloud. We evaluate our method with respect to state-of-the-art projection-based semantic segmentation approaches using well-known public datasets. We demonstrate that the spherical representation enables us to provide more accurate segmentation and to have a better generalization to sensors with different field-of-view and number of beams than what was seen during training.

Most recently proposed methods for robotic per-ception are based on deep learning, which require very large datasets to perform well. The accuracy of a learned model is mainly dependent on the data distribution it was trained on. Thus for deploying such models, it is crucial to use training data belonging to the robot's environment. However, collecting and labeling data is a significant bottleneck, necessitating efficient data collection and labeling pipelines. This paper presents a method to compute high-quality object segmentation maps for RGB-D video sequences using minimal human labeling effort. We leverage the density learned by a Neural Radiance Field (NeRF) to infer the geometry of the scene, which we use to compute dense segmentation maps using a single 3D bounding box provided by a user. We study the accuracy of the computed segmentation maps and present a way to generate additional synthetic training examples observing the scene from novel viewpoints using the learned radiance fields. Our results show that our method is able to compute accurate segmentation maps, outperforming baseline and state-of-the-art methods. We also show that using the synthetic training examples improves performance on a downstream object detection task.

This work presents the mechanical design and control of a novel small-size and lightweight Micro Aerial Vehicle (MAV) for aerial manipulation. To our knowledge, with a total take-off mass of only 2.0 kg, the proposed system is the most lightweight Aerial Manipulator (AM) that has 8-DOF independently controllable: 5 for the aerial platform and 3 for the articulated arm. We designed the robot to be fully-actuated in the body forward direction. This allows independent pitching and instantaneous force generation, improving the platform's performance during physical interaction. The robotic arm is an origami delta manipulator driven by three servomotors, enabling active motion compensation at the end-effector. Its composite multimaterial links help reduce the weight, while their flexibility allow for compliant aerial interaction with the environment. In particular, the arm's stiffness can be changed according to its configuration. We provide an in depth discussion of the system design and characterize the stiffness of the delta arm. A control architecture to deal with the platform's overactuation while exploiting the delta arm is presented. Its capabilities are experimentally illustrated both in free flight and physical interaction, highlighting advantages and disadvantages of the origami's folding mechanism.

In this paper we provide a practical demonstration of how the modularity in a Behavior Tree (BT) decreases the effort in programming a robot task when compared to a Finite State Machine (FSM). In recent years the way to represent a task plan to control an autonomous agent has been shifting from the standard FSM towards BTs. Many works in the literature have highlighted and proven the benefits of such design compared to standard approaches, especially in terms of modularity, reactivity and human readability. However, these works have often failed in providing a tangible comparison in the implementation of those policies and the programming effort required to modify them. This is a relevant aspect in many robotic applications, where the design choice is dictated both by the robustness of the policy and by the time required to program it. In this work, we compare backward chained BTs with a fault-tolerant design of FSMs by evaluating the cost to modify them. We validate the analysis with a set of experiments in a simulation environment where a mobile manipulator solves an item fetching task.

Small uncrewed aerial systems (sUASs) are useful tools for 3D reconstruction due to their speed, ease of use, and ability to access high-utility viewpoints. Today, most aerial survey approaches generate a preplanned coverage pattern assuming a planar target region. However, this is inefficient since it results in superfluous overlap and suboptimal viewing angles and does not utilize the entire flight envelope. In this work, we propose active path planning for photogrammetric reconstruction. Our main contribution is a view utility function based on Fisher information approximating the offline reconstruction uncertainty. The metric enables online path planning to make in-flight decisions to collect geometrically informative image data in complex terrain. We evaluate our approach in a photorealistic simulation. A viewpoint selection study shows that our metric leads to faster and more precise reconstruction than state-of-the-art active planning metrics and adapts to different camera resolutions. Comparing our online planning approach to an ordinary fixed-wing aerial survey yields 3.2 × faster coverage of 16 ha undulated terrain without sacrificing precision.

Numerous applications require robots to operate in environments shared with other agents, such as humans or other robots. However, such shared scenes are typically subject to different kinds of long-term semantic scene changes. The ability to model and predict such changes is thus crucial for robot autonomy. In this work, we formalize the task of semantic scene variability estimation and identify three main varieties of semantic scene change: changes in the position of an object, its semantic state, or the composition of a scene as a whole. To represent this variability, we propose the Variable Scene Graph (VSG), which augments existing 3D Scene Graph (SG) representations with the variability attribute, representing the likelihood of discrete long-term change events. We present a novel method, DeltaVSG, to estimate the variability of VSGs in a supervised fashion. We evaluate our method on the 3RScan long-term dataset, showing notable improvements in this novel task over existing approaches. Our method DeltaVsgachieves an accuracy of 77.1% and a recall of 72.3%, often mimicking human intuition about how indoor scenes change over time. We further show the utility of VSG prediction in the task of active robotic change detection, speeding up task completion by 66.0% compared to a scene-change-unaware planner. We make our code available as open-source.

This paper presents a novel method for using Riemannian Motion Policies on volumetric maps, shown in the example of obstacle avoidance for Micro Aerial Vehicles (MAVs), Today, most robotic obstacle avoidance algorithms rely on sampling or optimization-based planners with volumetric maps. However, they are computationally expensive and often have inflexible monolithic architectures. Riemannian Motion Policies are a modular, parallelizable, and efficient navigation alternative but are challenging to use with the widely used voxel-based environment representations. We propose using GPU raycasting and tens of thousands of concurrent policies to provide direct obstacle avoidance using Riemannian Motion Policies in voxelized maps without needing map smoothing or pre-processing. Additionally, we present how the same method can directly plan on LiDAR scans without any intermediate map. We show how this reactive approach compares favorably to traditional planning methods and can evaluate up to 200 million rays per second. We demonstrate the planner successfully on a real MAV for static and dynamic obstacles. The presented planner is made available as an open-source package11https://github.com/ethz-asl/reactive_avoidance.

Micro aerial vehicles (MAVs) hold the potential for performing autonomous and contactless land surveys for the detection of landmines and explosive remnants of war (ERW). Metal detectors are the standard detection tool but must be operated close to and parallel to the terrain. A successful combination of MAVs with metal detectors has not been presented yet, as it requires advanced flight capabilities. To this end, we present an autonomous system to survey challenging undulated terrain using a metal detector mounted on a 5 degrees of freedom (DOF) MAV. Based on an online estimate of the terrain, our receding-horizon planner efficiently covers the area, aligning the detector to the surface while considering the kinematic and visibility constraints of the platform. As the survey requires resilient and accurate localization in diverse terrain, we also propose a factor graph-based online fusion of GNSS, IMU, and LiDAR measurements. We validate the robustness of the solution to individual sensor degeneracy by flying under the canopy of trees and over featureless fields. A simulated ablation study shows that the proposed planner reduces coverage duration and improves trajectory smoothness. Real-world flight experiments showcase autonomous mapping of buried metallic objects in undulated and obstructed terrain.

Hybrid unmanned aerial vehicles (H-UAVs) are highly versatile platforms with the ability to transition between rotary- and fixed-wing flight. However, their (aero)dynamics tend to be highly nonlinear which increases the risk of introducing safety-critical modeling errors in a controller. Designing a safe, yet not too cautious controller, requires a credible model which provides accurate dynamics uncertainty quantification. We present a data-efficient, probabilistic semi-parametric dynamics modeling approach that allows for online, filter-based inference. The proposed model leverages prior knowledge using a nominal parametric model, and combines it with residuals in the form of sparse Gaussian processes to account for possibly unmodeled forces and moments. Uncertain nominal and residual parameters are jointly estimated using Bayesian filtering. The resulting model accuracy and the reliability of its predicted uncertainty are analyzed for both a simulated and a real example, where we learn the 6DoF nonlinear dynamics of a tiltwing H-UAV from a few minutes of flight data. Compared to a residual-free nominal model, the proposed semi-parametric approach provides increased model accuracy in relevant parts of the flight envelope and substantially higher credibility overall.

Interactions with articulated objects are a challenging but important task for mobile robots. To tackle this challenge, we propose a novel closed-loop control pipeline, which integrates manipulation priors from affordance estimation with sampling-based whole-body control. We introduce the concept of agent-aware affordances which fully reflect the agent's capabilities and embodiment and we show that they outperform their state-of-the-art counterparts which are only conditioned on the end-effector geometry. Additionally, closed-loop affordance inference is found to allow the agent to divide a task into multiple non-continuous motions and recover from failure and unexpected states. Finally, the pipeline is able to perform long-horizon mobile manipulation tasks, i.e. opening and closing an oven, in the real world with high success rates (opening: 71%, closing: 72%).

Autonomous vehicles that operate in urban environments shall comply with existing rules and reason about the interactions with other decision-making agents. In this paper, we introduce a decentralized and communication-free interaction-aware motion planner and apply it to Autonomous Surface Vessels (ASVs) in urban canals. We build upon a sampling-based method, namely Model Predictive Path Integral control (MPPI), and employ it to, in each time instance, compute both a collision-free trajectory for the vehicle and a prediction of other agents' trajectories, thus modeling interactions. To improve the method's efficiency in multi-agent scenarios, we introduce a two-stage sample evaluation strategy and define an appropriate cost function to achieve rule compliance. We evaluate this decentralized approach in simulations with multiple vessels in real scenarios extracted from Amsterdam's canals, showing superior performance than a state-of-the-art trajectory optimization framework and robustness when encountering different types of agents.

From construction materials, such as sand or asphalt, to kitchen ingredients, like rice, sugar, or salt; the world is full of granular materials. Despite impressive progress in robotic manipulation of single objects, granular materials remain a challenge due to difficulties in modelling these highly deformable and inhomogeneous materials, which are governed by dynamics that are hard to capture analytically. We argue that despite the high degrees of freedom and the complex underlying dynamics of granular materials, many practical problems that require manipulating them can be solved by leveraging simple models, informative motion priors, and a fast feedback loop. In this work, we show that computational Optimal Transport (OT) can be leveraged to derive informative, robot-agnostic motion priors for transforming a pile of granular materials from a source into a target distribution and generate robot motion plans with a next-best sweep planner that uses a simple material-agnostic sweep model. We plan sweeps directly on a height map representation of the material distribution and hence avoid a costly particle-level treatment of the problem. We validate our approach with a large set of simulation and hardware experiments that demonstrate several complex shaping tasks, including gathering, separating, and writing letters with different types of granular materials.

Recently, groundbreaking results have been presented on open-vocabulary semantic image segmentation. Such methods segment each pixel in an image into arbitrary categories provided at run-time in the form of text prompts, as opposed to a fixed set of classes defined at training time. In this work, we present a zero-shot volumetric open-vocabulary semantic scene segmentation method. Our method builds on the insight that we can fuse image features from a vision-language model into a neural implicit representation. We show that the resulting feature field can be segmented into different classes by assigning points to natural language text prompts. The implicit volumetric representation enables us to segment the scene both in 3D and 2D by rendering feature maps from any given viewpoint of the scene. We show that our method works on noisy real-world data and can run in real-time on live sensor data dynamically adjusting to text prompts. We also present quantitative comparisons on the ScanNet dataset.

Methods have recently been proposed that densely segment 3D volumes into classes using only color images and expert supervision in the form of sparse semantically annotated pixels. While impressive, these methods still require a relatively large amount of supervision and segmenting an object can take several minutes in practice. Such systems typically only optimize the representation on the scene they are fitting, without leveraging prior information from previously seen images. In this paper, we propose to use features extracted with models pre-trained on large existing datasets to improve segmentation performance on novel scenes. We bake this feature representation into a Neural Radiance Field (NeRF) by volu-metrically rendering feature maps and supervising on features extracted from each input image. We show that by baking this representation into the NeRF, we make the subsequent classification task much easier. Our experiments show that our method achieves higher segmentation accuracy with fewer semantic annotations than existing methods over a wide range of scenes.

Reliable obstacle detection on railways could help prevent collisions that result in injuries and potentially damage or derail the train. Unfortunately, generic object detectors do not have enough classes to account for all possible scenarios, and datasets featuring objects on railways are challenging to obtain. We propose utilizing a shallow network to learn railway segmentation from normal railway images. The limited receptive field of the network prevents overconfident predictions and allows the network to focus on the locally very distinct and repetitive patterns of the railway environment. Additionally, we explore the controlled inclusion of global information by learning to hallucinate obstacle-free images. We evaluate our method on a custom dataset featuring railway images with artificially augmented obstacles. Our proposed method outperforms other learning-based baseline methods.

The field of aerial manipulation has seen rapid advances, transitioning from push-and-slide tasks to interaction with articulated objects. The motion trajectory of these complex actions is usually hand-crafted or a result of online optimization methods like Model Predictive Control (MPC) or Model Predictive Path Integral (MPPI) control. However, these methods rely on heuristics or model simplifications to efficiently run on onboard hardware, limiting their robustness, and making them sensitive to disturbances and differences between the real environment and its model. In this work, we propose a Reinforcement Learning (RL) approach to learn reactive motion behaviors for a manipulation task while producing policies that are robust to disturbances and modeling errors. Specifically, we train a policy to perform a door-opening task with an Omnidirectional Micro Aerial Vehicle (OMAV). The policy is trained in a physics simulator and shown in the real world, where it is able to generalize also to door closing tasks never seen in training. We also compare our method against a state-of-the-art MPPI solution in simulation, showing a considerable increase in robustness and speed.

Drilling, grinding, and setting anchors on vertical walls are fundamental processes in everyday construction work. Manually doing these works is error-prone, potentially dangerous, and elaborate at height. Today, heavy mobile ground robots can perform automatic power tool work. However, aerial vehicles could be deployed in untraversable environments and reach inaccessible places. Existing drone designs do not provide the large forces, payload, and high precision required for using power tools. This work presents the first aerial robot design to perform versatile manipulation tasks on vertical concrete walls with continuous forces of up to 150 N. The platform combines a quadrotor with active suction cups for perching on walls and a lightweight, tiltable linear tool table. This combination minimizes weight using the propulsion system for flying, surface alignment, and feed during manipulation and allows precise positioning of the power tool. We evaluate our design in a concrete drilling application - a challenging construction process that requires high forces, accuracy, and precision. In 30 trials, our design can accurately pinpoint a target position despite perching imprecision. Nine visually guided drilling experiments demonstrate a drilling precision of 6 mm without further automation. Aside from drilling, we also demonstrate the versatility of the design by setting an anchor into concrete.

Autonomous pedestrian-aware navigation in shared human-robot environments is a challenging problem. Here we consider a common situation in which a large crowd of pedestrians moves together in a limited space. Traditional planners struggle to find collision-free paths in such situations since the free space is limited and always changing. To solve this problem, we proposed a flow map-based RRT* method (FM-RRT*) containing a velocity layer and a minimally-intrusive layer. The proposed method models the velocity of the pedestrian flow and the area where the robot is less invasive to pedestrians. Furthermore, we propose an adaptive bias sampling, which drives the robot considering relative velocity, or minimal intrusion, according to the pedestrian flow. The evaluation is conducted in the Crowdbot Challenge simulator. The results show that our method can find a feasible path considering collision risk while simultaneously avoiding intrusive human movement.

In this article, we investigate and deploy sampling-based control techniques for the challenging task of the mobile manipulation of articulated objects. By their nature, manipulation tasks necessitate environment interactions, which require the handling of nondifferentiable switching contact dynamics. These dynamics represent a strong limitation for traditional gradient-based optimization methods, such as model-predictive control and differential dynamic programming, which often rely on heuristics for trajectory generation.Sampling-basedtechniques alleviate these constraints but do not ensure robots' stability and input/state constraints either. On the other hand, real-world applications in human environments require safety and robustness to unexpected events. For this reason, we propose a novel framework for safe robotic manipulation of movable articulated objects. The framework combines sampling-based control together withcontrol barrier functionsandpassivity theorythat, thanks to formal stability guarantees, enhance the safety and robustness of the method. We also provide the practical insights that enable robust deployment of stochastic control using a conventional central processing unit. We deploy the algorithm on a ten-degree-of-freedom mobile manipulator robot. Finally, we open source our generic and multithreaded implementation.

This article tackles the problem of active planning to achieve cooperative localization for multirobot systems under measurement uncertainty in GNSS-limited scenarios. Specifically, we address the issue of accurately predicting the probability of a future connection between two robots equipped with range-based measurement devices. Due to the limited range of the equipped sensors, edges in the network connection topology will be created or destroyed as the robots move with respect to one another. Accurately predicting the future existence of an edge, given imperfect state estimation and noisy actuation, is therefore a challenging task. An adaptive power series expansion (or APSE) algorithm is developed based on current estimates and control candidates. Such an algorithm applies the power series expansion formula of the quadratic positive form in a normal distribution. Finite-term approximation is made to realize the computational tractability. Further analyses are presented to show that the truncation error in the finite-term approximation can be theoretically reduced to a desired threshold by adaptively choosing the summation degree of the power series. Several sufficient conditions are rigorously derived as the selection principles. Finally, extensive simulation results and comparisons, with respect to both single and multirobot cases, validate that a formally computed and therefore more accurate probability of future topology can help improve the performance of active planning under uncertainty.

Creating computer vision datasets requires careful planning and lots of time and effort. In robotics research, we often have to use standardized objects, such as the YCB object set, for tasks such as object tracking, pose estimation, grasping and manipulation, as there are datasets and pre-learned methods available for these objects. This limits the impact of our research since learning-based computer vision methods can only be used in scenarios that are supported by existing datasets. In this work, we present a full object keypoint tracking toolkit, encompassing the entire process from data collection, labeling, model learning and evaluation. We present a semi-automatic way of collecting and labeling datasets using a wrist mounted camera on a standard robotic arm. Using our toolkit and method, we are able to obtain a working 3D object keypoint detector and go through the whole process of data collection, annotation and learning in just a couple hours of active time.

Bilateral teleoperation offers an intriguing solution towards shared autonomy with aerial vehicles in contact-based inspection and manipulation tasks. Omnidirectional aerial robots allow for full pose operations, making them particularly attractive in such tasks. Naturally, the question arises whether standard bilateral teleoperation methodologies are suitable for use with these vehicles. In this work, a fully decoupled 6DoF bilateral teleoperation framework for aerial physical interaction is designed and tested for the first time. The method is based on the well established rate control, recentering and interaction force feedback policy. However, practical experiments evince the difficulty of performing de-coupled motions in a single axis only. As such, this work shows that the trivial extension of standard methods is insufficient for omnidirectional teleoperation, due to the operator's physical inability to properly decouple all input DoFs. This suggests that further studies on enhanced haptic feedback are necessary.

Visual-inertial sensors have a wide range of applications in robotics. However, good performance often requires different sophisticated motion routines to accurately calibrate camera intrinsics and inter-sensor extrinsics. This work presents a novel formulation to learn a motion policy to be executed on a robot arm for automatic data collection for calibrating intrinsics and extrinsics jointly. Our approach models the calibration process compactly using model-free deep reinforcement learning to derive a policy that guides the motions of a robotic arm holding the sensor to efficiently collect measurements that can be used for both camera intrinsic calibration and camera-IMU extrinsic calibration. Given the current pose and collected measurements, the learned policy generates the subsequent transformation that optimizes sensor calibration accuracy. The evaluations in simulation and on a real robotic system show that our learned policy generates favorable motion trajectories and collects enough measurements efficiently that yield the desired intrinsics and extrinsics with short path lengths. In simulation, we are able to perform calibrations 10× faster than hand-crafted policies, which transfers to a real-world speed up of 3× over a human expert. The code of this work is publicly available at: https://github.com/ethz-asl/Learn-to-Calibrate.

In this paper, we deal with the problem of creating globally consistent pose graphs in a centralized multi-robot SLAM framework. For each robot to act autonomously, individual onboard pose estimates and maps are maintained, which are then communicated to a central server to build an optimized global map. However, inconsistencies between onboard and server estimates can occur due to onboard odometry drift or failure. Furthermore, robots do not benefit from the collaborative map if the server provides no feedback in a computationally tractable and bandwidth-efficient manner. Motivated by this challenge, this paper proposes a novel collaborative mapping framework to enable accurate global mapping among robots and server. In particular, structural differences between robot and server graphs are exploited at different spatial scales using graph spectral analysis to generate necessary constraints for the individual robot pose graphs. The proposed approach is thoroughly analyzed and validated using several real-world multi-robot field deployments where we show improvements of the onboard system up to 90%.

Although manipulation capabilities of aerial robots greatly improved in the last decade, only few works addressed the problem of aerial physical interaction with dynamic environments, proposing strongly model-based approaches. However, in real scenarios, modeling the environment with high accuracy is often impossible. In this work, we aim at developing a control framework for Omnidirectional Micro Aerial Vehicles (OMAVs) for reliable physical interaction tasks with articulated and movable objects in the presence of possibly unforeseen disturbances, and without relying on an accurate model of the environment. Inspired by previous applications of energy-based controllers for physical interaction, we propose a passivity-based impedance and wrench tracking controller in combination with a momentum-based wrench estimator. This is combined with an energytank framework to guarantee the stability of the system, while energy and power flow-based adaptation policies are deployed to enable safe interaction with any type of passive environment. The control framework provides formal guarantees of stability, which is validated in practice considering the challenging task of pushing a cart of unknown mass, moving on a surface of unknown friction, as well as subjected to unknown disturbances. For this scenario, we present, evaluate and discuss three different policies.

This paper presents a novel strategy for autonomous teamed exploration of subterranean environments using legged and aerial robots. Tailored to the fact that subterranean settings, such as cave networks and underground mines, often involve complex, large-scale and multi-branched topologies, while wireless communication within them can be particularly challenging, this work is structured around the synergy of an onboard exploration path planner that allows for resilient long-term autonomy, and a multi-robot coordination framework. The onboard path planner is unified across legged and flying robots and enables navigation in environments with steep slopes, and diverse geometries. When a communication link is available, each robot of the team shares submaps to a centralized location where a multi-robot coordination framework identifies global frontiers of the exploration space to inform each system about where it should re-position to best continue its mission. The strategy is verified through a field deployment inside an underground mine in Switzerland using a legged and a flying robot collectively exploring for 45 min, as well as a longer simulation study with three systems.

For robotic interaction in environments shared with other agents, access to volumetric and semantic maps of the scene is crucial. However, such environments are inevitably subject to long-term changes, which the map needs to account for. We thus propose panoptic multi-TSDFs as a novel representation for multi-resolution volumetric mapping in changing environments. By leveraging high-level information for 3D reconstruction, our proposed system allocates high resolution only where needed. Through reasoning on the object level, semantic consistency over time is achieved. This enables our method to maintain up-to-date reconstructions with high accuracy while improving coverage by incorporating previous data. We show in thorough experimental evaluation that our map can be efficiently constructed, maintained, and queried during online operation, and that the presented approach can operate robustly on real depth sensors using non-optimized panoptic segmentation as input.

Autonomous robots deal with unexpected scenarios in real environments. Given input images, various visual perception tasks can be performed, e.g., semantic segmentation, depth estimation and normal estimation. These different tasks provide rich information for the whole robotic perception system. All tasks have their own characteristics while sharing some latent correlations. However, some of the task predictions may suffer from the unreliability dealing with complex scenes and anomalies. We propose an attention-based failure detection approach by exploiting the correlations among multiple tasks. The proposed framework infers task failures by evaluating the individual prediction, across multiple visual perception tasks for different regions in an image. The formulation of the evaluations is based on an attention network supervised by multi-task uncertainty estimation and their corresponding prediction errors. Our proposed framework11Code link https://github.com/ethz-asl/uncertainty_with_multiple_tasks. generates more accurate estimations of the prediction error for the different task's predictions.

Picking a specific object from clutter is an essential component of many manipulation tasks. Partial observations often require the robot to collect additional views of the scene before attempting a grasp. This paper proposes a closed-loop next-best-view planner that drives exploration based on occluded object parts. By continuously predicting grasps from an up-to-date scene reconstruction, our policy can decide online to finalize a grasp execution or to adapt the robot's trajectory for further exploration. We show that our reactive approach decreases execution times without loss of grasp success rates compared to common camera placements and handles situations where the fixed baselines fail. Video and code are available at https://github.com/ethz-asl/active_grasp.

Autonomously navigating a robot in everyday crowded spaces requires solving complex perception and planning challenges. When using only monocular image sensor data as input, classical two-dimensional planning approaches cannot be used. While images present a significant challenge when it comes to perception and planning, they also allow capturing potentially important details, such as complex geometry, body movement, and other visual cues. In order to successfully solve the navigation task from only images, algorithms must be able to model the scene and its dynamics using only this channel of information. We investigate whether the world model concept, which has shown state-of-the-art results for modeling and learning policies in Atari games as well as promising results in 2D LiDAR-based crowd navigation, can also be applied to the camera-based navigation problem. To this end, we create simulated environments where a robot must navigate past static and moving humans without colliding in order to reach its goal. We find that state-of-the-art methods are able to achieve success in solving the navigation problem, and can generate dream-like predictions of future image-sequences which show consistent geometry and moving persons. We are also able to show that policy performance in our high-fidelity sim2real simulation scenario transfers to the real world by testing the policy on a real robot. We make our simulator, models and experiments available at https://github.com/danieldugas/NavDreams.

Autonomous navigation among people is a com-plex problem that also exhibits considerable variation depending on the type of environment and people involved. Here we consider navigation among crowds that exhibit flow-like behavior like people moving through a train station. We propose a novel pseudo-fluid model of crowd flow for such problems. These have an intuitive physical interpretation and do not require much tuning. We further formalize an observation model to infer flow properties from discrete sensor observations, including support for partial observability, and pair it with a flow-aware planner. We demonstrate the potential of the approach in simulated navigation scenarios. We achieve state of the art results on the CrowdBot navigation benchmark, and also compare favorably against a standard ROS planner on a partially observable environment, demonstrating that the flow-aware planner successfully estimates and plans around counter-flows in the crowd in real time. We conclude that flow-based planning shows great promise for crowded environments that may exhibit such flow-like behavior.

Visual localization is an important task for many robotic and augmented reality applications. As localizing within large scale maps can be memory and computationally de-manding, cloud-based localization services are appealing for developers. However, such services raise important privacy concerns for both passive and active users. In particular, some sensitive information might be revealed by an attacker who intercepts data during the data-sharing process. Therefore, the sensitive data in the image should be concealed before it is shared. As a motivating case, we demonstrated the exposure generated by the common feature descriptor SIFT when attempting to recover private content. In this paper, we propose a pipeline to effectively conceal privacy-sensitive image regions from possible attacks to the transmission or localization services, by making use of learning-based image inpainting techniques while preserving, and even boosting, the localization performance. We tested our pipeline with two off-the-shelf localization services based on deep neural networks on the publicly available Oxford Robotcar dataset, showing that the localization performance on our generated private concealed images is on par with the non-private baseline.1

In this work we address the challenging problem of manipulating a flexible link, like a rope, with an aerial robot. Inspired by spraying tasks in construction and maintenance scenarios, we consider the case in which an autonomous end-effector (e.g., a spray nozzle moved by a robot or a human operator) is connected to a fixed point by a rope (e.g., a hose). To avoid collisions between the rope and the environment while the end-effector moves, we propose the use of an aerial robot as a flying companion to properly manipulate the rope away from collisions. The aerial robot is attached to the rope between the end-effector and the fixed point. Assuming no direct control of the end-effector (e.g., when operated by a human), we design a reactive and fast motion planner for the aerial robot. Grounding on the theory of Forced Geometric Fabrics, we design a motion planner that generates trajectories to drive the aerial robot to follow the end-effector, while manipulating the rope to avoid collisions in cluttered environments. To include the complex behavior of the flexible link, we propose a rope model that estimates its real-time state under forces and position-based interactions, as well as collisions with obstacle surfaces. Finally, we evaluate the system behavior and the motion planner performance in simulations, as well as in real-world experiments on an original spray painting application.

We present a novel 3D mapping method leveraging the recent progress in neural implicit representation for 3D reconstruction. Most existing state-of-the-art neural implicit representation methods are limited to object-level reconstructions and can not incrementally perform updates given new data. In this work, we propose a fusion strategy and training pipeline to incrementally build and update neural implicit representations that enable the reconstruction of large scenes from sequential partial observations. By representing an arbitrarily sized scene as a grid of latent codes and performing updates directly in latent space, we show that incrementally built occupancy maps can be obtained in real-time even on a CPU. Compared to traditional approaches such as Truncated Signed Distance Fields (TSDFs), our map representation is significantly more robust in yielding a better scene completeness given noisy inputs. We demonstrate the performance of our approach in thorough experimental validation on real-world datasets with varying degrees of added pose noise.

The inability of state-of-the-art semantic segmentation methods to detect anomaly instances hinders them from being deployed in safety-critical and complex applications, such as autonomous driving. Recent approaches have focused on either leveraging segmentation uncertainty to identify anomalous areas or re-synthesizing the image from the semantic label map to find dissimilarities with the input image. In this work, we demonstrate that these two methodologies contain complementary information and can be combined to produce robust predictions for anomaly segmentation. We present a pixel-wise anomaly detection framework that uses uncertainty maps to improve over existing re-synthesis methods in finding dissimilarities between the input and generated images. Our approach works as a general framework around already trained segmentation networks, which ensures anomaly detection without compromising segmentation accuracy, while significantly out performing all similar methods. Top-2 performance across a range of different anomaly datasets shows the robustness of our approach to handling different anomaly instances.

In this paper, we propose a robust end-to-end multi-modal pipeline for place recognition where the sensor systems can differ from the map building to the query. Our approach operates directly on images and LiDAR scans without requiring any local feature extraction modules. By projecting the sensor data onto the unit sphere, we learn a multi-modal descriptor of partially overlapping scenes using a spherical convolutional neural network. The employed spherical projection model enables the support of arbitrary LiDAR and camera systems readily without losing information. Loop closure candidates are found using a nearest-neighbor lookup in the embedding space. We tackle the problem of correctly identifying the closest place by correlating the candidates’ power spectra, obtaining a confidence value per prospect. Our estimate for the correct place corresponds then to the candidate with the highest confidence. We evaluate our proposal w.r.t. state-of-the-art approaches in place recognition using real-world data acquired using different sensors. Our approach can achieve a recall that is up to 10% and 5% higher than for a LiDAR- and vision-based system, respectively, when the sensor setup differs between model training and deployment. Additionally, our place selection can correctly identify up to 95% matches from the candidate set.

Autonomous exploration allows mobile robots to navigate in initially unknown territories in order to build complete representations of the environments. In many real-life applications, environments often contain dynamic obstacles which can compromise the exploration process by temporarily blocking passages, narrow paths, exits or entrances to other areas yet to be explored. In this work, we formulate a novel exploration strategy capable of explicitly handling dynamic obstacles, thus leading to complete and reliable exploration outcomes in populated environments. We introduce the concept of dynamic frontiers to represent unknown regions at the boundaries with dynamic obstacles together with a cost function which allows the robot to make informed decisions about when to revisit such frontiers. We evaluate the proposed strategy in challenging simulated environments and show that it outperforms a state-of-the-art baseline in these populated scenarios.

Robot navigation is a task where reinforcement learning approaches are still unable to compete with traditional path planning. State-of-the-art methods differ in small ways, and do not all provide reproducible, openly available implementations. This makes comparing methods a challenge. Recent research has shown that unsupervised learning methods can scale impressively, and be leveraged to solve difficult problems. In this work, we design ways in which unsupervised learning can be used to assist reinforcement learning for robot navigation. We train two end-to-end, and 18 unsupervised-learning-based architectures, and compare them, along with existing approaches, in unseen test cases. We demonstrate our approach working on a real life robot. Our results show that unsupervised learning methods are competitive with end-to-end methods. We also highlight the importance of various components such as input representation, predictive unsupervised learning, and latent features. We make all our models publicly available, as well as training and testing environments, and tools1. This release also includes OpenAI-gym-compatible environments designed to emulate the training conditions described by other papers, with as much fidelity as possible. Our hope is that this helps in bringing together the field of RL for robot navigation, and allows meaningful comparisons across state-of-the-art methods.

The ability to safely navigate through complex and cluttered environments is required for a wide range of robotics applications. This paper introduces a framework to compute safe global paths in maps represented as collections of 3D Signed Distance Function (SDF) submaps. Such maps are able to maintain global consistency in spite of odometry drift. However, computationally efficient global path planning in this context remains a challenging problem. We present a planning approach based on pre-computed local graphs, computed in each submap, that are linked to form a global path at planning time. To ensure globally safe paths, planning algorithms make frequent queries to the submap collection, which grows over time as the agent collects observational data. We present an efficient algorithm for performing these queries, through the use of a spatial hash table. We analyze the performance of our proposal extensively in simulation and real-world environments, and compare our approach to state- of-the-art planning approaches designed for monolithic maps, extended to submap-based maps. We show the efficacy of our method at adapting to global map deformations, while significantly reducing the planning time to an average of ~1.2 seconds, a reduction by 90 % compared to classical monolithic approaches.

The ability to simultaneously track and reconstruct multiple objects moving in the scene is of the utmost importance for robotic tasks such as autonomous navigation and interaction. Virtually all of the previous attempts to map multiple dynamic objects have evolved to store individual objects in separate reconstruction volumes and track the relative pose between them. While simple and intuitive, such formulation does not scale well with respect to the number of objects in the scene and introduces the need for an explicit occlusion handling strategy. In contrast, we propose a map representation that allows maintaining a single volume for the entire scene and all the objects therein. To this end, we introduce a novel multi-object TSDF formulation that can encode multiple object surfaces at any given location in the map. In a multiple dynamic object tracking and reconstruction scenario, our representation allows maintaining accurate reconstruction of surfaces even while they become temporarily occluded by other objects moving in their proximity. We evaluate the proposed TSDF++ formulation on a public synthetic dataset and demonstrate its ability to preserve reconstructions of occluded surfaces when compared to the standard TSDF map representation. Code is available at https://github.com/ethz-asl/tsdf-plusplus.

The evaluation of robot capabilities to navigate human crowds is essential to conceive new robots intended to operate in public spaces. This paper initiates the development of a benchmark tool to evaluate such capabilities; our long term vision is to provide the community with a simulation tool that generates virtual crowded environment to test robots, to establish standard scenarios and metrics to evaluate navigation techniques in terms of safety and efficiency, and thus, to install new methods to benchmarking robots’ crowd navigation capabilities. This paper presents the architecture of the simulation tools, introduces first scenarios and evaluation metrics, as well as early results to demonstrate that our solution is relevant to be used as a benchmark tool.

We present a novel hierarchical POMDP framework to solve an object search and delivery task where the agent is given a prior belief about the possible item locations. Solving POMDPs is computationally demanding and, as such, applications have typically been limited to small environments. The proposed hierarchical POMDP framework performs reasoning on multiple spatial scales in order to reduce computation time. The problem is first solved in the top layer of the hierarchy with a coarsely discretized state space. Its solution is refined in the lower layers with increasing resolution. Three different methods for propagating information down the spatial hierarchy are discussed and validated in simulation. We show that a two-layer multi-scale POMDP decreases computation time by an order of magnitude allowing for real-time applications while maintaining high solution quality. For large problems that require three layers to reach the desired resolution, computation time speedups by two orders of magnitude are achieved.

In this paper, we present a model predictive controller for a fully actuated aerial manipulator to track a hybrid force and pose trajectory at the end-effector in an aerial interaction task. A force sensor at the end-effector is used to detect contact and to directly control the interaction force. We propose an approach for automatic transition between three operation modes which reflect the state of contact constraints, including free flight and two modes for force control based on static or dynamic friction at the end-effector. This division into three modes allows for different mode-specific controller tunings to optimize the desired performance throughout an interaction task. Results from flight experiments which combine force, position, and attitude tracking, show the performance of the controller in terms of accuracy and precision. The performance is further benchmarked against a hybrid force/impedance controller.

Highly dynamic environments, with moving objects such as cars or humans, can pose a performance challenge for LiDAR SLAM systems that assume largely static scenes. To overcome this challenge and support the deployment of robots in real world scenarios, we propose a complete solution for a dynamic object aware LiDAR SLAM algorithm. This is achieved by leveraging a real-time capable neural network that can detect dynamic objects, thus allowing our system to deal with them explicitly. To efficiently generate the necessary training data which is key to our approach, we present a novel end-to-end occupancy grid based pipeline that can automatically label a wide variety of arbitrary dynamic objects. Our solution can thus generalize to different environments without the need for expensive manual labeling and at the same time avoids assumptions about the presence of a predefined set of known objects in the scene. Using this technique, we automatically label over 12000 LiDAR scans collected in an urban environment with a large amount of pedestrians and use this data to train a neural network, achieving an average segmentation IoU of 0.82. We show that explicitly dealing with dynamic objects can improve the LiDAR SLAM odometry performance by 39.6% while yielding maps which better represent the environments. A supplementary video1as well as our test data2are available online.

For asymptotically optimal sampling-based path planners such as RRT*, path quality improves as the number of samples added to the motion tree increases. However, each additional sample requires a nearest-neighbor search. Calculating state transition costs can be particularly difficult in cases with complex dynamics such as aerial vehicles in non-isotropic cost fields like wind. Computationally costly nearest neighbor searches increase the time required to add new samples to the search tree, thereby reducing the likelihood of finding low-cost paths in a given computational time. In this paper, we propose the use of a lightweight neural network to approximate nearest neighbor cost calculations. The network approach uses a low-dimensional encoding of the cost space along with a start and goal query pair and returns an estimate of the path cost that can be used for nearest neighbor and path validity estimation. We demonstrate our method for a Dubins airplane model in a 3D wind field and show that the network method achieves equivalent path lengths as an existing iterative solver 32% faster and, when given the same search time, up to 10.8% shorter.

With the advent of powerful, light-weight 3D LiDARs, they have become the hearth of many navigation and SLAM algorithms on various autonomous systems. Pointcloud registration methods working with unstructured pointclouds such as ICP are often computationally expensive or require a good initial guess. Furthermore, 3D feature-based registration methods have never quite reached the robustness of 2D methods in visual SLAM. With the continuously increasing resolution of LiDAR range images, these 2D methods not only become applicable but should exploit the illumination-independent modalities that come with it, such as depth and intensity. In visual SLAM, deep learned 2D features and descriptors perform exceptionally well compared to traditional methods. In this publication, we use a state-of-the-art 2D feature network as a basis for 3D3L, exploiting both intensity and depth of LiDAR range images to extract powerful 3D features. Our results show that these keypoints and descriptors extracted from LiDAR scan images outperform state-of-the-art on different benchmark metrics and allow for robust scan-to-scan alignment as well as global localization.

Model-based controllers on real robots require accurate knowledge of the system dynamics to perform optimally. For complex dynamics, first-principles modeling is not sufficiently precise, and data-driven approaches can be leveraged to learn a statistical model from real experiments. However, the efficient and effective data collection for such a data-driven system on real robots is still an open challenge. This paper introduces an optimization problem formulation to find an informative trajectory that allows for efficient data collection and model learning. We present a sampling-based method that computes an approximation of the trajectory that minimizes the prediction uncertainty of the dynamics model. This trajectory is then executed, collecting the data to update the learned model. We experimentally demonstrate the capabilities of our proposed framework when applied to a complex omnidirectional flying vehicle with tiltable rotors. Using our informative trajectories results in models which outperform models obtained from non-informative trajectory by 13.3% with the same amount of training data. Furthermore, we show that the model learned from informative trajectories generalizes better than the one learned from non-informative trajectories, achieving better tracking performance on different tasks.

This paper presents an online informative path planning approach for active information gathering on three-dimensional surfaces using aerial robots. Most existing works on surface inspection focus on planning a path offline that can provide full coverage of the surface, which inherently assumes the surface information is uniformly distributed hence ignoring potential spatial correlations of the information field. In this paper, we utilize manifold Gaussian processes (mGPs) with geodesic kernel functions for mapping surface information fields and plan informative paths online in a receding horizon manner. Our approach actively plans information-gathering paths based on recent observations that respect dynamic constraints of the vehicle and a total flight time budget. We provide planning results for simulated temperature modeling for simple and complex 3D surface geometries (a cylinder and an aircraft model). We demonstrate that our informative planning method outperforms traditional approaches such as 3D coverage planning and random exploration, both in reconstruction error and information-theoretic metrics. We also show that by taking spatial correlations of the information field into planning using mGPs, the information gathering efficiency is significantly improved.

Localization is an essential task for mobile autonomous robotic systems that want to use pre-existing maps or create new ones in the context of SLAM. Today, many robotic platforms are equipped with high-accuracy 3D LiDAR sensors, which allow a geometric mapping, and cameras able to provide semantic cues of the environment. Segment-based mapping and localization have been applied with great success to 3D point-cloud data, while semantic understanding has been shown to improve localization performance in vision based systems. In this paper we combine both modalities in SemSegMap, extending SegMap into a segment based mapping framework able to also leverage color and semantic data from the environment to improve localization accuracy and robustness. In particular, we present new segmentation and descriptor extraction processes. The segmentation process benefits from additional distance information from color and semantic class consistency resulting in more repeatable segments and more overlap after re-visiting a place. For the descriptor, a tight fusion approach in a deep-learned descriptor extraction network is performed leading to a higher descriptiveness for landmark matching. We demonstrate the advantages of this fusion on multiple simulated and real-world datasets and compare its performance to various baselines. We show that we are able to find 50.9 % more high-accuracy prior-less global localizations compared to SegMap on challenging datasets using very compact maps while also providing accurate full 6 DoF pose estimates in real-time.

Camera anomalies like rain or dust can severely degrade image quality and its related tasks, such as localization and segmentation. In this work we address this important issue by implementing a pre-processing step that can effectively mitigate such artifacts in a real-time fashion, thus supporting the deployment of autonomous systems with limited compute capabilities. We propose a shallow generator with aggregation, trained in an adversarial setting to solve the ill-posed problem of reconstructing the occluded regions. We add an enhancer to further preserve high-frequency details and image colorization. We also produce one of the largest publicly available datasets1to train our architecture and use realistic synthetic raindrops to obtain an improved initialization of the model. We benchmark our framework on existing datasets and on our own images obtaining state-of-the-art results while enabling real-time performance, with up to 40x faster inference time than existing approaches.

Suitable datasets are an integral part of robotics research, especially for training neural networks in robot perception. However, in many domains, suitable real-world data are scarce and cannot be easily obtained. This problem is especially prevalent for unstructured outdoor environments, in particular, planetary ones. Recent advances in photorealistic simulations help researchers to simulate close-to-real data in many domains. Yet, there exists no high-quality synthetic data for planetary exploration tasks. Also, existing simulators lack the fidelity required for generating planetary data, which is inherently less structured than human environments. Synthetic planetary data requires careful modeling and annotation of many different terrain aspect and details, such as textures and distributions of rocks, to become a valuable test-bed for robotics. To fill this gap, we present a novel simulator specifically designed for the needs of planetary robotics visual tasks, but also applicable for other outdoor environments. Our simulator is capable of generating large varieties of (planetary) outdoor scenes with rich generation of meta data, such as multilevel semantic and instance annotations. To demonstrate the wide applicability of this new simulator, we evaluate its performance on typical robotics applications, i.e. semantic segmentation, instance segmentation, and visual SLAM. Our simulator is accessible under https://github.com/DLR-RM/oaisys.

In this paper, we propose a resource-efficient approach to provide an autonomous UAV with an on-board perception method to detect safe, hazard-free landing sites during flights over complex 3D terrain. We aggregate 3D measurements acquired from a sequence of monocular images by a Structure-from-Motion approach into a local, robot-centric, multi-resolution elevation map of the overflown terrain, which fuses depth measurements according to their lateral surface resolution (pixel-footprint) in a probabilistic framework based on the concept of dynamic Level of Detail. Map aggregation only requires depth maps and the associated poses, which are obtained from an on-board Visual Odometry algorithm. An efficient landing site detection method then exploits the features of the underlying multi-resolution map to detect safe landing sites based on slope, roughness, and quality of the reconstructed terrain surface. The evaluation of the performance of the mapping and landing site detection modules are analyzed independently and jointly in simulated and real-world experiments in order to establish the efficacy of the proposed approach.

Many mobile robotic platforms rely on an accurate knowledge of the extrinsic calibration parameters, especially systems performing visual stereo matching. Although a number of accurate stereo camera calibration methods have been developed, which provide good initial “factory” calibrations, the determined parameters can lose their validity over time as the sensors are exposed to environmental conditions and external effects. Thus, on autonomous platforms on-board diagnostic methods for an early detection of the need to repeat calibration procedures have the potential to prevent critical failures of crucial systems, such as state estimation or obstacle detection. In this work, we present a novel data-driven method to estimate the quality of extrinsic calibration and detect discrepancies between the original calibration and the current system state for stereo camera systems. The framework consists of a novel dataset generation pipeline to train CalQNet, a deep convolutional neural network. CalQNet can estimate the extrinsic calibration quality using a new metric that approximates the degree of miscalibration in stereo setups. We show the framework's ability to predict the divergence of a state-of-the-art stereo-visual odometry system following a degraded calibration in two real-world experiments.

Abstract Deep learning has enabled impressive progress in the accuracy of semantic segmentation. Yet, the ability to estimate uncertainty and detect failure is key for safety-critical applications like autonomous driving. Existing uncertainty estimates have mostly been evaluated on simple tasks, and it is unclear whether these methods generalize to more complex scenarios. We present Fishyscapes, the first public benchmark for anomaly detection in a real-world task of semantic segmentation for urban driving. It evaluates pixel-wise uncertainty estimates towards the detection of anomalous objects. We adapt state-of-the-art methods to recent semantic segmentation models and compare uncertainty estimation approaches based on softmax confidence, Bayesian learning, density estimation, image resynthesis, as well as supervised anomaly detection methods. Our results show that anomaly detection is far from solved even for ordinary situations, while our benchmark allows measuring advancements beyond the state-of-the-art. Results, data and submission information can be found at https://fishyscapes.com/ .

This article presents and validates active interaction force control and planning for fully actuated and omnidirectional aerial manipulation platforms, with the goal of aerial contact inspection in unstructured environments. We present a variable axis-selective impedance control which integrates direct force control for intentional interaction, using feedback from an on-board force sensor. The control approach aims to reject disturbances in free flight, while handling unintentional interaction and actively controlling desired interaction forces. A fully actuated and omnidirectional tilt-rotor aerial system is used to show capabilities of the control and planning methods. Experiments demonstrate disturbance rejection, push-and-slide interaction, and force-controlled interaction in different flight orientations. The system is validated as a tool for nondestructive testing of concrete infrastructure, and statistical results of interaction control performance are presented and discussed.

Current research on visual place recognition mostly focuses on aggregating local visual features of an image into a single vector representation. Therefore, high-level information such as the geometric arrangement of the features is typically lost. In this paper, we introduce a novel learning-based approach to place recognition, using RGB-D cameras and line clusters as visual and geometric features. We state the place recognition problem as a problem of recognizing clusters of lines instead of individual patches, thus maintaining structural information. In our work, line clusters are defined as lines that make up individual objects, hence our place recognition approach can be understood as object recognition. 3D line segments are detected in RGB-D images using state-of-the-art techniques. We present a neural network architecture based on the attention mechanism for frame-wise line clustering. A similar neural network is used for the description of these clusters with a compact embedding of 128 floating point numbers, trained with triplet loss on training data obtained from the InteriorNet dataset. We show experiments on a large number of indoor scenes and compare our method with the bag-of-words image-retrieval approach using SIFT and SuperPoint features and the global descriptor NetVLAD. Trained only on synthetic data, our approach generalizes well to real-world data captured with Kinect sensors, while also providing information about the geometric arrangement of instances.

This work presents a method to control omnidirectional micro aerial vehicles (OMAVs) for the tracking of 6-DoF trajectories in free space. A rigid body model based approach is applied in a receding horizon fashion to generate optimal wrench commands that can be constrained to meet limits given by the mechanical design and actuators of the platform. Allocation of optimal actuator commands is performed in a separate step. A disturbance observer estimates forces and torques that may arise from unmodeled dynamics or external disturbances and fuses them into the optimization to achieve offset-free tracking. Experiments on a fully overactuated MAV show the tracking performance and compare it against a classical PD-based controller.

Self-diagnosis and self-repair are some of the key challenges in deploying robotic platforms for long-term real-world applications. One of the issues that can occur to a robot is miscalibration of its sensors due to aging, environmental transients, or external disturbances. Precise calibration lies at the core of a variety of applications, due to the need to accurately perceive the world. However, while a lot of work has focused on calibrating the sensors, not much has been done towards identifying when a sensor needs to be recalibrated. This paper focuses on a data-driven approach to learn the detection of miscalibration in vision sensors, specifically RGB cameras. Our contributions include a proposed miscalibration metric for RGB cameras and a novel semi-synthetic dataset generation pipeline based on this metric. Additionally, by training a deep convolutional neural network, we demonstrate the effectiveness of our pipeline to identify whether a recalibration of the camera’s intrinsic parameters is required or not. The code is available at http://github.com/ethz-asl/camera_miscalib_detection.

Robots require a detailed understanding of the 3D structure of the environment for autonomous navigation and path planning. A popular approach is to represent the environment using metric, dense 3D maps such as 3D occupancy grids. However, in large environments the computational power required for most state-of-the-art 3D dense mapping systems is compromising precision and real-time capability. In this work, we propose a novel mapping method that is able to build and maintain 3D dense representations for large indoor environments using standard CPUs. Topological global representations and 3D dense submaps are maintained as hybrid global map. Submaps are generated for every new visited place. A place (room) is identified as an isolated part of the environment connected to other parts through transit areas (doors). This semantic partitioning of the environment allows for a more efficient mapping and path-planning. We also propose a method for autonomous exploration that directly builds the hybrid representation in real time.We validate the real-time performance of our hybrid system on simulated and real environments regarding mapping and path-planning. The improvement in execution time and memory requirements upholds the contribution of the proposed work.

Object finding in clutter is a skill that requires perception of the environment and in many cases physical interaction. In robotics, interactive perception defines a set of algorithms that leverage actions to improve the perception of the environment, and vice versa use perception to guide the next action. Scene interactions are difficult to model, therefore, most of the current systems use predefined heuristics. This limits their ability to efficiently search for the target object in a complex environment. In order to remove heuristics and the need for explicit models of the interactions, in this work we propose a reinforcement learning based active and interactive perception system for scene exploration and object search. We evaluate our work both in simulated and in real-world experiments using a robotic manipulator equipped with an RGB and a depth camera, and compare our system to two baselines. The results indicate that our approach, trained in simulation only, transfers smoothly to reality and can solve the object finding task efficiently and with more than 88% success rate.

Information gathering algorithms play a key role in unlocking the potential of robots for efficient data collection in a wide range of applications. However, most existing strategies neglect the fundamental problem of the robot pose uncertainty, which is an implicit requirement for creating robust, high-quality maps. To address this issue, we introduce an informative planning framework for active mapping that explicitly accounts for the pose uncertainty in both the mapping and planning tasks. Our strategy exploits a Gaussian Process (GP) model to capture a target environmental field given the uncertainty on its inputs. For planning, we formulate a new utility function that couples the localization and field mapping objectives in GP-based mapping scenarios in a principled way, without relying on manually-tuned parameters. Extensive simulations show that our approach outperforms existing strategies, reducing mean pose uncertainty and map error. We present a proof of concept in an indoor temperature mapping scenario.

Globally localizing in a given map is a crucial ability for robots to perform a wide range of autonomous navigation tasks. This paper presents OneShot - a global localization algorithm that uses only a single 3D LiDAR scan at a time, while outperforming approaches based on integrating a sequence of point clouds. Our approach, which does not require the robot to move, relies on learning-based descriptors of point cloud segments and computes the full 6 degree-of-freedom pose in a map. The segments are extracted from the current LiDAR scan and are matched against a database using the computed descriptors. Candidate matches are then verified with a geometric consistency test. We additionally present a strategy to further improve the performance of the segment descriptors by augmenting them with visual information provided by a camera. For this purpose, a custom-tailored neural network architecture is proposed. We demonstrate that our LiDAR-only approach outperforms a state-of-the-art baseline on a sequence of the KITTI dataset and also evaluate its performance on the challenging NCLT dataset. Finally, we show that fusing in visual information boosts segment retrieval rates by up to 26% compared to LiDAR-only description.

Painting textures on 3D surfaces requires an understanding of the surface geometry, paint flow and paint mixing. This work formulates automated painting as a planning problem and proposes a solution based on a self-supervised learning framework that enables a robot to paint monochromatic non-uniform textures on 3D surfaces. We developed a method that iteratively decides the actions to take based on constant feedback of the painting process. Inspired by recent results, we formulate our solution using a recurrent neural network (RNN) to decide where and what to paint on the surface at each time instant. Specifically, the paint delivery tool's flow rate, orientation and position relative to the surface at each time instant are evaluated. This data can then be processed by a robot's planner of choice for generating a painting mission that can achieve the desired end result. We evaluate the proposed approach by providing qualitative and quantitative results of the different components. Furthermore, we validate the effectiveness of the approach for the application by providing renderings from a paint simulation environment and show how a robot executes the planned painting mission on a generic 3D surface.

This paper presents a method for predicting the probability of future connectivity between mobile robots with range-limited communication. In particular, we focus on its application to active motion planning for cooperative localization (CL). The probability of connection is modeled by the distribution of quadratic forms in random normal variables and is computed by the infinite power series expansion theorem. A finite-term approximation is made to realize the computational feasibility and three more modifications are designed to handle the adverse impacts introduced by the omission of the higher order series terms. On the basis of this algorithm, an active and CL problem with leader-follower architecture is then reformulated into a Markov Decision Process (MDP) with a one-step planning horizon, and the optimal motion strategy is generated by minimizing the expected cost of the MDP. Extensive simulations and comparisons are presented to show the effectiveness and efficiency of both the proposed prediction algorithm and the MDP model.

This paper presents the perception, mapping, and planning pipeline implemented on an autonomous race car. It was developed by the 2019 AMZ driverless team for the Formula Student Germany (FSG) 2019 driverless competition, where it won 1st place overall. The presented solution combines early fusion of camera and LiDAR data, a layered mapping approach, and a planning approach that uses Bayesian filtering to achieve high-speed driving on unknown race tracks while creating accurate maps. We benchmark the method against our team's previous solution, which won FSG 2018, and show improved accuracy when driving at the same speeds. Furthermore, the new pipeline makes it possible to reliably raise the maximum driving speed in unknown environments from 3 m/s to 12 m/s while still mapping with an acceptable RMSE of 0.29 m.

State-of-the-art approaches for robot navigation among humans are typically restricted to planar movement actions. This work addresses the question of whether it can be beneficial to use interaction actions, such as saying, touching, and gesturing, for the sake of allowing robots to navigate in unstructured, crowded environments. To do so, we first identify challenging scenarios to traditional motion planning methods. Based on the hypothesis that the variation in modality for these scenarios calls for significantly different planning policies, we design specific navigation behaviors as interaction planners for actuated, mobile robots. We further propose a high level planning algorithm for multi-behavior navigation, named Interaction Actions for Navigation (IAN). Through both real-world and simulated experiments, we validate the selected behaviors and the high-level planning algorithm, and discuss the impact of our obtained results on our stated assumptions.

Dynamic obstacle avoidance is one crucial component for compliant navigation in crowded environments. In this paper we present a system for accurate and reliable detection and tracking of dynamic objects using noisy point cloud data generated by stereo cameras. Our solution is real-time capable and specifically designed for the deployment on computationally-constrained unmanned ground vehicles. The proposed approach identifies individual objects in the robot's surroundings and classifies them as either static or dynamic. The dynamic objects are labeled as either a person or a generic dynamic object. We then estimate their velocities to generate a 2D occupancy grid that is suitable for performing obstacle avoidance. We evaluate the system in indoor and outdoor scenarios and achieve real-time performance on a consumergrade computer. On our test-dataset, we reach a MOTP of 0.07 ± 0.07m, and a MOTA of 85.3% for the detection and tracking of dynamic objects. We reach a precision of 96.9% for the detection of static objects.

In this paper, we introduce IDOL, an optimization-based framework for IMU-DVS Odometry using Lines. Event cameras, also called Dynamic Vision Sensors (DVSs), generate highly asynchronous streams of events triggered upon illumination changes for each individual pixel. This novel paradigm presents advantages in low illumination conditions and high-speed motions. Nonetheless, this unconventional sensing modality brings new challenges to perform scene reconstruction or motion estimation. The proposed method offers to leverage a continuous-time representation of the inertial readings to associate each event with timely accurate inertial data. The method's front-end extracts event clusters that belong to line segments in the environment whereas the back-end estimates the system's trajectory alongside the lines' 3D position by minimizing point-to-line distances between individual events and the lines' projection in the image space. A novel attraction/repulsion mechanism is presented to accurately estimate the lines' extremities, avoiding their explicit detection in the event data. The proposed method is benchmarked against a state-of-the-art frame-based visual-inertial odometry framework using public datasets. The results show that IDOL performs at the same order of magnitude on most datasets and even shows better orientation estimates. These findings can have a great impact on new algorithms for DVS.

Mobile robots operating in public environments require the ability to navigate among humans and other obstacles in a socially compliant and safe manner. This work presents a combined imitation learning and deep reinforcement learning approach for motion planning in such crowded and cluttered environments. By separately processing information related to static and dynamic objects, we enable our network to learn motion patterns that are tailored to real-world environments. Our model is also designed such that it can handle usual cases in which robots can be equipped with sensor suites that only offer limited field of view. Our model outperforms current state-of-the-art approaches, which is shown in simulated environments containing human-like agents and static obstacles. Additionally, we demonstrate the real-time performance and applicability of our model by successfully navigating a robotic platform through real-world environments.

Visually poor scenarios are one of the main sources of failure in visual localization systems in outdoor environments. To address this challenge, we present MOZARD, a multi-modal localization system for urban outdoor environments using vision and LiDAR. By fusing key point based visual multi-session information with semantic data, an improved localization recall can be achieved across vastly different appearance conditions. In particular we focus on the use of curbstone information because of their broad distribution and reliability within urban environments. We present thorough experimental evaluations on several driving kilometers in challenging urban outdoor environments, analyze the recall and accuracy of our localization system and demonstrate in a case study possible failure cases of each subsystem. We demonstrate that MOZARD is able to bridge scenarios where our previous key point based visual approach, VIZARD, fails, hence yielding an increased recall performance, while a similar localization accuracy of 0.2m is achieved.

LiDAR sensors are crucial in automotive perception for accurate object detection. However, LiDAR data is hard to interpret for humans and consequently time-consuming to label. Whereas camera data is easy interpretable and thus, comparably simpler to label. Within this work we present a transductive transfer learning approach to transfer the knowledge for the object detection task from images to point cloud data. We propose a multi-modal adversarial Auto Encoder architecture which disentangles uni-modal features into two groups: common (transferable) features, and complementary (modality-specific) features. This disentanglement is based on the hypothesis that a set of common features exist. An important point of our framework is that the disentanglement is learned in an unsupervised manner. Furthermore, the results show that only a small amount of multi-modal data is needed to learn the disentanglement, and thus to transfer the knowledge between modalities. As a result we our experiments show that training with 75% less data of the KITTI objects, the classification accuracy achieved is of 71.75%, only 3.12% less than when using the full data set. The implications of these findings can have great impact in perception pipelines based on LIDAR data.

This paper presents a novel design for a large-scale interactive tactile display. Fast dynamic tactile effects are created at high spatial resolution on a flexible screen, using directable nozzles that spray water jets onto the rear of the screen. The screen further has back-projected visual content and touch interaction. The technology is demonstrated in Feeling Fireworks, a tactile firework show. The goal is to make fireworks more inclusive for the Blind and Low-Vision (BLV) community. A BLV focus group provided input during the development process, and a user study with BLV users showed that Feeling Fireworks is an enjoyable and meaningful experience. A user study with sighted users showed that users could accurately label the correspondence between the designed tactile firework effects and corresponding visual fireworks. Beyond the Feeling Fireworks application, this is a novel approach for scalable tactile displays with potential for broader use.

Robust and accurate visual localization is a fundamental capability for numerous applications, such as autonomous driving, mobile robotics, or augmented reality. It remains, however, a challenging task, particularly for large-scale environments and in presence of significant appearance changes. State-of-the-art methods not only struggle with such scenarios, but are often too resource intensive for certain real-time applications. In this paper we propose HF-Net, a hierarchical localization approach based on a monolithic CNN that simultaneously predicts local features and global descriptors for accurate 6-DoF localization. We exploit the coarse-to-fine localization paradigm: we first perform a global retrieval to obtain location hypotheses and only later match local features within those candidate places. This hierarchical approach incurs significant runtime savings and makes our system suitable for real-time operation. By leveraging learned descriptors, our method achieves remarkable localization robustness across large variations of appearance and sets a new state-of-the-art on two challenging benchmarks for large-scale localization.

Obtaining an accurate estimate of the local wind remains a significant challenge for small unmanned aerial vehicles (UAVs). Small UAVs often operate at low altitudes near terrain, where the wind environment can be more complex than at higher altitudes. Combined with their relatively low mass, this makes small UAVs particularly susceptible to wind. In this paper we present an approach for predicting high-resolution wind fields based on a terrain elevation model and known inflow conditions. Our approach uses a deep convolutional neural network (CNN) to generate 3D wind estimates. We show that our approach produces wind estimates with lower prediction error than existing methods, and that inference can be performed on an on-board computer in less than two seconds. By providing the wind estimate to a sampling-based planner we show that the improved estimates allow the planner to generate safer paths in strong wind scenarios than with alternative wind estimation techniques.

In this paper we present an end-to-end deep learning framework to turn images that show dynamic content, such as vehicles or pedestrians, into realistic static frames. This objective encounters two main challenges: detecting all the dynamic objects, and inpainting the static occluded background with plausible imagery. The former challenge is addressed by the use of a convolutional network that learns a multiclass semantic segmentation of the image. The second problem is approached with a conditional generative adversarial model that, taking as input the original dynamic image and its dynamic/static binary mask, is capable of generating the final static image. These generated images can be used for applications such as augmented reality or vision-based robot localization purposes. To validate our approach, we show both qualitative and quantitative comparisons against other state-of-the-art inpainting methods by removing the dynamic objects and hallucinating the static structure behind them. Furthermore, to demonstrate the potential of our results, we carry out pilot experiments that show the benefits of our proposal for visual place recognition.

This paper presents a pipeline for semantic segmentation of trees into their components. Given a single RGB-D image of a tree, we employ a deep network to predict labels to classify each pixel of the tree into trunk, branches, twigs and leaves. Multiple convolutional neural network architectures to combine the complementary modalities of depth and colour data are investigated. An asynchronous training approach where two networks trained separately on RGB and depth encoded as a 3-channel HHA image are combined using a late fusion architecture with different learning rates performs the best. Training and evaluation are performed on a synthetic dataset of 6 species of broadleaf trees. We further demonstrate the network's generalization capabilities, across various tree species on the synthetic dataset, achieving an accuracy of upto 92.5%. Furthermore, we present a qualitative evaluation of our approach on real-world data.

Compared to other vertical take-off and landing (VTOL) systems, a tailsitter minimizes the number of actuators and moving parts necessary. The downside of having a minimalistic actuation is its inherent low fault-tolerance. The failure of an actuator usually results in a loss of controllability, resulting in a crash. In this paper we analyze possible actuator failures and the constraints they pose on the capabilities of the system. We further present light-weight adaptations to the nominal flight controller to make it fault-tolerant. The fault-tolerant controller is implemented on a small tailsitter VTOL aircraft and adjusted to the system by means of extensive experimental studies. Finally, the capabilities and performance under failures are demonstrated and analyzed.

In autonomous racing, vehicles operate close to the limits of handling and a sensor failure can have critical consequences. To limit the impact of such failures, this paper presents the redundant perception and state estimation approaches developed for an autonomous race car. Redundancy in perception is achieved by estimating the color and position of the track delimiting objects using two sensor modalities independently. Specifically, learning-based approaches are used to generate color and pose estimates, from LiDAR and camera data respectively. The redundant perception inputs are fused by a particle filter based SLAM algorithm that operates in real-time. Velocity is estimated using slip dynamics, with reliability being ensured through a probabilistic failure detection algorithm. The sub-modules are extensively evaluated in real-world racing conditions using the autonomous race car gotthard driverless, achieving lateral accelerations up to 1. 7G and a top speed of 90km/h.

Accurate environment representation is one of the key challenges in autonomous ground vehicle navigation in unstructured environments. We propose a real-time optimization-based approach to terrain modeling and path planning in off-road and rough environments. Our method uses an irregular, hierarchical, graph-like environment model. A space-dividing tree is used to define a compact data structure capturing vertex positions and establishing connectivity. The same unique underlying data structure is used for both terrain modeling and path planning without memory reallocation. Local plans are generated by graph search algorithms and are continuously regenerated for on-the-fly obstacle avoidance inside the scope of the local terrain map. We show that implementing a hierarchical model over a regular space division reduces graph edge expansions by up to 84%. We illustrate the applicability of the method through experiments with an unmanned ground vehicle in both structured and unstructured environments.

Applications of mobile ground robots demand high speed and agility while navigating in complex indoor environments. These present an ongoing challenge in mobile robotics. A system with these specifications would be of great use for a wide range of indoor inspection tasks. This paper introduces Ascento, a compact wheeled bipedal robot that is able to move quickly on flat terrain, and to overcome obstacles by jumping. The mechanical design and overall architecture of the system is presented, as well as the development of various controllers for different scenarios. A series of experiments1with the final prototype system validate these behaviors in realistic scenarios.

Target search with unmanned aerial vehicles (UAVs) is relevant problem to many scenarios, e.g., search and rescue (SaR). However, a key challenge is planning paths for maximal search efficiency given flight time constraints. To address this, we propose the Obstacle-aware Adaptive Informative Path Planning (OA-IPP) algorithm for target search in cluttered environments using UAVs. Our approach leverages a layered planning strategy using a Gaussian Process (GP)based model of target occupancy to generate informative paths in continuous 3D space. Within this framework, we introduce an adaptive replanning scheme which allows us to trade off between information gain, field coverage, sensor performance, and collision avoidance for efficient target detection. Extensive simulations show that our OA-IPP method performs better than state-of-the-art planners, and we demonstrate its application in a realistic urban SaR scenario.

For autonomous driving applications it is critical to know which type of road users and road side infrastructure are present to plan driving manoeuvres accordingly. Therefore autonomous cars are equipped with different sensor modalities to robustly perceive its environment. However, for classification modules based on machine learning techniques it is challenging to overcome unseen sensor noise. This work presents an object classification module operating on unsupervised learned multi-modal features with the ability to overcome gradual or total sensor failure. A two stage approach composed of an unsupervised feature training and a uni-modal and multimodal classifiers training is presented. We propose a simple but effective decision module switching between uni-modal and multi-modal classifiers based on the closeness in the feature space to the training data. Evaluations on the ModelNet 40 data set show that the proposed approach has a 14% accuracy gain compared to a late fusion approach operating on a noisy point cloud data and a 6% accuracy gain when operating on noisy image data.

We present a fully-integrated sensing and control system which enables mobile manipulator robots to execute building tasks with millimeter-scale accuracy on building construction sites. The approach leverages multi-modal sensing capabilities for state estimation, tight integration with digital building models, and integrated trajectory planning and whole-body motion control. A novel method for high-accuracy localization updates relative to the known building structure is proposed. The approach is implemented on a real platform and tested under realistic construction conditions. We show that the system can achieve sub-cm end-effector positioning accuracy during fully autonomous operation using solely onboard sensing.

Many multirotor Unmanned Aerial Systems applications have a critical need for precise position control in environments with strong dynamic external disturbances such as wind gusts or ground and wall effects. Moreover, to maximize flight time, small multirotor platforms have to operate within strict constraints on payload and thus computational performance. In this paper, we present the design and experimental comparison of Model Predictive and PID multirotor position controllers augmented with a disturbance estimator to reject strong wind gusts up to 12 m/s and ground effect. For disturbance estimation, we compare Extended and Unscented Kalman filtering. In extensive in- and outdoor flight tests, we evaluate the suitability of the developed control and estimation algorithms to run on a computationally constrained platform. This allows to draw a conclusion on whether potential performance improvements justify the increased computational complexity of MPC for multirotor position control and UKF for disturbance estimation.

In this paper, we propose a visual-inertial framework able to efficiently estimate the camera poses of a non-rigid trinocular baseline for long-range depth estimation on-board a fast moving aerial platform. The estimation of the time-varying baseline is based on relative inertial measurements, a photometric relative pose optimizer, and a probabilistic wing model fused in an efficient Extended Kalman Filter (EKF) formulation. The estimated depth measurements can be integrated into a geo-referenced global map to render a reconstruction of the environment useful for local replanning algorithms. Based on extensive real-world experiments we describe the challenges and solutions for obtaining the probabilistic wing model, reliable relative inertial measurements, and vision-based relative pose updates and demonstrate the computational efficiency and robustness of the overall system under challenging conditions.

In many applications, maintaining a consistent map of the environment is key to enabling robotic platforms to perform higher-level decision making. Detection of already visited locations is one of the primary ways in which map consistency is maintained, especially in situations where external positioning systems are unavailable or unreliable. Mapping in 2D is an important field in robotics, largely due to the fact that man-made environments such as warehouses and homes, where robots are expected to play an increasing role, can often be approximated as planar. Place recognition in this context remains challenging: 2D lidar scans contain scant information with which to characterize, and therefore recognize, a location. This paper introduces a novel approach aimed at addressing this problem. At its core, the system relies on the use of the distance function for representation of geometry. This representation allows extraction of features which describe the geometry of both surfaces and free-space in the environment. We propose a feature for this purpose. Through evaluations on public datasets, we demonstrate the utility of free-space in the description of places, and show an increase in localization performance over a state-of-the-art descriptor extracted from surface geometry.

We introduce a novel method for oriented place recognition with 3D LiDAR scans. A Convolutional Neural Network is trained to extract compact descriptors from single 3D LiDAR scans. These can be used both to retrieve near-by place candidates from a map, and to estimate the yaw discrepancy needed for bootstrapping local registration methods. We employ a triplet loss function for training and use a hard-negative mining strategy to further increase the performance of our descriptor extractor. In an extensive evaluation on the NCLT and KITTI datasets, we demonstrate that our method outperforms related state-of-the-art approaches based on both data-driven and handcrafted data representation in challenging long-term outdoor conditions.

Small, low-speed fixed-wing Unmanned Aerial Vehicles (UAVs) operating autonomously, beyond-visual-line-of-sight (BVLOS) will inevitably encounter winds rising to levels near or exceeding the vehicles' nominal airspeed. In this paper, we develop a nonlinear lateral-directional path following guidance law with explicit consideration of online wind estimates. Energy efficient airspeed reference compensation logic is developed for excess wind scenarios (i.e. when the wind speed rises above the airspeed), enabling either mitigation, prevention, or over-powering of excess wind induced run-away from a given path. The developed guidance law is demonstrated on a representative small, low-speed test UAV in two flight experiments conducted in mountainous regions of Switzerland with strong, turbulent wind conditions, gusts reaching up to 13 meters per second. We demonstrate track-keeping errors of less than 1 meter consistently maintained during a representative duration of gusting, excess winds and a mean ground speed undershoot of 0.5 meters per second from the commanded minimum forward ground speed demonstrated in over 5 minutes of the showcased flight results.

Changes in appearance is one of the main sources of failure in visual localization systems in outdoor environments. To address this challenge, we present VIZARD, a visual localization system for urban outdoor environments. By combining a local localization algorithm with the use of multi-session maps, a high localization recall can be achieved across vastly different appearance conditions. The fusion of the visual localization constraints with wheel-odometry in a state estimation framework further guarantees smooth and accurate pose estimates. In an extensive experimental evaluation on several hundreds of driving kilometers in challenging urban outdoor environments, we analyze the recall and accuracy of our localization system, investigate its key parameters and boundary conditions, and compare different types of feature descriptors. Our results show that VIZARD is able to achieve nearly 100% recall with a localization accuracy below 0.5m under varying outdoor appearance conditions, including at night-time.

Robust prediction of pedestrian behavior is one of the most challenging problems for autonomous driving. Particularly, predicting pedestrian crossings at crosswalks is of considerable importance for avoiding accidents on the one hand and not unnecessarily slowing down traffic on the other hand. Traditional model-based motion tracking and prediction approaches have difficulties in capturing abrupt changes in motions, as humans can perform them. In this paper, an approach for predicting pedestrian motions that combines established motion tracking algorithms with data-driven methods is presented. The approach is built upon a hierarchical structure, where first, the intent of each pedestrian is classified. Then, the approach computes several qualitative metrics, such as time-to-cross, for the pedestrians classified as crossing. The approach is evaluated on a challenging urban data set collected for different types of crosswalks such as roundabouts and straight roads. The evaluation also provides a thorough analysis of the generalization performance of the proposed approach.

Visual robot navigation within large-scale, semistructured environments deals with various challenges such as computation intensive path planning algorithms or insufficient knowledge about traversable spaces. Moreover, many state-of-the-art navigation approaches only operate locally instead of gaining a more conceptual understanding of the planning objective. This limits the complexity of tasks a robot can accomplish and makes it harder to deal with uncertainties that are present in the context of real-time robotics applications. In this work, we present Topomap, a framework which simplifies the navigation task by providing a map to the robot which is tailored for path planning use. This novel approach transforms a sparse feature-based map from a visual Simultaneous Localization And Mapping (SLAM) system into a three-dimensional topological map. This is done in two steps. First, we extract occupancy information directly from the noisy sparse point cloud. Then, we create a set of convex free-space clusters, which are the vertices of the topological map. We show that this representation improves the efficiency of global planning, and we provide a complete derivation of our algorithm. Planning experiments on real world datasets demonstrate that we achieve similar performance as RRT* with significantly lower computation times and storage requirements. Finally, we test our algorithm on a mobile robotic platform to prove its advantages.

This paper proposes a computationally efficient method to estimate the time-varying relative pose between two visual-inertial sensor rigs mounted on the flexible wings of a fixed-wing unmanned aerial vehicle (UAV). The estimated relative poses are used to generate highly accurate depth maps in real-time and can be employed for obstacle avoidance in low-altitude flights or landing maneuvers. The approach is structured as follows: Initially, a wing model is identified by fitting a probability density function to measured deviations from the nominal relative baseline transformation. At runtime, the prior knowledge about the wing model is fused in an Extended Kalman filter (EKF) together with relative pose measurements obtained from solving a relative perspective N-point problem (PNP), and the linear accelerations and angular velocities measured by the two inertial measurement units (IMU) which are rigidly attached to the cameras. Results obtained from extensive synthetic experiments demonstrate that our proposed framework is able to estimate highly accurate baseline transformations and depth maps.

Use of low-cost depth sensors, such as a stereo camera setup with illuminators, is of particular interest for numerous applications ranging from robotics and transportation to mixed and augmented reality. The ability to quantify noise is crucial for these applications, e.g., when the sensor is used for map generation or to develop a sensor scheduling policy in a multi-sensor setup. Range error models provide uncertainty estimates and help weigh the data correctly in instances where range measurements are taken from different vantage points or with different sensors. Such a model is derived in this work. We show that the range error for stereo systems with integrated illuminators is cubic and validate the proposed model experimentally with an off-the-shelf structured light stereo system. The experiments confirm the validity of the model and simplify the application of this type of sensor in robotics.

Driven by the promise of leveraging the benefits of collaborative robot operation, this paper presents an approach to estimate the relative transformation between two small Unmanned Aerial Vehicles (UAVs), each equipped with a single camera and an inertial sensor, comprising the first step of any meaningful collaboration. Formation flying and collaborative object manipulation are some of the few tasks that the proposed work has direct applications on, while forming a variable-baseline stereo rig using two UAVs carrying a monocular camera each promises unprecedented effectiveness in collaborative scene estimation. Assuming an overlap in the UAVs' fields of view, in the proposed framework, each UAV runs monocular-inertial odometry onboard, while an Extended Kalman Filter fuses the UAVs' estimates and common image measurements to estimate the metrically scaled relative transformation between them, in realtime. Decoupling the direction of the baseline between the cameras of the two UAVs from its magnitude, this work enables consistent and robust estimation of the uncertainty of the relative pose estimation. Our evaluation on both on simulated data and benchmarking datasets consisting of real aerial data, reveals the power of the proposed methodology in a variety of scenarios. Video - https://youtu.be/AmkkaXa2601.

Achieving accurate, high-rate pose estimates from proprioceptive and/or exteroceptive measurements is the first step in the development of navigation algorithms for agile mobile robots such as Unmanned Aerial Vehicles (UAVs). In this paper, we propose a decoupled Graph-Optimization based Multi-Sensor Fusion approach (GOMSF) that combines generic 6 Degree-of-Freedom (DoF) visual-inertial odometry poses and 3 DoF globally referenced positions to infer the global 6 DoF pose of the robot in real-time. Our approach casts the fusion as a real-time alignment problem between the local base frame of the visual-inertial odometry and the global base frame. The alignment transformation that relates these coordinate systems is continuously updated by optimizing a sliding window pose graph containing the most recent robot's states. We evaluate the presented pose estimation method on both simulated data and large outdoor experiments using a small UAV that is capable to run our system onboard. Results are compared against different state-of-the-art sensor fusion frameworks, revealing that the proposed approach is substantially more accurate than other decoupled fusion strategies. We also demonstrate comparable results in relation with a finely tuned Extended Kalman Filter that fuses visual, inertial and GPS measurements in a coupled way and show that our approach is generic enough to deal with different input sources in a straightforward manner. Video - https//youtu.be/GIZNSZ2soL8.

Many robotic applications, such as search-and-rescue, require multiple agents to search for and perform actions on targets. However, such missions present several challenges, including cooperative exploration, task selection and allocation, time limitations, and computational complexity. To address this, we propose a decentralized multi-agent decision-making framework for the search and action problem with time constraints. The main idea is to treat time as an allocated budget in a setting where each agent action incurs a time cost and yields a certain reward. Our approach leverages probabilistic reasoning to make near-optimal decisions leading to maximized reward. We evaluate our method in the search, pick, and place scenario of the Mohamed Bin Zayed International Robotics Challenge (MBZIRC), by using a probability density map and reward prediction function to assess actions. Extensive simulations show that our algorithm outperforms benchmark strategies, and we demonstrate system integration in a Gazebo-based environment, validating the framework's readiness for field application.

This paper reports on a data-driven, interaction-aware motion prediction approach for pedestrians in environments cluttered with static obstacles. When navigating in such workspaces shared with humans, robots need accurate motion predictions of the surrounding pedestrians. Human navigation behavior is mostly influenced by their surrounding pedestrians and by the static obstacles in their vicinity. In this paper we introduce a new model based on Long-Short Term Memory (LSTM) neural networks, which is able to learn human motion behavior from demonstrated data. To the best of our knowledge, this is the first approach using LSTMs, that incorporates both static obstacles and surrounding pedestrians for trajectory forecasting. As part of the model, we introduce a new way of encoding surrounding pedestrians based on a 1d-grid in polar angle space. We evaluate the benefit of interaction-aware motion prediction and the added value of incorporating static obstacles on both simulation and real-world datasets by comparing with state-of-the-art approaches. The results show, that our new approach outperforms the other approaches while being very computationally efficient and that taking into account static obstacles for motion predictions significantly improves the prediction accuracy, especially in cluttered environments.

This paper introduces jlüela driverless: the first autonomous racecar to win a Formula Student Driverless competition. In this competition, among other challenges, an autonomous racecar is tasked to complete 10 laps of a previously unknown racetrack as fast as possible and using only onboard sensing and computing. The key components of flüela's design are its modular redundant sub-systems that allow robust performance despite challenging perceptual conditions or partial system failures. The paper presents the integration of key components of our autonomous racecar, i.e., system design, EKF-based state estimation, LiDAR-based perception, and particle filter-based SLAM. We perform an extensive experimental evaluation on real-world data, demonstrating the system's effectiveness by outperforming the next-best ranking team by almost half the time required to finish a lap. The autonomous racecar reaches lateral and longitudinal accelerations comparable to those achieved by experienced human drivers.

Predominately, robotic construction is applied as prefabrication in structured indoor environments with standard building materials. Our work, on the other hand, focuses on utilizing irregular materials found on-site, such as rubble and rocks, for autonomous construction. We present a pipeline to detect arbitrarily placed objects in a scene and form a structure out of the detected objects. The next best stacking pose is selected using a searching method employing gradient descent with random initial orientations, exploiting a physics engine. This approach is validated in an experimental setup using a robotic manipulator by constructing balancing vertical stacks without mortars and adhesives. We show the results of eleven consecutive trials to form such towers autonomously using four arbitrarily in front of the robot placed rocks.

Sensor fusion is a fundamental process in robotic systems as it extends the perceptual range and increases robustness in real-world operations. Current multi-sensor deep learning based semantic segmentation approaches do not provide robustness to under-performing classes in one modality, or require a specific architecture with access to the full aligned multi-sensor training data. In this work, we analyze statistical fusion approaches for semantic segmentation that overcome these drawbacks while keeping a competitive performance. The studied approaches are modular by construction, allowing to have different training sets per modality and only a much smaller subset is needed to calibrate the statistical models. We evaluate a range of statistical fusion approaches and report their performance against state-of-the-art baselines on both realworld and simulated data. In our experiments, the approach improves performance in IoU over the best single modality segmentation results by up to 5%. We make all implementations and configurations publicly available.

The growing popularity of autonomous systems creates a need for reliable and efficient metric pose retrieval algorithms. Currently used approaches tend to rely on nearest neighbor search of binary descriptors to perform the 2D-3D matching and guarantee realtime capabilities on mobile platforms. These methods struggle, however, with the growing size of the map, changes in viewpoint or appearance, and visual aliasing present in the environment. The rigidly defined descriptor patterns only capture a limited neighborhood of the keypoint and completely ignore the overall visual context. We propose LandmarkBoost - an approach that, in contrast to the conventional 2D-3D matching methods, casts the search problem as a landmark classification task. We use a boosted classifier to classify landmark observations and directly obtain correspondences as classifier scores. We also introduce a formulation of visual context that is flexible, efficient to compute, and can capture relationships in the entire image plane. The original binary descriptors are augmented with contextual information and informative features are selected by the boosting framework. Through detailed experiments, we evaluate the retrieval quality and performance of Landmark-Boost, demonstrating that it outperforms common state-of-the-art descriptor matching methods.

Reliable long-term localization is key for robotic systems in dynamic environments. In this paper, we propose a novel approach for long-term localization using 3D LiDARs, coined PoseMap. In essence, we extract distinctive features from range measurements and bundle these into local views along with observation poses. The sensor's trajectory is then estimated in a sliding window fashion by matching current and old features and minimizing the distances in-between. The map representation facilitates finding a suitable set of old features, by selecting the closest local map(s) for matching. Similarly to a visibility analysis, this procedure provides a suitable set of features for localization but at a fraction of the computational cost. PoseMap also allows for updates and extensions of the map at any time by replacing and adding local maps when necessary. We evaluate our approach using two platforms both equipped with a 3D LiDAR and an IMU, demonstrating localization at 8 Hz and robustness to changes in the environment such as moving vehicles and changing vegetation. PoseMap was implemented on an autonomous vehicle allowing it to drive autonomously over a period of 18 months through a mix of industrial and unstructured off-road environments, covering more than 100 kms without a single localization failure.

Many industrial facilities require periodic visual inspections. Often the points of interest are out of reach or in potentially hazardous environment. Multi-copters are ideal platforms to automate this expensive and tedious task. This video presents a system that enables a human operator to teach a visual inspection task to an autonomous aerial vehicle by simply demonstrating the task using a tablet. The system employs the Google Tango visual-inertial mapping framework as the only source of pose estimates, thus enabling operation in GPS-denied environments. In a first step the operator records the desired inspection path using the tablet. Inspection points are automatically inserted if the operator pauses, holding a viewpoint. The mapping framework then computes a feature-based localization map, which is shared with the robot. After take-off, the robot estimates its pose based on this map and plans a smooth trajectory through the way points defined by the operator. Furthermore, the system is able to track the global pose of other robots or the operator, localized in the same map, and follow them in real-time, while avoiding collision. This was demonstrated in the second part of the video, where the robot is following the operator in real-time through a hedge maze.

Collecting 3D object data sets involves a large amount of manual work and is time consuming. Getting complete models of objects either requires a 3D scanner that covers all the surfaces of an object or one needs to rotate it to completely observe it. We present a system that incrementally builds a database of objects as a mobile agent traverses a scene. Our approach requires no prior knowledge of the shapes present in the scene. Object-like segments are extracted from a global segmentation map, which is built online using the input of segmented RGB-D images. These segments are stored in a database, matched among each other, and merged with other previously observed instances. This allows us to create and improve object models on the fly and to use these merged models to reconstruct also unobserved parts of the scene. The database contains each (potentially merged) object model only once, together with a set of poses where it was observed. We evaluate our pipeline with one public dataset, and on a newly created Google Tango dataset containing four indoor scenes with some of the objects appearing multiple times, both within and across scenes.

System identification of High Altitude Long Endurance fixed-wing aerial vehicles is challenging as its operating flight envelope covers wide ranges of altitudes and Mach numbers. We present a new global system identification framework geared towards such fixed-wing aerial platforms where the aim is to build a global aerodynamic model without many repetitions of local system identification procedures or the use of any aerodynamic database. Instead we apply parameter identification techniques to virtually created system identification data and update the identified parameters with available flight test data. The proposed framework was evaluated using data set outside the flight envelope of the available flight test data, i.e. at different airspeeds considering both interpolation and extrapolation scenarios. The error analysis has shown that the obtained longitudinal aerodynamic model can accurately predict the pitch rate and pitch angle, mostly within a tolerance of +1.5 degrees/s and +2 degrees respectively. Such a cost and time efficient model development framework enables high fidelity simulation and precise control which ultimately leads to higher success rates in autonomous missions.

In many applications, maintaining a consistent dense map of the environment is key to enabling robotic platforms to perform higher level decision making. Several works have addressed the challenge of creating precise dense 3D maps from visual sensors providing depth information. However, during operation over longer missions, reconstructions can easily become inconsistent due to accumulated camera tracking error and delayed loop closure. Without explicitly addressing the problem of map consistency, recovery from such distortions tends to be difficult. We present a novel system for dense 3D mapping which addresses the challenge of building consistent maps while dealing with scalability. Central to our approach is the representation of the environment as a collection of overlapping Truncated Signed Distance Field (TSDF) subvolumes. These subvolumes are localized through feature-based camera tracking and bundle adjustment. Our main contribution is a pipeline for identifying stable regions in the map, and to fuse the contributing subvolumes. This approach allows us to reduce map growth while still maintaining consistency. We demonstrate the proposed system on a publicly available dataset and simulation engine, and demonstrate the efficacy of the proposed approach for building consistent and scalable maps. Finally we demonstrate our approach running in real-time onboard a lightweight Micro Aerial Vehicle (MAV).

Micro-Aerial Vehicles (MAVs) have the advantage of moving freely in 3D space. However, creating compact and sparse map representations that can be efficiently used for planning for such robots is still an open problem. In this paper, we take maps built from noisy sensor data and construct a sparse graph containing topological information that can be used for 3D planning. We use a Euclidean Signed Distance Field, extract a 3D Generalized Voronoi Diagram (GVD), and obtain a thin skeleton diagram representing the topological structure of the environment. We then convert this skeleton diagram into a sparse graph, which we show is resistant to noise and changes in resolution. We demonstrate global planning over this graph, and the orders of magnitude speed-up it offers over other common planning methods. We validate our planning algorithm in real maps built onboard an MAV, using RGB-D sensing.

Many scenarios require a robot to be able to explore its 3D environment online without human supervision. This is especially relevant for inspection tasks and search and rescue missions. To solve this high-dimensional path planning problem, sampling-based exploration algorithms have proven successful. However, these do not necessarily scale well to larger environments or spaces with narrow openings. This paper presents a 3D exploration planner based on the principles of Next-Best Views (NBVs). In this approach, a Micro-Aerial Vehicle (MAV)equipped with a limited field-of-view depth sensor randomly samples its configuration space to find promising future viewpoints. In order to obtain high sampling efficiency, our planner maintains and uses a history of visited places, and locally optimizes the robot's orientation with respect to unobserved space. We evaluate our method in several simulated scenarios, and compare it against a state-of-the-art exploration algorithm. The experiments show substantial improvements in exploration time (2 ⨯ faster), computation time, and path length, and advantages in handling difficult situations such as escaping dead-ends (up to 20 ⨯ faster). Finally, we validate the on-line capability of our algorithm on a computational constrained real world MAV.

We present a complete map management process for a visual localization system designed for multi-vehicle long-term operations in resource constrained outdoor environments. Outdoor visual localization generates large amounts of data that need to be incorporated into a lifelong visual map in order to allow localization at all times and under all appearance conditions. Processing these large quantities of data is non-trivial, as it is subject to limited computational and storage capabilities both on the vehicle and on the mapping backend. We address this problem with a two-fold map update paradigm capable of, either, adding new visual cues to the map, or updating co-observation statistics. The former, in combination with offline map summarization techniques, allows enhancing the appearance coverage of the lifelong map while keeping the map size limited. On the other hand, the latter is able to significantly boost the appearance-based landmark selection for efficient online localization without incurring any additional computational or storage burden. Our evaluation in challenging outdoor conditions shows that our proposed map management process allows building and maintaining maps for precise visual localization over long time spans in a tractable and scalable fashion.

In this paper, we present a method, namely€CCA, for collision avoidance in dynamic environments among interacting agents, such as other robots or humans. Given a preferred motion by a global planner or driver, the method computes a collision-free local motion for a short time horizon, which respects the actuator constraints and allows for smooth and safe control. The method builds on the concept of reciprocal velocity obstacles and extends it to respect the kinodynamic constraints of the robot and account for a grid-based map representation of the environment. The method is best suited for large multirobot settings, including heterogeneous teams of robots, in which computational complexity is of paramount importance and the robots interact with one another. In particular, we consider a set of motion primitives for the robot and solve an optimization in the space of control velocities with additional constraints. Additionally, we propose a cooperative approach to compute safe velocity partitions in the distributed case. We describe several instances of the method for distributed and centralized operation and formulated both as convex and nonconvex optimizations. We compare the different variants and describe the benefits and tradeoffs both theoretically and in extensive experiments with various robotic platforms: robotic wheelchairs, robotic boats, humanoid robots, small unicycle robots, and simulated cars.

This paper reports on an algorithm for planning trajectories that allow a multirotor micro aerial vehicle (MAV) to quickly identify a set of unknown parameters. In many problems like self calibration or model parameter identification some states are only observable under a specific motion. These motions are often hard to find, especially for inexperienced users. Therefore, we consider system model identification in an active setting, where the vehicle autonomously decides what actions to take in order to quickly identify the model. Our algorithm approximates the belief dynamics of the system around a candidate trajectory using an extended Kalman filter (EKF). It uses sampling-based motion planning to explore the space of possible beliefs and find a maximally informative trajectory within a user-defined budget. We validate our method in simulation and on a real system showing the feasibility and repeatability of the proposed approach. Our planner creates trajectories which reduce model parameter convergence time and uncertainty by a factor of four.

To many unmanned aerial vehicle (UAV) designs, the lack of information about the wind speed and direction is a limiting factor in achieving robust outdoor flight. This paper addresses the problem of wind estimation onboard a hovering vertical take-off and landing (VTOL) tailsitter UAV. The proposed estimation framework makes use of the standard onboard sensor suite: inertial measurement unit (IMU), global positioning system (GPS) and a magnetometer. No additional airspeed sensor is needed. As a result, the autopilot is provided with an estimate of the wind velocity vector in the horizontal (north-east) plane. An aerodynamic model of the vehicle has been derived and used in a Kalman filter framework to estimate the horizontal wind velocity vector in real-time. The wind estimator has been implemented onboard the UAVs autopilot and validated in real flight. As a result, we successfully obtain the direction and speed of the wind with an estimation accuracy close to the accuracy range of the ground truth measurement. Furthermore, the derived grey-box model allows to generalise the framework to different airframes.

Place recognition in 3D data is a challenging task that has been commonly approached by adapting image-based solutions. Methods based on local features suffer from ambiguity and from robustness to environment changes while methods based on global features are viewpoint dependent. We propose SegMatch, a reliable place recognition algorithm based on the matching of 3D segments. Segments provide a good compromise between local and global descriptions, incorporating their strengths while reducing their individual drawbacks. SegMatch does not rely on assumptions of `perfect segmentation', or on the existence of `objects' in the environment, which allows for reliable execution on large scale, unstructured environments. We quantitatively demonstrate that SegMatch can achieve accurate localization at a frequency of 1Hz on the largest sequence of the KITTI odometry dataset. We furthermore show how this algorithm can reliably detect and close loops in real-time, during online operation. In addition, the source code for the SegMatch algorithm is made publicly available.

Robots that are operating for extended periods of time need to be able to deal with changes in their environment and represent them adequately in their maps. In this paper, we present a novel 3D reconstruction algorithm based on an extended Truncated Signed Distance Function (TSDF) that enables to continuously refine the static map while simultaneously obtaining 3D reconstructions of dynamic objects in the scene. This is a challenging problem because map updates happen incrementally and are often incomplete. Previous work typically performs change detection on point clouds, surfels or maps, which are not able to distinguish between unexplored and empty space. In contrast, our TSDF-based representation naturally contains this information and thus allows us to more robustly solve the scene differencing problem. We demonstrate the algorithms performance as part of a system for unsupervised object discovery and class recognition. We evaluated our algorithm on challenging datasets that we recorded over several days with RGB-D enabled tablets. To stimulate further research in this area, all of our datasets are publicly available3.

Predominately, robotic construction is applied as prefabrication in structured indoor environments with standard building materials. Our work, on the other hand, focuses on utilizing irregular materials found on-site, such as rubble and rocks, for autonomous construction. We present a pipeline that detects randomly placed objects in a scene that are used by our next best stacking pose searching method employing gradient descent with a random initial orientation, exploiting a physics engine. This approach is validated in an experimental setup using a robotic manipulator by constructing balancing vertical stacks without mortars and adhesives. We show the results of eleven consecutive trials to form such towers autonomously using four arbitrarily in front of the robot placed rocks.

Autonomous delivery of goods using a Micro Air Vehicle (MAV) is a difficult problem, as it poses high demand on the MAV's control, perception and manipulation capabilities. This problem is especially challenging if the exact shape, location and configuration of the objects are unknown. In this paper, we report our findings during the development and evaluation of a fully integrated system that is energy efficient and enables MAVs to pick up and deliver objects with partly ferrous surface of varying shapes and weights. This is achieved by using a novel combination of an electro-permanent magnetic gripper with a passively compliant structure and integration with detection, control and servo positioning algorithms. The system's ability to grasp stationary and moving objects was tested, as well as its ability to cope with different shapes of the object and external disturbances. We show that such a system can be successfully deployed in scenarios where an object with partly ferrous parts needs to be gripped and placed in a predetermined location.

We propose a novel scoring concept for visual place recognition based on nearest neighbor descriptor voting and demonstrate how the algorithm naturally emerges from the problem formulation. Based on the observation that the number of votes for matching places can be evaluated using a binomial distribution model, loop closures can be detected with high precision. By casting the problem into a probabilistic framework, we not only remove the need for commonly employed heuristic parameters but also provide a powerful score to classify matching and non-matching places. We present methods for both a 2D-2D image matching and a 2D-3D landmark matching based on the above scoring. The approach maintains accuracy while being efficient enough for online application through the use of compact (low-dimensional) descriptors and fast nearest neighbor retrieval techniques. The proposed methods are evaluated on several challenging datasets in varied environments, showing state-of-the-art results with high precision and high recall.

Perception systems for outdoor robotics have to deal with varying environmental conditions. Variations in illumination in particular, are currently the biggest challenge for vision-based perception. In this paper we present an approach for radiometric characterization of multispectral cameras. To enable spatio-temporal mapping we also present a procedure for in-situ illumination estimation, resulting in radiometric calibration of the collected images. In contrast to current approaches, we present a purely data driven, parameter free approach, based on maximum likelihood estimation which can be performed entirely on the field, without requiring specialised laboratory equipment. Our routine requires three simple datasets which are easily acquired using most modern multispectral cameras. We evaluate the framework with a cost-effective snapshot multispectral camera. The results show that our method enables the creation of quatitatively accurate relative reflectance images with challenging on field calibration datasets under a variety of ambient conditions.

Many robotics and Augmented Reality (AR) systems that use sparse keypoint-based visual maps operate in large and highly repetitive environments, where pose tracking and localization are challenging tasks. Additionally, these systems usually face further challenges, such as limited computational power, or insufficient memory for storing large maps of the entire environment. Thus, developing compact map representations and improving retrieval is of considerable interest for enabling large-scale visual place recognition and loop-closure. In this paper, we propose a novel approach to compress descriptors while increasing their discriminability and match-ability, based on recent advances in neural networks. At the same time, we target resource-constrained robotics applications in our design choices. The main contributions of this work are twofold. First, we propose a linear projection from descriptor space to a lower-dimensional Euclidean space, based on a novel supervised learning strategy employing a triplet loss. Second, we show the importance of including contextual appearance information to the visual feature in order to improve matching under strong viewpoint, illumination and scene changes. Through detailed experiments on three challenging datasets, we demonstrate significant gains in performance over state-of-the-art methods.

Unmanned aerial vehicles (UAVs) and other robots in smart farming applications offer the potential to monitor farm land on a per-plant basis, which in turn can reduce the amount of herbicides and pesticides that must be applied. A central information for the farmer as well as for autonomous agriculture robots is the knowledge about the type and distribution of the weeds in the field. In this regard, UAVs offer excellent survey capabilities at low cost. In this paper, we address the problem of detecting value crops such as sugar beets as well as typical weeds using a camera installed on a light-weight UAV. We propose a system that performs vegetation detection, plant-tailored feature extraction, and classification to obtain an estimate of the distribution of crops and weeds in the field. We implemented and evaluated our system using UAVs on two farms, one in Germany and one in Switzerland and demonstrate that our approach allows for analyzing the field and classifying individual plants.

A variety of end-user devices involving keypoint-based mapping systems are about to hit the market e.g. as part of smartphones, cars, robotic platforms, or virtual and augmented reality applications. Thus, the generated map data requires automated evaluation procedures that do not require experienced personnel or ground truth knowledge of the underlying environment. A particularly important question enabling commercial applications is whether a given map is of sufficient quality for localization. This paper proposes a framework for predicting localization performance in the context of visual landmark-based mapping. Specifically, we propose an algorithm for predicting performance of vision-based localization systems from different poses within the map. To achieve this, a metric is defined that assigns a score to a given query pose based on the underlying map structure. The algorithm is evaluated on two challenging datasets involving indoor data generated using a handheld device and outdoor data from an autonomous fixed-wing unmanned aerial vehicle (UAV). Using these, we are able to show that the score provided by our method is highly correlated to the true localization performance. Furthermore, we demonstrate how the predicted map quality can be used within a belief based path planning framework in order to provide reliable trajectories through high-quality areas of the map.

Learning from demonstration for motion planning is an ongoing research topic. In this paper we present a model that is able to learn the complex mapping from raw 2D-laser range findings and a target position to the required steering commands for the robot. To our best knowledge, this work presents the first approach that learns a target-oriented end-to-end navigation model for a robotic platform. The supervised model training is based on expert demonstrations generated in simulation with an existing motion planner. We demonstrate that the learned navigation model is directly transferable to previously unseen virtual and, more interestingly, real-world environments. It can safely navigate the robot through obstacle-cluttered environments to reach the provided targets. We present an extensive qualitative and quantitative evaluation of the neural network-based motion planner, and compare it to a grid-based global approach, both in simulation and in real-world experiments.

In this paper, we introduce an informative path planning (IPP) framework for active classification using unmanned aerial vehicles (UAVs). Our algorithm uses a combination of global viewpoint selection and evolutionary optimization to refine the planned trajectory in continuous 3D space while satisfying dynamic constraints. Our approach is evaluated on the application of weed detection for precision agriculture. We model the presence of weeds on farmland using an occupancy grid and generate adaptive plans according to information-theoretic objectives, enabling the UAV to gather data efficiently. We validate our approach in simulation by comparing against existing methods, and study the effects of different planning strategies. Our results show that the proposed algorithm builds maps with over 50% lower entropy compared to traditional “lawnmower” coverage in the same amount of time. We demonstrate the planning scheme on a multirotor platform with different artificial farmland set-ups.

Conventional camera calibration techniques rely on discrete reference points extracted from a set of input images. While these approaches have been applied successfully for a long time, omitting all image information apart from reference point positions at the initial stage of the calibration pipeline renders correct treatment of uncertainties difficult and gives rise to complications in timestamping measurements in applications where exposure time cannot be neglected. Drawing inspiration from visual state estimation, we employ a direct formulation of the camera measurement model. To this end, we render a view of the target given all calibration parameters, enabling a maximum likelihood estimator formulated on image intensities as measurements. We demonstrate the advantages of avoiding abstraction from image measurements for determining the line delay of a rolling shutter camera and by estimating camera exposure time from motion blur.

Environmental conditions and external effects, such as shocks, have a significant impact on the calibration parameters of visual-inertial sensor systems. Thus long-term operation of these systems cannot fully rely on factory calibration. Since the observability of certain parameters is highly dependent on the motion of the device, using short data segments at device initialization may yield poor results. When such systems are additionally subject to energy constraints, it is also infeasible to use full-batch approaches on a big dataset and careful selection of the data is of high importance. In this paper, we present a novel approach for resource efficient self-calibration of visual-inertial sensor systems. This is achieved by casting the calibration as a segment-based optimization problem that can be run on a small subset of informative segments. Consequently, the computational burden is limited as only a predefined number of segments is used. We also propose an efficient information-theoretic selection to identify such informative motion segments. In evaluations on a challenging dataset, we show our approach to significantly outperform state-of-the-art in terms of computational burden while maintaining a comparable accuracy.

This paper shows a strategy based on passive force control for collaborative object transportation using Micro Aerial Vehicles (MAVs), focusing on the transportation of a bulky object by two hexacopters. The goal is to develop a robust approach which does not rely on: (a) communication links between the MAVs, (b) the knowledge of the payload shape and (c) the position of grasping point. The proposed approach is based on the master-slave paradigm, in which the slave agent guarantees compliance to the external force applied by the master to the payload via an admittance controller. The external force acting on the slave is estimated using a non-linear estimator based on the Unscented Kalman Filter (UKF) from the information provided by a Visual-Inertial (VI) navigation system. Experimental results (online video [1]) demonstrate the performance of the force estimator and show the collaborative transportation of a 1.2 m long object.

This paper addresses the problem of trajectory optimization for the transition of a Vertical Take-off and Landing (VTOL) tailsitter Unmanned Aerial Vehicle (UAV). The proposed strategy performs a model based optimization, where the model represents the closed-loop dynamics of the UAV with low-level control, ensuring attitude stabilization over the whole trajectory. We discuss the design of an optimization framework, vehicle modeling, and elaborate on the cost function construction. An additional feedback gain is implemented on the throttle channel with altitude discrepancies as its input to provide some level of robustness to wind disturbances. The overall approach is verified in simulation and experimental results with a focus on optimization of the back-transition (cruise-to-hover) of the Wingtra S100 VTOL tailsitter.

Using multiple cooperative robots is advantageous for time critical Search and Rescue (SaR) missions as they permit rapid exploration of the environment and provide higher redundancy than using a single robot. A considerable number of applications such as autonomous driving and disaster response could benefit from merging mapping data from several agents. Online multi-robot localization and mapping has mainly been addressed for robots equipped with cameras or 2D LiDARs. However, in unstructured and ill-lighted real-life scenarios, a mapping system can potentially benefit from a rich 3D geometric solution. In this work, we present an online localization and mapping system for multiple robots equipped with 3D LiDARs. This system is based on incremental sparse pose-graph optimization using sequential and place recognition constraints, the latter being identified using a 3D segment matching approach. The result is a unified representation of the world and relative robot trajectories. The complete system runs in real-time and is evaluated with two experiments in different environments: one urban and one disaster scenario. The system is available open source and easy-to-run demonstrations are publicly available.

When several Multirotor Micro Aerial Vehicles (MAVs) share the same airspace, reliable and robust collision avoidance is required. In this paper we address the problem of multi-MAV reactive collision avoidance. We employ a model-based controller to simultaneously track a reference trajectory and avoid collisions. Moreover, to achieve a higher degree of robustness, our method also accounts for the uncertainty of the state estimator and of the position and velocity of the other agents. The proposed approach is decentralized, does not require a collision-free reference trajectory and accounts for the full MAV dynamics. We validated our approach in simulation and experimentally with two MAV.

Micro Aerial Vehicles (MAVs) that operate in unstructured, unexplored environments require fast and flexible local planning, which can replan when new parts of the map are explored. Trajectory optimization methods fulfill these needs, but require obstacle distance information, which can be given by Euclidean Signed Distance Fields (ESDFs). We propose a method to incrementally build ESDFs from Truncated Signed Distance Fields (TSDFs), a common implicit surface representation used in computer graphics and vision. TSDFs are fast to build and smooth out sensor noise over many observations, and are designed to produce surface meshes. We show that we can build TSDFs faster than Octomaps, and that it is more accurate to build ESDFs out of TSDFs than occupancy maps. Our complete system, called voxblox, is available as open source and runs in real-time on a single CPU core. We validate our approach on-board an MAV, by using our system with a trajectory optimization local planner, entirely on-board and in real-time.

Unmanned aerial vehicles (UAVs) can offer timely and cost-effective delivery of high-quality sensing data. However, deciding when and where to take measurements in complex environments remains an open challenge. To address this issue, we introduce a new multiresolution mapping approach for informative path planning in terrain monitoring using UAVs. Our strategy exploits the spatial correlation encoded in a Gaussian Process model as a prior for Bayesian data fusion with probabilistic sensors. This allows us to incorporate altitude-dependent sensor models for aerial imaging and perform constant-time measurement updates. The resulting maps are used to plan information-rich trajectories in continuous 3-D space through a combination of grid search and evolutionary optimization. We evaluate our framework on the application of agricultural biomass monitoring. Extensive simulations show that our planner performs better than existing methods, with mean error reductions of up to 45% compared to traditional “lawnmower” coverage. We demonstrate proof of concept using a multirotor to map color in different environments.

Deployment of camera and laser based motion estimation systems for controlling platforms operating at high speeds, such as cars or trains, is posing increasingly challenging precision requirements on the temporal calibration of these sensors. In this work, we demonstrate a simple, low-cost system for calibrating any combination of cameras and time of flight LIDARs with respect to the CPU clock (and therefore, also to each other). The newly proposed device is based on widely available off-the-shelf components, such as the Raspberry Pi 3, which is synchronized using the Precision Time Protocol (PTP) with respect to the CPU of the sensor carrying system. The obtained accuracy can be shown to be below 0.1 ms per measurement for LIDARs and below minimal exposure time per image for cameras. It outperforms state-of-the-art approaches also not relying on hardware synchronization by more than a factor of 10 in precision. Moreover, the entire process can be carried out at a high rate allowing the study of how offsets evolve over time. In our analysis, we demonstrate how each building block of the system contributes to this accuracy and validate the obtained results using real-world data.

In this paper, we propose a GPU parallelized SLAM system capable of using photometric and inertial data together with depth data from an active RGB-D sensor to build accurate dense 3D maps of indoor environments. We describe several extensions to existing dense SLAM techniques that allow us to operate in real-time onboard memory constrained robotic platforms. Our primary contribution is a memory management algorithm that scales to large scenes without being limited by GPU memory resources. Moreover, by integrating a visual-inertial odometry system, we robustly track the camera pose even on an agile platform such as a quadrotor UAV. Our robust camera tracking framework can deal with fast camera motions and varying environments by relying on depth, color and inertial motion cues. Global consistency is achieved via regular checking for loop closures in conjunction with a pose graph, as a basis for corrective deformation of the 3D map. Our efficient SLAM system is capable of producing highly dense meshes up to 5mm resolution at rates close to 60Hz fully onboard a UAV. Experimental validations both in simulation and on a real-world platform, show that our approach is fast, more robust and more memory efficient than state-of-the-art techniques, while obtaining better or comparable accuracy.

An increasing number of simultaneous localization and mapping (SLAM) systems are using appearance-based localization to improve the quality of pose estimates. However, with the growing time-spans and size of the areas we want to cover, appearance-based maps are often becoming too large to handle and are consisting of features that are not always reliable for localization purposes. This paper presents a method for selecting map features that are persistent over time and thus suited for long-term localization. Our methodology relies on a CNN classifier based on image patches and depth maps for recognizing which features are suitable for life-long matchability. Thus, the classifier not only considers the appearance of a feature but also takes into account its expected lifetime. As a result, our feature selection approach produces more compact maps with a high fraction of temporally-stable features compared to the current state-of-the-art, while rejecting unstable features that typically harm localization. Our approach is validated on indoor and outdoor datasets, that span over a period of several months.

A novel method for visual place recognition is introduced and evaluated, demonstrating robustness to perceptual aliasing and observation noise. This is achieved by increasing discrimination through a more structured representation of visual observations. Estimation of observation likelihoods are based on graph kernel formulations, utilizing both the structural and visual information encoded in covisibility graphs. The proposed probabilistic model is able to circumvent the typically difficult and expensive posterior normalization procedure by exploiting the information available in visual observations. Furthermore, the place recognition complexity is independent of the size of the map. Results show improvements over the state-of-theart on a diverse set of both public datasets and novel experiments, highlighting the benefit of the approach.

This paper presents a novel path planning algorithm for the autonomous exploration of unknown space using aerial robotic platforms. The proposed planner employs a receding horizon “next-best-view” scheme: In an online computed random tree it finds the best branch, the quality of which is determined by the amount of unmapped space that can be explored. Only the first edge of this branch is executed at every planning step, while repetition of this procedure leads to complete exploration results. The proposed planner is capable of running online, onboard a robot with limited resources. Its high performance is evaluated in detailed simulation studies as well as in a challenging real world experiment using a rotorcraft micro aerial vehicle. Analysis on the computational complexity of the algorithm is provided and its good scaling properties enable the handling of large scale and complex problem setups.

As the applications of Micro Aerial Vehicles (MAVs) get more and more complex, and require highly dynamic motions, it becomes essential to have an accurate dynamic model of the MAV. Such a model can be used for reliable state estimation, control, and for realistic simulation. A good model requires accurate estimates of physical parameters of the system, which we aim to estimate from recorded flight data. In this paper, we present a detailed physical model of the MAV and a maximum likelihood estimation scheme for determining the dominant parameters, such as inertia matrix, center of gravity (CoG) with respect to the IMU, and parameters related to the aerodynamics. To incorporate all information given by the IMU and the physical MAV model, we propose to use two process models in the optimization. We show the effectiveness of the method on simulated data, as well as on a real platform.

Place recognition is a core component in simultaneous localization and mapping (SLAM), limiting positional drift over space and time to unlock precise robot navigation. Determining which previously visited places belong together continues to be a highly active area of research as robotic applications demand increasingly higher accuracies. A large number of place recognition algorithms have been proposed, capable of consuming a variety of sensor data including laser, sonar and depth readings. The best performing solutions, however, have utilized visual information by either matching entire images or parts thereof. Most commonly, vision based approaches are inspired by information retrieval and utilize 3D-geometry information about the observed scene as a post-verification step. In this paper we propose to use the 3D-scene information from sparse-visual feature maps directly at the core of the place recognition pipeline. We propose a novel structural descriptor which aggregates sparse triangulated landmarks from SLAM into a compact signature. The resulting 3D-features provide a discriminative fingerprint to recognize places over seasonal and viewpoint changes which are particularly challenging for approaches based on sparse visual descriptors. We evaluate our system on publicly available datasets and show how its complementary nature can provide an improvement over visual place recognition.

Sliding window estimation is widely used for online simultaneous localization and mapping. While increasing the sliding window size generally yields improved accuracy, it also comes at an increase in computational cost. In order to reduce this cost, we propose smarter non-uniform sampling of the trajectory representation over the sliding window. This non-uniform temporal resolution is possible with continuous-time representations that allow freely adjustable knots location. Four strategies for selecting the knots location are presented and evaluated based on a real data laser-odometry SLAM problem. The results clearly show that non-uniform distributions of knots can be superior to uniform distribution in terms of accuracy per computation time.

An accurate estimate of the 3D-structure in the environment is key to robotic applications such as autonomous inspection, obstacle avoidance and manipulation. Recent years have seen substantial algorithmic advances towards creating highly accurate models of small objects as well as large scale architectural structures. Most commonly a rich set of images covering a static scene are used to jointly estimate the pose of the cameras and the observed 3D-structure. For many practical application however the assumption of static scenes and sufficient coverage by images does not hold. In fact for industrial inspection the change in the scene is of most interest and the limited resources on mobile platforms don't allow for extensive data captures. In this paper we investigate the potential of combining multiple independent captures of a place to selectively reconstruct a scene over time. We propose an incremental reconstruction algorithm which identifies and fuses novel data into a joint model of the scene. Being able to identify changing parts of the scene is particularly interesting for mobile applications where bandwidth, storage and processing power are limited. Through detailed experiments, we show the potential of our approach to use multiple mobile devices to reconstruct and update a model of the static part of the environment over time.

This paper presents a framework for real-time, full-state feedback, unconstrained, nonlinear model predictive control that combines trajectory optimization and tracking control in a single, unified approach. The proposed method uses an iterative optimal control algorithm, namely Sequential Linear Quadratic (SLQ), in a Model Predictive Control (MPC) setting to solve the underlying nonlinear control problem and simultaneously derive the optimal feedforward and feedback terms. Our customized solver can generate trajectories of multiple seconds within only a few milliseconds. The performance of the approach is validated on two different hardware platforms, an AscTec Firefly hexacopter and the ball balancing robot Rezero. In contrast to similar approaches, we perform experiments that require leveraging the full system dynamics.

An increasing number of robotic systems feature multiple inertial measurement units (IMUs). Due to competing objectives-either desired vicinity to the center of gravity when used in controls, or an unobstructed field of view when integrated in a sensor setup with an exteroceptive sensor for ego-motion estimation-individual IMUs are often mounted at considerable distance. As a result, they sense different accelerations when the platform is subjected to rotational motions. In this work, we derive a method for spatially calibrating multiple IMUs in a single estimator based on the open-source camera/IMU calibration toolbox kalibr. We further extend the toolbox to determine IMU intrinsics, enabling accurate calibration of low-cost IMUs. The results suggest that the extended estimator is capable of precisely determining these intrinsics and even of localizing individual accelerometer axes inside a commercial grade IMU to millimeter precision.

This paper addresses the challenges of the design, development and control of a new convertible VTOL tailsitter unmanned aerial vehicle that combines the advantages of both fixed wing and rotary wing systems. Wind tunnel measurements are used to get an understanding of the control allocation and to model the static forces and moments acting on the system. Based on the derived model, a novel controller that operates in SO(3) and handles the dynamics of the vehicle at any attitude configuration, including the rotorcraft and fixed-wing regimes as well as their transitions, is presented. This unified controller allows the autonomous transition of the system without discontinuities of switching, as well as its overall high performance flight control. The capabilities and flying qualities of the platform and the controller are demonstrated and evaluated by means of extensive experimental studies.

Although Point Clouds Registration is a very well studied problem, with many different solutions, most of the approaches in the literature aims at aligning two dense point clouds. Instead, we tackle the problem of aligning a dense point cloud with a sparse one: a problem that has to be solved, for example, to merge maps produced by different sensors, such as a vision-based sensor and laser scanner or two different laser-based sensors. The most used approach to point clouds registration, Iterative Closest Point (ICP), is also applicable to this sub-problem. We propose an improvement over the standard ICP data association policy and we called it Probabilistic Data Association. It was derived applying statistical inference techniques on a fully probabilistic model. In our proposal, each point in the source point cloud is associated with a set of points in the target point cloud; each association is then weighted so that the weights form a probability distribution. The result is an algorithm similar to ICP but more robust w.r.t. noise and outliers. While we designed our approach to deal with the problem of dense-sparse registration, it can be successfully applied also to standard point clouds registration.

In this paper, we present an online landmark selection method for distributed long-term visual localization systems in bandwidth-constrained environments. Sharing a common map for online localization provides a fleet of autonomous vehicles with the possibility to maintain and access a consistent map source, and therefore reduce redundancy while increasing efficiency. However, connectivity over a mobile network imposes strict bandwidth constraints and thus the need to minimize the amount of exchanged data. The wide range of varying appearance conditions encountered during long-term visual localization offers the potential to reduce data usage by extracting only those visual cues which are relevant at the given time. Motivated by this, we propose an unsupervised method of adaptively selecting landmarks according to how likely these landmarks are to be observable under the prevailing appearance condition. The ranking function this selection is based upon exploits landmark co-observability statistics collected in past traversals through the mapped area. Evaluation is performed over different outdoor environments, large time-scales and varying appearance conditions, including the extreme transition from day-time to night-time, demonstrating that with our appearance-dependent selection method, we can significantly reduce the amount of landmarks used for localization while maintaining or even improving the localization performance.

In this paper we present a new estimation algorithm that allows for the combination of information from any number of process and measurement models. This adds more flexibility to the design of the estimator and in our case avoids the need for state augmentation. We achieve this by adapting the maximum likelihood formulation of the Kalman Filter, and thereby represent all measurement models as residuals. Posing the problem in this form allows for the straightforward integration of any number of (nonlinear) constraints between two subsequent states. To solve the optimization we present a closed form recursive set of equations that directly marginalizes out information that is not required, this leads to an efficient and generic implementation. The new algorithm is applied to parameter estimation on MAVs which have two dynamic models, the MAV dynamic model and the IMU-driven model. We show the benefits and limitations of the new filtering approach on a simplified simulation example and on a real MAV system.

Precisely estimating the pose of an agent in a global reference frame is a crucial goal that unlocks a multitude of robotic applications, including autonomous navigation and collaboration. In order to achieve this, current state-of-the-art localization approaches collect data provided by one or more agents and create a single, consistent localization map, maintained over time. However, with the introduction of lengthier sorties and the growing size of the environments, data transfers between the backend server where the global map is stored and the agents are becoming prohibitively large. While some existing methods partially address this issue by building compact summary maps, the data transfer from the agents to the backend can still easily become unmanageable. In this paper, we propose a method that is designed to reduce the amount of data that needs to be transferred from the agent to the backend, functioning in large-scale, multi-session mapping scenarios. Our approach is based upon a landmark selection method that exploits information coming from multiple, possibly weak and correlated, landmark utility predictors; fused using learned feature coefficients. Such a selection yields a drastic reduction in data transfer while maintaining localization performance and the ability to efficiently summarize environments over time. We evaluate our approach on a data set that was autonomously collected in a dynamic indoor environment over a period of several months.

Flying and walking robots can use their complementary features in terms of viewpoint and payload capability to the best in a heterogeneous team. To this end, we present our online collaborative navigation framework for unknown and challenging terrain. The method leverages the flying robot's onboard monocular camera to create both a map of visual features for simultaneous localization and mapping and a dense representation of the environment as an elevation map. This shared knowledge from the flying platform enables the walking robot to localize itself against the global map, and plan a global path to the goal by interpreting the elevation map in terms of traversability. While following the planned path, the absolute pose corrections are fused with the legged state estimation and the elevation map is continuously updated with distance measurements from an onboard laser range sensor. This allows the legged robot to safely navigate towards the goal while taking into account any changes in the environment. In this setup, our approach is independent of external localization, relative observations between the robots, and does not require an initial guess about the pose of the robots. The presented methods are fully integrated and we demonstrate their capabilities in an experiment with a hexacopter and a quadrupedal robot.

Persistent merging of maps created by different sensor modalities is an insufficiently addressed problem. Current approaches either rely on appearance-based features which may suffer from lighting and viewpoint changes or require pre-registration between all sensor modalities used. This work presents a framework using structural descriptors for matching LIDAR point-cloud maps and sparse vision keypoint maps. The matching algorithm works independently of the sensors' viewpoint and varying lighting and does not require pre-registration between the sensors used. Furthermore, we employ the approach in a novel vision-laser map-merging algorithm. We analyse a range of structural descriptors and present results of the method integrated within a full mapping framework. Despite the fact that we match between the visual and laser domains, we can successfully perform map-merging using structural descriptors. The effectiveness of the presented structure-based vision-laser matching is evaluated on the public KITTI dataset and furthermore demonstrated on a map merging problem in an industrial site.

Accurate and robust real-time map generation onboard of a fixed-wing UAV is essential for obstacle avoidance, path planning, and critical maneuvers such as autonomous take-off and landing. Due to the computational constraints, the required robustness and reliability, it remains a challenge to deploy a fixed-wing UAV with an online-capable, accurate and robust map generation framework. While photogrammetric approaches have underlying assumptions on the structure and the view of the camera, generic simultaneous localization and mapping (SLAM) approaches are computationally demanding. This paper presents a framework that uses the autopilot's state estimate as a prior for sliding window bundle adjustment and map generation. Our approach outputs an accurate geo-referenced dense point-cloud which was validated in simulation on a synthetic dataset and on two real-world scenarios based on ground control points.

In this paper, the design, modeling and experimental verification of a large workspace, parallel aerial manipulator is presented. The proposed manipulator has 3 Degrees of Freedom (DoFs) and enables physical interaction on the sides, as well as below the aerial robot. The design parameters of the manipulator are chosen such that it achieves large and singularity-free workspace in combination with high dexterity. A Global Conditioning Index (GCI) is defined and used as a performance index for the manipulator over its complete workspace. Given the manipulator design parameters, a holistic model of the redundant aerial manipulation system is derived capturing the coupled dynamics of the aerial vehicle and the manipulator. Free flight experimental studies are utilized to validate the system, demonstrate its redundant DoFs, and evaluate its performance.

With robotic systems reaching considerable maturity in basic self-localization and environment mapping, new research avenues open up pushing for interaction of a robot with its surroundings for added autonomy. However, the transition from traditionally sparse feature-based maps to dense and accurate scene-estimation imperative for realistic manipulation is not straightforward. Moreover, achieving this level of scene perception in real-time from a computationally constrained and highly shaky and agile platform, such as a small an Unmanned Aerial Vehicle (UAV) is perhaps the most challenging scenario for perception for manipulation. Drawing inspiration from otherwise computationally constraining Computer Vision techniques, we present a system combining visual, inertial and depth information to achieve dense, local scene reconstruction of high precision in real-time. Our evaluation testbed is formed using ground-truth not only in the pose of the sensor-suite, but also the scene reconstruction using a highly accurate laser scanner, offering unprecedented comparisons of scene estimation to ground-truth using real sensor data. Given the lack of any real, ground-truth datasets for environment reconstruction, our V4RL Dense Surface Reconstruction dataset is publicly available.

Multirotor unmanned aerial vehicles (UAVs) are rapidly gaining popularity for many applications. However, safe operation in partially unknown, unstructured environments remains an open question. In this paper, we present a continuous-time trajectory optimization method for real-time collision avoidance on multirotor UAVs. We then propose a system where this motion planning method is used as a local replanner, that runs at a high rate to continuously recompute safe trajectories as the robot gains information about its environment. We validate our approach by comparing against existing methods and demonstrate the complete system avoiding obstacles on a multirotor UAV platform.

This paper reports on a data-driven motion planning approach for interaction-aware, socially-compliant robot navigation among human agents. Autonomous mobile robots navigating in workspaces shared with human agents require motion planning techniques providing seamless integration and smooth navigation in such. Smooth integration in mixed scenarios calls for two abilities of the robot: predicting actions of others and acting predictably for them. The former requirement requests trainable models of agent behaviors in order to accurately forecast their actions in the future, taking into account their reaction on the robot's decisions. A human-like navigation style of the robot facilitates other agents-most likely not aware of the underlying planning technique applied-to predict the robot motion vice versa, resulting in smoother joint navigation. The approach presented in this paper is based on a feature-based maximum entropy model and is able to guide a robot in an unstructured, real-world environment. The model is trained to predict joint behavior of heterogeneous groups of agents from onboard data of a mobile platform. We evaluate the benefit of interaction-aware motion planning in a realistic public setting with a total distance traveled of over 4 km. Interestingly the motion models learned from human-human interaction did not hold for robot-human interaction, due to the high attention and interest of pedestrians in testing basic braking functionality of the robot.

Having a team of robots to perform a task such as mapping is faster and more reliable than doing the same with a single robot, which can be crucial in scenarios such as search and rescue. We are developing a fully distributed framework for collaborative mapping with large robot swarms that is robust to abrupt departure of robots due to malfunctions or network problems. While several approaches to multi-robot mapping have been proposed, most of them either build a collection of local sub-maps, or rely on a central authority to merge maps built by individual robots. Our framework is unique in that it requires no central authority, yet allows robots to simultaneously contribute to a single global map, which is stored in a decentralized fashion. This greatly improves the scalability of our system with respect to number of robots. However, our approach requires systematic coordination among robots in order to make modifications to the map. Unannounced departure of the robots makes coordination challenging, and can potentially make the map inconsistent or result in loss of data. We borrow ideas from the domain of distributed computing to address those challenges. Further, we demonstrate the robustness of the proposed system by subjecting it to various conditions in which participating robots fail.

We present an aerial robotic platform for remote tree cavity inspection, based on a hexacopter Micro-Aerial vehicle (MAV) equipped with a dexterous manipulator. The goal is to make the inspection process safer and more efficient and facilitate data collection about tree cavities, which are important for the conservation of biodiversity in forest ecosystems. This work focuses on two key enabling technologies, namely a vision-based cavity detection system and strategies for high level control of the MAV and manipulator. The results of both simulation and real-world experiments are discussed at the end of the paper and demonstrate the effectiveness of our approach.

This paper presents a framework for planning safe and efficient paths for a legged robot in rough and unstructured terrain. The proposed approach allows to exploit the distinctive obstacle negotiation capabilities of legged robots, while keeping the complexity low enough to enable planning over considerable distances in short time. We compute typical terrain characteristics such as slope, roughness, and steps to build a traversability map. This map is used to assess the costs of individual robot footprints as a function of the robot-specific obstacle negotiating capabilities for steps, gaps and stairs. Our sampling-based planner employs the RRT* algorithm to optimize path length and safety. The planning framework has a hierarchical architecture to frequently replan the path during execution as new terrain is perceived with onboard sensors. Furthermore, a cascaded planning structure makes use of different levels of simplification to allow for fast search in simple environments, while retaining the ability to find complex solutions, such as paths through narrow passages. The proposed navigation planning framework is integrated on the quadrupedal robot StarlETH and extensively tested in simulation as well as on the real platform.

Automated valet parking services provide great potential to increase the attractiveness of electric vehicles by mitigating their two main current deficiencies: reduced driving ranges and prolonged refueling times. The European research project V-Charge aims at providing this service on designated parking lots using close-to-market sensors only. For this purpose the project developed a prototype capable of performing fully automated navigation in mixed traffic on designated parking lots and GPS-denied parking garages with cameras and ultrasonic sensors only. This paper summarizes the work of the project, comprising advances in network communication and parking space scheduling, multi-camera calibration, semantic mapping concepts, visual localization and motion planning. The project pushed visual localization, environment perception and automated parking to centimetre precision. The developed infrastructure-based camera calibration and semi-supervised semantic mapping concepts greatly reduce maintenance efforts. Results are presented from extensive month-long field tests.

Future automated driving systems will require a comprehensive scene understanding. Considering these systems in an urban environment it becomes immediately clear that reasoning about the future behavior and trajectories of pedestrians represents one major challenge. In this paper we focus on predicting the pedestrians' time-to-cross when approaching a crosswalk. Due to the complexity of the underlying model, we propose a data-driven approach that by means of regression models learns the target variable. Instead of utilizing a standard mean regression, we propose the use of Quantile Regression. We show that this special type of regression is more suited to describe the variability of real world pedestrian trajectories. We examine and compare two approaches: Linear Quantile Regression and Quantile Regression Forest, which is an extended version of Random Forests. We present evaluations with real data and a detailed analysis emphasizing strengths and weaknesses of quantile regression for the target application.

This paper presents underwater 3D capture using a commercial depth camera. Previous underwater capture systems use ordinary cameras, and it is well-known that a calibration procedure is needed to handle refraction. The same is true for a depth camera being used underwater. We describe a calibration method that corrects the depth maps of refraction effects. Another challenge is that depth cameras use infrared light (IR) which is heavily attenuated in water. We demonstrate scanning is possible with commercial depth cameras for ranges up to 20 cm in water. The motivation for using a depth camera under water is the same as in air — it provides dense depth data and higher quality 3D reconstruction than multi-view stereo. Underwater 3D capture is being increasingly used in marine biology and oceanology; our approach offers exciting prospects for such applications. To the best of our knowledge, ours is the first approach that successfully demonstrates underwater 3D capture using low cost depth cameras like Intel RealSense. We describe a complete system, including protective housing for the depth camera which is suitable for handheld use by a diver. Our main contribution is an easy-to-use calibration method, which we evaluate on exemplar data as well as 3D reconstructions in a lab aquarium. We also present initial results of ocean deployment.

With growing demands for accuracy in sensor fusion, increasing attention is being paid to temporal offsets as a source of deterministic error when processing data from multiple devices. Established approaches for the calibration of temporal offsets exploit domain-specific heuristics of common sensor suites and utilize simplifications to circumvent some of the challenges arising when both temporal and spatial parameters are not accurately known a priori. These properties make it difficult to generalize the work to other applications or different combinations of sensors. This work presents a general and principled approach to joint estimation of temporal offsets and spatial transformations between sensors. Our framework exploits recent advances in continuous-time batch estimation and thus exists within the rigorous theoretical framework of maximum likelihood estimation. The derivation is presented without relying on unique properties of specific sensors and, therefore, represents the first general technique for temporal calibration in robotics. The broad applicability of this approach is demonstrated through spatiotemporal calibration of a camera with respect to an inertial measurement unit as well as between a stereo camera and a laser range finder. The method is shown to be more repeatable and accurate than the current state of the art, estimating spatial displacements to millimeter precision and temporal offsets to a fraction of the fastest measurement interval.

Recent developments in Unmanned Aerial Vehicles (UAVs) have made them ideal tools for remotely monitoring agricultural fields. Complementary advancements in computer vision have enabled automated post-processing of images to generate dense 3D reconstructions in the form of point clouds. In this paper we present a monitoring pipeline that uses a readily available, low cost UAV and camera for quickly surveying a winter wheat field, generate a 3D point cloud from the collected imagery and present methods for automated crop height estimation from the extracted point cloud and compare our estimates with those using standardized techniques.

M-estimators are the de-facto standard method of robust estimation in robotics. They are easily incorporated into iterative non-linear least-squares estimation and provide seamless and effective handling of outliers in data. However, every M-estimator's robust loss function has one or more tuning parameters that control the influence of different data. The choice of M-estimator and the manual tuning of these parameters is always a source of uncertainty when applying the technique to new data or a new problem. In this paper we develop the concept of self-tuning M-estimators. We first make the connection between many common M-estimators and elliptical probability distributions. This connection shows that the choice of M-estimator is an assumption that the residuals belong to a well-defined elliptical distribution. We exploit this implication in two ways. First, we develop an algorithm for tuning the M-estimator parameters during iterative optimization. Second, we show how to choose the correct M-estimator for your data by examining the likelihood of the data given the model. We fully derive these algorithms and show their behavior on a representative example of visual simultaneous localization and mapping.

This paper presents a formalism that exploits deformability during manipulation of soft objects by robot teams. A hybrid centralized/distributed approach restricts centralized planning to high-level global guidance of the object for consensus. Low-level control is thus delegated to the individual manipulator robots, which retain manipulation and collision avoidance guarantees by passing forces to one another through the object. A distributed receding horizon planner provides local control, formulated as a convex optimization problem in velocity space and incorporating constraints for both collision avoidance and shape maintenance. We demonstrate teams of mobile manipulators autonomously carrying various deformable objects.

A taxonomy for gesture-based interaction between a human and a group (swarm) of robots is described. Methods are classified into two categories. First, free-form interaction, where the robots are unconstrained in position and motion and the user can use deictic gestures to select subsets of robots and assign target goals and trajectories. Second, shape-constrained interaction, where the robots are in a configuration shape that can be modified by the user. In the later, the user controls a subset of meaningful degrees of freedom defining the overall shape instead of each robot directly. A multi-robot interactive display is described where a depth sensor is used to recognize human gesture, determining the commands sent to a group comprising tens of robots. Experimental results with a preliminary user study show the usability of the system.

Within this paper, a new fast algorithm that provides efficient solutions to the problem of inspection path planning for complex 3D structures is presented. The algorithm assumes a triangular mesh representation of the structure and employs an alternating two-step optimization paradigm to find good viewpoints that together provide full coverage and a connecting path that has low cost. In every iteration, the viewpoints are chosen such that the connection cost is reduced and, subsequently, the tour is optimized. Vehicle and sensor limitations are respected within both steps. Sample implementations are provided for rotorcraft and fixed-wing unmanned aerial systems. The resulting algorithm characteristics are evaluated using simulation studies as well as multiple real-world experimental test-cases with both vehicle types.

In this work, we present a model-based estimation scheme for multi-rotor Micro Aerial Vehicles (MAVs). Although modeling approaches for MAVs have been presented in the past, these models have rarely been used for real-time state estimation onboard MAVs. Building on this work, we identify the most dominant effects and propose an easy-to-use calibration scheme for estimation of the model parameters. Given the calibration estimates for these parameters, we derive a state estimator where the state prediction of the indirect Extended Kalman Filter (EKF) is driven by a MAV model. Solely using measurements from the Inertial Measurement Unit (IMU) and a barometric pressure sensor - both available on almost every MAV - our model-based formulation keeps the estimated velocity of the MAV bounded in all directions, as opposed to state of the art IMU-model driven state estimators onboard MAVs. This is crucial for keeping MAVs airborne safely, for instance in the case of failures or re-initialization of vision based localization systems.

Large scale, long-term, distributed mapping is a core challenge to modern field robotics. Using the sensory output of multiple robots and fusing it in an efficient way enables the creation of globally accurate and consistent metric maps. To combine data from multiple agents into a global map, most existing approaches use a central entity that collects and manages the information from all agents. Often, the raw sensor data of one robot needs to be made available to processing algorithms on other agents due to the lack of computational resources on that robot. Unfortunately, network latency and low bandwidth in the field limit the generality of such an approach and make multi-robot map building a tedious task. In this paper, we present a distributed and decentralized back-end for concurrent and consistent robotic mapping. We propose a set of novel approaches that reduce the bandwidth usage and increase the effectiveness of inter-robot communication for distributed mapping. Instead of locking access to the map during operations, we define a version control system which allows concurrent and consistent access to the map data. Updates to the map are then shared asynchronously with agents which previously registered notifications. A technique for data lookup is provided by state-of-the-art algorithms from distributed computing. We validate our approach on real-world datasets and demonstrate the effectiveness of the proposed algorithms.

Robust, scalable place recognition is a core competency for many robotic applications. However, when revisiting places over and over, many state-of-the-art approaches exhibit reduced performance in terms of computation and memory complexity and in terms of accuracy. For successful deployment of robots over long time scales, we must develop algorithms that get better with repeated visits to the same environment, while still working within a fixed computational budget. This paper presents and evaluates an algorithm that alternates between online place recognition and offline map maintenance with the goal of producing the best performance with a fixed map size. At the core of the algorithm is the concept of a Summary Map, a reduced map representation that includes only the landmarks that are deemed most useful for place recognition. To assign landmarks to the map, we use a scoring function that ranks the utility of each landmark and a sampling policy that selects the landmarks for each place. The Summary Map can then be used by any descriptor-based inference method for constant-complexity online place recognition. We evaluate a number of scoring functions and sampling policies and show that it is possible to build and maintain maps of a constant size and that place-recognition performance improves over multiple visits.

This paper presents the conceptual design, detailed development and flight testing of AtlantikSolar, a 5.6m-wingspan solar-powered Low-Altitude Long-Endurance (LALE) Unmanned Aerial Vehicle (UAV) designed and built at ETH Zurich. The UAV is required to provide perpetual endurance at a geographic latitude of 45°N in a 4-month window centered around June 21st. An improved conceptual design method is presented and applied to maximize the perpetual flight robustness with respect to local meteorological disturbances such as clouds or winds. Airframe, avionics hardware, state estimation and control method development for autonomous flight operations are described. Flight test results include a 12-hour flight relying solely on batteries to replicate night-flight conditions. In addition, we present flight results from Search-And-Rescue field trials where a camera and processing pod were mounted on the aircraft to create high-fidelity 3D-maps of a simulated disaster area.

Real-time dense mapping and pose estimation is essential for a wide range of navigation tasks in mobile robotic applications. We propose an odometry and mapping system that leverages the full photometric information from a stereo-vision system as well as inertial measurements in a probabilistic framework while running in real-time on a single low-power Intel CPU core. Instead of performing mapping and localization on a set of sparse image features, we use the complete dense image intensity information in our navigation system. By incorporating a probabilistic model of the stereo sensor and the IMU, we can robustly estimate the ego-motion as well as a dense 3D model of the environment in real-time. The probabilistic formulation of the joint odometry estimation and mapping process enables to efficiently reject temporal outliers in ego-motion estimation as well as spatial outliers in the mapping process. To underline the versatility of the proposed navigation system, we evaluate it in a set of experiments on a multi-rotor system as well as on a quadrupedal walking robot. We tightly integrate our framework into the stabilization-loop of the UAV and the mapping framework of the walking robot. It is shown that the dense framework exhibits good tracking and mapping performance in terms of accuracy as well as robustness in scenarios with highly dynamic motion patterns while retaining a relatively small computational footprint. This makes it an ideal candidate for control and navigation tasks in unstructured GPS-denied environments, for a wide range of robotic platforms with power and weight constraints. The proposed framework is released as an open-source ROS package.

This paper presents a fast collision-detection method for sampling-based motion planners based on bounding volume hierarchies in workspace-time space. By introducing time as an additional dimension to the robot's workspace, the method is able to quickly evaluate time-indexed candidate trajectories for collision with the known future motions of other agents. The approach makes no assumptions on the shape of the objects and is able to handle arbitrary motions. We highlight implementation details regarding the application of the collision detection technique within an online planning framework for automated driving. Furthermore, we give detailed profiling information to show the capability for real-time operation.

In this paper, we present a monocular visual-inertial odometry algorithm which, by directly using pixel intensity errors of image patches, achieves accurate tracking performance while exhibiting a very high level of robustness. After detection, the tracking of the multilevel patch features is closely coupled to the underlying extended Kalman filter (EKF) by directly using the intensity errors as innovation term during the update step. We follow a purely robocentric approach where the location of 3D landmarks are always estimated with respect to the current camera pose. Furthermore, we decompose landmark positions into a bearing vector and a distance parametrization whereby we employ a minimal representation of differences on a corresponding σ-Algebra in order to achieve better consistency and to improve the computational performance. Due to the robocentric, inverse-distance landmark parametrization, the framework does not require any initialization procedure, leading to a truly power-up-and-go state estimation system. The presented approach is successfully evaluated in a set of highly dynamic hand-held experiments as well as directly employed in the control loop of a multirotor unmanned aerial vehicle (UAV).

In this work, we present an MAV system that is able to relocalize itself, create consistent maps and plan paths in full 3D in previously unknown environments. This is solely based on vision and IMU measurements with all components running onboard and in real-time. We use visual-inertial odometry to keep the MAV airborne safely locally, as well as for exploration of the environment based on high-level input by an operator. A globally consistent map is constructed in the background, which is then used to correct for drift of the visual odometry algorithm. This map serves as an input to our proposed global planner, which finds dynamic 3D paths to any previously visited place in the map, without the use of teach and repeat algorithms. In contrast to previous work, all components are executed onboard and in real-time without any prior knowledge of the environment.

Robust, scalable localization unlocks path-planning, obstacle avoidance as well as manipulation and thus is a core competency for many robotic applications. However, as we leave the lab and move out in the world, models of the environment no longer span distances of meters but kilometers in length. Now, gigabytes instead of megabytes of memory are required to hold the model of the environment required for localization. Discarding data and keeping the map representation compact is thus essential for any meaningful application. This paper presents and evaluates a map compression algorithm that approaches this data-reduction as an constrained optimization problem. At the core of the algorithm is the concept of a Summary Map, a reduced map representation that includes only the landmarks that are deemed most useful for place recognition. To assign landmarks to the map we have to satisfy the conflicting goals of map coverage and localizability as well as our tight memory budget. While using an optimization approach for compression is not novel, in this paper we propose adaptations to drastically reduce the computational requirements. Our approach improves scalability from trajectories of a few tens of meters manageable by the state of the art to virtually unlimited dataset sizes in our system. We evaluate the performance of various compression levels as well as several methods for selecting the best localization landmarks from outdoor datasets.

Quadrupedal locomotion on sloped terrains poses different challenges than walking in a mostly flat environment. The robot's configuration needs to be explicitly controlled in order to avoid slipping and kinematic limits. To this end, information about the terrain's inclination is required for carefully planning footholds, the pose of the main body, and modulation of the ground reaction forces. This is even more important for dynamic trotting, as only two support legs are available to compensate for gravity and drive a desired motion. We propose a reliable method for estimating the parameters of the terrain quadrupedal robots move on, in the face of limited perception capabilities and drifting robot pose estimates. By fusing inertial measurements, kinematic data from joint encoders and contact information from force sensors, the local inclination can be robustly estimated and used to optimize the contact forces to reduce slippage. The estimated terrain information, namely the pitch and roll angles of the ground plane, is exploited in an extended version of our previous model-based control approach. Our improved control framework enabled StarlETH, a medium-sized, fully autonomous, torque-controllable quadrupedal robot, to trot on slopes of up to 21°.

For autonomous navigation of Micro Aerial Vehicles (MAVs) in cluttered environments, it is essential to detect potential obstacles not only in the direction of flight but in their entire local environment. While there exist systems that do vision based obstacle detection, most of them are limited to a single perception direction. Extending these systems to a multi-directional sensing approach would exhaust the payload limit in terms of weight and computational power. We present a novel light-weight sensor setup comprising of four stereo heads and an inertial measurement unit (IMU) to perform FPGA-based dense reconstruction for obstacle detection in all directions. As the data-rate scales up with the number of cameras we use an FPGA to perform streaming based tasks in real-time and show a light-weight polar-coordinate map to allow a companion computer to fully process the data of all the cameras and perform obstacle detection in real-time. The system is able to process up to 80 frames per second (fps) freely distributed on the four stereo heads while maintaining a low power budget. The perception system including FPGA, image sensors and stereo mounts is 235 g in weight.

Methods of optimizing a single trajectory are mature enough for planning in many applications. Yet such optimization methods applied to high Degree-Of-Freedom robots either consume too much time to be real-time or approximate the dynamics such that they lack physical consistency. In this paper, we present a method of precomputing optimized trajectories and compressing the information to get a compact representation of the optimal policy function. By varying the initial configuration of a robot and optimizing multiple trajectories, the controller gains knowledge about the optimal policy function. Such computation can be performed on a powerful workstation or even supercomputers instead of an onboard computer of the robot. The precomputed optimal trajectories are stored in a Single-hidden Layer Feedforward neural Network (SLFN) using Optimally Pruned Extreme Learning Machine (OP-ELM). This ensures minimal representation of the model and fast evaluation of the SLFN. We first explain our method using a simple time-optimal control problem with an analytical solution. We then demonstrate how this method can work even for high dimensional state by optimizing a foothold strategy of a full quadruped robot in simulation.

In this paper, we present a novel approach in combining visual and inertial measurements in non-static environments for first order characterization of the metric motion of non-static objects in the scene. Our approach leverages online estimated ego motion states and uses a novel inertial-optical flow (IOF) measurement analysis to identify moving objects and to characterize them in their angular and linear velocities. The novelty of our algorithm lies in the identification and segmentation of consistent optical flow outliers in the so-called kinematic space. These consistent outliers in combination with the IOF information for ego-motion estimation yield a first order estimation of the moving object in full 3D and in metric units. The approach is highly efficient as it only requires matched features in two consecutive images. We evaluate and demonstrate our algorithm in simulations and in real world tests.

Localization is essential for robots to operate autonomously, especially for extended periods of time, when estimator drift tends to destroy alignment to any global map. Though there has been extensive work in vision-based localization in recent years, including several systems that show real-time performance, none have been demonstrated running entirely on-board in closed loop on robotic platforms. We propose a fast, real-time localization system that keeps the existing local visual-inertial odometry frame consistent for controllers and collision avoidance, while correcting drift and alignment to a global coordinate frame. We demonstrate our localization system entirely on-board an aerial and ground robot, showing a collaboration experiment where both robots are able to localize against the same map accurately enough to allow the multicopter to land on top of the ground robot. We also perform extensive evaluations for the proposed closed-loop system on ground-truth datasets from MAV flight in an industrial setting.

This paper describes human classifiers that are 'viewpoint specific', meaning specific to subjects being observed by a particular camera in a particular scene. The advantages of the approach are (a) improved human detection in the presence of perspective foreshortening from an elevated camera, (b) ability to handle partial occlusion of subjects e.g. partial occlusion by furniture in an indoor scene, and (c) ability to detect subjects when partially truncated at the top, bottom or sides of the image. Elevated camera views will typically generate truncated views for subjects at the image edges but our viewpoint specific method handles such cases and thereby extends overall detection coverage. The approach is - (a) define a tiling on the ground plane of the 3D scene, (b) generate training images per tile using virtual humans, (c) train a classifier per tile (d) run the classifiers on the real scene. The approach would be prohibitive if each new deployment required real training images, but it is feasible because training is done with a virtual humans inserted into a scene model. The classifier is a linear SVM and HOGs. Experimental results provide a comparative analysis with existing algorithms to demonstrate the advantages described above.

Place recognition is a core competency for any visual simultaneous localization and mapping system. Identifying previously visited places enables the creation of globally accurate maps, robust relocalization, and multi-user mapping. To match one place to another, most state-of-the-art approaches must decide a priori what constitutes a place, often in terms of how many consecutive views should overlap, or how many consecutive images should be considered together. Unfortunately, depending on thresholds such as these, limits their generality to different types of scenes. In this paper, we present a placeless place recognition algorithm using a novel vote-density estimation technique that avoids heuristically discretizing the space. Instead, our approach considers place recognition as a problem of continuous matching between image streams, automatically discovering regions of high vote density that represent overlapping trajectory segments. The resulting algorithm has a single free parameter and all remaining thresholds are set automatically using well-studied statistical tests. We demonstrate the efficiency and accuracy of our methodology on three outdoor sequences: A comprehensive evaluation against ground-truth from publicly available datasets shows that our approach outperforms several state-of-the-art algorithms for place recognition.

This work shows experimental results with three systems that take real-time user input to direct a robot swarm formed by tens of small robots. These are: real-time drawing, gesture based interaction with an RGB-D sensor and control via a hand-held tablet computer.

This paper presents a method for shared control of a vehicle. The driver commands a preferred velocity which is transformed into a collision-free local motion that respects the actuator constraints and allows for smooth and safe control. Collision-free local motions are achieved with an extension of velocity obstacles that takes into account dynamic constraints and a grid-based map representation. To limit the freedom of the driver, a global guidance trajectory can be included, which specifies the areas where the vehicle is allowed to drive in each time instance. The low computational complexity of the method makes it well suited for multi-agent settings and high update rates and both a centralized and a distributed algorithm are provided that allow for real-time control of tens of vehicles. Extensive experimental results with real robotic wheelchairs at relatively high speeds in tight scenarios are presented.

The challenge of aerial robotic physical interaction towards inspection of infrastructure facilities through contact is the main motivation of this paper. A hybrid model predictive control framework is proposed, based on which a typical quadrotor vehicle becomes capable of stable physical interaction, accurate trajectory tracking on environmental surfaces as well as force control with only minor structural adaptations. Convex optimization techniques enabled the explicit computation of such a controller which accounts for the dynamics in free-flight and during physical interaction, ensures the stability of the hybrid system as well as response optimality, while respecting system constraints and imposed logical rules. This control framework is further extended to include obstacle avoidance capabilities. Extensive experimental studies that included complex “aerial-writing” tasks, interaction with non-planar and textured surfaces and obstacle avoidance maneuvers, indicate the efficiency of the approach and the potential capabilities of such aerial robotic physically interacting operations.

Developing control methods that allow legged robots to move with skill and agility remains one of the grand challenges in robotics. In order to achieve this ambitious goal, legged robots must possess a wide repertoire of motor skills. A scalable control architecture that can represent a variety of gaits in a unified manner is therefore desirable. Inspired by the motor learning principles observed in nature, we use an optimization approach to automatically discover and fine-tune parameters for agile gaits. The success of our approach is due to the controller parameterization we employ, which is compact yet flexible, therefore lending itself well to learning through repetition. We use our method to implement a flying trot, a bound and a pronking gait for StarlETH, a fully autonomous quadrupedal robot.

Legged robots come in a range of sizes and capabilities. By combining these robots into heterogeneous teams, joint locomotion and perception tasks can be achieved by utilizing the diversified features of each robot. In this work we present a framework for using a heterogeneous team of legged robots to detect slippery terrain. StarlETH, a large and highly capable quadruped uses the VelociRoACH as a novel remote probe to detect regions of slippery terrain. StarlETH localizes the team using internal state estimation. To classify slippage of the VelociRoACH, we develop several Support Vector Machines (SVM) based on data from both StarlETH and VelociRoACH. By combining the team's information about the motion of VelociRoACH, a classifier was built which could detect slippery spots with 92% (125/135) accuracy using only four features.

The online recalibration of multi-sensor systems is a fundamental problem that must be solved before complex automated systems are deployed in situations such as automated driving. In such situations, accurate knowledge of calibration parameters is critical for the safe operation of automated systems. However, most existing calibration methods for multisensor systems are computationally expensive, use installations of known fiducial patterns, and require expert supervision. We propose an alternative approach called infrastructure-based calibration that is efficient, requires no modification of the infrastructure, and is completely unsupervised. In a survey phase, a computationally expensive simultaneous localization and mapping (SLAM) method is used to build a highly accurate map of a calibration area. Once the map is built, many other vehicles are able to use it for calibration as if it were a known fiducial pattern. We demonstrate the effectiveness of this method to calibrate the extrinsic parameters of a multi-camera system. The method does not assume that the cameras have an overlapping field of view and it does not require an initial guess. As the camera rig moves through the previously mapped area, we match features between each set of synchronized camera images and the map. Subsequently, we find the camera poses and inlier 2D-3D correspondences. From the camera poses, we obtain an initial estimate of the camera extrinsics and rig poses, and optimize these extrinsics and rig poses via non-linear refinement. The calibration code is publicly available as a standalone C++ package.

Robotic sensors are promising instruments for monitoring spatial phenomena. Oftentimes, rather than aiming to achieve low prediction error everywhere, one is interested in determining whether the phenomenon exhibits certain critical behavior. In this paper, we consider the problem of focusing autonomous sampling to determine whether and where the sensed spatial field exceeds a given threshold value. We introduce a receding horizon path planner, LSE-DP, which plans efficient paths for sensing in order to reduce our uncertainty specifically around the threshold value. We report fully autonomous field experiments with an Autonomous Surface Vessel (ASV) in an aquatic monitoring setting, which demonstrate the effectiveness of the proposed method. LSE-DP is able to reduce the uncertainty around the threshold value of interest to 68% when compared to non-adaptive methods.

This paper describes a hierarchical operational space control (OSC) method based on least square optimization and outlines different ways to reduce the dimensionality of the optimization vector. The framework allows to emulate various behaviors by prioritized task-space motion, joint torque, and contact force optimization. Moreover, a methodology is introduced to partially excite the natural dynamics of the robot by open-loop motor regulation while the entire behavior is stabilized by hierarchical OSC. As a major contribution, the presented control strategies are tested and validated in real hardware walking, trotting, and pronking experiments using a fully torque controllable quadrupedal robot.

Mobile wheeled- or tracked-robots drive in 2.5-dimensional (2.5D) environments, where the traversable surface can be considered as a 2D-manifold embedded in a three-dimensional (3D) ambient space. In this work, we aim at solving the 2.5D navigation problem solely on point-cloud. The proposed method is independent of traditional surface parametrization or reconstruction methods, such as a meshing process, which generally has high computational complexity. Instead, we utilize the output of 3D tensor voting framework (TVF) using raw point-clouds. A novel local Riemannian metric is defined based on the saliency components of TVF, which helps the modeling of the latent traversable surface. Using this metric, we prove that the geodesic in the 3D tensor space leads to rational path-planning results. Compared to traditional methods, the results reveal the advantages of the proposed method in terms of facilitating the robot maneuver with minimum movement.

Robust, accurate pose estimation and mapping at real-time in six dimensions is a primary need of mobile robots, in particular flying Micro Aerial Vehicles (MAVs), which still perform their impressive maneuvers mostly in controlled environments. This work presents a visual-inertial sensor unit aimed at effortless deployment on robots in order to equip them with robust real-time Simultaneous Localization and Mapping (SLAM) capabilities, and to facilitate research on this important topic at a low entry barrier. Up to four cameras are interfaced through a modern ARM-FPGA system, along with an Inertial Measurement Unit (IMU) providing high-quality rate gyro and accelerometer measurements, calibrated and hardware-synchronized with the images. This facilitates a tight fusion of visual and inertial cues that leads to a level of robustness and accuracy which is difficult to achieve with purely visual SLAM systems. In addition to raw data, the sensor head provides FPGA-pre-processed data such as visual keypoints, reducing the computational complexity of SLAM algorithms significantly and enabling employment on resource-constrained platforms. Sensor selection, hardware and firmware design, as well as intrinsic and extrinsic calibration are addressed in this work. Results from a tightly coupled reference visual-inertial motion estimation framework demonstrate the capabilities of the presented system.

New applications of mobile robotics in dynamic urban areas require more than the single-session geometric maps that have dominated simultaneous localization and mapping (SLAM) research to date; maps must be updated as the environment changes and include a semantic layer (such as road network information) to aid motion planning in dynamic environments. We present an algorithm for long-term localization and mapping in real time using a three-dimensional (3D) laser scanner. The system infers the static or dynamic state of each 3D point in the environment based on repeated observations. The velocity of each dynamic point is estimated without requiring object models or explicit clustering of the points. At any time, the system is able to produce a most-likely representation of underlying static scene geometry. By storing the time history of velocities, we can infer the dominant motion patterns within the map. The result is an online mapping and localization system specifically designed to enable long-term autonomy within highly dynamic environments. We validate the approach using data collected around the campus of ETH Zurich over seven months and several kilometers of navigation. To the best of our knowledge, this is the first work to unify long-term map update with tracking of dynamic objects.

Detection and tracking of people in visible-light images has been subject to extensive research in the past decades with applications ranging from surveillance to search-and-rescue. Following the growing availability of thermal cameras and the distinctive thermal signature of humans, research effort has been focusing on developing people detection and tracking methodologies applicable to this sensing modality. However, a plethora of challenges arise on the transition from visible-light to thermal images, especially with the recent trend of employing thermal cameras onboard aerial platforms (e.g. in search-and-rescue research) capturing oblique views of the scenery. This paper presents a new, publicly available dataset of annotated thermal image sequences, posing a multitude of challenges for people detection and tracking. Moreover, we propose a new particle filter based framework for tracking people in aerial thermal images. Finally, we evaluate the performance of this pipeline on our dataset, incorporating a selection of relevant, state-of-the-art methods and present a comprehensive discussion of the merits spawning from our study.

This paper presents a method to optimize the position and trajectory of each aerial vehicle within a large group that displays objects and animations in 3D space. The input is a single object or an animation created by an artist. In a first step, goal positions for the given number of vehicles and representing the object are optimized with respect to a known viewpoint. For displaying an animation, an optimal trajectory satisfying the dynamic constraints of each vehicle is computed using B-splines. Finally, a trajectory following controller is described, which provides the preferred velocity, later optimized to be collision-free with respect to all neighboring vehicles.

In this paper we present a method for fusing optical flow and inertial measurements. To this end, we derive a novel visual error term which is better suited than the standard continuous epipolar constraint for extracting the information contained in the optical flow measurements. By means of an unscented Kalman filter (UKF), this information is then tightly coupled with inertial measurements in order to estimate the egomotion of the sensor setup. The individual visual landmark positions are not part of the filter state anymore. Thus, the dimensionality of the state space is significantly reduced, allowing for a fast online implementation. A nonlinear observability analysis is provided and supports the proposed method from a theoretical side. The filter is evaluated on real data together with ground truth from a motion capture system.

This work presents a novel approach to spatio-temporal calibration of a laser range finder (LRF) with respect to a combination of a stereo camera and an inertial measurement unit (IMU). Spatial calibration between an LRF and a camera has been extensively studied, but so far the temporal relationship between the two has largely been neglected. While this may be sufficient for applications where the setup is mounted on a vehicle, which imposes bounds on the dynamics, we aim for employment on a hand-held scanning device, where angular velocities can easily exceed hundreds of degrees per second. Employing a continuous-time batch estimation framework, this work demonstrates that the transformation between the LRF and the visual/inertial setup-but also its temporal relationship-can be estimated accurately. In contrast to the majority of established calibration approaches, our approach does not require an overlap in the field of view of the LRF and camera, allowing for previously infeasible sensor configurations to be calibrated. Preliminary results for a novel hand-held scanning device suggest improvements in 3D reconstructions and image based point cloud coloring, especially for highly dynamic motions.

In this paper, we present a vision-based path following algorithm for a non-holonomic wheeled platform. The algorithm is based on choosing control actions that minimise the value of a potential field cost function calculated directly from the image plane. The algorithm is suitable for teach and replay or leader follower implementations where the desired path is represented as a collection of images. The algorithm computes the cost function based on the relative bearings of features matched between the current and previously observed images. A forward prediction step is then used to determine the control action that will lead to the greatest reduction in the cost function. The algorithm is demonstrated on a 400 m path in an outdoor environment where the accuracy is shown to be similar to that of differential GPS.

We implemented single-session workshops using the Thymio-II--a small, self-contained robot designed for young students, and VPL--a graphical software development environment based upon event handling. Our goal was to investigate if the students could learn this core computer science concept while enjoying themselves in the robotics context. A visual questionnaire was developed based upon the combined Bloom and SOLO taxonomies, although it proved difficult to construct a questionnaire appropriate for young students. We found that--despite the short duration of the workshop--all but the youngest students achieved the cognitive level of Unistructural Understanding, while some students achieved higher levels of Unistructural Applying. and Multistructural Understanding and Applying.

This paper provides an overview about StarlETH: a compliant quadrupedal robot that is designed to study fast, efficient, versatile, and robust locomotion. The platform is driven by highly compliant series elastic actuation, which makes the system fully torque controllable, energetically efficient, and well suited for dynamic maneuvers. Using model-based control strategies, this medium dog-sized machine is capable of various gaits ranging from static walking to dynamic running over challenging terrain. StarlETH is equipped with an onboard PC, batteries, and various sensor equipment that enables enduring autonomous operation. In this paper, we provide an overview about the underlying locomotion control algorithms, outline a real-time control and simulation environment, and conclude the work with a number of experiments to demonstrate the performance of the presented hardware and controllers.

Scene recognition problems for mobile robots have been extensively studied. This is important for tasks such as visual topological mapping. Usually, sophisticated key-point-based descriptors are used, which can be computationally expensive. In this paper, we describe a lightweight novel scene recognition method using an adaptive descriptor, which is based on color features and geometric information that are extracted from an uncalibrated omnidirectional camera. The proposed method enables a mobile robot to perform online registration of new scenes onto a topological representation automatically and solve the localization problem to topological regions simultaneously, all in real time. We adopt a Dirichlet process mixture model (DPMM) to describe the online inference process. It is based on an approximation of conditional probabilities of the new measurements given incrementally estimated reference models. It enables online inference speeds of up to 50 Hz for a normal CPU. We compare it with state-of-the-art key-point descriptors and show the advantage of the proposed algorithm in terms of performance and computational efficiency. A real-world experiment is carried out with a mobile robot equipped with an omnidirectional camera. Finally, we show the results on extended datasets.

Rolling Shutter (RS) cameras are used across a wide range of consumer electronic devices-from smart-phones to high-end cameras. It is well known, that if a RS camera is used with a moving camera or scene, significant image distortions are introduced. The quality or even success of structure from motion on rolling shutter images requires the usual intrinsic parameters such as focal length and distortion coefficients as well as accurate modelling of the shutter timing. The current state-of-the-art technique for calibrating the shutter timings requires specialised hardware. We present a new method that only requires video of a known calibration pattern. Experimental results on over 60 real datasets show that our method is more accurate than the current state of the art.

With navigation algorithms reaching a certain maturity in the field of mobile robots, the community now focuses on more advanced tasks like path planning towards increased autonomy. While the goal is to efficiently compute a path to a target destination, the uncertainty in the robot's perception cannot be ignored if a realistic path is to be computed. With most state of the art navigation systems providing the uncertainty in motion estimation, here we propose to exploit this information. This leads to a system that can plan safe avoidance of obstacles, and more importantly, it can actively aid navigation by choosing a path that minimizes the uncertainty in the monitored states. Our proposed approach is applicable to systems requiring certain excitations in order to render all their states observable, such as a MAV with visual-inertial based localization. In this work, we propose an approach which takes into account this necessary motion during path planning: by employing Rapidly exploring Random Belief Trees (RRBT), the proposed approach chooses a path to a goal which allows for best estimation of the robot's states, while inherently avoiding motion in unobservable modes. We discuss our findings within the scenario of vision-based aerial navigation as one of the most challenging navigation problem, requiring sufficient excitation to reach full observability.

A new type of coaxial-rotor unmanned helicopter capable of physically interacting with its environment is the subject of this paper. Its design is optimized in order to provide the means of robust environmental interaction through contact (e.g. docking and sliding on walls). Due to the rapid change of the dynamics from the free-flying helicopter to the helicopter subject to the forces and moments during contact a hybrid systems modeling approach is followed. This global model of the system's dynamics is the basis for the design of a hybrid model predictive controller that guarantees the stability of the hybrid system and provides the capability of controlled docking on walls as well as sliding on them. The capabilities of the platform and the efficiency of the control law are illustrated through experimental studies.

In this paper a centralized method for collision avoidance among multiple agents is presented. It builds on the velocity obstacle (VO) concept and its extensions to arbitrary kino-dynamics and is applicable to heterogeneous groups of agents (with respect to size, kino-dynamics and aggressiveness) moving in 2D and 3D spaces. In addition, both static and dynamic obstacles can be considered in the framework. The method maximizes a joint utility function and is formulated as a mixed-integer quadratic program, where online computation can be achieved as a trade-off with solution optimality. In experiments with groups of two to 50 agents the benefits of the joint utility optimization are shown. By construction, it's suboptimal variant is at least as good as comparable decentralized methods, while retaining online capability for small groups of agents. In its optimal variant, the proposed algorithm can provide a benchmark for distributed collision avoidance methods, in particular for those based on the VO concept that take interaction into account.

This paper introduces a novel batch optimization based calibration framework for legged robots. Given a non-degenerate calibration dataset and considering the stochastic models of the sensors, the task is formulated as a maximum likelihood problem. In order to facilitate the derivation of consistent measurement equations, the trajectory of the robot and other auxiliary variables are included into the optimization problem. This formulation can be transformed into a nonlinear least squares problem which can be readily solved. Applied to our legged robot StarIETH, the framework estimates kinematic parameters (segment lengths, body dimensions, angular offsets), accelerometer and gyroscope biases, as well as full inter-sensor calibrations. The generic structure easily allows the inclusion of additional sensor modalities. Based on datasets obtained on the real robot the consistency and performance of the presented approach are successfully evaluated.

In this work, we propose a novel RFID-based hybrid metric-topological Simultaneous Localization and Mapping (SLAM) algorithm which enables autonomous navigation in GPS-denied environments. A method based on the normalized-cut is proposed for online clustering of strongly connected Radio Frequency Identification (RFID) tags to form topological nodes. A particle filter together with a sensor model which characterizes the received signal strength (RSS) as well as the tag detection probability is used to create metric submaps for each topological node. The hybrid framework is highly scalable, simplifies path planning and promises precision and robustness. The algorithm requires only odometry and RFID measurements to localize the RFID tags with a relative accuracy of approximately 0.3 meters. The ideas presented here are supported by experimental results.

Quadrupedal animals move through their environments with unmatched agility and grace. An important part of this is the ability to choose between different gaits in order to travel optimally at a certain speed or to robustly deal with unanticipated perturbations. In this paper, we present a control framework for a quadrupedal robot that is capable of locomoting using several gaits. We demonstrate the flexibility of the algorithm by performing experiments on StarlETH, a recently-developed quadrupedal robot. We implement controllers for a static walk, a walking trot, and a running trot, and show that smooth transitions between them can be performed. Using this control strategy, StarlETH is able to trot unassisted in 3D space with speeds of up to 0.7m/s, it can dynamically navigate over unperceived 5-cm high obstacles and it can recover from significant external pushes.

This paper presents a method for state estimation on a ballbot; i.e., a robot balancing on a single sphere. Within the framework of an extended Kalman filter and by utilizing a complete kinematic model of the robot, sensory information from different sources is combined and fused to obtain accurate estimates of the robot's attitude, velocity, and position. This information is to be used for state feedback control of the dynamically unstable system. Three incremental encoders (attached to the omniwheels that drive the ball of the robot) as well as three rate gyroscopes and accelerometers (attached to the robot's main body) are used as sensors. For the presented method, observability is proven analytically for all essential states in the system, and the algorithm is experimentally evaluated on the Ballbot Rezero.

This paper addresses the problem of evaluating and estimating the mechanical robustness of footholds for legged robots in unstructured terrain. In contrast to approaches that rely on human expert knowledge or human defined criteria to identify appropriate footholds, our method uses the robot itself to assess whether a certain foothold is adequate or not. To this end, one of the robot's legs is employed to haptically explore an unknown foothold. The robustness of the foothold is defined by a simple metric as a function of the achievable ground reaction forces. This haptic feedback is associated with the foothold shape to estimate the robustness of untouched footholds. The underlying shape clustering principles are tested on synthetic data and in hardware experiments using a single-leg testbed.

This paper describes an accurate and extendable rotorcraft dynamics simulator developed to support the design and control of autonomous coaxial rotor vehicles. This simulator is capable of accurately predicting the dynamic flight response of coaxial rotor vehicles purely based on geometric, inertial and aerodynamic specifications. The simulator is fully configurable and implements the typical mechanical layouts found in model-size coaxial helicopters. The corresponding software framework as well as the underlying theory is presented in detail. System parameters for a coaxial rotor prototype have been estimated and the resulting simulation results compared with real flight-data demonstrating the capabilities of the presented simulation software.

This paper introduces two novel solutions to the generalized-camera exterior orientation problem, which has a vast number of potential applications in robotics: (i) a minimal solution requiring only three point correspondences, and (ii) gPnP, an efficient, non-iterative n-point solution with linear complexity in the number of points. Already existing minimal solutions require exhaustive algebraic derivations. In contrast, our novel minimal solution is solved in a straightforward manner using the Gröbner basis method. Existing n-point solutions are mostly based on iterative optimization schemes. Our n-point solution is non-iterative and outperforms existing algorithms in terms of computational efficiency. Our results present an evaluation against state-of-the-art single-camera algorithms, and a comparison of different multi-camera setups. It demonstrates the superior noise resilience achieved when using multi-camera configurations, and the efficiency of our algorithms. As a further contribution, we illustrate a possible robotic use-case of our non-perspective orientation computation algorithms by presenting visual odometry results on real data with a non-overlapping multi-camera configuration, including a comparison to a loosely coupled alternative.

Omnidirectional cameras are widely used for robotic applications in structured environments. However, because of the distorted field of view (FOV), it is hard to describe the primitive features extracted from them robustly. In this paper, we tackle the problem by using Histogram of Gradient (HoG) statistics for the regions of interest (ROI) in the neighborhood of major vertical lines extracted from the panoramic image. As a validation, we compare the proposed algorithm with state-of-the-art based on two widely used data-sets, leading to evidently better performance. We also introduce a scene reconstruction scenario using the proposed descriptor based on 1D Trifocal Tensor framework. The comparative results show the competence of the descriptor.

In this paper we present a vision-based path following algorithm for a non-holonomic wheeled platform capable of keeping the vehicle on a desired path using only a single camera. The algorithm is suitable for teach and replay or leader-follower implementations where the desired path is represented by a collection of images obtained along the path. The algorithm makes use of the 1D trifocal tensor to estimate parameters required for the path following controller through a structure from motion approach. Our algorithm provides the benefits of a position-based visual servoing method without having to explicitly recover the 3D structure of the environment. By using the trifocal tensor, the proposed algorithm overcomes several problems usually associated with visual servoing techniques. The singularities commonly encountered with image-based visual servoing and epipolar methods are eliminated; and the unknown scale problem is also resolved by incorporating the scale into the control parameters. This simultaneously removes the velocity dependence of the controller gains and the need for odometric sensors on the platform. In addition, we also propose a novel method of resolving the ambiguities often associated with structure from motion when using the 1D trifocal tensor. The proposed algorithm is validated using both simulated and experimental results where robustness to a large degree of feature noise is demonstrated.

In this work, we present a powerful, albeit simple position control approach for Micro Aerial Vehicles (MAVs) targeting specifically multicopter systems. Exploiting the differential flatness of four of the six outputs of multicopters, namely position and yaw, we show that the remaining outputs of pitch and roll need not be controlled states, but rather just need to be known. Instead of the common approach of having multiple cascaded control loops (position - velocity - acceleration/attitude - angular rates), the proposed method employs an outer control loop based on dynamic inversion, which directly commands angular rates and thrust. The inner control loop then reduces to a simple proportional controller on the angular rates. As a result, not only does this combination allow for higher bandwidth compared to common control approaches, but also eliminates many mathematical operations (only one trigonometric function is called), speeding up the necessary processing especially on embedded systems. This approach assumes a reliable state estimation framework, which we are able to provide with through previous work. As a result, with this work, we provide the missing elements necessary for a complete approach on autonomous navigation of MAVs.

This paper presents a state estimation approach for legged robots based on stochastic filtering. The key idea is to extract information from the kinematic constraints given through the intermittent contacts with the ground and to fuse this information with inertial measurements. To this end, we design an unscented Kalman filter based on a consistent formulation of the underlying stochastic model. To increase the robustness of the filter, an outliers rejection methodology is included into the update step. Furthermore, we present the nonlinear observability analysis of the system, where, by considering the special nature of 3D rotations, we obtain a relatively simple form of the corresponding observability matrix. This yields, that, except for the global position and the yaw angle, all states are in general observable. This also holds if only one foot is in contact with the ground. The presented filter is evaluated on a real quadruped robot trotting over an uneven and slippery terrain.

This paper presents Skye, a novel blimp design. Skye is a helium-filled sphere of diameter 2.7m with a strong inelastic outer hull and an impermeable elastic inner hull. Four tetrahedrally-arranged actuation units (AU) are mounted on the hull for locomotion, with each AU having a thruster which can be rotated around a radial axis through the sphere center. This design provides redundant control in the six degrees of freedom of motion, and Skye is able to move omnidirectionally and to rotate around any axis. A multi-camera module is also mounted on the hull for capture of aerial imagery or live video stream according to an ‘eyeball’ concept — the camera module is not itself actuated, but the whole blimp is rotated in order to obtain a desired camera view. Skye is safe for use near people — the double hull minimizes the likelihood of rupture on an unwanted collision; the propellers are covered by grills to prevent accidental contact; and the blimp is near neutral buoyancy so that it makes only a light impact on contact and can be readily nudged away. The system is portable and deployable by a single operator — the electronics, AUs, and camera unit are mounted externally and are detachable from the hull during transport; operator control is via an intuitive touchpad interface. The motivating application is in entertainment robotics. Skye has a varied motion vocabulary such as swooping and bobbing, plus internal LEDs for visual effect. Computer vision enables interaction with an audience. Experimental results show dexterous maneuvers in indoor and outdoor environments, and non-dangerous impacts between the blimp and humans.

One milestone for autonomous mobile robotics is to endow robots with the capability to compute the plans and motor commands necessary to reach a defined goal position. For indoor or car-like robots moving on flat terrain, this problem is well mastered and open-source software can be deployed to such robots. However, for many applications such as search and rescue, ground robots must handle three-dimensional terrain. In this article, we present a system that is able to plan and execute a path in a complex environment starting from noisy sensor input. In order to cope with the complexity of a high-dimensional configuration space, we separate position and configuration planning. We demonstrate our system on a search and rescue robot with flippers by climbing up and down a difficult curved staircase.

This paper presents the application of reinforcement learning to improve the performance of highly dynamic single legged locomotion with compliant series elastic actuators. The goal is to optimally exploit the capabilities of the hardware in terms of maximum jump height, jump distance, and energy efficiency of periodic hopping. These challenges are tackled with the reinforcement learning method Policy Improvement with Path Integrals (PI2) in a model-free approach to learn parameterized motor velocity trajectories as well as highlevel control parameters. The combination of simulation and hardware-based optimization allows to efficiently obtain optimal control policies in an up to 10-dimensional parameter space. The robotic leg learns to temporarily store energy in the elastic elements of the joints in order to improve the jump height and distance. In addition, we present a method to learn time-independent control policies and apply it to improve the energetic efficiency of periodic hopping.

In order to increase accuracy and robustness in state estimation for robotics, a growing number of applications rely on data from multiple complementary sensors. For the best performance in sensor fusion, these different sensors must be spatially and temporally registered with respect to each other. To this end, a number of approaches have been developed to estimate these system parameters in a two stage process, first estimating the time offset and subsequently solving for the spatial transformation between sensors. In this work, we present on a novel framework for jointly estimating the temporal offset between measurements of different sensors and their spatial displacements with respect to each other. The approach is enabled by continuous-time batch estimation and extends previous work by seamlessly incorporating time offsets within the rigorous theoretical framework of maximum likelihood estimation. Experimental results for a camera to inertial measurement unit (IMU) calibration prove the ability of this framework to accurately estimate time offsets up to a fraction of the smallest measurement period.

This video presents experiments conducted within the final review meeting demonstration session of the AIRobots project. AIRobots started at 2010 and the final review meeting took place on 22 of March, 2013. The presented experiments cover a wide area of the challenges related with aerial industrial inspection. In particular, multiple test-cases related with both vision-based and contact-based inspection and in general physical interaction are shown. It is highlighted that these experiments were recorded live during the project demonstration and evaluation process.

It has been long known that fusing information from multiple sensors for robot navigation results in increased robustness and accuracy. However, accurate calibration of the sensor ensemble prior to deployment in the field as well as coping with sensor outages, different measurement rates and delays, render multi-sensor fusion a challenge. As a result, most often, systems do not exploit all the sensor information available in exchange for simplicity. For example, on a mission requiring transition of the robot from indoors to outdoors, it is the norm to ignore the Global Positioning System (GPS) signals which become freely available once outdoors and instead, rely only on sensor feeds (e.g., vision and laser) continuously available throughout the mission. Naturally, this comes at the expense of robustness and accuracy in real deployment. This paper presents a generic framework, dubbed MultiSensor-Fusion Extended Kalman Filter (MSF-EKF), able to process delayed, relative and absolute measurements from a theoretically unlimited number of different sensors and sensor types, while allowing self-calibration of the sensor-suite online. The modularity of MSF-EKF allows seamless handling of additional/lost sensor signals during operation while employing a state buffering scheme augmented with Iterated EKF (IEKF) updates to allow for efficient re-linearization of the prediction to get near optimal linearization points for both absolute and relative state updates. We demonstrate our approach in outdoor navigation experiments using a Micro Aerial Vehicle (MAV) equipped with a GPS receiver as well as visual, inertial, and pressure sensors.

This paper presents a novel underwater robot based on biological locomotion principle. A robotic platform imitating sea-turtle fin propulsion is described and tested. As fin locomotion is a novel and complex research area, basic control concepts are analyzed and implemented. Based on a simulation, a fin-trajectory morphing control strategy is developed in order to control the robots roll, pitch and yaw rates, thus allowing the robot to follow a given vector. Absolute position control or depth control, however, is not yet implemented. The paper concludes with the presentation of a working system that demonstrated motion capabilities in air as well as the first dive test in a swimming pool.

Future requirements for drastic reduction of CO2production and energy consumption will lead to significant changes in the way we see mobility in the years to come. However, the automotive industry has identified significant barriers to the adoption of electric vehicles, including reduced driving range and greatly increased refueling times. Automated cars have the potential to reduce the environmental impact of driving, and increase the safety of motor vehicle travel. The current state-of-the-art in vehicle automation requires a suite of expensive sensors. While the cost of these sensors is decreasing, integrating them into electric cars will increase the price and represent another barrier to adoption. The V-Charge Project, funded by the European Commission, seeks to address these problems simultaneously by developing an electric automated car, outfitted with close-to-market sensors, which is able to automate valet parking and recharging for integration into a future transportation system. The final goal is the demonstration of a fully operational system including automated navigation and parking. This paper presents an overview of the V-Charge system, from the platform setup to the mapping, perception, and planning sub-systems.

We present a generic algorithm for self calibration of robotic systems that utilizes two key innovations. First, it uses information theoretic measures to automatically identify and store novel measurement sequences. This keeps the computation tractable by discarding redundant information and allows the system to build a sparse but complete calibration dataset from data collected at different times. Second, as the full observability of the calibration parameters may not be guaranteed for an arbitrary measurement sequence, the algorithm detects and locks unobservable directions in parameter space using a truncated QR decomposition of the Gauss-Newton system. The result is an algorithm that listens to an incoming sensor stream, builds a minimal set of data for estimating the calibration parameters, and updates parameters as they become observable, leaving the others locked at their initial guess. Through an extensive set of simulated and real-world experiments, we demonstrate that our method outperforms state-of-the-art algorithms in terms of stability, accuracy, and computational efficiency.

In this paper a generic framework for sampling-based partial motion planning along a reference path is presented. The sampling mechanism builds on the specification of a vehicle model and a control law, both of which are freely selectable. Via a closed-loop forward simulation, the vehicle model is regulated onto a carefully chosen set of terminal states aligned with the reference path, generating system-compliant sample trajectories in accordance with the specified system and environmental constraints. The consideration of arbitrary state and input limits make this framework appealing to nonholonomic systems. The rich trajectory set is evaluated in an online sampling-based planning framework, targeting realtime motion planning in dynamic environments. In an example application, a Volkswagen Golf is modeled via a kinodynamic single-track system that is further constrained by steering angle/rate and velocity/acceleration limits. Control is implemented via state-feedback onto piecewise C0-continuous reference paths. Experiments demonstrate the planner's applicability to online operation, its ability to cope with discontinuous reference paths as well as its capability to navigate in a realistic traffic scenario.

Visual homing enables a mobile robot to move to a reference position using only visual information. The approaches that we present in this paper utilize matched image key points (e.g., scale-invariant feature transform) that are extracted from an omnidirectional camera as inputs. First, we propose three visual homing methods that are based on feature scale, bearing, and the combination of both, under an image-based visual servoing framework. Second, considering computational cost, we propose a simplified homing method which takes an advantage of the scale information of key-point features to compute control commands. The observability and controllability of the algorithm are proved. An outlier rejection algorithm is also introduced and evaluated. The results of all these methods are compared both in simulations and experiments. We report the performance of all related methods on a series of commonly cited indoor datasets, showing the advantages of the proposed method. Furthermore, they are tested on a compact dataset of omnidirectional panoramic images, which is captured under dynamic conditions with ground truth for future research and comparison.

This paper addresses decentralized motion planning among a homogeneous set of feedback-controlled, decision-making agents. It introduces the continuous control obstacle ( Cn-CO), which describes the set of Cn-continuous control sequences (and thus trajectories) that lead to a collision between interacting agents. By selecting a feasible trajectory from Cn-CO's complement, a collision-free motion is obtained. The approach represents an extension to the reciprocal velocity obstacle (RVO, ORCA) collision-avoidance methods so that trajectory segments verify Cncontinuity rather than piecewise linearity. This allows the large class of robots capable of tracking Cn-continuous trajectories to employ it for partial motion planning directly-rather than as a mere tool for collision checking. This paper further establishes that both the original velocity obstacle method and several of its recently developed reciprocal extensions (which treat specific robot physiologies only) correspond to particular instances of Cn-CO. In addition to the described extension in trajectory continuity, Cn-CO thus represents a unification of existing RVO theory. Finally, the presented method is validated in simulation-and a parameter study reveals under which environmental and control conditions Cn-CO with admits significantly improved navigation performance compared with inflated approaches based on ORCA.

In this paper, we present a framework for 6D absolute scale motion and structure estimation of a multi-camera system in challenging indoor environments. It operates in real-time and employs information from two cameras with non-overlapping fields of view. Monocular Visual Odometry supplying up-to-scale 6D motion information is carried out in each of the cameras, and the metric scale is recovered via a linear solution by imposing the known static transformation between both sensors. The redundancy in the motion estimates is finally exploited by a statistical fusion to an optimal 6D metric result. The proposed technique is robust to outliers and able to continuously deliver a reasonable measurement of the scale factor. The quality of the framework is demonstrated by a concise evaluation on indoor datasets, including a comparison to accurate ground truth data provided by an external motion tracking system.

In this paper a method for distributed reciprocal collision avoidance among multiple non-holonomic robots with bike kinematics is presented. The proposed algorithm, bicycle reciprocal collision avoidance (B-ORCA), builds on the concept of optimal reciprocal collision avoidance (ORCA) for holonomic robots but furthermore guarantees collision-free motions under the kinematic constraints of car-like vehicles. The underlying principle of the B-ORCA algorithm applies more generally to other kinematic models, as it combines velocity obstacles with generic tracking control. The theoretical results on collision avoidance are validated by several simulation experiments between multiple car-like robots.

This paper presents a novel approach to tracking dynamic objects in 3D range data. Its key contribution lies in the generative object detection algorithm which allows the tracker to robustly extract objects of varying sizes and shapes from the observations. In contrast to tracking methods using discriminative detectors, we are thus able to generalize over a wide range of object classes matching our assumptions. Whilst the generative model underlying our framework inherently scales with the complexity and the noise characteristics of the environment, all parameters involved in the detection process obey a clean probabilistic interpretation. Nevertheless, our unsupervised object detection and tracking algorithm achieves real-time performance, even in highly dynamic scenarios covering a significant amount of moving objects. Through an application to populated urban settings, we are able to show that the tracking performance of the presented approach yields results which are comparable to state-of-the-art discriminative methods.

A typical Inertial Navigation System (INS) fuses acceleration and angular rate readings with aiding measurements obtained by GPS and a compass. Here we present a robust state estimation framework based on the Extended Kalman Filter (EKF) applied to low-cost electronics typically installed on-board small unmanned airplanes. It uses airspeed measurements as a backup operation mode replacing GPS updates when temporarily unavailable. We demonstrate the applicability of the proposed approach to real-world scenarios using a challenging dataset recorded on-board a manned glider including long-term circling. A comparison between the normal operation mode and the backup solution reveals minimal difference between the respective orientation estimates, a position error growth sub-linear with time during GPS outage and a seamless transition back to GPS-based operation.

Visual Homing is the process by which a mobile robot moves to a Home position using only information extracted from visual data. The approach we present in this paper uses image keypoints (e.g. SIFT) extracted from omnidirectional images and matches the current set of keypoints with the set recorded at the Home location. In this paper, we first formulate three different visual homing problems using uncalibrated omnidirectional camera within the Image Based Visual Servoing (IBVS) framework; then we propose a novel simplified homing approach, which is inspired by IBVS, based only on the scale information of the SIFT features, with its computational cost linear to the number of features. This paper reports on the application of our method on a commonly cited indoor database where it outperforms other approaches. We also briefly present results on a real robot and allude on the integration into a topological navigation framework.

Topological mapping and scene recognition problems are still challenging, especially for online realtime vision-based applications. We develop a hierarchical probabilistic model to tackle them using color information. This work is stimulated by our previous work [1] which defined a lightweight descriptor using color and geometry information from segmented panoramic images. Our novel model uses a Dirichlet Process Mixture Model to combine color and geometry features which are extracted from omnidirectional images. The inference of the model is based on an approximation of conditional probabilities of observations given estimated models. It allows online inference of the mixture model in real-time (at 50Hz), which outperforms other existing approaches. A real experiment is carried out on a mobile robot equipped with an omnidirectional camera. The results show the competence against the state-of-art.

In this paper, we address the problem of curb detection for a pedestrian robot navigating in urban environments. We devise an unsupervised method that is mostly view-independent, makes no assumptions about the environment, restricts the set of hand-tuned parameters, and builds on sound probabilistic reasoning from the input data to the outcome of the algorithm. In our approach, we construct a piecewise planar model of the environment and determine curbs at plane segment boundaries. Initially, we sense the environment with a nodding laser range-finder and project the 3D measurements into an efficient Digital Elevation Map (DEM). Each cell of the DEM maintains an error model that is propagated throughout the entire algorithm. Plane segments are further estimated with a mixture of linear regression models on the DEM. Here, we propose an original formulation of the standard Expectation-Maximization (EM) algorithm for mixture models. Specifically, in the E-step, responsibilities are computed with a Conditional Random Field (CRF) that introduces dependencies between the covariates of the mixture model. A graph-based segmentation of the DEM provides an estimate of the number of planes and initial parameters for the EM. We show promising results of the algorithm on simulated and real-world data.

In this work we apply optimal control to create running gaits for the model of an electrically driven one legged hopper, and compare the results obtained for five different objective functions. By using high compliant series elastic actuators, the motions of joint and motor are decoupled, which allows the exploitation of natural dynamics. Depending on the cost function, this exploitation varies. Energy is injected at different points of time, the amplitude of actuator action changes significantly, and the optimal gear ratios differ by a factor of two. Variations are, however, comparable over a wide range of hopping heights and running velocities. Purely force-based cost functions prove to be ill-suited for such non-conservative systems, and it is shown that thermal electrical losses, in contrast to common belief, do not dominate energy expenditure. The numerical results are corroborated by detailed analytical considerations which give general insights into optimal excitation with electric actuators.

In this paper, we present a versatile framework to enable autonomous flights of a Micro Aerial Vehicle (MAV) which has only slow, noisy, delayed and possibly arbitrarily scaled measurements available. Using such measurements directly for position control would be practically impossible as MAVs exhibit great agility in motion. In addition, these measurements often come from a selection of different onboard sensors, hence accurate calibration is crucial to the robustness of the estimation processes. Here, we address these problems using an EKF formulation which fuses these measurements with inertial sensors. We do not only estimate pose and velocity of the MAV, but also estimate sensor biases, scale of the position measurement and self (inter-sensor) calibration in real-time. Furthermore, we show that it is possible to obtain a yaw estimate from position measurements only. We demonstrate that the proposed framework is capable of running entirely onboard a MAV performing state prediction at the rate of 1 kHz. Our results illustrate that this approach is able to handle measurement delays (up to 500ms), noise (std. deviation up to 20 cm) and slow update rates (as low as 1 Hz) while dynamic maneuvers are still possible. We present a detailed quantitative performance evaluation of the real system under the influence of different disturbance parameters and different sensor setups to highlight the versatility of our approach.

The combination of visual and inertial sensors has proved to be very popular in robot navigation and, in particular, Micro Aerial Vehicle (MAV) navigation due the flexibility in weight, power consumption and low cost it offers. At the same time, coping with the big latency between inertial and visual measurements and processing images in real-time impose great research challenges. Most modern MAV navigation systems avoid to explicitly tackle this by employing a ground station for off-board processing. In this paper, we propose a navigation algorithm for MAVs equipped with a single camera and an Inertial Measurement Unit (IMU) which is able to run onboard and in real-time. The main focus here is on the proposed speed-estimation module which converts the camera into a metric body-speed sensor using IMU data within an EKF framework. We show how this module can be used for full self-calibration of the sensor suite in real-time. The module is then used both during initialization and as a fall-back solution at tracking failures of a keyframe-based VSLAM module. The latter is based on an existing high-performance algorithm, extended such that it achieves scalable 6DoF pose estimation at constant complexity. Fast onboard speed control is ensured by sole reliance on the optical flow of at least two features in two consecutive camera frames and the corresponding IMU readings. Our nonlinear observability analysis and our real experiments demonstrate that this approach can be used to control a MAV in speed, while we also show results of operation at 40Hz on an onboard Atom computer 1.6 GHz.

The SFly project is an EU-funded project, with the goal to create a swarm of autonomous vision controlled micro aerial vehicles. The mission in mind is that a swarm of MAV's autonomously maps out an unknown environment, computes optimal surveillance positions and places the MAV's there and then locates radio beacons in this environment. The scope of the work includes contributions on multiple different levels ranging from theoretical foundations to hardware design and embedded programming. One of the contributions is the development of a new MAV, a hexacopter, equipped with enough processing power for onboard computer vision. A major contribution is the development of monocular visual SLAM that runs in real-time onboard of the MAV. The visual SLAM results are fused with IMU measurements and are used to stabilize and control the MAV. This enables autonomous flight of the MAV, without the need of a data link to a ground station. Within this scope novel analytical solutions for fusing IMU and vision measurements have been derived. In addition to the realtime local SLAM, an offline dense mapping process has been developed. For this the MAV's are equipped with a payload of a stereo camera system. The dense environment map is used to compute optimal surveillance positions for a swarm of MAV's. For this an optimiziation technique based on cognitive adaptive optimization has been developed. Finally, the MAV's have been equipped with radio transceivers and a method has been developed to locate radio beacons in the observed environment.

In this video, we present our latest results towards fully autonomous flights with a small helicopter. Using a monocular camera as the only exteroceptive sensor, we fuse inertial measurements to achieve a self-calibrating power-on-and-go system, able to perform autonomous flights in previously unknown, large, outdoor spaces. Our framework achieves Simultaneous Localization And Mapping (SLAM) with previously unseen robustness in onboard aerial navigation for small platforms with natural restrictions on weight and computational power. We demonstrate successful operation in flights with altitude between 0.2-70 m, trajectories with 350 m length, as well as dynamic maneuvers with track speed of 2 m/s. All flights shown are performed autonomously using vision in the loop, with only high-level waypoints given as directions.

This paper presents a fully automated method to display objects and animations in 3D with a group of aerial vehicles. The system input is a single object or an animation (sequence of objects) created by an artist. The first stage is to generate physical goal configurations and robot colors to represent the objects with the available number of robots. The run-time system includes algorithms for goal assignment, path planning and local reciprocal collision avoidance that guarantee smooth, fast and oscillation-free motion. The presented algorithms are tested in simulations and verified with real quadrotor helicopters and scale to large robot swarms.

In this paper, we present a novel semi-supervised clustering approach based on Markov process. It deals with data which include abundant local constraints. We apply the designed model to a topological region extraction problem, where topological segmentation is constructed based on sparse human inputs (potentially provided by human experts). The model considers human indications as seeds for topological regions, i.e. the partially labeled data. It results in a regional topological segmentation of connected free space.

Visual homing has been widely studied in the past decade. It enables a mobile robot to move to a Home position using only information extracted from visual data. However, integration of homing algorithms into real applications is not widely studied and poses a number of significant challenges. Failures often occur due to moving people within the scene and variations in illumination. We present a novel integrated indoor topological navigation framework, which combines odometry motion with visual homing algorithms. We show robustness to scene variation and real-time performance through a series of tests conducted in four real apartments and several typical indoor scenes, including doorways, offices etc.

Hearing is amongst the most important senses a modern robot must exhibit. Perceiving the acoustic world enables capabilities such as natural interaction with humans, interpreting spoken commands or the localization of victims during search and rescue tasks. Real-world robotic operations often take place in noisy, reverberant environments while requiring features such as source separation, accurate direction of arrival estimation or high performance noise suppression. This work presents a methodology to design, calibrate and operate large microphone arrays that enable such features. Recent micro electro-mechanical microphones in conjunction with reconfigurable logic tackle the weight, size, power consumption and cost constraints of robotic systems. A novel, automatic array shape calibration algorithm is developed for 2D and 3D arrays to face common experimental problems such as reverberation and poor signal-to-noise ratio when calibrating the array. The special case of a 2D array calibrated using sources moving in 3D is addressed. No prior information on array geometry is required, the process is fully automated and does not require any specific calibration equipment. The example application of an acoustic camera is presented as a proof of concept. High-quality acoustic images are computed in real-time by generalized inverse beamforming. This demonstrates the effectiveness of the proposed design and illustrates the usefulness of such sensing capabilities for various robotic applications.

Pseudo-omnidirectional robots with independently steerable wheels require a method to synchronize the steering motion of the wheels in order to keep a unique instantaneous center of rotation (ICR). For standard wheels, the instantaneous center of rotation is defined as the intersection point of all wheel axes. We present a novel approach to deal with the problem of continuously shifting the center of rotation of a pseudo-omnidirectional rover from an initial to a demanded position in the Cartesian plane. The main contribution is the consideration of substantial velocity and acceleration limits on the steering units, as well as mechanical constraints and noise affected sensor measurements. We solve this problem by deriving a relationship between the steering accelerations of the single wheels and the acceleration of the center of rotation. We furthermore provide a contribution to the tracking of the ICR in the presence of significant sensor noise. Our results are evaluated by tests on the rover breadboard developed during the activities for the ExoMars mission.

Consumers and producers of mobility products have been co-creating a mobile world - all within the limits of political regulation and infrastructure realities. With the advent of electric vehicles, the existing mobile world requires adaptation: Producers need to create new ecosystems and mobility concepts ([1], [2]), infrastructure requires adaptation ([1]) and lastly consumers might revisit their expectations. Particularly challenging is the market introduction phase of electric vehicles. Neither have the potentials of the infrastructure and the electric vehicles been fully exploited, nor have consumers become accustomed to electric vehicles and shaped their expectations accordingly. Making electric vehicles a success story requires the satisfaction of customer expectations in the face of both electric vehicle and infrastructure realities. This paper suggests an optimization approach which maximizes customer satisfaction for existing electric vehicle and infrastructure realities. For various degrees-of-freedom (DoF) of the mobility system, the improvement potential is analysed with respect to consumption, charging time, cost and travel time. Moreover, the optimization complexity is analysed, which scales with the number of DoF. The approach enables market entry of electric vehicles and provides the means for future e-navigation and e-travel-planning.

The increasing demand for real-time high-precision Visual Odometry systems as part of navigation and localization tasks has recently been driving research towards more versatile and scalable solutions. In this paper, we present a novel framework for combining the merits of inertial and visual data from a monocular camera to accumulate estimates of local motion incrementally and reliably reconstruct the trajectory traversed. We demonstrate the robustness and efficiency of our methodology in a scenario with challenging camera dynamics, and present a comprehensive evaluation against ground-truth data. 1

The Perspective-Three-Point (P3P) problem aims at determining the position and orientation of the camera in the world reference frame from three 2D-3D point correspondences. This problem is known to provide up to four solutions that can then be disambiguated using a fourth point. All existing solutions attempt to first solve for the position of the points in the camera reference frame, and then compute the position and orientation of the camera in the world frame, which alignes the two point sets. In contrast, in this paper we propose a novel closed-form solution to the P3P problem, which computes the aligning transformation directly in a single stage, without the intermediate derivation of the points in the camera frame. This is made possible by introducing intermediate camera and world reference frames, and expressing their relative position and orientation using only two parameters. The projection of a world point into the parametrized camera pose then leads to two conditions and finally a quartic equation for finding up to four solutions for the parameter pair. A subsequent backsubstitution directly leads to the corresponding camera poses with respect to the world reference frame. We show that the proposed algorithm offers accuracy and precision comparable to a popular, standard, state-of-the-art approach but at much lower computational cost (15 times faster). Furthermore, it provides improved numerical stability and is less affected by degenerate configurations of the selected world points. The superior computational efficiency is particularly suitable for any RANSAC-outlier-rejection step, which is always recommended before applying PnP or non-linear optimization of the final solution.

Effective and efficient generation of keypoints from an image is a well-studied problem in the literature and forms the basis of numerous Computer Vision applications. Established leaders in the field are the SIFT and SURF algorithms which exhibit great performance under a variety of image transformations, with SURF in particular considered as the most computationally efficient amongst the high-performance methods to date. In this paper we propose BRISK1, a novel method for keypoint detection, description and matching. A comprehensive evaluation on benchmark datasets reveals BRISK's adaptive, high quality performance as in state-of-the-art algorithms, albeit at a dramatically lower computational cost (an order of magnitude faster than SURF in cases). The key to speed lies in the application of a novel scale-space FAST-based detector in combination with the assembly of a bit-string descriptor from intensity comparisons retrieved by dedicated sampling of each keypoint neighborhood.

In this paper, we present our latest achievements towards the goal of autonomous flights of an MAV in unknown environments, only having a monocular camera as exteroceptive sensor. As MAVs are highly agile, it is not sufficient to directly use the visual input for position control at the framerates that can be achieved with small onboard computers. Our contributions in this work are twofold. First, we present a solution to overcome the issue of having a low frequent onboard visual pose update versus the high agility of an MAV. This is solved by filtering visual information with inputs from inertial sensors. Second, as our system is based on monocular vision, we present a solution to estimate the metric visual scale aid of an air pressure sensor. All computation is running onboard and is tightly integrated on the MAV to avoid jitter and latencies. This framework enables stable flights indoors and outdoors even under windy conditions.

This paper describes work on multi-robot pattern formation. Arbitrary target patterns are represented with an optimal robot deployment, using a method that is independent of the number of robots. Furthermore, the trajectories are visually appealing in the sense of being smooth, oscillation free, and showing fast convergence. A distributed controller guarantees collision free trajectories while taking into account the kinematics of differentially driven robots. Experimental results are provided for a representative set of patterns, for a swarm of up to ten physical robots, and for fifty virtual robots in simulation.

DisCoverage is a distributed strategy for frontier-based multi-robot exploration. The robots coordinate by a partition of the environment, and choose their target points by optimizing a locally decomposable objective function. In [9] DisCoverage for convex regions was proposed. In this work, we extend DisCoverage to support arbitrary non-convex real-world environments with obstacles. Therefore, we introduce a transformation of non-convex environments to robot centric star-shaped domains. This results in a general solution with broader applications for exploration and path planning. Simulations as well as experiments with real robots demonstrate the exploration progress.

This paper presents a closed-form solution for metric velocity estimation of a single camera using inertial measurements. It combines accelerometer and attitude measurements with feature observations in order to compute both the distance to the feature and the speed of the camera inside the camera frame. Notably, we show that this is possible by just using three consecutive camera positions and a single feature correspondence. Our approach represents a compact linear and multirate solution for estimating complementary information to regular essential matrix computation, namely the scale of the problem. The algorithm is thoroughly validated on simulated and real data and conditions for good quality of the results are identified.

When people communicate with robots, the most intuitive mean is by naming the different regions in the environment. The capability that robots are able to identify different regions highly depends on the unsupervised topological segmentation results. This paper addresses the problem of segmenting a metric map into regions. Nowadays many researches in this direction develop approaches based on spectral clustering. However there are inherent drawbacks of spectral clustering algorithms. In this paper, we first discuss these drawbacks using several testing results; then we propose our approach based on information theory which uses Chow-Liu tree to segment the composed graph according to the weight differences. The results show that our method provides more flexible and faster results in the sense of facilitating semantic mapping or further applications.

This paper presents a novel self-supervised online learning method to discover driving behaviors from data acquired with an inertial measurement unit (IMU) and a camera. Both sensors where mounted in a car that was driven by a human through a typical city environment with intersections, pedestrian crossings and traffic lights. The presented system extracts motion segments from the IMU data and relates them to visual cues obtained from camera data. It employs a Bayesian on-line estimation method to discover the motion segments based on change-point detection and uses a Dirichlet Compound Multinomial (DCM) model to represent the visual features extracted from the camera images. By incorporating these visual cues into the on-line estimation process, labels are computed that are equal for similar motion segments. As a result, typical traffic situations such as braking maneuvers in front of a red light can be identified automatically. Furthermore, appropriate actions in form of observed motion changes are associated to the discovered traffic situations. The approach is evaluated on a real data set acquired in the center of Zurich.

Single camera solutions such as monocular visual odometry or monoSLAM approaches - found a wide echo in the community. All the monocular approaches, however, suffer from the lack of metric scale. In this paper, we present a solution to tackle this issue by adding an inertial sensor equipped with a three-axis accelerometer and gyroscope. In contrast to previous approaches, our solution is independent of the underlying vision algorithm which estimates the camera poses. As a direct consequence, the algorithm presented here operates at a constant computational complexity in real time. We treat the visual framework as a black box and thus the approach is modular and widely applicable to existing monocular solutions. It can be used with any pose estimation algorithm such as visual odometry, visual SLAM, monocular or stereo setups or even GPS solutions with gravity and compass attitude estimation. In this paper, we show the thorough development of the metric state estimation based on an Extended Kalman Filter. Furthermore, even though we treat the visual framework as a black box, we show how to detect failures and estimate drifts in it. We implement our solution on a monocular vision pose estimation framework and show the results both in simulation and on real data.

Terrainability is mostly dependant on the suspension mechanism and the control of a space rover. For the six wheeled CRAB rover, this paper presents the composite control design with torque control and adaptive Kriging control to improve the terrainability, somewhat related to minimizing wheel slip. As CRAB is moving slowly, the torque control is processed by minimizing the variance of the required friction coefficient based on the static model. Adaptive Kriging control is used to track the commanded velocity. The system uncertainty is compensated by Kriging estimation based on the velocity dynamics. Experiment results with two different tires show the effectiveness of the control scheme.

In this paper, we propose a method to recover the relative pose of two robots in absolute scale and in real-time using one monocular camera on each robot. We achieve this by fusing measurements from the onboard inertial sensors on each platform with information obtained from feature correspondences between the two cameras using an Extended Kalman Filter (EKF). This forms a flexible stereo rig, providing the ability to treat the two robots as one single dynamic sensor, which can adapt to the environment and thus improve environmental mapping, obstacle avoidance and navigation. We demonstrate the power of this approach on both simulation and real datasets, employing two micro aerial vehicles (MAVs) to illustrate successful operation over general 3D motion.

The objective of this paper is the full 6D relative localization of mobile devices, and direct robot-robot localization in particular. We present a novel relative localization system that consists of two complementary modules: a monocular vision module and a target module with four active or passive markers. The core localization algorithm running on the modules determines the marker positions in the camera image and derives the relative robot pose in 3D space. The system is supported by a prediction mechanism based on regression. The modules are tested successfully in experiments with a quadrotor helicopter as well as on a team of two e-puck robots performing a coverage task. The relative localization system provides accuracies of a few centimeters in position and up to a few degrees in orientation. Furthermore, the system is lightweight, with low complexity and system requirements, which enables its application to a wide range of mobile robot platforms.

This paper deals with the problem of deploying a team of flying robots to perform surveillance coverage missions over a terrain of arbitrary morphology. In such missions, a key factor for the successful completion is the knowledge of the terrain's morphology. In this paper, we introduce a two-step centralized procedure to align optimally a swarm of flying vehicles for the aforementioned task. Initially, a single robot constructs a map of the area of interest using a novel monocular-vision-based approach. A state-of-the-art visual-SLAM algorithm tracks the pose of the camera while, simultaneously, building an incremental map of the surrounding environment. The map generated is processed and serves as an input in an optimization procedure using the cognitive adaptive methodology initially introduced in [1], [2]. The output of this procedure is the optimal arrangement of the robot team, which maximizes the monitored area. The efficiency of our approach is demonstrated using real data collected from aerial robots in different outdoor areas.

The dynamics of micro coaxial helicopters are coupled, especially in the presence of a stabilizer bar and in dynamic maneuvers. This paper presents a model-based approach for active decoupling of the dynamics of a micro coaxial helicopter. This allows for easier and more accurate operation of the system. The nonlinear model covers all degrees of freedom for attitude and altitude. It accounts for hover and cruise flight situations and explicitly captures the off-axis dynamics and the dynamics of the stabilizer bar. A six-axis force/torque sensor and an RPM measurement system are used in a custom built test bench. It is applied for analysis of the forces and torques generated by the rotors in combination with the dynamics of the drive train and the swashplate. The parameter identification and the model validation is obtained with flight data recorded with a vision-based motion tracking system. The decoupling controller is implemented on the commercial robotic helicopter CoaX and its performance is shown via a motion experiment.

This paper introduces the mechanical design and the control concept of the Series Compliant Articulated Robotic Leg ScarlETH which was developed at ETH Zurich for fast, efficient, and versatile locomotion. Inspired by biological systems, we seek to achieve this through large compliances in the joints which enable natural dynamics, allow temporary energy storage, and improve the passive adaptability. A sophisticated chain and cable pulley design minimizes the segment masses, places the overall CoG close to the hip joint, and maximizes the range of motion. Nonlinearities in the damping and an appropriate low-level controller allow for precise torque control during stance and for fast task space position control during swing. This paved the road for the combined application of a virtual model controller for ground contact and a modified Raibert style controller for flight phase which was successfully tested in planar running.

In this work, we present a novel, deterministic closed-form solution for computing the scale factor and the gravity direction of a moving, loosely-coupled, and monocular vision-inertial system. The methodology is based on analysing delta-velocities. On one hand, they are obtained from a differentiation of the up-to-scale camera pose computation by a visual odometry or visual SLAM algorithm. On the other hand, they can also be retrieved from the gravity-affected short-term integration of acceleration signals. We derive a method for separating the gravity contribution and recovering the metric scale factor of the vision algorithm. The method thus also recovers the offset in roll and pitch angles of the vision reference frame with respect to the direction of the gravity vector. It uses only a single inertial integration period, and no absolute orientation information is required. For optimal sensor-fusion and metric scale-estimation filters in the loosely-coupled case, it has been shown that the convergence of the fusion of an up-to-scale pose information with inertial measurements largely depends on the availability of a good initial value for the scale factor. We show how this problem can be tackled by applying the method presented in this paper. Finally, we present results in simulation and on real data, demonstrating the suitability of the method in real scenarios.

The increasing number of ICP variants leads to an explosion of algorithms and parameters. This renders difficult the selection of the appropriate combination for a given application. In this paper, we propose a state-of-the-art, modular, and efficient implementation of an ICP library. We took advantage of the recent availability of fast depth cameras to demonstrate one application example: a 3D pose tracker running at 30 Hz. For this application, we show the modularity of our ICP library by optimizing the use of lean and simple descriptors in order to ease the matching of 3D point clouds. This tracker is then evaluated using datasets recorded along a ground truth of millimeter accuracy. We provide both source code and datasets to the community in order to accelerate further comparisons in this field.

This work introduces a framework for the creation and analysis of efficient gaits for legged systems based on the exploitation of natural dynamics. It summarizes the theory behind hybrid dynamic modeling, the identification of optimal periodic motions with single shooting and direct collocation, and the analysis of first order stability. Three examples introduce various aspects of gait creation and analysis: a stability study of a passive dynamic walker determines the ideal position of the leg's center of mass, the cost of transportation is minimized for a prismatic monopod hopper based on series elastic actuators, and a basic controller is created for the model of a bounding robot. The presented tools and examples are freely available at www.asl.ethz.ch/people/cremy/personal/GaitCreation.

This work presents a method for estimating the egomotion of an aerial vehicle in challenging industrial environments. It combines binocular visual and inertial cues in a tightly-coupled fashion and operates in real time on an embedded platform. An extended Kalman filter fuses measurements and makes motion estimation rely more on inertial data if visual feature constellation is degenerate. Errors in roll and pitch are bounded implicitly by the gravity vector. Inertial sensors are used for efficient outlier detection and enable operation in poorly and repetitively textured environments. We demonstrate robustness and accuracy in an industrial scenario as well as in general indoor environments. The former is accompanied by a detailed performance evaluation supported with ground truth measurements from an external tracking system.

This paper presents a semi-autonomous navigation strategy aimed at the control of assistive devices (e.g. an intelligent wheelchair) using low throughput interfaces. A mobile robot proposes the most probable action, as analyzed from the environment, to a human user who can either accept or reject the proposition. In case of rejection, the robot will propose another action, until both entities agree on what needs to be done. In a known environment, the system infers the intended goal destination based on the first executed actions. Furthermore, we endowed the system with learning capabilities, so as to learn the user habits depending on contextual information (e.g. time of the day or if a phone rings). This additional knowledge allows the robot to anticipate the user intention and propose appropriate actions, or goal destinations.

In this letter, we propose a learning system, active decision fusion learning (ADFL), for active fusion of decisions. Each decision maker, referred to as a local decision maker, provides its suggestion in the form of a probability distribution over all possible decisions. The goal of the system is to learn the active sequential selection of the local decision makers in order to consult with and thus learn the final decision based on the consultations. These two learning tasks are formulated as learning a single sequential decision-making problem in the form of a Markov decision process (MDP), and a continuous reinforcement learning method is employed to solve it. The states of this MDP are decisions of the attended local decision makers, and the actions are either attending to a local decision maker or declaring final decisions. The learning system is punished for each consultation and wrong final decision and rewarded for correct final decisions. This results in minimizing the consultation and decision-making costs through learning a sequential consultation policy where the most informative local decision makers are consulted and the least informative, misleading, and redundant ones are left unattended. An important property of this policy is that it acts locally. This means that the system handles any nonuniformity in the local decision maker's expertise over the state space. This property has been exploited in the design of local experts. ADFL is tested on a set of classification tasks, where it outperforms two well-known classification methods, Adaboost and bagging, as well as three benchmark fusion algorithms: OWA, Borda count, and majority voting. In addition, the effect of local experts design strategy on the performance of ADFL is studied, and some guidelines for the design of local experts are provided. Moreover, evaluating ADFL in some special cases proves that it is able to derive the maximum benefit from the informative local decision makers and to minimize attending to redundant ones.

People tracking is a key technology for autonomous systems, intelligent cars and social robots operating in populated environments. What makes the task difficult is that the appearance of humans in range data can change drastically as a function of body pose, distance to the sensor, self-occlusion and occlusion by other objects. In this paper we propose a novel approach to pedestrian detection in 3D range data based on supervised learning techniques to create a bank of classifiers for different height levels of the human body. In particular, our approach applies AdaBoost to train a strong classifier from geometrical and statistical features of groups of neighboring points at the same height. In a second step, the AdaBoost classifiers mutually enforce their evidence across different heights by voting into a continuous space. Pedestrians are finally found efficiently by mean-shift search for local maxima in the voting space. Experimental results carried out with 3D laser range data illustrate the robustness and efficiency of our approach even in cluttered urban environments. The learned people detector reaches a classification rate up to 96% from a single 3D scan.

The large collections of datasets for researchers working on the Simultaneous Localization and Mapping problem are mostly collected from sensors such as wheel encoders and laser range finders mounted on ground robots. The recent growing interest in doing visual pose estimation with cameras mounted on micro-aerial vehicles however has made these datasets less useful. In this paper, we describe our work in creating new datasets collected from a sensor suite mounted on a quadrotor platform. Our sensor suite includes a forward looking camera, a downward looking camera, an inertial measurement unit and a Vicon system for groundtruth. We propose the use our datasets as benchmarking tools for future works on visual pose estimation for micro-aerial vehicles. We also show examples of how the datasets could be used for benchmarking visual pose estimation algorithms.

Within the research on Micro Aerial Vehicles (MAVs), the field on flight control and autonomous mission execution is one of the most active. A crucial point is the localization of the vehicle, which is especially difficult in unknown, GPS-denied environments. This paper presents a novel vision based approach, where the vehicle is localized using a downward looking monocular camera. A state-of-the-art visual SLAM algorithm tracks the pose of the camera, while, simultaneously, building an incremental map of the surrounding region. Based on this pose estimation a LQG/LTR based controller stabilizes the vehicle at a desired setpoint, making simple maneuvers possible like take-off, hovering, setpoint following or landing. Experimental data show that this approach efficiently controls a helicopter while navigating through an unknown and unstructured environment. To the best of our knowledge, this is the first work describing a micro aerial vehicle able to navigate through an unexplored environment (independently of any external aid like GPS or artificial beacons), which uses a single camera as only exteroceptive sensor.

This paper presents a solution to decentralized Voronoi coverage in non-convex polygonal environments. We show that complications arise when existing approaches to Voronoi coverage are applied for deploying a group of robots in non-convex environments. We present an algorithm that is guaranteed to converge to a local optimum. Our algorithm combines classical Voronoi coverage with the Lloyd algorithm and the local path planning algorithm TangentBug to compute the motion of the robots around obstacles and corners. We present the algorithm and prove convergence and optimality. We also discuss experimental results from an implementation with five robots.

In this paper, we are presenting a method to estimate terrain properties (such as small-scale geometry or surface friction) to improve the assessment of stability and the guiding of foot placement of legged robots in rough terrain. Haptic feedback, expressed through joint motor currents and ground contact force measurements that arises when prescribing a predefined motion was collected for a variety of ground samples (four different shapes and four different surface properties). Features were extracted from this data and used for training and classification by a multiclass AdaBoost machine learning algorithm. In a single leg testbed, the algorithm could correctly classify about 94% of the terrain shapes, and about 73% of the surface samples.

Homing is one of the fundamental functions for both the mobile robot and the flying robot. Furthermore, homing can be introduced into a topological navigation system by cyclically setting Home positions at the keypoints/nodes in a topological map. In this work, we describe a bearing-only homing method based on only few matching keypoints to grant the mobile robot the homing ability. Our method considers the homing problem as a visual servoing problem in 2D plane and even in 3D space, using an omnidirectional camera as the visual sensor. It doesn't require the distance information to the reference feature points. The proof of the convergence for the algorithm is also given. The simulation results confirm the feasibility and robustness of our method.

Most large-scale public environments provide direction signs to facilitate the orientation for humans and to find their way to a goal location in the environment. Thus, for a robot operating in the same environment, it would be beneficial to interpret such signs correctly for a safe and efficient navigation. In this work, we propose a novel approach to infer the meaning of direction signs and to use that for navigation, i.e., to find a mapping of a detected sign to a motion direction. Our method uses a hierarchical extension of the Implicit Shape Model framework called HISM that does not require any hand-labeled training data to detect the signs. On the lower level of this two-stage hierarchy, ISM is applied to image descriptors as in the standard approach. On the higher level, ISM operates on subparts of signs called tokens, using weights learned from data. The interpretation of the signs is inferred by associating navigation data to direction instructions. We conducted experiments from image data acquired in an airport terminal, aiming towards the implementation of a robotic guide, with promising results.

In search of underwater locomotion methods as alternatives to propellers, systems relying on the propagation of waves along a fin have already been designed and evaluated by several scientists. Considerable effort has been undertaken to optimise their efficiency both by fluid dynamic analysis and experiments on physical prototypes. One drawback of the systems hitherto has been their electro-mechanical complexity in that they required many actuators and refined control strategies to generate the desired fin undulation. Our approach has been to translate the result of these optimisations into a simpler, purely mechanical model relying on the principle of camshafts to achieve a similar undulatory fin motion. The goal was to evaluate whether this type of propulsion system is feasible and whether it was a viable alternative to propellers in Autonomous Underwater Vehicles. The prototype built during the project, CUTTLEFIN, reached comparable speeds to other undulating robot solutions. Force measurements also showed that the thrust produced is in qualitative accordance to a simplified fluid dynamics model. This makes the camshaft approach a promising option for generating an undulating wave in a membrane-based fin propulsion system, if one is willing to pay the price of lower flexibility compared to current dexterously actuated solutions.

In the present study, we applied the principles of passive dynamic walking onto the three dimensional motion of a simplified quadrupedal model. We extended the simulation framework of a planar system to include a rolling degree of freedom and searched for limit cycles that represent periodic gaits. Among the eight different gaits that we identified, were three kinds of trots and paces, as well as a lateral and diagonal single foot sequence. We could show that a distinct relation exists between the lateral spacing of the legs and the relative phase of the front and the back legs, and a certain trade-off between efficiency and dynamic stability. In agreement with established bipedal models, our results showed that the lateral rolling motion is invariably unstable.

In this paper we describe a novel and simple to implement yet effective lattice design algorithm, which simultaneously produces input and state-space sampled lattice graphs. The presented method is an extension to the ideas suggested by Bicchi et al. on input lattices and is applicable to systems which can be brought into (2,n) chained form, such as kinematic models of unicycles, bicycles, differential-drive robots and car-like vehicles (pulling several trailers). We further show that a transformation from chained form to path coordinates allows the resulting lattice to be bent along any C1continuous path. We exploit this fact by shaping it along the skeleton of arbitrary structured environments, such as the center of road lanes and corridors. In our experiments in both structured (i.e. on-road) and unstructured (i.e. parking lot) scenarios, we successfully demonstrate for the first time the applicability of lattice-based planning approaches to search queries in arbitrary environments.

In this paper we present a topological map building algorithm based on a Vocabulary Tree that is robust to features present in dynamic or similar environments. The algorithm recognises incorrect loop closures, not supported by the odometry, and uses this information to update the feature weights in the tree to suppress further associations from these features. Two methods of adjusting these feature entropies are proposed, one decreasing entropy related to incorrect features in a uniform manner and the other proportional to the contribution of the said feature. Preliminary results showing the performance of the proposed method are presented where it is found that by adjusting the feature entropies, the number of incorrect associations can be reduced while improving the quality of the correct matches.

We present a novel approach for unsupervised discovery of repetitive objects from 3D point clouds. Our method assumes that objects are non-deformable and uses multiple occurrences of an object as the evidence for its existence. We segment input range data by superpixel segmentation and extract features for each segment. We search for a group of segments where each segment matches a segment in another group using a joint compatibility test. The discovered objects are then verified by the Iterative Closest Point algorithm to remove false matches. The presented method was tested on real data of complex objects. The experiments demonstrate that the proposed approach is capable of finding objects that occur multiple times in a scene and distinguish apart those objects of different types.

Safe navigation through corridors plays a major role in the autonomous use of Micro Aerial Vehicles (MAVs) in indoor environments. In this paper, we present an approach for wall collision avoidance using a depth map based on optical flow from on board camera images. An omnidirectional fisheye camera is used as a primary sensor, while IMU data is needed for compensating rotational effects of the optical flow. The here presented approach is designed for safely maneuvering a helicopter through an indoor corridor. Results based on real images taken in a corridor with textured walls are shown at the end of this paper.

This paper addresses the problem of deploying a group of networked robots on a non-planar surface embedded in 3D space. Two distributed coverage control algorithms are presented that both provide a solution to the problem by discrete coverage of a graph. The first method computes shortest paths and runs the Lloyd algorithm on the graph to obtain a centroidal Voronoi tessellation. The second method uses the Euclidean distance measure and locally exchanges mesh cells between approximated Voronoi regions to reach an optimal robot configuration. Both methods are compared and evaluated in simulations and in experiments with five robots on a curved surface.

SLIP models are generally known as one of the best and simplest abstractions describing the spring-like leg behavior found in human and animal running, and have thus been subject to exhaustive investigation. To exploit these findings in real robots, we utilize an operational space controller that projects the behavior of the SLIP model onto the dynamics of an actual segmented robotic leg. Additionally, we introduce a method to compensate for the energetic losses at the impact collisions, which are not accounted for in the simplified SLIP assumptions. This allows the direct application of existing dead-beat control strategies to arbitrary robotic legs, for which we can show that the collision and compensation effects in the actual leg enlarge the regions of stable running and reduce the minimally required locomotion speed. The necessary joint torque profiles can be generated in large part passively, for example by using high compliance series elastic actuators.

The present paper is focusing on the initialization of statistical optimum filters for motion estimation in robotics. It shows that if certain conditions concerning the stability of a system are fulfilled, and some knowledge about the mean of the state is given, an initial error covariance matrix that is optimal with regard to the convergence behavior of the filter estimate might be analytically obtained. Easy algorithms for the n-dimensional continuous and discrete cases are presented. The applicability to non-linear systems is also pointed out. The convergence of a normal Kalman filter is analyzed in simulation using the discrete model of a theoretical example.

The capability to overcome terrain irregularities or obstacles, named terrainability, is mostly dependant on the suspension mechanism of the rover and its control. For a given wheeled robot, the terrainability can be improved by using a sophisticated control, and is somewhat related to minimizing wheel slip. The proposed control method, named torque control, improves the rover terrainability by taking into account the whole mechanical structure. The rover model is based on the Newton-Euler equations and knowing the complete state of the mechanical structures allows us to compute the force distribution in the structure, and especially between the wheels and the ground. Thus, a set of torques maximizing the traction can be used to drive the rover. The torque control algorithm is presented in this paper, as well as tests showing its impact and improvement in terms of terrainability. Using the CRAB rover platform, we show that the torque control not only increases the climbing performance but also limits odometric errors and reduces the overall power consumption.

In structure-from-motion with a single camera it is well known that the scene can be only recovered up to a scale. In order to compute the absolute scale, one needs to know the baseline of the camera motion or the dimension of at least one element in the scene. In this paper, we show that there exists a class of structure-from-motion problems where it is possible to compute the absolute scale completely automatically without using this knowledge, that is, when the camera is mounted on wheeled vehicles (e.g. cars, bikes, or mobile robots). The construction of these vehicles puts interesting constraints on the camera motion, which are known as “nonholonomic constraints”. The interesting case is when the camera has an offset to the vehicle's center of motion. We show that by just knowing this offset, the absolute scale can be computed with a good accuracy when the vehicle turns. We give a mathematical derivation and provide experimental results on both simulated and real data over a large image dataset collected during a 3 Km path. To our knowledge this is the first time nonholonomic constraints of wheeled vehicles are used to estimate the absolute scale. We believe that the proposed method can be useful in those research areas involving visual odometry and mapping with vehicle mounted cameras.

This paper presents a detailed characterization of the Hokuyo URG-04LX 2D laser range finder. While the sensor specifications only provide a rough estimation of the sensor accuracy, the present work analyzes issues such as time drift effects and dependencies on distance, target properties (color, brightness and material) as well as incidence angle. Since the sensor is intended to be used for measurements of a tubelike environment on an inspection robot, the characterization is extended by investigating the influence of the sensor orientation and dependency on lighting conditions. The sensor characteristics are compared to those of the Sick LMS 200 which is commonly used in robotic applications when size and weight are not critical constraints. The results show that the sensor accuracy is strongly depending on the target properties (color, brightness, material) and that it is consequently difficult to establish a calibration model. The paper also identifies cases for which the sensor returns faulty measurements, mainly when the surface has low reflectivity (dark surfaces, foam) or for high incidence angles on shiny surfaces. On the other hand, the repeatability of the sensor seems to be competitive with the LMS 200.

In this paper we describe a novel path planning approach for mobile robots operating in indoor environments. In such scenarios, robots must be able to maneuver in crowded spaces, partially filled with static and dynamic obstacles (such as people). Our approach produces smooth, complex maneuvers over large distances through the use of an anytime graph search algorithm applied to a novel multi-resolution state lattice, where the resolution is adapted based on both environmental characteristics and task characteristics. In addition, we present a novel approach for generating fast globally optimal trajectories in constrained spaces (i.e. rooms connected via doors and hallways). This approach exploits offline precomputation to provide extremely efficient online performance and is applicable to a wide range of both indoor and outdoor navigation scenarios. By combining an anytime, multi-resolution lattice-based search algorithm with our precomputation technique, globally optimal trajectories in up to four dimensions (2D position, heading and velocity) are obtained in real-time.

This paper presents a system capable of recovering the trajectory of a vehicle from the video input of a single camera at a very high frame-rate. The overall frame-rate is limited only by the feature extraction process, as the outlier removal and the motion estimation steps take less than 1 millisecond with a normal laptop computer. The algorithm relies on a novel way of removing the outliers of the feature matching process.We show that by exploiting the nonholonomic constraints of wheeled vehicles it is possible to use a restrictive motion model which allows us to parameterize the motion with only 1 feature correspondence. Using a single feature correspondence for motion estimation is the lowest model parameterization possible and results in the most efficient algorithms for removing outliers. Here we present two methods for outlier removal. One based on RANSAC and the other one based on histogram voting. We demonstrate the approach using an omnidirectional camera placed on a vehicle during a peak time tour in the city of Zurich. We show that the proposed algorithm is able to cope with the large amount of clutter of the city (other moving cars, buses, trams, pedestrians, sudden stops of the vehicle, etc.). Using the proposed approach, we cover one of the longest trajectories ever reported in real-time from a single omnidirectional camera and in cluttered urban scenes, up to 3 kilometers.

This paper proposes an object classification framework based on a geometric grammar aimed for mobile robotic applications. The paper first discusses the geometric grammar as a compact representation form for object categories with primitive parts as its constituent elements. The paper then discusses the object classification implemented as parsing of primitive parts. In particular, two approaches are discussed that constrain the search space in order to render the parsing of the primitive parts practical. The two approaches are experimentally verified, first, for a generic object category of chair applied to real range images acquired with a range camera mounted on a mobile robot and, second, for multiple generic object categories applied to synthetic range images. The experimental results show the practicability of the framework.

In this paper, we present the mechanical design of a compact magnetic wheeled robot - with the goal to do inspection and vibration measurements in the housings of large generators and similar environments in power plants. After a detailed analysis of the specifications in this application, we present a new vehicle structure that allows for passing sharp concave corners - even with low friction coefficient between wheels and surface - and without using an additional active DOF. The advantages of this structure and the core parameters for its optimization are described in a quasi-static 2D calculation model. A prototype was implemented and successfully tested both in laboratory and real environments. The paper concludes pointing out the future improvements for a final industrial version.

In the present study, we investigate a control strategy for hopping motions of an articulated leg that is driven by series elastic actuation. A highly compliant spring in the knee joint allows the exploitation of periodic energy storage but creates a major control challenge by severely limiting the bandwidth of closed-loop position or force control. This handicap is intensified by slow actuators, substantial delays, and the kinematic coupling of the articulated design. With classic closed-loop control strategies failing, an adaptive open-loop control algorithm is presented, that, over a series of jumps, estimates the compression of the actuator springs, and gradually modifies the motor inputs in order to minimize slipping and create a purely vertical motion.

Mobile robots rely on their ability of scene recognition to build a topological map of the environment and perform location-related tasks. In this paper, we describe a novel lightweight scene recognition method using an adaptive descriptor which is based on color features and geometric information for omnidirectional vision. Our method enables the robot to add nodes to a topological map automatically and solve the localization problem of mobile robot in realtime. The descriptor of a scene is extracted in the YUV color space and its dimension is adaptive depending on the segmentation result of the panoramic image. Furthermore, the descriptor is invariant to rotation and slight changes of illumination. The robustness of the scene matching and recognition is tested through real experiments in a dynamic indoor environment. The experiment is carried out on a mobile robot equipped with an omnidirectional camera. In our tests, the average processing time is 30 ms for each frame including feature extraction, matching, and the adding of new nodes.

This paper presents a novel image based detection method for pedestrians at very small scales (between 16 × 20 and 32 × 40). We propose a set of new distinctive image features based on collections of local image gradients grouped by a superpixel segmentation. Features are collected and classified using AdaBoost. The positive classified features then vote for potential hypotheses that are collected using a mean shift mode estimation approach. The presented method overcomes the common limitations of a sliding window approach as well as those of standard voting approaches based on interest points. Extensive tests have been produced on a dataset with more than 20000 images showing the potential of this approach.

This paper presents a comparison between six different ways to convey navigational information provided by a robot to a human. Visual, auditory, and tactile feedback modalities were selected and designed to suggest a direction of travel to a human user, who can then decide if he agrees or not with the robot's proposition. This work builds upon a previous research on a novel semi-autonomous navigation system in which the human supervises an autonomous system, providing corrective monitoring signals whenever necessary.

This paper presents an incremental object part detection algorithm using a particle filter. The method infers object parts from 3D data acquired with a range camera. The range information is quantized and enhanced by local structure to partially cope with considerable measurement noise and distortion. The augmented voxel representation allows the adaptation of known track-before-detect algorithms to infer multiple object parts in a range image sequence even when each single observation does not contain enough information to do the detection. The appropriateness of the method is successfully demonstrated by two experiments for chair legs.

This paper presents a novel human detection method based on a Bayesian fusion approach using laser range data and camera images. Laser range data analysis groups data points with a novel graph cutting method. Therefore, it computes a belief to each cluster based on the evaluation of multidimensional features that describe geometrical properties. A person detection algorithm based on dense overlapping grid of Histograms of Oriented Gradients (HOG) is processed on the image area determined by each laser cluster. The selection of HOG features and laser features is obtained through a learning process based on a cascade of linear Support Vector Machines (SVM). A technique to obtain conditional probabilities from a cascade of SVMs is here proposed in order to combine the two information together. The resulting human detection consists in a rich information that takes into account the distance of the cluster and the confidence level of both detection methods. We demonstrate the performance of this work on real-world data and different environments.

Whenever the realization of a swash plate mechanism is not feasible (e.g. due to miniaturization limitations), center of gravity steering is an interesting alternative to swash plate steering. We present an approach to describe the dynamic behavior of a coaxial micro helicopter steered by a center of gravity shifting mechanism. The mechanical design of an existing system is improved to increase mechanical robustness and steering quality. In parallel, a simulation model is developed and implemented. It is used to estimate the system response to steering inputs, and to compare center of gravity to swash plate steering. Experimental flight results show an improvement of the helicopter performance due to the mechanical redesign.

The study presented in this paper explores people perception of robots, with a particular focus on domestic use. We addressed issues related to positive and negative attitudes toward robots, needs for domestic robots as well as preferences in terms of appearance and interaction modalities. We used a combined qualitative and quantitative approach using interviews and questionnaires. In total, 240 people participated in our survey. Results indicate that a large proportion of participants have a very positive attitude towards robots. They expect concrete help for a variety of tasks from these devices. They prefer a small machine and they would like to interact with robots using speech.

Most of the existing camera calibration toolboxes require the observation of a checkerboard shown by the user at different positions and orientations. This paper presents an algorithm for the automatic detection of checkerboards, described by the position and the arrangement of their corners, in blurred and heavily distorted images. The method can be applied to both perspective and omnidirectional cameras. An existing corner detection method is evaluated and its strengths and shortcomings in detecting corners on blurred and distorted test image sets are analyzed. Starting from the results of this analysis, several improvements are proposed, implemented, and tested. We show that the proposed algorithm is able to consistently identify 80% of the corners on omnidirectional images of as low as VGA resolution and approaches 100% correct corner extraction at higher resolutions, outperforming the existing implementation significantly. The performance of the proposed method is demonstrated on several test image sets of various resolution, distortion, and blur, which are exemplary for different kinds of camera-mirror setups in use.

In robotics, vertical lines have been always very useful for autonomous robot localization and navigation in structured environments. This paper presents a robust method for matching vertical lines in omnidirectional images. Matching robustness is achieved by creating a descriptor which is very distinctive and is invariant to rotation and slight changes of illumination. We characterize the performance of the descriptor on a large image dataset by taking into account the sensitiveness to the different parameters of the descriptor. The robustness of the approach is also validated through a real navigation experiment with a mobile robot equipped with an omnidirectional camera.

This paper presents a novel people detection and tracking method based on a combined multimodal sensor approach that utilizes 2D and 3D laser range and camera data. Laser data points are clustered and classified with a set of geometrical features using an SVM AdaBoost method. The clusters define a region of interest in the image that is adjusted using the ground plane information extracted from the 3D laser. In this areas a novel vision based people detector based on Implicit Shape Model (ISM) is applied. Each detected person is tracked using a greedy data association technique and multiple Extended Kalman Filters that use different motion models. This way, the filter can cope with a variety of different motion patterns. The tracker is asynchronously updated by the detections from the laser and the camera data. Experiments conducted in real-world outdoor scenarios with crowds of pedestrians demonstrate the usefulness of our approach.

In this paper, we describe a real-time algorithm for computing the ego-motion of a vehicle relative to the road. The algorithm uses as input only those images provided by a single omnidirectional camera mounted on the roof of the vehicle. The front ends of the system are two different trackers. The first one is a homography-based tracker that detects and matches robust scale-invariant features that most likely belong to the ground plane. The second one uses an appearance-based approach and gives high-resolution estimates of the rotation of the vehicle. This planar pose estimation method has been successfully applied to videos from an automotive platform. We give an example of camera trajectory estimated purely from omnidirectional images over a distance of 400 m. For performance evaluation, the estimated path is superimposed onto a satellite image. In the end, we use image mosaicing to obtain a textured 2-D reconstruction of the estimated path.

This paper presents two methods able to exploit the information at the loop closure in the SLAM problem. Both methods have three fundamental advantages. The first one is that to apply the loop closure constraint they do not require to compute any correlation among the features which are not observed simultaneously. The second advantage is that the loop closure constraint can be applied only once at the end (even after more than one loop) in a single step with low computational complexity. Hence, the computational complexity during the robot exploration is independent of the number of features. Finally, the third advantage is that the linearization does not affect the estimation process. This especially holds for the second method, which is based on the Levenberg-Marquardt algorithm. The first method is based on the extended Kalman filter. Simulations show that these approaches significantly outperform the conventional EKF based SLAM both in the computational cost and in the map precision.

Map learning is a fundamental task in mobile robotics because maps are required for a series of high level applications. In this paper, we address the problem of building maps of large-scale areas like villages or small cities. We present our modified car-like robot which we use to acquire the data about the environment. We introduce our localization system which is based on an information filter and is able to merge the information obtained by different sensors. We furthermore describe out mapping technique that is able to compactly model three-dimensional scenes and allows us efficient and accurate incremental map learning. We additionally apply a global optimization techniques in order to accurately close loops in the environment. Our approach has been implemented and deeply tested on a real car equipped with a series of sensors. Experiments described in this paper illustrate the accuracy and efficiency of the presented techniques.

The research on autonomous miniature flying robots has intensified considerably thanks to the recent growth of civil and military interest in unmanned aerial vehicles (UAV). This paper summarizes the final results of the modeling and control parts of OS4 project, which focused on design and control of a quadrotor. It introduces a simulation model which takes into account the variation of the aerodynamical coefficients due to vehicle motion. The control parameters found with this model are successfully used on the helicopter without re-tuning. The last part of this paper describes the control approach (integral backstepping) and the scheme we propose for full control of quadrotors (attitude, altitude and position). Finally, the results of autonomous take-off, hover, landing and collision avoidance are presented.

In this paper describes a pair of wall climbing robots that use magnetic wheels for adhesion. They are designed for inspecting the interior surfaces of gas-tanks in oversea ships. Environments like this were impossible to access by previous climbing robots, as they are made out of very thin sheet metal that cannot support a high robot mass and contain several types of difficult obstacles. In order to master these challenges, the system described in this work uses two robots in a "mother-child"-structure: the smaller robot is built very lightweight and simple, without the ability to steer or climb vertically. It just moves horizontally and uses ridges in the gas tank surface as guidance rails. In order to pass from one ridge to the next one, the smaller robot docks to the bigger "mother"-robot. This robot always stays in an area where the sheet metal is thicker and never enters the extremely fragile surfaces. Thus, its mass is not critical for the structural stability of the environment and the robot can hence be built big enough to be able to climb in all directions of gravity and to pass difficult obstacles along its path. In the present paper, the basic concept, the mechanical design of all important components and the proposed control strategy are described briefly, followed by test results of the most critical components. An outlook to similar applications to which the basic idea of this work can be successfully transferred concludes this article.

Toward obtaining a compact and multiresolution representation of 2D range scans, a wavelet framework is proposed for encoding an orientation measure called running angle (RA). A new shrinkage algorithm is developed using discrete wavelet transform of the RA signal, which leads to a simplified polyline approximation of the initial scanned points. This approach is evaluated in terms of segmentation of 2D range scans as a line extractor. As a proof of concept, an experiment is performed in our laboratory hallway by a mobile robot equipped with two SICK laser range finders which shows that it is possible to successfully segment raw measurements of the scanner using the proposed approach and obtain proper linear abstraction. Besides a simple, fast heuristic line extraction algorithm is also proposed for the sake of comparison. It is based on thresholding the changes of the incident angles between the laser beam and the vertices of the initial polygon observed by the scanner. Despite its simplicity, this approach performs rather well and can be used in structured environments with low measurement noise. Both approaches are experimentally evaluated and compared with some well known and commonly used line extraction algorithms.

This paper focuses on development of a dynamic model for an Ackermann-like vehicle based on a static tire-road friction model and laws of technical mechanics. The model takes as input the steering angle of the wheels in front and the rotational velocities of the drive wheels in the back of the vehicle. It delivers a 3-DOF output in terms of CoG vehicle velocity, body slip angle and the yaw rate of the vehicle in the x-y plane, as well as estimates on the forces acting on the system. It is suitable for modeling dynamic vehicle regimes in e.g. overtaking maneuvers/obstacle avoidance and lane-keeping, enabling active steering control by stabilizing the dynamics of the vehicle. The physical model description is based on previous works combined with a suitable friction model that is tractable in practice. Experimental verification of the obtained model is given for the Smart testing vehicle platform, where a separate analysis is done for directly measured as opposed to estimated/optimized parameters of the model.

Simple, fast and lightweight SLAM algorithms are necessary in many embedded robotic systems which soon will be used in houses and offices in order to do various service tasks. In this paper the Orthogonal SLAM algorithm is presented as an answer to this need. In continuation of our previous work, the algorithm is extended to generate 3D maps and empirically validated by mapping the long corridor of our lab with the accuracy comparable with hand measured ground truth. The main contribution resides in the idea of reducing the complexity by using orthogonality constraint in indoor environments. This is done by mapping only planes that are parallel or perpendicular to each other which represent the main structure of most indoor environments. Having this assumption, we use an inclined sensor setup (fixed 2D SICK laser range finders) to generate 3D orthogonal maps. The algorithm is extremely fast since in each step it just processes one line of laser measurements.

In this paper, we describe a new approach for the extrinsic calibration of a camera with a 3D laser range finder, that can be done on the fly. This approach does not require any calibration object. Only few point correspondences are used, which are manually selected by the user from a scene viewed by the two sensors. The proposed method relies on a novel technique to visualize the range information obtained from a 3D laser scanner. This technique converts the visually ambiguous 3D range information into a 2D map where natural features of a scene are highlighted. We show that by enhancing the features the user can easily find the corresponding points of the camera image points. Therefore, visually identifying laser- camera correspondences becomes as easy as image pairing. Once point correspondences are given, extrinsic calibration is done using the well-known PnP algorithm followed by a noninear refinement process. We show the performance of our approach through experimental results. In these experiments, we will use an omnidirectional camera. The implication of this method is important because it brings 3D computer vision systems out of the laboratory and into practical use.

This paper describes a compact robot with two magnetic wheels in a bicycle arrangement, which is intended for inspecting the inner casing of pipes with complex shaped structures. The locomotion concept is based on an adapted magnetic wheel unit integrating two lateral lever arms. These arms allow for slightly lifting off the wheel in order to locally decrease the magnetic force, as well as laterally stabilizing the wheel unit. The robot has the main advantage to be compact and mechanically simple. It features 5 active degrees of freedom: 2 driven wheels each equipped with an active lifter-stabilizer and 1 steering unit. This paper also presents the design and implementation of a prototype robot and its high mobility is shown. It is able to pass 90deg convex and concave obstacles with any inclination regarding the gravity. Finally, it only requires limited space to maneuver, since turning on spot around the rear wheel is possible.

The future of robots, as our companions is dependent on their ability to understand, interpret and represent the environment in a human compatible manner. Towards this aim of making robots more spatially cognizant, the presented work is part of an attempt to create a hierarchical probabilistic concept-oriented representation of space, based on objects. Specifically, this work details efforts taken towards learning and generating concepts from the perceived objects and attempts to classify places using the concepts gleaned. The approach is based on learning from exemplars, clustering and the use of Bayesian network classifiers. Experiments on conceptualization and place classification are reported. Thus, the theme of the work is - conceptualization and classification for representation and spatial cognition.

This paper presents a new technique to estimate the extrinsic parameters of a robot-vision sensor system. More in general, this technique can be adopted to calibrate any robot bearing sensor. It is based on the extended Kalman filter. It is very simple and allows an automatic self-calibration during the robot motion. It only requires a source of light in the environment and an odometry system on the robot. The strategy is theoretically validated through an observability analysis which takes into account the system nonlinearities. This analysis shows that the system contains all the necessary information to perform the self-calibration. Furthermore, many accurate simulations and experiments performed on a real platform equipped with encoder sensors and an omnidirectional conic vision sensor, show the exceptional performance of the strategy

In this work we address the problem of perception, spatial cognition and topological navigation for a mobile robot. The objective of this work is to enable the navigation of an autonomous mobile robot (or vehicle) in an indoor (or outdoor) structured environment without relying on maps a priori learned and without using artificial landmarks. A new method for incremental and automatic topological mapping and global localization using fingerprints of places is presented. The fingerprint-based representation permits a reliable, compact and distinctive environment-modeling. Experimental results for mapping indoor and outdoor environments with a mobile robot and a "SMART" vehicle, both equipped with a multi-sensor system composed of two 180deg laser range finders and an omnidirectional camera are also reported

This paper presents an experimental evaluation of a plane extraction method using various line extraction algorithms. Four different algorithms are chosen, which are well known in mobile robotics and computer vision. Experiments are performed on two sets of 25 range images either obtained by simulation or acquired by a proprietary 3D laser scanner. The performance of the range image segmentation is measured in terms of an average segment classification ratio. Moreover, the speed of the method is measured to conclude on the suitability for service robot applications.

In this paper, we present a stereovision algorithm for real-time 6DoF ego-motion estimation, which integrates image intensity information and 3D stereo data in the well-known Iterative Closest Point (ICP) scheme. The proposed method addresses a basic problem of standard ICP, i.e. its inability to perform the segmentation of data points and to deal with large displacements. Neither a-priori knowledge of the motion nor inputs from other sensors are required, while the only assumption is that the scene always contains visually distinctive features which can be tracked over subsequent stereo pairs. This generates what is usually called Visual Odometry. The paper details the various steps of the algorithm and presents the results of experimental tests performed with an allterrain mobile robot, proving the method to be as accurate as effective for autonomous navigation purposes.

In this paper, we present a flexible new technique for single viewpoint omnidirectional camera calibration. The proposed method only requires the camera to observe a planar pattern shown at a few different orientations. Either the camera or the planar pattern can be freely moved. No a priori knowledge of the motion is required, nor a specific model of the omnidirectional sensor. The only assumption is that the image projection function can be described by a Taylor series expansion whose coefficients are estimated by solving a two-step least-squares linear minimization problem. To test the proposed technique, we calibrated a panoramic camera having a field of view greater than 200 in the vertical direction, and we obtained very good results. To investigate the accuracy of the calibration, we also used the estimated omni-camera model in a structure from motion experiment. We obtained a 3D metric reconstruction of a scene from two highly distorted omnidirectional images by using image correspondences only. Compared with classical techniques, which rely on a specific parametric model of the omnidirectional camera, the proposed procedure is independent of the sensor, easy to use, and flexible.

Using each other's knowledge and expertise in learning - what we call cooperation in learning- is one of the major existing methods to reduce the number of learning trials, which is quite crucial for real world applications. In situated systems, robots become expert in different areas due to being exposed to different situations and tasks. As a consequence, areas of expertise (AOE) of the other agents must be detected before using their knowledge, especially when the exchanged knowledge is not abstract, and simple information exchange might result in incorrect knowledge, which is the case for Q-learning agents. In this paper we introduce an approach for extraction of AOE of agents for cooperation in learning using their Q-tables. The evaluating robot uses a behavioral measure to evaluate itself, in order to find a set of states it is expert in. That set is used, then, along with a Q-table-based feature for extraction of areas of expertise of other robots by means of a classifier. Extracted areas are merged in the last stage. The proposed method is tested both in extensive simulations and in real world experiments using mobile robots. The results show effectiveness of the introduced approach, both in accurate extraction of areas of expertise and increasing the quality of the combined knowledge, even when, there are uncertainty and perceptual aliasing in the application and the robot

Decision trees, being human readable and hierarchically structured, provide a suitable mean to derive state-space abstraction and simplify the inclusion of the available knowledge for a reinforcement learning (RL) agent. In this paper, we address two approaches to combine and purify the available knowledge in the abstraction trees, stored among different RL agents in a multi-agent system, or among the decision trees learned by the same agent using different methods. Simulation results in nondeterministic football learning task provide strong evidences for enhancement in convergence rate and policy performance

This paper presents the design and preliminary control of an indoor coaxial helicopter "CoaX" based on a concept of hybrid active and passive control. The robot is capable of passive roll and pitch stabilization against small disturbances while actively stabilizing the yaw and the altitude. We achieve omnidirectional horizontal motion by shifting the center of gravity. The structure is made out of lightweight polyurethane and carbon fiber. Preliminary tests show that "CoaX" is able to hover for short moments and the lateral control is effective

Recently, intelligent transportation systems have been introduced for tasks like automated parking and highway driving. This is one of many contact points between human and robot intelligence, in that a human driver is sharing the driving task with intelligent computer systems. In this video we present an automated passenger vehicle that is able to autonomously navigate through both structured and unstructured B23environments without relying on prior environmental information or known waypoints. The system uses ego motion estimation based on an inertial measurement unit and internal vehicle sensors, and combines this with a laser range finder to map its environment. It uses a combination of global planning and local planning to safely navigate through the environment to a desired goal location.

The creation of a rover for a specific task requires designing and selecting the mechanical structure specifically for its mission. This can be done by modelling a chassis and evaluating it with specific criteria, which is the aim of the performance optimization tool presented here. This software makes it possible to compare and improve existing and new designs in a quick and efficient way. The tool presented in this paper is based on a quasi-static approach including optimization of the friction coefficients to model and evaluate the rover

This paper focuses on development of a motion planning strategy for car-like vehicles in dynamic urban-like scenarios. The strategy can be summarized as a search for a collision-free trajectory among linearly moving obstacles applying rapidly-exploring random trees (RRT) and B-splines. Collision avoidance is based on geometric search in transformed state space of chained form kinematic model decomposition. The time criterion for avoiding obstacles is based on relative robot to obstacle motion and is checked iteratively for possible collisions within the RRT exploration phase. The line segment geometric path is interpolated with a B-spline curve in order to generate a feasible trajectory that takes into account nonholonomic constraints. The exploration strategy aims at finding an optimal steering and longitudinal control of the vehicle in minimum time and steering activity sense. In order to test the strategy a MatLab based simulator was developed. This simulator reproduces a simple 2D urban-like environment with parked and moving cars, buses, trucks, people, buildings, streets, and trees. The test vehicle, a modified smart car equipped with several sensors was kinematically modeled. The sensor data are extracted from the environment based on its geometrical description and used as input data for the motion planning strategy which was verified in a dynamic urban scenario simulation

In this paper we derive theoretical results for the problem of on-line sensor calibration for a mobile robot. We consider the case of the odometry sensor. A first series of results regards the problem of understanding if a given system (consisting of a robot with several sensors) contains the necessary information to perform the on-line self calibration of one of its sensors. We consider several cases corresponding to different odometry systems and different types of robot sensors. Finally, we also consider the problem of maximizing the calibration accuracy and we formulate this problem as an optimal control problem. For the special case of a holonomic vehicle, we derive an analytical solution, i.e. we find the best trajectory which maximizes the calibration accuracy

Today, lightweight SLAM algorithms are needed in many embedded robotic systems. In this paper the orthogonal SLAM (OrthoSLAM ) algorithm is presented and empirically validated. The algorithm has constant time complexity in the state estimation and is capable to run real-time. The main contribution resides in the idea of reducing the complexity by means of an assumption on the environment. This is done by mapping only lines that are parallel or perpendicular to each other which represent the main structure of most indoor environments. The combination of this assumption with a Kalman filter and a relative map approach is able to map our laboratory hallway with the size of 80 m times 50 m and a trajectory of more than 500 m. The precision of the resulting map is similar to the measurements done by hand which are used as the ground-truth

In this paper, the independence relative map algorithm is presented. The algorithm aims to achieve the independence of relative map states. We show that using dependent relative quantities from the same observation creates a bias to the state covariance matrix, leading to an inaccurate and inconsistent relative map. Having independent map states improves the map consistency. Two case studies are presented in which we apply the proposed algorithm together with two popular relative map methods. Experimental results on simulated data show that the integrated algorithms outperform the original methods in term of map consistency and algorithm speed

This paper presents work on sensor-based motion planning in initially unknown dynamic environments. Motion detection and modeling are combined with a smooth navigation function to perform on-line path planning in cluttered dynamic environments. The SLIP algorithm, an extension of iterative closest point, combines motion detection from a mobile platform with position estimation. This information is used via probabilistic prediction to estimate a traversal risk function that unifies dynamic and static obstacles. The risk is fed to E* and leads to smooth paths that trade off collision risk versus detours

In this paper, we present a novel technique for calibrating central omnidirectional cameras. The proposed procedure is very fast and completely automatic, as the user is only asked to collect a few images of a checker board, and click on its corner points. In contrast with previous approaches, this technique does not use any specific model of the omnidirectional sensor. It only assumes that the imaging function can be described by a Taylor series expansion whose coefficients are estimated by solving a four-step least-squares linear minimization problem, followed by a non-linear refinement based on the maximum likelihood criterion. To validate the proposed technique, and evaluate its performance, we apply the calibration on both simulated and real data. Moreover, we show the calibration accuracy by projecting the color information of a calibrated camera on real 3D points extracted by a 3D sick laser range finder. Finally, we provide a Toolbox which implements the proposed calibration procedure

Information about the locomotion performance of known rovers is sparse. A comprehensive evaluation of wheeled passive systems is presented in this work. It is based on a static 2D approach that includes optimization of the wheel torques in order to minimize the required friction which is an important performance metric. The evaluation comprises well known rover concepts and new suspension systems. The performance of the systems is compared and interesting effects of some concepts are discussed in more detail including torques and load distribution. The rovers MER (NASA) and CRAB (EPFL) show good performance which is topped only by the eight wheeled DoubleSpring system

This paper presents an improved feature-based 3D SLAM approach for a mobile robot equipped with a rotating laser scanner. The features are represented using the SPmodel, with associated planar segment information based on decimated polygon sets. An extended Kalman filter is used to build a three-dimensional map of the environment and track the robot's pose. As shown, the resulting maps are highly detailed, useful for higher-level robotic tasks and small in size

The latest technological progress in sensors, actuators and energy storage devices enables the developments of miniature VTOL systems. In this paper we present the results of two nonlinear control techniques applied to an autonomous micro helicopter called Quadrotor. A backstepping and a sliding-mode techniques. We performed various simulations in open and closed loop and implemented several experiments on the test-bench to validate the control laws. Finally, we discuss the results of each approach. These developments are part of the OS4 project in our lab.

This paper presents a method for wheel-ground contact angle measurement and a traction control strategy minimizing slip in rough terrain. The slip minimization algorithm has been tested and compared with a standard speed control in simulation, which allows to verify the validity of the assumptions taken during the modeling phase. The simulations show clearly the advantage of torque control versus speed control. Furthermore, the proposed method has the advantage to avoid relying on complex wheel-soil interaction models, whose parameters are generally unknown in challenging terrains.

In this paper we consider the problem of simultaneously localizing all members of a team of robots. Each robot is equipped with proprioceptive sensors and exteroceptive sensors. The latter provide relative observations between the robots. Proprioceptive and exteroceptive data are fused with an Extended Kalman Filter. We derive the equations for this estimator for the most general relative observation between two robots. Then we consider three special cases of relative observations and we present the structure of the filter for each case. Finally, we study the performance of the approach through many accurate simulations.

In [5], a version of Relative Map Filter (RMF) is proposed to solve the simultaneous localization and map building (SLAM) problem. In the RMF, the map states contain only quantities invariant under shift and rotation. The estimation of the map states and their correlations is carried out in an optimal way using the Kalman filter. However, the dependency among the map states is not taken into account, thus the resulting map states are inconsistent. This paper presents two methods to enforce the consistency of the relative map states. The idea is to maintain a geometrically consistent map by solving a set of constraints between the map states. Experimental results obtained by using the proposed methods on real platform data show better performance than those deduced from the original RMF.

The E* algorithm is a path planning method capable of dynamic replanning and user-configurable path cost interpolation. It calculates a navigation function as a sampling of an underlying smooth goal distance that takes into account a continuous notion of risk that can be controlled in a fine-grained manner. E* results in more appropriate paths during gradient descent. Dynamic replanning means that changes in the environment model can be repaired to avoid the expenses of complete replanning. This helps compensating for the increased computational effort required for interpolation. We present the theoretical basis and a working implementation, as well as measurements of the algorithm’s precision, topological correctness, and computational effort.

This paper presents the latest developments around our mobile microrobot Alice. This small robot is the starting point driving and enabling enhancements in locomotion, energy, communication, perception and control. A set of new features and new HW modules is described. The robot itself is /spl sim/2/spl times/2/spl times/2 cm/sup 3/ and is able to move, sense, receive remote commands and locally communicate with neighbor robots. Extension modules implement a long range sensor, radio communication, a linear camera and an energy pack. The current projects in biomimetic, collective and evolutionary robotics using this set of tools are also shortly explained.

In this paper, the problem of localize two mobile robots is considered. The robots are equipped with proprioceptive sensors (like encoders) and exteroceptive sensors able to provide relative observations between them. In these observations, one robot detects and identifies the other one and measures some relative quantity. An observability analysis is performed by taking into account the system nonlinearities and for four different relative observations. The theoretical results are validated by simulations and experiments carried out on real platforms. In these experiments, an extended Kalman filter is adopted to fuse the information coming from the encoders and the sensors performing the observations.

This paper addresses the closing loop problem as the challenge of using all the information from the observation gathered when closing the loop in order to optimally adjust the whole map (assuming a correct data association). The proposed approach is an approximation, which allows the calculation of the gain without keeping track of all the correlations (i.e. with a complexity independent of the number of the map elements). Furthermore, the paper presents an explicit mathematical demonstration showing that the correlations computed by the EKF-based SLAM are overestimated. More precisely, it is shown that these correlations decrease exponentially with respect to the heading error of the robot. The approach is empirically demonstrated by means of meaningful simulations. The results are then discussed and conclusions are pointed out in the last section.

This paper presents an experimental evaluation of different line extraction algorithms on 2D laser scans for indoor environment. Six popular algorithms in mobile robotics and computer vision are selected and tested. Experiments are performed on 100 real data scans collected in an office environment with a map size of 80m /spl times/ 50m. Several comparison criteria are proposed and discussed to highlight the advantages and drawbacks of each algorithm, including speed, complexity, correctness and precision. The results of the algorithms are compared with the ground truth using standard statistical methods.

We introduce an initial implementation of a real-time component-based software framework for autonomous mobile robots. We argue that real-world autonomous mobile robots shall be controlled by self-contained software systems able to meet hard timing constraints. The proposed solution empowers specialized roboticists to contribute software components that can be integrated into complete real-time systems. The framework also facilitates robotic software components reuse and portability across hardware platforms. Based on an indoor navigation case study we evaluate the advantages and the limitations of the framework in terms of ease of use, modularity and real-time capabilities.

Abstract − Even today, robot mapping is one of the biggest challenges in mobile robotics. Geometric or topological maps can be used by a robot to navigate in the environment. Automatic creation of such maps is still problematic if the robot tries to map large environments. This paper presents a new method for incremental mapping using fingerprints of places. This type of representation permits a reliable, compact, and distinctive environment-modeling and makes navigation and localization easier for the robot. Experimental results for incremental mapping using a mobile robot equipped with a multi-sensor system composed of two 180 ° laser range finders and an omni-directional camera are also reported.

This paper presents the extension and experimental validation of the widely used EKF-based SLAM algorithm to 3D space. It uses planar features extracted probabilistically from dense three-dimensional point clouds generated by a rotating 2D laser scanner. These features are represented in compliance with the symmetries and perturbation model (SPmodel) in a stochastic map. As the robot moves, this map is updated incrementally while its pose is tracked by using an extended Kalman filter. After showing how three-dimensional data can be generated, the probabilistic feature extraction method is described, capable of robustly extracting (infinite) planes from structured environments. The SLAM algorithm is then used to track a robot moving through an indoor environment and its capabilities in terms of 3D reconstruction are analyzed.

Progresses in sensor technology, data processing and integrated actuators has made the development of miniature flying robots fully possible. Micro VTOL systems represent a useful class of flying robots because of their strong capabilities for small-area monitoring and building exploration. In this paper we describe the approach that our lab has taken to micro VTOL evolving towards full autonomy, and present the mechanical design, dynamic modelling, sensing, and control of our indoor VTOL autonomous robot OS4.

This paper presents general ideas on design and implementation of robots to be used together with groups of animals. This work is part of the European project LEURRE aiming to study, model and control mixed societies of animals and robots. After a short presentation of the general project, the implications on the special requirements are discussed. The design process of the robot are described leading to the implementation of the autonomous mini robot called InsBot. This is very compact (19/spl times/30/spl times/41 mm/sup 3/), has many sensors (12 IR proximity, linear camera and temperature) and is able to interact with gregarious cockroaches.

Navigating in rough terrain is a complex task that requires the robot to be considered as a holistic system. Algorithms, which don't consider the physical dimensions and capabilities of the mobile robot lead to inefficient motion and suffer from a lack of robustness. A physical model of the robot is necessary for trajectory control. In this paper, quasi-static modeling of a six-wheeled robot with a passive suspension mechanism is presented together with a method for selecting the optimal torques considering the system constraints: maximal and minimal torques, positive normal forces. The aim of this method is to limit wheel slip and to improve climbing capabilities. The modeling and the optimization are applied to the shrimp rover.

This paper starts with a discussion of the open challenges in the SLAM problem. In our opinion they can be grouped in two main and distinct areas: convergence of the built map and computation requirement for real world application. To deal with the previous problems, a solution in the stochastic map framework based on the concept of the relative map is proposed. The idea consists in introducing a map state, which only contains quantities invariant under shift and rotation and to carry out the estimation of this relative map in an optimal way. This is a possible way in order to have a decoupling between the robot motion and the landmark estimation and therefore not to rely the landmark estimation on the unmodeled error sources of the robot motion. Moreover, the proposed solution scales linearly with the number of landmark allowing real-time application. Experimental results, carried out on a real platform, show the better performance of this method with respect to the joint vehicle-landmark approach (absolute map filter) when the odometry is affected by undetected systematic errors or by large or unmodeled non-systematic errors.

This work presents a localization algorithm for indoor environments. The environmental model is topological and the approach describes how a multimodal perception increases the reliability for the topological localization problem for mobile robots, by using the Bayesian programming formalism. For the topological framework the fingerprint concept is used. This type of representation permits a reliable and distinctive environment modeling. Experimental results of a mobile robot equipped with a multi sensor system composed of two 180/spl deg/ laser range finders and an omni-directional camera are reported.

Programming efficient and reliable code can be considered a non-trivial task, as it requires deep understanding of the problem to be solved along with good programming skills. However, software frameworks and programming paradigms can provide a dependable infrastructure upon which better programs can be written and deployed. This allows engineers to focus mainly on their task, while relying on the underlying run-time environment for taking care of low-level programming issues, such as memory allocation and disposal, typing consistency and interface compliance. In this paper, we argue that strong-typed programming languages and paradigms offer a valid support for the production of reliable programs. Aware of the challenges of formal measurement metrics for code quality, we present the benefits of strong-typing by considering a practical application: the design and implementation of RoboX, a tour-guide robot for the Swiss National Exhibition Expo.02. The example is extremely well suited for such a discussion, since complex mechatronic applications can be considered critical systems - i.e., systems whose failure may endanger missions, lives and society - thus their reliability has to be made a prime concern.

This work presents a method for probabilistic plane fitting and an application to robotic 3D mapping. The plane is fitted in an orthogonal least-square sense and the output complies with the conventions of the symmetries and perturbation model (SPmodel). In the second part of the paper, the presented plane fitting method is used within a 3D mapping application. It is shown that by using probabilistic information, high precision 3D maps can be generated.

This paper presents a solution to the simultaneous localization and mapping (SLAM) problem in the stochastic map framework for a mobile robot navigating in an indoor environment. The approach is based on the concept of the relative map. The idea consists in introducing a map state, which only contains quantities invariant under translation and rotation. This is done in order to have a decoupling between the robot motion and the landmark estimation and therefore not to rely the landmark estimation on the unmodeled error sources of the robot motion. The case of the corner feature is here considered. The relative state estimated through the Kalman filter contains the distances and the relative orientations among the corners observed at the same tune. Therefore, this state is invariant with respect to the robot configuration (translation and rotation). Finally, an environment containing structures consisting of several corners is also investigated. Real experiments carried out with a mobile robot equipped with a 360/spl deg/ laser range finder show the performance of the approach.

The development of miniature flying robots has become a reachable dream, thanks to the new sensing and actuating technologies. Micro VTOL systems represent a useful class of flying robots because of their strong abilities for small-area monitoring and building exploration. In this paper, we present the results of two model-based control techniques applied to an autonomous four-rotor micro helicopter called quadrotor. A classical approach (PID) assumed a simplified dynamics and a modern technique (LQ) based on a more complete model. Various simulations were performed and several tests on the bench validate the control laws. Finally, we present the results of the first test in flight with the helicopter released. These developments are part of the OS4 project in our lab.

Scan alignment estimates the relative robot position from corresponding sets of data by identifying the transformation that minimizes a distance metric on these sets. Here, we present a method (SLIP) establishing correspondences between points based on a novel probabilistic distance metric to allow robust detection of outliers. This metric takes into account sensor noise and robot position uncertainty. Outliers are detected as elements with none but low probability links among all correspondences. To achieve scan alignment an inverse model is applied on the links, estimating robot position and reducing position uncertainty. Results of SLIP preserving all links and a computationally more efficient variant retaining only the most probable link are compared to standard ICP for tests with scan data and artificially inserted outliers. Additionally SLIP was used to built maps of an office environment from scan series. It was found to correct position errors and reject outliers from artificial data and real scans successfully.

This paper proposes a method that reasons intent of motion through task order such as "switch on the light." Intent of motion means kinds of path of a contact point between a target and an end effector when a robot recreates ordered task. The intent is used for teaching. After a robot fixes a framework of task using the intent, it asks questions to a user to specify concrete values in the framework, the user replies them. This interaction creates motions of the robot. The kinds of path are defined as linear, circular and point to point. Moreover, impossible task is also considered. Four sets of P/sub f/(w): Probability that a word represents a path, and C/sub p/(w}: Certainty of P/sub p/(w) are added to each word in an electric thesaurus to enable the reasoning. After values are input in part of the sets by 8 production rules, they are propagated to the other sets through hierarchical relationship of words in the thesaurus. When the user enters two words, namely task and target, argmax/sub p/{P/sub p/(w)*C/sub p/(w)} becomes answer. Moreover, update and re-propagation of values in the sets using answers from the user make the reasoning more precise.

Many algorithms related to localization need good pose prediction in order to produce accurate results. This is especially the case for data association algorithms, where false feature matches can lead to the localization system failure. In rough terrain, the field of view can vary significantly between two feature extraction steps, so a good position prediction is necessary to robustly track features. This paper presents a method for combining dead reckoning sensor information in order to provide an initial estimate of the six degrees of freedom of a rough terrain rover. An inertial navigation system (INS) and the wheel encoders are used as sensory inputs. The sensor fusion scheme is based on an extended information filter (EIF) and is extensible to any kind and number of sensors. In order to test the system, the rover has been driven on different kind of obstacles while computing both pure 3D-odometric and fused INS/3D-odometry trajectories. The results show that the use of the INS significantly improves the pose prediction.

This paper proposes an approach allowing topology learning and recognition in indoor environments by using a probabilistic approach called Bayesian Programming. The main goal of this approach is to cope with the uncertainty, imprecision and incompleteness of handled information. The Bayesian Program for topology recognition and door detection is presented. The method has been successfully tested in indoor environments with the BIBA robot, a fully autonomous robot. The experiments address both the topology learning and topology recognition capabilities of the approach.

This paper relates first experiences using a state-of-the-art, time-of-flight sensor that is able to deliver 3D images. The properties and capabilities of the sensor make it a potential powerful tool for applications within mobile robotics especially for real-time tasks, as the sensor features a frame rate of up to 30 frames per second. Its capabilities in terms of basic obstacle avoidance and local path-planning are evaluated and compared to the performance of a standard laser scanner.