Many robotic systems rely on visual sensing to accomplish simultaneously the tasks of state estimation, mapping, and path planning. One one hand, the usage of camera sensors represents a power-efficient and lightweight option for solving this problem. On the other hand, these tasks pose requirements on the quality of the visual input (e.g. number of tracked features for Visual Odometry) that are often in contrast to the optimal viewpoint planning for local mapping and obstacle avoidance. Dealing with this constraint is actively researched in the field of perception-aware planning. The approaches delivered by this field mostly concern Micro air vehicles (MAVs), but could be applied to a larger group of robotic systems. We propose a perception-aware trajectory planner for a class of robotic systems that can orient their cameras independently from their direction of travel. By using motion primitives, our planner does not require differentiable models for motion and perception objectives. We evaluate our method in simulation, showing increased capabilities in localization-aware motions around obstacles, and demonstrate its run-time capability on a real planetary rover. The code is released publicly under github.com/DLR-RM/palp.

We present a Multi-Criteria Decision Making (MCDM) framework specifically designed for planetary exploration. Our work is based on PROMETHEE II, which allows operators to add task-specific criteria and conditions. We extended this algorithm to improve its resource usage by reducing the number of candidate exploration goals that have to be evaluated and compared. This is crucial when given a large number of goals, as is typical for outdoor environments. In addition, we identified five different criteria for planetary exploration, including a novel criterion that we call Direction of Interest (DOI), and use a categorization of these for further resource optimizations. We thereby ensure that the CPU usage of our decision making method can meet the limited budget of a space rover. We present simulated and real-world experiments with the Lightweight Rover Unit (LRU) and show a reduction of the processing time for decision making of approx. 70%.

Future planetary missions will rely on rovers that can autonomously explore and navigate in unstructured environments. An essential element is the ability to recognize places that were already visited or mapped. In this work, we leverage the ability of stereo cameras to provide both visual and depth information, guiding the search and validation of loop closures from a multi-modal perspective. We propose to augment submaps that are created by aggregating stereo point clouds, with visual keyframes. Point clouds matches are found by comparing CSHOT descriptors and validated by clustering, while visual matches are established by comparing keyframes using Bag-of-Words (BoW) and ORB descriptors. The relative transformations resulting from both keyframe and point cloud matches are then fused to provide pose constraints between submaps in our graph-based SLAM framework. Using the LRU rover, we performed several tests in both an indoor laboratory environment as well as a challenging planetary analog environment on Mount Etna, Italy, consisting of areas where either keyframes or point clouds alone failed to provide adequate matches demonstrating the benefit of the proposed multi-modal approach.

The here presented flying system uses two pairs of wide-angle stereo cameras and maps a large area of interest in a short amount of time. We present a multicopter system equipped with two pairs of wide-angle stereo cameras and an inertial measurement unit (IMU) for robust visual-inertial navigation and time-efficient omni-directional 3D mapping. The four cameras cover a 240 degree stereo field of view (FOV) vertically, which makes the system also suitable for cramped and confined environments like caves. In our approach, we synthesize eight virtual pinhole cameras from four wide-angle cameras. Each of the resulting four synthesized pinhole stereo systems provides input to an independent visual odometry (VO). Subsequently, the four individual motion estimates are fused with data from an IMU, based on their consistency with the state estimation. We describe the configuration and image processing of the vision system as well as the sensor fusion and mapping pipeline on board the MAV. We demonstrate the robustness of our multi-VO approach for visual-inertial navigation and present results of a 3D-mapping experiment.

Despite the substantial progression of autonomous driving systems, their application is often limited e.g. due to safety margins which can be caused by uncertainties in the environment reconstruction. Then, via teleoperation as a fallback solution, a human-in-the-loop can be introduced as the main decision maker. However, high delay in the communication channel distorts the performance of direct force feedback teleoperation for example in space or disaster scenarios. On the other hand, model-mediated teleoperation can provide instantaneous and even predictive force feedback to the user, but the performance is limited due to state mismatches, incomplete models, model errors and the modeling challenges of complex wheel-ground contacts. Therefore, in this paper we introduce the concept of extended model-mediated teleoperation with a car like interface for mobile robots by fusing local fictitious and remote force feedback, which can be measured, computed or fictitious. We provide a method to guarantee stability of the extended model-mediated teleoperation (involving time delay, multilateral coupling, fictitious force feedback and permanent updates of the local model) based on the passivity theorem. The benefits of the approach are highlighted by human-in-the-loop experiments with a wheeled mobile robot.

A robotic system for search and rescue missions needs to efficiently explore and map new areas. In this paper, we present an integrated exploration strategy with active loop closing, which balances between the exploration speed and map quality. Specifically, it finds a trade-off between moving towards unknown space to gather new information and revisiting previous locations to improve localization accuracy and map quality through loop closures. Our integrated exploration is built upon a submap-based 6D SLAM system. Loop closure constraints originate from pairwise submap matches, which allow the optimization of an underlying SLAM graph. During exploration, we employ the expected information gain as well as the robot's localization uncertainty estimates to weigh exploration and revisiting actions online. We introduce the match effect as the expected impact of a loop closure on global optimization and consider this novel criterion together with the match likelihood and cost when evaluating the utility of revisiting previous locations. To demonstrate our approach, we present simulated and real-world experiments, comparing two variants of our novel method to a frontier-based exploration.

Autonomous robots operating in semi- or unstructured environments, e.g. during search and rescue missions, require methods for online on-board creation of maps to support path planning and obstacle avoidance. Perception based on stereo cameras is well suited for mixed indoor/outdoor environments. The creation of full 3D maps in GPS-denied areas however is still a challenging task for current robot systems, in particular due to depth errors resulting from stereo reconstruction. State-of-the-art 6D SLAM approaches employ graph-based optimization on the relative transformations between keyframes or local submaps. To achieve loop closures, correct data association is crucial, in particular for sensor input received at different points in time. In order to approach this challenge, we propose a novel method for submap matching. It is based on robust keypoints, which we derive from local obstacle classification. By describing geometrical 3D features, we achieve invariance to changing viewpoints and varying light conditions. We performed experiments in indoor, outdoor and mixed environments. In all three scenarios we achieved a final 3D position error of less than 0.23% of the full trajectory. In addition, we compared our approach with a 3D RBPF SLAM from previous work, achieving an improvement of at least 27% in mean 2D localization accuracy in different scenarios.

Teams of mobile robots can be deployed in search and rescue missions to explore previously unknown environments. Methods for joint localization and mapping constitute the basis for (semi-)autonomous cooperative action, in particular when navigating in GPS-denied areas. As communication losses may occur, a decentralized solution is required. With these challenges in mind, we designed a submap-based SLAM system that relies on inertial measurements and stereo-vision to create multi-robot dense 3D maps. For online pose and map estimation, we integrate the results of keyframe-based local reference filters through incremental graph SLAM. To the best of our knowledge, we are the first to combine these two methods to benefit from their particular advantages for 6D multi-robot localization and mapping: Local reference filters on each robot provide real-time, long-term stable state estimates that are required for stabilization, control and fast obstacle avoidance, whereas online graph optimization provides global multi-robot pose and map estimates needed for cooperative planning. We propose a novel graph topology for a decoupled integration of local filter estimates from multiple robots into a SLAM graph according to the filters' uncertainty estimates and independence assumptions and evaluated its benefits on two different robots in indoor, outdoor and mixed scenarios. Further, we performed two extended experiments in a multi-robot setup to evaluate the full SLAM system, including visual robot detections and submap matches as inter-robot loop closure constraints.

The creation of local and global maps is crucial for (semi-)autonomous operation of mobile robots in previously unknown environments, e.g. during search and rescue missions. We developed an on-board stereo-vision based mapping system, thereby introducing local obstacle maps that can directly be used for fast local obstacle avoidance and path planning. In addition, we designed them to constitute a suitable input to a widely-used simultaneous localization and mapping (SLAM) algorithm. We performed experiments in unknown indoor, unstructured outdoor as well as mixed environments and demonstrated the applicability of our method to camera setups with small as well as wide field of view. In all three scenarios, we achieved a final 2D position error of less than 0.08% of the full trajectory.

In the context of robotic assistants in human everyday environments, pick and place tasks are beginning to be competently solved at the technical level. The question of where to place objects or where to pick them up from, among other higher-level reasoning tasks, is therefore gaining practical relevance. In this work, we consider the problem of identifying the organizational structure within an environment, i.e. the problem of determining organizational principles that would allow a robot to infer where to best place a particular, previously unseen object or where to reasonably search for a particular type of object given past observations about the allocation of objects to locations in the environment. This problem can be reasonably formulated as a classification task. We claim that organizational principles are governed by the notion of similarity and provide an empirical analysis of the importance of various features in datasets describing the organizational structure of kitchens. For the aforementioned classification tasks, we compare standard classification methods, reaching average accuracies of at least 79% in all scenarios. We thereby show that, in particular, ontology-based similarity measures are well-suited as highly discriminative features. We demonstrate the use of learned models of organizational principles in a kitchen environment on a real robot system, where the robot identifies a newly acquired item, determines a suitable location and then stores the item accordingly.