Vision-based self-localization is a crucial technology for enabling autonomous robot navigation in GPS-deprived environments. However, standard frame cameras are subject to motion blur and suffer from a limited dynamic range. This research focuses on efficient feature tracking for self-localization by using event-based cameras. Such cameras do not provide regular snapshots of the environment but asynchronously collect events that correspond to a small delta of illumination in each pixel independently, thus addressing the issue of motion blur during fast motion and high dynamic range. Specifically, we propose a continuous real-time asynchronous event-based feature tracking pipeline, named RATE. This pipeline integrates (i) a corner detector node utilizing a time slice of the Surface of Active Events to initialize trackers continuously, along with (ii) a tracker node with a proposed "tracking manager", consisting of a grid-based distributor to reduce redundant trackers and to remove feature tracks of poor quality. Evaluations using public datasets reveal that our method maintains a stable number of tracked features, and performs real-time tracking efficiently while maintaining or even improving tracking accuracy compared to state-of-the-art event-only tracking methods. Our ROS implementation is released as open-source: https://github.com/mikihiroikura/RATE

Future planetary missions will rely on rovers that can autonomously explore and navigate in unstructured environments. An essential element is the ability to recognize places that were already visited or mapped. In this work, we leverage the ability of stereo cameras to provide both visual and depth information, guiding the search and validation of loop closures from a multi-modal perspective. We propose to augment submaps that are created by aggregating stereo point clouds, with visual keyframes. Point clouds matches are found by comparing CSHOT descriptors and validated by clustering, while visual matches are established by comparing keyframes using Bag-of-Words (BoW) and ORB descriptors. The relative transformations resulting from both keyframe and point cloud matches are then fused to provide pose constraints between submaps in our graph-based SLAM framework. Using the LRU rover, we performed several tests in both an indoor laboratory environment as well as a challenging planetary analog environment on Mount Etna, Italy, consisting of areas where either keyframes or point clouds alone failed to provide adequate matches demonstrating the benefit of the proposed multi-modal approach.

Suitable datasets are an integral part of robotics research, especially for training neural networks in robot perception. However, in many domains, suitable real-world data are scarce and cannot be easily obtained. This problem is especially prevalent for unstructured outdoor environments, in particular, planetary ones. Recent advances in photorealistic simulations help researchers to simulate close-to-real data in many domains. Yet, there exists no high-quality synthetic data for planetary exploration tasks. Also, existing simulators lack the fidelity required for generating planetary data, which is inherently less structured than human environments. Synthetic planetary data requires careful modeling and annotation of many different terrain aspect and details, such as textures and distributions of rocks, to become a valuable test-bed for robotics. To fill this gap, we present a novel simulator specifically designed for the needs of planetary robotics visual tasks, but also applicable for other outdoor environments. Our simulator is capable of generating large varieties of (planetary) outdoor scenes with rich generation of meta data, such as multilevel semantic and instance annotations. To demonstrate the wide applicability of this new simulator, we evaluate its performance on typical robotics applications, i.e. semantic segmentation, instance segmentation, and visual SLAM. Our simulator is accessible under https://github.com/DLR-RM/oaisys.

In the future, extraterrestrial expeditions will not only be conducted by rovers but also by flying robots. The technical demonstration drone Ingenuity, that just landed on Mars, will mark the beginning of a new era of exploration unhindered by terrain traversability. Robust self-localization is crucial for that. Cameras that are lightweight, cheap and information-rich sensors are already used to estimate the ego-motion of vehicles. However, methods proven to work in man-made environments cannot simply be deployed on other planets. The highly repetitive textures present in the wastelands of Mars pose a huge challenge to descriptor matching based approaches.In this paper, we present an advanced robust monocular odometry algorithm that uses efficient optical flow tracking to obtain feature correspondences between images and a refined keyframe selection criterion. In contrast to most other approaches, our framework can also handle rotation-only motions that are particularly challenging for monocular odometry systems. Furthermore, we present a novel approach to estimate the current risk of scale drift based on a principal component analysis of the relative translation information matrix. This way we obtain an implicit measure of uncertainty. We evaluate the validity of our approach on all sequences of a challenging real-world dataset captured in a Mars-like environment and show that it outperforms state-of-the-art approaches. The source code is publicly available at: https://github.com/DLR-RM/granit.

The ability to recognize previously mapped locations is an essential feature for autonomous systems. Unstructured planetary-like environments pose a major challenge to these systems due to the similarity of the terrain. As a result, the ambiguity of the visual appearance makes state-of-the-art visual place recognition approaches less effective than in urban or man-made environments. This paper presents a method to solve the loop closure problem using only spatial information. The key idea is to use a novel continuous and probabilistic representations of terrain elevation maps. Given 3D point clouds of the environment, the proposed approach exploits Gaussian Process (GP) regression with linear operators to generate continuous gradient maps of the terrain elevation information. Traditional image registration techniques are then used to search for potential matches. Loop closures are verified by leveraging both the spatial characteristic of the elevation maps (SE (2) registration) and the probabilistic nature of the GP representation. A submap-based localization and mapping framework is used to demonstrate the validity of the proposed approach. The performance of this pipeline is evaluated and benchmarked using real data from a rover that is equipped with a stereo camera and navigates in challenging, unstructured planetary-like environments in Morocco and on Mt. Etna.

We propose an appearance-based along-route localization algorithm that relies on robust place recognition by matching image sequences instead of individual frames. Our approach extends state of the art place recognition framework SeqSLAM in several aspects to realize real-time localization along routes for autonomous navigation. First, our method is online in that we only rely on the recently observed image frames. Second, we provide a homing mechanism based on rotations computed from frame matches. And third, we use a more flexible mechanism to search for matching locations, not restricting the search to straight lines in the cost matrix as in SeqSLAM, but allowing for a wide variety of route traversal conditions such as varying velocities or rotational and translational viewpoint differences. We investigate different image preprocessing steps as well as image similarity metrics wrt. their influence on illumination and viewpoint invariance for a more robust place recognition. On a new challenging dataset, recorded in real world experiments with a planetary rover, in the course of a Moon-analogue mission on Sicily's Mount Etna, we show the feasibility of our direct, sequence-based approach to along-route localization.

The here presented flying system uses two pairs of wide-angle stereo cameras and maps a large area of interest in a short amount of time. We present a multicopter system equipped with two pairs of wide-angle stereo cameras and an inertial measurement unit (IMU) for robust visual-inertial navigation and time-efficient omni-directional 3D mapping. The four cameras cover a 240 degree stereo field of view (FOV) vertically, which makes the system also suitable for cramped and confined environments like caves. In our approach, we synthesize eight virtual pinhole cameras from four wide-angle cameras. Each of the resulting four synthesized pinhole stereo systems provides input to an independent visual odometry (VO). Subsequently, the four individual motion estimates are fused with data from an IMU, based on their consistency with the state estimation. We describe the configuration and image processing of the vision system as well as the sensor fusion and mapping pipeline on board the MAV. We demonstrate the robustness of our multi-VO approach for visual-inertial navigation and present results of a 3D-mapping experiment.

A lightweight visual compass system is presented as well as a direct method for estimating sun direction and its covariance. The optical elements of the system are described enabling estimation of sky polarization in a FOV of approx. 56° with a single standard camera sensor. Using the proposed direct method, the sun direction and its covariance matrix can be estimated based on the polarization measured in the image plane. Experiments prove the applicability of the polarization sensor and the proposed estimation method, even in difficult conditions. It is also shown that in case the sensor is not leveled, combination with an IMU allows to determine all degrees of orientation. Due to the low weight of the sensor and the low complexity of the estimation method the polarization system is well suited for MAVs which have limited payload and computational resources. Furthermore, since not just the sun direction but also its covariance is estimated an integration in a multi-sensor navigation framework is straight forward.

Map based navigation is a crucial task for any mobile robot. On many platforms this problem is addressed by applying Simultaneous Localization and Mapping (SLAM) based on metric grid-maps. Such solutions work well on robots with adequate resources and limited workspaces. Platforms with limited payload which operate in unbounded workspaces, do often have insufficient resources to keep a metric world representation. Nevertheless, many applications demand that the robot can autonomously navigate between different operation areas. In this work the Landmark-Tree map (LT-map), a resource efficient topological map concept, is for the first time applied to a mobile robotic platform equipped with an omnidirectional camera. It enables the robot to efficiently adapt the acquired map online to the available memory. During map acquisition and navigation the motion is estimated by the Z∞-algorithm. Both methods are based on similar concepts, which results in a mutual benefit. An efficient navigation strategy based on the LT-map allows the robot to reliably follow previously recorded paths. The presented approach is evaluated on a mobile robot in indoor and outdoor scenarios. The experiments prove its feasibility and show that pruning the map just smooths the trajectories, which is the expected and desired behaviour.

We present a generalization of the Koenderink-van Doorn (KvD) algorithm that allows robust monocular localization with large motion between the camera frames for a wide range of optical systems including omnidirectional systems and standard perspective cameras. The KvD algorithm estimates simultaneously ego-motion parameters, i.e. rotation, translation, and object distances in an iterative way. However due to the linearization of the rotational component of optic flow, the original algorithm fails for larger rotations. We present a generalization of the algorithm to arbitrary rotations that is especially suited for omnidirectional cameras where features can be tracked for long sequences. This reduces the need for vector summation of several individual motion estimates that leads to accumulation of odometry errors. The significant improvement in the performance of the proposed generalized algorithm compared to the original KvD implementation is validated using simulated data. The algorithm is also tested in a real-world experiment with ground-truth data obtained from an external tracking system. The experiment was carried out using a novel compact omnidirectional camera that is designed for small aerial vehicles. It consists of an off-the-shelf webcam that is combined with a reflective surface machined into acrylic glass.