In this paper, we present a local, robot-centric navigation map optimized for autonomous mobile robots operating in unknown environments, enhancing their onboard perception systems for collision-free operation with far look-ahead distances. Utilizing a novel converging covariance cell representation, our approach effectively analyzes hazards such as obstacles and hazardous slopes in both terrestrial and aerial navigation contexts. The new technique specifically targets mapping from stereo scenarios with ultra short baseline and highly oblique viewpoints close to the ground.Our methodology surpasses traditional window-based hazard analysis by resolving sub-cell size obstacles and terrain gradients at the individual cell level, thereby avoiding the computational overhead typically associated with such analyses. It leverages a multi-resolution strategy adaptive to the range errors common in stereo vision systems, making it particularly suitable for embedded systems with computational limitations.Functionality includes constant-time queries for height, obstacle presence, and slope details, boasting improvements in run time, memory usage, precision, and resolvable obstacle size compared to existing grid-based mapping algorithms. We validate our approach through rigorous simulation and real-world testing. This technique will be used for the local mapping and collision avoidance on NASA’s CADRE lunar rovers.

The next generation of Mars rotorcrafts requires on-board autonomous hazard avoidance landing. To this end, this work proposes a system that performs continuous multi-resolution height map reconstruction and safe landing spot detection. Structure-from-Motion measurements are aggregated in a pyramid structure using a novel Optimal Mixture of Gaus-sians formulation that provides a comprehensive uncertainty model. Our multiresolution pyramid is built more efficiently and accurately than past work by decoupling pyramid filling from the measurement updates of different resolutions. To detect the safest landing location, after an optimized hazard segmentation, we use a mean shift algorithm on multiple distance transform peaks to account for terrain roughness and uncertainty. The benefits of our contributions are evaluated on real and synthetic flight data.

Monocular Visual-Inertial Odometry (VIO) has become ubiquitous for navigation of autonomous Micro Air Vehicles (MAVs). Yet, state-of-the-art VIO is still very failure-prone, which can have dramatic consequences. To prevent this, VIO must be able to re-initialize in mid-air, either during a free fall or on a constant velocity trajectory after attitude control has been re-established. However, for both of these trajectories, the visual scale cannot be observed with VIO batch initializers because of the absence of acceleration change. We propose to use a small and lightweight laser-range finder (LRF) and a scene facet model to initialize vision-based navigation at the right scale under any motion condition and over any scene structure. This new range constraint is integrated into a visual-inertial bundle-adjustment initializer. We evaluate our approach in simulation, including robustness to various parameters, and demonstrate on real data how this approach can address midair state estimation failure in real-time.

In this paper, we propose a resource-efficient approach to provide an autonomous UAV with an on-board perception method to detect safe, hazard-free landing sites during flights over complex 3D terrain. We aggregate 3D measurements acquired from a sequence of monocular images by a Structure-from-Motion approach into a local, robot-centric, multi-resolution elevation map of the overflown terrain, which fuses depth measurements according to their lateral surface resolution (pixel-footprint) in a probabilistic framework based on the concept of dynamic Level of Detail. Map aggregation only requires depth maps and the associated poses, which are obtained from an on-board Visual Odometry algorithm. An efficient landing site detection method then exploits the features of the underlying multi-resolution map to detect safe landing sites based on slope, roughness, and quality of the reconstructed terrain surface. The evaluation of the performance of the mapping and landing site detection modules are analyzed independently and jointly in simulated and real-world experiments in order to establish the efficacy of the proposed approach.

The ability to find safe landing sites over complex 3D terrain is an essential safety feature for fully autonomous small unmanned aerial systems (UAS), which requires on-board perception for 3D reconstruction and terrain analysis if the overflown terrain is unknown. This is a challenge for UAS that are limited in size, weight and computational power, such as small rotorcrafts executing autonomous missions on Earth, or in planetary applications such as the Mars Helicopter. For such a computationally constraint system, we propose a structure from motion approach that uses inputs from a single downward facing camera to produce dense point clouds of the overflown terrain in real time. In contrast to existing approaches, our method uses metric pose information from a visual-inertial odometry algorithm as camera pose priors, which allows deploying a fast pose refinement step to align camera frames such that a conventional stereo algorithm can be used for dense 3D reconstruction. We validate the performance of our approach with extensive evaluations in simulation, and demonstrate the feasibility with data from UAS flights.

Many multirotor Unmanned Aerial Systems applications have a critical need for precise position control in environments with strong dynamic external disturbances such as wind gusts or ground and wall effects. Moreover, to maximize flight time, small multirotor platforms have to operate within strict constraints on payload and thus computational performance. In this paper, we present the design and experimental comparison of Model Predictive and PID multirotor position controllers augmented with a disturbance estimator to reject strong wind gusts up to 12 m/s and ground effect. For disturbance estimation, we compare Extended and Unscented Kalman filtering. In extensive in- and outdoor flight tests, we evaluate the suitability of the developed control and estimation algorithms to run on a computationally constrained platform. This allows to draw a conclusion on whether potential performance improvements justify the increased computational complexity of MPC for multirotor position control and UKF for disturbance estimation.

We propose an approach to the throw-and-go (TnG) problem for micro air vehicles (MAVs) using visual and inertial sensors. The key challenge is the fast on-board initialization of the visual odometry (VO) system, which usually requires user input to recover the visual scale. Our approach is based on the identification of the gravity vector from the acceleration data computed with images of the ground during in free fall. This enables scaling of the poses reconstructed with visual information. The proposed framework use inertial data to control the MAV attitude so the ground is visible after the throw. Using image to image homography a metric scale is estimated with which the MAV's height is propagated. Unlike existing literature, this approach requires no additional sensor nor user input or pre-throw assumptions and can recover from any initial attitude. We show results on both simulation and real data.

Many unmanned aerial vehicle surveillance and monitoring applications require observations at precise locations over long periods of time, ideally days or weeks at a time (e.g. ecosystem monitoring), which has been impractical due to limited endurance and the requirement of humans in the loop for operation. To overcome these limitations, we propose a fully autonomous small rotorcraft UAS that is capable of performing repeated sorties for long-term observation missions without any human intervention. We address two key technologies that are critical for such a system: full platform autonomy including emergency response to enable mission execution independently from human operators, and the ability of vision-based precision landing on a recharging station for automated energy replenishment. Experimental results of up to 11 hours of fully autonomous operation in indoor and outdoor environments illustrate the capability of our system.

Sensing the 3D environment of a moving robot is essential for collision avoidance. Most 3D sensors produce dense depth maps, which are subject to imperfections due to various environmental factors. Temporal fusion of depth maps is crucial to overcome those. Temporal fusion is traditionally done in 3D space with voxel data structures, but it can be approached by temporal fusion in image space, with potential benefits in reduced memory and computational cost for applications like reactive collision avoidance for micro air vehicles. In this paper, we present an efficient Gaussian Mixture Models based depth map fusion approach, introducing an online update scheme for dense representations. The environment is modeled from an ego-centric point of view, where each pixel is represented by a mixture of Gaussian inverse-depth models. Consecutive frames are related to each other by transformations obtained from visual odometry. This approach achieves better accuracy than alternative image space depth map fusion techniques at lower computational cost.

Fusing data from multiple sensors on-board a mobile platform can significantly augment its state estimation abilities and enable autonomous traversals of different domains by adapting to changing signal availabilities. However, due to the need for accurate calibration and initialization of the sensor ensemble as well as coping with erroneous measurements that are acquired at different rates with various delays, multi-sensor fusion still remains a challenge. In this paper, we introduce a novel multi-sensor fusion approach for agile aerial vehicles that allows for measurement validation and seamless switching between sensors based on statistical signal quality analysis. Moreover, it is capable of self-initialization of its extrinsic sensor states. These initialized states are maintained in the framework such that the system can continuously self-calibrate. We implement this framework on-board a small aerial vehicle and demonstrate the effectiveness of the above capabilities on real data. As an example, we fuse GPS data, ultra-wideband (UWB) range measurements, visual pose estimates, and IMU data. Our experiments demonstrate that our system is able to seamlessly filter and switch between different sensors modalities during run time.

In this paper, we present a novel approach in combining visual and inertial measurements in non-static environments for first order characterization of the metric motion of non-static objects in the scene. Our approach leverages online estimated ego motion states and uses a novel inertial-optical flow (IOF) measurement analysis to identify moving objects and to characterize them in their angular and linear velocities. The novelty of our algorithm lies in the identification and segmentation of consistent optical flow outliers in the so-called kinematic space. These consistent outliers in combination with the IOF information for ego-motion estimation yield a first order estimation of the moving object in full 3D and in metric units. The approach is highly efficient as it only requires matched features in two consecutive images. We evaluate and demonstrate our algorithm in simulations and in real world tests.

In this paper, we describe a novel method using only optical flow from a single camera and inertial information to quickly initialize, deploy, and autonomously stabilize an inherently unstable aerial vehicle. Our approach requires a minimal number of tracked features in only two consecutive frames and inertial readings eliminating the need of long feature tracks or local maps and rendering it inherently failsafe. We show theoretically, in simulation, and in real experiments that we can reliably estimate and control the vehicle velocity, full attitude, and metric distance to the scene while self-calibrating inertial intrinsics and sensor extrinsics. In fact, the fast initialization, self-calibration, and inherent fail-safe property leads to the first visual-inertial throw-and-go capable system.

We address obstacle avoidance for outdoor flight of micro air vehicles. The highly textured nature of outdoor scenes enables camera-based perception, which will scale to very small size, weight, and power with very wide, two-axis field of regard. In this paper, we use forward-looking stereo cameras for obstacle detection and a downward-looking camera as an input to state estimation. For obstacle representation, we use image space with the stereo disparity map itself. We show that a C-space-like obstacle expansion can be done with this representation and that collision checking can be done by projecting candidate 3-D trajectories into image space and performing a z-buffer-like operation with the disparity map. This approach is very efficient in memory and computing time. We do motion planning and trajectory generation with an adaptation of a closed-loop RRT planner to quadrotor dynamics and full 3D search. We validate the performance of the system with Monte Carlo simulations in virtual worlds and flight tests of a real quadrotor through a grove of trees. The approach is designed to support scalability to high speed flight and has numerous possible generalizations to use other polar or hybrid polar/Cartesian representations and to fuse data from additional sensors, such as peripheral optical flow or radar.

In this paper, we describe a novel approach in fusing optical flow with inertial cues (3D acceleration and 3D angular velocities) in order to navigate a Micro Aerial Vehicle (MAV) drift free in 4DoF and metric velocity. Our approach only requires two consecutive images with a minimum of three feature matches. It does not require any (point) map nor any type of feature history. Thus it is an inherently failsafe approach that is immune to map and feature-track failures. With these minimal requirements we show in real experiments that the system is able to navigate drift free in all angles including yaw, in one metric position axis, and in 3D metric velocity. Furthermore, it is a power-on-and-go system able to online self-calibrate the inertial biases, the visual scale and the full 6DoF extrinsic transformation parameters between camera and IMU.

A Feature and Pose Constrained Extended Kalman Filter (FPC-EKF) is developed for highly dynamic computationally constrained micro aerial vehicles. Vehicle localization is achieved using only a low performance inertial measurement unit and a single camera. The FPC-EKF framework augments the vehicle's state with both previous vehicle poses and critical environmental features, including vertical edges. This filter framework efficiently incorporates measurements from hundreds of opportunistic visual features to constrain the motion estimate, while allowing navigating and sustained tracking with respect to a few persistent features. In addition, vertical features in the environment are opportunistically used to provide global attitude references. Accurate pose estimation is demonstrated on a sequence including fast traversing, where visual features enter and exit the fleld-of-view quickly, as well as hover and ingress maneuvers where drift free navigation is achieved with respect to the environment.

This paper presents a new stereo algorithm for computing dense disparity maps from stereo image pairs by a global cost relaxation, realized as an optimization problem, where the disparity map is the momentary state of a dynamic process. Following the natural role model of the human visual system, we assign a set of possible disparities to each image pixel described by cooperating probability variables. In the first step a correlation-based similarity measure is performed to initialize the relaxation process. The relaxation itself is formulated as an optimization of a global cost function taking into account both the stereoscopic continuity constraint and considerations of the pixel similarity. A special formulation guarantees the existence of a unique cost minimum which can be easily and rapidly found by standard numerical procedures. In a post-processing step, occluded areas are detected and a sub-pixel precise disparity map is computed.