Mobile AR applications benefit from fast initialization to display world-locked effects instantly. However, standard visual odometry or SLAM algorithms require motion parallax to initialize (see Figure 1) and, therefore, suffer from delayed initialization. In this paper, we present a 6-DoF monocular visual odometry that initializes instantly and without motion parallax. Our main contribution is a pose estimator that decouples estimating the 5-DoF relative rotation and translation direction from the 1-DoF translation magnitude. While scale is not observable in a monocular vision-only setting, it is still paramount to estimate a consistent scale over the whole trajectory (even if not physically accurate) to avoid AR effects moving erroneously along depth. In our approach, we leverage the fact that depth errors are not perceivable to the user during rotation-only motion. However, as the user starts translating the device, depth becomes perceivable and so does the capability to estimate consistent scale. Our proposed algorithm naturally transitions between these two modes. Our second contribution is a novel residual in the relative pose problem to further improve the results. The residual combines the Jacobians of the functional and the functional itself and is minimized using a Levenberg-Marquardt optimizer on the 5-DoF manifold. We perform extensive validations of our contributions with both a publicly available dataset and synthetic data. We show that the proposed pose estimator outperforms the classical approaches for 6-DoF pose estimation used in the literature in low-parallax configurations. Likewise, we show our relative pose estimator outperforms state-of-the-art approaches in an odometry pipeline configuration where we can leverage initial guesses. We release a dataset for the relative pose problem using real data to facilitate the comparison with future solutions for the relative pose problem. Our solution is either used as a full odometry or as a pre-SLAM component of any supported SLAM system (ARKit, ARCore) in world-locked AR effects on platforms such as Instagram and Facebook.

This paper reports on an algorithm for planning trajectories that allow a multirotor micro aerial vehicle (MAV) to quickly identify a set of unknown parameters. In many problems like self calibration or model parameter identification some states are only observable under a specific motion. These motions are often hard to find, especially for inexperienced users. Therefore, we consider system model identification in an active setting, where the vehicle autonomously decides what actions to take in order to quickly identify the model. Our algorithm approximates the belief dynamics of the system around a candidate trajectory using an extended Kalman filter (EKF). It uses sampling-based motion planning to explore the space of possible beliefs and find a maximally informative trajectory within a user-defined budget. We validate our method in simulation and on a real system showing the feasibility and repeatability of the proposed approach. Our planner creates trajectories which reduce model parameter convergence time and uncertainty by a factor of four.

Environmental conditions and external effects, such as shocks, have a significant impact on the calibration parameters of visual-inertial sensor systems. Thus long-term operation of these systems cannot fully rely on factory calibration. Since the observability of certain parameters is highly dependent on the motion of the device, using short data segments at device initialization may yield poor results. When such systems are additionally subject to energy constraints, it is also infeasible to use full-batch approaches on a big dataset and careful selection of the data is of high importance. In this paper, we present a novel approach for resource efficient self-calibration of visual-inertial sensor systems. This is achieved by casting the calibration as a segment-based optimization problem that can be run on a small subset of informative segments. Consequently, the computational burden is limited as only a predefined number of segments is used. We also propose an efficient information-theoretic selection to identify such informative motion segments. In evaluations on a challenging dataset, we show our approach to significantly outperform state-of-the-art in terms of computational burden while maintaining a comparable accuracy.

As the applications of Micro Aerial Vehicles (MAVs) get more and more complex, and require highly dynamic motions, it becomes essential to have an accurate dynamic model of the MAV. Such a model can be used for reliable state estimation, control, and for realistic simulation. A good model requires accurate estimates of physical parameters of the system, which we aim to estimate from recorded flight data. In this paper, we present a detailed physical model of the MAV and a maximum likelihood estimation scheme for determining the dominant parameters, such as inertia matrix, center of gravity (CoG) with respect to the IMU, and parameters related to the aerodynamics. To incorporate all information given by the IMU and the physical MAV model, we propose to use two process models in the optimization. We show the effectiveness of the method on simulated data, as well as on a real platform.

This paper addresses the challenges of the design, development and control of a new convertible VTOL tailsitter unmanned aerial vehicle that combines the advantages of both fixed wing and rotary wing systems. Wind tunnel measurements are used to get an understanding of the control allocation and to model the static forces and moments acting on the system. Based on the derived model, a novel controller that operates in SO(3) and handles the dynamics of the vehicle at any attitude configuration, including the rotorcraft and fixed-wing regimes as well as their transitions, is presented. This unified controller allows the autonomous transition of the system without discontinuities of switching, as well as its overall high performance flight control. The capabilities and flying qualities of the platform and the controller are demonstrated and evaluated by means of extensive experimental studies.

In this paper we present a new estimation algorithm that allows for the combination of information from any number of process and measurement models. This adds more flexibility to the design of the estimator and in our case avoids the need for state augmentation. We achieve this by adapting the maximum likelihood formulation of the Kalman Filter, and thereby represent all measurement models as residuals. Posing the problem in this form allows for the straightforward integration of any number of (nonlinear) constraints between two subsequent states. To solve the optimization we present a closed form recursive set of equations that directly marginalizes out information that is not required, this leads to an efficient and generic implementation. The new algorithm is applied to parameter estimation on MAVs which have two dynamic models, the MAV dynamic model and the IMU-driven model. We show the benefits and limitations of the new filtering approach on a simplified simulation example and on a real MAV system.

Multirotor unmanned aerial vehicles (UAVs) are rapidly gaining popularity for many applications. However, safe operation in partially unknown, unstructured environments remains an open question. In this paper, we present a continuous-time trajectory optimization method for real-time collision avoidance on multirotor UAVs. We then propose a system where this motion planning method is used as a local replanner, that runs at a high rate to continuously recompute safe trajectories as the robot gains information about its environment. We validate our approach by comparing against existing methods and demonstrate the complete system avoiding obstacles on a multirotor UAV platform.

Within this paper, a new fast algorithm that provides efficient solutions to the problem of inspection path planning for complex 3D structures is presented. The algorithm assumes a triangular mesh representation of the structure and employs an alternating two-step optimization paradigm to find good viewpoints that together provide full coverage and a connecting path that has low cost. In every iteration, the viewpoints are chosen such that the connection cost is reduced and, subsequently, the tour is optimized. Vehicle and sensor limitations are respected within both steps. Sample implementations are provided for rotorcraft and fixed-wing unmanned aerial systems. The resulting algorithm characteristics are evaluated using simulation studies as well as multiple real-world experimental test-cases with both vehicle types.

In this work, we present a model-based estimation scheme for multi-rotor Micro Aerial Vehicles (MAVs). Although modeling approaches for MAVs have been presented in the past, these models have rarely been used for real-time state estimation onboard MAVs. Building on this work, we identify the most dominant effects and propose an easy-to-use calibration scheme for estimation of the model parameters. Given the calibration estimates for these parameters, we derive a state estimator where the state prediction of the indirect Extended Kalman Filter (EKF) is driven by a MAV model. Solely using measurements from the Inertial Measurement Unit (IMU) and a barometric pressure sensor - both available on almost every MAV - our model-based formulation keeps the estimated velocity of the MAV bounded in all directions, as opposed to state of the art IMU-model driven state estimators onboard MAVs. This is crucial for keeping MAVs airborne safely, for instance in the case of failures or re-initialization of vision based localization systems.

In this work, we present an MAV system that is able to relocalize itself, create consistent maps and plan paths in full 3D in previously unknown environments. This is solely based on vision and IMU measurements with all components running onboard and in real-time. We use visual-inertial odometry to keep the MAV airborne safely locally, as well as for exploration of the environment based on high-level input by an operator. A globally consistent map is constructed in the background, which is then used to correct for drift of the visual odometry algorithm. This map serves as an input to our proposed global planner, which finds dynamic 3D paths to any previously visited place in the map, without the use of teach and repeat algorithms. In contrast to previous work, all components are executed onboard and in real-time without any prior knowledge of the environment.

Localization is essential for robots to operate autonomously, especially for extended periods of time, when estimator drift tends to destroy alignment to any global map. Though there has been extensive work in vision-based localization in recent years, including several systems that show real-time performance, none have been demonstrated running entirely on-board in closed loop on robotic platforms. We propose a fast, real-time localization system that keeps the existing local visual-inertial odometry frame consistent for controllers and collision avoidance, while correcting drift and alignment to a global coordinate frame. We demonstrate our localization system entirely on-board an aerial and ground robot, showing a collaboration experiment where both robots are able to localize against the same map accurately enough to allow the multicopter to land on top of the ground robot. We also perform extensive evaluations for the proposed closed-loop system on ground-truth datasets from MAV flight in an industrial setting.

The challenge of aerial robotic physical interaction towards inspection of infrastructure facilities through contact is the main motivation of this paper. A hybrid model predictive control framework is proposed, based on which a typical quadrotor vehicle becomes capable of stable physical interaction, accurate trajectory tracking on environmental surfaces as well as force control with only minor structural adaptations. Convex optimization techniques enabled the explicit computation of such a controller which accounts for the dynamics in free-flight and during physical interaction, ensures the stability of the hybrid system as well as response optimality, while respecting system constraints and imposed logical rules. This control framework is further extended to include obstacle avoidance capabilities. Extensive experimental studies that included complex “aerial-writing” tasks, interaction with non-planar and textured surfaces and obstacle avoidance maneuvers, indicate the efficiency of the approach and the potential capabilities of such aerial robotic physically interacting operations.

Robust, accurate pose estimation and mapping at real-time in six dimensions is a primary need of mobile robots, in particular flying Micro Aerial Vehicles (MAVs), which still perform their impressive maneuvers mostly in controlled environments. This work presents a visual-inertial sensor unit aimed at effortless deployment on robots in order to equip them with robust real-time Simultaneous Localization and Mapping (SLAM) capabilities, and to facilitate research on this important topic at a low entry barrier. Up to four cameras are interfaced through a modern ARM-FPGA system, along with an Inertial Measurement Unit (IMU) providing high-quality rate gyro and accelerometer measurements, calibrated and hardware-synchronized with the images. This facilitates a tight fusion of visual and inertial cues that leads to a level of robustness and accuracy which is difficult to achieve with purely visual SLAM systems. In addition to raw data, the sensor head provides FPGA-pre-processed data such as visual keypoints, reducing the computational complexity of SLAM algorithms significantly and enabling employment on resource-constrained platforms. Sensor selection, hardware and firmware design, as well as intrinsic and extrinsic calibration are addressed in this work. Results from a tightly coupled reference visual-inertial motion estimation framework demonstrate the capabilities of the presented system.

This paper presents Skye, a novel blimp design. Skye is a helium-filled sphere of diameter 2.7m with a strong inelastic outer hull and an impermeable elastic inner hull. Four tetrahedrally-arranged actuation units (AU) are mounted on the hull for locomotion, with each AU having a thruster which can be rotated around a radial axis through the sphere center. This design provides redundant control in the six degrees of freedom of motion, and Skye is able to move omnidirectionally and to rotate around any axis. A multi-camera module is also mounted on the hull for capture of aerial imagery or live video stream according to an ‘eyeball’ concept — the camera module is not itself actuated, but the whole blimp is rotated in order to obtain a desired camera view. Skye is safe for use near people — the double hull minimizes the likelihood of rupture on an unwanted collision; the propellers are covered by grills to prevent accidental contact; and the blimp is near neutral buoyancy so that it makes only a light impact on contact and can be readily nudged away. The system is portable and deployable by a single operator — the electronics, AUs, and camera unit are mounted externally and are detachable from the hull during transport; operator control is via an intuitive touchpad interface. The motivating application is in entertainment robotics. Skye has a varied motion vocabulary such as swooping and bobbing, plus internal LEDs for visual effect. Computer vision enables interaction with an audience. Experimental results show dexterous maneuvers in indoor and outdoor environments, and non-dangerous impacts between the blimp and humans.