In this paper, we present the vision-aided inertial navigation (VISINAV) algorithm that enables precision planetary landing. The vision front-end of the VISINAV system extracts 2-D-to-3-D correspondences between descent images and a surface map (mapped landmarks), as well as 2-D-to-2-D feature tracks through a sequence of descent images (opportunistic features). An extended Kalman filter (EKF) tightly integrates both types of visual feature observations with measurements from an inertial measurement unit. The filter computes accurate estimates of the lander's terrain-relative position, attitude, and velocity, in a resource-adaptive and hence real-time capable fashion. In addition to the technical analysis of the algorithm, the paper presents validation results from a sounding-rocket test flight, showing estimation errors of only 0.16 m/s for velocity and 6.4 m for position at touchdown. These results vastly improve current state of the art for terminal descent navigation without visual updates, and meet the requirements of future planetary exploration missions.

Visual odometry can augment or replace wheel odometry when navigating in high slip terrain which is quite important for autonomous navigation on Mars. We present a computationally efficient and robust visual odometry algorithm developed for the Mars Science Laboratory mission. This algorithm is a significant improvement over the algorithm developed for the Mars Exploration Rover Mission because it is at least four time more computationally efficient and it tracks significantly more features. The core of the algorithm is an integrated motion estimation and stereo feature tracking loop that allows for feature recovery while guiding feature correlation search to minimize computation. Results on thousands of terrestrial and Martian stereo pairs show that the algorithm can operate with no initial motion estimate while still obtaining subpixel attitude estimation performance.

Unmanned planetary landers to date have landed blind, without the benefit of onboard landing hazard detection and avoidance systems. This constrains landing sites to very benign terrain and limits the scientific goals of missions. We review sensor options for landing hazard detection, then identify an approach based on stereo vision and shadow analysis that appears to address the broadest set of missions with the lowest cost. We describe algorithms for slope estimation and rock detection with this approach, develop models of their performance, and validate those models experimentally. Instantiating our model of rock detection reliability for Mars predicts that this approach would reduce the probability of failed landing by at least a factor of 4 compared to blind landing. Conversely, for the safety level desired for the 2009 Mars lander, this approach would increase the fraction of the planet that is accessible for landing from about 1/3 to nearly 100%.

During the Mars Exploration Rover (MER) landings, the descent image motion estimation system (DIMES) was used for horizontal velocity estimation. The DIMES algorithm combined measurements from a descent camera, a radar altimeter, and an inertial measurement unit. To deal with large changes in scale and orientation between descent images, the algorithm used altitude and attitude measurements to rectify images to a level ground plane. Feature selection and tracking were employed in the rectified images to compute the horizontal motion between images. Differences of consecutive motion estimates were then compared to inertial measurements to verify correct feature tracking. DIMES combined sensor data from multiple sources in a novel way to create a low-cost, robust, and computationally efficient velocity estimation solution, and DIMES was the first use of computer vision to control a spacecraft during planetary landing. This paper presents the detailed implementation of the DIMES algorithm and the results from the two landings on Mars.

Future robotic space missions will employ a precision soft-landing capability that will enable exploration of previously inaccessible sites that have strong scientific significance. To enable this capability, a fully autonomous onboard system that identifies and avoids hazardous features such as steep slopes and large rocks is required. Such a system will also provide greater functionality in unstructured terrain to unmanned aerial vehicles. This paper describes an algorithm for landing hazard avoidance based on images from a single moving camera. The core of the algorithm is an efficient application of structure from motion to generate a dense elevation map of the landing area. Hazards are then detected in this map and a safe landing site is selected. The algorithm has been implemented on an autonomous helicopter testbed and demonstrated four times resulting in the first autonomous landing of an unmanned helicopter in unknown and hazardous terrain.

The Mars Exploration Rover (MER) Descent Image Motion Estimation System (DIMES) is the first autonomous machine vision system used to safely land a robotics payload on another planet. DIMES consists of a descent camera and an algorithm for estimating horizontal velocity using image, inertial and altitude measurements. Before DIMES was accepted by MER for inclusion in the mission, its performance was validated through field testing using a manned helicopter to image three Mars analog test sites. Statistical analysis of the resulting 1900+ test cases showed that DIMES met its velocity estimation requirement. This paper describes the DIMES field test approach and associated results.

Numerous upcoming NASA missions need to land safely and precisely on planetary bodies. Accurate and robust state estimation during the descent phase is necessary. Towards this end, we have developed an approach for improved state estimation by augmenting traditional inertial navigation techniques with image-based motion estimation (IBME). A Kalman filter that processes rotational velocity and linear acceleration measurements provided from an inertial measurement unit has been enhanced to accommodate relative pose measurements from the IBME. In addition to increased state estimation accuracy, IBME convergence time is reduced while robustness of the overall approach is improved. The methodology is described in detail and experimental results with a 5 DOF gantry testbed are presented.

In this paper we present an algorithm for robust absolute position estimation in natural terrain based on landmarks extracted from dense 3-D surfaces. Our landmarks are constructed by concatenating pose dependent oriented surface points with pose invariant surface signatures into a single feature vector; this definition of landmarks allows a priori pose information to be used to constrain the search for landmark matches. The first step in our algorithm is to extract landmarks from stable and salient surface patches. These landmarks are then stored in a closest point search structure with which landmarks are matched efficiently using available pose constraints and invariant values. Finally, an iterative pose estimation algorithm, based on least median squares, is wrapped around landmark matching to eliminate outliers and estimate absolute position. To validate our algorithm, we show hundreds of absolute position estimation results from three different natural scenes. These results show that our algorithm can incorporate constraints on position and attitude for efficient landmark matching and match small and dense scene surface patches to large and coarse model surfaces.

This paper describes an algorithm for translational motion estimation near a comet a surface using scanning laser rangefinder data. Our technique is based on terrain map generation from rangefinder data followed by terrain map alignment. The output of our algorithm is estimates of rigid translational motion and motion covariance between scans. Our algorithms have been tested using data acquired with a scanning laser rangefinder designed for comet landing, and results indicated motion estimation accuracies of 0.5 m over 70 m of motion with a processing rate of 4.4 Hz on 10000 sample range scans.

We present a 3D shape-based object recognition system for simultaneous recognition of multiple objects in scenes containing clutter and occlusion. Recognition is based on matching surfaces by matching points using the spin image representation. The spin image is a data level shape descriptor that is used to match surfaces represented as surface meshes. We present a compression scheme for spin images that results in efficient multiple object recognition which we verify with results showing the simultaneous recognition of multiple objects from a library of 20 models. Furthermore, we demonstrate the robust performance of recognition in the presence of clutter and occlusion through analysis of recognition trials on 100 scenes.

We present a 3-D shape-based object recognition system for simultaneous recognition of multiple objects in scenes containing clutter and occlusion. Recognition is based on matching surfaces by matching points using the spin-image representation. The spin-image is a data level shape descriptor that is used to match surfaces represented as surface meshes. We present a compression scheme for spin-images that results in efficient multiple object recognition which we verify with results showing the simultaneous recognition of multiple objects from a library of 20 models. Furthermore, we demonstrate the robust performance of recognition in the presence of clutter and occlusion through analysis of recognition trials on 100 scenes.

We present an approach to recognition of complex objects in cluttered 3-D scenes that does not require feature extraction or segmentation. Our object representation comprises descriptive images associated with each oriented point on the surface of an object. Using a single point basis constructed from an oriented point, the position of other points on the surface of the object can be described by two parameters. The accumulation of these parameters for many points on the surface of the object results in an image at each oriented point. These images, localized descriptions of the global shape of the object, are invariant to rigid transformations. Through correlation of images, point correspondences between a model and scene data are established and then grouped using geometric consistency. The effectiveness of our algorithm is demonstrated with results showing recognition of complex objects in cluttered scenes with occlusion.

This paper describes a system that semi-automatically builds a virtual world for remote operations by constructing 3-D models of a robot's work environment. With a minimum of human interaction, planar and quadric surface representations of objects typically found in man-made facilities are generated from laser rangefinder data. The surface representations are used to recognize complex models of objects in the scene. These object models are incorporated into a larger world model that can be viewed and analyzed by the operator, accessed by motion planning and robot safeguarding algorithms, and ultimately used by the operator to command the robot through graphical programming and other high level constructs. Limited operator interaction, combined with assumptions about the robots task environment, make the problem of modeling and recognizing objects tractable and yields a solution that can be readily incorporated into many telerobotic control schemes.