This paper describes the hardware and software components of a general-purpose humanoid robot system for autonomously driving several different types of utility vehicles. The robot recognizes which vehicle it is in, localizes itself with respect to the dashboard, and self-aligns in order to interface with the steering wheel and accelerator pedal. Low- and higher-level methods are presented for speed control, environment perception, and trajectory planning and following suitable for operation in planar areas with discrete obstacles as well as along road-like paths.

In this paper, we focus on 3D point cloud classification by assigning semantic labels to each point in the scene. We propose to use simplified Markov networks to model the contextual relations between points, where the node potentials are calculated from point-wise classification results using off-the-shelf classifiers, such as Random Forest and Support Vector Machines, and the edge potentials are set by physical distance between points. Our experimental results show that this approach yields comparable if not better results with improved speed compared with state-of-the-art methods. We also propose a novel robust neighborhood filtering method to exclude outliers in the neighborhood of points, in order to reduce noise in local geometric statistics when extracting features and also to reduce number of false edges when constructing Markov networks. We show that applying robust neighborhood filtering improves the results when classifying point clouds with more object categories.

We propose a simple contrast-based method for tree detection and shape estimation from ground-plane perspective images for purposes of counting, classification, modeling, or robotic obstacle avoidance. Under the assumption that tree trunks are relatively narrow and vertical shapes which strongly differ in appearance from the scene background and have boundaries of opposite contrast, we apply a bank of bar filters parametrized from camera intrinsics combined with trunk location and diameter limits, and integrate results vertically. Non-maximum suppression is applied to candidates in the resulting trunk likelihood image. We present results demonstrating the effectiveness of our tree detection algorithm on a variety of forested images obtained directly from our robot as well as sampled from the web, and quantify performance using ground truth on a subset of those images. We also compare the results of our method with several related published approaches.

We describe a system which follows “trails” for autonomous outdoor robot navigation. Through a combination of appearance and structural cues derived from stereo omnidirectional color cameras, the algorithm is able to detect and track rough paths despite widely varying tread material, border vegetation, and illumination conditions. The approaching trail region is modeled as a circular arc segment of constant width. Using likelihood formulations which measure color, brightness, and/or height contrast between a hypothetical region and flanking areas, the tracker performs a robust randomized search for the most likely trail region and robot pose relative to it with no a priori appearance model. The addition of the structural information, which is derived from a semi-global dense stereo algorithm with ground-plane fitting, is shown to improve trail segmentation accuracy and provide an additional layer of safety beyond solely ladar-based obstacle avoidance. Our system's ability to follow a variety of trails is demonstrated through live runs as well as analysis of offline runs on several long sequences with diverse appearance and structural characteristics using ground-truth segmentations.

Continuous trails are extended regions along the ground such as roads, hiking paths, rivers, and pipelines which can be navigationally useful for ground-based or aerial robots. Finding trails in an image and determining possible obstacles on them are important tasks for robot navigation systems. Assuming that a rough initial segmentation or outline of the region of interest is available, our goal is to refine the initial guess to obtain a more accurate and detail representation of the true trail borders. In this paper, we compare the suitability of several previously published segmentation algorithms both in terms of agreement with ground truth and speed on a range of trail images with diverse appearance characteristics. These algorithms include generic graph cut, a shape-based version of graph cut which employs a distance penalty, Grab Cut, and an iterative superpixel grouping method.

We describe a system which follows “trails” for autonomous outdoor robot navigation. Through a combination of visual cues provided by stereo omnidirectional color cameras and ladar-based structural information, the algorithm is able to detect and track rough paths despite widely varying tread material, border vegetation, and illumination conditions. The approaching trail region is simply modeled as a circular arc of constant width. Using an adaptive measure of color and brightness contrast between a hypothetical region and flanking areas, the tracker performs a robust randomized search for the most likely trail region and robot pose relative to it with no a priori appearance model. Stereo visual odometry improves tracker dynamics on uneven terrain and permits local obstacle map maintenance. A motion planner is also described which takes the trail shape estimate and local map to plan smooth trajectories around in-trail and near-trail hazards. Our system's performance is analyzed on several long sequences with diverse appearance and structural characteristics using ground-truth segmentations.

We describe a framework for finding and tracking ¿trails¿ for autonomous outdoor robot navigation. Through a combination of visual cues and ladar-derived structural information, the algorithm is able to follow paths which pass through multiple zones of terrain smoothness, border vegetation, tread material, and illumination conditions. Our shape-based visual trail tracker assumes that the approaching trail region is approximately triangular under perspective. It generates region hypotheses from a learned distribution of expected trail width and curvature variation, and scores them using a robust measure of color and brightness contrast with flanking regions. The structural component analogously rewards hypotheses which correspond to empty or low-density regions in a groundstrike-filtered ladar obstacle map. Our system's performance is analyzed on several long sequences with diverse appearance and structural characteristics. Ground-truth segmentations are used to quantify performance where available, and several alternative algorithms are compared on the same data.

We describe a framework for detecting and tracking continuous ldquotrailsrdquo in images and image sequences for autonomous robot navigation. Continuous trails are extended regions along the ground such as roads, hiking paths, rivers, and pipelines which can be navigationally useful for ground-based or aerial robots. Our approach to single-image trail segmentation incorporates both bottom-up and top-down processes. First, good grouping hypotheses are efficiently generated by probabilistic clustering of superpixels based on color similarity. Second, hypotheses are robustly ranked with an objective function comprising shape, appearance, and deformation terms. The shape term measures how well a triangle, the approximate template for a trail viewed under perspective, can be fit to the groupingpsilas boundary. The appearance term reflects the visual contrast between the grouping and its surroundings using a between-class/within-class scatter measure. Finally, the deformation term measures the closeness of the fitted triangle to a learned distribution which captures expected size, location, and other degrees of shape variation. Although trail detection is accurate and reasonably fast on a variety of isolated images, we describe how introducing temporal filtering to both the bottom-up and top-down stages increases segmentation accuracy and per-frame speed over image sequences. Results are shown on varied sequences collected from flying and driving platforms, as well as images sampled from the Web.

In this paper we investigate the problem of automatically detecting 2D grid structures such as windows on building facades from images taken in urban settings. The key assumption that the background is strongly structured allows searching for near-regular textures in the image. We describe a probabilistic framework using Markov Random Field modeling and Markov Chain Monte Carlo (MCMC) optimization to explicitly recognize and group rectangular structures that appear in a grid-like pattern. Results on a variety of images of building facades are shown.

We present a technique for constructing a "clean" texture map of a partially occluded building facade from a series of images taken from a moving camera. Building regions blocked by trees, signs, people, and other foreground objects in a minority of views can be recovered via temporal median filtering on a registered image mosaic of the planar facade. However, when such areas are occluded in the majority of camera views, appearance information from other visible portions of the facade provides a critical cue to correctly complete the mosaic. In this paper, we apply a robust measure of spread to infer whether a particular mosaic pixel is occluded in a majority of views, and introduce a novel spatiotemporal timeline-based inpainting algorithm that uses appearance and motion cues in order to fill the texture map in majority-occluded regions. We describe methods for automatically training appearance-based classifiers from a coarse motion-based segmentation to efficiently recognize foreground and background patches in static imagery. Results of recovered building facades are shown for various sequences.

We present a vision- and ladar-based approach to autonomous driving on rural and desert roads that has been tested extensively in a closed-loop system. The vision component uses Gabor wavelet filters for texture analysis to find ruts and tracks from which the road vanishing point can be inferred via Hough-style voting, yielding a direction estimate for steering control. The ladar component projects detected obstacles along the road direction onto the plane of the front of the vehicle and tracks the 1-D obstacle "gap" due to the road to yield a lateral offset estimate. Several image- and state-based tests to detect failure conditions such as off-road poses (i.e., there is no road to follow) and poor lighting due to sun glare or distracting shadows are also explained. The system's efficacy is demonstrated with full control of a vehicle over 10+ miles of difficult roads at up to 25 mph, as well as analysis of logged data in diverse situations

This paper studies the problem of aligning images from multiple cameras with minimally- or non-overlapping fields of view using frame-to-frame transformations calculated for sequences from each camera. We examine implementation issues for the algorithm of Caspi and Irani that performs the alignment for two cameras which are fixed relative to each other and have approximately the same center of projection. Furthermore, we extend it to compute the camera-to-camera homographies for an N/spl ges/2 multi-camera network by simultaneously solving for all parameters of the unknown transformations. This enforces tighter constraints on the solution than performing the alignment for each pair independently. We show the efficacy of the approach on both synthetic as well as real sequences captured using a polycamera built in our lab. The aligned images can be mosaiced together to obtain a wider field of view virtual camera for subsequent processing.

We present a technique for constructing a "clean" texture map of a partially occluded building facade from a series of images taken from a moving camera. Building regions blocked by trees, signs, people, and other foreground objects in a minority of views can be recovered via temporal median filtering on a registered image mosaic of the planar facade. However, when such areas are occluded in the majority of camera views, appearance information from other visible portions of the facade provides a critical cue to correctly complete the mosaic. In this paper we apply a robust measure of spread to infer whether a particular mosaic pixel is occluded in a majority of views, and introduce a novel spatiotemporal timeline-based inpainting algorithm that uses an additional motion cue in order to fill the texture map in majority-occluded regions.

We present two key parts of an ongoing robotic, vision-based architectural modeling project. The first component is a randomized approach to view planning for a single ground robot scanning a building perimeter to recover a series of texture map mosaics. This algorithm generates paths that simultaneously address coverage and quality (i.e., real-valued distance and foreshortening factors). The second part is a technique for "cleaning" the captured texture maps in the presence of occlusions caused by trees, signs, people, and other foreground objects. When such occlusions comprise a minority of views a background feature can be recovered via temporal median filtering, but when they are in the majority, appearance information from other visible portions of the facade provides a critical cue to correctly complete the mosaic. We describe a novel spatiotemporal timeline-based inpainting algorithm that identifies and corrects such areas.

Many rural roads lack sharp, smoothly curving edges and a homogeneous surface appearance, hampering traditional vision-based road-following methods. However, they often have strong texture cues parallel to the road direction in the form of ruts and tracks left by other vehicles. This paper describes an unsupervised algorithm for following ill-structured roads in which dominant texture orientations computed with Gabor wavelet filters vote for a consensus road vanishing point location. The technique is first described for estimating the direction of straight-road segments, then extended to curved and undulating roads by tracking the vanishing point indicated by a differential “strip ” of voters moving up toward the nominal vanishing line. Finally, the vanishing point is used to constrain a search for the road boundaries by maximizing texture- and color-based region discriminant functions. Results are shown for a variety of road scenes including gravel roads, dirt trails, and highways. 1.

In an outdoor, off-road mobile robotics environment, it is important to identify objects that can affect the vehicle's ability to traverse its planned path, and to determine their three-dimensional characteristics. In the paper, a combination of three elements is used to accomplish this task. An imaging ladar collects range images of the scene. A color camera, whose position relative to the ladar is known, is used to gather color images. Information extracted from these sensors is used to build a world model, a representation of the current state of the world. The world model is used actively in the sensing to predict what should be visible in each of the sensors during the next imaging cycle. The paper explains how the combined use of these three types of information leads to a robust understanding of the local environment surrounding the robotic vehicle for two important tasks: puddle/pond avoidance and road sign detection.

We describe results on combining depth information from a laser range-finder and color and texture image cues to segment ill-structured dirt, gravel, and asphalt roads as input to an autonomous road following system. A large number of registered laser and camera images were captured at frame-rate on a variety,of rural roads, allowing laser features such as 3-D height and smoothness to be correlated with image features such as color histograms and Gabor filter responses. A small set of road models was generated by training separate neural networks on labeled feature vectors clustered by road "type." By first classifying the type of a novel road image, an appropriate second-stage classifier was selected to segment individual pixels, achieving a high degree of accuracy on arbitrary images from the dataset. Segmented images combined with laser range information and the vehicle's inertial navigation data were used to construct 3-D maps suitable for path planning.

We describe a framework that explicitly reasons about data association to improve tracking performance in many difficult visual environments. A hierarchy of tracking strategies results from ascribing ambiguous or missing data to: 1) noise-like visual occurrences, 2) persistent, known scene elements (i.e., other tracked objects), or 3) persistent, unknown scene elements. First, we introduce a randomized tracking algorithm adapted from an existing probabilistic data association filter (PDAF) that is resistant to clutter and follows agile motion. The algorithm is applied to three different tracking modalities-homogeneous regions, textured regions, and snakes-and extensibly defined for straightforward inclusion of other methods. Second, we add the capacity to track multiple objects by adapting to vision a joint PDAF which oversees correspondence choices between same-modality trackers and image features. We then derive a related technique that allows mixed tracker modalities and handles object overlaps robustly. Finally, we represent complex objects as conjunctions of cues that are diverse both geometrically (e.g., parts) and qualitatively (e.g., attributes). Rigid and hinge constraints between part trackers and multiple descriptive attributes for individual parts render the whole object more distinctive, reducing susceptibility to mistracking. Results are given for diverse objects such as people, microscopic cells, and chess pieces.

Common objects such as people and cars comprise many visual parts and attributes, yet image-based tracking algorithms are often keyed to only one of a target's identifying characteristics. In this paper, we present a framework for combining and sharing information among several state estimation processes operating on the same underlying visual object. Well-known techniques for joint probabilistic data association are adapted to yield increased robustness when multiple trackers attuned to disparate visual cues are deployed simultaneously. We also formulate a measure of tracker confidence, based on distinctiveness and occlusion probability, which permits the deactivation of trackers before erroneous state estimates adversely affect the ensemble. We discuss experiments focusing on color-region- and snake-based tracking that demonstrate the efficacy of this approach.

Robots relying on vision as a primary sensor frequently need to track common objects such as people, cars, and tools in order to successfully perform autonomous navigation or grasping tasks. These objects may comprise many visual parts and attributes, yet image-based tracking algorithms are often keyed to only one of a target's identifying characteristics. In this paper, we present a framework for sharing information among disparate state estimation processes operating on the same underlying visual object. Well-known techniques for joint probabilistic data association are adapted to yield increased robustness when multiple trackers attuned to different visual cues are deployed simultaneously. We also formulate a measure of tracker confidence, based on distinctiveness and occlusion probability, which permits the deactivation of trackers before erroneous state estimates adversely affect the ensemble. We will discuss experiments using color-region- and snake-based tracking in tandem that demonstrate the efficacy of this approach.

We have been developing an architecture for vision-based navigation which relies on continuous feedback from visual "landmarks" to control robot motion, In this approach, landmarks are consistently located and acquired as they come into view. To make this process efficient and robust, it is important that the image locations of these features can be predicted from available image information. In this article, we discuss methods for direct image-based prediction of point and line features for a mobile system operating on a planar surface. Preliminary experimental results suggest that image-based prediction con be performed efficiently and with sufficient accuracy to ensure robust acquisition of navigational landmarks.