The XWand is a novel wireless sensor package that enables styles of natural interaction with intelligent environments. For example, a user may point the wand at a device and control it using simple gestures. The XWand system leverages the intelligence of the environment to best determine the user's intention. We detail the hardware device, signal processing algorithms to recover position and orientation, gesture recognition techniques, a multimodal (wand and speech) computational architecture and a preliminary user study examining pointing performance under conditions of tracking availability and audio feedback.

Context-aware intelligent environments are computing systems embedded within physical spaces. They are equipped with input and output computing devices for users and sensors to provide contextual information to the system. These environments provide new challenges to interface designers due to a number of differences from typical desktop computing environments, including the lack of a single focal point for the user, a dynamic set of interaction devices, the sensor-rich nature of the environment, the potential of multiple simultaneous users, and the opportunity for diverse interaction modalities. This essay describes these challenges and focuses on issues involving multiple interaction modalities and automatic system behaviors.

We previously presented a framework for segmentation of complex scenes using multiple physical hypotheses for simple image regions. A consequence of that framework was a proposal for a new approach to the segmentation of complex scenes into regions corresponding to coherent surfaces rather than merely regions of similar color. Herein we present an implementation of this new approach and show example segmentations for scenes containing multi-colored piece-wise uniform objects. Using our approach we are able to intelligently segment scenes with objects of greater complexity than previous physics-based segmentation algorithms. The results show that by using general physical models we obtain segmentations that correspond more closely to coherent surfaces in the scene than segmentations found using only color.

Uniformly textured surfaces in 3D scenes provide important cues for image understanding. Texture can be used for both segmentation and for 3D shape inference. Unfortunately, virtually all current algorithms are based on assumptions that make it impossible to do texture segmentation and shape-from-texture in the same image. Texture segmentation algorithms rely on an absence of 3D effects that tend to distort the texture. Shape-from-texture algorithms depend on these effects, relying instead on the texture being already segmented. To really understand texture in images, texture segmentation and shape-from-texture must be viewed as a combined problem to be solved simultaneously. We present a solution to this problem with a region-growing algorithm that explicitly accounts for perspective distortions of otherwise uniform texture. We use the image spectrogram to compute local surface normals, which are in turn used to "frontalize" the texture. These frontalized texture patches are then subjected to a region-growing algorithm based on similarity in the local frequency domain and a minimum description length criteria. We show results of our algorithm on real texture images taken in the lab and outdoors.>

This paper proposes new "hypergeometric" filters for the problem of image matching under the translational and affine model. This new set of filters has the following advantages: (1) High-precision registration of two images under the translational and affine model. Because the window effects are eliminated, we are able to achieve superb performance in both translational and affine matching. (2) Affine matching without exhaustive search or image warping. Due to the recursiveness of the filters in the spatial domain, we are able to analytically express the relation between filter outputs and the six affine parameters. This analytical relation enables us to directly compute these affine parameters. (3) Generality. The approach we demonstrate here can be applied to a broad class of matching problems, as long as the transformation between the two image patches can be mathematically represented in the frequency domain.>

Many aspects of the real world continue to plague stereo matching systems. One of these is perspective foreshortening, an effect that occurs when a surface is viewed at a sharp angle. Because each stereo camera has a slightly different view, the image of the surface is more compressed and occupies a smaller area in one view. These effects cause problems because most stereo methods compare similarly sized regions (using the same-sized windows in both images), tacitly assuming that objects occupy the same extents in both images. Clearly this condition is violated by perspective foreshortening. We show how to overcome this problem using a local spatial frequency representation. A simple geometric analysis leads to an elegant solution in the frequency domain which, when applied to a Gabor filter-based stereo system, increases the system's maximum matchable surface angle from 30 degrees to over 75 degrees.

Many low level visual computation problems such as focus, stereo, etc. can be formulated as problems of extracting one or more parameters of a non-stationary transformation between two images. Because of the non-stationary nature, finite-width windows are widely used in various algorithms to extract spatially local information from images. While the choice of window width has a very profound impact on the quality of results of algorithms, there has been no quantitative way to measure or eliminate the negative effects of finite width windows. To address this problem, we introduce a new set of filters, moment filters. Due to their recursiveness in the Fourier and spatial domains, these filters allow the effects of finite-width windows and fore-shortening to be explicitly analyzed and eliminated.

This paper presents a new approach to segmentation using explicit hypotheses about the physics that creates images. We propose an initial segmentation that identifies image regions exhibiting constant color; but possibly varying intensity. For each region, we propose a set of hypotheses, each of which specifically models the illumination, reflectance, and shape of the 3-D patch which caused that region. Each hypothesis represents a distinct, plausible explanation for the color and intensity variation of that patch. Hypotheses for adjacent patches can be compared for similarity and merged when appropriate, resulting in more global hypotheses which group elementary regions.>

There are two basic problems concerned with Gabor filterings that we will address in this paper. One is the window size problem, in which we will adopt a set of 2D variable window Gabor filters, and compare its performance with those of fixed window filters. We will show that the variable window scheme is more adaptive to image contents. The other problem we will address is the stability of amplitude and phase information resulting from convolving the filters with images. We will extend Fleet's 1D phase stability analysis to 2D phase and amplitude stability analysis based upon the assumption of local resemblance of filter outputs to a single sinusoid. Applications on focus quality measurement and 2D correspondence are described, and the results demonstrate improvements of performance by detecting unstable information using the criterion developed.>

A taxonomy that includes 15 different definitions of image center is presented. It is concluded that the accuracy of the image center can be an important factor in the accuracy of the overall camera calibration, and that the large variation in the position of the image center across different definitions and different lens settings makes the calibration problem complex. With proper modeling and by using appropriate definitions for all image centers in a system, the accuracy of the camera calibration can be improved.>

The problem of obtaining depth information from focusing and defocusing is studied. In depth from focusing, instead of the Fibonacci search, which is often trapped in local maxima, the combination of Fibonacci search and curve fitting is proposed. This combination leads to an unprecedentedly accurate result. A model of the blurring effect that takes geometric blurring as well as imaging blurring into consideration in the calibration of the blurring model is proposed. In spectrogram-based depth from defocusing, a maximal resemblance estimation method is proposed to decrease or eliminate the window effect.>

It is shown how local spatial image frequency is related to the surface normal of a textured surface. It is found that the Fourier power spectra of any two similarly textured patches on a plane are approximately related to each other by an affine transformation. The transformation parameters are a function of the plane's surface normal. This relationship is used as the basis of an algorithm for finding surface normals of textured shapes using the spectrogram, which is one type of local spatial frequency representation. The relationship is validated by testing the algorithm on real textures. By analyzing shape and texture in terms of the local spatial frequency representation, the advantages of the representation for the shape-from-texture problem can be exploited. Specifically, the algorithm requires no feature detection and can give correct results even when the texture is aliased.>

It is shown that the color histogram has an even closer relationship to scene properties than has been previously described. Color histograms have identifiable features that relate in a precise mathematical way to scene properties. Object color and illumination color are the most obvious properties that are related to color distribution, and their extraction has already been described. It is shown here that the histogram of color variation may be further exploited to relate its shape to surface roughness and imaging geometry. An understanding of these features allows an improved estimate of illumination color and object color to be made.>

Lenses used in computer vision systems have many nonideal behaviors that can have a significant effect on vision tasks if not taken into account or compensated for. The authors show how two nonideal lens behaviors, chromatic aberration and focus magnification, can cause significant problems in color image analysis and focus ranging respectively. They show how precise active lens control can be used to reduce these effects significantly.>

Real-world scenes contain many interacting phenomena that lead to complex images which are difficult to interpret automatically. Some effects are best described in the spatial domain, while others are more naturally expressed in frequency. In order to resolve this dichotomy, the authors present the combined space/frequency representation which, for each point in an image, shows the spatial frequencies at that point. This representation is useful for developing theories about many important vision phenomena, leading to deeper understanding and better algorithms. The authors show how the representation can be used for the shape from texture problem and to analyze aliasing simply and naturally. The space/frequency representation should be a key aid in untangling the complex interaction of phenomena in images, allowing automatic understanding of real-world scenes.>

In this paper, we present an approach to color image understanding that can be used to segment and analyze surfaces with color variations due to highlights and shading. We begin with a theory - the Dichromatic Reflection Model - that relates the reflected light from dielectric materials, such as plastic, to fundamental physical reflection processes, and describes the color of the reflected light as a linear combination of the color of the light due to surface reflection (highlights) and body reflection (object color). This dichromatic theory is used in an algorithm that separates a color image into two parts: an image of just the highlights, and the original image with the highlights removed. In the past, we have applied this method to handsegmented images. This paper shows how to perform automatic segmentation by applying this theory in stages to identify the object and highlight colors. The result is a combination of segmentation and reflection analysis that is better than traditional heuristic segmentation methods and provides important physical information about the surface geometry and material properties at the same time. This line of research can lead to physics-based image understanding methods that are both more reliable and more useful than traditional methods.

In stereo navigation, a mobile robot estimates its position by tracking landmarks with on-board cameras. Previous systems for stereo navigation have suffered from poor accuracy, in part because they relied on scalar models of measurement error in triangulation. Using three-dimensional (3D) Gaussian distributions to model triangulation error is shown to lead to much better performance. How to compute the error model from image correspondences, estimate robot motion between frames, and update the global positions of the robot and the landmarks over time are discussed. Simulations show that, compared to scalar error models, the 3D Gaussian reduces the variance in robot position estimates and better distinguishes rotational from translational motion. A short indoor run with real images supported these conclusions and computed the final robot position to within two percent of distance and one degree of orientation. These results illustrate the importance of error modeling in stereo vision for this and other applications.

This paper describes sensor fusion in the context of an autonomous mobile robot. The requirements of a complex mission, real-world operation, and real-time control dictate many facets of the system architecture. The hardware architecture must include both general-purpose and special-purpose computers, and multiple sensors of various modalities (vision, range, etc.). The software architecture must allow modular development of a parallel system that supports many perceptual modalities and navigation planning tasks, but at the same time enforces global consistency regarding position and orientation of the vehicle and sensors. We are building such a system at CMU, called the NAVLAB system, based on a commercial truck with computer controls and studded with cameras and other sensors. This paper describes the software architecture of the NAVLAB, consisting of two parts: a "whiteboard" system called CODGER that is similar to a blackboard but supports parallelism in the knowledge source modules, and an organized collection of perceptual and navigational modules tied together by the CODGER system. In general, the system philosophy is to provide as much top-down guidance as possible, and to exploit sensor modality differences to produce complementary rather than competing perceptual processes in the system. In this way, the limitations of each sensor modality are compensated for as much as possible by other sensors or by higher level knowledge. The NAVLAB is being produced as part of the DARPA Strategic Computing Initiative, in conjunction with the Autonomous Land Vehicle project.

Recursive segmentation of an image into regions using histograms is one of the most widely used techniques for image segmentation. At CMU, several versions of a region segmentation program have been developed based on this technique (Ohlander, Price, Shafer and Kanade). Based on these experiences, this paper discusses issues of recursive region segmentation in the context of PHOENIX, the newest version of region segmentation program, running on a VAX 11/780 under UNIX. The issues discussed in this paper include: Image features to be used in histogramming; comparison of the algorithm with other techniques; important improvements made in PHOENIX over its predecessor (Ohlander and Price); and some inherent problems in histogram-based segmentation together with suggestions for minimizing them. PHOENIX is being incorporated into the ARPA Image Understanding Testbed, under construction at SRI International.