Ultimately, a display device should be capable of reproducing the visual effects observed in reality. In this paper we introduce an autostereoscopic display that uses a scalable array of digital light projectors and a projection screen augmented with microlenses to simulate a light field for a given three-dimensional scene. Physical objects emit or reflect light in all directions to create a light field that can be approximated by the light field display. The display can simultaneously provide many viewers from different viewpoints a stereoscopic effect without head tracking or special viewing glasses. This work focuses on two important technical problems related to the light field display; calibration and rendering. We present a solution to automatically calibrate the light field display using a camera and introduce two efficient algorithms to render the special multi-view images by exploiting their spatial coherence. The effectiveness of our approach is demonstrated with a four-projector prototype that can display dynamic imagery with full parallax.

In recent work, the authors introduced a multiplicative, low dimensional model of illumination that is computed as a linear combination of a set of simple-to-compute Legendre basis functions. The basis coefficients describing illumination change, are can be combined with the "shape" vector to define a joint "shape"-illumination space for tracking. The increased dimensionality of the state vector necessitates an increase in the number of particles required to maintain tracking accuracy. In this paper, we utilize the recently proposed PF-MT algorithm to estimate the illumination vector. This is motivated by the fact that, except in case of occlusions, multimodality of the state posterior is usually due to multimodality in the "shape" vector (e.g. there may be multiple objects in the scene that roughly match the template). In other words, given the "shape" vector at time t, the posterior of the illumination (probability distribution of illumination conditioned on the "shape" and illumination at previous time) is unimodal. In addition, it is also true that this posterior is usually quite narrow since illumination changes over time are slow. The choice of the illumination model permits the illumination coefficients to be solved in closed form as a solution of a regularized least squares problem. We demonstrate the use of our method for the problem of face tracking under variable lighting conditions existing in the scene.

Visual tracking involves generating an inference about the motion of an object from measured image locations in a video sequence. In this paper we present a unified framework that incorporates shape and illumination in the context of visual tracking. The contribution of the work is twofold. First, we introduce a a multiplicative, low dimensional model of illumination that is defined by a linear combination of a set of smoothly changing basis functions. Secondly, we show that a small number of centroids in this new space can be used to represent the illumination conditions existing in the scene. These centroids can be learned from ground truth and are shown to generalize well to other objects of the same class for the scene. Finally we show how this illumination model can be combined with shape in a probabilistic sampling framework. Results of the joint shape-illumination model are demonstrated in the context of vehicle and face tracking in challenging conditions.

In this paper we introduce a multiplicative, low dimensional model for appearance change in the context of visual tracking. The model is defined by a linear combination of smoothly changing basis functions. Secondly, we show that a small number of cluster centers in this new space can be used to represent the illumination conditions existing in the scene. These cluster centers can be learned from ground truth and are shown to generalize well to other objects of the same class for the scene. Finally we show how this illumination model can be combined with shape in a probabilistic sampling framework to improve tracking robustness under challenging conditions.

We explore the integration of projected imagery with a physical book that acts as a tangible interface to multimedia data. Using a camera and projector pair, a tracking framework is presented wherein the 3D position of planar pages are monitored as they are turned back and forth by a user, and data is correctly warped and projected onto each page at interactive rates. The book pages are blank, so traditional approaches to tracking physical features on the display surface do not apply. Instead, in each frame, feature points are independently extracted from the camera and projector images, and matched in order to recover the geometry of the pages in motion. The book can be loaded with multimedia content, including images, videos, and volumetric datasets (in which case a page can be removed from the book and used to navigate through a virtual 3D volume).

Ground truth is important not only for performance evaluation but also for a principled development of computer vision algorithms. Unfortunately obtaining ground truth data is difficult and often very labor intensive. This is particularly true of video analysis due to the immense cost of producing pixel-wise ground truth in potentially thousands of frames. In this paper, we propose a method to produce foreground/background segmentation for video sequences captured by a stationary camera, that requires very little human labor as compared to complete manual segmentation, while still producing high quality results. Given a sequence, we use a few hand labeled images and Adaboost to train a classifier that segments the rest of the sequence. We demonstrate the effectiveness of our approach on two sequences and discuss the new horizons opened by these encouraging results. 1

Uncertainty estimates related to the position of image features are seeing increasing use in several computer vision problems. Many of these have been recast from standard least squares model fitting to techniques that minimize the Mahalanobis distance, which weighs each error vector by covariance of the observations. These include structure from motion and traditional geometric camera calibration. Uncertainty estimates previously derived for the case of corner localization are based on implicit assumptions that preclude sophisticated image noise models. Uncertainties associated with these features tend to be over estimated. In this work, we introduce a new formulation for feature location uncertainty that supports arbitrary pixel covariance to derive a more accurate positional uncertainty estimate. The method is developed and evaluated in the case of a traditional interest operator that is in widespread use. Results show that uncertainty estimates based on this new formulation better reflect the error distribution in feature location.

We present a quality assessment procedure for correspondence estimation based on geometric coherence rather than ground truth. The procedure can be used for performance evaluation of correspondence extraction schemes developed by researchers, as well as for online learning and adaptation aimed at better system performance. A very important aspect of the proposed procedure is that it considers uncertainty in the correspondence extraction, and encourages the evaluated methods to deal correctly with uncertainty. Other important strengths of the procedure are that it does not use any manual work, and that it does not put any strong constraints on the scene, but rather relies on geometric coherence in the motion. Thanks to these strengths, it can therefore be used with large amounts of real, potentially application specific data, or even data acquired during system operation. In the evaluation the correspondence extractor is handled as a black box producing a probability distribution for the local motion vector between a pair of image patches. The procedure is therefore quite general. We are making the evaluation procedure available for public use.

Object motions can be represented as a sequence of shape deformations and translations which can be interpretated as a sequence of points in N-dimensional shape space. These spaces range from simple 2D translations to more inclusive spaces such as the affine. In this case, tracking is the problem of inferring the most likely point in the space for the next frame given a current set of hypotheses. A robust method for achieving this is the particle filter. In this case, likely points within shape space are selected in a two step process. First, image measurements assign likelihoods to proposed points. Likely points are then propagated forward using a dynamical model to derive a set of new points that are perturbed according to some sampling distribution. These distributions play an important role in tracking performance because dynamical models are seldom known and a Gauss Markov model is often assumed for the model dynamics. This paper address the problems inherent in utilizing uninformed sampling distributions for visual tracking. We introduce a principled adaptive sampling approach that takes into account constraints on each component of the shape vector. Further a more appropriate sampling distribution that takes place in a linear subspace representing the predominant motion in the shape space. Results demonstrate improved tracking performance in challenging conditions where targets exhibit changing motion models.

Front-projection displays are a cost-effective and increasingly popular method for large format visualization and immersive rendering of virtual models. New approaches to projector tiling, automatic calibration, and color balancing have made multiprojector display systems feasible without undue infrastructure changes and maintenance. As a result, front-projection displays are being used to generate seamless, visually immersive worlds for virtual reality and visualization applications with reasonable cost and maintenance overhead. However, these systems suffer from a fundamental problem: Users and other objects in the environment can easily and inadvertently block projectors, creating shadows on the displayed image. Shadows occlude potentially important information and detract from the sense of presence an immersive display may have conveyed. We introduce a technique that detects and corrects shadows in a multiprojector display while it is in use. Cameras observe the display and compare observations with an expected image to detect shadowed regions. These regions are transformed to the appropriate projector frames, where corresponding pixel values are increased and/or attenuated. In display regions where more than one projector contributes to the image, shadow regions are eliminated.

Face recognition has primarily focused on recognizing and matching face images against large, controlled databases of frontal views. Many of these techniques perform well against databases that have been collected from a reduced set of viewpoints, under controlled lighting, and are normalized for scale. Acquisition of these databases, however, particularly in unconstrained environments, remains a challenge. We present a real-time technique to acquire a mugshot database automatically from a video surveillance network. Mugshot extraction is a twofold problem. First, faces are detected and tracked in all cameras of the network. Face targets are analyzed to determine which frames represent actual mugshots capable of supporting subsequent matching and recognition. Next, mugshot candidates are evaluated based on their ability to improve the quality of the incrementally constructed database. We introduce a database quality measure, which assigns high value to mugshots of previously unseen subjects or mugshots that do not decrease separability of existing clusters. The quality measure is discounted for mugshots that are redundant or increase the intra-cluster spread. Results demonstrate that automatic acquisition of a high-quality database from a twelve-camera network is feasible. The quality of these databases is demonstrated using traditional methods to match faces accurately against the acquired database.

We are developing a novel display system that physically realizes a sampling of the light field emitted by a three-dimensional scene. An array of projectors, each with a two-dimensional framebuffer, populates the 4D space of the light field. The view of the scene is simultaneously correct for all head positions within a volume. This eliminates the need for head tracking, produces binocular disparities without the need for glasses, and supports any number of viewers. Input to our display could be streamed from a light field sensor [1], or can be efficiently rendered in parallel using a cluster of standard computer graphics pipelines. For static scenes, no run-time rendering is necessary. We demonstrate the feasibility of the approach using a prototype cluster of projectors.

The Rich Photo-Realistic Content (RPC) image-based model defined and used by ArchVision represents a photo-realistic scene as an ordered set of images. A desired view from the represented viewing directions can quickly be extracted for rendering and display. The benefit of the RPC is its photo-realistic quality. One weakness, however, is that there is no support for view interpolation between closely related basis views. This paper presents results from a tool for creating a new RPC-like image-based model, which detects and discards captured views that are determined to be faithfully and automatically interpolated from the remaining images in the RPC. We exploit the geometric constraints of the acquisition system and provide an automated matching algorithm, and interpolation scheme, and an error metric displayed in a graphical tool for monitoring the quality impact of discarding particular frames.

Front-projection display environments suffer from a fundamental problem: users and other objects in the environment can easily and inadvertently block projectors, creating shadows on the displayed image. We introduce a technique that detects and corrects transient shadows in a multi-projector display. Our approach is to minimize the difference between predicted (generated) and observed (camera) images by continuous modification of the projected image values for each display device. We speculate that the general predictive monitoring framework introduced here is capable of addressing more general radiometric consistency problems. Using an automatically-derived relative position of cameras and projectors in the display environment and a straightforward color correction scheme, the system renders an expected image for each camera location. Cameras observe the displayed image, which is compared with the expected image to detect shadowed regions. These regions are transformed to the appropriate projector frames, where corresponding pixel values are increased. In display regions where more than one projector contributes to the image, shadow regions are eliminated. We demonstrate an implementation of the technique in a multiprojector system.

A technique is introduced for extracting and reconstructing a wide class of building types from a registered range image and optical image. An attentional focus stage, followed by model indexing, allows top-down robust surface fitting to reconstruct the 3D nature of the buildings in the data. Because of the effectiveness of model selection, top-down processing of noisy range data still succeeds and the algorithm is capable of detecting and reconstructing several different building roof classes, including flat single level, flat multi-leveled, peaked, and curved rooftops. The algorithm is applicable to range data that may have been collected from several different range sensor types. We demonstrate reconstructions of different buildings classes in the presence of large amounts of noise. Our results underline the usefulness of range data when processed in the context of a focus-of-attention area derived from the monocular optical image.

A system has been developed to acquire, extend and refine 3D geometric site models from aerial imagery. This system hypothesizes potential building roofs in an image, automatically locates supporting geometric evidence in other images, and determines the precise shape and position of the new buildings via multiimage triangulation. Model-to-image registration techniques are applied to align new, incoming images against the site model. Model extension and refinement procedures are then performed to add previously unseen buildings and to improve the geometric accuracy of the existing 3D building models.>

A new method for extracting planar polygonal rooftops in monocular aerial imagery is proposed. Structural features are extracted and hierarchically related using perceptual grouping techniques. Top-down feature verification is used so that features, and links between the features, are verified with local information in the image and weighed in a graph. Cycles in the graph correspond to possible building rooftop hypotheses. Virtual features are hypothesized for the perceptual completion of partially occluded rooftops. Extraction of the "best" grouping features into a building rooftop hypothesis is posed as a graph search problem. The maximally weighted, independent set of cycles in the graph is extracted as the final set of roof boundaries.>

A perception framework for intelligent inspection and reverse engineering is addressed. The use of discrete event dynamic systems (DEDS) to guide the sensing of mechanical parts is investigated. Dynamic recursive finite state machines (DRFSMs) are introduced as a new DEDS tool for utilizing the recursive nature of the mechanical parts under consideration. The proposed framework uses DRFSM DEDS for constructing an observer for exploration and inspection purposes. A sensing /spl rarr/ CAD interface is constructed for the automatic reconstruction of parts from visual data. A graphical interface for designing DRFSM DEDS controllers is implemented.>