Image recognition is one of the fundamental problems in multimedia analysis. Typically in the training database, there will be more than one image for each object, however most existing bag-of-features based approaches treat them independently and completely ignore the feature correspondence relationship among them. As a result, features corresponding to the same physical point may be clustered into different clusters, which finally leads to inaccurate image representations for recognition. To tackle the problem, we present a supervised codebook construction algorithm exploiting the feature correspondence constraints in feature clustering. Features in different images of the same object are first matched, then ho-mography between images are computed to remove outliers as well as recover the feature correspondences that are not correctly matched. Features belonging to the same physical point are enforced to be in the same cluster. We show via experiments that codebook constructed using this approach can improve the recognition performance.

We describe methods, accomplishments, and direction in a laboratory-corporate collaborative project over this past year and a half. With the goal of taking new immersive technologies to a growing customer base through in-situ demonstrations and experiments, we are combining novel capture and display capabilities in delivering life-sized immersive 3D entertainment experiences. We describe the adaptations and innovations needed in delivering immersive life sized 3D to a large audience.

The paper presents decoding positional and color information using visual coded patterns for efficient geometric calibration and color consistency across multiple cameras. The patterns are generated from an alphabet of basis colors placed in unique spatial configurations called ChromaCodes. The imaged patterns are automatically decoded to derive the information needed for both geometric and color calibration. Previous 2-D structured patterns are typically designed for obtaining only geometry whereas the proposed patterns capture color information as well. Moreover, the unique decodability of the codes allows them to overcome the problem of pattern visibility in all views and enables effective geometric calibration and color consistency with even partial and/or occluded view of the pattern. Experimental results demonstrate that a single shot of the pattern is sufficient to encapsulate enough information to compute both geometry and color across one or more cameras, especially important for real-time or interactive applications.

We describe a robust feature descriptor called soft ordinal spatial intensity distribution (soft OSID) that is invariant to any monotonically increasing brightness changes. In traditional histogram-based feature descriptors, each pixel is explicitly assigned to a single histogram bin, making them not robust to image deformations and appearance changes. In this paper, we present a feature descriptor that is obtained by assigning each pixel to more than one bin where the fraction is determined by a weight function to put more weight on close bins. This makes the descriptor more robust to image changes like viewpoint changes, image blur, and JPEG compression. Extensive experiments show that the proposed descriptor significantly outperforms many state-of-the-art descriptors such as OSID, SIFT, GLOH, and PCA-SIFT under complex brightness changes. The proposed descriptor has far reaching implications for many applications in computer vision.

We describe a novel and robust feature descriptor called ordinal spatial intensity distribution (OSID) which is invariant to any monotonically increasing brightness changes. Many traditional features are invariant to intensity shift or affine brightness changes but cannot handle more complex nonlinear brightness changes, which often occur due to the nonlinear camera response, variations in capture device parameters, temporal changes in the illumination, and viewpoint-dependent illumination and shadowing. A configuration of spatial patch sub-divisions is defined, and the descriptor is obtained by computing a 2-D histogram in the intensity ordering and spatial sub-division spaces. Extensive experiments show that the proposed descriptor significantly outperforms many state-of-the-art descriptors such as SIFT, GLOH, and PCA-SIFT under complex brightness changes. Moreover, the experiments demonstrate the proposed descriptor's superior performance even in the presence of image blur, viewpoint changes, and JPEG compression. The proposed descriptor has far reaching implications for many applications in computer vision including motion estimation, object tracking/recognition, image classification/retrieval, 3D reconstruction, and stereo.

Supersampling is widely used by graphics hardware to render anti-aliased images. In conventional supersampling, multiple scene samples are computationally combined to produce a single screen pixel. We consider a novel imaging paradigm that we calldisplay supersampling, where multiple display samples are physically combined via the superimposition of multiple image subframes. Conventional anti-aliasing and texture mapping techniques are shown inadequate for the task of rendering high-quality images on supersampled displays. Instead of requiring anti-aliasing filters, supersampled displays actually require alias generation filters to cancel the aliasing introduced by nonuniform sampling. We present fundamental theory and efficient algorithms for the real-time rendering of high-resolution anti-aliased images on supersampled displays. We show that significant image quality gains are achievable by taking advantage of display supersampling. We prove that alias-free resolution beyond the Nyquist limits of a single subframe may be achieved by designing a bank of alias-canceling rendering filters. In addition, we derive a practical noniterative filter bank approach to real-time rendering and discuss implementations on commodity graphics hardware.

We consider the problem of rendering high-resolution images on a display composed of multiple superimposed lower-resolution projectors. A theoretical analysis of this problem in the literature previously concluded that the multi-projector superimposition of low resolution projectors cannot produce high resolution images. In our recent work, we showed to the contrary that super-resolution via multiple superimposed projectors is indeed theoretically achievable. This paper derives practical algorithms for real multi-projector systems that account for the intra- and inter-projector variations and that render high-quality, high-resolution content at real-time interactive frame rates. A camera is used to estimate the geometric, photometric, and color properties of each component projector in a calibration step. Given this parameter information, we demonstrate novel methods for efficiently generating optimal sub-frames so that the resulting projected image is as close as possible to the given high resolution images.

Multi-projector super-resolution is the dual of multi-camera super-resolution. The goal of projector super-resolution is to produce a high resolution frame via superimposition of multiple low resolution subframes. Prior work claims that it is impossible to improve resolution via superimposed projection except in specialized circumstances. Rigorous analysis has been previously restricted to the special case of uniform display sampling, which reduces the problem to a simple shift-invariant deblurring. To understand the true behavior of superimposed projection as an inverse of classical camera super-resolution, one must consider the effects of non-uniform displacements between component subframes. In this paper, we resolve two fundamental theoretical questions concerning resolution enhancement via superimposed projection. First, we show that it is possible to reproduce frequencies that are well beyond the Nyquist limit of any of the component subframes. Second, we show that nonuniform sampling and pixel reconstruction functions impose fundamental limits on achievable resolution.

We present a general framework for the modeling and optimization of scalable multi-projector displays. Based on this framework, we derive algorithms that can robustly optimize the visual quality of an arbitrary combination of projectors without manual adjustment. When the projectors are tiled, we show that our framework automatically produces blending maps that outperform state-of-the-art projector blending methods. When all the projectors are superimposed, the framework can produce high-resolution images beyond the Nyquist resolution limits of component projectors. When a combination of tiled and superimposed projectors are deployed, the same framework harnesses the best features of both tiled and superimposed multi-projector projection paradigms. The framework creates for the first time a new unified paradigm that is agnostic to a particular configuration of projectors yet robustly optimizes for the brightness, contrast, and resolution of that configuration. In addition, we demonstrate that our algorithms support high resolution video at real-time interactive frame rates achieved on commodity graphics platforms. This work allows for inexpensive, compelling, flexible, and robust large scale visualization systems to be built and deployed very efficiently.

While popular as a visual interface, current interactive 3D environments are often painstakingly created by hand. The paper proposes interactive 3D environments that are automatically designed on-the-fly based on various design rules, thereby separating layout and content. The resulting environments are compelling, customizable, and intuitive for information exploration. The interface also serves as an integrated framework for different types of rich media. The complete system for designing, constructing, and rendering the environments in real lime is discussed and shown to have implications in a variety of applications, including e-commerce.

This paper presents a practical framework for creating and visualizing interactive 3-D media using a system of uncalibrated projector-cameras. The proposed solution uses light patterns that temporally encode the projector’s coordinate system to solve the traditionally challenging multiframe correspondence problem by straightforward decoding instead of computational multiframe optimization. Two sets of coded light patterns (black/white stripes and colored 2x2 blocks, both of varying spatial resolutions) are presented and compared. The resulting correspondences are directly used as a compelling form of interactive 3-D media through described techniques including three-frame view synthesis, multiframe view synthesis using multiple three-frame groupings, and even single-camera view interpolation. It is shown that adapting the rendering order of the correspondences with respect to the projector’s coordinate system ensures the correct visibility for the synthesized views. Experimental results demonstrate that the framework works well for various real-world scenes, even including those with multiple objects and textured surfaces. The framework, along with the resulting correspondences, also has implications in many other computer vision and image processing applications, especially those that require multiframe correspondences.

We propose a new depth-based representation of 3-D scenes primarily for the problem of arbitrary view synthesis. The information contained in a given image sequence is compacted into a single multivalued array, where points are organized by occlusions rather than by coherent affine motions. This grouping facilitates an automatic process to determine the number of layers and helps to reduce the artifacts caused by occlusions in the scene. In addition, an iterative multiframe dynamic programming algorithm is described to produce piecewise smooth depth maps. A novel multiframe segmentation, tracking, and plane fitting algorithm is also proposed to handle the traditionally difficult low-contrast regions. Reconstructed views as well as arbitrary flyarounds of real scenes are presented to demonstrate the effectiveness of the approach.

We propose a computational framework for assessing uncertainty of structure estimates under perspective projection for a given camera configuration. Unlike probabilistic models, the approach takes into account finite sensor areas and relates the uncertainty in estimating a set of points to the volumes formed by the intersections of their associated frustums. Simulations demonstrate and compare the effectiveness of lateral and semicircular motion for structure recovery of arbitrarily-shaped objects.

This paper focuses on the representation and view generation of three-dimensional (3-D) scenes. In contrast to existing methods that construct a full 3-D model or those that exploit geometric invariants, our representation consists of dense depth maps at several preselected viewpoints from an image sequence. Furthermore, instead of using multiple calibrated stationary cameras or range scanners, we derive our depth maps from image sequences captured by an uncalibrated camera with only approximately known motion. We propose an adaptive matching algorithm that assigns various confidence levels to different regions in the depth maps. Nonuniform bicubic spline interpolation is then used to fill in low confidence regions in the depth maps. Once the depth maps are computed at preselected viewpoints, the intensity and depth at these locations are used to reconstruct arbitrary views of the 3-D scene. Specifically, the depth maps are regarded as vertices of a deformable 2-D mesh, which are transformed in 3-D, projected to 2-D, and rendered to generate the desired view. Experimental results are presented to verify our approach.

This paper focuses on the representation and arbitrary view generation of three dimensional (3-D) scenes. In contrast to existing methods that construct a full 3-D model or those that exploit geometric invariants, our representation consists of dense depth maps at several preselected viewpoints from an image sequence. Furthermore, instead of using multiple calibrated stationary cameras or range data, we derive our depth maps from image sequences captured by an uncalibrated camera. We propose an adaptive matching algorithm which assigns various confidence levels to different regions. Nonuniform bicubic spline interpolation is then used to fill in low confidence regions in the depth maps. Once the depth maps are computed at preselected viewpoints, the intensity and depth at these locations are used to reconstruct arbitrary views of the 3-D scene. Experimental results are presented to verify our approach.