Stereo matching has a long history in image processing and computer vision. In fact, there are inumerous approaches reported in the literature, and quantitative evaluation is usually performed by comparing the obtained disparity maps with ground truth data (using the MSE, for instance). One important application of stereo matching is view interpolation, where it is desired to produce a new synthetic view from (at least) a pair of images and the corresponding disparity maps. In view interpolation, evaluation is mostly qualitative (visual quality of the synthesized image), and quantitative approaches compute objective similarity metrics between the synthesized image and the actual image at the same position (e.g. PSNR). The main goal of this paper is to evaluate the impact of several different stereo matching algorithms in a view interpolation context, relating the quality of the disparity maps with the quality of the corresponding synthesized views using standardized datasets. In this paper, experiments using the MPEG reference software for view interpolation and more than twenty datasets are presented and discussed. Our results indicate that the use of the common percentage of bad pixels as a metric for stereo matching methods does not translate well to the quality of view interpolation.

This paper presents a new approach for stereo matching and view interpolation problems based on triangular tessellations suitable for a linear array of rectified cameras. The domain of the reference image is initially partitioned into triangular regions using edge and scale information, aiming to place vertices along image edges and increase the number of triangles in textured regions. A region-based matching algorithm is then used to find an initial disparity for each triangle, and a refinement stage is applied to change the disparity at the vertices of the triangles, generating a piecewise linear disparity map. A simple post-processing procedure is applied to connect triangles with similar disparities generating a full 3D mesh related to each camera (view), which are used to generate new synthesized views along the linear camera array. With the proposed framework, view interpolation reduces to the trivial task of rendering polygonal meshes, which can be done very fast, particularly when GPUs are employed. Furthermore, the generated views are hole-free, unlike most point-based view interpolation schemes that require some kind of post-processing procedures to fill holes.

In this paper we propose a new disparity map estimation algorithm from multiple rectified images. The reference image is initially segmented into triangular regions, and each triangle is assigned to an initial disparity, leading to a piece-wise constant disparity map. Then, a refinement step is applied, where the disparity of the triangle vertices is adjusted aiming to impose spatial consistency to the disparity map, smoothing the map within the objects and keeping the discontinuities between them. The final result is a piece-wise linear depth map on a triangular domain that can be explored for view synthesis.

The reflectance function of a scene point captures the appearance of that point as a function of lighting direction. We present an approach to printing the reflectance functions of an object or scene so that its appearance is modified correctly as a function of the lighting conditions when viewing the print. For example, such a “photograph” of a statue printed with our approach appears to cast shadows to the right when the “photograph” is illuminated from the left. Viewing the same print with lighting from the right will cause the statue's shadows to be cast to the left. Beyond shadows, all effects due to the lighting variation, such as Lambertian shading, specularity, and inter-reflection can be reproduced. We achieve this ability by geometrically and photometrically controlling specular highlights on the surface of the print. For a particular viewpoint, arbitrary reflectance functions can be built up at each pixel by controlling only the specular highlights and avoiding significant diffuse reflections. Our initial binary prototype uses halftoning to approximate continuous grayscale reflectance functions.

This paper presents a new bilayer video segmentation algorithm focusing on videoconferencing application. A face tracking algorithm is used to guide a generic Ω-shaped template of the head and shoulders. A region of interest (ROI) is created around the generic template, and an energy function based on edge, color and motion cues is used to define the boundary between the person and the background. Our experimental results indicate that the silhouettes can be effectively extracted in common videoconferencing scenarios.

Desktop video conferencing often suffers from bad lighting, which may be caused by harsh shadowing, saturated regions, etc. The primary reason for this is the lack of control over lighting in the user's environment. A hardware-based solution to this problem would be to place lights near the video camera, but these would be distracting to the user. We use a set of infrared lights placed around the computer monitor to gather a sequence of frames which is used to infer surface normals of the scene. These are used in combination with a visible spectrum image to create an improved relighting result.

Polynomial Texture Maps (PTM)[SIGGRAPH 2001] form an alternative method for apprehending surface colour and albedo that extends a simple model of image formation from the Lambertian variant of Photometric Stereo (PST) to more general reflectances. Here we consider solving such a model in a robust version, not to date attempted for PTM. But the main upshot of utilizing robust regression is in the identification of both shadows and specularities automatically, without the need for any thresholds, in a tripar-tite set of weights for pixels that are labelled as matte, shadow, or specularity. Original images are captured using a hemispherical set of lights, and pixel values across the light-ing directions are then labelled as inliers, or outliers of two types. A per-pixel robust regression on luminance is carried out using Least Median of Squares, and automatically-identified outlier pixels are labelled as shadows if they are darker than matte and corre-spondingly, specular outliers are too bright. Inlier identification generates correct values for chromaticity and for surface albedo and thus matte luminance and colour. Then a robust version of PST, using only PTM inliers, improves estimates of normal vectors and albedo recovered. With specular pixel values over the lights in hand we model specu-larity using a radial basis function (RBF) regression, and non-specular pixel departures from matte using a second RBF set. Then for a new lighting direction, we can readily interpolate both specular content as well as shadows. 1

Coliseum is a multiuser immersive remote teleconferencing system designed to provide collaborative workers the experience of face-to-face meetings from their desktops. Five cameras are attached to each PC display and directed at the participant. From these video streams, view synthesis methods produce arbitrary-perspective renderings of the participant and transmit them to others at interactive rates, currently about 15 frames per second. Combining these renderings in a shared synthetic environment gives the appearance of having all participants interacting in a common space. In this way, Coliseum enables users to share a virtual world, with acquired-image renderings of their appearance replacing the synthetic representations provided by more conventional avatar-populated virtual worlds. The system supports virtual mobility---participants may move around the shared space---and reciprocal gaze, and has been demonstrated in collaborative sessions of up to ten Coliseum workstations, and sessions spanning two continents.Coliseum is a complex software system which pushes commodity computing resources to the limit. We set out to measure the different aspects of resource, network, CPU, memory, and disk usage to uncover the bottlenecks and guide enhancement and control of system performance. Latency is a key component of Quality of Experience for video conferencing. We present how each aspect of the system---cameras, image processing, networking, and display---contributes to total latency. Performance measurement is as complex as the system to which it is applied. We describe several techniques to estimate performance through direct light-weight instrumentation as well as use of realistic end-to-end measures that mimic actual user experience. We describe the various techniques and how they can be used to improve system performance for Coliseum and other network applications. This article summarizes the Coliseum technology and reports on issues related to its performance---its measurement, enhancement, and control.

A new paradigm for automatic nonphotorealistic rendering (NPR) is introduced in this paper. Existing NPR approaches can be categorized in two groups depending on the type of input they use: image based and object based. Using multiple images as input to the NPR scheme, we propose a novel hybrid model that simultaneously uses information from the image and object domains. The benefit not only comes from combining the features of each approach, but most important, it minimizes the need for manual or user assisted tasks in extracting scene features and geometry, as employed in virtually all state-of-the-art NPR approaches. We describe a particular implementation of such an hybrid system and present a number of automatically generated pen-and-ink style drawings. This work then shows how to use and extend well developed techniques in computer vision to address fundamental problems in image representation and rendering.

Coliseum is a multiuser immersive remote teleconferencing system designed to provide collaborative workers the experience of face-to-face meetings from their desktops. Five cameras are attached to each PC display and directed at the participant. From these video streams, view synthesis methods produce arbitrary-perspective renderings of the participant and transmit them to others at interactive rates, currently about 15 frames per second. Combining these renderings in a shared synthetic environment gives the appearance of having all participants interacting in a common space. In this way, Coliseum enables users to share a virtual world, with acquired-image renderings of their appearance replacing the synthetic representations provided by more conventional avatar-populated virtual worlds. The system supports virtual mobility--participants may move around the shared space--and reciprocal gaze, and has been demonstrated in collaborative sessions of up to ten Coliseum workstations, and sessions spanning two continents. This paper summarizes the technology, and reports on issues related to its performance.

In this paper we present a new form of texture mapping that produces increased photorealism. Coefficients of a biquadratic polynomial are stored per texel, and used to reconstruct the surface color under varying lighting conditions. Like bump mapping, this allows the perception of surface deformations. However, our method is image based, and photographs of a surface under varying lighting conditions can be used to construct these maps. Unlike bump maps, these Polynomial Texture Maps (PTMs) also capture variations due to surface self-shadowing and interreflections, which enhance realism. Surface colors can be efficiently reconstructed from polynomial coefficients and light directions with minimal fixed-point hardware. We have also found PTMs useful for producing a number of other effects such as anisotropic and Fresnel shading models and variable depth of focus. Lastly, we present several reflectance function transformations that act as contrast enhancement operators. We have found these particularly useful in the study of ancient archeological clay and stone writings.

Gradient information is used in volume rendering to classify and color samples along a ray. In this paper, we present an analysis of the theoretically ideal gradient estimator and compare it to some commonly used gradient estimators. A new method is presented to calculate the gradient at arbitrary sample positions, using the derivative of the interpolation filter as the basis for the new gradient filter. As an example, we will discuss the use of the derivative of the cubic spline. Comparisons with several other methods are demonstrated. Computational efficiency can be realized since parts of the interpolation computation can be leveraged in the gradient estimation.

In computer graphics we have traditionally rendered images of data sets specified spatially, Here, we present a volume rendering technique that operates on a frequency domain representation of the data set and that efficiently generates line integral projections of the spatial data it represents, The motivation for this approach is that the Fourier Projection-Slice Theorem allows us to compute 2-D projections of 3-D data seta using only a 2-D slice of the data in the frequency domain.In general, these "X-ray-like" images can be rendered at a significantly lower computational cost than images generated by current volume rendering techniques, Additionally, assurances of image accuracy can he made.