In this paper we address calibration of camera arrays for life-size 3D video acquisition and display where mosaicking and multiviewpoint stereo are combined to provide an immersive experience which, by its size, resolution, and three-dimensionality, is meant to rival being there. This coupling of multiview and mosaicking requires integration of numerous synchronized video streams in a single presentation, aligned for both panoramic blending and epipolar rectification. The calibration framework we have developed extends the classical checkerboard approach through a modular multi-stage pipeline performing global optimization across intrinsics, extrinsics, panoramas, multiview epipolar alignments, and color correction. We demonstrate the methodology on several multiview camera arrays with various configurations aimed at mosaicking and epipolar light-field analysis. The results of this analysis have driven real-time life-sized panoramic 3D displays of captured events such as a concert, a fashion show, and sports activity.

We describe methods, accomplishments, and direction in a laboratory-corporate collaborative project over this past year and a half. With the goal of taking new immersive technologies to a growing customer base through in-situ demonstrations and experiments, we are combining novel capture and display capabilities in delivering life-sized immersive 3D entertainment experiences. We describe the adaptations and innovations needed in delivering immersive life sized 3D to a large audience.

In this paper we describe an exploratory study of a mobile cosmetic advisory system that enables women to select appropriate colors of cosmetics. This system is intended for commercial use to address the problem of foundation color selection. Although women are primarily responsible for making most purchasing decisions in the US, we found very few studies to assess the adoption of retail related mobile services by women. Based on surveys, semi-structured interviews, and focus groups, we have identified a number of design factors that should be considered when designing mobile services for women consumers. The results of our study indicate that while usefulness is an important factor, other design aspects such as mobile vs. kiosk, installed vs. existing software, technical comfort vs. social comfort, social vs. individual, privacy and trust should also be accounted for.

We describe an automatic method for classifying skin color, independent of lighting and imaging device characteristics, using consumer digital cameras and a simple color calibration target. After color normalization and face detection is performed, pixels of each face image are clustered in an unsupervised fashion. Pixels likely to be representative of skin color, rather than of distractors such as shadows, specularities, eyes, and lips, are identified by selecting the dominant clusters that have large number of pixels assigned per volume. A Gauss mixture model (GMM) of a person's skin color is formed from the pixels belonging to the selected clusters. When a set of exemplar images with skin color labels by an expert, we show that the label assigned by the same expert to a new, test face image can be predicted by comparison of the GMMs of the test image and the exemplars. Specifically, we use the label of the exemplar whose GMM has smallest KL divergence from that of the test image.

Little prior image processing work has addressed estimation and classification of skin color in a manner that is independent of camera and illuminant. To this end, we first present new methods for 1) fast, easy-to-use image color correction, with specialization toward skin tones, and 2) fully automated estimation of facial skin color, with robustness to shadows, specularities, and blemishes. Each of these is validated independently against ground truth, and then combined with a classification method that successfully discriminates skin color across a population of people imaged with several different cameras. We also evaluate the effects of image quality and various algorithmic choices on our classification performance. We believe our methods are practical for relatively untrained operators, using inexpensive consumer equipment.

Coliseum is a multiuser immersive remote teleconferencing system designed to provide collaborative workers the experience of face-to-face meetings from their desktops. Five cameras are attached to each PC display and directed at the participant. From these video streams, view synthesis methods produce arbitrary-perspective renderings of the participant and transmit them to others at interactive rates, currently about 15 frames per second. Combining these renderings in a shared synthetic environment gives the appearance of having all participants interacting in a common space. In this way, Coliseum enables users to share a virtual world, with acquired-image renderings of their appearance replacing the synthetic representations provided by more conventional avatar-populated virtual worlds. The system supports virtual mobility---participants may move around the shared space---and reciprocal gaze, and has been demonstrated in collaborative sessions of up to ten Coliseum workstations, and sessions spanning two continents.Coliseum is a complex software system which pushes commodity computing resources to the limit. We set out to measure the different aspects of resource, network, CPU, memory, and disk usage to uncover the bottlenecks and guide enhancement and control of system performance. Latency is a key component of Quality of Experience for video conferencing. We present how each aspect of the system---cameras, image processing, networking, and display---contributes to total latency. Performance measurement is as complex as the system to which it is applied. We describe several techniques to estimate performance through direct light-weight instrumentation as well as use of realistic end-to-end measures that mimic actual user experience. We describe the various techniques and how they can be used to improve system performance for Coliseum and other network applications. This article summarizes the Coliseum technology and reports on issues related to its performance---its measurement, enhancement, and control.

Coliseum is a multiuser immersive remote teleconferencing system designed to provide collaborative workers the experience of face-to-face meetings from their desktops. Five cameras are attached to each PC display and directed at the participant. From these video streams, view synthesis methods produce arbitrary-perspective renderings of the participant and transmit them to others at interactive rates, currently about 15 frames per second. Combining these renderings in a shared synthetic environment gives the appearance of having all participants interacting in a common space. In this way, Coliseum enables users to share a virtual world, with acquired-image renderings of their appearance replacing the synthetic representations provided by more conventional avatar-populated virtual worlds. The system supports virtual mobility--participants may move around the shared space--and reciprocal gaze, and has been demonstrated in collaborative sessions of up to ten Coliseum workstations, and sessions spanning two continents. This paper summarizes the technology, and reports on issues related to its performance.

The previous implementations of the authors' epipolar-plane image-analysis mapping technique demonstrated the feasibility and benefits of the approach, but were carried out for restricted camera geometries. The question of more general geometries made the technique's utility for autonomous navigation uncertain. The authors have developed a generalization of the analysis that: (1) enables varying view direction, including varying over time; (2) provides three-dimensional connectivity information for building coherent spatial descriptions of observed objects; and (3) operates sequentially, allowing initiation and refinement of scene feature estimates while the sensor is in motion. To implement this generalization it was necessary to develop an explicit description of the evolution of images over time. They achieved this by building a process that creates a set of two-dimensional manifolds defined at the zeros of a three-dimensional spatiotemporal Laplacian. These manifolds represent explicitly both the spatial and temporal structure of the temporally evolving imagery and are termed spatiotemporal surfaces.>

This article presents a facility developed in support of image sequence procesiiiig for an autonoliious x-eliicle. As in man the success of the sequence analysis tlel?ends 011 t,he ability to operate explicitly in both space and tinic, and 011 exploiting the massive redundancy present iii the hundreds of view that can lie obtained when moving through a scene. The mechanism in our analysis that integrates the is a 3-D surrace-buildins process called the il'envzrzg Wall this paper focuses on the construction and use of siirfaces from this process In robotic navigation mork the surfaces represent the space-time evolulzon of scene images, and this rcpresen(ation, in coiijuiictioii with geometric constraints, allows us to determine tlie thrcc-dimensional structure of a scene. In other domains where there is a similar gradual evolution of data oyer a third dimension, in medical tomography or scale-space studies, for example, the surfaces constructed by the Weaving Wall are iiniiiediately of value for their topographic structure. The designs of both the surface-building and scene-rcconstriictiiig processes make them well-suited for real-time operation given appropriate hardware The principlc hehiiid tlie \Veaviiig TT.sl1 may have a dramatic impact 011 vision; medical iniagiiig, antl other fields.