In everyday meetings, automatic association of co-located mobile devices would ease sharing of web-links, media, and other information. We propose a method that compares patterns of silence from device microphones to detect co-location of those devices. This method works with unsynchronized audio capture, requires only 100bps and preserves privacy. We show how to formulate pattern matching in a sequential hypothesis framework so that changes in co-location status (when people leave or join a meeting) can be determined promptly, and how to compute the likelihood ratio in practice. Using 16 hours of captured audio, we show that our approach can correctly determine device co-location with a low error rate of 0.05%, and can detect co-location changes 10 seconds faster than a similar decision rule based on a constant time window. Compared to a prior audio signature method, we achieve higher accuracy at 1/7 the bit rate.

Stereo matching has a long history in image processing and computer vision. In fact, there are inumerous approaches reported in the literature, and quantitative evaluation is usually performed by comparing the obtained disparity maps with ground truth data (using the MSE, for instance). One important application of stereo matching is view interpolation, where it is desired to produce a new synthetic view from (at least) a pair of images and the corresponding disparity maps. In view interpolation, evaluation is mostly qualitative (visual quality of the synthesized image), and quantitative approaches compute objective similarity metrics between the synthesized image and the actual image at the same position (e.g. PSNR). The main goal of this paper is to evaluate the impact of several different stereo matching algorithms in a view interpolation context, relating the quality of the disparity maps with the quality of the corresponding synthesized views using standardized datasets. In this paper, experiments using the MPEG reference software for view interpolation and more than twenty datasets are presented and discussed. Our results indicate that the use of the common percentage of bad pixels as a metric for stereo matching methods does not translate well to the quality of view interpolation.

This document describes the technology behind the accompanying video, which gives a demonstration of determining the dynamic group membership of a meeting by matching patterns of relative audio silence, or "silence signatures," sensed by mobile devices.

A list of the dynamically changing group membership of a meeting supports a variety of meeting-related activities. Effortless content sharing might be the most important application, but we can also use it to provide business card information for attendees, feed information into calendar applications to simplify scheduling of follow-up meetings, populate the membership of collaborative editing applications, mailing lists, and social networks, and perform many other tasks.

This paper presents a new approach for stereo matching and view interpolation problems based on triangular tessellations suitable for a linear array of rectified cameras. The domain of the reference image is initially partitioned into triangular regions using edge and scale information, aiming to place vertices along image edges and increase the number of triangles in textured regions. A region-based matching algorithm is then used to find an initial disparity for each triangle, and a refinement stage is applied to change the disparity at the vertices of the triangles, generating a piecewise linear disparity map. A simple post-processing procedure is applied to connect triangles with similar disparities generating a full 3D mesh related to each camera (view), which are used to generate new synthesized views along the linear camera array. With the proposed framework, view interpolation reduces to the trivial task of rendering polygonal meshes, which can be done very fast, particularly when GPUs are employed. Furthermore, the generated views are hole-free, unlike most point-based view interpolation schemes that require some kind of post-processing procedures to fill holes.

In this paper we propose a new disparity map estimation algorithm from multiple rectified images. The reference image is initially segmented into triangular regions, and each triangle is assigned to an initial disparity, leading to a piece-wise constant disparity map. Then, a refinement step is applied, where the disparity of the triangle vertices is adjusted aiming to impose spatial consistency to the disparity map, smoothing the map within the objects and keeping the discontinuities between them. The final result is a piece-wise linear depth map on a triangular domain that can be explored for view synthesis.

Many recent prototypes for video collaboration, digital media sharing and gesture interfaces provide a video signal for display on a screen or surface and capture another video signal through the same screen or surface. The media captured in such systems, for transmission or for gesture user interfaces, needs to be separated from the displayed video. Otherwise, video cross-talk occurs. The prior, widely used temporal multiplexing avoids cross-talk by synchronizing camera capture with screen display so that the camera only captures when the screen does not display signal. This approach suffers from light loss (both displayed and captured) and increased display flicker due to the lower duty cycle of the displayed signal. This paper describes a new method, computational temporal modulation, that temporally modulates the displayed signal. The intentionally mixed signals captured by the camera are subsequently separated using computations. Our approach results in brighter display with less flicker and more signal captured by the camera. Experiments using a prototype collaboration system show good quality cross-talk reduction with light-weight real-time computation.

The reflectance function of a scene point captures the appearance of that point as a function of lighting direction. We present an approach to printing the reflectance functions of an object or scene so that its appearance is modified correctly as a function of the lighting conditions when viewing the print. For example, such a “photograph” of a statue printed with our approach appears to cast shadows to the right when the “photograph” is illuminated from the left. Viewing the same print with lighting from the right will cause the statue's shadows to be cast to the left. Beyond shadows, all effects due to the lighting variation, such as Lambertian shading, specularity, and inter-reflection can be reproduced. We achieve this ability by geometrically and photometrically controlling specular highlights on the surface of the print. For a particular viewpoint, arbitrary reflectance functions can be built up at each pixel by controlling only the specular highlights and avoiding significant diffuse reflections. Our initial binary prototype uses halftoning to approximate continuous grayscale reflectance functions.

Professional studios produce good lighting by using well-positioned, bright studio lights to illuminate the subjects, while casual desktop video conferencing often suffers from uneven and unreliable lighting. Using bright lights is not an option - they annoy and distract non-professional subjects. In this work, we illuminate the subjects using invisible near infrared (IR) lights and capture the subject simultaneously in IR and visible light. The IR video is used as a reference to enhance the visible video. The advantages of IR illumination for video conferencing have previously been shown, but the prior works require 4 to 8 times higher IR frame rate, with concomitant complexity and bandwidth increases. The enhancement developed here performs well with low IR frame rates equal to the visible frame rate. The technique works well even when uneven lighting causes bright visible light pixels to clip. After describing the technique, comparisons with prior methods are provided.

Desktop video conferencing participants are often poorly positioned in their outgoing video - unaware of their appearance on camera. We combine face detection, feature tracking and motion detection for automatic real-time detection of poorly framed participants and subsequently provide framing feedback by compositing their incoming and outgoing video streams. Our solution provides participants visual feedback only while they have framing problems. Otherwise the display shows only the remote participants, allowing users to focus fully on the conference. We analyze the system components and describe a user study that found advantages of our approach over the existing mirror window solution.

This paper presents a new bilayer video segmentation algorithm focusing on videoconferencing application. A face tracking algorithm is used to guide a generic Ω-shaped template of the head and shoulders. A region of interest (ROI) is created around the generic template, and an energy function based on edge, color and motion cues is used to define the boundary between the person and the background. Our experimental results indicate that the silhouettes can be effectively extracted in common videoconferencing scenarios.

In this paper, we address the problem of no-reference quality assessment for digital pictures corrupted with blur. We start with the generation of a large real image database containing pictures taken by human users in a variety of situations, and the conduction of subjective tests to generate the ground truth associated to those images. Based upon this ground truth, we select a number of high quality pictures and artificially degrade them with different intensities of simulated blur (gaussian and linear motion), totalling 6000 simulated blur images. We extensively evaluate the performance of state-of-the-art strategies for no-reference blur quantification in different blurring scenarios, and propose a paradigm for blur evaluation in which an effective method is pursued by combining several metrics and low-level image features. We test this paradigm by designing a no-reference quality assessment algorithm for blurred images which combines different metrics in a classifier based upon a neural network structure. Experimental results show that this leads to an improved performance that better reflects the images' ground truth. Finally, based upon the real image database, we show that the proposed method also outperforms other algorithms and metrics in realistic blur scenarios.

Desktop video conferencing often suffers from bad lighting, which may be caused by harsh shadowing, saturated regions, etc. The primary reason for this is the lack of control over lighting in the user's environment. A hardware-based solution to this problem would be to place lights near the video camera, but these would be distracting to the user. We use a set of infrared lights placed around the computer monitor to gather a sequence of frames which is used to infer surface normals of the scene. These are used in combination with a visible spectrum image to create an improved relighting result.

Recent two-way collaboration prototypes attempt to improve natural interactivity, correct eye contact and gaze direction, and media sharing using novel configurations of projectors, screens, and video cameras. These systems are often afflicted by video cross-talk where the content displayed for viewing by the local participant is unintentionally captured by the camera and delivered to the remote participant. Prior attempts to reduce this cross-talk purely in hardware through various forms of multiplexing (e.g., temporal, wavelength (color), polarization) have performance and cost limitations. In this work, careful system characterization and subsequent signal processing algorithms allow us to reduce video cross-talk. The signals themselves are used to detect temporal synchronization offsets which then allow subsequent reduction of the cross-talk signal. Our software-based approach enables the effective use of simpler hardware and optics than prior methods. Results show substantial cross-talk reduction in a system with unsynchronized projector and camera.

Modern digital presentation systems use rich media to bring highly sophisticated information visualization and highly effective storytelling capabilities to classrooms and corporate boardrooms. In this paper we address a number of issues that arise when the ubiquitous computer-projector setup is used in large venues like the cavernous auditoriums and hotel ballrooms often used in large scale academic meetings and industrial conferences. First, when the presenter is addressing a large audience the slide display needs to be very large and placed high enough so that it is clearly visible from all corners of the room. This makes it impossible for a presenter to walk up to the display and interact with the display with gestures, gaze, and other forms of paralanguage. Second, it is hard for the audience to know which part of the slide the presenter is looking at when he/she has to look the opposite way from the audience while interacting with the slide material. It is also hard for the presenter to see the audience in these cases. Even though there may be video captures of the presenter, slides, and even the audience, the above factors add up to make it very difficult for a user viewing either a live feed or a recording to grasp the interaction between all the components and participants of a presentation. We address these problems with a novel presentation system which creates a live video view that seamlessly combines the presenter and the presented material, capturing all graphical, verbal, and nonverbal channels of communication. The system also allows the local and remote audiences to have highly interactive exchanges with the presenter while creating a comprehensive view for recording or remote streaming.

A printed digital photograph is difficult to reuse because the digital information that generated the print may no longer be available. This paper describes a method for approximating the original digital image by combining a scan of the printed photograph with digital auxiliary information kept together with the print. We formulate and solve the approximation problem using a Wyner-Ziv coding framework. During encoding, the Wyner-Ziv auxiliary information consists of a small amount of digital data composed of a number of sampled luminance pixel blocks and a number of sampled color pixel values to enable subsequent accurate registration and color-reproduction during decoding. The registration and color information is augmented by an additional amount of digital data encoded using Wyner-Ziv coding techniques that recovers residual errors and lost high spatial frequencies. The decoding process consists of scanning the printed photograph, together with a two step decoding process. The first decoding step, using the registration and color auxiliary information, generates a side-information image which registers and color corrects the scanned image. The second decoding step uses the additional Wyner-Ziv layer together with the side-information image to provide a closer approximation of the original, reducing residual errors and restoring the lost high spatial frequencies. The experimental results confirm the reduced digital storage needs when the scanned print assists in the digital reconstruction.

The information about the blur and noise of an original image is lost when a standard image thumbnail is generated by filtering and subsampling. Image browsing becomes difficult since the standard thumbnails do not distinguish between high-quality and low-quality originals. In this paper, an efficient algorithm with a blur-generating component and a noise-generating component preserves the local blur and the noise of the originals. The local blur is rapidly estimated using a scale-space expansion of the standard thumbnail and subsequently used to apply a space-varying blur to the thumbnail. The noise is estimated and rendered by using multirate signal transformations that allow most of the processing to occur at the lower spatial sampling rate of the thumbnail. The new thumbnails provide a quick, natural way for users to identify images of good quality. A subjective evaluation shows the new thumbnails are more representative of their originals for blurry images. The noise generating component improves the results for noisy images, but degrades the results for textured images. The blur generating component of the new thumbnails may always be used to advantage. The decision to use the noise generating component of the new thumbnails should be based on testing with the particular image mix expected for the application.

The process of adding high-density information onto printed material enables and improves interesting hardcopy document applications, such as: security, authentication, physical-electronic round tripping, item-level tagging as well as consumer/product interaction. This investigation on robust and high capacity print codes aims to maximize information payload in a given printed page area, subject to robustness to distortions originated by printing and scanning processes and also to degradations introduced by user manipulation of printed documents. The novel approach includes statistical print-and-scan channel characterization, designing of robust segmentation, unsupervised Bayesian color classification with expectation-maximization algorithm for parameters estimation of a mixture of Gaussians model and design of error correction codes. Results illustrate the performance evaluated under real channel and distortions conditions. High payload is achieved with sufficient robustness to distortions resulting of regular office hardcopy document handling: print-and-scan channel and user manipulation.

Video conferencing without controlled lighting suffers from the spurious automatic exposure (AE) errors commonly seen in Webcams. These errors cause problems for the subsequent processing and compression. For example, since video encoders do not model intensity changes, these AE errors in turn cause severe blocking artifacts. We develop a pixel-domain AE conditioning algorithm for stationary cameras that: (1) effectively reduces spurious AE changes, resulting in natural and artifact-free video; (2) allows maximum compatibility with third party components (may be transparently inserted between any camera driver and encoder/video processing engine); and (3) is fast and requires little memory. This algorithm allows inexpensive cameras to provide higher quality video conferencing. We describe the algorithm, analyze its performance exactly for a specific video source model and validate its performance experimentally using captured video.

Adding high-density information to printed materials enables and improves interesting hardcopy document applications involving security, authentication, physical-electronic round tripping, item-level tagging, and consumer/product interaction. This investigation of robust and high capacity print codes aims to maximize information payload in a given printed page area, subject to robustness to channel errors including distortions introduced by the printing and scanning processes and also due to the usual degradations introduced by user manipulation of printed documents. The novel approach includes statistical print-and-scan channel characterization, designing of robust segmentation using visual cues, unsupervised Bayesian color classification with expectation-maximization algorithm for parameters estimation of a mixture of Gaussians model and design of error correction codes. Results illustrate the performance evaluated under real channel and distortions conditions. High payload is achieved with sufficient robustness to distortions resulting of regular office hardcopy document handling: print-and-scan channel and user manipulation.

We present ConnectBoard, a new system for remote collaboration where users experience natural interaction with one another, seemingly separated only by a vertical, transparent sheet of glass. It overcomes two key shortcomings of conventional video communication systems: the inability to seamlessly capture natural user interactions, like using hands to point and gesture at parts of shared documents, and the inability of users to look into the camera lens without taking their eyes off the display. We solve these problems by placing the camera behind the screen, where the remote user is virtually located. The camera sees through the display to capture images of the user. As a result, our setup captures natural, frontal views of users as they point and gesture at shared media displayed on the screen between them. Users also never have to take their eyes off their screens to look into the camera lens. Our novel optical solution based on wavelength multiplexing can be easily built with off-the-shelf components and does not require custom electronics for projector-camera synchronization.

A printed photograph is difficult to reuse because the digital pixels that generated it are no longer available. This paper describes approximating the original digital pixels by combining a scan of the printed photograph with small amounts of digital auxiliary information kept with the print, using an encoding such as a dense barcode. The auxiliary information enables accurate registration and color-reproduction, and the recovery of information lost due to the print-scan channel, by distributed source coding techniques. Approximating the original digital image enables many uses, including making high-fidelity reprints, as well as robust watermark detection. A block entropy coder using terminated tree codes for cosets of DCT coefficients is proposed for distributed image coding.

When a standard image thumbnail is generated by filtering and subsampling the original, information about the blur and noise of the original is lost. This loss of information makes it difficult to use standard thumbnails for image browsing since they do not distinguish between good quality and bad quality originals. This paper addresses image noise, developing a fast algorithm for preserving noise while reducing image size. Multirate signal transformations modify a traditional denoising and subsampling approach so that most of the processing occurs at the low spatial sampling rate of the reduced-sized output image. Experiments with noisy images and simulations show that the results of the new algorithm are very similar to the results of traditional denoising and subsampling. For the subsampling ratios typical between originals and image thumbnails, the new algorithm is an order of magnitude faster.

Image thumbnails are commonly used for selecting images for display, sharing or printing. Standard thumbnails, generated with current techniques, do not distinguish between high and low quality originals. Both sharp and blurry originals appear sharp in the thumbnails, and both clean and noisy originals appear clean in the thumbnails. This leads to errors and inefficiencies during image selection. In this paper, thumbnails generated using image analysis better represent the local blur and the noise of the originals. The new thumbnails provide a quick, natural way for users to identify images of good quality, while allowing the viewer's knowledge to select desired subject matter. Computer simulations with added blur and noise show the new thumbnails better represent images of differing qualities. Validation of these findings is found in a subjective evaluation reported elsewhere but summarized below.

Automatic generation of music thumbnails based on content analysis is important for efficient management and better consumption of digital music. In this paper, a unique method for extracting thumbnails from a song based on structure analysis of the music piece is presented. First, paragraphs in the song with repeated melody are detected; next, vocal portions in the song are identified. With such information, the structure of the song is derived. Then, one or more thumbnails of the song can be selected according to the song structure and heuristic rules. This approach has proved to be effective when tested on a variety of music pieces. It takes only a couple of seconds to process each song. The method was integrated into a music search system. Also, a demo was built based on it, which lets the user instantly view the structure and thumbnails of any song he selects.

Tagging digital media, such as photos and videos, with capture time and location information has previously been proposed to enhance its organization and presentation. We believe that the full path traveled during media capture, rather than just the media capture locations, provides a much richer context for understanding and "re-living" a trip experience, and offers many possibilities for novel applications. We introduce the concept of path-enhanced media, in which media is associated and stored together with a densely sampled path in time and space and we present the MediaBeads architecture for capturing, representing, browsing, editing, presenting, and searching this data. The architecture includes, among other things, novel data representations, new algorithms for automatically building movie-like presentations of trips, and novel search applications.

Central to capturing a trip is knowing where you were, and when you were there. Combining continuous path data with media (path-enhanced media or PEM) offers substantial advantages over the previous approach of tagging individual media with time and location. Prototype systems, collectively called PathMarker, are used for gathering, editing, presenting and browsing PEM. We have developed: (1) a methodology for gathering PEM with off-the-shelf hardware; (2) software for automatic conversion of the raw path data and media into an application independent XML representation; (3) two example PEM applications. The first application provides map-overlaid trip editing, presentation and browsing. The second application provides a 3D immersive environment with digital elevation maps for automatic trip flybys and for browsing. Experience with a number of recorded trips confirms that PathMarker systems seem to capture the essence of a trip.

This correspondence describes an algorithm for estimating the proportions of classes in an SAR image by first assuming that an image consists of a mixture of a known number of different pixel types. A maximum likelihood estimate of the parameters of the resulting mixture distribution is then evaluated using the EM algorithm. An advantage of the finite mixtures approach is that the quantities of interest, the proportions, are directly estimated. The technique is applied to aircraft synthetic aperture radar (SAR) images of sea ice. In addition to finding the proportions of the classes, knowledge of the mixture components allows image displays tailored to a user's requirements.