Alignment of many 3D point clouds, possibly captured by multiple devices at different times, is a critical step for increasingly popular applications such as 3D model construction and augmented reality. For very large data sets, traditional methods such as ICP can become computationally intractable, or produce poor results. We present an efficient method for accurately aligning very large numbers of dense 3D point clouds, and apply it to a city-scale data set. The method relies on the novel combination of 1) partitioning the point clouds based on loop structures detected across a combined network of all device capture paths, and 2) making use of the loop closure property to accurately align point clouds within each sub-problem. Final global alignment of the loop-based results is formulated as a least squares optimization with closed form solution. Experimental results are shown for aligning 3D points across the entire city of San Francisco with centimeter-scale accuracy, via an efficient parallelized architecture.

In this paper we describe an exploratory study of a mobile cosmetic advisory system that enables women to select appropriate colors of cosmetics. This system is intended for commercial use to address the problem of foundation color selection. Although women are primarily responsible for making most purchasing decisions in the US, we found very few studies to assess the adoption of retail related mobile services by women. Based on surveys, semi-structured interviews, and focus groups, we have identified a number of design factors that should be considered when designing mobile services for women consumers. The results of our study indicate that while usefulness is an important factor, other design aspects such as mobile vs. kiosk, installed vs. existing software, technical comfort vs. social comfort, social vs. individual, privacy and trust should also be accounted for.

We describe an automatic method for classifying skin color, independent of lighting and imaging device characteristics, using consumer digital cameras and a simple color calibration target. After color normalization and face detection is performed, pixels of each face image are clustered in an unsupervised fashion. Pixels likely to be representative of skin color, rather than of distractors such as shadows, specularities, eyes, and lips, are identified by selecting the dominant clusters that have large number of pixels assigned per volume. A Gauss mixture model (GMM) of a person's skin color is formed from the pixels belonging to the selected clusters. When a set of exemplar images with skin color labels by an expert, we show that the label assigned by the same expert to a new, test face image can be predicted by comparison of the GMMs of the test image and the exemplars. Specifically, we use the label of the exemplar whose GMM has smallest KL divergence from that of the test image.

In prior work, we introduced adaptive plan-view height and occupancy templates, derived from stereo camera data, for person tracking and activity recognition. These templates efficiently capture current details of each tracked person's body pose, thereby enabling good tracking performance even when multiple people occlude and interact with each other. However, the templates ignore useful color information, and their rapid evolution makes them poorly suited for recognizing the same person at well-separated times. In this paper, we seek to remedy both of these shortcomings, by 1) adding novel plan-view color templates to our short-term, template-based models of person appearance, and 2) augmenting our person descriptions with longer-term models that describe invariants of each person's shape and color. We demonstrate how each of these improves our real-time tracking performance on challenging, multi-person sequences.

Little prior image processing work has addressed estimation and classification of skin color in a manner that is independent of camera and illuminant. To this end, we first present new methods for 1) fast, easy-to-use image color correction, with specialization toward skin tones, and 2) fully automated estimation of facial skin color, with robustness to shadows, specularities, and blemishes. Each of these is validated independently against ground truth, and then combined with a classification method that successfully discriminates skin color across a population of people imaged with several different cameras. We also evaluate the effects of image quality and various algorithmic choices on our classification performance. We believe our methods are practical for relatively untrained operators, using inexpensive consumer equipment.

Tagging digital media, such as photos and videos, with capture time and location information has previously been proposed to enhance its organization and presentation. We believe that the full path traveled during media capture, rather than just the media capture locations, provides a much richer context for understanding and "re-living" a trip experience, and offers many possibilities for novel applications. We introduce the concept of path-enhanced media, in which media is associated and stored together with a densely sampled path in time and space and we present the MediaBeads architecture for capturing, representing, browsing, editing, presenting, and searching this data. The architecture includes, among other things, novel data representations, new algorithms for automatically building movie-like presentations of trips, and novel search applications.

Central to capturing a trip is knowing where you were, and when you were there. Combining continuous path data with media (path-enhanced media or PEM) offers substantial advantages over the previous approach of tagging individual media with time and location. Prototype systems, collectively called PathMarker, are used for gathering, editing, presenting and browsing PEM. We have developed: (1) a methodology for gathering PEM with off-the-shelf hardware; (2) software for automatic conversion of the raw path data and media into an application independent XML representation; (3) two example PEM applications. The first application provides map-overlaid trip editing, presentation and browsing. The second application provides a 3D immersive environment with digital elevation maps for automatic trip flybys and for browsing. Experience with a number of recorded trips confirms that PathMarker systems seem to capture the essence of a trip.

We present MSA (media services architecture), a flexible, general architecture for requesting, configuring, and running services that operate on streaming audio and video as it flows through the network. MSA decomposes requested media services into modular processing components that may be distributed to servers throughout the network and which intercommunicate via standard streaming protocols. Use of standard protocols also affords seamless interoperability between MSA and media content delivery networks. MSA manages media services by monitoring the networked servers and assigning service components to them in a manner that uses available computational and network resources efficiently. We describe some implemented example services to illustrate the concepts and benefits of the architecture.

A new algorithm for background estimation and removal in video sequences obtained with stereo cameras is presented. Per-pixel Gaussian mixtures are used to model recent scene observations in the combined space of depth and luminance-invariant color. These mixture models adapt over time, and are used to build a new model of the background at each time step. This combination in itself is novel, but we also introduce the idea of modulating the learning rate of the background model according to the scene activity level on a per-pixel basis, so that dynamic foreground objects are incorporated into the background more slowly than are static scene changes. Our results show much greater robustness than prior state-of-the-art methods to challenging phenomena such as video displays, non-static background objects, areas of high foreground traffic, and similar color of foreground and background. Our method is also well-suited for use in real-time systems.

This paper explores several approaches for articulated-pose estimation, assuming that video-rate depth information is available, from either stereo cameras or other sensors. We use these depth measurements in the traditional linear brightness constraint equation, as well as in a depth constraint equation. To capture the joint constraints, we combine the brightness and depth constraints with twist mathematics. We address several important issues in the formation of the constraint equations, including updating the body rotation matrix without using a first-order matrix approximation and removing the coupling between the rotation and translation updates. The resulting constraint equations are linear on a modified parameter set. After solving these linear constraints, there is a single closed-form non-linear transformation to return the updates to the original pose parameters. We show results for tracking body pose in oblique views of synthetic walking sequences and in moving-camera views of synthetic jumping-jack sequences. We also show results for tracking body pose in side views of a real walking sequence.

Background estimation and removal based on the joint use of range and color data produces superior results than can be achieved with either data source alone. This is increasingly relevant as inexpensive, real-time, passive range systems become more accessible through novel hardware and increased CPU processing speeds. Range is a powerful signal for segmentation which is largely independent of color and hence not effected by the classic color segmentation problems of shadows and objects with color similar to the background. However range alone is also not sufficient for the good segmentation: depth measurements are rarely available at all pixels in the scene, and foreground objects may be indistinguishable in depth when they are close to the background. Color segmentation is complementary in these cases. Surprisingly, little work has been done to date on joint range and color segmentation. We describe and demonstrate a background estimation method based on a multidimensional (range and color) clustering at each image pixel. Segmentation of the foreground in a given frame is performed via comparison with background statistics in range and normalized color. Important implementation issues such as treatment of shadows and low confidence measurements are discussed in detail.

This paper explores the direct motion estimation problem assuming that video-rate depth information is available, from either stereo cameras or other sensors. We use these depth measurements in the traditional linear brightness constraint equations, and we introduce a new depth constraint equation. As a result, estimation of certain types of motion, such as translation in depth and rotations out of the image plane, becomes more robust. We derive linear brightness and depth change constraint equations that govern the velocity field in 3-D for both perspective and orthographic camera projection models. These constraints are integrated jointly over image regions according to a rigid-body motion model, yielding a single linear system to robustly track 3D object pose. Results are shown for tracking the pose effaces in sequences of synthetic and real images.

We present an approach to real-time person tracking in crowded and/or unknown environments using multi-modal integration. We combine stereo, color and face detection modules into a single robust system, and show an initial application in an interactive, face-responsive display. Dense, real-time stereo processing is used to isolate users from other objects and people in the background. Skin-hue classification identifies and tracks likely body parts within the silhouette of a user. Face pattern detection discriminates and localizes the face within the identified body parts. Faces and bodies of users are tracked over several temporal scales: short-term (user stay's within the field of view), medium-term (user exits/reenters within minutes), and long term (user returns after hours or days). Short-term tracking is performed using simple region position and size correspondences, while medium and long-term tracking are based on statistics of user appearance. We discuss the failure modes of each individual module, describe our integration method, and report results with the complete system in trials with thousands of users.

The demo presents a virtual mirror interface which reacts to the viewer using robust, real-time face tracking. The display directly combines a user's face with various graphical distortions, performed only on the face region in the image. The face detection and tracking is done in real-time, so the graphical effect stays with the user and continues to adapt al; they move within the viewing space, increasing in intensity as the user approaches the display.The tracking system performs well in crowded environments with open and moving backgrounds. This robust performance is achieved using multi-modal integration, combining stereo, color, and grey-scale pattern matching modules into 2 single realtime system. Stereo processing is used to isolate the figure of a user from other objects and people in the background. Skinhue classification identifies and tracks likely body parts within the foreground region. Face pattern detection discriminates and localizes the face within the tracked body parts. A second set of displays, located to the side of the main viewing area, shows intermediate results of each processing module of the system.