This paper describes an augmented reality system that incorporates a real-time dense stereo vision system. Analysis of range and intensity data is used to perform two functions: 1) 3D detection and tracking of the user's fingertip or a pen to provide natural 3D pointing gestures, and 2) computation of the 3D position and orientation of the user's viewpoint without the need for fiducial mark calibration procedures, or manual initialization. The paper describes the stereo depth camera, the algorithms developed for pointer tracking and camera pose tracking, and demonstrates their use within an application in the field of oil and gas exploration.

A new algorithm for background estimation and removal in video sequences obtained with stereo cameras is presented. Per-pixel Gaussian mixtures are used to model recent scene observations in the combined space of depth and luminance-invariant color. These mixture models adapt over time, and are used to build a new model of the background at each time step. This combination in itself is novel, but we also introduce the idea of modulating the learning rate of the background model according to the scene activity level on a per-pixel basis, so that dynamic foreground objects are incorporated into the background more slowly than are static scene changes. Our results show much greater robustness than prior state-of-the-art methods to challenging phenomena such as video displays, non-static background objects, areas of high foreground traffic, and similar color of foreground and background. Our method is also well-suited for use in real-time systems.

Background estimation and removal based on the joint use of range and color data produces superior results than can be achieved with either data source alone. This is increasingly relevant as inexpensive, real-time, passive range systems become more accessible through novel hardware and increased CPU processing speeds. Range is a powerful signal for segmentation which is largely independent of color and hence not effected by the classic color segmentation problems of shadows and objects with color similar to the background. However range alone is also not sufficient for the good segmentation: depth measurements are rarely available at all pixels in the scene, and foreground objects may be indistinguishable in depth when they are close to the background. Color segmentation is complementary in these cases. Surprisingly, little work has been done to date on joint range and color segmentation. We describe and demonstrate a background estimation method based on a multidimensional (range and color) clustering at each image pixel. Segmentation of the foreground in a given frame is performed via comparison with background statistics in range and normalized color. Important implementation issues such as treatment of shadows and low confidence measurements are discussed in detail.

This paper explores the direct motion estimation problem assuming that video-rate depth information is available, from either stereo cameras or other sensors. We use these depth measurements in the traditional linear brightness constraint equations, and we introduce a new depth constraint equation. As a result, estimation of certain types of motion, such as translation in depth and rotations out of the image plane, becomes more robust. We derive linear brightness and depth change constraint equations that govern the velocity field in 3-D for both perspective and orthographic camera projection models. These constraints are integrated jointly over image regions according to a rigid-body motion model, yielding a single linear system to robustly track 3D object pose. Results are shown for tracking the pose effaces in sequences of synthetic and real images.

We present an approach to real-time person tracking in crowded and/or unknown environments using multi-modal integration. We combine stereo, color and face detection modules into a single robust system, and show an initial application in an interactive, face-responsive display. Dense, real-time stereo processing is used to isolate users from other objects and people in the background. Skin-hue classification identifies and tracks likely body parts within the silhouette of a user. Face pattern detection discriminates and localizes the face within the identified body parts. Faces and bodies of users are tracked over several temporal scales: short-term (user stay's within the field of view), medium-term (user exits/reenters within minutes), and long term (user returns after hours or days). Short-term tracking is performed using simple region position and size correspondences, while medium and long-term tracking are based on statistics of user appearance. We discuss the failure modes of each individual module, describe our integration method, and report results with the complete system in trials with thousands of users.

The demo presents a virtual mirror interface which reacts to the viewer using robust, real-time face tracking. The display directly combines a user's face with various graphical distortions, performed only on the face region in the image. The face detection and tracking is done in real-time, so the graphical effect stays with the user and continues to adapt al; they move within the viewing space, increasing in intensity as the user approaches the display.The tracking system performs well in crowded environments with open and moving backgrounds. This robust performance is achieved using multi-modal integration, combining stereo, color, and grey-scale pattern matching modules into 2 single realtime system. Stereo processing is used to isolate the figure of a user from other objects and people in the background. Skinhue classification identifies and tracks likely body parts within the foreground region. Face pattern detection discriminates and localizes the face within the tracked body parts. A second set of displays, located to the side of the main viewing area, shows intermediate results of each processing module of the system.

Face recognition from a representation based on features extracted from range images is explored. Depth and curvature features have several advantages over more traditional intensity-based features. Specifically, curvature descriptors have the potential for higher accuracy in describing surface-based events, are better suited to describe properties of the face in areas such as the cheeks, forehead, and chin, and are viewpoint invariant. Faces are represented in terms of a vector of feature descriptors. Comparisons between two faces is made based on their relationship in the feature space. The author provides a detailed analysis of the accuracy and discrimination of the particular features extracted, and the effectiveness of the recognition system for a test database of 24 faces. Recognition rates are in the range of 80% to 100%. In many cases, feature accuracy is limited more by surface resolution than by the extraction process.>