We describe an efficient method to detect and track planar objects using a template of edge segments. Such segments are selected at multiple scales based on gradient magnitude; their positions and orientations are used to determine a canonical reference frame where the descriptor is computed based on quantized orientation. The resulting descriptors are efficiently matched using logical operations, and tracked between frames. The method yields pose estimates that are robust to scale changes, foreshortening, partial occlusions, and is suitable for use in augmented reality and human-computer interaction.

We describe a system to learn an object template from a video stream, and localize and track the corresponding object in live video. The template is decomposed into a number of local descriptors, thus enabling detection and tracking in spite of partial occlusion. Each local descriptor aggregates contrast invariant statistics (normalized intensity and gradient orientation) across scales, in a way that enables matching under significant scale variations. Low-level tracking during the training video sequence enables capturing object-specific variability due to the shape of the object, which is encapsulated in the descriptor. Salient locations on both the template and the target image are used as hypotheses to expedite matching.

We describe a method to select edgels and to calculate gradient orientation-based template descriptors for edgel features. An edgel is selected within a grid block based on gradient magnitude; its position and orientation are used to determine a canonical frame where the descriptor is computed based on quantized orientation. The resulting descriptor is efficiently matched using logical operations. We demonstrate the use of the resulting edgel detection and description method for planar object detection and pose estimation.

We demonstrate a visual recognition system operating on a hand-held device, with the help of an efficient and robust feature tracking and an object recognition mechanism that can be used for interactive mobile applications. In our recognition system, corner features are detected from captured video frames in a multi-scale image pyramid, and are tracked between consecutive frames efficiently. In order to perform object recognition, local descriptors are calculated on the tracked features, and quantized using a vocabulary tree. For each object, a bag-of-words model is learned from multiple views. The learned objects are recognized by computing the ranking score for the set of features in a single video frame. Our feature tracking algorithm and local descriptors are different than the Lucas-Kanade algorithm in image pyramid or the SIFT descriptor, however improving the efficiency and accuracy. For our implementation on a mobile phone, we used an iPhone 3GS with a 600MHz ARM chip CPU. The video frame is captured from a camera preview screen at a rate of 15 frames per second using the public API. The task of object recognition on a mobile phone runs at around 7 frames per second, including the feature tracking and descriptor calculation.

We describe a novel markerless camera tracking approach and user interaction methodology for augmented reality (AR) on unprepared tabletop environments. We propose a real-time system architecture that combines two types of feature tracking. Distinctive image features of the scene are detected and tracked frame-to-frame by computing optical flow. In order to achieve real-time performance, multiple operations are processed in a synchronized multi-threaded manner: capturing a video frame, tracking features using optical flow, detecting distinctive invariant features, and rendering an output frame. We also introduce user interaction methodology for establishing a global coordinate system and for placing virtual objects in the AR environment by tracking a user's outstretched hand and estimating a camera pose relative to it. We evaluate the speed and accuracy of our hybrid feature tracking approach, and demonstrate a proof-of-concept application for enabling AR in unprepared tabletop environments, using bare hands for interaction.

We describe a novel markerless camera tracking approach and user interaction methodology for augmented reality (AR) on unprepared tabletop environments. We propose a real-time system architecture that combines two types of feature tracking methods. Distinctive image features of the scene are detected and tracked frame- to-frame by computing optical flow. In order to achieve real-time performance, multiple operations are processed in a multi-threaded manner for capturing a video frame, tracking features using optical flow, detecting distinctive invariant features, and rendering an output frame. We also introduce a user interaction for establishing a global coordinate system and for locating virtual objects in the AR environment. A user's bare hand is used for the user interface by estimating a camera pose relative to the user's outstretched hand. We evaluate the speed and accuracy of our hybrid feature tracking approach, and demonstrate a proof-of-concept application for enabling AR in unprepared tabletop environments using hands for interaction.

We introduce a technique to establish a coordinate system for augmented reality (AR) on tabletop environments. A user's hand is tracked and the fingertips on the outstretched hand are detected, providing a camera pose estimation relative to the hand. As a user places the hand on the surface of a tabletop environment, the hand's coordinate system is propagated to the environment, detecting distinctive image features in the scene. The features are tracked fast and robustly using optical flow. In this way, a new tabletop AR environment is set up without having to carry a marker or a sophisticated tracking system to the environment itself. We also demonstrate a proof-of-concept application for establishing a tabletop AR environment and recognizing a scene when detecting its features.

We present a method for stabilizing live video from a moving camera for the purpose of a tele-meeting, in which a participant with an AR view onto a shared canvas collaborates with a remote user. The AR view is established without markers and using no other tracking equipment than a head-worn camera. The remote user is allowed to directly annotate the local user's view in real time on a desktop or tablet PC. The planar homographies between the reference frame and the other following frames are maintained. In effect, both the local and remote participants can annotate the physical meeting space, the local AR user through physical interaction, the remote user through our stabilized video. When tracking is lost, the remote user can still continue annotating on a frozen video frame. We tested several small demo applications with this new form of transient AR collaboration that can be established easily, on a per need basis, and without complicated equipment or calibration requirements.