Natural feature tracking systems for augmented reality are highly accurate, but can suffer from lost tracking. When registration is lost, the system must be able to re-localize and recover tracking. Likewise, when a camera is new to a scene, it must be able to perform the related task of localization. Localization and re-localization can only be performed at certain points or when viewing particular objects or parts of the scene with a sufficient number and quality of recognizable features to allow for tracking recovery. We explore how the density of such recovery locations/poses influences the time it takes users to resume tracking. We focus our evaluation on two generalized techniques for localization: keyframe-based and model-based. For the keyframe-based approach we assume a constant collection rate for keyframes. We find that at practical collection rates, the task of localization to a previously acquired keyframe that is shown to the user does not become more time-consuming as the interval between keyframes increases. For a localization approach using model data, we consider a grid of points around the model at which localization is guaranteed to succeed. We find that the user interface is crucial to successful localization. Localization can occur quickly if users do not need to orient themselves to marked localization points. When users are forced to mentally register themselves with a map of the scene, localization quickly becomes impractical as the distance to the next localization point increases. We contend that our results will help future designers of localization techniques to better plan for the effects of their proposed solutions.

The acquisition of surround-view panoramas using a single hand-held or head-worn camera relies on robust real-time camera orientation tracking and relocalization. This paper presents robust methodology and evaluation for camera orientation relocalization, using virtual keyframes for online environment map construction. In the case of tracking loss, incoming camera frames are matched against known-orientation keyframes to re-estimate camera orientation. Instead of solely using real keyframes from incoming video, the proposed approach employs virtual keyframes which are distributed strategically within completed portions of an environment map. To improve tracking speed, we introduce a new variant of our system which carries out relocalization only when tracking fails and uses inexpensive image-patch descriptors. We compare different system variants using three evaluation methods to show that the proposed system is useful in a practical sense. To improve relocalization robustness against lighting changes in indoor and outdoor environments, we propose a new approach based on illumination normalization and saturated area removal. We examine the performance of our solution over several indoor and outdoor video sequences, evaluating relocalization rates based on ground truth from a pan-tilt unit.

We present an analysis of four orientation tracking systems used for construction of environment maps. We discuss the analysis necessary to determine the robustness of tracking systems in general. Due to the difficulty inherent in collecting user evaluation data, we then propose a metric which can be used to obtain a relative estimate of these values. The proposed metric will still require a set of input videos with an associated distance to ground truth, but not an additional user evaluation.

The acquisition of surround-view panoramas using a single hand-held or head-worn camera relies on robust real-time camera orientation tracking. In absence of robust tracking recovery methods, the complete acquisition process has to be re-started when tracking fails. This paper presents methodology for camera orientation relocalization, using virtual keyframes for online environment map construction. Instead of relying on real keyframes from incoming video, the proposed approach enables camera orientation relocalization by employing virtual keyframes which are distributed strategically within an environment map. We discuss our insights about a suitable number and distribution of virtual keyframes, as suggested by our experiments on virtual keyframe generation and orientation relocalization. After a shading correction step, we relocalize camera orientation in real-time by comparing the current camera frame to virtual keyframes. While expanding the captured environment map, we continue to simultaneously generate virtual keyframes within the completed portion of the map, as descriptors to estimate camera orientation. We implemented our camera orientation relocalizer with the help of a GPU fragment shader for real-time application, and evaluated the speed and accuracy of the proposed approach.

This paper presents methodology for integrating a small, single-point laser range finder into a wearable augmented reality system. We first present a way of creating object-aligned annotations with very little user effort. Second, we describe techniques to segment and pop-up foreground objects. Finally, we introduce a method using the laser range finder to incrementally build 3D panoramas from a fixed observerpsilas location. To build a 3D panorama semi-automatically, we track the systempsilas orientation and use the sparse range data acquired as the user looks around in conjunction with real-time image processing to construct geometry around the userpsilas position. Using full 3D panoramic geometry, it is possible for new virtual objects to be placed in the scene with proper lighting and occlusion by real world objects, which increases the expressivity of the AR experience.