Monocular SLAM has the potential to turn inexpensive cameras into powerful pose sensors for applications such as robotics and augmented reality. We present a relocalization module for such systems which solves some of the problems encountered by previous monocular SLAM systems--tracking failure, map merging, and loop closure detection. This module extends recent advances in keypoint recognition to determine the camera pose relative to the landmarks within a single frame time of 33 ms. We first show how this module can be used to improve the robustness of these systems. Blur, sudden motion, and occlusion can all cause tracking to fail, leading to a corrupted map. Using the relocalization module, the system can automatically detect and recover from tracking failure while preserving map integrity. Extensive tests show that the system can then reliably generate maps for long sequences even in the presence of frequent tracking failure. We then show that the relocalization module can be used to recognize overlap in maps, i.e., when the camera has returned to a previously mapped area. Having established an overlap, we determine the relative pose of the maps using trajectory alignment so that independent maps can be merged and loop closure events can be recognized. The system combining all of these abilities is able to map larger environments and for significantly longer periods than previous systems.

Video see-through Augmented Reality adds computer graphics to the real world in real time by overlaying graphics onto a live video feed. To achieve a realistic integration of the virtual and real imagery, the rendered images should have a similar appearance and quality to those produced by the video camera. This paper describes a compositing method which models the artifacts produced by a small low-cost camera, and adds these effects to an ideal pinhole image produced by conventional rendering methods. We attempt to model and simulate each step of the imaging process, including distortions, chromatic aberrations, blur, Bayer masking, noise, sharpening, and color-space compression, all while requiring only an RGBA image and an estimate of camera velocity as inputs.

A successful approach in the recovery of video- rate structure from motion is to allow the camera to keep track of its position in every frame assuming the recovered set of scene landmarks is fixed in 3D, and then to use the poses in a subset of separated frames, or keyframes, to initialise further landmark structure. The landmark structure and keyframe poses are optimised in a bundle adjustment. Unfortunately this monolithic bundle adjustment has cubic complexity. This paper shows how representing landmarks and camera poses in relative frames, and by temporarily removing certain measurements, introduces a conditional indepedence which allows the bundle adjustment to be split into two parts. One "local" part involves the most recent keyframes and associated landmarks, and runs in constant time. The other "global" part deals with older keyframes and structure, and runs, as ever, in cubic time. Three important outcomes are: (i) the fixed- time local adjustment allows exploratory map-building to keep pace with camera pose tracking; (ii) it produces statistically consistent results; and (iii) referencing to relative frames means that any update in positions from the global adjustment are immediately incorporated in the local fixed-time adjustment. The relative frame approach is applied to the parallel tracking and mapping method for structure from motion, and its results shown to be identical, and the exploratory map building phase shown to maintain fixed time performance.

Camera phones are a promising platform for hand-held augmented reality. As their computational resources grow, they are becoming increasingly suitable for visual tracking tasks. At the same time, they still offer considerable challenges: Their cameras offer a narrow field-of-view not best suitable for robust tracking; images are often received at less than 15 Hz; long exposure times result in significant motion blur; and finally, a rolling shutter causes severe smearing effects. This paper describes an attempt to implement a keyframe-based SLAMsystem on a camera phone (specifically, the Apple iPhone 3 G). We describe a series of adaptations to the Parallel Tracking and Mapping system to mitigate the impact of the device's imaging deficiencies. Early results demonstrate a system capable of generating and augmenting small maps, albeit with reduced accuracy and robustness compared to SLAM on a PC.

This paper develops a Square Root Unscented Kalman Filter (SRUKF) for performing video-rate visual simultaneous localization and mapping (SLAM) using a single camera. The conventional UKF has been proposed previously for SLAM, improving the handling of nonlinearities compared with the more widely used Extended Kalman Filter (EKF). However, no account was taken of the comparative complexity of the algorithms: In SLAM, the UKF scales as O(N;{3}) in the state length, compared to the EKF's O(N;{2}), making it unsuitable for video-rate applications with other than unrealistically few scene points. Here, it is shown that the SRUKF provides the same results as the UKF to within machine accuracy and that it can be reposed with complexity O(N;{2}) for state estimation in visual SLAM. This paper presents results from video-rate experiments on live imagery. Trials using synthesized data show that the consistency of the SRUKF is routinely better than that of the EKF, but that its overall cost settles at an order of magnitude greater than the EKF for large scenes.

This paper introduces a square root unscented Kalman filter (SRUKF) solution to the problem of performing visual simultaneous localization and mapping (SLAM) using a single camera. Several authors have proposed the conventional UKF for SLAM to improve the handling of non-linearities compared with the more widely used EKF, but at the expense increasing computational complexity from O(N2) to O(N3) in the map size, making it unattractive for video-rate application. Van der Merwe and Wan's general SRUKF delivers identical results to a general UKF along with computational savings, but remains O(N3) overall. This paper shows how the SRUKF for the SLAM problem can be re-posed with O(N2) complexity, matching that of the EKF. The paper also shows how the method of inverse depth feature initialization developed by Montiel et al. for the EKF can be reformulated to work with the SRUKF. Experimental results confirm that the SRUKF and the UKF produce identical estimates, and that the SRUKF is more consistent than the EKF. Although the complexity is the same, the SRUKF remains more expensive to compute.

To achieve a realistic integration of virtual and real imagery in video see-through augmented reality, the rendered images should have a similar appearance and quality to those captured by the video camera. This paper describes a compositing method which models the artefacts produced by a small low-cost camera, and adds these effects to an ideal pinhole image produced by conventional rendering methods. We attempt to model and simulate each step of the imaging process, including distortions, chromatic aberrations, blur, Bayer masking, noise and colour-space compression, all while requiring only an RGBA image and an estimate of camera velocity as inputs.

Using simultaneous localization and mapping to determine the 3D surroundings and pose of a wearable or hand-held camera provides the geometrical foundation for several capabilities of value to an autonomous wearable vision system. The one explored here is the ability to incorporate recognized objects into the map of the surroundings and refer to them. Established methods for feature cluster recognition are used to identify and localize known planar objects, and their geometry is incorporated into the map of the surrounds using a minimalist representation. Continued measurement of these mapped objects improves both the accuracy of estimated maps and the robustness of the tracking system. In the context of wearable (or hand-held) vision, the system’s ability to enhance generated maps with known objects increases the map’s value to human operators, and also enables meaningful automatic annotation of the user’s surroundings. 1

Monocular SLAM has the potential to turn inexpensive cameras into powerful pose sensors for applications such as robotics and augmented reality. However, current implementations lack the robustness required to be useful outside laboratory conditions: blur, sudden motion and occlusion all cause tracking to fail and corrupt the map. Here we present a system which automatically detects and recovers from tracking failure while preserving map integrity. By extending recent advances in keypoint recognition the system can quickly resume tracking - i.e. within a single frame time of 33 ms - using any of the features previously stored in the map. Extensive tests show that the system can reliably generate maps for long sequences even in the presence of frequent tracking failure.

This paper presents a system which combines single-camera SLAM (simultaneous localization and mapping) with established methods for feature recognition. Besides using standard salient image features to build an on-line map of the camera's environment, this system is capable of identifying and localizing known planar objects in the scene, and incorporating their geometry into the world map. Continued measurement of these mapped objects improves both the accuracy of estimated maps and the robustness of the tracking system. In the context of hand-held or wearable vision, the system's ability to enhance generated maps with known objects increases the map's value to human operators, and also enables meaningful automatic annotation of the user's surroundings. The presented solution lies between the high order enriching of maps such as scene classification, and the efforts to introduce higher geometric primitives such as lines into probabilistic maps.

This paper presents a method of estimating camera pose in an unknown scene. While this has previously been attempted by adapting SLAM algorithms developed for robotic exploration, we propose a system specifically designed to track a hand-held camera in a small AR workspace. We propose to split tracking and mapping into two separate tasks, processed in parallel threads on a dual-core computer: one thread deals with the task of robustly tracking erratic hand-held motion, while the other produces a 3D map of point features from previously observed video frames. This allows the use of computationally expensive batch optimisation techniques not usually associated with real-time operation: The result is a system that produces detailed maps with thousands of landmarks which can be tracked at frame-rate, with an accuracy and robustness rivalling that of state-of-the-art model-based systems.

This paper demonstrates a real-time, full-3D edge tracker based on a particle filter. In contrast to previous methods this system is capable of tracking complex self-occluding three-dimensional structures. The system exploits graphics hardware in a novel manner, allowing it not only to perform hidden line removal for each particle but also to evaluate pose likelihoods directly on the graphics card. This approach allows video-rate filtering with hundreds of particles on a standard workstation. 1