Without knowledge of the absolute baseline between images, the scale of a map from a single-camera simultaneous localization and mapping system is subject to calamitous drift over time. We describe a monocular approach that in addition to point measurements also considers object detections to resolve this scale ambiguity and drift. By placing an expectation on the size of the objects, the scale estimation can be seamlessly integrated into a bundle adjustment. When object observations are available, the local scale of the map is then determined jointly with the camera pose in local adjustments. Unlike many previous visual odometry methods, our approach does not impose restrictions such as constant camera height or planar roadways, and is therefore more widely applicable. We evaluate our approach on the KITTI data set and show that it reduces scale drift over long-range outdoor sequences with a total length of 40 km. As the scale of objects is known absolutely, metric accuracy is obtained for all sequences. Qualitative evaluation is also performed on video footage from a hand-held camera.

Reconstructing dense, volumetric models of real-world 3D scenes is important for many tasks, but capturing large scenes can take significant time, and the risk of transient changes to the scene goes up as the capture time increases. These are good reasons to want instead to capture several smaller sub-scenes that can be joined to make the whole scene. Achieving this has traditionally been difficult: joining sub-scenes that may never have been viewed from the same angle requires a high-quality camera relocaliser that can cope with novel poses, and tracking drift in each sub-scene can prevent them from being joined to make a consistent overall scene. Recent advances, however, have significantly improved our ability to capture medium-sized sub-scenes with little to no tracking drift: real-time globally consistent reconstruction systems can close loops and re-integrate the scene surface on the fly, whilst new visual-inertial odometry approaches can significantly reduce tracking drift during live reconstruction. Moreover, high-quality regression forest-based relocalisers have recently been made more practical by the introduction of a method to allow them to be trained and used online. In this paper, we leverage these advances to present what to our knowledge is the first system to allow multiple users to collaborate interactively to reconstruct dense, voxel-based models of whole buildings using only consumer-grade hardware, a task that has traditionally been both time-consuming and dependent on the availability of specialised hardware. Using our system, an entire house or lab can be reconstructed in under half an hour and at a far lower cost than was previously possible.

Monocular visual localization and mapping algorithms are able to estimate the environment only up to scale, a degree of freedom which leads to scale drift, difficulty closing loops, and eventual failure. This paper describes an image-driven approach for scale-drift correction which uses a convolutional neural network to infer the speed of the camera from successive monocular video frames. We obtain continuous drift correction, avoiding the need for explicit higher-level representations of the map to resolve scale. We also propose a novel method of including speed estimates as a regularizer in bundle adjustment which avoids the pitfalls of sudden imposition of scale knowledge. We demonstrate our approach using long-distance sequences for which ground truth is available, and find output that is essentially free of scale drift. We compare the performance with number of other methods for scale-drift correction from monocular data, and show that our solution achieves more accurate results.

We describe a novel probabilistic framework for real-time tracking of multiple objects from combined depth-colour imagery. Object shape is represented implicitly using 3D signed distance functions. Probabilistic generative models based on these functions are developed to account for the observed RGB-D imagery, and tracking is posed as a maximum a posteriori problem. We present first a method suited to tracking a single rigid 3D object, and then generalise this to multiple objects by combining distance functions into a shape union in the frame of the camera. This second model accounts for similarity and proximity between objects, and leads to robust real-time tracking without recourse to bolt-on or ad-hoc collision detection.

Without knowledge of the absolute baseline between images, the scale of a map from single-camera simultaneous localization and mapping system is subject to calamitous drift over time. We describe a monocular approach that in addition to point measurements also considers object detections to resolve this scale ambiguity and drift. By placing a prior on the size of the objects, the scale estimation can be seamlessly integrated into a bundle adjustment. When object observations are available, the local scale of the map is then determined jointly with the camera pose in local adjustments. Unlike many previous visual odometry methods, our approach does not impose restrictions such as approximately constant camera height or planar roadways, and is therefore applicable to a much wider range of applications. We evaluate our approach on the KITTI dataset and show that it reduces scale drift over long-range outdoor sequences with a total length of 40 km. Qualitative evaluation is also performed on video footage from a hand-held camera.

Our abilities in scene understanding, which allow us to perceive the 3D structure of our surroundings and intuitively recognise the objects we see, are things that we largely take for granted, but for robots, the task of understanding large scenes quickly remains extremely challenging. Recently, scene understanding approaches based on 3D reconstruction and semantic segmentation have become popular, but existing methods either do not scale, fail outdoors, provide only sparse reconstructions or are rather slow. In this paper, we build on a recent hash-based technique for large-scale fusion and an efficient mean-field inference algorithm for densely-connected CRFs to present what to our knowledge is the first system that can perform dense, large-scale, outdoor semantic reconstruction of a scene in (near) real time. We also present a `semantic fusion' approach that allows us to handle dynamic objects more effectively than previous approaches. We demonstrate the effectiveness of our approach on the KITTI dataset, and provide qualitative and quantitative results showing high-quality dense reconstruction and labelling of a number of scenes.

Volumetric methods provide efficient, flexible and simple ways of integrating multiple depth images into a full 3D model. They provide dense and photorealistic 3D reconstructions, and parallelised implementations on GPUs achieve real-time performance on modern graphics hardware. To run such methods on mobile devices, providing users with freedom of movement and instantaneous reconstruction feedback, remains challenging however. In this paper we present a range of modifications to existing volumetric integration methods based on voxel block hashing, considerably improving their performance and making them applicable to tablet computer applications. We present (i) optimisations for the basic data structure, and its allocation and integration; (ii) a highly optimised raycasting pipeline; and (iii) extensions to the camera tracker to incorporate IMU data. In total, our system thus achieves frame rates up 47 Hz on a Nvidia Shield Tablet and 910 Hz on a Nvidia GTX Titan XGPU, or even beyond 1.1 kHz without visualisation.

We present a novel framework for jointly tracking a camera in 3D and reconstructing the 3D model of an observed object. Due to the region based approach, our formulation can handle untextured objects, partial occlusions, motion blur, dynamic backgrounds and imperfect lighting. Our formulation also allows for a very efficient implementation which achieves real-time performance on a mobile phone, by running the pose estimation and the shape optimisation in parallel. We use a level set based pose estimation but completely avoid the, typically required, explicit computation of a global distance. This leads to tracking rates of more than 100 Hz on a desktop PC and 30 Hz on a mobile phone. Further, we incorporate additional orientation information from the phone's inertial sensor which helps us resolve the tracking ambiguities inherent to region based formulations. The reconstruction step first probabilistically integrates 2D image statistics from selected keyframes into a 3D volume, and then imposes coherency and compactness using a total variational regularisation term. The global optimum of the overall energy function is found using a continuous max-flow algorithm and we show that, similar to tracking, the integration of per voxel posteriors instead of likelihoods improves the precision and accuracy of the reconstruction.

Most current approaches for 3D object tracking rely on distinctive object appearances. While several such trackers can be instantiated to track multiple objects independently, this not only neglects that objects should not occupy the same space in 3D, but also fails when objects have highly similar or identical appearances. In this paper we develop a probabilistic graphical model that accounts for similarity and proximity and leads to robust real-time tracking of multiple objects from RGB-D data, without recourse to bolton collision detection.

We introduce a probabilistic framework for simultaneous tracking and reconstruction of 3D rigid objects using an RGB-D camera. The tracking problem is handled using a bag-of-pixels representation and a back-projection scheme. Surface and background appearance models are learned online, leading to robust tracking in the presence of heavy occlusion and outliers. In both our tracking and reconstruction modules, the 3D object is implicitly embedded using a 3D level-set function. The framework is initialized with a simple shape primitive model (e.g. a sphere or a cube), and the real 3D object shape is tracked and reconstructed online. Unlike existing depth-based 3D reconstruction works, which either rely on calibrated/fixed camera set up or use the observed world map to track the depth camera, our framework can simultaneously track and reconstruct small moving objects. We use both qualitative and quantitative results to demonstrate the superior performance of both tracking and reconstruction of our method.

A novel framework for joint monocular 3D tracking and reconstruction is described that can handle untextured objects, occlusions, motion blur, changing background and imperfect lighting, and that can run at frame rate on a mobile phone. The method runs in parallel (i) level set based pose estimation and (ii) continuous max flow based shape optimisation. By avoiding a global computation of distance transforms typically used in level set methods, tracking rates here exceed 100Hz and 20Hz on a desktop and mobile phone, respectively, without needing a GPU. Tracking ambiguities are reduced by augmenting orientation information from the phone's inertial sensor. Reconstruction involves probabilistic integration of the 2D image statistics from keyframes into a 3D volume. Per-voxel posteriors are used instead of the standard likelihoods, giving increased accuracy and robustness. Shape coherency and compactness is then imposed using a total variational approach solved using globally optimal continuous max flow.

The recovery of structure from motion in real time over extended areas demands methods that mitigate the effects of computational complexity and arithmetical inconsistency. In this paper, we develop SCISM, an algorithm based on relative frame bundle adjustment, which splits the recovered map of 3D landmarks and keyframes poses so that the camera can continue to grow and explore a local map in real time while, at the same time, a bulk map is optimized in the background. By temporarily excluding certain measurements, it ensures that both maps are consistent, and by using the relative frame representation, new results from the bulk process can update the local process without disturbance. The paper first shows how to apply this representation to the parallel tracking and mapping (PTAM) method, a real-time bundle adjuster, and compares results obtained using global and relative frames. It then explains the relative representation's use in SCISM and describes an implementation using PTAM. The paper provides evidence of the algorithm's real-time operation in outdoor scenes, and includes comparison with a more conventional submapping approach.

This paper addresses scene understanding in the context of a moving camera, integrating semantic reasoning ideas from monocular vision with 3D information available through structure-from-motion. We combine geometric and photometric cues in a Bayesian framework, building on recent successes leveraging the indoor Manhattan assumption in monocular vision. We focus on indoor environments and show how to extract key boundaries while ignoring clutter and decorations. To achieve this we present a graphical model that relates photometric cues learned from labeled data, stereo photo-consistency across multiple views, and depth cues derived from structure-from-motion point clouds. We show how to solve MAP inference using dynamic programming, allowing exact, global inference in ~100 ms (in addition to feature computation of under one second) without using specialized hardware. Experiments show our system out-performing the state-of-the-art.

Three methods are explored which help indicate whether feature points are potentially visible or occluded in the matching phase of the keyframe-based real-time visual SLAM system. The first derives a measure of potential visibility from the angular proximity to keyframes in which they were observed and globally adjusted, and preferentially selects those with high visibility when tracking the camera position between keyframes. It is found that sorting and selecting features within image bins spread over the image improves tracking stability. The second method automatically recognizes and locates 3D polyhedral objects alongside the point map, and uses them to determine occlusion. The third method uses the map points themselves to grow surfaces. The performance of each is tested on live and recorded sequences.

Though modern Visual Simultaneous Localisation and Mapping (vSLAM) systems are capable of localising robustly and efficiently even in the case of a monocular camera, the maps produced are typically sparse point-clouds that are difficult to interpret and of little use for higher-level reasoning tasks such as scene understanding or human- machine interaction. In this paper we begin to address this deficiency, presenting progress on expanding the competency of visual SLAM systems to build richer maps. Specifically, we concentrate on modelling indoor scenes using semantically meaningful surfaces and accompanying labels, such as “floor”, “wall”, and “ceiling” - an important step towards a representation that can support higher-level reasoning and planning. We leverage the Manhattan world assumption and show how to extract vanishing directions jointly across a video stream. We then propose a guided line detector that utilises known vanishing points to extract extremely subtle axis- aligned edges. We utilise recent advances in single view structure recovery to building geometric scene models and demonstrate our system operating on-line.

Video see-through Augmented Reality adds computer graphics to the real world in real time by overlaying graphics onto a live video feed. To achieve a realistic integration of the virtual and real imagery, the rendered images should have a similar appearance and quality to those produced by the video camera. This paper describes a compositing method which models the artifacts produced by a small low-cost camera, and adds these effects to an ideal pinhole image produced by conventional rendering methods. We attempt to model and simulate each step of the imaging process, including distortions, chromatic aberrations, blur, Bayer masking, noise, sharpening, and color-space compression, all while requiring only an RGBA image and an estimate of camera velocity as inputs.

A successful approach in the recovery of video- rate structure from motion is to allow the camera to keep track of its position in every frame assuming the recovered set of scene landmarks is fixed in 3D, and then to use the poses in a subset of separated frames, or keyframes, to initialise further landmark structure. The landmark structure and keyframe poses are optimised in a bundle adjustment. Unfortunately this monolithic bundle adjustment has cubic complexity. This paper shows how representing landmarks and camera poses in relative frames, and by temporarily removing certain measurements, introduces a conditional indepedence which allows the bundle adjustment to be split into two parts. One "local" part involves the most recent keyframes and associated landmarks, and runs in constant time. The other "global" part deals with older keyframes and structure, and runs, as ever, in cubic time. Three important outcomes are: (i) the fixed- time local adjustment allows exploratory map-building to keep pace with camera pose tracking; (ii) it produces statistically consistent results; and (iii) referencing to relative frames means that any update in positions from the global adjustment are immediately incorporated in the local fixed-time adjustment. The relative frame approach is applied to the parallel tracking and mapping method for structure from motion, and its results shown to be identical, and the exploratory map building phase shown to maintain fixed time performance.

To work at video rate, the maps that monocular SLAM builds are bound to be sparse, making them sensitive to the erroneous inclusion of moving points and to the deletion of valid points through temporary occlusion. This paper describes the parallel implementation of monoSLAM with a 3D object tracker, allowing reasoning about moving objects and occlusion. The SLAM process provides the object tracker with information to register objects to the map's frame, and the object tracker allows the marking of features, either those on objects, or those created by their occluding edges, or those occluded by objects. Experiments are presented to verify the recovered geometry and to indicate the impact on camera pose in monoSLAM of including and avoiding moving features.

This paper demonstrates how objects can be recognized, reconstructed, and localized within a 3D map, using observations and matching of SIFT features in keyframes. The keyframes arise as part of a frame-rate process of parallel camera tracking and mapping, in which the keyframe camera poses and 3D map points are refined using bundle adjustment. The object reconstruction process runs independently, and in parallel to, the tracking and mapping processes. Detected objects are automatically labelled on the user's display using predefined annotations. The annotations are also used to highlight areas of interest upon the objects to the user.

Camera phones are a promising platform for hand-held augmented reality. As their computational resources grow, they are becoming increasingly suitable for visual tracking tasks. At the same time, they still offer considerable challenges: Their cameras offer a narrow field-of-view not best suitable for robust tracking; images are often received at less than 15 Hz; long exposure times result in significant motion blur; and finally, a rolling shutter causes severe smearing effects. This paper describes an attempt to implement a keyframe-based SLAMsystem on a camera phone (specifically, the Apple iPhone 3 G). We describe a series of adaptations to the Parallel Tracking and Mapping system to mitigate the impact of the device's imaging deficiencies. Early results demonstrate a system capable of generating and augmenting small maps, albeit with reduced accuracy and robustness compared to SLAM on a PC.

This paper develops a Square Root Unscented Kalman Filter (SRUKF) for performing video-rate visual simultaneous localization and mapping (SLAM) using a single camera. The conventional UKF has been proposed previously for SLAM, improving the handling of nonlinearities compared with the more widely used Extended Kalman Filter (EKF). However, no account was taken of the comparative complexity of the algorithms: In SLAM, the UKF scales as O(N;{3}) in the state length, compared to the EKF's O(N;{2}), making it unsuitable for video-rate applications with other than unrealistically few scene points. Here, it is shown that the SRUKF provides the same results as the UKF to within machine accuracy and that it can be reposed with complexity O(N;{2}) for state estimation in visual SLAM. This paper presents results from video-rate experiments on live imagery. Trials using synthesized data show that the consistency of the SRUKF is routinely better than that of the EKF, but that its overall cost settles at an order of magnitude greater than the EKF for large scenes.

This paper discusses two of the most important design considerations for a wearable device with visual sensing: what kind of sensor to use and where to place it. While nature and computer vision have explored a wide range of imaging techniques, wearables have mostly viewed the world through conventional narrow-view passive cameras designed for nonwearable applications, which are attached to the wearer's head. The rationale presented here for sensor selection and the novel methodology developed for objectively studying sensor placement have informed the development of a number of visual wearables.

This paper introduces a square root unscented Kalman filter (SRUKF) solution to the problem of performing visual simultaneous localization and mapping (SLAM) using a single camera. Several authors have proposed the conventional UKF for SLAM to improve the handling of non-linearities compared with the more widely used EKF, but at the expense increasing computational complexity from O(N2) to O(N3) in the map size, making it unattractive for video-rate application. Van der Merwe and Wan's general SRUKF delivers identical results to a general UKF along with computational savings, but remains O(N3) overall. This paper shows how the SRUKF for the SLAM problem can be re-posed with O(N2) complexity, matching that of the EKF. The paper also shows how the method of inverse depth feature initialization developed by Montiel et al. for the EKF can be reformulated to work with the SRUKF. Experimental results confirm that the SRUKF and the UKF produce identical estimates, and that the SRUKF is more consistent than the EKF. Although the complexity is the same, the SRUKF remains more expensive to compute.

To achieve a realistic integration of virtual and real imagery in video see-through augmented reality, the rendered images should have a similar appearance and quality to those captured by the video camera. This paper describes a compositing method which models the artefacts produced by a small low-cost camera, and adds these effects to an ideal pinhole image produced by conventional rendering methods. We attempt to model and simulate each step of the imaging process, including distortions, chromatic aberrations, blur, Bayer masking, noise and colour-space compression, all while requiring only an RGBA image and an estimate of camera velocity as inputs.

Using simultaneous localization and mapping to determine the 3D surroundings and pose of a wearable or hand-held camera provides the geometrical foundation for several capabilities of value to an autonomous wearable vision system. The one explored here is the ability to incorporate recognized objects into the map of the surroundings and refer to them. Established methods for feature cluster recognition are used to identify and localize known planar objects, and their geometry is incorporated into the map of the surrounds using a minimalist representation. Continued measurement of these mapped objects improves both the accuracy of estimated maps and the robustness of the tracking system. In the context of wearable (or hand-held) vision, the system’s ability to enhance generated maps with known objects increases the map’s value to human operators, and also enables meaningful automatic annotation of the user’s surroundings. 1

This paper presents a system which combines single-camera SLAM (simultaneous localization and mapping) with established methods for feature recognition. Besides using standard salient image features to build an on-line map of the camera's environment, this system is capable of identifying and localizing known planar objects in the scene, and incorporating their geometry into the world map. Continued measurement of these mapped objects improves both the accuracy of estimated maps and the robustness of the tracking system. In the context of hand-held or wearable vision, the system's ability to enhance generated maps with known objects increases the map's value to human operators, and also enables meaningful automatic annotation of the user's surroundings. The presented solution lies between the high order enriching of maps such as scene classification, and the efforts to introduce higher geometric primitives such as lines into probabilistic maps.

This paper presents a method of estimating camera pose in an unknown scene. While this has previously been attempted by adapting SLAM algorithms developed for robotic exploration, we propose a system specifically designed to track a hand-held camera in a small AR workspace. We propose to split tracking and mapping into two separate tasks, processed in parallel threads on a dual-core computer: one thread deals with the task of robustly tracking erratic hand-held motion, while the other produces a 3D map of point features from previously observed video frames. This allows the use of computationally expensive batch optimisation techniques not usually associated with real-time operation: The result is a system that produces detailed maps with thousands of landmarks which can be tracked at frame-rate, with an accuracy and robustness rivalling that of state-of-the-art model-based systems.

This paper demonstrates a real-time, full-3D edge tracker based on a particle filter. In contrast to previous methods this system is capable of tracking complex self-occluding three-dimensional structures. The system exploits graphics hardware in a novel manner, allowing it not only to perform hidden line removal for each particle but also to evaluate pose likelihoods directly on the graphics card. This approach allows video-rate filtering with hundreds of particles on a standard workstation. 1

The RVM-based learning method for whole body pose estimation proposed by Agarwal and Triggs is adapted to hand pose recovery. To help overcome the difficulties presented by the greater degree of self-occlusion and the wider range of poses exhibited in hand imagery, the adaptation proposes a method for combining multiple views. Comparisons of performance using single versus multiple views are reported for both synthesized and real imagery, and the effects of the number of image measurements and the number of training samples on performance are explored.

In this paper we consider a single hand-held camera performing SLAM at video rate with generic 6DOF motion. The aim is to optimise both the localisation of the sensor and building of the feature map by computing the most appropriate control actions or movements. The actions belong to a discrete set (e.g. go forward, go left, go up, turn right, etc), and are chosen so as to maximise the mutual information gain between posterior states and measurements. Maximising the mutual information helps the camera avoid making ill-conditioned measurements appropriate to bearing-only SLAM. Moreover, orientation changes are determined by maximising the trace of the Fisher information matrix. In this way, we allow the camera to continue looking at those landmarks with large uncertainty, but from better-posed directions. Various position and gaze control strategies are first tested in a simulated environment, and then validated in a video-rate implementation. Given that our system is capable of producing motion commands for a real-time 6DOF visual SLAM, it could be used with any type of mobile platform, without the need of other sensors

We contrast the performance of two methods of imposing constraints during the tracking of articulated objects, the first method preimposing the kinematic constraints during tracking and, thus, using the minimum degrees of freedom, and the second imposing constraints after tracking and, hence, using the maximum. Despite their very different formulations, the methods recover the same pose change. Further comparisons are drawn in terms of computational speed and algorithmic simplicity and robustness, and it is the last area which is the most telling. The results suggest that using built-in constraints is well-suited to tracking individual articulated objects, whereas applying constraints afterward is most suited to problems involving contact and breakage between articulated (or rigid) objects, where the ability to test tracking performance quickly with constraints turned on or off is desirable.

MLESAC is an established algorithm for maximum-likelihood estimation by random sampling consensus, devised for computing multiview entities like the fundamental matrix from correspondences between image features. A shortcoming of the method is that it assumes that little is known about the prior probabilities of the validities of the correspondences. This paper explains the consequences of that omission and describes how the algorithm's theoretical standing and practical performance can be enhanced by deriving estimates of these prior probabilities. Using the priors in guided-MLESAC is found to give an order of magnitude speed increase for problems where the correspondences are described by one image transformation and clutter. This paper describes two further modifications to guided-MLESAC. The first shows how all putative matches, ratherthan just the best, from a particularfeature can be taken forward into the sampling stage, albeit at the expense of additional computation. The second suggests how to propagate the output from one frame forward to successive frames. The additional information makes guided-MLESAC computationally realistic at video-rates for correspondence sets modeled by two transformations and clutter.

This paper is concerned with allowing the user of a wearable, portable, vision system to interact with the visual information using hand movements and gestures. Two example scenarios are explored. The first, in 2D, uses the wearer’s hand to both guide an active wearable camera and to highlight objects of interest using a grasping vector. The second is based in 3D, and builds on earlier work which recovers 3D scene structure at video-rate, allowing real-time purposive redirection of the camera to any scene point. Here, a range of hand gestures are used to highlight and select 3D points within the structure and in this instance used to insert 3D graphical objects into the scene. Structure recovery, gesture recognition, scene annotation and augmentation are achieved in parallel and at video-rate.

During tracking, lens zoom acts as a gain between scene dynamics and fixation errors, providing a trade-off between maximizing resolution and minimizing tracking error. Using a linear Kalman filter model, it is shown that when image measurement error scales with focal length, the filter is invariant to zoom. When the error is of fixed size, however, zooming alters the balance between process and measurement errors in a counter-intuitive manner. It is shown that this balance can be restored by appropriate adjustment of the process noise. With a zoom invariant filter, zoom can be used to ensure that fixation errors remain bounded. To this end, an error variance control method is proposed which gives high confidence that the target will not leave the image during tracking. To implement such a scheme, equipment delays and responses must be known, including those of the zoom lenses. Experiments to measure these are described, and overall results are presented for 30 Hz tracking of real scenes.

We present a general method for real-time, vision-only single-camera simultaneous localisation and mapping (SLAM) - an algorithm which is applicable to the localisation of any camera moving through a scene - and study its application to the localisation of a wearable robot with active vision. Starting from very sparse initial scene knowledge, a map of natural point features spanning a section of a room is generated on-the-fly as the motion of the camera is simultaneously estimated in full 3D. Naturally this permits the annotation of the scene with rigidly-registered graphics, but further it permits automatic control of the robot's active camera: for instance, fixation on a particular object can be maintained during extended periods of arbitrary user motion, then shifted at will to another object which has potentially been out of the field of view. This kind of functionality is the key to the understanding or "management" of a workspace which the robot needs to have in order to assist its wearer usefully in tasks. We believe that the techniques and technology developed are of particular immediate value in scenarios of remote collaboration, where a remote expert is able to annotate, through the robot, the environment the wearer is working in.

This demo showcases breakthrough results in the general field real-time simultaneous localization and mapping (SLAM) using vision and in particular its vital role in enabling a wearable robot to assists its user. In our approach, a wearable active vision system ("wearable robot") is mounted at the shoulder. As the wearer moves around his environment, typically browsing a workspace in which a task must be completed, the robot acquires images continuously and generates a map of natural visual features on-the-fly while estimating its ego-motion.

Algorithms for self-calibrating cameras whose changes in calibration parameters are confined to rotation and zooming are useful since many real-world imaging situations do not permit translations-consider, for instance, cameras mounted on tripods and desk or wall-mounted active heads. In practice, however, the assumption of pure rotation is often violated because the optic center of the camera and the rotation center do not completely coincide. This work determines how such misalignments affect the estimation of the camera focal length. Expressions for the errors in focal length and recovered rotations are derived and results are confirmed with experiments on synthetic data. We show that the approximation of pure rotation is indeed sufficient in many cases, especially since other sources of error, such as noise and particularly radial distortion, tend to be more detrimental.

Recent advances in wearable sensing allow active control of the orientation of a body-mounted camera worn by a remote user. In this paper we consider the control of the active camera from head movements. In the context of teleoperation, these may be the head movements of a remote operator, perhaps acting as the wearer’s assistant. The movements are likely to be larger than those in video-conference applications, and so frontal facial features are insufficient. The paper presents a model which incrementally combines a fixed 3D shape model with specific features found on the observed head. Robust methods, including the incorporation of a colour model, are used to mitigate the effect of mismatching, the main contribution being the use of both interest point and colour features within a single random-sampling framework.

We report on two methods we have developed to aid in the design of a wearable visual robot-a body mounted robot for which the main sensor is a camera. Specifically, we have first refined the analysis of sensor placement through the computation of the field of view and body motion using a 3D model of the human form. Second we have improved the design of the robot's morphology with the help of an optimization algorithm based on the Pareto front, within constraints set by the overall choice of robot kinematic chain and the need to specify obtainable actuators and sensors. The methods could be of use for the design and performance evaluation of rather different kinds of wearable robots and devices.

An active approach to sensing can provide the focused measurement capability over a wide field of view which allows correctly formulated simultaneous localization and map-building (SLAM) to be implemented with vision, permitting repeatable longterm localization using only naturally occurring, automatically-detected features. In this paper, we present the first example of a general system for autonomous localization using active vision, enabled here by a high-performance stereo head, addressing such issues as uncertainty-based measurement selection, automatic map-maintenance, and goal-directed steering. We present varied real-time experiments in a complex environment.

This paper describes a method for visual control of the zoom setting of an active camera during tracking. The method assumes an affine projection, and tracking is achieved using affine transfer, a process which is fundamentally invariant to zoom. However, the form of the projection matrices determined to achieve transfer allows the relative scale between the affine bases in different views to be determined, and hence controlled at unity. Unlike its projective equivalent, the method requires no self-calibration, the zooming camera may move quite generally, and no restriction need be applied to the nature of the scene being viewed. The performances of 3D and 2D versions of the algorithms are examined using synthetic data under varying levels of noise and under varying degrees of degeneracy in motion and structure. Real-time results are presented from imagery of laboratory scenes and off-line results obtained from an outdoor surveillance video sequence.

This paper describes a visual tool for teleoperative experimentation involving remote manipulation and contact tasks. Using modest hardware, it recovers in real time the pose of moving polyhedral objects, and presents a synthetic view of the scene to the operator of a teleoperated robot using any chosen viewpoint and viewing direction. To recover pose, the method of line tracking first introduced by Harris (1992) is extended to multiple calibrated cameras, and its dynamic performance improved using robust methods and iterative filtering. Experiments are reported which determine the static and dynamic performance of the vision system, and its use in teleoperation is illustrated in two experiments, a peg-in-hole manipulation task and an impact control task.

In this paper we will show how rotation matrices obtained from image measurements, for instance via self-calibration of rotating cameras, may be used to self-align an active stereo head such that the optic axes of the cameras are parallel, horizontal and perpendicular to the elevation axis. Whereas general rotation matrices have three degrees of freedom, with robot heads we may generate motions with fewer degrees of freedom, and use prior knowledge of the kinematic chain to constrain the rotation matrices to have a particular form. We demonstrate how these constraints provide sufficient alignment information, and we provide fast, accurate and stable algorithms to achieve this task. Both a linear method, which minimizes an algebraic error, and two nonlinear methods which minimize meaningful errors, are presented, and we show in theory and experiments that they give virtually identical results. Results from both simulated data and from real images obtained from an active stereo head are presented.

We show that radial distortion of images invalidates the geometric constraint on which self-calibration of a rotating camera is based on, that is, 3D lines drawn between matched features all intersect at the rotation centre. We develop a geometric picture showing how radial distortion violates this constraint and discuss the implications for self-calibration of a rotating camera. In particular we show that the behaviour of self-calibration is markedly different for pin-cushion and barrel distortion, the latter causing self-calibration to be unreliable or to fail completely. A method is presented for automatically estimating the radial distortion over a sequence of images, when both distortion and camera internal parameters vary. We discuss when such an approach will work and whether accurate automatic calibration of a rotating camera is really possible.

The paper describes the design and construction of a wearable active vision platform which is able to achieve substantial decoupling of the camera motion from the wearer's motion. Design issues in sensor placement, robot kinematics and their relation to wearability are discussed and the prototype platform's performance is evaluated in a number of important visual tasks. The paper also considers potential application scenarios for this kind of wearable visual robot.

Visual feedback is an essential part of any telerobotic system. Traditional approaches have provided the operator with a view of the remote workcell either through TV monitors fed by cameras dotted around the workcell or by direct viewing. More recently, virtual viewing systems have been used for operator feedback because they allow the operator a choice of viewpoint. Care, however, must be taken to ensure that objects displayed by the virtual view are in true relation to their remote workcell counterparts. In this paper, the effects of two types of errors on operator performance are reported. The first type of error is due to offsets (biases) of the virtual object from their true workcell locations which are caused by modelling errors of the remote workcell. The second is caused by noise and dynamics induced by the sensors and filters used to track the position and orientation (pose) of the remote objects. Experiments are conducted to examine the performance of operators completing peg-in-hole tasks subject to the two errors. We conclude the paper by commenting on the validity of current operator motor control models.

In this paper we specialize the projective unifocal, bifocal, and trifocal tensors to the affine case, and show how the tensors obtained relate to the registered tensors encountered in previous work. This enables us to obtain an affine specialization of known projective relations connecting points and lines across two or three views. In the simpler case of affine cameras we give neccessary and sufficient constraints on the components of the trifocal tensor together with a simple geometric interpretation. Finally, we show how the estimation of the tensors from point correspondences is achieved through factorization, and discuss the estimation from line correspondences.

This paper describes methods for tracking using point and line features in affine views to provide fundamental invariance to changes of focal length. It first demonstrates how an earlier method of transfer-based tracking using spatio-temporal matching of point features in a stereo active head is indeed zoom-invariant. In order to also make use of lines the paper then illustrates how the affine tri- and quadrifocal tensors may be applied to tracking with zoom in monocular and stereo systems respectively. The usefulness of the tensors is evident from their ability to transfer a fixation point in two uncalibrated images into novel views. Whereas the trifocal tensor is already familiar and in common use for matching and reconstruction we believe this to be the first practical application of a quadrifocal tensor. We develop expressions for affine tri- and quadrifocal tensors, and using novel affine specializations of existing projective algorithms we show how computation of the tensors is faster simpler and more stable. Experiments on real images are presented.

Derives two discrete motion models for three-dimensional (3-D) pose recovery starting from the stochastic differential equations that describe the object's motion in continuous time. The velocity is considered first as a Wiener process, which underlies the very often used constant velocity model, and second as an Ornstein-Uhlenbeck process. Analysis of the autocorrelation signal derived in experiment from visual tracking of both translational and rotational movements of a human head demonstrates that such muscular motion is better suited to modeling by the Ornstein-Uhlenbeck velocity process. The model is embedded in an iterated extended Kalman filter, which linearizes about the predicted pose and in which care is taken properly to transform the state covariance, and incorporated in a system able visually to recover and track the pose of a human operator's head. The pose is copied onto a slave electromechanical stereo camera platform alone, providing two rotational degrees of freedom, or onto the same head carried by a robot arm to give complete six degree-of-freedom mapping of the operator's movements. Accuracy and frequency response are assessed.

This paper describes a system for the recovery in real-time of the pose of moving polyhedral objects using modest hardware. A method of line track-ing first introduced by Harris is extended to multiple calibrated cameras, and afforced by robust methods and filtering. The system, which uses three cam-eras a low-end commercial framegrabber and runs on single PC, has been devised to provide simple visual feedback for the tele-operator of a force re-flecting robot manipulator. Experimental results are given which demonstrate the accuracy of the vision system. 1

This paper describes the theory and real-time implementation using an active camera platform of a method of planar trajectory recovery, and of the use of those trajectories to facilitate prediction over delays in the visual feedback loop. Image-based position and velocity demands for tracking are generated by detecting and segmenting optical flow within a central region of the image, and a projective construct is used to map the camera platform's joint angles into a Euclidean coordinate system within a plane, typically the ground plane, in the scene. A set of extended Kalman filters with different dynamics is implemented to analyze the trajectories, and these compete to provide the best description of the motion within an interacting multiple model. Prediction from the optimum motion model is used within the visual feedback loop to overcome visual latency. It is demonstrated that prediction from the 3D planar description gives better tracking performance than prediction based on a filtered description of observer-based 2D motion trajectories.

The following presents a standardized approach to the kinematics of a generalized stereo robot head, providing both forward and inverse kinematic solutions as well as a discussion on the head Jacobian. The paper is intended as a comprehensive tutorial and as a standard notation reference for researchers in the field of active vision.

Zoom interacts strongly with both vision and control processes in an active visual system, causing problems for many commonly used tracking methods. This paper demonstrates the use of afne transfer to track while zooming, using clusters of corner features. Afne transfer not only is fundamentally invariant to zoom but also provides a natural mechanism to allow features to appear and disappear while tracking, events which will occur as detail sharpens and dissolves during zooming. The paper demonstrates ofine 3D afne transfer during zoom for objects undergoing substantial rotation, and describes real-time experiments using 2D afne transfer while zooming and tracking using an active camera platform.

This paper describes a system for visual telepresence that copies the motion of an operator’s head onto the neck axes of an electromechanical stereo head/eye platform. Three or more point features on the operator’s head are detected using an external static camera and tracked over time to recover the pose of the operator’s head The process of capturing images, tracking features, recovering pose, and generating demands runs at 50Hz with a latency of 23ms, with the controller and platform typically taking another 20ms to satisfy a demand. Experiments show that the system, including head platform, is able to copy head movements of up to s . The paper goes on to consider the possibility of copying individual eye movements onto the individual camera axes of the head, but simulation studies suggest that delays in the feedback will lead to unsmooth eye response.

Steering a motor vehicle around a winding but otherwise uncluttered road has been observed by Land and Lee (1994) to involve repeated periods of visual fixation upon the tangent point of the inside of each bend. We demonstrate a similar use of `active' fixation in the autonomous navigation of a robot vehicle around an obstacle, and show how the control law devised for steering in the robotic example is applicable to the observed human performance data. We discuss the merits of fixation for mobile robot localization.

This paper describes a system for visual telepresence that slaves the motion of an electromechanical stereo head/eye platform to that of an operator's head. Features on the operator's head are detected using an external static camera and tracked over time, using three or more features to find pose (orientation and translation) with a novel algorithm which avoids explicit recovery of the features 3D positions. The recovered pose is then transformed into demands for the pan and elevation axes of the head platform. The process of capturing images, tracking features, recovering pose, and generating demands runs at 50 Hz with a latency of 23 ms, with the controller and platform typically taking another 20 ms to satisfy a demand. Experiments show that the dynamic performance of the entire system, including head platform, is sufficient to copy head movements of up to 400/spl deg/ s/sup -1/.

This paper presents a new technique for calibrating a camera mounted on a controllable head/eye platform. It uses the trajectories of an arbitrary number of tracked corner features to improve the calibration parameter estimates over time, utilizing a novel variable state dimension form of recursive filter. No special visual stimuli are required and no assumptions are made about the structure of the scene, other than that it is stationary relative to the head. The algorithm runs at 4 frames per second on a single Inmos T805 transputer, and is fully integrated into a real-time active vision system. Updated calibration parameters are regularly passed to the vision modules that require them. Although the algorithm requires an initial estimate of camera focal length, results are presented from real experiments demonstrating that convergence is achieved for initial errors up to 50%.

This paper demonstrates an approach which exploits an active camera as a projective pointing mechanism. The optical centre of a static camera is notionally substituted by the centre of rotation of the active camera, as is the image plane by a frontal plane, a plane perpendicular to the optical axis of the active camera in its resting direction. Algorithms devised for 3D motion and 3D structure recovery using a single passive camera become immediately applicable to the active camera without need for reformulation. Furthermore, because the active camera can access a panoramic field of view, instabilities which may arise when the field of view is small, or because the shared field of view between successive after movement is small, are lessened. Two quite different applications of the idea are presented. In the first, the homography between a planar surface in the scene and the frontal plane is recovered and used to recover scene trajectories. In the second, the essential matrix between points in two frontal-plane views is recovered and used to determine the motion of a mobile vehicle.

Demonstrates a method of using nonmetric visual information derived from an uncalibrated active vision system to navigate an autonomous vehicle through free-space regions detected in a cluttered environment. The structure of 3-space is recovered modulo an affine transformation using an uncalibrated active stereo head carried by the vehicle. The plane at infinity, necessary for recovering affine structure from projective structure, is found in a novel manner by making controlled rotations of the head. The structure is composed of 3D points obtained by detecting and matching image corners through the stereo image sequence. Considerable care has been taken to ensure that the processing is reliable, robust and automatic. Driveable regions are determined from the projection of the affine structure onto a plane parallel to the ground determined using projective constructs. Two methods of negotiating the regions are explored. The first introduces metric information to allow control of a Euclidean vehicle. The second uses visual servoing of the active head to navigate in the affinely described free-space regions.>

This paper describes an algorithm for stereo tracking using 3D affine transfer of a body-centred fixation point. Transfer is based on corners detected in the image and matched over time and in stereo. The paper presents a method of basing the transfer on all the available data, providing immunity to noise and poor conditioning. The paper also shows an implementation at video rates on a four axis active camera platform. Graceful degradation in the presence of insufficient data and fixed latency tracking in parallel with the structure calculation provide robust performance. Recovered trajectories are shown in an approximately Euclidean frame while structure transfer is demonstrated by the evolution of the target's convex hull.>

The paper proposes a statistical framework that enables 3D structure and motion to be computed optimally from an image sequence, on the assumption that feature measurement errors are independent and Gaussian distributed. The analysis and results demonstrate that computing both camera/scene motion and 3D structure is essential to computing either with any accuracy. Having computed optimal estimates of structure and motion over a small number of initial images, a recursive version of the algorithm (previously reported) recomputes sub optimal estimates given new image data. The algorithm is designed explicitly for real time implementation, and the complexity is proportional to the number of tracked features. 3D projective, affine and Euclidean models of structure and motion recovery have been implemented, incorporating both point and line features into the computation. The framework can handle any feature type and camera model that may be encapsulated as a projection equation from scene to image.>

We describe an algorithm which uses affine transfer of the fixation point in a stereo pair to obtain stereo fixation of a moving object. The method runs in real-time using corners tracked temporally and in stereo. The fixation point is transferred to new left and right views using affine structure from four views old left and right and new left and right. We also present the singular value decomposition as a means by which all possible feature points contribute to the transfer, providing immunity to poor choice of basis features. Early results are presented for the method, showing its speed and reliability.

In this paper we describe the use of a steerable camera and active vision system in the real-time recovery of trajectories of objects moving on a planar surface in the scene. The system first self-calibrates the transformation from camera platform joint angles to scene plane coordinates by active observation of static scene geometry which has been stored as a model. The transformation is encoded as a homography. When distracted by motion, the camera saccades to the object and subsequently tracks it, and the 2D Cartesian components of the motion trajectory are recovered within the scene plane.>

We describe the implementation of, and results from, a realtime active surveillance vision system which detects moving objects in an everyday environment, directs the gaze of a headkye platform towards the objects and subsequently pursues them smoothly. Target detection and pursuit arc performed purely on the basis of image motion, and can continue over extended periods. Two independent parallel processes derive (i) coarse resolution motion over the entire image to direct saccadic shifts in attention over a wide field of view, and (ii) fine resolution motion in a small central region of the image used to perform smooth-pursuit. A gaze controller which selects results from the two visual processes and controls the movement of the head platform is implemented as a finite state machine.

The authors demonstrate four real-time reactive responses to movement in everyday scenes using an active head/eye platform. They first describe the design and realization of a high-bandwidth four-degree-of-freedom head/eye platform and visual feedback loop for the exploration of motion processing within active vision. The vision system divides processing into two scales and two broad functions. At a coarse, quasi-peripheral scale, detection and segmentation of new motion occurs across the whole image, and at fine scale, tracking of already detected motion takes place within a foveal region. Several simple coarse scale motion sensors which run concurrently at 25 Hz with latencies around 100 ms are detailed. The use of these sensors are discussed to drive the following real-time responses: (1) head/eye saccades to moving regions of interest; (2) a panic response to looming motion; (3) an opto-kinetic response to continuous motion across the image and (4) smooth pursuit of a moving target using motion alone.>

The authors describe a novel method of obtaining a fixation point on a moving object for a real-time gaze control system. The method makes use of a real-time implementation of a corner detector and tracker and reconstructs the image position of the desired fixation point from a cluster of corners detected on the object using the affine structure available from two or three views. The method is fast, reliable, viewpoint invariant, and insensitive to occlusion and/or individual corner dropout or reappearance. Results are presented for the method used with a high performance head/eye platform. The results are compared with two naive fixation methods.>

The authors present an active vision system which performs a surveillance task in everyday dynamic scenes. The system is based around simple, rapid motion processors and a control strategy which uses both position and velocity information. The surveillance task is defined in terms of two separate behavioral subsystems, saccade and smooth pursuit, which are demonstrated individually on the system. It is shown how these and other elementary responses to 2D motion can be built up into behavior sequences, and how judicious close cooperation between vision and control results in smooth transitions between the behaviors. These ideas are demonstrated by an implementation of a saccade to smooth pursuit surveillance system on a high-performance robotic hand/eye platform.

This paper describes a scheme in which optical flow information is used to guide stereo correspondence matching. The epipolar constraint, which is used in static stereo to confine correspondence search to a line, is replaced by a mechanism which confines search to an area around a point. The experimental setup consists of a pair of cameras mounted on a mobile robot vehicle. A sequence of stereo image pairs is taken during vehicle motion, and the images are processed by a corner detector. One of the first steps in extracting useful information from the corners is to solve the correspondence problem. For a stereo sequence, this requires both the temporal matching of corners along the sequence, and also the stereo matching of corners between left and right images. The results of

This paper aims to develop a simple algorithm for determining the direction of ego-motion of a monocular observer in a static environment. Rather than returning a single direction the algorithm returns a set of motion directions known to contain the true direct tion. Translational motion can be arbitrary. The algorithm is based on one invented by Horn and Weldon [1]. It uses only normal components of optical flow at discrete image locations and is only intended to function when the the contribution of rotation to the apparent motion is small. The algorithm uses the fact that imaged points must be at some positive distance from the viewer. No implicit assumptions are made about the scene being viewed. The algorithm is illustrated on an artificially generated set of normal components.

This paper presents results from computer experiments with an algorithm to perform scene disposition and motion segmentation from visual motion or optic flow. The maximum a posteriori (MAP) criterion is used to formulate what the best segmentation or interpretation of the scene should be, where the scene is assumed to be made up of some fixed number of moving planar surface patches. The Bayesian approach requires, first, specification of prior expectations for the optic flow field, which here is modeled as spatial and temporal Markov random fields; and, secondly, a way of measuring how well the segmentation predicts the measured flow field. The Markov random fields incorporate the physical constraints that objects and their images are probably spatially continuous, and that their images are likely to move quite smoothly across the image plane. To compute the flow predicted by the segmentation, a recent method for reconstructing the motion and orientation of planar surface facets is used. The search for the globally optimal segmentation is performed using simulated annealing.