This paper discusses two of the most important design considerations for a wearable device with visual sensing: what kind of sensor to use and where to place it. While nature and computer vision have explored a wide range of imaging techniques, wearables have mostly viewed the world through conventional narrow-view passive cameras designed for nonwearable applications, which are attached to the wearer's head. The rationale presented here for sensor selection and the novel methodology developed for objectively studying sensor placement have informed the development of a number of visual wearables.

We contrast the performance of two methods of imposing constraints during the tracking of articulated objects, the first method preimposing the kinematic constraints during tracking and, thus, using the minimum degrees of freedom, and the second imposing constraints after tracking and, hence, using the maximum. Despite their very different formulations, the methods recover the same pose change. Further comparisons are drawn in terms of computational speed and algorithmic simplicity and robustness, and it is the last area which is the most telling. The results suggest that using built-in constraints is well-suited to tracking individual articulated objects, whereas applying constraints afterward is most suited to problems involving contact and breakage between articulated (or rigid) objects, where the ability to test tracking performance quickly with constraints turned on or off is desirable.

To fill holes in photographs of structured, man made environments, we propose a technique which automatically adjusts and clones large image patches that have similar structure. These source patches can come from elsewhere in the same image, or from other images shot from different perspectives. Two significant developments of this work are the ability to automatically detect and adjust source patches whose macrostructure is compatible with the hole region, and alternately, to interactively specify a user's desired search regions. In contrast to existing photomontage algorithms which either synthesize microstructure or require careful user interaction to fill holes, our approach handles macrostructure with an adjustable degree of automation.

MLESAC is an established algorithm for maximum-likelihood estimation by random sampling consensus, devised for computing multiview entities like the fundamental matrix from correspondences between image features. A shortcoming of the method is that it assumes that little is known about the prior probabilities of the validities of the correspondences. This paper explains the consequences of that omission and describes how the algorithm's theoretical standing and practical performance can be enhanced by deriving estimates of these prior probabilities. Using the priors in guided-MLESAC is found to give an order of magnitude speed increase for problems where the correspondences are described by one image transformation and clutter. This paper describes two further modifications to guided-MLESAC. The first shows how all putative matches, ratherthan just the best, from a particularfeature can be taken forward into the sampling stage, albeit at the expense of additional computation. The second suggests how to propagate the output from one frame forward to successive frames. The additional information makes guided-MLESAC computationally realistic at video-rates for correspondence sets modeled by two transformations and clutter.

This paper is concerned with allowing the user of a wearable, portable, vision system to interact with the visual information using hand movements and gestures. Two example scenarios are explored. The first, in 2D, uses the wearer’s hand to both guide an active wearable camera and to highlight objects of interest using a grasping vector. The second is based in 3D, and builds on earlier work which recovers 3D scene structure at video-rate, allowing real-time purposive redirection of the camera to any scene point. Here, a range of hand gestures are used to highlight and select 3D points within the structure and in this instance used to insert 3D graphical objects into the scene. Structure recovery, gesture recognition, scene annotation and augmentation are achieved in parallel and at video-rate.

During tracking, lens zoom acts as a gain between scene dynamics and fixation errors, providing a trade-off between maximizing resolution and minimizing tracking error. Using a linear Kalman filter model, it is shown that when image measurement error scales with focal length, the filter is invariant to zoom. When the error is of fixed size, however, zooming alters the balance between process and measurement errors in a counter-intuitive manner. It is shown that this balance can be restored by appropriate adjustment of the process noise. With a zoom invariant filter, zoom can be used to ensure that fixation errors remain bounded. To this end, an error variance control method is proposed which gives high confidence that the target will not leave the image during tracking. To implement such a scheme, equipment delays and responses must be known, including those of the zoom lenses. Experiments to measure these are described, and overall results are presented for 30 Hz tracking of real scenes.

Recent advances in wearable sensing allow active control of the orientation of a body-mounted camera worn by a remote user. In this paper we consider the control of the active camera from head movements. In the context of teleoperation, these may be the head movements of a remote operator, perhaps acting as the wearer’s assistant. The movements are likely to be larger than those in video-conference applications, and so frontal facial features are insufficient. The paper presents a model which incrementally combines a fixed 3D shape model with specific features found on the observed head. Robust methods, including the incorporation of a colour model, are used to mitigate the effect of mismatching, the main contribution being the use of both interest point and colour features within a single random-sampling framework.

We report on two methods we have developed to aid in the design of a wearable visual robot-a body mounted robot for which the main sensor is a camera. Specifically, we have first refined the analysis of sensor placement through the computation of the field of view and body motion using a 3D model of the human form. Second we have improved the design of the robot's morphology with the help of an optimization algorithm based on the Pareto front, within constraints set by the overall choice of robot kinematic chain and the need to specify obtainable actuators and sensors. The methods could be of use for the design and performance evaluation of rather different kinds of wearable robots and devices.

This paper describes a method for visual control of the zoom setting of an active camera during tracking. The method assumes an affine projection, and tracking is achieved using affine transfer, a process which is fundamentally invariant to zoom. However, the form of the projection matrices determined to achieve transfer allows the relative scale between the affine bases in different views to be determined, and hence controlled at unity. Unlike its projective equivalent, the method requires no self-calibration, the zooming camera may move quite generally, and no restriction need be applied to the nature of the scene being viewed. The performances of 3D and 2D versions of the algorithms are examined using synthetic data under varying levels of noise and under varying degrees of degeneracy in motion and structure. Real-time results are presented from imagery of laboratory scenes and off-line results obtained from an outdoor surveillance video sequence.

We show that radial distortion of images invalidates the geometric constraint on which self-calibration of a rotating camera is based on, that is, 3D lines drawn between matched features all intersect at the rotation centre. We develop a geometric picture showing how radial distortion violates this constraint and discuss the implications for self-calibration of a rotating camera. In particular we show that the behaviour of self-calibration is markedly different for pin-cushion and barrel distortion, the latter causing self-calibration to be unreliable or to fail completely. A method is presented for automatically estimating the radial distortion over a sequence of images, when both distortion and camera internal parameters vary. We discuss when such an approach will work and whether accurate automatic calibration of a rotating camera is really possible.

The paper describes the design and construction of a wearable active vision platform which is able to achieve substantial decoupling of the camera motion from the wearer's motion. Design issues in sensor placement, robot kinematics and their relation to wearability are discussed and the prototype platform's performance is evaluated in a number of important visual tasks. The paper also considers potential application scenarios for this kind of wearable visual robot.