Aging and chronic conditions affect older adults’ daily lives, making the early detection of developing health issues crucial. Weakness, which is common across many conditions, can subtly alter physical movements and daily activities. However, these behavioral changes can be difficult to detect because they are gradual and often masked by natural day-to-day variability. To isolate the behavioral phenotype of weakness while controlling for confounding factors, this study simulates physical weakness in healthy adults through exercise-induced fatigue, providing interpretable insights into potential behavioral indicators for long-term monitoring. A non-intrusive camera sensor is used to monitor individuals’ daily sitting and relaxing activities over multiple days, allowing us to observe behavioral changes before and after simulated weakness. The system captures fine-grained features related to body motion, inactivity, and environmental context in real time while prioritizing privacy. A Bayesian Network models the relationships among activities, contextual factors, and behavioral indicators. Fine-grained features, including non-dominant upper-body motion speed and scale, together with inactivity distribution, are most effective when used with a 300-second window. Personalized models achieve 0.97 accuracy at distinguishing simulated weak days from normal days, and no universal set of optimal features or activities is observed across participants.

Video understanding has made huge strides in recent years, relying largely on the power of transformers. As this architecture is notoriously expensive and video data is highly redundant, research into improving efficiency has become particularly relevant. Some creative solutions include token selection and merging. While most methods succeed in reducing the cost of the model and maintaining accuracy, an interesting pattern arises: most methods do not outperform the baseline of randomly discarding tokens. In this paper we take a closer look at this phenomenon and observe 5 principles of the nature of visual tokens. For example, we observe that the value of tokens follows a clear Pareto-distribution where most tokens have remarkably low value, and just a few carry most of the perceptual information. We build on these and further insights to propose a lightweight video model, LITE, that can select a small number of tokens effectively, outperforming state-of-the-art and existing baselines across datasets (Kinetics-400 and Something-Something-V2) in the challenging trade-off of computation (GFLOPs) vs accuracy. Experiments also show that LITE generalizes across datasets and even other tasks without the need for retraining.

Deformable linear object (DLO) manipulation is needed in many fields. Previous research on deformable linear object (DLO) manipulation has primarily involved parallel jaw gripper manipulation with fixed grasping positions. However, the potential for dexterous manipulation of DLOs using an anthropomorphic hand is under-explored. We present DexDLO, a model-free framework that learns dexterous dynamic manipulation policies for deformable linear objects with a fixed-base dexterous hand in an end-to-end way. By abstracting several common DLO manipulation tasks into goal-conditioned tasks, DexDLO can perform tasks such as DLO grabbing, DLO pulling, DLO end-tip position controlling, etc. Using the Mujoco physics simulator, we demonstrate that our framework can efficiently and effectively learn five different DLO manipulation tasks with the same framework parameters. We further provide a thorough analysis of learned policies, reward functions, and reduced observations for a comprehensive understanding of the framework.

A new application for real-time monitoring of the lack of movement in older adults’ own homes is proposed, aiming to support people’s lives and independence in their later years. A lightweight camera monitoring system, based on an RGB-D camera and a compact computer processor, was developed and piloted in community homes to observe the daily behavior of older adults. Instances of body inactivity were detected in everyday scenarios anonymously and unobtrusively. These events can be explained at a higher level, such as a loss of consciousness or physiological deterioration. The accuracy of the inactivity monitoring system is assessed, and statistics of inactivity events related to the daily behavior of older adults are provided. The results demonstrate that our method achieves high accuracy in inactivity detection across various environments and camera views. It outperforms existing state-of-the-art vision-based models in challenging conditions like dim room lighting and TV flickering. However, the proposed method does require some ambient light to function effectively.

Abstract Three-dimensional registration is an established yet challenging problem that is key in many different applications, such as mapping the environment for autonomous vehicles, or modeling people for avatar creation, among others. Registration refers to the process of mapping multiple data into the same coordinate system by means of matching correspondences and transformation estimation. Novel proposals exploit the benefits of deep learning architectures for this purpose, as they learn the best features for the data, providing better matches and hence results. However, the state of the art is usually focused on cases of relatively small transformations, although in certain applications and in a real and practical environment, large transformations are very common. In this paper, we present ReLaTo (Registration for Large Transformations), an architecture that addresses the cases where large transformations happen while maintaining good performance for local transformations. This proposal uses a novel Softmax pooling layer to find correspondences in a bilateral consensus manner between two point sets, sampling the most confident matches. These matches estimate a coarse and global registration using weighted Singular Value Decomposition (SVD). A target-guided denoising step is applied to both the obtained matches and latent features to estimate the final fine registration considering the local geometry. All these steps are carried out following an end-to-end approach, which has been shown to perform better than 7 state-of-the-art registration methods in two datasets commonly used for this task (ModelNet40 and the Karlsruhe Institute of Technology and Toyota Technological Institute dataset, KITTI), especially in the case of large transformations. Graphic abstract

Scene understanding is an important area in robotics and autonomous driving. To accomplish these tasks, the 3D structures in the scene have to be inferred to know what the objects and their locations are. To this end, semantic segmentation and disparity estimation networks are typically used, but running them individually is inefficient since they require high-performance resources. A possible solution is to learn both tasks together using a multi-task approach. Some current methods address this problem by learning semantic segmentation and monocular depth together. However, monocular depth estimation from single images is an ill-posed problem. A better solution is to estimate the disparity between two stereo images and take advantage of this additional information to improve the segmentation. This work proposes an efficient multi-task method that jointly learns disparity and semantic segmentation. Employing a Siamese backbone architecture for multi-scale feature extraction, the method integrates specialized branches for disparity estimation and coarse and refined segmentations, leveraging progressive task-specific feature sharing and attention mechanisms to enhance accuracy for solving both tasks concurrently. The proposal achieves state-of-the-art results for joint segmentation and disparity estimation on three distinct datasets: Cityscapes, TrimBot2020 Garden, and S-ROSeS, using only 1/3 of the parameters of previous approaches.

Current datasets for computer vision-based action recognition and localization cover a wide range of classes and challenging scenarios. However, these datasets don't cater to healthcare applications that involve long-term monitoring, tracking minor changes in movements over time for healthcare purposes, or completely modeling a specific human behavior that includes multiple sub-actions. Specifically, there are no existing datasets for research on either health monitoring on atomic-action-based eating behavior or for a full range of eating sub-actions that fully segment the main action. Addressing these gaps is valuable for extending research on the health monitoring of elderly people and is needed for creating richer and more complete descriptions of actions. This paper introduces a new benchmark dataset named EatSense that targets both the computer vision and healthcare communities and fills in the aforementioned gaps. EatSense is recorded while a person eats in an uncontrolled dining setting. The key features of EatSense are the introduction of challenging atomic actions for action recognition, the significantly diverse durations of actions that make it difficult for current temporal action localization frameworks to localize, the capability to model comprehensive eating behavior in terms of a sequence of action-based behaviors, and the simulation of minor variations in motion or performance. We conduct extensive experiments on EatSense with baseline deep learning-based approaches for benchmarking and hand-crafted feature-based approaches for explainable applications. We believe this dataset will benefit future researchers in building robust temporal action localization networks, behavior recognition, and performance assessment models for eating.

Abstract The elderly population is increasing at a rapid rate, and the need for effectively supporting independent living has become crucial. Wearable sensors can be helpful, but these are intrusive as they require adherence by the elderly. Thus, a semi-anonymous (no image records) vision-based non-intrusive monitoring system might potentially be the answer. As everyone has to eat, we introduce a first investigation into how eating behavior might be used as an indicator of performance changes. This study aims to provide a comprehensive model of the eating behavior of individuals. This includes creating a visual representation of the different actions involved in the eating process, in the form of a state diagram, as well as measuring the level of performance or decay over time during eating. Also, in studies that involve humans, getting a generalized model across numerous human subjects is challenging, as indicative features that parametrize decay/performance changes vary significantly from person to person. We present a two-step approach to get a generalized model using distinctive micro-movements, i.e., (1) get the best features across all subjects (all features are extracted from 3D poses of subjects) and (2) use an uncertainty-aware regression model to tackle the problem. Moreover, we also present an extended version of EatSense, a dataset that explores eating behavior and quality of motion assessment while eating.

Scene classification is a well-established area of computer vision research that aims to classify a scene image into pre-defined categories such as playground, beach and airport. Recent work has focused on increasing the variety of pre-defined categories for classification, but so far failed to consider two major challenges: changes in scene appearance due to lighting and open set classification (the ability to classify unknown scene data as not belonging to the trained classes). Our first contribution, SceneVLAD, fuses scene classification and visual place recognition CNNs for appearance invariant scene classification that outperforms state-of-the-art scene classification by a mean F1 score of up to 0.1. Our second contribution, OpenSceneVLAD, extends the first to an open set classification scenario using intra-class splitting to achieve a mean increase in F1 scores of up to 0.06 compared to using state-of-the-art openmax layer. We achieve these results on three scene class datasets extracted from large scale outdoor visual localisation datasets, one of which we collected ourselves.

The lip is a dominant dynamic facial unit when a person is speaking. Detecting lip events is beneficial to speech analysis and support for the hearing impaired. This paper proposes a 3D lip event detection pipeline that automatically determines the lip events from a 3D speaking lip sequence. We define a motion divergence measure using 3D lip landmarks to quantify the interframe dynamics of a 3D speaking lip. Then, we cast the interframe motion detection in a multi-temporal-resolution framework that allows the detection to be applicable to different speaking speeds. The experiments on the S3DFM Dataset investigate the overall 3D lip dynamics based on the proposed motion divergence. The proposed 3D pipeline is able to detect opening and closing lip events across 100 sequences, achieving a state-of-the-art performance.

Three-dimensional registration is a well-known topic in computer vision that aims to align two datasets (e.g. point clouds). Recent approaches to this problem are based on learning techniques. In this paper, we present an improved solution to the problem of registration with a novel architecture that, given two 3D point clouds as input, estimates the rotation to map one into the other. The network architecture is conceptually divided into two parts, the first part is a feature selection based on PointNet and PointNet++. The second part estimates the rotation with Euler angles by calculating the correspondences with a FlowNet-based network, and finally the rotations in yaw, pitch, and roll. The generalization capability of the proposal allows mapping two point clouds in a wide range of angles with a stable error over the whole range. Experiments have been carried out using the ModelNet10 objects dataset, varying the axis and the angle of rotation to provide a sufficiently complete evaluation of the architecture. The results show an average distance Mean Square Error of 4.94 × 10−5within a unit sphere and a rotation error of 1.18 degrees. The results with noisy point clouds are sufficiently accurate providing the network was trained only with noise-free data, demonstrating good generalization of the approach.

Learning object-centric scene representations is essential for attaining structural understanding and abstraction of complex scenes. Yet, as current approaches for unsupervised object-centric representation learning are built upon either a stationary observer assumption or a static scene assumption, they often: i) suffer single-view spatial ambiguities, or ii) infer incorrectly or inaccurately object representations from dynamic scenes. To address this, we propose Dynamics-aware Multi-Object Network (DyMON), a method that broadens the scope of multi-view object-centric representation learning to dynamic scenes. We train DyMON on multi-view-dynamic-scene data and show that DyMON learns---without supervision---to factorize the entangled effects of observer motions and scene object dynamics from a sequence of observations, and constructs scene object spatial representations suitable for rendering at arbitrary times (querying across time) and from arbitrary viewpoints (querying across space). We also show that the factorized scene representations (w.r.t. objects) support querying about a single object by space and time independently.

In the context of supervised statistical learning, it is typically assumed that the training set comes from the same distribution that draws the test samples. When this is not the case, the behavior of the learned model is unpredictable and becomes dependent upon the degree of similarity between the distribution of the training set and the distribution of the test set. One of the research topics that investigates this scenario is referred to as domain adaptation (DA). Deep neural networks brought dramatic advances in pattern recognition and that is why there have been many attempts to provide good DA algorithms for these models. Herein we take a different avenue and approach the problem from an incremental point of view, where the model is adapted to the new domain iteratively. We make use of an existing unsupervised domain-adaptation algorithm to identify the target samples on which there is greater confidence about their true label. The output of the model is analyzed in different ways to determine the candidate samples. The selected samples are then added to the source training set by self-labeling, and the process is repeated until all target samples are labeled. This approach implements a form of adversarial training in which, by moving the self-labeled samples from the target to the source set, the DA algorithm is forced to look for new features after each iteration. Our results report a clear improvement with respect to the non-incremental case in several data sets, also outperforming other state-of-the-art DA algorithms.

We present a groundtruthing approach which is applicable to large video datasets collected for studying people's behavior, and which are recorded at a low frame per second (fps) rate. Groundtruthing a large dataset manually is a time consuming task and is prone to errors. The proposed approach is semi-automated (using a combination of deepnet and traditional image analysis) to minimize human labeler's interaction with the video frames. The framework employs mask-rcnn as a people counter followed by human assisted semi-automated tests to correct the wrong labels. Subsequently, a bounding box extraction algorithm is used which is fully automated for frames with a single person and semi-automated for frames with two or more people. We also propose a methodology for anomaly detection i.e., collapse on table or floor. Behavior recognition is performed by using a fine-tuned alexnet convolutional neural network. The people detection and behavior analysis components of the framework are primarily designed to help reduce human labor in ground-truthing so that minimal human involvement is required. They are not meant to be employed as fully automated state-of-the-art systems. The proposed approach is validated on a new dataset presented in this paper, containing human activity in an indoor office environment and recorded at 1 fps as well as an indoor video sequence recorded at 15 fps. Experimental results show a significant reduction in human labor involved in the process of ground-truthing i.e., the number of potential clicks for office dataset was reduced by 99.2% and for the additional test video by 99.7%.

The paper presents a working pipeline which integrates hardware and software in an automated robotic rose cutter. To the best of our knowledge, this is the first robot able to prune rose bushes in a natural environment. Unlike similar approaches like tree stem cutting, the proposed method does not require to scan the full plant, have multiple cameras around the bush, or assume that a stem does not move. It relies on a single stereo camera mounted on the end-effector of the robot and real-time visual servoing to navigate to the desired cutting location on the stem. The evaluation of the whole pipeline shows a good performance in a garden with unconstrained conditions, where finding and approaching a specific location on a stem is challenging due to occlusions caused by other stems and dynamic changes caused by the wind.

With the emergence of low cost 3D sensors, the focus is moving towards the recognition and scene understanding of tridimensional data. This kind of representation is really challenging in terms of computation, and it needs the development of new strategies and algorithms to be handled and interpreted.In this work, we propose NurbsNet, a novel approach for 3D object classification based on local similarities with free form surfaces modeled as Nurbs.The proposal has been tested in ModelNet10 and ModelNet40 with results that are promising with less training iterations than state-of-the-art methods and very low memory consumption.

Learning object-centric representations of multi-object scenes is a promising approach towards machine intelligence, facilitating high-level reasoning and control from visual sensory data. However, current approaches for \textit{unsupervised object-centric scene representation} are incapable of aggregating information from multiple observations of a scene. As a result, these ``single-view'' methods form their representations of a 3D scene based only on a single 2D observation (view). Naturally, this leads to several inaccuracies, with these methods falling victim to single-view spatial ambiguities. To address this, we propose \textit{The Multi-View and Multi-Object Network (MulMON)}---a method for learning accurate, object-centric representations of multi-object scenes by leveraging multiple views. In order to sidestep the main technical difficulty of the \textit{multi-object-multi-view} scenario---maintaining object correspondences across views---MulMON iteratively updates the latent object representations for a scene over multiple views. To ensure that these iterative updates do indeed aggregate spatial information to form a complete 3D scene understanding, MulMON is asked to predict the appearance of the scene from novel viewpoints during training. Through experiments we show that MulMON better-resolves spatial ambiguities than single-view methods---learning more accurate and disentangled object representations---and also achieves new functionality in predicting object segmentations for novel viewpoints.

Fusing disparity maps from different methods is an useful technique to get a refined disparity map by leveraging the complimentary advantage. We present a model for disparity fusion that uses an adversarial network, which can be trained without using ground truth disparity data. We input two initial disparity maps (from the left view) along with auxiliary information (gradient, left & right intensity image) into the generator and train the generator to output a refined disparity map registered on the left view. The refined left disparity map and left intensity image are used to reconstruct a fake right intensity image. Finally, the fake and real right intensity images (from the right stereo vision camera) are fed into a discriminator. The trained network's architecture is effective for the fusion task (90 fps on Kitti2015). The accuracy is on par or even better than the state-of-art supervised methods. A demo video is available https://youtu.be/XTHOF3kZGsU.

Images of co-planar points in 3-dimensional space taken from different camera positions are a homography apart. Homographies are at the heart of geometric methods in computer vision and are used in geometric camera calibration, 3D reconstruction, stereo vision and image mosaicking among other tasks. In this paper we show the surprising result that homographies are the apposite tool for relating image colors of the same scene when the capture conditions-illumination color, shading and device-change. Three applications of color homographies are investigated. First, we show that color calibration is correctly formulated as a homography problem. Second, we compare the chromaticity distributions of an image of colorful objects to a database of object chromaticity distributions using homography matching. In the color transfer problem, the colors in one image are mapped so that the resulting image color style matches that of a target image. We show that natural image color transfer can be re-interpreted as a color homography mapping. Experiments demonstrate that solving the color homography problem leads to more accurate calibration, improved color-based object recognition, and we present a new direction for developing natural color transfer algorithms.

Color transfer is an image editing process that naturally transfers the color theme of a source image to a target image. In this paper, we propose a 3D color homography model which approximates photo-realistic color transfer algorithm as a combination of a 3D perspective transform and a mean intensity mapping. A key advantage of our approach is that the re-coded color transfer algorithm is simple and accurate. Our evaluation demonstrates that our 3D color homography model delivers leading color transfer re-coding performance. In addition, we also show that our 3D color homography model can be applied to color transfer artifact fixing, complex color transfer acceleration, and color-robust image stitching.

Accurately registering point clouds from a cheap low resolution sensor is a challenging task. Existing rigid registration methods failed to use the physical 3D uncertainty distribution of each point from a real sensor in the dynamic alignment process. It is mainly because the uncertainty model for a point is static and invariant and it is hard to describe the change of these physical uncertainty models in different views. Additionally, the existing Gaussian mixture alignment architecture cannot efficiently implement these dynamic changes. This paper proposes a simple architecture combining error estimation from sample covariances and dual dynamic global probability alignment using the convolution of uncertainty-based Gaussian Mixture Models (GMM) from point clouds. Firstly, we propose an efficient way to describe the change of each 3D uncertainty model, which represents the structure of the point cloud much better. Unlike the invariant GMM (representing a fixed point cloud) in traditional Gaussian mixture alignment, we use two uncertainty-based GMMs that change and interact with each other in each iteration. In order to have a wider basin of convergence than other local algorithms, we design a more robust energy function by convolving efficiently the two GMMs over the whole 3D space. Tens of thousands of trials have been conducted on hundreds of models from multiple datasets to demonstrate the proposed method’s superior performance compared with the current state-of-the-art methods. All the materials including our code is available from https://github. com/Canpu999/DUGMA.

Face-based behaviometrics focus on dynamic biological signatures generated from face behaviors, which are informative and subject-specific for identity recognition. Most existing face behaviometrics rely on 2D visual features and thus are sensitive to pose or intensity variations. This paper presents a dual-modality behaviometrics algorithm (talking-metrics) that integrates 3D video and audio cues from a human face speaking a passphrase. Static and dynamic 3D face features are extracted algorithmically and audio features are transformed through a few learning models. We concatenate the top 18 discriminative 3D visual-audio features to represent the bi-modality and utilize an linear discrimant analysis (LDA) classifier for identity recognition. The experiments were conducted on a new publicly released dataset (S3DFM). Both qualitative feature distributions and quantitative comparison results show the feasibility of the proposed pipeline and the superiority over using each modality independently. A 98.5% cross-validation recognition rate over 60 subjects and 10 trials was achieved. An anti-spoofing test also demonstrates the robustness of the proposed method.

Visual servoing is a well-known task in robotics. However, there are still challenges when multiple visual sources are combined to accurately guide the robot or occlusions appear. In this paper we present a novel visual servoing approach using hybrid multi-camera input data to lead a robot arm accurately to dynamically moving target points in the presence of partial occlusions. The approach uses four RGBD sensors as Eye-to-Hand (EtoH) visual input, and an arm-mounted stereo camera as Eye-in-Hand (EinH). A Master supervisor task selects between using the EtoH or the EinH, depending on the distance between the robot and target. The Master also selects the subset of EtoH cameras that best perceive the target. When the EinH sensor is used, if the target becomes occluded or goes out of the sensor's view-frustum, the Master switches back to the EtoH sensors to re-track the object. Using this adaptive visual input data, the robot is then controlled using an iterative planner that uses position, orientation and joint configuration to estimate the trajectory. Since the target is dynamic, this trajectory is updated every time-step. Experiments show good performance in four different situations: tracking a ball, targeting a bulls-eye, guiding a straw to a mouth and delivering an item to a moving hand. The experiments cover both simple situations such as a ball that is mostly visible from all cameras, and more complex situations such as the mouth which is partially occluded from some of the sensors.

Extracting a statistically significant result from video of natural phenomenon can be difficult for two reasons: (i) there can be considerable natural variation in the observed behaviour and (ii) computer vision algorithms applied to natural phenomena may not perform correctly on a significant number of samples. This study presents one approach to clean a large noisy visual tracking dataset to allow extracting statistically sound results from the image data. In particular, analyses of 3.6 million underwater trajectories of a fish with the water temperature at the time of acquisition are presented. Although there are many false detections and incorrect trajectory assignments, by a combination of data binning and robust estimation methods, reliable evidence for an increase in fish speed as water temperature increases are demonstrated. Then, a method for data cleaning which removes outliers arising from false detections and incorrect trajectory assignments using a deep learning‐based clustering algorithm is proposed. The corresponding results show a rise in fish speed as temperature goes up. Several statistical tests applied to both cleaned and not‐cleaned data confirm that both results are statistically significant and show an increasing trend. However, the latter approach also generates a cleaner dataset suitable for other analysis.

Abstract The first hybrid CPU‐GPU based method for estimating a point light source position in a scene recorded by an RGB‐D camera is presented. The image and depth information from the Kinect is enough to estimate a light position in a scene, which allows for the rendering of synthetic objects into a scene that appears realistic enough for augmented reality purposes. This method does not require a light probe or other physical device. To make this method suitable for augmented reality, we developed a hybrid implementation that performs light estimation in under 1second. This is sufficient for most augmented reality scenarios because both the position of the light source and the position of the Kinect are typically fixed. The method is able to estimate the angle of the light source with an average error of 20°. By rendering synthetic objects into the recorded scene, we illustrate that this accuracy is good enough for the rendered objects to look realistic. Copyright © 2015 John Wiley & Sons, Ltd.

Coral reefs exhibit significant within-class variations, complex between-class boundaries and inconsistent image clarity. This makes coral classification a challenging task. In this paper, we report the application of generic CNN representations combined with hand-crafted features for coral reef classification to take advantage of the complementary strengths of these representation types. We extract CNN based features from patches centred at labelled pixels at multiple scales. We use texture and color based hand-crafted features extracted from the same patches to complement the CNN features. Our proposed method achieves a classification accuracy that is higher than the state-of-art methods on the MLC benchmark dataset for corals.

A reject option in classification is useful to filter less confident decisions of known classes or to detect and remove untrained classes. This paper presents a novel rejection system in a hierarchical classification method for fish species recognition. Since hierarchical methods accumulate errors along the decision path, the rejection system provides an alternative channel to discover misclassified samples at the leaves of the classification hierarchy. This is also applied to probe test samples from new classes. We apply a Gaussian Mixture Model (GMM) to evaluate the posterior probability of testing samples. 2626 dimensions of features, e.g. color and shape and texture properties, from different parts of the fish are computed and normalized. We use forward sequential feature selection (FSFS), which utilizes SVM as a classifier, to select a subset of effective features that distinguishes samples of a given class from others. After learning the mixture models, the reject function is integrated with a Balance-Guaranteed Optimized Tree (BGOT) hierarchical method. We compare three rejection methods. The experimental results demonstrate a reduction in the accumulated errors from hierarchical classification and an improvement in discovering unknown classes.

We address the analysis of fish trajectories in unconstrained underwater videos environmental changes which can be observed from the abnormal behaviour of fish. The fish trajectories are separated into normal and abnormal classes which indicate the common behaviour of fish and the behaviours that are rare/ unusual respectively. The proposed solution is based on a novel type of hierarchical classifier which builds the tree using clustered and labelled data based on similarity of data while using different feature sets at different levels of hierarchy. The paper presents a new method for fish trajectory analysis which has better performance compared to state-of-the-art techniques while the results are significant considering the challenges of underwater environments, low video quality, erratic movement of fish and highly imbalanced trajectory data that we used. Moreover, the proposed method is also powerful enough to classify highly imbalanced real-world datasets.

Estimation of the point light source position in the scene enhances the experience for augmented reality. The image and depth information from the RGB-D camera allows estimation of the point light source position in a scene, where our approach does not need any probe objects or other measuring devices. The approach uses the Lambertian reflectance model, where the RGB-D camera provides the image and the surface model and the remaining unknowns are the albedo and light parameters (light intensity and direction). In order to determine the light parameters, we assume that segments with a similar colour have the same albedo, which allows us to find the point light source that explains the illumination in the scene. The performance of this method is evaluated on multiple scenes, where a single light bulb is used to illuminate the scene. In this case, the average error in the angle between the true light position vector and our estimate is around 10 degrees. This allows realistic rendering of synthetic objects into the recorded scene, which is used to improve the experience of augmented reality.

We propose an approach for the analysis of fish trajectories in unconstrained underwater videos. Trajectories are classified into two classes: normal trajectories which contain the usual behavior of fish and abnormal trajectories which indicate the behaviors that are not as common as the normal class. The paper presents two innovations: 1) a novel approach to abnormal trajectory detection and 2) improved performance on video based abnormal trajectory analysis of fish in unconstrained conditions. First we extract a set of features from trajectories and apply PCA. We then perform clustering on a subset of features. Based on the clustering, outlier detection is applied to each cluster. Improved results are obtained which is significant considering the challenges of underwater environments, low video quality, and erratic movement of fish.

We present a method for restoring the recordings obtained from surveillance cameras whose quality deteriorates due to dirt or water that gathers on the camera's lens. The method is designed to operate in the surveillance setting and makes use of good quality frames from the beginning of the recorded sequence to remove the blur at later stages caused by the dirty lens. A background subtraction method allows us to obtain a stable background of the scene. Based on this background, a multiframe blind deconvolution algorithm is used to estimate the Point Spread Function (PSF) of the blur. Once the PSF is obtained it can be used to deblur the entire scene. This restoration method was tested on both synthetic and real data with improvements of 15 dB in PSNR being achieved by using clean frames from the beginning of the recorded sequence.

This paper presents an empirical comparison of strategies for representing motion trajectories with fixed-length vectors. We compare four techniques, which have all previously been adopted in the trajectory classification literature: least-squares cubic spline approximation, the Discrete Fourier Transform, Chebyshev polynomial approximation, and the Haar wavelet transform. We measure the class separability of five different trajectory datasets- ranging from vehicle trajectories to pen trajectories- when described in terms of these representations. Results obtained over a range of dimensionalities indicate that the different representations yield similar levels of class separability, with marginal improvements provided by Chebyshev and Spline representations. For the datasets considered here, each representation appears to yield better results when used in conjunction with a curve parametrisation strategy based on arc-length, rather than time. However, we illustrate a situation- pertinent to surveillance applications- where the converse is true.

A novel learning framework is proposed for anomalous behaviour detection in a video surveillance scenario, so that a classifier which distinguishes between normal and anomalous behaviour patterns can be incrementally trained with the assistance of a human operator. We consider the behaviour of pedestrians in terms of motion trajectories, and parametrise these trajectories using the control points of approximating cubic spline curves. This paper demonstrates an incremental semi-supervised one-class learning procedure in which unlabelled trajectories are combined with occasional examples of normal behaviour labelled by a human operator. This procedure is found to be effective on two different datasets, indicating that a human operator could potentially train the system to detect anomalous behaviour by providing only occasional interventions (a small percentage of the total number of observations). 1

Common 3D acquisition techniques, such as laser scanning and stereo capture, are realistically only 2.5D in nature. Here we consider the automated completion of hidden or missing portions in 3D scenes originally acquired from 2.5D (or 3D) capture. We propose an approach based on the non-parametric propagation of available scene knowledge from the known (visible) scene areas to these unknown (invisible) 3D regions in conjunction with an initial underlying geometric surface completion.

The addition of colour information to the computation of range/scene flow is proposed to improve its accuracy and robustness to ambiguities. This is applied in the form of additional optical flow constraints from aligned colour image data. Combining constraints gives improved velocity displacement fields for both synthetic and real datasets over using depth alone, or in using depth plus intensity. This ultimately has benefits for the processing of dense, temporal depth data obtainable from novel video-rate 3D capture systems.

This paper presents and analyzes 12 quality measures that characterize robotic grips according to their stability and reliability. The measures are designed to assess three-finger grips of two-dimensional parts performed in a real environment, taking into account both theoretical aspects and unavoidable uncertainties of a grasping action. They build on the existing literature and on physical and mechanical considerations. The measures constitute a feature space that pattern recognition methods can use in order to classify robotic grips according to their quality. Six of the measures depend on the actual finger configuration of the gripper, and they have shown to be critical for better characterization. The kinematics of the Barrett Hand have been used. As a validation step, the measures are merged in two global quality values (with different practical applicability) that can be used to rank feasible candidate grips.

This paper describes a computational model for deriving a decomposition of objects from laser rangefinder data. The process aims to produce a set of parts defined by compactness and smoothness of surface connectivity. Relying on a general decomposition rule, any kind of objects made up of free-form surfaces are partitioned. A robust method to partition the object based on Markov random fields (MRF), which allows to incorporate prior knowledge, is presented. Shape index and curvedness descriptors along with discontinuity and concavity distributions are introduced to classify region labels correctly. In addition, a novel way to classify the shape of a surface is proposed resulting in a better distinction of concave, convex and saddle shapes. To achieve a reliable classification a multiscale method provides a stable estimation of the shape index.

This paper presents and analyses ten criteria that assess the quality of a set of three-finger grips suitable for dextrous manipulation on real 2D parts. The set of candidate hand configurations is the result of a previous process of grasp generation from the object image. The proposed criteria include six that depend on the actual finger configuration of the gripper. The kinematics of the Barrett hand has been used. The criteria are merged to give a global quality value that can be used to select the best grip to execute. Experimental results include tests on stability and the effect of parameter variation.

This paper presents new techniques for improving the structural quality of automatically acquired architectural 3D models. Common architectural features like parallelism and orthogonality of walls and edges are exploited. The location of these features is extracted from the model by using a probabilistic technique (RANSAC). The relationships among the planes and edges are inferred automatically using a knowledge-based architectural model. A numerical algorithm is used to optimise the orientations of the features. Small irregularities in the model are removed by projecting the triangulation vertices onto the features. Planes and edges in the resulting model are aligned to each other. The techniques produce models with improved appearance. We show results for synthetic and real data with consideration of noise.

This paper summarizes recent research at Edinburgh University on applying domain knowledge of standard shapes and relationships to solve or improve reverse engineering problems. The problems considered are how to enforce known relationships when data fitting, how to extract features even in noisy data, how to get better shape parameter estimates and how to infer data about unseen features.

Analysis and reconstruction of range images usually focuses on complex objects completely contained in the field of view; little attention has been devoted so far to the reconstruction of simply shaped wide areas like parts of a wall hidden behind furniture pieces in an indoor range image. The work presented in the paper is aimed at such reconstruction. First of all, the range image is partitioned based on depth discontinuities and fold edges. Next, the planes best fitting each of the regions constituting the partition of the image are determined. A third step locates potentially contiguous surfaces, while a final step reconstructs the hidden regions. The paper presents results for reconstruction of the shape of planar surfaces behind arbitrary occluding surfaces. The system proved to be effective and the reconstructed surfaces appear to be reasonable. Some examples of results are presented from the Bornholm church range images.

The ability to construct CAD or other object models from edge and range data has a fundamental meaning in building a recognition and positioning system. While the problem of model fitting has been successfully addressed, the problem of efficient high accuracy and stability of the fitting is still an open problem. In the past researchers have used approximate distance functions rather than the real Euclidean distance because of computational efficiency. We now feel that machine speeds are sufficient to ask whether it is worth considering Euclidean fitting again. This paper address the problem of estimation of elliptical cylinder and cone surfaces to 3D data by a constrained Euclidean fitting. We study and compare the performance of various distance functions in terms of correctness, robustness and pose invariance, and present our results improving known fitting methods by closed form expressions of the real Euclidean distance. 1

We present a method for estimating the viewpoint from which a 3D image has been taken using a central-projection range sensor. We assume we have the 3D coordinates of the points, organized with a known topology, but considerable noise is present in the data. At points in the scene where there are surface discontinuities, we estimate step rays through a linear interpolation. The viewpoint is found as the point of minimum distance to the set of step rays. To cope with noise, we define an unbiased distance measure. The minimization of the sum of distances provides the viewpoint. We present results of several experiments carried out with 3D images of an old church.

In this paper we present an algorithm for automatically building models of articulated objects from range data. These models not only describe the surface shape of the object but also describe the kinematics that constrain the movement of one object component in relation to another. This is more difficult than building models of rigid objects because the association of surface measurements to object components must be determined. The algorithm is demonstrated on a difficult object with free-form surfaces. 1 Introduction The ability to automatically acquire geometric models from example objects is useful in a growing number of application areas. In the field of computer graphics, the need for improvements in realism requires more complex models, but manual model construction is time-consuming and difficult. Users of Computer-Aided Design technology would like to be able to make improvements to a manufactured part and then update their CAD model to reflect this. This provides a very eff...

The paper proposes a reliable method for estimating second-order surfaces from 3D range data in the framework of object recognition and localization or object modelling. Instead of estimating such surface individually the approach ts all the surfaces captured in the scene together, taking into account the geometric relationships between them and their speci c characteristics. The technique is compared with other methods through experiments performed on real objects and demonstrates that the use of constrained relationships improves shape estimates.

This work presents a new efficient method for fitting ellipses to scattered data. Previous algorithms either fitted general conics or were computationally expensive. By minimizing the algebraic distance subject to the constraint 4ac-b/sup 2/=1, the new method incorporates the ellipticity constraint into the normalization factor. The proposed method combines several advantages: It is ellipse-specific, so that even bad data will always return an ellipse. It can be solved naturally by a generalized eigensystem. It is extremely robust, efficient, and easy to implement.

This paper presents a method for solving the Best Next View problem. This problem arises while gathering range data for the purpose of building 3Dmodels of objects. The novelty of our solution is the introduction of a quality criterion in addition to the visibility criterion used by previous researchers. This quality criterion aims at obtaining views that improve the overall range data quality of the imaged surfaces. Results demonstrate that this method selects views which generate reasonable volumetric models for convex, concave and curved objects. Keywords: Best Next Views, Sensor Planning, Voxelmap, Sphere Tessellation, 3D Scene Reconstruction, Quality Criterion. 1 Introduction When building a complete 3D model of an object, it is necessary to obtain views of the object from several directions, so that data for all surfaces of the object is acquired. Since obtaining a range scanner view is a time-consuming process, a method of automating the data acquisition process would be desir...

In this paper we consider one aspect of the problem of automatically constructing geometric models of articulated objects from multiple range images. Automatic model construction has been investigated for rigid objects, but the techniques used do nor extend easily to the articulated case. The problem arises because of the need to register surface measurements taken from different viewpoints into a common reference frame. Registration algorithms generally assume that an object does not change shape from one view to the next, but when automatically building a model of an articulated object, it is necessary for the modes of articulation to be present in the example data. To avoid this problem we propose that raw surface data of articulated objects is first segmented into rigid subsets, corresponding to rigid subcomponents of the object. This allows a model of each subcomponent to be constructed using the conventional approaches and a final, articulated model to be constructed by assembling each of the subcomponent models. We describe an algorithm developed to segment range data into rigid subsets based on surface patch correspondences and present some results for the planar patch case.

Traditional image processing teaching has used materials where the theory and drill are separated into textbooks and image processing packages. HTML and JAVA might allow easier construction of an integrated teaching resource. Such a resource would have widespread, platform-independent accessibility. This paper reports our assessment of this potential, which is explored through extensions of the HIPR teaching materials. Our conclusions are that the approach is feasible and attractive to students, that a few standard programming protocols reduce development time, and that use of compiled JAVA is essential.

This paper presents a novel approach to recognizing 3D complex ob-jects that have similar geometric structure but belong to dierent sub-classes. Test scenes are acquired by a laser striper as range images, and the objects are modelled using a composite volumetric representation of superquadrics and geons. Matching is decomposed into two stages: rst, an indexing scheme designed to make eective use of the symbolic keys of the representation is implemented in order to direct the search to the models containing the parts identied; second, a method is pro-posed where the hypotheses picked from the index are searched using an Interpretation Tree algorithm combined with a quality measure to evaluate the bindings and the nal valid hypotheses based on Possibil-ity Theory, or Theory of Fuzzy Sets. The valid hypotheses ranked by the matching process are then passed to the pose estimation module. 1

In this paper, a novel algorithm for colour recovery is presented. It assumes that the 3-D geometry of the scene is known. The spectral power distribution of a point illumination source, and the response function of the sensor are calibrated jointly. This algorithm has been used for the colour recovery part of an integrated system, developed in our laboratory, for environmental modelling. The geometry of the scene is recovered using a laser stripe range-finder and this information is exploited by the colour recovery algorithm. A point light source, attached to the whole system, has been used for the illumination of the scene, in order to confine undesirable side-effects of the ambient light. The joint spectral power distribution of this point light source and the response function of the camera are obtained with off-line calibration. 1 Introduction The objective of this paper is to present a new algorithm for obtaining surface colour information in a controled enviroment. ...

In this paper we present the attention sub-system of an iconic, picture based, vision system. Our system is based on an interest map that re-cords the saliency of potential foveation points. These saliency scores are computed on the basis of both photometric features and know-ledge of the likely relationships between object sub-components. We demonstrate our attention algorithm on articial and natural images. 1

In the context of part segmentation from 2D edge images, this paper presents some interesting results with a novel method that addresses the problem of filtering a redundant set of part hypotheses that retains only those that are likely to correspond to actual parts. In the proposed method, supporting evidence for hypotheses competes in a minimum description-length (MDL) framework to select part hypotheses that most economically represent supporting edges in the 'language' of generic parts.

This paper addresses the following problem: How can we make a complicated mathematical shape model simpler while keeping a comparable level of representational power? The proposed solution is to use the original model itself -- which represents a class of shapes -- to train a Point Distribution Model. In this paper the idea is applied to the case of deformable superellipses.

Ellipse fitting is one of the classic problems of pattern recognition and has been subject to considerable attention because of its many applications. This article presents the first direct method for specifically fitting ellipses in the least squares sense. Previous approaches used either generic conic fitting or relied on iterative methods to recover elliptic solutions. The proposed method is (i) ellipse-specific, (ii) directly solved by a generalised eigen-system, (iii) has a desirable low-eccentricity bias, and (iv) is robust to noise. We provide a theoretical demonstration, several examples and the Matlab coding of the algorithm.

When reverse engineering a CAD model, it is necessary to integrate information from several views of an object into a common reference frame. Given a rough initial alignment, further pose refinement here uses an improved version of the interactive closes point algorithm. Incremental adjustments are computed simultaneously for all data sets, resulting in a more globally optimal set of transformations. Also, thresholds for removing outlier correspondences are not needed, as the merging data sets are considered as a whole. Motion updates are computed through force-based optimization, using implied springs between data sets. Experiments indicate that even for very rough initial positionings, registration accuracy approaches 25% of the interpoint sampling resolution of the images.

This paper presents a new efficient method for fitting ellipses to scattered data. Previous algorithms either fitted general conics or were computationally expensive. By minimizing the algebraic distance subject to the constraint 4ac-b/sup 2/=1 the new method incorporates the ellipticity constraint into the normalization factor. The new method combines several advantages: 1) it is ellipse-specific so that even bad data will always return an ellipse; 2) it can be solved naturally by a generalized eigensystem, and 3) it is extremely robust, efficient and easy to implement. We compare the proposed method to other approaches and show its robustness on several examples in which other nonellipse-specific approaches would fail or require computationally expensive iterative refinements.

A methodology for evaluating range image segmentation algorithms is proposed. This methodology involves (1) a common set of 40 laser range finder images and 40 structured light scanner images that have manually specified ground truth and (2) a set of defined performance metrics for instances of correctly segmented, missed, and noise regions, over- and under-segmentation, and accuracy of the recovered geometry. A tool is used to objectively compare a machine generated segmentation against the specified ground truth. Four research groups have contributed to evaluate their own algorithm for segmenting a range image into planar patches.

This paper introduces the use of a multi-variate correlation function for region-based image matching and extends it to a modified crosscorrelation function that works well when matching image areas are required have the same intensity contrast. It also shows that the multivariate case is a straightforward generalisation of the monochrome image case. Experiments with both MRI and RGB colour imagery are shown, along with comparisons with the Euclidean, Manhatten and L1 matching metrics. 1 Introduction With the increase in available computing power, both through fast microprocessors and also through special purpose VLSI and board-level products, vision researchers have been again investigating image region-based matching processes. For example, research projects have investigated area-based stereo [3, 8], Ugaritic character stroke location [2], general template matching [1, 5, 10], MRI image correspondence determination over time [11], corner detection [6] and face recognition. What ch...

In this paper we evaluate several methods of tting data to conic sec-tions. Conic tting is a commonly required task in machine vision, but many algorithms perform badly on incomplete or noisy data. We evalu-ate several algorithms under various noise and degeneracy conditions, identify the key parameters which aect sensitivity, and present the results of comparative experiments which emphasize the algorithms' behaviours under common examples of degenerate data. In addition, complexity analyses in terms of \nop counts are provided in order to further inform the choice of algorithm for a specic application. 1

A common need in machine vision is to compute the 3-D rigid transformation that exists between two sets of points for which corresponding pairs have been determined. In this paper a comparative analysis of four popular and efficient algorithms is given. Each computes the translational and rotational components of the transform in closed-form as the solution to a least squares formulation of the problem. They differ in terms of the representation of the transform and the method of solution, using respectively: singular value decomposition of a matrix, orthonormal matrices, unit quaternions and dual quaternions. This comparison presents results of several experiments designed to determine the (1) accuracy in the presence of noise, (2) stability with respect to degenerate data sets, and (3) relative computation time of each approach.

The Hough transform is a standard technique for finding features such as lines in images.Typically edgels or other features are mapped into a partitioned parameter or Hough space as individual votes.The target image features are detected as peaks in the Hough space.In this paper we consider not just the peaks but the mapping of the entire shape boundary from image space to the Hough parameter space.We analyse this mapping and illustrate correspondences between features in Hough space and image space.Using this knowledge we present an algorithm to construct convex hulls of arbitrary 2D shapes with smooth and polygonal boundaries as well as isolated point sets.We also demonstrate its extension to the 3D case.We then show how this mapping changes as we move the origin in image space.The origin can be considered as a vantage point from which to view the object and the occluding contour can be extracted easily from Hough space as those points where R = 0. We demonstrate the potential for tracking of transitions in the mapping to be used to construct an aspect graph of arbitrary 2D and 3D shapes.

This paper focuses on the experimental evaluation of a range image segmentation system which partitions range data into homogeneous surface patches using estimates of the sign of the mean and Gaussian curvatures. The authors report the results of an extensive testing program aimed at investigating the behavior of important experimental parameters such as the probability of correct classification and the accuracy of curvature estimates, measured over variations of significant segmentation variables. Evaluation methods in computer vision are often unstructured and subjective: this paper contributes a useful example of extensive experimental assessment of surface-based range segmentation.>

This paper addresses two aspects of triangulation-based range sensors using structured laser light: calibration and measurements consistency. We present a direct calibration technique which does not require modelling any specific sensor component or phenomena, therefore is not limited in accuracy by the inability to model error sources. We also sketch some consistency tests based on two-camera geometry which make it possible to acquire satisfactory range images of highly reflective surfaces with holes. Experimental results indicating the validity of the methods are reported.

Addresses two aspects of triangulation-based range sensors using structured laser light: calibration and measurement consistency. We present a direct calibration technique which does not require the modelling of any specific sensor component or phenomena, so is not limited in accuracy by the inability to model error sources. We also introduce some consistency tests based on two-camera geometry which make it possible to acquire satisfactory range images of highly reflective surfaces with holes. Experimental results indicating the validity of the methods are reported.>

Automatic model building is a crucial requirement of any model-based vision system which must work in unknown environments. Even in the specialised environments where CAD models are available for the small number of parts, these models often lack visual saliency, impacting on the robustness (more than the mean accuracy) of the system using them. Moreover, the time needed to make such models by hand may be prohibitive, particularly with freeform curved objects; hence automatic model acquisition is greatly desirable. We present a procedure which merges multiple range images of an unmodelled object to create a 3-D body-centred model of the part. By explicitly considering the specific problems of vision systems, we achieve a high-level, robust and accurate description of the unknown scene, where visual applicability of our generated model is guaranteed. The models are characterized by a pleasant `intuitive' feel which allows easy operator intervention if they must be altered, perhaps to g...

Segmentation of 3D articulated objects into a small number of parts is an important step for recognition of objects in 3D space. Range data provides ready-to-use information to derive a reasonably stable and accurate differential structure of the surfaces, which allow us to track discontinuities between these parts. We present a mixed and comprehensive approach to segmenting and extracting prototypical parts from 3D articulated objects. Discontinuity points detected from the images are combined using a "snake-type" minimization technique, and deformable superquadric models are fitted to the resulting regions afterwards. We compare our approach with others for part-based segmentation. 1

In Fisher[1] we introduced a non-wildcard model matching algorithm that has speed advantages over the standard Interpretation Tree model matching algorithm. This paper describes a hierarchical model-matching algorithm that has improved performance over both the standard and non-wildcard algorithms. 1 Introduction The most well-known control algorithm for high-level model matching in computer vision is the Interpretation Tree(IT) expansion algorithm, as used by Grimson and Lozano-Perez[2, 3]. In Fisher[1] we introduced a variation on this algorithm that did not use a wildcard which gave performance advantages of 4-10. Both algorithms search a tree of model-to-data correspondences, such that each node in the tree represents one correspondence and the path of nodes from the current node back to the root of the tree is a set of simultaneous pairings. The non-wildcard algorithm avoids the many matches requiring wildcards and only investigates the single model-to-data pairings once, while s...

Besl and Jain's variable order surface fitting algorithm [1] is a useful method of constructing a noise-free reconstruction of 2jD range images with a small number of primitive regions. The use of bivariate polynomials as the approximation basis functions is linear, fast and easy to render robust. Seeding fits from regions classified by differential geometry is an important step towards a viewpoint invariant segmentation. However, in order to better approximate arbitrarily shaped surfaces, polynomials of high degree are needed. For a region-growing paradigm, the poor extrapolation power of high order polynomials slows convergence and generates "non-intuitive " segmentations when crossing curvature discontinuities. Such segmentations are difficult to match against traditional CAD-like models. Further, the instability of the segmentation makes invocation of the correct model from a large database extremely difficult. We show that these algorithms must of necessity trade representational richness for repeatability. In this paper we describe a new method of satisfying the requirement for high representational richness while retaining the ease of manipulation and recognition of single-extremum surface patches. By introducing a canonical reparameterised coordinate system, biquadratic patches can be made to approximate arbitrary single-extremum shapes in a viewpoint invariant manner. An iterative fitting algorithm is presented, which quickly converges to the appropriate description. Examples of the abilities of the new approach are supplied, and compared with alternative strategies. 1

We explore the potential of variance matrices to represent not just statistical error on object pose estimates but also partially constrained degrees of freedom. Using an iterated extended Kalman filter as an estimation tool, we generate, combine and predict partially constrained pose estimates from 3D range data. We find that partial constraints on the translation component of pose which occur frequently in practice are handled well by the method. However, coupled partial constraints on rotation and translation are in general to non-linear to be adequately represented. Keywords: model-based vision, geometric constraints, pose estimation 1 Introduction Most model-based part recognition or location vision systems establish all model-to-data pairings during an initial matching phase, and then estimate the pose from the consistent pairings. This is less than ideal, as insufficient features may have been segmented to estimate fully the pose, or it may be desirable to improve the pose est...

This paper describes an implementation of a distributed parallel blackboard system that runs on a Meiko multitransputer system. The blackboard is split up amongst the transputers to allow for distributed local processing, yet the access of the blackboard is transparent to the user processes, irrespective of whether the data is local or remote. Multiple expert processes are invoked as processing resources become available and task dependencies are resolved. The implementation allows both task and data parallelism. A Canny edge detector implementation achieved a speedup of 21 times on 64 transputers.

Second-order volumetric features (e.g. ridges, dents, bumps, etc) were previously defined to extend the SMS object modeling system. Here, we show that one can extract surface features from range data that can be described in this vocabulary of second-order features. The process is based on a classification of regions found by an approach based on local surface shape, and has a natural scale structure. Algorithms and results are given.

We describe a robot vision system which produces a depth map in real time by means of motion parallax or kinetic depth. A video camera is held by a robot which moves so that a given point in space is kept fixated on the centre of the camera's imaging surface. The optical flow is calculated in a Datacube MaxVideo system and a full-frame depth map is produced 12.5 times per second. Calculated depths show an average 10% discrepancy with measured depths over 7 nonconsecutive images. sponding to the direction of the target point which are closer to the robot than the target point would indicate the presence of an obstacle. The first step in this process is the construction of the kinetic depth system and the second step is to evaluate its sensitivity to system parameters. We describe the kinetic depth system here and give preliminary results of the evaluation. For an observer fixating a point in space and moving perpendicularly to the direction of gaze, objects in front of the fixation point appear to move in the opposite direction and objects behind the fixation point appear to move in the same direction. The speed of apparent motion is proportional to the distance of the object from the fixation point. This phenomenon is known as kinetic depth. If the parameters of the observer's optical system and the details of the observer's motion and fixation point are known, then measurements of the speed and direction of apparent motion of an image point (optical flow) give the distance of the corresponding object point from the observer.

An examination is made of the problem of predicting when a model surface patch on a three-dimensional object is totally back-facing, and hence need not be searched for during object recognition. Examining every point on the surface patch is inelegant and impractical, yet difficulties arise with curved surface patches. The authors conclude that visibility can be determined from an analysis of the surface orientation at the patch boundary for a wide class of model surface patch types (i.e. those having constant principal curvature signs), under orthographic projection.>