Presents an integrated approach to the derivation of scene descriptions from a pair of stereo images, where the steps of feature correspondence and surface reconstruction are addressed within the same framework. Special attention is given to the development of a methodology with general applicability. In order to handle the issues of noise, lack of image features, surface discontinuities and regions that are visible in one image only, we adopt a tensor representation for the data and introduce a robust computational technique called tensor voting for information propagation. The key contributions of this paper are twofold. First, we introduce "saliency" instead of correlation scores as the criterion to determine the correctness of matches and the integration of feature matching and structure extraction. Second, our tensor representation and voting as a tool enables us to perform the complex computations associated with the formulation of the stereo problem in 3D at a reasonable computational cost. We illustrate the steps on an example, then provide results on both random dot stereograms and real stereo pairs, all processed with the same parameter set.

Pointing at planar surfaces such as TV and computer monitors or projection screens can be a useful mode of interaction between humans and machines. To a large extent what seems to hinder the use of vision in such practical applications is the difficulty of the computational task, which is typically defined as 3-D reconstruction from uncalibrated 2-D images of a non-static scene. We describe below two designs where, using one or two cameras, the target of pointing on a flat monitor or screen is identified without 3-D inference, using only image morphing and line intersection. This is accomplished by registering the images with the target plane. When used to identify a pointing target on a surface hidden from the camera (e.g., a computer monitor which supports the camera itself as in most PC configurations), we add aperture(s) coplanar with the target surface in front of the camera(s). We describe experimental results showing a fully automated procedure for pointing target detection with high accuracy. The simplicity of our method and its robustness, as well as the relative accuracy of our results, can make pointing a practical means of human-machine interaction.

This article describes a 30 frames/second VGA format image sensor made in a standard CMOS process with an embedded massively parallel processor. The processor is fully programmable and therefore the sensor IC itself is able to run a variety of algorithms with data and processing in close vicinity of the sensor. Because of the parallel architecture comprising processor array and parallel memory accesses, high computational performances of up to 5 GOPS at 16 MHz are achieved. This high performance allowed us to implement skin tone detection on the camera itself as part of a larger system for face recognition, releasing the host computer of cumbersome pixel processing tasks and minimizing the data transfer between camera and computer.

We address the problem of epipolar geometry estimation by formulating it as one of hyperplane inference from a sparse and noisy point set in an 8D space. Given a set of noisy point correspondences in two images of a static scene without correspondences, even in the presence of moving objects, our method extracts good matches and rejects outliers. The methodology is novel and unconventional, since, unlike most other methods optimizing certain scalar, objective functions, our approach does not involve initialization or any iterative search in the parameter space. Therefore, it is free of the problem of local optima or poor convergence. Further, since no search is involved, it is unnecessary to impose simplifying assumption to the scene being analyzed for reducing the search complexity. Subject to the general epipolar constraint only, we detect wrong matches by a computation scheme, 8D tensor voting, which is an instance of the more general N-dimensional tensor voting framework. In essence, the input set of matches is first transformed into a sparse 8D point set. Dense, 8D tensor kernels are then used to vote for the most salient hyperplane that captures all inliers inherent in the input. With this filtered set of matches, the normalized eight-point algorithm can be used to estimate the fundamental matrix accurately. By making use of efficient data structure and locality, our method is both time and space efficient despite the higher dimensionality. We demonstrate the general usefulness of our method using example image pairs for aerial image analysis, with widely different views, and from nonstatic 3D scenes. Each example contains a considerable number of wrong matches.

Motion is an important visual cue for scene analysis. It is particularly useful when the scene is cluttered, such as in typical home or office environments. We present a motion segmentation algorithm that makes use of temporal differencing to detect moving people in cluttered indoor scenes. The algorithm is devised based on a couple of perceptual organization principles. To deal with missing data, noise and outliers, a robust segmentation and grouping technique called tensor voting is employed. The resulting real-time people detector can handle the presence of multiple persons, and varying body sizes and poses. It requires no initialization, uses subjective threshold, which defines the minimum saliency of "significant" motion, and the only two parameters are the scales (sizes) of the local neighborhood for region and contour analysis.

We present a novel, efficient, initialization free approach to the problem of epipolar geometry estimation, by formulating it as one of hyperplane inference from a sparse and noisy point set in an 8D space. Given a set of noisy point correspondences in two images as obtained from two views of a static scene without correspondences, even in the presence of moving objects, our method pulls out inlier matches while rejecting outliers. Unlike most methods which optimize certain objective function, our approach does not involve initialization or any search in the parameter space, and therefore is free of the problem of local optima or poor convergence. Since no search is involved, it is unnecessary to impose simplifying assumption (such as affine camera or local planar homography) to the scene being analyzed for reducing the search complexity. Subject to the general epipolar constraint only, we detect wrong matches by establishing salient "extremalities" via a naval approach, 8D Tensor Voting: the input set of matches is first transformed into a sparse and discrete 8D point set. Dense tensor kernels are then applied to vote for the most salient hyperplane (normal and intercept) that captures all inliers inherent in the input. With this filtered set of matches, the normalized Eight-point Algorithm suffices for the accurate estimation of the fundamental matrix. By using efficient data structure and locality, our method is both time and space efficient despite the higher dimensionality. We demonstrate the general usefulness of our method using example image pairs (i) for aerial image analysis, (ii) with widely different views, and (iii) from non-static 3D scenes (e.g. basketball game in an indoor stadium). Each example contains a considerable amount of wrong matches.

We present an integrated approach to the derivation of scene description from binocular stereo images. By inferring the scene description directly from local measurements of both point and line correspondences, we address both the stereo correspondence problem and the surface reconstruction problem simultaneously. We introduce a robust computational technique called tensor voting for the inference of scene description in terms of surfaces, junctions, and region boundaries. The methodology is grounded in two elements: tensor calculus for representation, and non-linear voting for data communication. By efficiently and effectively collecting and analyzing neighborhood information, we are able to handle the tasks of interpolation, discontinuity detection, and outlier identification simultaneously. The proposed method is non-iterative, robust to initialization and thresholding in the preprocessing stage, and the only critical free parameter is the size of the neighborhood. We illustrate the approach with results on a variety of images.

Synthesizing the image of a 3‐D scene as it would be captured by a camera from an arbitrary viewpoint is a central problem in Computer Graphics. Given a complete 3‐D model, it is possible to render the scene from any viewpoint. The construction of models is a tedious task. Here, we propose to bypass the model construction phase altogether, and to generate images of a 3‐D scene from any novel viewpoint from prestored images. Unlike methods presented so far, we propose to completely avoid inferring and reasoning in 3‐D by using projective invariants. These invariants are derived from corresponding points in the prestored images. The correspondences between features are established off‐line in a semi‐automated way. It is then possible to generate wireframe animation in real time on a standard computing platform. Well understood texture mapping methods can be applied to the wireframes to realistically render new images from the prestored ones. The method proposed here should allow the integration of computer generated and real imagery for applications such as walkthroughs in realistic virtual environments. We illustrate our approach on synthetic and real indoor and outdoor images.