In this paper, we propose an efficient method to reconstruct surface-from-gradients (SfG). Our method is formulated under the framework of discrete geometry processing. Unlike the existing SfG approaches, we transfer the continuous reconstruction problem into a discrete space and efficiently solve the problem via a sequence of least-square optimization steps. Our discrete formulation brings three advantages: 1) the reconstruction preserves sharp-features, 2) sparse/incomplete set of gradients can be well handled, and 3) domains of computation can have irregular boundaries. Our formulation is direct and easy to implement, and the comparisons with state-of-the-arts show the effectiveness of our method.

We address how a human-computer interface with small device size, large display, and touch-input facility can be made possible by a mere projector and camera. The realization is through the use of a properly embedded structured light sensing scheme that enables a regular light-colored table surface to serve the dual roles of both a projection screen and a touch-sensitive display surface. A random binary pattern is employed to code structured light in pixel accuracy, which is embedded into the regular projection display in a way that the user perceives only regular display but not the structured pattern hidden in the display. With the projection display on the table surface being imaged by a camera, the observed image data, plus the known projection content, can work together to probe the 3-D workspace immediately above the table surface, like deciding if there is a finger present and if the finger touches the table surface, and if so, at what position on the table surface the contact is made. All the decisions hinge upon a careful calibration of the projector-camera-table surface system, intelligent segmentation of the hand in the image data, and exploitation of the homography mapping existing between the projector's display panel and the camera's image plane. Extensive experimentation including evaluation of the display quality, hand segmentation accuracy, touch detection accuracy, trajectory tracking accuracy, multitouch capability and system efficiency are shown to illustrate the feasibility of the proposed realization.

We address the problem of recovering camera motion from video data, which does not require the establishment of feature correspondences or computation of optical flows but from normal flows directly. We have designed an imaging system that has a wide field of view by fixating a number of cameras together to form an approximate spherical eye. With a substantially widened visual field, we discover that estimating the directions of translation and rotation components of the motion separately are possible and particularly efficient. In addition, the inherent ambiguities between translation and rotation also disappear. Magnitude of rotation is recovered subsequently. Experimental results on synthetic and real image data are provided. The results show that not only the accuracy of motion estimation is comparable to those of the state-of-the-art methods that require explicit feature correspondences or optical flows, but also a faster computation time.

We address the problem of motion recovery and depth map reconstruction from a sequence of stereo images. The proposed method utilizes normal flows which are directly observable in image sequence. No explicit correspondence establishment is required. We first determine the direction of translation and the full rotation for each of the cameras in the stereo system using an iterative algorithm. Then we recover the translation magnitude using a geometrical method. Depth map in metric scale is reconstructed once the motion parameters are estimated. Experimental results on the synthetic and the real image data show that the proposed method is effective in both motion determination and depth map reconstruction.

We present a direct method for estimating general camera motion from a rectified stereo image sequence using merely normal flows. We require neither point-to-point feature correspondences nor special assumption about the imaged scene. It has a closed-form expression, requiring no iterative computation. Experimental results on both synthetic and real image data are provided to demonstrate the performance of the proposed method.

We describe a system of embedding codes into projection display for structured light-based sensing, with the purpose of letting the projector serve as both a display device and a 3-D sensor. The challenge is to make the codes imperceptible to human eyes so as not to disrupt the content of the original projection. There is the temporal resolution limit of human vision that one can exploit, by having a higher than necessary frame rate in the projection and stealing some frames for code projection. Yet, there is still the conflict between imperceptibility of the embedded codes and the robustness of code retrieval that has to be addressed. We introduce noise-tolerant schemes to both coding and decoding stages. At the coding end, specifically designed primitive shapes and large Hamming distance are employed to enhance tolerance toward noise. At the decoding end, pretrained primitive shape detectors are used to detect and identify the embedded codes, a task that is difficult to achieve by segmentation that is used in general structured light methods, because the weakly embedded information is generally interfered by substantial noise. Extensive experiments show that the proposed system is effective, even with the prerequisite of incurring minimum disturbance to the original projection.

We describe a method of embedding imperceptible codes, that are for structured light sensing (SLS), into regular display in video projection. To resolve the conflict between imperceptibility of the embedded codes and feasibility and robustness of SLS code retrieval, we introduce antinoise schemes to the coding and decoding stages simultaneously. On one hand, specially designed primitive shapes and large Hamming distance are employed in the spatial SLS coding to enhance the error-tolerance capability. On the other hand, pre-trained primitive shape detectors are applied to detect and identify the embedded codes in the decoding stage - a task difficult to achieve by segmentation that is adopted in classical structured light methods, when weakly embedded information is inevitably interfered by noise. Extensive experiments show that the proposed method is effective and accurate in code retrieval, even with the prerequisite of incurring minimum disturbance to arbitrary video projection. Some potential applications to a robotic system are also demonstrated.

We describe a method of estimating head pose in space by imperceptible structured light sensing. Firstly, through elaborate pattern projection strategy and camera-projector synchronization, pattern-illuminated images of the subject and the corresponding scene-texture image are captured under imperceptible patterned illumination. 3D positions of the key facial feature points are then derived by a combined use of (1) the 2D facial feature points in the scene-texture image that are localized by AAM, and (2) the point cloud generated by structured light sensing. Eventually, the head orientation and translation are estimated by SVD of a correlation matrix that is generated from the 3D corresponding feature point pairs over the various image frames. Extensive experiments show that the proposed method is effective, accurate, and fast in 6-DOF head pose estimation, making it suitable for use in real-time applications.

We address how human pose in 3D can be tracked from a monocular video using a probabilistic inference method. Human body is modeled as a number of cylinders in space, each with an appearance facet as well as a pose facet. The appearance facets are acquired in a learning phase from some beginning frames of the input video. On this the visual hull description of the target human subject constructed from multiple images is found to be instrumental. In the operation phase, the 3D pose of the target subject in the subsequent frames of the input video is tracked. A bottom-up framework is used, which for any current image frame extracts firstly the tentative candidates of each body part in the image space. The human model, with the appearance facets already learned, and with the pose entries initialized with those for the previous image frame, is then brought in under a belief propagation algorithm, to establish correlation with the above 2D body part candidates while enforcing the proper articulation between the body parts, thereby determining the 3D pose of the human body in the current frame. The tracking performance on a number of monocular videos is shown.

Position and orientation profiles are two principal descriptions of shape in space. We describe how a structured light system, coupled with the illumination of a pseudorandom pattern and a suitable choice of feature points, can allow not only the position but also the orientation of individual surface elements to be determined independently. Unlike traditional designs which use the centroids of the illuminated pattern elements as the feature points, the proposed design uses the grid points between the pattern elements instead. The grid points have the essences that their positions in the image data are inert to the effect of perspective distortion, their individual extractions are not directly dependent on one another, and the grid points possess strong symmetry that can be exploited for their precise localization in the image data. Most importantly, the grid lines of the illuminated pattern that form the grid points can aid in determining surface normals. In this paper, we describe how each of the grid points can be labeled with a unique color code, what symmetry they possess and how the symmetry can be exploited for their precise localization at subpixel accuracy in the image data, and how 3D orientation in addition to 3D position can be determined at each of them. Both the position and orientation profiles can be determined with only a single pattern illumination and a single image capture.

The problem we address is: Given line correspondences over three views, what is the condition of the line correspondences for the spatial relation of the three associated camera positions to be uniquely recoverable? The observed set of lines in space is called critical if there are multiple projectively nonequivalent configurations of the camera positions that can picture the same image triplet of the lines. We tackle the problem from the perspective of trifocal tensor, a quantity that captures the relative pose of the cameras in relation to the captured views. We show that the rank of a matrix that leads to the estimation of the tensor is reduced to 7, 11, 15 if the observed lines come from a line pencil, a line bundle, and a line field, respectively, which are line families belonging to linear line space; and 12, 19, 23 if the lines come from a general linear ruled surface, a general linear line congruence, and a general linear line complex, which are subclasses of linear line structures. We show that the above line structures, with the exception of linear line congruence and linear line complex, ought to be critical line structures. All of these structures are quite typical in reality, and thus, the findings are important to the validity and stability of practically all algorithms related to structure from motion and projective reconstruction using line correspondences.

Structured light system together with the use of pseudo-randomly coded pattern in the projection is an effective solution for 3D reconstruction; it requires only a single image capture to operate. In this article, a 2D pseudo-random pattern consisting of rhombic color elements is proposed, and the grid-points between the pattern elements as opposed to the centroids of the elements are adopted as the feature points. Two possible types of grid-point are described, and a scheme that allows each grid-point to be uniquely distinguished by a codeword and the grid-point type it belongs to is described. We also present a grid-point detector that is based upon a certain symmetry the grid-points of rhombic pattern own — the 2-fold rotation symmetry — which is largely preserved under pattern projection, reflection, perspective distortion, image noise, and image blur.

We address the question of, what structure of a set of lines in space constitutes a critical configuration to three generally positioned cameras. By critical configuration, it is meant a set of lines in space whose image projections to the cameras do not allow unique determination of the cameraspsila relative positions in space. We approach the question by looking into the trifocal tensor of the cameraspsila a quantity tightly related to the camera geometry. We focus on structures of lines in space that are common in reality - the linear structures - and examine which of them would not allow the tensor to be uniquely determined. Linear structures of lines include linear ruled surface, linear line congruence, and linear line complex, and more specifically line pencil, point star, and ruled plane. We offer a summary of by how much such families of line set are shy of the general enough structure for a unique determination of the tensor. We also point out how many lines need be visible minimally in each of them, for the full information of the associated structure to be revealed in the image data. We present synthetic and real image results to confirm the findings. The findings are important to the validity and stability of algorithms related to structure from motion and projective reconstruction using line correspondences.

In advanced electronic manufacturing that involves say die-to-die bonding, microscopic surfaces like solder bumps on wafers have to be inspected in 3D. However, because the bumps are of hemispherical shape, light projected onto the bumps could be reflected and illuminate other regions. Such multi-reflections could greatly disturb the intensity distribution in the image data and limit the use of gray level intensities for accurate 3D reconstruction of the bumps. In a previous work, we described a new solution mechanism that was based upon the concept of binary pattern projection, but unlike the traditional mechanisms which use an array of light sources it uses only a single light source. The light source in combination with a binary fringe grating could induce binary pattern on the target surface to be imaged, and the displacements of the binary fringe grating could allow the binary pattern to be varied. In this work, we describe under that solution framework how multi-reflections could be detected and the correct binary signals could be restored. Experimental results on solder bumps validate the feasibility of the proposed approach.

Recommender Systems belong to a class of systems intended to assist individuals make evaluations about entities in meaningful ways. In this paper we discuss the issues in the design of integrated recommender systems and suggest a framework that takes the perspective of an individual functioning in multiple domains. This is particularly applicable today with the rapidly increasing diffusion of personalized, networked mobile devices. We present some preliminary design ideas in the form of a functional prototype.

Calibrating the relative geometry between cameras which would move against one another from time to time is an important problem in multi-camera system. Most of the existing calibration technologies are based on the cross-camera feature correspondences. This paper presents a new solution method. The method demands image data captured under a rigid motion of the camera pair, but unlike the existing motion correspondence-based calibration methods, it does not estimate optical flows nor motion correspondences explicitly. Instead it estimates the inter-camera geometry from the observations that are directly available from the two image streams -the monocular normal flows. Experimental results on real image data are shown to illustrate the feasibility of the solution.

Brittle material like ceramic and rock has the nature that it is hard to crack, but if it is to crack it will crack into not dangling pieces, but separate pieces whose boundaries are contours that link up positions of strong stress in a continuous and often smooth fashion. Image segmentation desires similar results: it is to partition the image so that each partition corresponds to a separate surface or object in the imaged scene, whose boundaries are continuous and often smooth contours with a dense concentration of large intensity gradients. This work explores the adaptation of brittle fracture mechanics model toward image segmentation. An image is regarded as a plate experiencing brittle fracture when external forces arc exerted on it according to intensity gradient distribution in the image. Stress concentrations due to the external forces and its change over time due to changing crack distribution are what determine the development of cracks. Real image results indicate that the model is promising.

Robot tracking and intercepting fast-maneuvering object is a classical and important issue. Many research results were published in recent years. Most of them employed model-based methods which require robot's model in advance. However, it is difficult and time-consuming to obtain robot's mathematical model. In this paper, we present a novel approach which needs no mathematical model. The proposed approach is based on learning tracking strategy from human beings. With human's demonstrations, the robot can learn and abstract human tracking and intercepting skill using cascade neural network. Preliminarily simulation results attest the feasibility of this novel approach. Furthermore, experiment is done on a real-time human face tracking system and the results verify the validity and efficiency of the approach.

Synthesizing images of a real scene at novel viewpoints other than those of some given images is important for image-based rendering, hybrid reality, and other applications that involve real scenes. Novel view synthesis has been dealt with via explicit 3D reconstruction, image transfer, or plenoptic modeling. In this paper we present an interpolation-based solution that avoids the need of explicit 3D reconstruction. The key difficulty in the interpolation approach is, as viewpoint changes, not only could a scene feature's intensity profile in the image change, its image position could also shift. We show that a novel interpolation scheme, which we refer to as iEBI (indexed function example-based interpolation) mechanism, could tackle the difficulty and acquire quality novel views even from only a few example views. Experimental result on some benchmarking image data is shown to illustrate the solution's performance.

The established approach of stereo vision is first to establish correspondences across the input images and then to determine 3D at the intersection of the corresponding projection rays. However, correspondence establishment is often difficult. Aloimonos and Herve (1990) presented an algorithm that could recover the 3D structure of planar surface without establishing point-to-point correspondences. Yet, the algorithm is restricted to images taken under the parallel-axis stereo geometry. In this paper, we present a correspondence-free stereo vision algorithm that could handle image data taken from arbitrarily positioned cameras. The new algorithm has a closed-form expression, requiring no iterative computation. The algorithm is applicable to both planar scenes and scenes that are pictured from a far distance. Experiment results with real image data are presented to illustrate the performance of the algorithm.

This paper addresses the problem of geometry determination of a stereo rig that undergoes general rigid motions. Neither known reference objects nor stereo correspondence are required. With almost no exception, all existing online solutions attempt to recover stereo geometry by first establishing stereo correspondences. We first describe a mathematical framework that allows us to solve for stereo geometry, i.e., the rotation and translation between the two cameras, using only motion correspondence that is far easier to acquire than stereo correspondence. Second, we show how to recover the rotation and present two linear methods, as well as a nonlinear one to solve for the translation. Third, we perform a stability study for the developed methods in the presence of image noise, camera parameter noise, and ego-motion noise. We also address accuracy issues. Experiments with real image data are presented. The work allows the concept of online calibration to be broadened, as it is no longer true that only single cameras can exploit structure-from-motion strategies; even the extrinsic parameters of a stereo rig of cameras can do so without solving stereo correspondence. The developed framework is applicable for estimating the relative three-dimensional (3D) geometry associated with a wide variety of mounted devices used in vision and robotics, by exploiting their scaled ego-motion streams.

An obstacle detection and avoidance system that uses a stereo pair of cameras but needs no 3D reconstruction of the surroundings is described. The system is designed for a challenging mobile platform: a legged robot, whose locomotion is not as smooth as the wheeled robots' and thus induces more disturbance to the visual data. The concept of the image-to-image mapping named homography induced by the ground is exploited for detecting obstacles. The described system takes advantage of the fact that obstacles are not on the ground and thus their projections in the images would not agree with such a mapping. Such a homography is allowed to change over time as the legged motion takes place. Obstacles can thus be detected in terms of a 2D distribution on the ground we call the local obstacle map (LOM). Based on the LOM which is further time-delay compensated, a fuzzy control mechanism is then used to command the robot motion. Experiments show that the system is effective for avoiding walls, making turns at corners, and withdrawing from dead-ends during navigation in textured environments, in real-time.

This paper presents a new approach of combining stereo vision and dynamic vision with the objective of retaining their advantages and removing their disadvantages. It is shown that, by assuming affine cameras, the stereo correspondences and motion correspondences, if organized in a particular way in a matrix, can be decomposed into: the 3D structure of the scene, the camera parameters, the motion parameters, and the stereo geometry. With this, the approach can infer stereo correspondences from motion correspondences, requiring only a time linear with respect to the size of the available image data. The approach offers the advantages of simpler correspondence, as in dynamic vision, and accurate reconstruction, as in stereo vision, even with short image sequences.

This paper introduces a new framework for stereo correspondence recovery using one motion of a stereo rig. Both the stereo correspondence and the motion of the stereo rig are unknown. By combining the stereo geometry and the motion correspondence we are able to infer the stereo correspondence from motion correspondence without having to systematically use the intensity-based stereo matching algorithms. The stereo correspondence recovery consists of two consecutive steps: the first step uses metric data associated with the stereo rig while the second step uses feature correspondences only. Experiments involving real stereo pairs indicate the feasibility and robustness of the approach.

Recovering three-dimensional (3D) information of a scene from its images is a fundamental problem in computer vision. There exists two major multi-ocular cue for it, namely the motion cue and the stereo cue. This paper presents a new approach of integrating the two cues when two cameras that move through the scene while taking pictures repeatedly are available. The approach is based on the Singular Value Decomposition (SVD) technique, with which the 3D structure of the scene, the image projection parameters, the motion parameters, and the stereo geometry are all separated. The approach offers the advantages of both cues: simple correspondence as well as accurate reconstruction. It can also work with relatively short image sequences.

Reconstructing the 3D boundary of a structure from tomographic data is an important problem in medical imaging. Various forms of the active surface model have been proposed to tackle it, but they are either computationally expensive or requiring extensive guidance from the user. We propose an active model that is conformable to any 3D shape, easy to use, efficient, and is highly parallelizable. It requires only an initial contour on one of the tomographs to start with. The essential idea is to break down the highly interconnected 3D deformation system into 2D pieces that work independently to save computations, and that deform in all directions to wrap around noise spots while conforming to the structure's shape.

Recovering three-dimensional shape of an object from a single line drawing is a classical problem in computer vision. Methods proposed range from Huffman–Clowes junction labeling, to Kanade's gradient space and skew symmetry analysis, to Sugihara's necessary and sufficient condition for a realizable polyhedral object, to Marill's MSDA shape recovery procedure, to Leclerc–Fischler's shape recovery procedure which assures planar faces, and to the recent Baird–Wang's gradient-descent algorithm which has a favorable time complexity. Yet all these assume perfect line drawings as the input. We propose a method that through the use of iterative clustering interprets an imperfect line drawing of a polyhedral scene. It distinguishes the true surface boundaries from the extraneous ones like the surface markings, fill-in the missing surface boundaries, and recovers 3-D shapes satisfying constraints of planarity of faces and parallel symmetry of lines, all at the same time. Experiments also show that the 3-D interpretation agrees with human perception.

Recovering three-dimensional shape of an object from a single line drawing is a classical problem in computer vision. Methods proposed range from Huffman-Clowes junction labeling, to Kanade's gradient space and skew symmetry analysis, to Sugihara's necessary and sufficient condition for a realizable polyhedral object, to Marill's MSDA shape recovery procedure, and to the recent Leclerc-Fischler's shape recovery procedure which assures planar faces. Yet all these assume perfect line drawings as the input. We propose a method that through the use of iterative clustering interprets an imperfect line drawing of a polyhedral scene. It distinguishes the true surface boundaries from the false ones like the surface markings, fill-in the missing surface boundaries, and recovers 3-D shapes satisfying constraints of planarity of faces and parallel symmetry of lines, all at the same time. Experiments also show that the 3-D interpretation agrees with human perception.

Stereo images have to be calibrated before stereo vision can recover the 3D information of the imaged scene. Position constraints via point correspondences are traditionally used to solve the calibration problem. We describe a method that uses correspondences of projections of orthogonal trihedral vertices for calibration. The method has a closed-form solution, as opposed to many other calibration methods which are iterative. It also requires only two vertex correspondences at minimum to recover all the transformation parameters which are recoverable from a stereo image pair. Extensive experimental results including those on real images are presented, and they show that our method is more robust than those using position constraints alone.

Addresses the problem of extracting polyhedral building structures from a stereo pair of aerial intensity images. The authors describe a system that computes a hierarchy of descriptions such as segments, junctions, and links between junctions from each view, and matches these features at the different levels. Such high level features not only help reduce correspondence ambiguity during stereo matching, but also allow us to infer surface boundaries even though the boundaries may be broken because of noise and weak contrast. The authors hypothesize surface boundaries by examining global information such as continuity and coplanarity of linked edges in 3-D, rather than merely by looking at local depth information. When the walls of the buildings are visible, they also exploit the relationship among adjacent surfaces in a polyhedral object to help confirm the different levels of descriptions. The authors give some experimental results for aerial images taken from overhead views and oblique views.>

A hierarchical stereo system is described that uses structural descriptions up to the surface level. Surface descriptions are computed from monocular images, by using a perceptual grouping technique. Occlusion can be a major problem in stereo analysis and is often not treated explicitly. An analysis is presented of occlusion effects in stereo, and it is shown how structural descriptions can be used to deal with them. Experimental results are given for scenes with curved objects and significant occlusions.>