A novel patch-based correspondence model is presented in this paper. Many segment-based correspondence approaches have been proposed in recent years. Untextured pixels and boundaries of discontinuities are imposed with hard constraints by the discontinuity assumption that large disparity variation only happens at the boundaries of segments in the above approaches. Significant improvements on performance of untextured and discontinuity area have been reported. But, the performance near occlusion is not satisfactory because a segmented region in one image may be only partially visible in the other one. To solve this problem, we utilize the observation that the shared edge of a visible area and an occluded area corresponds to the discontinuity in the other image. So, the proposed model conducts color segmentation on both images first and then a segment in one image is further cut into smaller patches corresponding to the boundaries of segments in the other when it is assigned with a disparity. Different visibility of patches in one segment is allowed. The uniqueness constraint in a segment level is used to compute the occlusions. An energy minimization framework using graph-cuts is proposed to find a global optimal configuration including both disparities and occlusions. Besides, some measurements are taken to make our segment-based algorithm suffer less from violation of the discontinuity assumption. Experimental results have shown superior performance of the proposed approach, especially on occlusions, untextured areas, and near discontinuities.

Occlusion is one of the challenging problems in stereo. In this paper, we solve the problem in a segment-based style. Both images are segmented, and we propose a novel patch-based stereo algorithm that cuts the segments of one image using the segments of the other, and handles occlusion areas in a proper way. A symmetric graph-cuts optimization framework is used to find correspondence and occlusions simultaneously. The experimental results show superior performance of the proposed algorithm, especially on occlusions, untextured areas and discontinuities

In many vision problems, the appearances of the observed images, e.g. the human facial images, are often influenced by multiple underlying factors. In this paper, a kernel-based factorization framework is proposed to analyze a multifactor dataset. Specifically, we perform N-mode singular value decomposition (N-mode SVD) in a higher dimensional feature space instead of the input space by using kernel approaches. Given an input sample, its specific underlying factors which may be all absent in the training set can be extracted and translated from one sample to another by using kernel-based 'translation'. Therefore our framework is suitable for tasks of new image synthesis and underlying factor recognition. We demonstrate the capabilities of our framework on ensembles of facial images subjected to different person identities, viewpoints and illuminations with high-quality synthetic faces and high face recognition accuracy.

Performance of conventional image restoration methods is sensitive to signal-to-noise ratios. For heavily blurred and noisy human facial images, information contained in the eigen-face subspace can be used to compensate for the lost details. The blurred image is decomposed into the eigen-face subspace and then restored with a regularized total constrained least square method. With Generalized cross-validation, a cost function is deduced to include two unknown parameters: the regularization factor and one parameter relevant to point spread function. It is shown that, in minimizing the cost function, the cost function dependence of any one unknown parameter can be separated from the other one, which means the cost function can be considered roughly, depending on single variable in an iterative algorithm. With realistic constraints on the regularized factor, a global minimum for the cost function is achieved to determine the unknown parameters. Experiments are presented to demonstrate the effectiveness and robustness of the new method.

Face hallucination is to synthesize a high-resolution facial image from a low-resolution input. In this paper, we present a new two-step approach to hallucinating faces motivated by the two-step algorithm of Liu et al. First, a linear relationship between both high-resolution and low-resolution facial images is established by applying PCA on both of them, and the global image, which is similar to the original high-resolution image, is reconstructed under a MAP criterion. Second, a linear model between the residual image (the difference between the original image and the global image) and the low-resolution residual image (the difference between the low-resolution input and the manually down-sampled global image) are built, and, following a MRF prior, the optimal residual image is estimated under a MAP criterion again. Experiments demonstrate that our approach can be applied to yield 4-8 fold super-resolution with high-quality hallucinated results.

Facial images captured by camera are usually influenced by several different kinds of factor, such as human identity, facial expression, and etc. Therefore facial expressive image analysis plays an important role in many different applications, such as expression recognition, face recognition with expressive images, facial expressive image synthesis and etc. In this paper, "human identity" and "facial expression" are regarded as two influence factors of face appearance, and a novel kernel-based bilinear factorization method is proposed to decouple the interaction from each other. The trained non-linear factorization model has been used for facial expressive image translation, and to person and/or expression recognition, and got promising results. In this paper, the principle of our method is described and some preliminary experimental results are included.

In spectral clustering algorithms, selecting the cluster number and determining the parameter of affinity function are two generally unsolved problems. In this paper we analyze in detail the influence of these two parameters on clustering results and show their close relationship. We further extend one of them, the cluster number, to intrinsic cluster number family, which is designed to achieve stable clustering hierarchy. Specifically, we use random walk on graph and eigengap to discover the intrinsic structure of the data. We proposed an algorithm to simultaneously determination the cluster number family and the parameter of affinity function. The experimental results on both simulated data clustering and natural image segmentation show that our proposed algorithm has many advantages.

It is important and challenging to make the growing image repositories easy to search and browse. Image clustering is a technique that helps in several ways, including image data preprocessing, user interface designing, and search result representation. Spectral clustering method has been one of the most promising clustering methods in the last few years, because it can cluster data with complex structure, and the (near) global optimum is guaranteed. However, existing spectral clustering algorithms, like Normalized Cut, are difficult to handle data points out of training set. In this paper, we propose a clustering algorithm named Locality Preserving Clustering (LPC), which shares many of the data representation properties of nonlinear spectral method. Yet LPC provides an explicit mapping function which is defined everywhere, both on training data points and testing points. Experimental results show that LPC is more accurate than both "direct Kmeans" and "PCA + Kmeans". We also show that LPC produces in general comparable results with Normalized Cut, yet is more efficient than Normalized Cut.

This paper, aiming at real-time gesture-controlled interaction, describes visual modeling, analysis, and recognition of continuous dynamic hand gestures. By hierarchically integrating multiple cues, a spatio-temporal appearance model and novel approaches are proposed for modeling and analysis of dynamic gestures respectively. At low level, fusion of flesh chrominance analysis and coarse image motion detection is employed to detect and segment hand gestures; at high level, parameters of the spatio-temporal appearance model are recovered by combining robust parameterized image motion estimation and hand shape analysis. The approach, therefore, fulfils real-time processing as well as high recognition rates. Without resorting to any special marks, twelve kinds of hand gestures can be recognized with average accuracy over 89%. A prototype system, gesture-controlled panoramic map browser is designed and implemented to demonstrate the usability of gesture-controlled interaction.

This paper presents a novel approach for generating and analyzing epipolar plane images (EPIs) from video sequences taken from a moving platform subject to vibration so that the 3D model of an arbitrary scene can be constructed. Two problems are solved in our approach: (1) how to generate EPIs from video under a more general motion than a pure translation; (2) how to analyze the huge amount of data in the EPIs robustly and efficiently. For the first problem, a 3D image stabilization method is proposed which decouples the vibration from the vehicle's motion so that good EPIs and panoramic view images (PVIs) can be generated. For the second problem, we propose an efficient panoramic EPI analysis (PEPIA) method in which only one scanline of each EPI is processed. The PEPIA combines advantages of PVIs and EPIs and consists of three important steps: locus orientation detection, motion boundary localization, and occlusion/resolution recovery. The output of the PEPIA-a layered 3D panorama, is very useful in visual navigation and virtual reality modeling. Since camera calibration, image segmentation, feature extraction and matching are avoided, all the proposed algorithms are fully automatic and rather general. Results on real image sequences are given.

This paper presents a system for analyzing and generating human mouth motion. We apply model-based tracking to a set of typical mouth image sequences and obtain model motion sequences, which are used to build the mouth motion space by applying principal component analysis (PCA). Given an abstract description of the mouth motion in the mouth motion space, our system can generate a new mouth motion image sequence.

This paper presents the results of integrating omnidirectional view image analysis and a set of adaptive backpropagation networks to understand the outdoor road scene by a mobile robot. Both the road orientations used for robot heading and the road categories used for robot localization are determined by the integrated system, the road understanding neural networks (RUNN). Classification is performed before orientation estimation so that the system can deal with road images with different types effectively and efficiently. An omni-view image (OVI) sensor captures images with 360 degree view around the robot in real-time. The rotation-invariant image features are extracted by a series of image transformations, and serve as the inputs of a road classification network (RCN). Each road category has its own road orientation network (RON), and the classification result (the road category) activates the corresponding RON to estimate the road orientation of the input image. Several design issues, including the network model, the selection of input data, the number of the hidden units, and learning problems are studied. The internal representations of the networks are carefully analyzed. Experimental results with real scene images show that the method is fast and robust.

A novel stereo matching algorithm has been proposed in this paper based on the observation that there are plenty of horizontal features in indoor environments. In this algorithm, the shape identity of all horizontal feature correspondences is achieved by means of a special image transformation-reprojection transformation. By using edge string as an intermediate structure, a generalized Hough transformation strategy fuses a variety of criteria, such as shape similarity disparity continuity and pixel attribution similarity, etc., together. In this way it copes with the correspondence of horizontal and non-horizontal features for 3D environment model building. First the principle of reprojection transformation using calibrated or uncalibrated images are discussed, then the generalized Hough transform strategy is addressed. Preliminary experimental results indicate that the algorithm is an effective and efficient method for indoor environment model building by using stereo methodology.

A novel 3D CAD-based vision system by using sparse structured light images is presented in this paper. By using an eye-on-hand structured light sensor, sparse range images are collected from several different positions based upon the requirements in the process of object recognition and localization. In order to recognize object with minimal sensing activities, a novel concept of maximum expected rate of hypothesis reduction (MERHR) for sensor planning is proposed its implementational procedure is carefully designed. By pre-computing the sensor's optimal configuration and storing it into a lookup table, the heavy computation burden for sensor planning during on-line recognition phase can be avoided.

This paper proposes an effective next view planning strategy for the object recognition and localization task in a model-based robot vision system. A set of rules are designed to automatically predict new features and calculate the next optimal placement of the sensor so that the most useful information can be gathered from multi-views. A state vector (i,r,t) is defined to describe the current state of the vision system and each possible state corresponds to a subset of rules to deal with it. The recognition and location task can be described as a process of rule calling and state conversions. The most suitable rule is selected at each step to try to acquire more useful information as soon as possible. Experiments are shown in the paper.>

Presents a new framework for detection of dynamic obstacles in the unstructured outdoor road environment by integrating binocular color image sequences. In the authors' system, color image segmentation, stereo obstacle detection, visual egomotion estimation, and moving object analysis, are all built-in task-oriented modules and hence are efficient and robust. Most of these functions can be performed in real-time. They are activated and integrated adaptively in the manner of neural parallel distributed processing (PDP). Experimental results are given to validate the authors' philosophy of so-called purposive vision, retinal mapping, adaptive integration and parallel distributed processing.>

Based on flat ground assumption, a correlation method is developed to estimate the orientation of a vehicle relative to its running environment. A weight function is introduced to account for figure variation caused by 3D dynamic scene and perspective effects caused by the camera system. The estimated parameters can be qualitatively used in visual navigation. Results indicate that robust estimation can be achieved in real time by using the powerful image processing system PIPE (Pipelined Image Processing Engine).>

A novel method for outdoor natural color image segmentation for road following is presented. From the results of experiments it can be seen that the hue, saturation, and intensity (HSI) system should be used rather than the red, green, blue (RGB) system, and segmentation can be done in S-I space with good performance. An automatic adaptive threshold selection method is developed. The preliminary results indicate the effectiveness and efficiency of this method.>

A novel method called height from motion (HFM) is developed to estimate the motion and structure under planar motion. By using this method both translational and rotational motion (three degrees of freedom) can be treated in a unified manner. Based on the HFM method, the correspondence problem becomes easy to deal with, especially under translational motion. Experiments of real scene image sequences and the error analysis (theoretically and experimentally) have shown the efficiency and robustness of the method.>

Our goal is to develop a vision system using range image for the industrial application. In the system, simple basic surface primitives are considered and are extracted from the range image. From the robust geometric features contained in the primitives, rotation-invariant characteristics are then extracted to form smallest distinctive feature sets (SDFS). The set of SDFSs and the relationships between elements in SDFS and the kernel are stored to form the object model. The recognition process involves first extracting a set of SDFSs from the image and matching the extracted SDFSs with the stored SDFSs. If the kernels determined for SDFSs matched are consistent, object can be detected and its location and orientation in the viewer-centered coordinate system (VCC) can be determined.

The original algorithm of Hough transform was for detecting and finding straight lines in two dimensional images. Since then, it has been extended to detect and locate special planar curves such as circles, parabolas, etc. Recently, generalized Hough transform has been applied for recognizing and locating three dimensional objects using range data. Our work is based on the same principle by decomposing the overall parameter set of nine elements representing a quadric into three decoupled parameter subsets: scalar parameter, translational parameter, and orientational parameters. The use of Hough transform on each decoupled subset in an independent but sequential manner rather than the whole enables us to reduce the computational cost and storage requirements by orders of magnitude as compared to all previous published applications.