In this paper, we propose an automatic approach to segment object from stereo videos, for which the viewpoints are widely apart. We first present a novel saliency analysis to emphasize the foreground object. The saliency map is estimated by combining the depth information recovered by feature matching and the boundary information revealed by color segmentation. The object mask is extracted initially based on the saliency map and then refined by graph-cut segmentation, where color and motion information are efficiently incorporated in both data and smoothness terms. Moreover, a background image is gradually reconstructed during video segmentation, based on which an additional constraint is imposed on the data term to further improve the video segmentation. The proposed method is tested on stereo videos with widely separated viewpoints and severe background clutters. Good experimental results demonstrate the feasibility of the proposed method.

A depth map enhancement algorithm based on Z-displacement of objects is presented in this paper. Most depth map enhancement algorithms utilize either the spatial or temporal information only. In this paper, information in another dimension, Z-displacement, is utilized to enhance the depth map of videos by estimating the scale change of objects. Our experimental results show that the proposed scale changing detection method has higher accuracy and lower complexity compared with the conventional scale changing detection approaches such as normalized gradient correlation. Based on the estimated scale change, the proposed depth map enhancement corrects the depth map by estimating the Z-displacement of objects in the 3D space. Experimental results demonstrate that better depth maps and visual quality of view synthesis are achieved by the proposed method as compared to the state-of-arts.

Local invariant features have been successfully used in image matching to cope with viewpoint change, partial occlusion, and clutters. However, when these factors become too strong, there will be a lot of mismatches due to the limited repeatability and discriminative power of features. In this paper, we present an efficient approach to remove the false matches and propagate the correct ones for the affine invariant features which represent the state-of-the-art local invariance. First, a pair-wise affine consistency measure is proposed to evaluate the consensus of the matches of affine invariant regions. The measure takes into account both the keypoint location and the region shape, size, and orientation. Based on this measure, a geometric filter is then presented which can efficiently remove the outliers from the initial matches, and is robust to severe clutters and non-rigid deformation. To increase the correct matches, we propose a global match refinement and propagation method that simultaneously finds a optimal group of local affine transforms to relate the features in two images. The global method is capable of producing a quasi-dense set of matches even for the weakly textured surfaces that suffer strong rigid transformation or non-rigid deformation. The strong capability of the proposed method in dealing with significant viewpoint change, non-rigid deformation, and low-texture objects is demonstrated in experiments of image matching, object recognition, and image based rendering.

In this paper, we propose the depth estimation and view synthesis approaches for the scene rendering of narrow-baseline videos. Depth estimation is performed by global energy minimization using graph cut technique. A novel smoothness energy term is proposed to enforce powerful smoothness of the estimated depth. In view synthesis, smart hole filling is proposed to efficiently remove the burr artifacts along the object boundary. Experimental results show that compared with the MPEG reference software, our method can generate more accurate depth maps and achieve much better objective (PSNR) and subjective quality in view synthesis.

Most existing feature detectors assume no surface discontinuity within the keypoints' support regions and, hence, have little chance to match the keypoints located on or near the surface boundaries. These keypoints, though not many, are salient and representative. In this paper, we show that they can be successfully matched by using the proposed scale- and affine-invariant Fan features. Specifically, the image neighborhood of a keypoint is depicted by multiple fan subregions, namely Fan features, to provide robustness to surface discontinuity and background change. These Fan features are made scale-invariant by using the automatic scale selection method based on the Fan Laplacian of Gaussian (FLOG). Affine invariance is further introduced to the Fan features based on the affine shape diagnosis of the mirror-predicted surface patch. The Fan features are then described by Fan-SIFT, which is an extension of the famous scale-invariant feature transform (SIFT) descriptor. Experimental results of quantitative comparisons show that the proposed Fan feature has good repeatability that is comparable to the state-of-the-art features for general structured scenes. Moreover, by using Fan features, we can successfully match image structures near surface discontinuities despite significant scale, viewpoint, and background changes. These structures are complementary to those found by the traditional methods and are especially useful for describing weakly textured scenes, which is demonstrated in our experiments on image matching and object rendering.

Invariant local image features have proven to be very successful in computer vision tasks involving partial occlusion and various image deformations. Even though the image features can be extracted in a high repeatability, their local appearance alone usually does not bring enough discriminative power to support a reliable matching, resulting in a relatively high number of outliers in the correspondence set. To reject these mismatches, various geometric filters have been proposed for different image features. In this paper, we present a novel and efficient geometric filter for the state-of-the-art affine invariant features. The proposed method detects the mismatches by examining the consistency of local affine geometry between neighboring matches of affine invariant features. Experimental results show that the proposed geometric filter not only achieves a higher inlier ratio than the standard Hough clustering, but also presents superior robustness to severe clutters, significant viewpoint changes and non-rigid deformation.

In this paper, we aim to address the problem of automatic scale selection for image features like corners and junctions. Image neighborhood of these features usually contain the background and multiple foreground surfaces, thus can not be correctly described by a single scale. We show that the proposed Fan Laplacian-of-Gaussian (FLOG) kernel is capable to select the appropriate scales for independent image partitions that can represent meaningful physical surfaces attached to the corner or junction. Support for the proposed method is given in terms of theoretical investigation and experiments on real images.

Conventional entropy coding such as UVLC (universal variable length coding) usually uses variable length coding (VLC) based on static codeword table mechanism to code syntax elements which are partitioned from motion vectors and residual coefficients. UVLC employs half-variable length code (HVLC) such as kth order Exp-Golomb-Rice code to code syntax elements. However, static tables which indicate the average probability can not adapt the real-time probability distribution satisfactorily, so there are some needs to construct a probability updating mechanism in the codeword tables. If the real-time probability could be evaluated, Huffman code is undoubtedly the best choice for its approximation to source entropy. In PUAHC, dynamic Huffman code which is based on probability updating and Golomb code are joined together to code different source information for their adaptations to different kinds of probability distributions. In the experiment, it is proved that when PUAHC is used, the bit rate can be decreased by 3.9% and the computational complexity of decoder is also acceptable compared to UVLC