The objective of human parsing is to partition a human in an image into constituent parts. This task involves labeling each pixel of the human image according to the classes. Since the human body comprises hierarchically structured parts, each body part of an image can have its sole position distribution characteristic. Probably, a human head is less likely to be under the feet, and arms are more likely to be near the torso. Inspired by this observation, we make instance class distributions by accumulating the original human parsing label in the horizontal and vertical directions, which can be utilized as supervision signals. Using these horizontal and vertical class distribution labels, the network is guided to exploit the intrinsic position distribution of each class. We combine two guided features to form a spatial guidance map, which is then superimposed onto the baseline network by multiplication and concatenation to distinguish the human parts precisely. We conducted extensive experiments to demonstrate the effectiveness and superiority of our method on three well-known benchmarks: LIP, ATR, and CIHP databases.††Our code is available at https://github.com/tjpulkl/CDGNet.

We present a depth-based fingertip recognition method for interactive projectors. We use a depth camera attached to a projector, so it is possible to change the relative pose between the projector and the projection surface without manual recalibration. For detection and classification of fingertips, we propose using cascaded random forests boosted by our 3-D pose-normalized pixel-difference features. The ensemble probabilities from the cascaded random forests are used to define a score function of a subset of detected fingertips. By finding the subset maximizing the score function, the fingertips in the subset are correctly classified, and the remaining incorrectly detected fingertips are rejected. Experiments show that the proposed method outperforms conventional random forest and convolutional neural network classifiers. In addition, our developed applications show the advantage of the proposed method in assigning different roles to different fingers.

In the recent decades, we have witnessed the advent of local or non-local filters for cost aggregation in stereo matching, pushing the envelope of local methods to the degree of global methods, while maintaining the efficiency. A specific filter with a specific parameter setting may have a potential to best work for an image pair, but may not guarantee equally good performance for other image pairs. To address this problem, we propose a mixture-of-experts model, which applies a heterogeneous set of filters on the cost volume and adaptively combines the results. We employ supervised learning to estimate per-pixel mixing coefficients, which are used to adaptively control the weight of the filter responses. Through experiments, we show that the mixture model significantly reduces errors in disparity estimation and even outperforms the strategy of selecting the best per-pixel filter from the pool of filters in the average sense.

Depth images that have the same spatial resolution as color images are required in many applications, such as multiview rendering and 3-D texture modeling. Since a depth sensor usually has poorer spatial resolution compared with a color image sensor, many depth image super-resolution methods have been investigated in the literature. With an assumption that no one super-resolution method can universally outperform the other methods, in this paper we introduce a learning-based selection scheme for different super-resolution methods. In our case study, three distinctive mean-type, max-type, and median-type filtering methods are selected as candidate methods. In addition, a new frequency-domain feature vector is designed to enhance the discriminability of the methods. Given the candidate methods and feature vectors, a classifier is trained such that the best method can be selected for each depth pixel. The effectiveness of the proposed scheme is first demonstrated using the synthetic data set. The noise-free and noisy low-resolution depth images are constructed, and the quantitative performance evaluation is performed by measuring the difference between the ground-truth high-resolution depth images and the resultant depth images. The proposed algorithm is then applied to real color and time-of-flight depth cameras. The experimental results demonstrate that the proposed algorithm outperforms the conventional algorithms both quantitatively and qualitatively.

This paper presents a method for increasing spatial resolution of a depth map using its corresponding high-resolution (HR) color image as a guide. Most of the previous methods rely on the assumption that depth discontinuities are highly correlated with color boundaries, leading to artifacts in the regions where the assumption is broken. To prevent scene texture from being erroneously transferred to reconstructed scene surfaces, we propose a framework for dividing the color image into different regions and applying different methods tailored to each region type. For the region classification, we first segment the low-resolution (LR) depth map into regions of smooth surfaces, and then use them to guide the segmentation of the color image. Using the consensus of multiple image segmentations obtained by different super-pixel generation methods, the color image is divided into continuous and discontinuous regions: in the continuous regions, their HR depth values are interpolated from LR depth samples without exploiting the color information. In the discontinuous regions, their HR depth values are estimated by sequentially applying more complicated depth-histogram-based methods. Through experiments, we show that each step of our method improves depth map upsampling both quantitatively and qualitatively. We also show that our method can be extended to handle real data with occluded regions caused by the displacement between color and depth sensors.

Time-of-Flight depth data suffer from spatially varying noise, whose variance is inversely proportional to the squared amplitude of the received signal. On the other hand, preservation of genuine discontinuities of the scene is an important quality for a denoising method to have. This paper presents a noise-aware and discontinuity-preserving Time-of-Flight depth de-noising method. To incorporate different constraints from the two philosophies, we recast depth denoising into an iteratively reweighted least squares problem, in which the cost function is iteratively updated and minimized in a manner of preserving the discontinuities and rejecting outliers while denoising the depth data. The experiments show that the proposed method delivers better results with lower error than existing methods, irrespective of the amount of noise and discontinuities.

Time-of-Flight cameras provide real-time depth information at each pixel. These cameras, however, suffer from modular errors, recording shortened distance measurements if the objects are farther than a certain maximum range. Previous error-correction methods use either a single depth map or successively acquired multi-frequency depth maps. In general, they are prone to incorrect range extension results when the scene is discontinuous or when the camera or objects are moving. Our method based on a stereo Time-of-Flight camera deals with moving cameras and objects by acquiring a pair of depth maps simultaneously. Challenges caused by noise and different viewpoints are resolved by an iterative Markov random field optimization technique that delivers the consistent recovery of actual depth values beyond the maximum range. The experiments show that the proposed method extends the range use of Time-of-Flight cameras beyond twice.

Nonlocal means filtering is an edge-preserving denoising method whose filter weights are determined by Gaussian weighted patch similarities. The nonlocal means filter shows superior performance in removing additive Gaussian noise at the expense of high computational complexity. In this paper, we propose an efficient and effective denoising method by introducing a separable implementation of the nonlocal means filter and adopting a bilateral kernel for computing patch similarities. Experimental results demonstrate that the proposed method provides comparable performance to the original nonlocal means, with lower computational complexity.

We present a bi-layer inpainting method for synthesizing novel views from a single color image and its corresponding depth map under the exemplar-based inpainting framework. Unlike conventional image inpainting, the decisions of the inpainting direction and the sample regions are important to inpaint disoccluded regions which disclose hidden background regions in the new viewpoint. The proposed algorithm first labels boundaries along the disoccluded regions whether it belongs to the foreground or background objects, and then separates their surrounding regions into the foreground and background regions using the graph cut algorithm. The disoc-cluded regions are then filled from the background boundary with best-match patches taken from the background regions. As demonstrated in the experimental results, the proposed method recovers the disoccluded region with visually plausible quality.

Time-of-Flight depth cameras provide a direct way to acquire range images, using the phase delay of the incoming reflected signal with respect to the emitted signal. These cameras, however, have a challenging problem called range folding, which occurs due to the modular error in phase delay—ranges are modulo the maximum range. To our best knowledge, we exploit the first approach to estimate the number of mods at each pixel from only a single range image. The estimation is recasted into an optimization problem in the Markov random field framework, where the number of mods is considered as a label. The actual range is then recovered using the optimal number of mods at each pixel, so-named range unfolding. As demonstrated in the experiments with various range images of real scenes, the proposed method accurately determines the number of mods. In result, the maximum range is practically extended at least twice of that specified by the modulation frequency.

This paper presents a novel algorithm for realistic 3D face modeling from a single pair of color and time-of-flight (TOF) depth images using a generic, deformable model. Most previous approaches have attempted to emphasize either facial feature or global shape consistency, resulting in either visually plausible or geometrically accurate models. In this paper, we introduce feature-preserving surface registration to achieve both reality and accuracy. The proposed algorithm takes advantages of both facial features from the color image and geometric information from the depth image in an iterative closest point (ICP) framework, where the non-rigid registration is achieved by iteratively minimizing the energy function consisting of feature distance term, surface distance term, and linear elasticity term. As demonstrated in the experimental results, the proposed algorithm builds the personalized 3D face model within a few seconds with high visual quality.

In this paper, feature point matching is formulated as an optimization problem in which the uniqueness condition is constrained. We propose a novel score function based on homography-induced pairwise constraints, and a novel optimization algorithm based on relaxation labeling. Homography-induced pairwise constraints are effective for image pairs with viewpoint or scale changes, unlike previous pairwise constraints. The proposed optimization algorithm searches for a uniqueness-constrained solution, while the original relaxation-labeling algorithm is appropriate for finding many-to-one correspondences. The effectiveness of the proposed method is shown by experiments involving image pairs with viewpoint or scale changes in addition to repeated textures and nonrigid deformation. The proposed method is also applied to object recognition, giving some promising results.

Recently, many vision-based robotic applications such as visual SLAM (Simultaneous Localization And Mapping) and autonomous navigation have achieved good performance using visual features. In these applications, robust feature tracking plays an important role, e.g., in scene recognition for autonomous navigation and in data association for visual SLAM. In this paper, we propose a hierarchical outlier detection algorithm for robust feature tracking; the algorithm uses a simple window-based correlation (NCC) and enforces angular and scale constraints. The proposed algorithm maximizes the inter-cluster score and detects outliers that do not satisfy the angular constraints. The remaining outliers are detected by enforcing scale constraints using SIFT descriptors. The proposed algorithm is efficient and particularly useful for scene recognition, in which an image corresponding to a query image is searched among data images. Experimental results demonstrate that the proposed algorithm is robust to outliers and image variations such as scale changes. One of the main applications of the proposed algorithm is global localization due to its low computational complexity and robustness to outliers.