This paper describes a non-convex model that is carefully designed for high quality depth upsampling. Modern depth sensors such as time-of-flight cameras provide a promising depth measurement with video rate, but suffer from noise and low resolution. To tackle these limitations, we formulate an optimization problem using a robust potential function. In this formulation, a nonlocal principle established in the high-dimensional feature space is used to disambiguate the up-sampling problem. We also derive a numerical algorithm based on the majorization-minimization approach for efficient optimization. The proposed model iteratively creates a new affinity space that determines the influence of neighboring pixels by jointly considering spatial distance, appearance, and current estimates. This behavior enables one to significantly reduce annoying artifacts on a variety of range dataset, including a challenging real measurement. Extensive experiments demonstrate that the proposed model achieves competitive performance with state-of-the-art methods.

Estimating depth from a single monocular image is a fundamental problem in computer vision. Traditional methods for such estimation usually require complicated and sometimes labor-intensive processing. In this paper, we propose a new perspective for this problem and suggest a new gradient-domain learning framework which is much simpler and more efficient. Inspired by the observation that there is substantial co-occurrence of image edges and depth discontinuities in natural scenes, we learn the relationship between local appearance features and corresponding depth gradients by making use of the K-means clustering algorithm within the image feature space. We then encode each cluster centroid with its associated depth gradients, which defines visual-depth words that model the image-depth relationship very well. This enables one to estimate the scene depth for an arbitrary image by simply selecting proper depth gradients from a compact dictionary of visual-depth words, followed by a Poisson surface reconstruction. Experimental results demonstrate that the proposed gradient-domain approach outperforms state-of-the-art methods both qualitatively and quantitatively and is generic over (unseen) scene categories which are not used for training.

Inferring scene depth from a single monocular image is a highly ill-posed problem in computer vision. This paper presents a new gradient-domain approach, called depth analogy, that makes use of analogy as a means for synthesizing a target depth field, when a collection of RGB-D image pairs is given as training data. Specifically, the proposed method employs a non-parametric learning process that creates an analogous depth field by sampling reliable depth gradients using visual correspondence established on training image pairs. Unlike existing data-driven approaches that directly select depth values from training data, our framework transfers depth gradients as reconstruction cues, which are then integrated by the Poisson reconstruction. The performance of most conventional approaches relies heavily on the training RGB-D data used in the process, and such a dependency severely degenerates the quality of reconstructed depth maps when the desired depth distribution of an input image is quite different from that of the training data, e.g., outdoor versus indoor scenes. Our key observation is that using depth gradients in the reconstruction is less sensitive to scene characteristics, providing better cues for depth recovery. Thus, our gradient-domain approach can support a great variety of training range datasets that involve substantial appearance and geometric variations. The experimental results demonstrate that our (depth) gradient-domain approach outperforms existing data-driven approaches directly working on depth domain, even when only uncorrelated training datasets are available.

This paper presents a texture flow estimation method that uses an appearance-space clustering and a correspondence search in the space of deformed exemplars. To estimate the underlying texture flow, such as scale, orientation, and texture label, most existing approaches require a certain amount of user interactions. Strict assumptions on a geometric model further limit the flow estimation to such a near-regular texture as a gradient-like pattern. We address these problems by extracting distinct texture exemplars in an unsupervised way and using an efficient search strategy on a deformation parameter space. This enables estimating a coherent flow in a fully automatic manner, even when an input image contains multiple textures of different categories. A set of texture exemplars that describes the input texture image is first extracted via a medoid-based clustering in appearance space. The texture exemplars are then matched with the input image to infer deformation parameters. In particular, we define a distance function for measuring a similarity between the texture exemplar and a deformed target patch centered at each pixel from the input image, and then propose to use a randomized search strategy to estimate these parameters efficiently. The deformation flow field is further refined by adaptively smoothing the flow field under guidance of a matching confidence score. We show that a local visual similarity, directly measured from appearance space, explains local behaviors of the flow very well, and the flow field can be estimated very efficiently when the matching criterion meets the randomized search strategy. Experimental results on synthetic and natural images show that the proposed method outperforms existing methods.

Exploring underlying texture flows defined with orientation and scale is of a great interest on a variety of vision-related tasks. However, existing methods often fail to capture accurate flows due to over-parameterization of texture deformation or employ a costly global optimization which makes the algorithm computationally demanding. In this paper, we address this inverse problem by casting it as a randomized correspondence search along with a locally-adaptive vector field smoothing. When a small example patch is given as a reference, a randomized deformable matching is performed on the very densely quantized label space, enabling an efficient estimation of texture deformation without quality degeneration, e.g., due to quantization artifacts which often appear in the optimization-driven discrete approaches. The visual similarity with respect to the deformation parameters is directly measured with an input texture image on an appearance space. The locally-adaptive smoothing is then applied to the intermediate flow field, resulting in a good continuation of the resultant texture flow. Experimental results on both synthetic and natural images show that the proposed method improves the performance in terms of both runtime efficiency and/or visual quality, compared to the existing methods.

In this paper, a data-driven approach is proposed for automatically estimating a plausible depth map from a single monocular image based on the weighted median statistics (WMS). Instead of using complicated parametric models for learning frameworks that are typically employed in existing methods, we cast the estimation as a simple yet effective statistical approach. It assigns perceptually proper depth values to an input image in accordance with a data-driven depth prior. Based on the assumption that similar scenes are likely to have similar depth structure, the depth prior is computed from the WMS of k-nearest neighbor 3D pairs in a large 3D image repository. We show that the WMS captures the underlying depth structure of the input image very well, even though the visual appearance of nearest neighbor images are not tightly aligned. The depth map is then inferred according to the depth prior by making use of the edge-aware image filtering technique, resulting in a discontinuity-preserving smooth depth map. Experimental results demonstrate that our method outperforms state-of-the-art methods in terms of both accuracy and efficiency.

This paper presents an efficient technique for performing a spatially inhomogeneous edge-preserving image smoothing, called fast global smoother. Focusing on sparse Laplacian matrices consisting of a data term and a prior term (typically defined using four or eight neighbors for 2D image), our approach efficiently solves such global objective functions. In particular, we approximate the solution of the memory-and computation-intensive large linear system, defined over a d-dimensional spatial domain, by solving a sequence of 1D subsystems. Our separable implementation enables applying a linear-time tridiagonal matrix algorithm to solve d three-point Laplacian matrices iteratively. Our approach combines the best of two paradigms, i.e., efficient edge-preserving filters and optimization-based smoothing. Our method has a comparable runtime to the fast edge-preserving filters, but its global optimization formulation overcomes many limitations of the local filtering approaches. Our method also achieves high-quality results as the state-of-the-art optimization-based techniques, but runs ∼10-30 times faster. Besides, considering the flexibility in defining an objective function, we further propose generalized fast algorithms that perform Lγ norm smoothing (0 < γ < 2) and support an aggregated (robust) data term for handling imprecise data constraints. We demonstrate the effectiveness and efficiency of our techniques in a range of image processing and computer graphics applications.

In this paper, we propose an image retargeting method that resizes an image by the axis-aligned importance scaling. The proposed method operates on the axis-aligned deformation space, where the mesh structure is parameterized in the 1D vector, i.e., quads on the same column (or row) share the single parameter. The unknown variables are thus dependent on the grid resolution only, allowing a fast and simple implementation on a moderate CPU. The optimal parameters are inferred in an iterative manner: these parameters are updated by scaling initial ones according to the deformation error. It is measured at each iteration by aggregating the transition cost for the deformed quad. Experimental results show that the proposed method preserves visually salient features without foldover artifacts better than competing methods. In addition, the optimal parameters can be calculated within 0.02 ms on a single-core CPU.

A well-known problem in stereoscopic videos is visual fatigue. However, conventional depth adjustment methods provide little guidance in deciding the amount of depth control or determining the condition for depth control. We propose a depth adjustment method based on binocular fusion characteristics, where the fusion time is used as the parameter for adjustment. Visual comfort enhancement is implemented with the horizontal image shift approach for depth adjustment. The speeded-up robust feature is used to estimate maximum disparity and disparity distribution, while a face detection algorithm is used to estimate the viewing distance with a single web camera. Binocular fusion characteristics are acquired in advance by measuring the time required for fusion under various conditions, including foreground disparity, background disparity, and focal distance for random-dot stereograms. Finally, a subjective evaluation is conducted in fixed and free-to-move viewing conditions, and the results show that comfortable videos were generated based on the proposed depth adjustment method.

In this paper, a space-time hole filling approach is presented to deal with a disocclusion when a view is synthesized for the 3D video. The problem becomes even more complicated when the view is extrapolated from a single view, since the hole is large and has no stereo depth cues. Although many techniques have been developed to address this problem, most of them focus only on view interpolation. We propose a space-time joint filling method for color and depth videos in view extrapolation. For proper texture and depth to be sampled in the following hole filling process, the background of a scene is automatically segmented by the random walker segmentation in conjunction with the hole formation process. Then, the patch candidate selection process is formulated as a labeling problem, which can be solved with random walks. The patch candidates that best describe the hole region are dynamically selected in the space-time domain, and the hole is filled with the optimal patch for ensuring both spatial and temporal coherence. The experimental results show that the proposed method is superior to state-of-the-art methods and provides both spatially and temporally consistent results with significantly reduced flicker artifacts.

Recently, various studies of human factors have been conducted to reveal stereoscopic characteristics of the human visual system and visual fatigue. In this paper, we investigate the effect of vergence-accommodation conflict and parallax difference on binocular fusion for random dot stereograms. The aim of this paper is to provide a study on visual fatigue induced by the conflict. We measured the time required for fusion under various conditions that include foreground parallax, background parallax, focal distance, aperture size, and corrugation frequency. The results show that foreground parallax and parallax difference between foreground parallax and background parallax have significant influences on fusion time. In addition, we verify the relationship between fusion time and visual fatigue by conducting a subjective evaluation of stereoscopic images.

In this paper, we propose a hole filling technique which coherently reconstructs the hole region during the view synthesis. The holes can be filled successfully in case that the virtual camera locates between real cameras by using interpolation. However, they cannot be handled in case that the virtual camera locates beyond the field of view of the real camera. We address this problem by jointly using image completion technique and random walks. First, occlusion constraint is imposed in order to guide the filling order. It is observed that the holes occur in a similar pattern because of the geometric characteristic of the camera configuration. This observation named vertical prior in this paper is also used to label each pixel on the fill front with foreground or background. Second, the probabilities estimated by random walks are utilized to find the patch candidates and to select the optimal patch. The experimental results show that the proposed method gives visually pleasing results over both interpolation and conventional image completion method.

A 3D video is expected to be a representative technique of realistic system but still has some problems such as visual fatigue and headache. In this paper, we propose a visual fatigue evaluation algorithm to predict the degree of visual fatigue from a 2D-plus-depth video. Spatial and temporal characteristics of the depth video are main factors of visual fatigue for autostereoscopic displays. Using depth image directly, we estimate spatial and temporal complexities, depth position and scene movement of the 3D video. Then, the overall visual fatigue of the 3D video is evaluated to have higher correlation with subjective fatigue evaluation by a linear regression. Moreover we control the pixel value of depth image from the 3D video which may induce severe fatigue to make more comfortable 3D video. The results of proposed algorithm show a considerable correlation with subjective visual fatigue.