This paper proposes a novel probabilistic representation of color image pixels (PRCI) and investigates its applications to similarity construction in motion estimation and image segmentation problems. The PRCI explores the mixture representation of the input image(s) as prior information and describes a given color pixel in terms of its membership in the mixture. Such representation greatly simplifies the estimation of the probability density function from limited observations and allows us to derive a new probabilistic pixel-wise similarity measure based on the continuous domain Bhattacharyya coefficient. This yields a convenient expression of the similarity measure in terms of the pixel memberships. Furthermore, this pixel-wise similarity is extended to measure the similarity between two image regions. The usefulness of the proposed pixel/region-wise similarities is demonstrated by incorporating them respectively in a dense image descriptor-based multi- layered motion estimation problem and an unsupervised image segmentation problem. Experimental results show that i) the integration of the proposed pixel-wise similarity in dense image-descriptor construction yields improved peak signal to noise ratio performance and higher tracking accuracy in the multi-layered motion estimation problem, and ii) the proposed similarity measures give the best performance in terms of all quantitative measurements in the unsupervised superpixel- based image segmentation of the MSRC and BSD300 datasets.

This paper proposes a new multi-view human body tracking and model-based rendering system with a textured deformable human body model and explores its practical application to view transfer in image-based rendering (IBR). The proposed approach first reconstructs an initial 3-D model of the human subject offline using Kinect depth cameras or from a general 3-D model if such information is unavailable in the original data. The human pose of the subject is then tracked with multi-view videos using an annealed particle filter (APF)-based tracker with a new color-based likelihood function and a Bayesian-Kalman filter smoother. The previous captured model can then be deformed to the new position for model-based rendering. An immediate application of the proposed approach is to support fly-over effects for view transfer in IBR systems with limited cameras. It avoids the reconstruction of the complete dynamic 3-D model where a large number of cameras may be required. Moreover, for static background, the background can be rendered using IBR with precaptured depth maps to further enhance the user's experience. To reduce the artifacts during fly-over caused by tracking errors and model deformation, a novel morphing technique utilizing a new free-form deformation-based artifacts suppression (FFD-AS) method and other user interface design techniques are also proposed. It allows smooth transition between the original view and the model-rendered views. The performance of the proposed algorithm is evaluated using the publicly available HumanEva dataset and our captured RGB-D multi-view dataset. Experimental results show that the proposed APF-based tracker offered improved tracking performance compared with the conventional bidirectional silhouette likelihood criterion. The proposed morphing approach is also shown to be effective in mitigating the rendering artifacts during view transfer.

This paper presents a depth map restoration scheme for both the raw and projected depth map from Kinect v2 sensor. Based on IR-depth consistency, erroneous depth readings around foreground objects are removed by an edge aware consistency correction method. Moreover, a joint adaptive kernel regression algorithm is designed to upsample the sparse depth map after the projection from Kinect v2 sensor's depth camera to its full HD video camera. The structural information in the high resolution color image is implicitly utilized to guide the upsampling of depth map. The effectiveness of the proposed upsampling algorithm is illustrated by experimental results and comparisons on both real Kinect v2 data and Middlebury dataset.