Depth image compression is important for compact representation of 3D visual data in texture-plus-depth format, where texture and depth maps from one or more viewpoints are encoded and transmitted. A decoder can then synthesize a freely chosen virtual view via depth-image-based rendering using nearby coded texture and depth maps as reference. Further, depth information can be used in other image processing applications beyond view synthesis, such as object identification, segmentation, and so on. In this paper, we leverage on the observation that neighboring pixels of similar depth have similar motion to efficiently encode depth video. Specifically, we divide a depth block containing two zones of distinct values (e.g., foreground and background) into two arbitrarily shaped regions (sub-blocks) along the dividing boundary before performing separate motion prediction (MP). While such arbitrarily shaped sub-block MP can lead to very small prediction residuals (resulting in few bits required for residual coding), it incurs an overhead to transmit the dividing boundaries for sub-block identification at decoder. To minimize this overhead, we first devise a scheme called arithmetic edge coding (AEC) to efficiently code boundaries that divide blocks into sub-blocks. Specifically, we propose to incorporate the boundary geometrical correlation in an adaptive arithmetic coder in the form of a statistical model. Then, we propose two optimization procedures to further improve the edge coding performance of AEC for a given depth image. The first procedure operates within a code block, and allows lossy compression of the detected block boundary to lower the cost of AEC, with an option to augment boundary depth pixel values matching the new boundary, given the augmented pixels do not adversely affect synthesized view distortion. The second procedure operates across code blocks, and systematically identifies blocks along an object contour that should be coded using sub-block MP via a rate-distortion optimized trellis. Experimental results show an average overall bitrate reduction of up to 33% over classical H.264/AVC.

Enabling users to interactively navigate through different viewpoints of a static scene is a new interesting functionality in 3D streaming systems. While it opens exciting perspectives toward rich multimedia applications, it requires the design of novel representations and coding techniques to solve the new challenges imposed by the interactive navigation. In particular, the encoder must prepare a priori a compressed media stream that is flexible enough to enable the free selection of multiview navigation paths by different streaming media clients. Interactivity clearly brings new design constraints: the encoder is unaware of the exact decoding process, while the decoder has to reconstruct information from incomplete subsets of data since the server generally cannot transmit images for all possible viewpoints due to resource constrains. In this paper, we propose a novel multiview data representation that permits us to satisfy bandwidth and storage constraints in an interactive multiview streaming system. In particular, we partition the multiview navigation domain into segments, each of which is described by a reference image (color and depth data) and some auxiliary information. The auxiliary information enables the client to recreate any viewpoint in the navigation segment via view synthesis. The decoder is then able to navigate freely in the segment without further data request to the server; it requests additional data only when it moves to a different segment. We discuss the benefits of this novel representation in interactive navigation systems and further propose a method to optimize the partitioning of the navigation domain into independent segments, under bandwidth and storage constraints. Experimental results confirm the potential of the proposed representation; namely, our system leads to similar compression performance as classical inter-view coding, while it provides the high level of flexibility that is required for interactive streaming. Because of these unique properties, our new framework represents a promising solution for 3D data representation in novel interactive multimedia services.

Depth map compression is important for compact representation of 3D visual data in “texture-plus-depth” format, where texture and depth maps of multiple closely spaced viewpoints are encoded and transmitted. A decoder can then freely synthesize any chosen inter-mediate view via depth-image-based rendering (DIBR) using neighboring coded texture and depth maps as anchors. In this work, we leverage on the observation that “pixels of similar depth have similar motion” to efficiently encode depth video. Specifically, we divide a depth block containing two zones of distinct values (e.g., foreground and background) into two sub-blocks along the dividing edge before performing separate motion prediction. While doing such arbitrarily shaped sub-block motion prediction can lead to very small prediction residuals (resulting in few bits required to code them), it incurs an overhead to losslessly encode dividing edges for sub-block identification. To minimize this overhead, we first devise an edge prediction scheme based on linear regression to predict the next edge direction in a contiguous contour. From the predicted edge direction, we assign probabilities to each possible edge direction using the von Mises distribution, which are subsequently inputted to a conditional arithmetic codec for entropy coding. Experimental results show an average overall bitrate reduction of up to 30% over classical H.264 implementation.

When transmitting the so-called “texture-plus-depth” video format, texture and depth maps from the same viewpoint exhibit high correlation. Coded bits from one map can then be used as side information to encode the other. In this paper, we propose to use the depth information to divide the corresponding block in texture map into arbitrarily shaped regions (sub-blocks) for separate motion estimation (ME) and motion compensation (MC). We implemented our proposed sub-block motion prediction (MP) method for texture map coding using depth information as a new coding mode (z-mode) in H.264. Nonetheless, in practical experiments one can observe either a misalignment between texture and depth edges, or an aliasing effect at the texture boundaries. To overcome this issue, z-mode offers two MC types: i) non-overlapping MC, and ii) overlapping MC. In the latter case, overlapped sub-blocks after ME are alpha-blended using a properly designed filter. Moreover, the MV of each sub-block in z-mode is predicted using a Laplacian-weighted average of MVs of neighboring blocks of similar depth. Experimental results show that using z-mode, coding performance of the texture map can be improved by up to 0.7dB compared to native H.264 implementation at high bitrate.

Recently it is relatively easy to produce digital point sampled 3D geometric models. In sight of the increasing capability of 3D scanning systems to produce models with millions of points, compression efficiency is of paramount importance. In this paper, we propose a novel competition-based predictive method for single-rate compression of 3D models represented as point cloud. In particular we aim at 3D scanning methods based on grid pattern. The proposed method takes advantage of the pattern characteristic made of vertical and horizontal lines, by assuming that the object surface is sampled in curve of points. We then designed and implemented a predictive coder driven by this curve-based point representation. Novel prediction techniques are specifically designed for a curve-based cloud of points, and been competing between them to achieve high quality 3D reconstruction. Experimental results demonstrate the effectiveness of the proposed method.

Depth Image Based Rendering (DIBR) technique has been recognized as a promising tool for supporting advanced 3D video services required in MultiView Video (MVV) systems. However, an inherent problem with DIBR is to fill newly exposed areas (holes) caused by disocclusions. This paper addresses the disocclusion problem. To deal with small disocclusions, hole-filling strategies have been designed by the state-of-the-art through pre-processing techniques of the depth video. For larger disocclusions, where depth pre-processing has some limitations, we propose an inpainting approach to retrieve missing pixels. Specifically, we propose in the texture and structure propagation process to take into account the depth information by distinguishing foreground and background parts of the scene. Experimental results illustrate the efficiency of the proposed method.

In this paper, a new filtering technique addresses the disocclusions problem issued from the depth image based rendering (DIBR) technique within 3DTV framework. An inherent problem with DIBR is to fill in the newly exposed areas (holes) caused by the image warping process. In opposition with multiview video (MVV) systems, such as free viewpoint television (FTV), where multiple reference views are used for recovering the disocclusions, we consider in this paper a 3DTV system based on a video-plus-depth sequence which provides only one reference view of the scene. To overcome this issue, disocclusion removal can be achieved by pre-processing the depth video and/or post-processing the warped image through hole-filling techniques. Specifically, we propose in this paper a pre-processing of the depth video based on a bilateral filtering according to the strength of the depth discontinuity. Experimental results are shown to illustrate the efficiency of the proposed method compared to the traditional methods.

Multiview video representation based on depth data, such as multiview video-plus-depth (MVD), is emerging 3D video communication services raising in the meantime the problem of coding and transmitting depth video in addition to classical texture video. Depth video is considered as a key side information in novel view synthesis within multiview video systems, such as three-dimensional television (3 DTV) or free viewpoint television (FTV), wherein the influence of depth compression on the novel synthesized view is still a contentious issue. In this paper, we propose to discuss and investigate the impact of wavelet-based compression of the depth video on the quality of the view synthesis. Experimental results show that significant gains can be obtained by improving depth edge preservation through shorter wavelet-based filtering on depth edges.

Multiview video coding is an emerging application where, in addition to classical temporal prediction, an efficient disparity prediction should be performed in order to achieve the best compression performance. A popular coder is the multiview video coding (MVC) extension of H.264/AVC, which uses a block-based disparity estimation (just like temporal prediction in H.264/AVC). In this paper, we propose to improve the MVC extension by using a dense estimation method that generates a smooth disparity map with ideally infinite precision. The obtained disparity is then segmented and efficiently encoded by using a rate-distortion optimization technique. Experimental results show that significant gains can be obtained compared to the block-based disparity estimation technique used in the MVC extension.

Multi-view video and 3D television are emerging applications raising the problem of efficient encoding of a depth map, in addition to classical texture images. This paper investigates depth image coding via an adaptive wavelet lifting scheme. Switching between long filters in homogeneous areas and short filters over the edges of the depth map is decided based on the contours detected in the texture image. The method takes thus into consideration the correlation existing between the edges in the texture and in the depth image, leading to an improved encoding of the latter one.

Depth image-based rendering (DIBR) is the process of synthesizing some new "virtual" views from one "real" view and the associated per-pixel depth information. The most important problem in this process is to deal with the newly exposed areas (holes) appearing in the virtual images. One common solution to decrease the number of holes is to pre-process the depth map, before the warping. In this paper, we present a new filtering technique for depth image-based rendering. In order to reduce or completely remove the newly exposed areas an efficient smoothing is necessary for the sharp depth changes near object boundaries. In the meantime it is useless to filter the smooth areas in the depth map. Our solution is based on a weighted Gaussian filter taking into account the distance to the contours. By this way, the geometric distortions and the computation time are reduced compared to a uniform filtering of the depth map. We present some results in the context of creation of stereoscopic views for 3D TV.