In order to enable new video applications such as 3DTV and free-viewpoint video, new data formats including both 2D video sequences and corresponding depth map sequences have been proposed. One major characteristic making the depth maps different from video frames is that they typically consist of homogeneous areas separated by sharp edges representing depth discontinuities. Another characteristic of depth map sequences is that the edges exhibit quite similar boundary behaviors as the edges in the corresponding video frames. In this paper, we propose a novel sparse dyadic mode in the design of an efficient depth map compression algorithm through appropriately exploiting these characteristics. With sparse representations of depth blocks and effective reference of edge information from the corresponding video frames, sparse dyadic mode can achieve up to 1.5 dB gain on rendering quality as compared to depth sequences coded using MVC at the same bitrate.

New data formats including 2D video and the corresponding depth maps enable new video applications in which virtual views can be rendered, such as 3DTV and free-viewpoint video (FVV). Different from video frames, depth maps typically consist of homogeneous areas (with no textures) separated by sharp edges representing depth value changes such as between foreground and background. Conventional video coding techniques with transforms followed by quantization typically result in large artifacts along such sharp edges. To suppress these coding artifacts while preserving edges, we propose in this paper a novel filtering method for depth coding, joint trilateral filter. The main contribution in the proposed filter design is the utilization of edge information in the collocated video frame as well as in the depth map. The filtering weights are determined by the following three factors: a domain (spatial) filter which measures the proximity of pixel positions, and two range filters. One range filter takes into account the similarity among depth samples and the other one considers the similarity among the collocated pixels in the video frame. By replacing the deblocking filter in H.264/AVC with the proposed trilateral filter, simulation results demonstrate up to 0.8 dB gain in rendering quality at given bitrate for depth signal.

This paper proposes a selective data pruning-based compression scheme to improve the rate-distortion relation of compressed images and video sequences. The original frames are pruned to a smaller size before compression. After decoding, they are interpolated back to their original size by an edge-directed interpolation method. The data pruning phase is optimized to obtain the minimal distortion in the interpolation phase. Furthermore, a novel high-order interpolation is proposed to adapt the interpolation to several edge directions in the current frame. This high-order filtering uses more surrounding pixels in the frame than the fourth-order edge-directed method and it is more robust. The algorithm is also considered for multiframe-based interpolation by using spatio-temporally surrounding pixels coming from the previous frame. Simulation results are shown for both image interpolation and coding applications to validate the effectiveness of the proposed methods.

This paper proposes a data pruning-based compression scheme to improve the rate-distortion relation of compressed images and video sequences. The original frames are pruned to a smaller size before compression. After decoding, they are interpolated to their original size by an edge-directed interpolation. The data pruning is optimized to obtain the minimal distortion in the interpolation phase. Furthermore, a novel high order interpolation is proposed to adapt the interpolation to many edge directions. This high order filtering uses extra surrounding pixels and achieves more robust edge-directed image interpolation. Simulation results are shown for both image interpolation and coding applications.

Video representations that support view synthesis based on depth maps, such as multiview plus depth (MVD), have been recently proposed, raising interest in efficient tools for depth map coding. In this paper, we derive a new distortion metric that takes into consideration camera parameters and global video characteristics in order to quantify the effect of lossy coding of depth maps on synthesized view quality. In addition, a new skip mode selection method is proposed based on local video characteristics. Experimental results with the proposed mode selection scheme show coding gains of up to 2 dB for the synthesized views, as well as better subjective quality.

We propose an iterative method for the optimization of a set of 2-D separable transforms for a given training data set. The method outputs orthornormal transforms, each one being optimal for a subset of the data with respect to a sparsity-based objective function. The vertical and horizontal directions of the transform may be different, thus allowing directional-adapted transforms (in contrast to the usual DCT). Additionally, we relate the reconstruction error and the sparsity cost terms through the quantization step. To prove the validity of our approach, experimental results concerning coding applications are provided.

In the last years, many sparsity based denoising approaches for image/video denoising have been proposed. Most of them exploit the image/video sparsity model under certain overcomplete basis. In this paper, we unify three sparsity-based denoising techniques and apply them to the problem of video compression artifacts removal. We compare and analyze the three techniques from the aspects of operation atom, transform dimensionality, and quantization impact. Based on the provided analysis, the paper may serve as a guideline to apply sparsity-based denoising techniques to related problems.

Rate control plays an important role in regulating the bit rate to meet the bandwidth and storage requirement. Most existing video encoders regulate the bit rate by adjusting the quantization step size. We propose to incorporate a new dimension: the quantization rounding offset into rate control. In this paper, we present a rate control algorithm with adaptive rounding offsets (ARO) that jointly adjusts the quantization step size and the rounding offset for high bit rate accuracy. Different from the quantization step size that has a limited number of choices, the rounding offset is a continuously adjustable variable that allows the rate control algorithm to reach any precision in principle. Our extensive experimental results show that the proposed ARO algorithm significantly improves the rate control accuracy at almost no extra computational complexity. Compared with the rho-domain rate control, the ARO algorithm reduces the rate control errors from about 2% to 0.5% for INTRA frames, and 5% to 1.5% for INTER frames. Our experiments also demonstrate that ARO provides the extra benefit of smoother visual quality.

Recent video coding strategies, such as H.264/AVC, incorporate an in-loop deblocking filter in order to reduce the effects of quantization noise. These techniques are limited to treating blocky artifacts on smooth regions. In order to solve this, sparsity-based filtering techniques have been recently proposed for efficient filtering of edge and textured areas. More recently, direction-adaptive sparsity-based filtering has also allowed to exploit directional features on intra encoded pictures. This paper proposes a high-performance in-loop filter for deartifacting intra and inter encoded video data. This work extends for Inter frames the use of direction-adaptive sparsity-based filtering techniques and it improves performance by adaptively selecting filtering thresholds consistent with quantization noise statistics, local encoding conditions, compression requirements and the original signal. Thresholds are both spatially and temporally adapted to optimize video quality and/or coding cost. Selected thresholds are encoded and transmitted as side information to the decoder. Experimental results show significant bit rate savings and visual quality enhancement when compared to the state-of-the- art H.264/AVC codec using in-loop deblocking filtering.

Block-based motion and disparity compensation are popular techniques to exploit correlation between video frames. Block sizes used for compensation can be chosen to achieve a good trade-off between signaling overhead and prediction accuracy. However, motion field boundaries correspond to objects having arbitrary shapes; this limits the accuracy of block-based compensation, even when small block sizes are chosen. In this paper we seek to enable compensation based on arbitrarily-shaped regions, while preserving an essentially block-based compensation architecture. To do so, we propose tools for implicit block-segmentation and predictor selection. Given two candidate block predictors, segmentation is applied to the difference of predictors. Then a weighted sum of predictors in each segment is selected for prediction. Simulation results show improvements in rate-distortion (R-D) performance, as compared to the standard quad tree approach in H.264/AVC.

In this paper, we consider compensation of focus mismatches for frames that are encoded with inter-view bi-prediction (B-frames) in multiview coding (MVC). We start with an analysis of a multiview system with focus mismatches, to demonstrate that a B-frame may suffer from different types of mismatches with respect to the frames from different views used as references. As compared to our previous work for inter-view P-frames, filter estimation for B-frames has to consider not only the depth-dependency of focus mismatches, but also i) the possibility that the two predictors, from different directions, exhibit different types of prediction mismatches, and ii) the effect of bi-predictive search on the generation of filtered references. We show that, designing filters only for the averaged bi- predictors could lead to a suboptimal solution when combined with conventional bi-predictive search schemes. Instead, we propose a filter design approach that estimates two sets of depth-related filters, each set compensating for the focus mismatches exhibit in one of the two references used for bi-directional prediction. Simulation results shows that for views coded with inter-view bi-prediction, the proposed method provides up to 0.8 dB gain over current H.264/AVC in the sequences we tested.

Rate control plays an important role in regulating the encoding bit rate to meet the bandwidth and storage requirement. Most existing works regulate the bit rate by adjusting the quantization step size. We propose to incorporate a new dimension: the quantization rounding offset into a rate control algorithm. Based on our previous work of multi-pass fine rate control, in this work, we present a unified one-pass rate control algorithm that jointly adjusts the quantization step size and the rounding offset for high bit rate accuracy. Unlike the quantization step size that has a limited number of choices, the rounding offset is a continuously adjustable variable that allows the rate control algorithm to reach any precision in principle. Our extensive experimental results show that the proposed algorithm greatly improves the rate control accuracy at almost no extra computational complexity.

Modern video coding standards such as H.264/AVC use intra prediction for efficient coding of Intra pictures. These usually exploit local directional signal correlations. More recently, intra prediction modes using non-local signal information have been introduced. A very popular approach is the so called template matching prediction (TMP), which uses template based texture synthesis for signal prediction. This, combined with regular directional prediction, significantly improves intra coding efficiency compared to H.264/AVC. However, current TMP techniques have trouble synthesizing picture data with non-uniform illumination characteristics. They assume that similar picture regions resemble at the same time in structure and illumination, which is often not the case. In order to solve this, we propose a template matching technique with locally adaptive illumination compensation. The proposed technique is based on a linear compensation model with a scaling and an offset parameters to compensate for contrast and brightness disparities respectively. The model parameters are calculated by solving an auto-regressive Least Square problem during the template search for TMP. This permits to synthesize signal structures while capturing the local characteristics of illumination without needing extra side information. The total improvement in intra coding efficiency with respect to H.264/AVC can be of up to 18%.

Abstract — The rate control algorithm is of essential importance to a video encoder. It enables the encoded bitstream to meet the bandwidth and storage requirement while maintaining good video quality. Most existing works adjust the quantization step size to achieve the required bit rate accuracy. This paper introduces a new dimension: the quantization rounding offset into a frame-level fine rate control algorithm. Specifically, we propose a novel fine rate control algorithm based on a linear model between the bit rate and the rounding offset. Unlike the quantization step size that has a limited number of choices, the quantization rounding offset is a continuously adjustable variable which allows the rate control algorithm to reach any precision in principle. Extensive experiment results show that the proposed algorithm greatly improves the bit rate accuracy and provides better visual quality by fine tuning of the rounding offset in addition to the quantization step size. I.

We propose an adaptive filtering approach for coding video that exhibits localized camera focus changes, e.g., such that different portions of a video frame can undergo different blurriness/sharpness changes with respect to corresponding areas in frames used for prediction. First we obtain, for each macroblock, the filter that provides maximum reduction in residual energy, when applied to the reference macroblock before to motion compensation. Then, starting from the block-wise filter parameters, we divide the macroblocks into classes, by clustering macroblocks that have been assigned "similar" filters. Finally, for each class, a two-dimensional filter is designed to minimize the average residual energy for all macroblocks in the class. The resulting filters are applied to the reference frames to generate better matches for motion compensation. Simulation results shows that the proposed method provides up to 1 dB gain over current H.264 for certain sequences. As compared to H.264 with multiple reference frames, the coding gain is about 0.5 dB.

Many modern video coding strategies, such as the H.264/AVC standard, use quadtree-based partition structures for coding intra macroblocks. Such a structure allows the coding algorithm to adapt to the complicated and non-stationary nature of natural images. Despite the adaptation flexibility of quadtree partitions, recent studies have shown that these are not efficient enough (in terms of rate-distortion performance) when images can be locally modeled as 2D piecewise-smooth signals. These observations motivate us to investigate the use of geometry based block partitioning for modeling intra data in video coding. In particular, in this paper, we study in detail the use of geometry-adaptive intra models, where wedgelet like discontinuities are used in order to define separate coding regions where different statistical/waveform modeling tools can be used. In order to implement this idea, we extend the existing H.264/AVC intra coding scheme by introducing two additional geometric modes: INTRA16X16GEO, and INTRA8X8GEO. Experimental results show that significantly improved R-D performance is achieved.

The masking property of human vision systems has been successfully applied in various image/video applications. To invoke the spatial masking effect, it is important to design a metric that effectively identifies the spatial activity of a region. This metric indicates which areas are more textured and more artifacts can be masked. We review three widely used metrics and evaluate their performance in context of film content. We observe that these metrics have strong dependencies on the brightness. More specifically, for smooth areas with film grain, these metrics usually assign greater degrees of texture to the bright regions and lower degrees to dark ones. This causes problems in the bright areas that are mistakenly identified as more textured than the dark areas. Utilizing the property of film grain, we explain the origin of this dependency and propose a new spatial activity metric that removes the dependency on the brightness. In our simulation, we use this new metric in the rate control algorithm of a MPEG-2 video encoder. The result shows more homogeneous film grain in the reconstructed pictures and improved visual quality.

Modern video compression codecs achieve high compression efficiency by exploiting the temporal and spatial redundancies present in video sequences. Although state of the nit block based intra and inter prediction have been refined to better adapt to the content in the scene, they still exhibit limitations in fully extracting information from the decoded data. We propose the addition of an in-loop prediction refinement stage that is initialized with the conventional intra or inter predicted result and is capable of further reducing the spatial redundancies present in the sequence. By extracting additional information from the decoded neighborhood, the refinement stage is able to bring the prediction closer to the original data, thus improving compression efficiency. In this work, we study one possible method of refinement for the particular case of intra prediction. This uses sparse decomposition of blocks that overlap decoded neighboring blocks and the current predicted block in order to make prediction more coherent with already encoded data.

We consider the problem of coding multi-view video that exhibits mismatches in frames from different views. Such mismatches could be caused by heterogeneous cameras and/or different shooting positions of the cameras. In particular, we consider focus mismatches across views, i.e., such that different portions of a video frame can undergo different blurriness/sharpness changes with respect to the corresponding areas in frames from the other views. We propose an adaptive filtering approach for cross-view prediction in multi-view video coding. The disparity fields are exploited as an estimation of scene depth. An Expectation-maximization (EM) algorithm is applied to classify the disparity vectors into groups. Based on the classification result, a video frame is partitioned into regions with different scene-depth levels. Finally, for each scene-depth level, a two-dimensional filter is designed to minimize the average residual energy of cross-view prediction for all blocks in the class. The resulting filters are applied to the reference frames to generate better matches for cross-view prediction. Simulation results show that, when encoding across views, the proposed method achieves up to 0.8dB gain over current H.264 video coding.

We propose new tools for multiview video coding (MVC) that aim to compensate for mismatches between video frames corresponding to different views. Such mismatches could be caused by different shooting positions of the cameras and/or heterogeneous camera settings. In particular, we consider illumination and focus mismatches across views, i.e., such that different portions of a video frame can undergo different illumination and blurriness/sharpness changes with respect to the corresponding areas in frames from the other views. Models for illumination and focus mismatches are proposed and new coding tools are developed from the models. We propose a block-based illumination compensation (IC) technique and a depth-dependent adaptive reference filtering (ARF) approach for cross-view prediction in multiview video coding. In IC, disparity field and illumination changes are jointly computed as part of the disparity estimation search. IC can be adaptively applied by taking into account the rate-distortion characteristics of each block. For ARF, the disparity fields are used to estimate scene depth, such that video frames are first divided into regions with different scene-depth levels. A 2-D filter is then selected for each scene-depth level. These filters are chosen to minimize residual energy, with the goal of compensating for focus mismatches. The resulting filters are applied to the reference frames to generate better matches for cross-view prediction. Furthermore, we propose a coding system that combines IC and ARF. Adjustments are made so as to maximize the gains achieved by using both coding tools, while reducing the complexity of the final integrated system. We analyze the complexity of all proposed methods and present simulation results of IC, ARF and combined system for different multiview sequences based on the H.264/AVC reference codec. When applying the proposed tool to cross-view coding we observe gains of up 1.3 dB as compared to directly using an H.264/AVC codec to perform predictive coding across views.

Attribute graphs offer a compact representation of 2D or 3D images, as each node represents a region with its attributes and the edges convey the neighborhood relations between adjacent regions. Such graphs may be used in the analysis of video sequences and the tracking of objects of interest. Each image of a sequence is segmented and represented as a region adjacency graph. Object tracking becomes a particular graph-matching problem, in which the nodes representing the same object are to be matched. The intrinsic complexity of graph matching is greatly reduced by coupling it with the segmentation. An attractive feature of our approach is its ability to keep track of occluded objects. Results on real sequences show the potential of this approach.

This paper focuses on a novel class of morphological filters called levelings and their extension to vector spaces. Unlike many filtering techniques reported in the literature, levelings suppress details while preserving perfectly the contours of the remaining objects. This feature makes them ideal tools for simplifying images in a large set of applications. They are particularly useful as preprocessing films for segmentation and coding. However when dealing with color images, new colors may be introduced if each color component is treated separately. In order to solve this drawback two extensions of scalar levelings to vector spaces are presented.