To achieve a target bit-rate reduction of 50% over H.264/AVC while maintaining equivalent perceptual video quality, High Efficiency Video Coding (HEVC) includes several new coding tools including a new set of integer transforms. Since these transforms are more complex than the H.264/AVC transforms, it is more challenging to design and develop high-performance integer transform hardware for HEVC. In this paper, we propose a series of high-throughput and low-cost hardware-oriented HEVC transform algorithms by using a butterfly structure and replacing multiplications by additions and shift operations in a way that minimizes critical computation paths. Compared to the algorithms using other methodologies like multiplierless multiple constant multiplication (MMCM) or decomposition to sparse matrices, our algorithms achieve around 20% shorter critical paths while consuming relatively small numbers of additions and shift operations. Hardware designs applying our proposed algorithms can increase their throughput by 20% while maintaining a reasonable resource consumption compared to when applying other algorithms.

Seam carving, an image re-targeting method, works by progressively finding and removing connected paths of low energy pixels in an image until a desired image aspect ratio is reached. In this paper, we first cast the problem of minimizing an energy function as that of minimizing a distortion cost. We then leverage on our understanding of image quality metrics/ distortion metrics in proposing a perceptually relevant energy function. Experimental results show that our proposed energy function can generate more desirable resized images in which the original structures of the images are better preserved.

Incorporating sample-based prediction during lossless coding can significantly improve coding performance. However, its use within a codec designed for lossy coding requires a modification of the available prediction scheme. When implementing the codec, two different prediction processes will have to be implemented. This paper describes a lossless coding scheme that delays the sample-based prediction till the residue coding stage of the codec and carries out prediction in the residual domain. In this way, the prediction scheme of the lossy coder can be retained while realizing the coding gains associated with sample-based prediction. The proposed scheme improves lossless intra coding performance in HEVC Main Profile by an average of 6.5%.

HEVC is an emerging video coding standard that can achieve significant compression gains compared to H.264/AVC due to the inclusion of numerous new coding tools. In particular, it allows for a flexible quadtree based block partitioning of each coding tree unit (CTU) and an ability to switch quantization parameters (QP) on a sub-CTU level. In this paper, we present an approach for selecting quantization parameters for each block of pixels on the basis of optimizing the SSIM of the entire picture. Our simulation results show that when SSIM is the quality metric, the proposed approach is able to give average BD-Rate gains of 5.5% to 7.4% compared to using a constant QP per picture while having a negligible increase in encoding runtime. In addition, our proposed method also significantly outperforms the MPEG-2 TM5 adaptive quantization algorithm implemented in the HEVC reference software.

Intra prediction improves coding performance by reducing inter pixel redundancy. However, to accommodate the use of block transforms, not all pixels can be predicted from reconstructed pixels that are located close to themselves. This causes prediction performance to suffer as pixel values further apart are less correlated. This paper presents additional intra coding modes designed with the goal of improving prediction performance. Experimental results show an average gain of about 2% in the key technical area software when the new modes are incorporated in the current 8$\,\times\,$8 prediction modes. Since the new coding modes (8$\,\times\,$8) are designed with transform size smaller than coding block size, the modes can also be useful when the source block is larger than the maximum transform size.

We present a method for analyzing dynamic range along the residual coding and reconstruction pathways during video coding by deriving bounds on the maximum absolute value of intermediary data using a simple combination of triangle inequality and reformulation using Kronecker products. The proposed method is then applied toward analyzing the residual coding and reconstruction process in the emerging High Efficiency Video Coding (HEVC) standard. Our analysis shows that, for an input residual with a bitdepth of (B+1) that uses a uniform quantizer, the dynamic range of quantized levels and dequantized coefficients are no more than (B+7) bits and 17 bits, respectively. Furthermore, a 16 bits transpose buffer is sufficient, while up to 5 bits of bitdepth expansion can occur in the reconstructed residual. The analysis is validated by simulation results with both randomly generated residual and encoding/decoding test video sequences using the HEVC reference software.

In this paper, we present a method for performing rate-distortion optimization (RDO) using a perceptual visual quality metric, the structural similarity index (SSIM), as the target of optimization. Rate-distortion optimization is widely used in modern video codecs to make various encoder decisions to optimize the rate-distortion tradeoff. Typically, the distortion measure used is either sum-of-square error or sum-of-absolute distance, both of which are convenient when used in the RDO framework but not always reflective of a perceptual visual quality. We show that SSIM can be used as the distortion metric in the RDO framework in a simple, yet effective, manner by scaling the Lagrange multiplier used in RDO based on the local variance in that region. The experimental results on the H.264/AVC reference software show that compared to traditional RDO approaches, for the same SSIM score, the proposed approach can achieve an average rate reduction of about 9% and 14% for random access and low-delay encoding configurations. At the same time, there is no significant change in the encoding runtime.

Image quality metrics (IQMs), such as the mean squared error (MSE) and the structural similarity index (SSIM), are quantitative measures to approximate perceived visual quality. In this paper, through analyzing the relationship between the MSE and the SSIM under an additive noise distortion model, we propose a perceptually relevant MSE-based IQM, MSE-SSIM, which is expressed in terms of the variance of the source image and the MSE between the source and distorted images. Evaluations on three publicly available databases (LIVE, CSIQ, and TID2008) show that the proposed metric, despite requiring less computation, compares favourably in performance to several existing IQMs. In addition, due to its simplicity, MSE-SSIM is amenable for the use in a wide range of image and video tasks that involve solving an optimization problem. As an example, MSE-SSIM is used as the objective function in designing a Wiener filter that aims at optimizing the perceptual visual quality of the output. Experimental results show that the images filtered with a MSE-SSIM-optimal Wiener filter have better visual quality than those filtered with a MSE-optimal Wiener filter.

Existing structural similarity (SSIM) index comprises of one term on luminance comparison and the other term on contrast and structure comparison. In this paper, the SSIM index is first improved by introducing three weighting factors to the second term such that it is adaptive to local intensities of two images to be compared. The improved SSIM (iSSIM) index is further extended for two images with possibly different exposures. Experimental results show that the proposed indices are more robust to large intensity changes of two images from the same scene and more sensitive to two images from different scenes than the existing SSIM index.

In the current HEVC test model (HM), a quad-tree based coding tree block (CTB) representation is used to signal mode, partition, prediction and residual information. The large number of combinations of quad-tree partitions and modes to be tested during rate-distortion optimization (RDO) results in a high encoding complexity. In this paper, we investigate and compare a variety of algorithms for fast CTB and mode decision. Experimental results from HM4-based implementations show that different strategies can provide a range of complexity-performance trade-offs. In particular, our proposed CU Depth Pruning algorithm can reduce encoding time by about 10% with only 0.1% coding loss, while a combination of our proposed Early Partition Decision and an early CU termination approach can reduce encoding time by about 40% with about 1% coding loss.

Rate-distortion optimization is widely used in modern video codecs to make various encoder decisions in order to optimize the rate-distortion trade-off. Typically, the distortion measure used is either sum-of-square error (SSE) or sum-of-absolute distance (SAD), both of which are convenient when used in the RDO framework but not always reflective of perceptual quality. In this paper, we show that by expressing SSIM in terms of SSE, SSIM can be used as the distortion metric in the RDO framework in an effective and efficient manner by simply scaling the Lagrange multiplier used in RDO based on the local variance in that region without further changes to the RDO engine. Experimental results show that compared to traditional RDO approaches, for the same SSIM score, the proposed approach can achieve an average rate decrease of 8% and 11% for random access and low-delay encoding configurations, with no significant change in encoding runtime.

One of the challenges in video rate control lies in determining a quantization parameter (Qp) that will be used for both the rate-distortion (R-D) optimization process and the quantization of transform coefficients. In this paper, we attempt to achieve effective rate control with a different approach. By modeling the relationships of distortion, texture bits, non-texture bits, and Qp, we derive the Qp required for both R-D optimization and quantization through Lagrangian optimization. From experiments with several video sequences, we found that our rate control scheme is capable of effective rate control with only a few model updates during encoding. The proposed rate control scheme adapts quickly to the characteristics of the source data and is particularly effective at controlling the rate of videos with high and unpredictable motion content.

The use of mode-dependent transforms for coding directional intra prediction residuals has been previously shown to provide coding gains, but the transform matrices have to be derived from training. In this paper, we derive a set of separable mode-dependent transforms by using a simple separable, directional, and anisotropic image correlation model. Our analysis shows that only one additional transform, the odd type-3 discrete sine transform (ODST-3), is required for the optimal implementation of mode-dependent transforms. In addition, the four-point ODST-3 also has a structure that can be exploited to reduce the operation count of the transform operation. Experimental results show that in terms of coding efficiency, our proposed approach matches or improves upon the performance of a mode-dependent transforms approach that uses transform matrices obtained through training.

Intra-prediction improves coding performance by reducing inter-pixel redundancy. However, to accommodate the use of block transforms, not all pixels can be predicted from reconstructed pixels that are located close to themselves. This causes prediction performance to suffer as pixel values further apart are less correlated. This paper presents additional intra-prediction modes designed with the goal of improving prediction performance. Experimental results show an average gain of about 2% in KTA when the new modes are incorporated in the current 8×8 prediction modes. Since the new coding modes (8×8) are designed with transform size smaller than coding block size, the modes can also be useful when the prediction unit is larger than the maximum transform size. The use of smaller transform sizes can potentially lead to reduction of decoder complexity and implementation costs.

Applying mode-dependent separable transforms, e.g., mode- dependent directional transform (MDDT), is an effective method for improving transform coding of intra prediction residuals. However, two transform matrices typically need to be stored for each intra prediction mode. By using a simple image correlation mode, we have previously derived and proposed a simplified mode-dependent separable transforms scheme that uses a combination of two well-known trans- forms: Discrete Cosine Transform (DCT) and Discrete Sine Transform (DST). In this paper, we propose an orthogonal 4-point integer DST that has a multiplier-less implementation consisting of only adds and bit-shifts. We also propose a simple set of mode-dependent scans for coefficient coding that can be used on top of mode-dependent transforms. Our experimental results on the current HEVC reference software show that in terms of coding efficiency, our proposed approach has comparable performance to MDDT. More importantly, compared to MDDT, our approach requires no training and has lower computational and storage costs.

Intra coding in the current H.264/AVC video coding standard achieves high compression efficiency, in part due to the highly effective intra prediction process that exploits spatial directional correlation. However, intra prediction of chroma components in YUV 4:2:0 videos uses a limited set of possible predictions available for coding of luma components. Furthermore, coding of chroma components proceeds somewhat independently of luma components, and ignores any possible correlation between them. In this paper, we show a way of using reconstructed luma pixels to help with intra prediction of chroma pixels. By making use of the reconstructed co-located luma block to perform template matching in the luma plane, we are able to use as predictors the co-located chroma blocks of the matched luma blocks. Simulations results indicate that the proposed approach is able to obtain up to 33% chroma bit-rate reduction and up to 8% overall bit-rate reduction over H.264/AVC.

In this paper, we derive separable KLTs for coding H.264/AVC intra prediction residuals, using a simple image correlation model. Our analysis shows that for some intra prediction modes, we can in fact just use the DCT for performing either the row-wise or column-wise transform. Furthermore, we also compute the KLT that should be used based on the image correlation model, which happens to have sinosuidal terms. The 4×4 transform also has a structure that can be exploited to reduce the operation count of the transform operation. In our simplified implementation of mode-dependent directional transforms (MDDT), we only need to make use of two matrices: the DCT and the derived KLT. Our experimental results show that in terms of coding efficiency, our proposed approach has similar performance when compared with MDDT. More importantly, compared to MDDT, our approach requires no training and has lower computational and storage costs.

In the current working draft of HEVC, residual quad-tree (RQT) coding is used to encode prediction residuals in both Intra and Inter coding units (CU). However, the rationale for using RQT as a coding tool is different in the two cases. For Intra prediction units, RQT provides an efficient syntax for coding a number of sub-blocks with the same intra prediction mode. For Inter CUs, RQT adapts to the spatial-frequency variations of the CU, using as large a transform size as possible while catering to local variations in residual statistics. While providing coding gains, effective use of RQT currently requires an exhaustive search of all possible combinations of transform sizes within a block. In this paper, we exploit our insights to develop two fast RQT algorithms, each designed to meet the needs of Intra and Inter prediction residual coding.

With an empirically-derived complexity-rate-distortion model of a complexity scalable video encoder, we study the distributed optimization of several encoders sharing a wireless network, using the PSNR of the decoded video as the utility function. We will demonstrate how universal frequency reuse and source rate control can help the system degrade gracefully with increasing users, show the value of a simple pricing mechanism on transmission power in improving performance and study the effect of power allocation between transmitter and source encoder.

The H.264/AVC video coding standard encapsulates the most advanced video coding tools. Since the various techniques that lead to better coding efficiency of the coding standard also inevitably increase the complexity of the video encoder, real-time H.264 encoding of video streams is a challenging task. If available computational resource does not allow the entire encoding process to be carried out in time, a complexity scalable technique that ensures a graceful degradation of coding performance will be a valuable tool. We designed a video encoding scheme that allows the rate distortion (R-D) process to be carried out in a complexity scalable fashion. Our proposed singularly parameterized complexity scalable scheme allows the control of complexity-coding performance tradeoff when available resources are limited and the optimal R-D performance is unattainable.

In this paper, we introduce a singularly-parameterized complexity scalable H.264-compliant encoder. Through modeling its complexity-rate-distortion relationships, we derive optimized operating mode of the encoder (rate and complexity) and show through experiment that such optimization can help a video encoder operate within rate and time constraints. The design of the complexity scalable encoding scheme enables the encoder to perform optimization while taking into consideration the availability of computational resource. This extension of traditional rate-distortion optimization is necessary when time or power constraints do not allow a video encoder to achieve rate-distortion optimized coding performance. Our optimization scheme outputs parameters that allow the encoder to be as close to being rate-distortion optimized as possible, within rate and complexity constraints.

The H.264/AVC video coding standard encapsulates the most advanced video coding tools. The various techniques that lead to better coding efficiency of the coding standard also inevitably increase the complexity of the video encoder. Thus, real-time encoding of video streams with H.264 coding standard is a challenging task. If available computational resource does not allow the entire encoding process to be carried out in time, a complexity scalable technique that ensures a graceful degradation of coding performance will be a useful tool. This work proposes a singularly-parameterized complexity scalable rate-distortion framework for H.264/AVC encoders.

In this paper, a singularly parameterized complexity-scalable scheme is proposed for designing power-aware and power-adaptive H.264 video encoder. The proposed scheme enables adaptive control of the trade-off between the encoder's complexity and coding performance by dynamically updating a control parameter during encoding. This enables the encoder to perform optimally on a variety of computing platforms and runtime environments by adaption to instantaneous available computing resources and varying source video characteristics. The proposed scheme also allows computational resource allocation across layers in SVC encoders.

We consider the on-demand delivery of layered video over peer-to-peer (P2P) networks. A parallel server architecture enables the service of requests by more than one server, thus reducing load at individual servers and dispersing network load. In a P2P network, peer upload bandwidth is limited and heterogenous, so a P2P streaming session may involve multiple supplying peers of different uplink. Partitioning the video data is a crucial first step in the collaborative delivery of data by multiple servers. If done right, the client's viewing experience can be significantly improved. In this paper, we describe a retrieval strategy that significantly reduces access delay necessary for continous playback when retrieving layers of heterogenous bitrate over multiple connections of heterogenous bandwidth.

Temporal scalability in H.264/SVC video compression standard can be achieved with the hierarchical B-pictures (HB) structure. When performing motion estimation (ME) in the HB structure, the temporal distance between the current frame and reference frame(s) can be large (up to 32 frames apart). This limits the performance of fast search algorithm as larger search window is often necessary. Extensive experiments showed that popular fast suboptimal block ME algorithms are ineffective at tracking large motions across several frames. In this paper, we propose a new framework called Lacing which integrates well with any fast block ME techniques to significantly improve the motion prediction accuracy in quality of the motion-compensated frame and also result in smoother motion vector fields with lower entropy.

In this paper, we introduce a low-complexity fine-granularity scalable (FGS) video encoder that refines both residue and motion information in the quality layers. The current scalable video coding (SVC) draft shows that significant gains can be achieved when each enhancement layer undergoes the motion estimation/motion compensation (ME/MC) process with its own motion vector field (MVF). However, given the high computational cost of ME/MC, a motion-refined FGS scheme can be expensive to implement. The proposed scheme controls the macroblock (MB) mode allowed in the base layer and channels computational resources to refine motion in enhancement layers. Through a proper selection of Lagrangian factor for the generation of the first MVF, it is possible to design a low-complexity FGS encoder that has good overall coding performance. A simplified motion-refinement scheme is also adopted for selected MBs in enhancement layers by exploiting the correlation of MB-type information between successive layers to further reduce the complexity of the FGS encoder. Meanwhile, a framework of rate-distortion-complexity optimization is proposed for the FGS by considering the interpolation complexity during the ME/MC in each layer. The FGS decoder can be simplified through the reduction in the number of interpolations.