Intra prediction is a crucial component of hybrid video coding framework due to its remarkable ability to reduce spatial redundancy in video signals. Unlike the single-mode based intra prediction in HEVC and VVC, intra fusion prediction methods, that combine the results of multiple angular prediction modes, were newly adopted by Enhanced Compression Model (ECM). However, intra fusion prediction over-relies on local spatial correlations and neglects potential texture similarities in non-adjacent regions. To overcome these limitations and elevate the accuracy of luma intra prediction, a Recurrent Intra Prediction Mode (RIPM) is proposed in this paper, which is composed of two sub-modules, i.e., Recurrent Intra Merge Mode (RIMM) and Recurrent Block Vector Substitution Module (RBVSM). RIMM utilizes the recurrent spatial texture information of the adjacent and non-adjacent spaces for adaptive mode derivation and prediction within the intra fusion prediction framework. RBVSM is a sophisticated mechanism for adaptive prediction mode selection and weight assignment during intra fusion prediction, resulting in enhanced coding performance with minimal impact on computational complexity. The proposed method, implemented on top of ECM-12.0, demonstrates a 0.095% BD-rate gain for the luma component under All Intra configuration, with negligible complexity increase. Currently, RIPM is under study in Exploration Experiments (EE) for ECM in JVET.

To explore the next standardization of future video coding technologies after Versatile Video Coding (VVC) standard was finalized, the Joint Video Exploration Team (JVET) with many video coding experts are working on the exploitation experiment for the compression enhancement beyond VVC capability. Intra template matching is one of the technologies which is studied and adopted in the experiment model. This paper provides an overview and technical detail discussions of intra template matching.

This paper presents an adaptive bilateral matching technique for decoder-side motion vector refinement in video coding. It allows encoder to choose not only the conventional bilateral matching mode with symmetric motion vector difference but also the asymmetric alternatives. To study the efficiency of the proposed technique, the proposed method is integrated in the Versatile Video Coding Test Model 11.0. The experimental result reports an overall of −2.78% luma Bjontegaard Delta rate for the random-access configurations. The compression efficiency on top of the Enhanced Compression Model that beyond the VVC capability are also reported.

This paper presents an overview of the technologies for in-loop processing and filtering in the Versatile Video Coding (VVC) standard. These processes comprise luma mapping with chroma scaling, deblocking filter, sample adaptive offset, adaptive loop filter and cross-component adaptive loop filter. They are qualified as “in-loop” because they are applied inside the encoding and decoding loops, before storing the pictures in the decoded picture buffer. The filters are complementary and address different purposes. Luma mapping with chroma scaling aims at adaptively modifying the coded samples distribution for improved coding efficiency. The deblocking filter aims at reducing blocking discontinuities. Sample adaptive offset mostly aims at reducing artifacts resulting from the quantization of transform coefficients. Adaptive loop filter and cross-component adaptive loop filter are adaptive filters enabling to enhance the reconstructed signal, using for instance Wiener-filter encoding approaches. The paper provides an overview of the in-loop filtering process and a detailed description of the filtering algorithms. Objective compression efficiency results are provided for each filter, with indication of cumulative coding gains. Subjective benefits are illustrated. Implementation issues considered during the design of the VVC in-loop filters are also discussed.

In an increasingly connected world, consumer video experiences have diversified away from traditional broadcast video into new applications with increased use of non-camera-captured content such as computer screen desktop recordings or animations created by computer rendering, collectively referred to as screen content. There has also been increased use of graphics and character content that is rendered and mixed or overlaid together with camera-generated content. The emerging Versatile Video Coding (VVC) standard, in its first version, addresses this market change by the specification of low-level coding tools suitable for screen content. This is in contrast to its predecessor, the High Efficiency Video Coding (HEVC) standard, where highly efficient screen content support is only available in extension profiles of its version 4. This paper describes the screen content support and the five main low-level screen content coding tools in VVC: transform skip residual coding (TSRC), block-based differential pulse-code modulation (BDPCM), intra block copy (IBC), adaptive color transform (ACT), and the palette mode. The specification of these coding tools in the first version of VVC enables the VVC reference software implementation (VTM) to achieve average bit-rate savings of about 41% to 61% relative to the HEVC test model (HM) reference software implementation using the Main 10 profile for 4:2:0 screen content test sequences. Compared to the HM using the Screen-Extended Main 10 profile and the same 4:2:0 test sequences, the VTM provides about 19% to 25% bit-rate savings. The same comparison with 4:4:4 test sequences revealed bit-rate savings of about 13% to 27% for$Y'C_{B}C_{R}$and of about 6% to 14% for$R'G'B'$screen content. Relative to the HM without the HEVC version 4 screen content coding extensions, the bit-rate savings for 4:4:4 test sequences are about 33% to 64% for$Y'C_{B}C_{R}$and 43% to 66% for$R'G'B'$screen content.

The paper provides an overview of the quantization and entropy coding methods in the Versatile Video Coding (VVC) standard. Special focus is laid on techniques that improve coding efficiency relative to the methods included in the High Efficiency Video Coding (HEVC) standard: The inclusion of trellis-coded quantization, the advanced context modeling for entropy coding of transform coefficient levels, the arithmetic coding engine with multi-hypothesis probability estimation, and the joint coding of chroma residuals. Beside a description of the design concepts, the paper also discusses motivations and implementation aspects. The effectiveness of the quantization and entropy coding methods specified in VVC is validated by experimental results.

In the past decade, the development of transform coding techniques has achieved significant progress and several advanced transform tools have been adopted in the new generation Versatile Video Coding (VVC) standard. In this paper, a brief history of transform coding development during VVC standardization is presented, and the transform coding tools in the VVC standard are described in detail together with their initial design, incremental improvements and implementation aspects. To improve coding efficiency, four new transform coding techniques are introduced in VVC, which are namely Multiple Transform Selection (MTS), Low-Frequency Non-separable Secondary Transform (LFNST) and Sub-Block Transform (SBT), as well as a large (64-point) type-2 DCT. The experimental results on VVC reference software (VTM-9.0) show that average 4.5% and 3.6% overall coding gain can be achieved by the VVC transform coding tools for All Intra and Random Access configurations, respectively.

Block-based partitioning is one of the fundamental techniques in video coding. Geometric-based block partitioning is a well-studied method to enable better spatial adaptation to the signal properties. This paper introduces the most recent proposal of advanced geometric-based inter prediction (GIP) made to the state-of-the-art are video coding standard - Versatile Video Coding (VVC). Implemented in the latest test model VTM-6.0 to generalize the existing triangle partition mode (TPM) and evaluated with the Joint Video Experts Team (JVET) Common Test Conditions (CTC) sequences, the proposed advanced GIP scheme provides luma BD-rate reduction of 0.56% for random access (RA) and 1.37% for low-delay (LB) test cases with 2% encoder runtime increase and negligible decoder runtime increase. Furthermore, BD-rate reductions up to 2.92% and 3.49% for RA and LB test cases can be achieved in the absence of multiple related VVC inter prediction tools.

In many video coding systems, separable transforms (such as two-dimensional DCT-2) have been used to code block residual signals obtained after prediction. This paper proposes a parametric approach to build graph-based separable transforms (GBSTs) for video coding. Specifically, a GBST is derived from a pair of line graphs, whose weights are determined based on two non-negative parameters. As certain choices of those parameters correspond to the discrete sine and cosine transform types used in recent video coding standards (including DCT-2, DST-7 and DCT-8), this paper further optimizes these graph parameters to better capture residual block statistics and improve video coding efficiency. The proposed GBSTs are tested on the Versatile Video Coding (VVC) reference software, and the experimental results show that about 0.4% average coding gain is achieved over the existing set of separable transforms constructed based on DCT-2, DST-7 and DCT-8 in VVC.

This paper provides an overview of the coding algorithms of the Joint Exploration Model (JEM) for video compression with capability beyond HEVC, which was developed by the Joint Video Exploration Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG). The goal of the JEM development and experimentation was to provide evidence that sufficient coding efficiency improvement over the High Efficiency Video Coding (HEVC) standard can be achieved, which would justify the need for a new video coding standard with a compression capability significantly exceeding that of HEVC. The development of the JEM provided an ability to conduct studies toward that goal in a verifiable and collaborative manner and led to the launching of the project to develop the new Versatile Video Coding (VVC) standard. Objective metric gains exceeding 30% were measured for most of the tested high-resolution video content that represents current demanding new applications, and subjective testing using human observers showed even more benefit.

This article describes the main video coding technologies included in a joint proposal submitted by Qualcomm and Technicolor, in response to a Call for Proposals (CfP) issued by ITU-T SG16 WP3 Q.6 (VCEG) and ISO/IEC JTC1/SC29/WG11 (MPEG) in Oct. 2017. The proposal contains the majority of the tools that have been adopted into the Joint Exploration Model (JEM), developed in the exploratory phase that preceded the CfP. A flexible multi-tree type (MTT) block-partitioning scheme is proposed to extend the quadtree and binary tree (QTBT) based partitioning in JEM by including triple tree (TT) and asymmetric binary tree (ABT) partitions. In addition, several JEM tools in intra and inter prediction, transforms and arithmetic coding are modified, and new tools such as sign prediction and motion compensated padding are proposed. Objective standard dynamic range (SDR) gains of 43.1% and 15.5% in terms of average luma BD-rate improvement have been achieved for the CfP constraint set 1 (random-access configuration) relative to HEVC/H.265 (HM) and JEM anchors, respectively. For the CfP constraint set 2 (low-delay configuration), the average luma BD-rate improvements are 33.7% relative to the HM anchor and 12.7% relative to the JEM anchor. The proposed codec scored highly in both subjective evaluations and objective metrics and was among the best-performing CfP proposals.

Video coding in the YCbCr color space has been widely used, since it is efficient for compression, but it can result in color distortion due to conversion error. Meanwhile, coding in the RGB color space maintains high color fidelity, having the drawback of a substantial bitrate increase with respect to YCbCr coding. Cross-component prediction (CCP) efficiently compresses video content by decorrelating color components while keeping high color fidelity. In this scheme, the chroma residual signal is predicted from the luma residual signal inside the coding loop. This paper gives a description of the CCP scheme from several points of view, from theoretical background to practical implementation. The proposed CCP scheme has been evaluated in standardization communities and adopted into H.265/High Efficiency Video Coding (HEVC) Range Extensions. The experimental results show significant coding performance improvements for both natural and screen content video, while the quality of all color components is maintained. The average coding gains for natural video are 17% and 5% bitrate reduction in the case of intra coding and 11% and 4% in the case of inter coding for RGB and YCbCr coding, respectively, while the average increment of encoding and decoding times in the HEVC reference software implementation are 10% and 4%, respectively.

This paper describes the coding block structure used in the joint Qualcomm/Technicolor responses to the JVET call for proposal on video compression with capability beyond HEVC. The proposed block structure relies on the known quadtree plus binary tree (QTBT) block structure and brings some higher degree of flexibility in it. Two sets of partitioning modes extend QTBT: the ternary tree (TT) and the asymmetric binary tree (ABT). The added split modes are used with a normative non-redundancy policy, which ensures any block topology can be reached through a unique series of split modes. Finally, some flexible binary and ternary partitioning is allowed near the bottom and right picture borders. Overall, the proposed split modes, non-redundancy and picture border strategies lead to 19.6% bitrate reduction over HEVC.

One key component of all block-based hybrid video codecs is transform coding of prediction residues, which consists of an orthogonal block transform, scalar quantization of transform coefficients, and entropy coding of the resulting quantization indexes. For improving coding efficiency relative to the state-of-the-art video coding standard HEVC, we propose the following modifications: (1) Replacing scalar quantization with trellis-coded quantization; and (2) utilizing additional statistical dependencies between quantization indexes for entropy coding. The proposed approach was integrated into the first test model VTM-1 for the new standardization project Versatile Video Coding (VVC). Our coding experiments showed average bit-rate savings of 4.9 % for intra-only, 3.4 % for random access, and 2.8 % for low-delay configurations.

Affine Motion Compensation (AMC) is a promising coding tool in Joint Exploration Model (JEM) developed by the Joint Video Exploration Team (JVET). AMC in JEM employs a 4-parameter affine model between the current block and its reference block. With this model, Motion Vectors (MV) of each sub-block can be derived from the MVs at two control points. In this paper, we present a practical framework to further improve the AMC in JEM. First, we introduce a multi-model AMC approach, which allows the encoder to select either the 4-parameter affine model or the 6-paramter affine model adaptively. Second, we improve the affine inter-mode in two aspects. For the normative part, we present an efficient affine motion coding method, which replaces the affine MV Prediction (MVP) candidates in JEM with more accurate but simpler ones, and employs a second-order MVP. For the non-normative part, we enhance the motion estimation process for AMC, by regulating the optimization algorithm. Finally, we propose to unify the affine merge-mode and the normal merge-mode into a unified merge-mode, which combine affine merge candidates and normal merge candidates in a single merge candidate list. Partial of these methods have been adopted into the next generation video coding standard named Versatile Video Coding (VVC). Simulation results show that the proposed methods can achieve 1.67% BD rate savings in average for the random access configurations.

In existing video coding standards such as H.264/AVC and HEVC, the intra prediction is typically derived using fixed, symmetric prediction filters along the prediction direction, e.g., in planar mode, top-right and bottom-left samples are predicted using symmetric prediction filters. However, in case of asymmetric availability of neighboring reference samples, the performance of intra prediction filters designed in HEVC may not be optimal. To further refine the intra prediction and achieve higher accuracy of prediction samples, this paper proposes low-complexity refinements over HEVC intra prediction, which are applied on frequently used planar, DC, horizontal and vertical modes. The proposed method only requires simple addition and bit-shift operations on top of HEVC's intra prediction implementation. Experimental results show that, an average of 0.7% coding gain is achieved for intra coding with no increase in run-time complexity.

Cross-Component Linear Model (CCLM) for chroma intra-prediction is a promising coding tool in Joint Exploration Model (JEM) developed by the Joint Video Exploration Team (JVET). CCLM assumes a linear correlation between the luma and chroma components in a coding block. With this assumption, the chroma components can be predicted by the Linear Model (LM) mode, which utilizes the reconstructed neighbouring samples to derive parameters of a linear model by linear regression. This paper presents three new methods to further improve the coding efficiency of CCLM. First, we introduce a multi-model CCLM (MM-CCLM) approach, which applies more than one linear models to a coding block. With MM-CCLM, reconstructed neighbouring luma and chroma samples of the current block are classified into several groups, and a particular set of linear model parameters is derived for each group. The reconstructed luma samples of the current block are also classified to predict the associated chroma samples with the corresponding linear model. Second, we propose a multi-filter CCLM (MF-CCLM) technique, which allows the encoder to select the optimal down-sampling filter for the luma component with the 4:2:0 colour format. Third, we present a LM-angular prediction (LAP) method, which synthesizes the angular intra-prediction and the MM-CCLM intra-prediction into a new chroma intra coding mode. Simulation results show that 0.55%, 4.66% and 5.08% BD rate savings in average on Y, Cb and Cr components respectively, are achieved for All Intra (AI) configurations with the proposed three methods. MM-CCLM and MF-CCLM have been adopted into the JEM by JVET.

Throughout the past few decades, the separable Discrete Cosine Transform (DCT), particularly the DCT type II, has been widely used in image and video compression. It is well known that, under first-order stationary Markov conditions, DCT is an efficient approximation of the optimal Karhunen-Loève transform. However, for natural image and video sources, the adaptivity of a single separable transform with fixed core is rather limited for the highly dynamic image statistics, e.g., textures and arbitrarily directed edges. It is also known that non-separable transforms can achieve better compression efficiency for images with directional texture patterns, yet they are computationally complex, especially when the transform size is large. In order to achieve higher transform coding gains with relatively low-complexity implementations, we propose a joint separable and non-separable transform. The proposed separable primary transform, named Enhanced Multiple Transform (EMT), applies multiple transform cores from a pre-defined subset of sinusoidal transforms, and the transform selection is signaled in a joint block level manner. Moreover, a Non-Separable Secondary Transform (NSST) method is proposed to operate in conjunction with EMT. Unlike the existing non-separable transform schemes which require excessive amounts of memory and computation, the proposed NSST efficiently improves coding gain with much lower complexity. Extensive experimental results show that the proposed methods, in a state-of-the-art video codec, such as HEVC, can provide significant coding gains (average 6.9% and 4.5% bitrate reductions for intra and random-access coding, respectively).

In this paper, a MV derivation method based on the idea of frame rate up-conversion (FRUC) is proposed. When a block is signaled as FRUC mode, the motion information of the block is derived without signaling. Moreover, derived MVs are refined at sub-block level for more accurate motion field. In addition, two matching methods, i.e., bilateral matching and template matching are supported to obtain good performance in both bi-directional and uni-directional prediction. Simulations under HEVC common test conditions show that over 4.2% average BD-rate reduction was achieved over HEVC reference software HM-16.6 in the case of random access configuration. The method has been adopted into the Joint Exploration Model (JEM) developed by the joint video exploration team (JVET) of MPEG and ITU-T VCEG for the study of next generation video coding standard.

In this paper, a multiple direct mode (MDM) method is presented for chroma intra coding. The main contributions of the proposed MDM method include two aspects: selection of multiple luma intra prediction modes from co-located luma blocks, and the derivation of chroma intra prediction modes from spatial neighbouring blocks. With the proposed method, both the cross-component correlation and spatial correlation of intra prediction modes can be better utilized for more efficient chroma intra coding. Simulation results have validated the efficiency of MDM especially under the decoupled luma-chroma partition trees. The proposed method has been adopted in the Joint Exploration Model (JEM) which is the test platform for future video coding technology exploration in Joint Video Exploration Team (JVET).

Cross-component Linear Model (CCLM) chroma intra prediction assumes a linear correlation between the luma and chroma components in a coding block. With this assumption, the chroma components can be predicted by LM mode, which utilizes the reconstructed neighbouring samples to derive parameters of the linear model by linear regression. This paper presents a multi-model CCLM (MM-CCLM) approach, which applies more than one linear models in a coding block. With MM-CCLM, reconstructed neighbouring luma and chroma samples of the current block are classified into several groups and each group is used as a training set to derive its own linear model. The reconstructed luma samples of the current block are also classified to use corresponding linear model to predict the associated chroma samples. Simulation results show that 0.26%, 1.89% and 1.96% BD rate savings on Y, Cb and Cr components are achieved for All Intra (AI) configurations in average. The proposed method has been adopted in the Joint Exploration Model (JEM) by Joint Video Exploration Team (JVET).

This paper presents a method for efficient compression of high dynamic range (HDR) and wide color gamut (WCG) video data. The proposed solution consists of two major elements: a conventional video codec (e.g., HEVC) and pre-and post-processing steps applied prior to encoding and after decoding process, respectively. The proposed HDR/WCG video coding system can be configured to provide two configurations: (1) a non-backward compatible bitstream with improved HDR video quality and (2) a SDR backward compatible bitstream with balanced visual quality between the reconstructed signal by the SDR and the HDR receivers. The simulations conducted under the MPEG Common Test Conditions for HDR demonstrate that the compression efficiency of the proposed solution outperforms the anchor solution on objective metrics. Additionally, subjective evaluations conducted under MPEG revealed improved visual quality for the proposed method.

The Discrete Cosine Transform (DCT), and in particular the DCT type II, has been widely used for image and video compression. Although DCT efficiently approximates the optimal Karhunen–Loève transform under first-order Markov conditions with low complexity, the energy packing efficiency is still limited since a fixed transform cannot always capture the highly dynamic statistics of natural video content. In this paper, to further improve the transform efficiency, an Enhanced Multiple Transform (EMT) scheme is proposed. In the proposed EMT, a few sinusoidal transforms, other than DCT, have also been utilized for coding both Intra and Inter prediction residuals. The best transform, as selected from a pre-defined transform subset specified by prediction mode, is explicitly signaled in a joint coding block level manner. Moreover, to accelerate encoding process, fast methods have also been proposed by skipping unnecessary transform rate-distortion evaluations using previously encoding statistics. The proposed method has been implemented on top of High-Efficiency Video Coding (HEVC) reference software, and significant coding gain has been verified.

Intra-frame prediction in the High Efficiency Video Coding (HEVC) standard can be empirically improved by applying sets of recursive two-dimensional filters to the predicted values. However, this approach does not allow (or complicates significantly) the parallel computation of pixel predictions. In this work we analyze why the recursive filters are effective, and use the results to derive sets of non-recursive predictors that have superior performance. We present an extension to HEVC intra prediction that combines values predicted using non-filtered and filtered (smoothed) reference samples, depending on the prediction mode, and block size. Simulations using the HEVC common test conditions show that a 2.0% bit rate average reduction can be achieved compared to HEVC, for All Intra (AI) configurations.

Recently, adaptive in-loop filter (ALF) for image/video coding has attracted increasing attention by its proven capability in improving coding performance. ALF is aiming to minimize the mean square error between original samples and decoded samples by using Wiener-based adaptive filter. Samples in a picture are classified into multiple categories and the samples in each category are then filtered with their associated adaptive filter. The filter coefficients may be signaled or inherited to optimize the tradeoff between the mean square error and the overhead. In this paper, a Geometry transformation-based ALF (GALF) scheme is proposed to further improve the performance of ALF, which introduces geometric transformations, such as rotation, diagonal and vertical flip, to be applied to the samples in filter support region depending on the orientation of the gradient of the reconstructed samples before ALF. With the introduction of geometric transformations, more spatial adaptation is supported without excessive signaling of filter coefficients. The experimental results show that GALF outperforms the existing ALF techniques and it has been adopted by the JEM reference software used as the test platform for future video coding technology exploration in JVET.

For the last few decades, the application of signal-adaptive transform coding to video compression has been stymied by the large computational complexity of matrix-based solutions. In this paper, we propose a novel parametric approach to greatly reduce the complexity without degrading the compression performance. In our approach, instead of following the conventional technique of identifying full transform matrices that yield best compression efficiency, we look for the best transform parameters defining a new class of transforms, called HyGTs, which have low complexity implementations that are easy to parallelize. The proposed HyGTs are implemented as an extension of High Efficiency Video Coding (HEVC), and our comprehensive experimental results demonstrate that proposed HyGTs improve average coding gain by 6% bit rate reduction, while using 6.8 times less memory than KLT matrices.

In traditional image and video coding schemes, separable transforms are typically employed due to their low-complexity implementations. However, the compression efficiency of separable transforms is limited for most natural image/video blocks which generally have arbitrarily directed edge and texture patterns. It is well known that non-separable transforms can achieve better compression efficiency for directional texture patterns, yet they are computationally complex, especially for larger block sizes. In order to achieve higher transform coding gains with relatively low-complexity implementations, in this paper, we propose non-separable secondary transforms (NSSTs). The proposed approach applies a secondary non-separable transform on a sub-block of low frequency coefficients generated using a primary separable transform, such as discrete cosine transform (DCT). Since the proposed NSST is a non-separable transform applied on low frequency coefficients in a much smaller block size, which typically captures most of the signal energy, better coding gains can be achieved with at a relatively low-computational cost. Experimental results show that, compared to the latest HEVC reference software (HM16.6), the proposed method achieves up to a significant 12% coding gain for Intra coding.

SHVC is the scalable extension of the latest video coding standard High Efficiency Video Coding (HEVC) and spatial resampling process is inevitable module to support spatial scalability. This paper describes in details the resampling process, including both texture and motion data resampling in SHVC, and using experimental evidence, demonstrate their benefits in terms of coding efficiency.

SHVC is the scalable extension of the latest video coding standard High Efficiency Video Coding (HEVC). Color Gamut Scalability (CGS) refers to a scalable use case in which base layer and enhancement layer have different color gamuts. In this case, special inter-layer prediction is needed to improve coding efficiency in SHVC. In this paper, a solution based on asymmetric 3D lookup table is presented for color gamut scalability. Compared to SHVC without CGS coding tool, the proposed solution provides 9.6% - 16.1% overall luma BD-rate reduction in different test cases.

Summary form only given. Screen content videos increasingly gain the popularity due to the rapid advances in cloud and multimedia technologies, which in turn requires highly efficient screen content compression. A recent standard, namely SCC is under development in JCT-VC, Joint Collaborative Team on Video Coding between ISO/IEC and ITU-T. In SCC, the most efficient new coding tool is Intra block copy (Intra BC). In this paper, we describe the Intra BC that has been proposed by the authors and adopted in the SCC standard and reference software for coding of screen content. Different from the conventional Intra prediction method where the prediction signal is derived from the spatially neighboring samples, the Intra BC mode greatly improves the prediction efficiency by fully exploiting the redundancy of repetitive patterns which typically appear in screen content. Experimental results suggest that the Intra BC mode can improve the coding efficiency significantly for typical screen content video sequences with 43.2% bit rate reduction on average.

This paper provides a technical overview of palette-based coding that was adopted into the test model for the screen content coding (SCC) extension of High Efficiency Video Coding (HEVC) standard at the 18th JCT-VC meeting. Key techniques that enable the palette mode to deliver significant coding gains for screen contents are highlighted, including palette table generation, palette table coding, and the coding methods for palette indices and escape colors. Proposed and adopted techniques up to the first version of the working draft of HEVC SCC extension and test model SCM-2.0 are presented. Experimental results are provided to evaluate the performance of the palette mode in the SCC extension of HEVC.

This paper presents an in-loop adaptive color-space transform for the HEVC Screen Content Coding extension. In the proposed method, the prediction residual is adaptively converted into a different color space to reduce the cross-component redundancy. After the ACT, the signal is coded following the existing HEVC framework. To keep the complexity as low as possible, fixed color-space transforms that are easily implemented with shift and add operations are utilized. Significant coding gains are achieved by this method in the current HEVC Screen Content Coding reference software with no increase of decoding runtime. The proposed method has been adopted to the HEVC Screen Content Coding extension.

With the prevalence of high speed Internet access, emerging video applications such as remote desktop sharing, virtual desktop infrastructure, and wireless display require high compression efficiency of screen contents. However, traditional intra and inter video coding tools were designed primarily for natural contents. Screen contents have significantly different characteristics compared with nature contents, e.g. sharp edges, less or no noise, which makes those traditional coding tools less sufficient. In this research, a new color palette based video coding tool is presented. Different from traditionally intra and inter prediction that mainly removes redundancy between different coding units, palette coding targets at the redundancy of repetitive pixel values/patterns within the coding unit. In the palette coding mode, a lookup table named palette which maps pixel values into table indices (also called palette indices) is signaled first. Then the mapped indice for a coding unit (which we call index block) are coded with a novel three-mode run-length entropy coding. Some encoder-side optimization for palette coding is also presented in detail in this paper. Simulation has been performed using the common screen content coding test condition defined by JCT-VC and the results show that palette coding can effectively improve screen content coding efficiency for both lossless and lossy scenarios.

Inter-layer filtering is a key module of the emerging Scalable Extension of High Efficiency Video Coding Standard (SHVC). In SHVC, up-sampled based layer reconstructed pictures are used as inter-layer references to predict enhancement layer frames such that inter-layer redundancy is reduced. To improve the coding performance of inter-layer filtering, luma plane based chroma plane enhancement was proposed at picture level. However, the efficiency of the picture level adaptation is not very promising when picture resolution is high. To address this issue, region based inter-layer cross-color filtering is proposed in this paper. Simulations under the common test conditions defined by Joint Collaborative Team on Video Coding (JCT-VC) showed that significant chroma coding gain and moderate luma improvement were achieved by the proposed method. When compared to the luma plane based chroma plane enhancement method, the coding gain over SHVC reference software SHM-2.0 is about doubled while the decoding complexity is kept even lower. Moreover, the proposed method outperforms other tools studied in SHVC core experiment on inter-layer filtering.

This paper presents a new coding tool named cross component decorrelation in the emerging High Efficiency Video Coding Range Extension (HEVC RExt) standard. Color video is generally composed of three color components, e.g., RGB or YCbCr. It has been known for over a decade that the three color components have correlation among each other. Although global out-of-loop color space conversion, e.g., RGB-to-YCbCr, can reduce the cross component correlation, local correlation still exists in YCbCr signal. Many methods have been developed in the literature to exploit such redundancy to improve coding efficiency. However, existing methods introduce high computational or implementation complexity, which makes them never be included in mainstream video coding standard such as H.264/AVC and HEVC version 1. In this research, a new hardware friendly cross component decorrelation method is presented which reduces implementation cost while achieving significant BD-rate reduction. For example, based on JCT-VC common test condition for HEVC RExt standardization, the proposed method results in (17.3%, 18.1%, 16.6%) BD-rate reduction for the three color components of the RGB test sequences, in the case of All Intra configuration.

3D-HEVC incorporates advanced inter-view prediction techniques based on more accurately derived disparity vector to better exploit the correlation between objects in different views. The efficient disparity vector derivation method, namely, Neighboring Block based Disparity Vector Derivation (NBDV) provides disparity without accessing any depth information. The NBDV has been developed as a part of the 3D-HEVC in Joint Collaborative Team on 3D Video Coding (JCT-3V) for several meeting cycles, and adopted as a common coding tool used for all the inter-view prediction techniques for high efficient coding of texture views. This paper presents a low complexity NBDV, which is the state-of-the-art disparity vector derivation method in 3D-HEVC.

This paper describes a scalable video codec that was submitted as a response to the joint call for proposals issued by ISO/IEC MPEG and ITU-T VCEG on HEVC scalable extension. The proposed codec uses a multi-loop decoding structure. Several inter-layer texture prediction methods are employed to remove the inter-layer redundancy. Inter-layer prediction is also used when coding enhancement layer syntax elements such as motion parameter and intra prediction mode, to further reduce bit overhead. Additionally, alternative transforms as well as adaptive coefficients scanning are used to code the prediction residues more efficiently. Experimental results are presented to demonstrate the effectiveness of the proposed scheme. When compared to HEVC single-layer coding, the additional rate overhead for the proposed scalable extension is 1.2% to 6.4% to achieve two layers of SNR and spatial scalability.

Scalable video coding extension of HEVC (SHVC) is being developed by Joint Collaborative Team on Video Coding (JCT-VC) of ISO/IEC MPEG and ITU-T VCEG. Different from scalable video coding extension of H.264/AVC (SVC), SHVC employs a multi-loop decoding framework so that the inter-layer residual prediction in SVC does not perform well in SHVC. In this paper, a method called generalized inter-layer residual prediction (GILRP) is proposed. To improve prediction accuracy, the residual predictor is derived with the information from both base and enhancement layers. Moreover, three additional weighting types are introduced on top of inter coding modes to further compensate errors caused by base layer quantization. Simulations under SHVC common test conditions defined by JCT-VC show that 2.9%, 5.1% and 4.9% overall luma BD-rate reduction on average were obtained over SHVC reference software for configurations of random access, low delay with P slices, and low delay with B slices, respectively.

In 3D-HEVC, depth modeling modes (DMM) are applied for efficient intra depth coding. With DMM modes, a prediction unit is partitioned into two parts, and each part is predicted by a single value. In one DMM mode, the partition pattern is implicitly derived at the decoder by searching all pre-defined Wedgelet patterns on a Co-located Texture Luma Block (CTLB), which increases the decoding complexity drastically. To simplify the design of this DMM mode, in this paper, we propose to utilize the Intra Prediction Mode (IPM) of CTLB to largely skip unnecessary Wedgelet searches at the decoder. To achieve this goal, for each IPM, only a limited number of Wedgelet patterns are selected as candidates in this DMM mode. Experimental results demonstrate that, with almost no coding performance degradation, the proposed method significantly reduces the decoding complexity by skipping 90% of Wedgelet searches. The proposed method has been adopted by 3D-HEVC.

Coding multiview video content often benefits significantly from disparity compensation. More advanced inter-view predictions, to predict e.g., motion vectors and residues among views require disparity vector to identify blocks in different views corresponding to the same objects. Block-level disparity vectors in typically approaches are either transmitted thus with additional overhead, or predicted with sophisticated algorithms. This paper describes a new method to derive disparity vectors, based on which advanced inter-view predictions can be better supported. The proposed method derives a disparity vector of a block only from spatial and temporal neighboring blocks. It was implemented and adopted into the 3D-HEVC codec, which is currently developed by Joint Collaborative Team on 3D Video coding development (JCT-3V). Experimental results show that with the proposed method, advanced inter-view prediction in 3D-HEVC provides in average 5.2% bitrate saving of the overall bitrate, for multiview video coding.

This paper introduces inter-layer filters for the scalable extension of High Efficiency Video Coding (SHVC) standard, which is being developed by the Joint Collaborative Team on Video Coding (JCT-VC). The major new coding tool in SHVC is inter-layer texture prediction. It provides about 18% average BD-rate reduction compared with HEVC two-layer simulcast. In the case of spatial scalability, base layer reconstructed pictures are up-sampled to the enhancement layer resolution to generate inter-layer texture prediction. A set of 2D separable 8 taps (luma) and 4 taps (chroma) DCT based interpolation filters, which follow the design principles of HEVC motion compensation interpolation filter, are used in the up-sampling process. In the case of SNR scalability, the up-sampling process is not needed since the reference layer has the same spatial resolution as the current layer but encoded with lower quality. This paper proposes a novel inter-layer filter with denoising effect for SNR scalability to improve enhancement layer coding efficiency and equalize the number of stages in inter-layer processing between SNR and spatial scalabilities. Experimental results show that the usage of inter-layer de-noising filter in SNR scalability provides up to 7.5% BD-rate reduction and has observable improvement on subjective visual quality.

Scalable extension of HEVC, a.k.a. SHVC, is being standardized by the Joint Collaborative Team on Video Coding (JCT-VC). SHVC employs one of the most important coding tools called interlayer prediction, in which. Reconstructed base layer pictures can be used as reference pictures to predict enhancement layer pictures. Therefore, how to efficiently generate interlayer reference pictures to improve coding efficiency is one of the core research topics for the new international standard. In this paper, we presented High Frequency SAO filter (HF-SAO), which extends Sample Adaptive Offset filter (SAO) in HEVC to SHVC. Experimental results based on SHVC reference software version 1.0 show that HFSAO achieves 1.2% (Luma), 1.4% (Cb), 1.4% (Cr) average BD-rate reduction for the enhancement layer coding, which makes itself one of the most promising candidate interlayer filters to the new generation of scalable video coding standard.

Scalable video coding has been a popular research topic for many years. As one of its key objectives, it aims to support different receiving devices connected through a network structure using a single bitstream. Scalable video coding extension of HEVC, also called as SHVC, is being developed by Joint Collaborative Team on Video Coding (JCT-VC) of ISO/IEC MPEG and ITU-T VCEG. Compared to previous standardized scalable video coding technologies, SHVC employs multi-loop decoding design with no low-level changes within any given layer compared to HEVC. With such a simplified extension it aims at solving some of the problems of previous scalable extensions that haven't been successful, and at the same time, aims at supporting all design features that are of vital importance for the success of SHVC. Supporting lightweight and finely tunable bandwidth adaptation is one such vital design feature important for the success of SHVC. This paper proposes novel high level syntax mechanism for SHVC quality scalability to support: (a) using the decoded pictures from higher quality layer as reference for lower layer pictures and key pictures concept to reduce drift; (b) single loop decoding design with encoder only constraints without introducing any normative low-level changes to the normal multi-loop decoding process. Experimental results based on SHVC reference software (SHM 2.0) show that the proposed key picture method achieves an average of 2.9% luma BD-rate reduction in multi-loop framework and an average of 4.4% luma BD-rate loss to attain the capability of single loop decoding.

ITU-T VCEG and ISO/IEC MPEG have undertaken a joint standardization activity on video coding called High Efficiency Video Coding (HEVC). This paper describes the transform coefficient coding in the HEVC Test Model (WD5) and the motivations driving the design. The coefficient coding description encompasses the scan patterns and the coding methods of the last significant coefficient, significance map and coefficient level. Special focus is given to the method for coding the last significant coefficient in the block.

In 3D video systems with Multiview Video plus Depth (MVD) representation, intermediate views can be rendered from transmitted texture views and corresponding depth maps by techniques such as Depth Image Based Rendering (DIBR). Recent standardization activities in MPEG include the development for such MVD based 3DV codecs. One codec which is H.264/AVC based is called 3DV-ATM. Because of compression, reconstructed depth maps often have certain distortions, such as blurry depth edges, which can result in noticeable artifacts in the rendered views. In this paper, a method of adaptive depth edge sharpening is proposed for 3D video coding, based on the 3DV-ATM. The proposed adaptive depth edge filtering and smoothing along depth edge techniques adaptively sharpen blurry edges of the reconstructed depth frames caused by compression. Compared to the anchor software 3DV-ATM, the proposed method achieves about 7.2% bitrate reduction on average for the rendered view PSNRs versus overall bitrates with comparable runtime complexity.

This paper describes transform coefficient coding in the draft international standard of High Efficiency Video Coding (HEVC) specification and the driving motivations behind its design. Transform coefficient coding in HEVC encompasses the scanning patterns and coding methods for the last significant coefficient, significance map, coefficient levels, and sign data. Special attention is paid to the new methods of last significant coefficient coding, multilevel significance maps, high-throughput binarization, and sign data hiding. Experimental results are provided to evaluate the performance of transform coefficient coding in HEVC.

Existing 3D video solutions take advantages of the 2D video infrastructure, by coding two views of the stereoscopic video content in a frame-compatible manner, such as side-by-side or top-bottom. Content providers and service providers are using the existing 2D video authorizing tools and delivery infrastructure to enhance the 2D video experience into 3D with relatively small additional cost. However, such a frame-compatible solution typically sacrifices the quality of each view, by providing a half-resolution representation. In the future, it is expected that more and more devices are capable of rendering full-resolution 1080p stereoscopic video content. Co-existing of services based on half-resolution stereoscopic video, full-resolution stereoscopic video, as well as full-resolution 2D video is expected in the marketplace. In this paper, a scalable codec is proposed to support the decoding and rendering of frame-compatible stereo, full-resolution 2D and full-resolution stereo video representations. Compared to codecs providing similar functionalities, significant coding gain can be achieved.

H.264/AVC uses motion compensated prediction with fractional-pixel precision to reduce temporal redundancy of the input video signal. It has been shown that the Adaptive Interpolation Filter (AIF) framework [3] can significantly improve accuracy of the motion compensated prediction. In this paper, we present the Enhanced Adaptive Interpolation Filters (E-AIF) scheme, which enhances the AIF framework with a number of useful features, aimed at both improving performance and reducing complexity. These features include the full-pixel position filter and the filter offset, the radial-shaped 12-position filter support, and a RD-based filter selection. Simulations show that E-AIF can achieve up to 20% bit rate reduction compared to H.264/AVC. Compared to all other AIF schemes, E-AIF further reduces the bit rate by up to 6%, and demonstrates the highest performance consistently.

This paper describes a video coding technology proposal submitted by Qualcomm in response to a joint call for proposals (CfP) issued by ITU-T SG16 Q.6 (VCEG) and ISO/IEC JTC1/SC29/WG11 (MPEG) in January 2010. The proposed video codec follows a hybrid coding approach based on temporal prediction, followed by transform, quantization, and entropy coding of the residual. Some of its key features are extended block sizes (up to 64$\,\times\,$64), single pass switched interpolation filters with offsets, mode-dependent directional transforms for intra-coding, luma and chroma high precision filtering, geometric motion partitions, adaptive motion vector resolution and efficient 16-point transforms. It also incorporates internal bit-depth increase and modified quadtree-based adaptive loop filtering. Simulation results are presented to demonstrate the high compression efficiency achieved by the proposed video codec at the expense of moderate increase in encoding and decoding complexity compared to the advanced video coding standard (AVC/H.264). For the random access and low delay configurations, it achieved average bit rate reductions of 30.9% and 33.0% for equivalent peak signal-to-noise ratio, respectively, compared to the corresponding AVC anchors. The proposed codec scored highly in both subjective evaluations and objective metrics and was among the best-performing CfP proposals.

In this paper, a novel intra coding scheme is proposed. The proposed scheme improves H.264 intra coding from three aspects: 1) H.264 intra prediction is enhanced with additional bi-directional intra prediction modes; 2) H.264 integer transform is supplemented with directional transforms for some prediction modes; and 3) residual coefficient coding in CAVLC is improved. Compared to H.264, together the improvements can bring on average 7% and 10% coding gain for CABAC and for CAVLC, respectively, with average coding gain of 12% for HD sequences.

This paper presents the design of a fine granularity SNR scalable video coder for low-delay applications. Specifically, temporal prediction is introduced into the FGS layer of closed-loop P-frames. The prediction is formed adaptively from the enhancement layer reference and the base layer reference, utilizing information coded in the base layer. This novel FGS coder can control drift that may occur due to partial decoding of the FGS layer, while at the same time achieving high coding efficiency. Effort is also spent on reducing the complexity of the new solution by using low-complexity motion compensation and by minimizing additional transform operations. With minimal increase in the complexity, the FGS coding performance of closed-loop P-frames can be improved by as much as 4.5 dB for one FGS layer on a base layer coded at QP 42.

This paper discusses two new frame types, SP-frames and SI-frames, defined in the emerging video coding standard, known as ITU-T Rec. H.264 or ISO/IEC MPEG-4/Part 10-AVC. The main feature of SP-frames is that identical SP-frames can be reconstructed even when different reference frames are used for their prediction. This property allows them to replace I-frames in applications such as splicing, random access, and error recovery/resilience. We also include a description of SI-frames, which are used in conjunction with SP-frames. Finally, simulation results illustrating the coding efficiency of SP-frames are provided. It is shown that SP-frames have significantly better coding efficiency than I-frames while providing similar functionalities.

This paper describes the adaptive deblocking filter used in the H.264/MPEG-4 AVC video coding standard. The filter performs simple operations to detect and analyze artifacts on coded block boundaries and attenuates those by applying a selected filter.

This paper presents an overview of the transform and quantization designs in H.264. Unlike the popular 8/spl times/8 discrete cosine transform used in previous standards, the 4/spl times/4 transforms in H.264 can be computed exactly in integer arithmetic, thus avoiding inverse transform mismatch problems. The new transforms can also be computed without multiplications, just additions and shifts, in 16-bit arithmetic, thus minimizing computational complexity, especially for low-end processors. By using short tables, the new quantization formulas use multiplications but avoid divisions.

This paper discusses the new frame types considered for the video coding standard JVT/H26L developed jointly by ITU-T VCEG and ISO-IEC MPEG. The new frame types are called SP and SI-frames and provide functionalities in bitstream switching, splicing, random access, fast forward, fast backward and error resilience/recovery. Specific applications are illustrated as examples on how the properties of SP and SI-frames can be exploited. Later, detailed descriptions of SP and SI-frame decoder and encoder are provided.

This paper presents an overview of the latest transform and quantization designs for H.26L. Unlike the popular discrete cosine transform (DCT) used in previous standards, the transforms in H.26L can be computed exactly in integer arithmetic, thus avoiding inverse transform mismatch problems. The new transforms can also be computed without multiplications, just additions and shifts, in 16-bit arithmetic, thus minimizing computational complexity, especially for low-end processors. By using short tables, the new quantization formulas use multiplications but avoid divisions.

In this correspondence, we present a new approach to two-dimensional (2-D) robust spline image smoothing based on the M-estimator algorithm. Unlike in other M-estimator based image processing algorithms, the new algorithm takes into consideration the spatial relations between picture elements. The contribution of the sample to the model depends not only on the current residual of that sample, but also on the neighboring residuals. A smoothing parameter is estimated separately for each processing window and it adapts to the local structure of the image. The proposed algorithm is applied to image filtering. The resulting filter preserves details and suppresses additive Gaussian and impulsive noise efficiently.

This paper discusses the problem of motion compensated prediction in a segmentation based video coding scheme. The problem is considered in the framework of a generic video coder utilizing spatial image segmentation and a polynomial model of the motion vector field for each image region. It is shown that very impressive reductions in prediction error can be achieved by this approach compared to traditional block matching. The cost of encoding the motion information is also addressed. We derive an analytical method for the reduction of the number of motion coefficients, which is optimal is the sense of the least increase in the prediction error. Finally, we derive a low complexity method for motion assisted merging of regions resulting from spatial segmentation which leads to a dramatic reduction in the number of regions.

In this work we present a new approach to two-dimensional robust spline smoothing. The proposed method is based on M-estimator algorithms but unlike in other M-estimator based image processing algorithms it takes into consideration spatial relations between picture elements. The contribution of the sample to the model depends not only on the current residual of that sample, but also on the neighboring residuals. The smoothing parameter (/spl lambda/) is estimated separately for each processing window and it adapts to the local structure of the image. In order to test the proposed algorithm we apply it to image filtering problem. We show that the filter based on our algorithm has excellent detail preserving properties while suppressing additive Gaussian and impulsive noise very efficiently.>