This paper presents a convolutional neural network (CNN)-based enhancement to inter prediction in Versatile Video Coding (VVC). Our approach aims at improving the prediction signal of inter blocks with a residual CNN that incorporates spatial and temporal reference samples. It is motivated by the theoretical consideration that neural network-based methods have a higher degree of signal adaptivity than conventional signal processing methods and that spatially neighboring reference samples have the potential to improve the prediction signal by adapting it to the reconstructed signal in its immediate vicinity. We show that adding a polyphase decomposition stage to the CNN results in a significantly better trade-off between computational complexity and coding performance. Incorporating spatial reference samples in the inter prediction process is challenging: The fact that the input of the CNN for one block may depend on the output of the CNN for preceding blocks prohibits parallel processing. We solve this by introducing a novel signal plane that contains specifically constrained reference samples, enabling parallel decoding while maintaining a high compression efficiency. Overall, experimental results show average bit rate savings of 4.07% and 3.47% for the random access (RA) and low-delay B (LB) configurations of the JVET common test conditions, respectively.

This paper presents a convolutional neural network (CNN) based solution for inter prediction in Versatile Video Coding (VVC). Our approach aims at improving the motion-compensated prediction signal of inter blocks with a residual CNN that incorporates spatial and temporal reference samples, i.e., intra-inter prediction. It is motivated by two considerations. Firstly, incorporating intra reference samples has the potential to improve the prediction signal by adapting it to the reconstructed signal in its immediate vicinity. Secondly, neural network-based methods offer a higher degree of signal adaptivity than conventional signal processing methods. The proposed network is integrated as a post-processing module for inter predicted blocks in VVC. Experimental results show that average bit rate savings of 3.2% to 4.0% can be achieved for the random access configuration.

This paper overviews the motion vector coding and block merging techniques in the Versatile Video Coding (VVC) standard developed by the Joint Video Experts Team (JVET). In general, inter-prediction techniques in VVC can be classified into two major groups: “whole block-based inter prediction” and “subblock-based inter prediction”. In this paper, we focus on techniques for whole block-based inter prediction. As in its predecessor, High Efficiency Video Coding (HEVC), whole block-based inter prediction in VVC is represented by adaptive motion vector prediction (AMVP) mode or merge mode. Newly introduced features purely for AMVP mode include symmetric motion vector difference and adaptive motion vector resolution. The features purely for merge mode include pairwise average merge, merge with motion vector difference, combined inter-intra prediction and geometric partitioning mode. Coding tools such as history-based motion vector prediction and bidirectional prediction with coding unit weights can be applied on both AMVP mode and merge mode. This paper discusses the design principles and the implementation of the new inter-prediction methods. Using objective metrics, simulation results show that the methods overviewed in the paper can jointly achieve 6.2% and 4.7% BD-rate savings on average with the random access and low-delay configurations, respectively. Significant subjective picture quality improvements of some tools are also reported when comparing the resulting pictures at same bitrates.

To reduce the residual energy of a video signal, motion compensated prediction with fractional-sample accuracy has been successfully employed in modern video coding technology. In contrast to the fixed quarter-sample motion vector resolution for the luma component in High Efficiency Video Coding standard, the current draft of a new Versatile Video Coding standard introduces a block-level adaptive motion vector resolution (AMVR) scheme. The AMVR allows coding of motion vector difference at different precisions. Nevertheless, the interpolation filters for each fractional sample position are fixed. In this paper, alternative half-luma-sample interpolation filters are proposed. Enabling the interpolation filter selection at a fine granularity allows to better adapt to the local image characteristics. Experimental results show that the proposed method can improve the average BD-rate of VTM 5.0 reference software by -0.40% for Random Access configuration and by -0.76% for Low Delay P configuration.

In this paper, we describe a video coding design that enables a higher coding efficiency than the HEVC standard. The proposed video codec follows the design of block-based hybrid video coding, but includes a number of advanced coding tools. A part of the incorporated advanced concepts was developed by the Joint Video Exploration Team, while others are newly proposed. The key aspects of these newly proposed tools are the following. A video frame is subdivided into rectangles of variable size using a binary partitioning with variable split ratios. Three new approaches for generating spatial intra prediction signals are supported: A line-wise application of conventional intra prediction modes, coupled with a mode-dependent processing order, a region-based template matching prediction method and intra prediction modes based on neural networks. For motion-compensated prediction, a multi-hypothesis mode with more than two motion hypotheses can be used. In transform coding, mode dependent combinations of primary and secondary transforms are applied. Moreover, scalar quantization is replaced by trellis-coded quantization and the entropy coding of the quantized transform coefficients is improved. The intra and inter prediction signals can be filtered using an edge-preserving diffusion filter or a non-linear DCT-based thresholding operation. The video codec includes an adaptive in-loop filter for which one of three classifiers can be chosen on a picture basis. We also incorporated an optional encoder control, which adjusts the quantization parameters based on a perceptually motivated distortion measure. In a random access scenario, our proposed video codec achieves luma BD-rate savings between 32.5% for HDR HLG UHD and 39.6% for SDR UHD over the HEVC (HM software) anchor for different categories of test sequences.

A key component of state-of-the art video coding is motion-compensated prediction, also called inter prediction. Current standards allow uni- and bi-prediction, i.e. linear superposition of up to two motion-compensated prediction signals. It is well-known that by a superposition of more than two prediction signals (or hypotheses), the energy of the prediction error can be further reduced. In this paper, it is shown that allowing the encoder to choose among different weights for the individual hypotheses is beneficial from a rate-distortion perspective. A practical multi-hypothesis inter prediction scheme based on the Versatile Video Coding Test Model (VTM) is presented. For VTM-1, in the Random Access configuration according to the JVET Common Test Conditions, the average luma BD bit rate is in the range of -1.6 % to -1.9 % for different settings using up to four prediction hypotheses. For VTM-2, the corresponding BD bit rate is -0.95 %. For higher bit rates (i.e., QP values 12, 17, 22, 27) the BD bit rates are -2.2 % for VTM-1 and -1.4 % for VTM-2.

In this paper we combine state of the art video compression and Partial Differential Equation (PDE) based image processing methods. We introduce a new signal adaptive method to filter the predictions of a hybrid video codec using a system of PDEs describing a diffusion process. The method can be applied to intra as well as inter predictions. The filter is embedded into the framework of HEVC. The efficiency of the HEVC video codec is improved by up to -2.76% for All Intra and -3.56% for Random Access measured in Bjøntegaard delta (BD) rate. Coding gains of up to -8.76% can be observed for individual test sequences.

In video coding, the coding decisions made during the encoding of a reference frame impact the achievable rate distortion performance of all inter-predicted frames which refer to it. Most encoders operate frame-by-frame and do not specifically consider this inter-dependency. In this paper, a multi-frame transform coefficient optimization method for H.265/HEVC is proposed. The inter-frame dependencies are described using a linear signal model, such that the optimization problem can be cast in the form of an ℓ1-regularized least squares problem. A simple functional relationship between the regularization parameter and the quantization paramter is empirically found. Bit rate savings are in the range of 7-10% for a simple IPPP... prediction structure and about 3% for a random access configuration.

This paper describes an extension of the high efficiency video coding (HEVC) standard for coding of multi-view video and depth data. In addition to the known concept of disparity-compensated prediction, inter-view motion parameter, and inter-view residual prediction for coding of the dependent video views are developed and integrated. Furthermore, for depth coding, new intra coding modes, a modified motion compensation and motion vector coding as well as the concept of motion parameter inheritance are part of the HEVC extension. A novel encoder control uses view synthesis optimization, which guarantees that high quality intermediate views can be generated based on the decoded data. The bitstream format supports the extraction of partial bitstreams, so that conventional 2D video, stereo video, and the full multi-view video plus depth format can be decoded from a single bitstream. Objective and subjective results are presented, demonstrating that the proposed approach provides 50% bit rate savings in comparison with HEVC simulcast and 20% in comparison with a straightforward multi-view extension of HEVC without the newly developed coding tools.

This paper presents an approach for 3D video coding that uses a format in which a small number of views as well as associated depth maps are coded and transmitted. At the receiver side, additional views required for displaying the 3D video on an autostereoscopic display can be generated based on the corresponding decoded signals by using depth image based rendering (DIBR) techniques. In terms of coding technology, the proposed coding scheme represents an extension of High Efficiency Video Coding (HEVC), similar to the Multiview Coding (MVC) extension of H.264/AVC. Besides the well-known disparity-compensated prediction, advanced techniques for inter-view and inter-component prediction, the representation of depth blocks, and the encoder control for depth signals have been developed and integrated. In comparison to simulcasting the different signals using HEVC, the proposed approach provides about 40% and 50% average bit rate savings for a whole test set when configured to comply with a 2- and 3-view scenario, respectively. The proposed codec was submitted as response to a Call for Proposals on 3D Video Technology issued by the ISO/IEC Moving Picture Experts Group (MPEG) and it was ranked as the overall best performing HEVC-based proposal in the related subjective tests.

The presented approach for 3D video coding uses the multiview video plus depth format, in which a small number of video views as well as associated depth maps are coded. Based on the coded signals, additional views required for displaying the 3D video on an autostereoscopic display can be generated by depth image based rendering techniques. The developed coding scheme represents an extension of HEVC, similar to the MVC extension of H.264/AVC. However, in addition to the well-known disparity-compensated prediction advanced techniques for inter-view and inter-component prediction, the representation of depth blocks, and the encoder control for depth signals have been integrated. In comparison to simulcasting the different signals using HEVC, the proposed approach provides about 40% and 50% bit rate savings for the tested configurations with 2 and 3 views, respectively. Bit rate reductions of about 20% have been obtained in comparison to a straightforward multiview extension of HEVC without the newly developed coding tools.

This paper describes a new mode for efficient encoding of depth map data that are associated with a video signal. Since the motion characteristics (i.e., block partitioning and corresponding motion vectors) of the video signal and its associated depth signal are typically similar, a motion vector inheritance (MVI) mode, where the data that are already transmitted for the video picture can be reused for efficient encoding of the depth map, has been introduced. Experimental results show averaged total bitrate reductions in terms of Bjontegaard-Delta bitrate of about 1.4% for a 2-view scenario and about 1.1% for a 3-view scenario, when averaged over all, coded and synthesized, views. This corresponds to bitrate savings for the depth signal of about 18% for the 2-view case and about 10% for the 3-view case.

Recently, ITU-T VCEG and ISO/IEC MPEG have started a new joint standardization activity on video coding, called High Efficiency Video Coding (HEVC). In this paper, the new transform coding techniques in the HEVC Test Model are de- scribed, including the residual quadtree (RQT) approach and coded block pattern signaling. Experimental results showing the advantage of using larger block size transforms, especially for high resolution video material, are presented. Furthermore, the trade-off between coding gain and encoder complexity in terms of the maximum RQT depth is discussed.

It is a well-known fact that, in order to overcome annoying blocking artifacts, transforms with block-overlapping basis functions have been proposed for image coding. Typically in transform coding, the encoder determines the transform coefficient values by applying the forward transform followed by scalar quantization. In this paper we present an approach, how rate-distortion optimized Lapped Biorthogonal Transform (LBT) coefficient values can be determined by solving ℓ1-regularized least squares problems. We compare a global version, where all the transform coefficients are obtained in one single optimization step, and a local version, where the optimization is done separately for each block, which results in losing optimality, but achieving highly reduced complexity. Our simulation results show gains of about 0.5 dB PSNR compared to ordinary forward transform and scalar quantization with only small losses (<; 0.1 dB) for the local variant.

This paper describes a novel video coding scheme that can be considered as a generalization of the block-based hybrid video coding approach of H.264/AVC. While the individual building blocks of our approach are kept simple similarly as in H.264/AVC, the flexibility of the block partitioning for prediction and transform coding has been substantially increased. This is achieved by the use of nested and pre-configurable quadtree structures, such that the block partitioning for temporal and spatial prediction as well as the space-frequency resolution of the corresponding prediction residual can be adapted to the given video signal in a highly flexible way. In addition, techniques for an improved motion representation as well as a novel entropy coding concept are included. The presented video codec was submitted to a Call for Proposals of ITU-T VCEG and ISO/IEC MPEG and was ranked among the five best performing proposals, both in terms of subjective and objective quality.

Abstract-A video coding architecture is described that is based on nested and pre-configurable quadtree structures for flexible and signal-adaptive picture partitioning. The primary goal of this partitioning concept is to provide a high degree of adaptability for both temporal and spatial prediction as well as for the purpose of space-frequency representation of prediction residuals. At the same time, a leaf merging mechanism is included in order to prevent excessive partitioning of a picture into prediction blocks and to reduce the amount of bits for signaling the prediction signal. For fractional-sample motion-compensated prediction, a fixed-point implementation of the maximal-order minimum-support algorithm is presented that uses a combination of infinite impulse response and FIR filtering. Entropy coding utilizes the concept of probability interval partitioning entropy codes that offers new ways for parallelization and enhanced throughput. The presented video coding scheme was submitted to a joint call for proposals of ITU-T Visual Coding Experts Group and ISO/IEC Moving Picture Experts Group and was ranked among the five best performing proposals, both in terms of subjective and objective quality.

The recently finalized scalable video coding (SVC) extension of the H.264/AVC video coding standard supports encoding of video sequences at different spatial resolutions within one single scalable bit-stream. In order to achieve an improved coding efficiency compared to simulcast, SVC allows so-called "inter-layer prediction," where texture, motion, and residual data of the lower resolution layer are exploited for the encoding of the higher resolution layer. In this paper, we extend the ideas presented in our previous work by M. Winken et al (2007), and describe an approach how the dependencies caused by the inter-layer residual prediction can be utilized in the selection of the transform coefficient values for the lower resolution layer. Our experimental results show gains of 0.5 dB luma PSNR compared to our previous approach for the high resolution layer with no impact on the base layer coding efficiency.

Scalable video coding (SVC) is an extension of H.264/AVC, which is currently being developed by the Joint Video Team. SVC supports any combination of temporal, spatial, and SNR scalability, but there is no special treatment for the case that two or more different bit-depths have to be supported by the scalable bit-stream. There is a number of applications (especially in the HD area) which would benefit from a design where an H.264/AVC conforming base layer with a sample bit-depth of typically 8 bit is extended in a backwards compatible way by an higher bit-depth enhancement layer. In this paper we present new techniques which can be used to achieve bit-depth scalability and show experimental results.

Coding of video sequences using hierarchical B pictures in the joint scalable video model (JSVM) for the scalability amendment of H.264/AVC has the benefit of improved rate distortion efficiency relative to other known temporal decomposition structures, besides providing temporal scalability. In the operational encoder control of the JSVM, the inter-picture dependencies within a hierarchical B picture structure are considered using a heuristic, where pictures that are more frequently used for motion compensation are coded with higher fidelity compared to pictures that are less often used for motion compensation. In this paper, we describe an approach where the dependencies introduced by motion compensation are also considered when selecting transform coefficient values. Our experimental results using an H.264/AVC conforming encoder show bit rate reductions of up to 10% compared to the quantization method used by JSVM.