Geometric Partitioning Mode (GPM) is an effective coding tool for inter prediction that splits a block into two partitions and blends their predictions. This paper presents a new coding mode, Regression-based Geometric Partitioning Mode (RGPM), which derives a sample-based blending for bi-predictions using a reconstructed template. The RGPM can enhance flexibility in splitting and blending methods compared to GPM. Moreover, two extensions of RGPM scheme are investigated: 1) extending RGPM with template matching (TM) and merge with motion vector difference (MMVD) methods; 2) extending RGPM principle to Spatial Geometric Partitioning Mode (SGPM) for intra prediction. Experimental results show that RGPM with extensions provide 0.12%, 0.24% and 0.23% average luma BD-rate savings on top of enhanced compression model (ECM) in all intra, random access and low delay configurations, respectively. The proposed RGPM is currently adopted in ECM and its two extensions are under study in exploration experiments for future ECM developments.

The Sample Adaptive Offset (SAO) filter has been introduced in HEVC to reduce general coding and banding artefacts in the reconstructed pictures, in complement to the De-Blocking Filter (DBF) which reduces artifacts at block boundaries specifically. The new video compression standard Versatile Video Coding (VVC) reduces the BD-rate by about 36% at the same reconstruction quality compared to HEVC. It implements an additional new in-loop Adaptive Loop Filter (ALF) on top of the DBF and the SAO filter, the latter remaining unchanged compared to HEVC. However, the relative performance of SAO in VVC has been lowered significantly. In this paper, it is proposed to revisit the SAO filter using Neural Networks (NN). The general principles of the SAO are kept, but the a-priori classification of SAO is replaced with a set of neural networks that determine which reconstructed samples should be corrected and in which proportion. Similarly to the original SAO, some parameters are determined at the encoder side and encoded per CTU. The average BD-rate gain of the proposed SAO improves VVC by at least 2.3% in Random Access while the overall complexity is kept relatively small compared to other NN-based methods.

The interactions between different tools added successively to a block-based video codec are critical to its ratedistortion efficiency. In particular, when deep neural network-based intra prediction modes are inserted into a block-based video codec, as the neural network-based prediction function cannot be easily characterized, the adaptation of the transform selection process to the new modes can hardly be performed manually. That is why this paper presents a combined neural network-based intra prediction and transform selection for a block-based video codec. When putting a single neural network-based intra prediction mode and the learned prediction of the selected LFNST pair index into VTM-8.0, -3.71%, -3.17%, and -3.37% of mean BD-rate reduction in all-intra is obtained.

This paper presents a learning-based method to improve bi-prediction in video coding. In conventional video coding solutions, the motion compensation of blocks from already decoded reference pictures stands out as the principal tool used to predict the current frame. Especially, the bi-prediction, in which a block is obtained by averaging two different motion-compensated prediction blocks, significantly improves the final temporal prediction accuracy. In this context, we introduce a simple neural network that further improves the blending operation. A complexity balance, both in terms of network size and encoder mode selection, is carried out. Extensive tests on top of the recently standardized VVC codec are performed and show a BD-rate improvement of −1.4% in random access configuration for a network size of fewer than 10k parameters. We also propose a simple CPU-based implementation and direct network quantization to assess the complexity/gains tradeoff in a conventional codec framework.

This paper presents an iterative training of neural networks for intra prediction in a block-based image and video codec. First, the neural networks are trained on blocks arising from the codec partitioning of images, each paired with its context. Then, iteratively, blocks are collected from the partitioning of images via the codec including the neural networks trained at the previous iteration, each paired with its context, and the neural networks are retrained on the new pairs. Thanks to this training, the neural networks can learn intra prediction functions that both stand out from those already in the initial codec and boost the codec in terms of rate-distortion. Moreover, the iterative process allows the design of training data cleansings essential for the neural network training. When the iteratively trained neural networks are put into H.265 (HM-16.15), -4.2% of mean BD-rate reduction is obtained, i.e. -1.8% above the state-of-the-art. By moving them into H.266 (VTM-5.0), the mean BD-rate reduction reaches -1.9%.

This article describes the main video coding technologies included in a joint proposal submitted by Qualcomm and Technicolor, in response to a Call for Proposals (CfP) issued by ITU-T SG16 WP3 Q.6 (VCEG) and ISO/IEC JTC1/SC29/WG11 (MPEG) in Oct. 2017. The proposal contains the majority of the tools that have been adopted into the Joint Exploration Model (JEM), developed in the exploratory phase that preceded the CfP. A flexible multi-tree type (MTT) block-partitioning scheme is proposed to extend the quadtree and binary tree (QTBT) based partitioning in JEM by including triple tree (TT) and asymmetric binary tree (ABT) partitions. In addition, several JEM tools in intra and inter prediction, transforms and arithmetic coding are modified, and new tools such as sign prediction and motion compensated padding are proposed. Objective standard dynamic range (SDR) gains of 43.1% and 15.5% in terms of average luma BD-rate improvement have been achieved for the CfP constraint set 1 (random-access configuration) relative to HEVC/H.265 (HM) and JEM anchors, respectively. For the CfP constraint set 2 (low-delay configuration), the average luma BD-rate improvements are 33.7% relative to the HM anchor and 12.7% relative to the JEM anchor. The proposed codec scored highly in both subjective evaluations and objective metrics and was among the best-performing CfP proposals.

This paper provides a technical overview of a deep-learning-based encoder method aiming at optimizing next generation hybrid video encoders for driving the block partitioning in intra slices. An encoding approach based on Convolutional Neural Networks is explored to partly substitute classical heuristics-based encoder speed-ups by a systematic and automatic process. The solution allows controlling the trade-off between complexity and coding gains, in intra slices, with one single parameter. This algorithm was proposed at the Call for Proposals of the Joint Video Exploration Team (JVET) on video compression with capability beyond HEVC. In All Intra configuration, for a given allowed topology of splits, a speed-up of ×2 is obtained without BD-rate loss, or a speed-up above ×4 with a loss below 1% in BD-rate.

Summary form only given. This paper presents two sets of modifications to band offset type of the Sample Adaptive Offset technique in HEVC. First, some constraints on the SAO semantics are added to solve sub-optimal syntax issue and to exploit the actual range information of reconstructed samples. Next, the classification process is adapted to the particular samples statistics.

This paper presents an adaptive clipping technique with optimized syntax in the video coding Joint Exploratory Model (JEM), which exploits the signal characteristics of the video sequence. The component-wise clipping bounds are coded for each slice. Two encoding methods leveraging the efficiency of the proposed technique are then described. The first one consists in modeling the errors induced by the clipping process in the Rate Distortion Optimization. The second one aims at reducing the cost of transform coefficients by smoothing the residuals. Finally, experimental results are provided and several variants are discussed.

Displays' new rendering capabilities combined with the ever-growing number of video applications have fueled the emergence of new video formats addressing wider color gamut and larger frame size. Thus, the need in scalable compression technology to provide backward compatibility with legacy devices and capitalize on the superior compression performance of High Efficiency Video Coding (HEVC) has increased significantly. This paper gives an overview of the work carried out in the Joint Collaborative Team on Video Coding of ITU-T Study Group 16 (VCEG) and ISO/IEC JTC1/SC29/WG11 motion picture experts group (MPEG) to define scalable extensions of HEVC (SHVC) targeting these market requirements. The color gamut scalability (CGS) tool of SHVC is specially designed to support efficient scalable coding with multiple layers in different color spaces. The genesis of the SHVC-CGS tool is presented and the performance of the various proposals is compared. Finally, the design of the recently adopted SHVC-CGS inter-layer prediction is detailed. The experimental results validate its efficiency in coding video with extended color gamut and high dynamic range.

The adoption of the Main 10 Profile, a 10 bits consumer profile, in the new HEVC standard jointly developed by ISO/IEC MPEG and ITU-T VCEG opens the opportunity to provide to the consumer a new range of video content, both preserving the source characteristics and delivering content with potentially larger intrinsic quality like UHD or premium HD video. This paper presents a scheme based on a content-adaptive color space transform of input video sequences to improve the video coding efficiency. A simple and fast method to determine optimal transform parameters is proposed. Experimental results are provided on top of the HEVC reference software with Main 10 Profile demonstrating the efficiency of this approach.

The HEVC extension for Scalable Video Coding (SHVC) is currently under definition. Color Gamut Scalability (CGS) has been identified as one MPEG requirement. This paper discuss what could be the stake of CGS in the context of video distribution. Two CGS models are presented: one simple scheme based on Gain-Offset transformation, and one based on 3D color LUT, more generic. A comparison of their performances and respective advantages is carried out using different test contents trying to address realistic use cases.

This paper presents a flexible stream switching technique for HEVC. The proposed scheme allows to switch at any frame type and to increase the quality of the reconstructed video compared to basic stream switching. Additional information is inserted and encoded using a subset of SHVC coding modes. This technique maintains a low insertion rate of IDR/CRA pictures. Experimental results show that the overall performance is improved significantly.

State-of-the-art video compression technologies, such as MPEG-4 AVC/H.264 or the new HEVC standard being developed by ISO MPEG and ITU-T VCEG, make use of tree-structured block partitioning for motion compensation. Such motion partitioning only captures horizontal and vertical motion boundaries. To better match actual motion frontiers, Geometry adaptive block partitioning (GEO) has been explored for several years. GEO enables splitting a block using non-horizontal or non-vertical line. Although noticeable coding efficiency gains can be obtained, GEO involves a significant increase of the number of modes to be tested with therefore a high impact on encoding complexity. This paper presents fast algorithms aiming at controlling the complexity while saving the coding efficiency gains of GEO. Experimental results are provided on top of the HEVC standard, demonstrating first the efficiency of the GEO tool. Simplified versions, offering noticeable complexity reduction with limited rate-distortion performance loss, are also demonstrated and compared.