VVC achieves up to 50% bit-rate savings over HEVC at the cost of increased encoding complexity, largely due to the QTMTT partitioning structure. This work surveys the evolution of the VVC Test Model (VTM) and evaluates partitioning acceleration techniques considering changes in complexity and internal heuristics across VTM versions.

Geometric Partitioning Mode (GPM) is an effective coding tool for inter prediction that splits a block into two partitions and blends their predictions. This paper presents a new coding mode, Regression-based Geometric Partitioning Mode (RGPM), which derives a sample-based blending for bi-predictions using a reconstructed template. The RGPM can enhance flexibility in splitting and blending methods compared to GPM. Moreover, two extensions of RGPM scheme are investigated: 1) extending RGPM with template matching (TM) and merge with motion vector difference (MMVD) methods; 2) extending RGPM principle to Spatial Geometric Partitioning Mode (SGPM) for intra prediction. Experimental results show that RGPM with extensions provide 0.12%, 0.24% and 0.23% average luma BD-rate savings on top of enhanced compression model (ECM) in all intra, random access and low delay configurations, respectively. The proposed RGPM is currently adopted in ECM and its two extensions are under study in exploration experiments for future ECM developments.

This paper presents a comparison study of two machine-learning techniques to accelerate the Versatile Video Coding (VVC) intra-partitioning process in QuadTree (QT) configuration: a regression model for predicting Rate-Distortion (RD) costs and a Deep Q-Network (DQN)-based Reinforcement Learning (RL) approach that models partitioning as a Markov Decision Process (MDP). Both methods are size-independent and utilize neighboring RD costs and threshold values to optimize the splits of Coding Units (CUs).

End-to-end image and video codecs are becoming increasingly competitive, compared to traditional compression techniques that have been developed through decades of manual engineering efforts. These trainable codecs have many advantages over traditional techniques, such as their straightforward adaptation to perceptual distortion metrics and high performance in specific fields thanks to their learning ability. However, current state-of-the-art neural codecs do not fully exploit the benefits of vector quantization and the existence of the entropy gradient in decoding devices. In this paper, we propose to leverage these two properties (vector quantization and entropy gradient) to improve the performance of off-the-shelf codecs. Firstly, we demonstrate that using non-uniform scalar quantization cannot improve performance over uniform quantization. We thus suggest using predefined optimal uniform vector quantization to improve performance. Secondly, we show that the entropy gradient, available at the decoder, is correlated with the reconstruction error gradient, which is not available at the decoder. We therefore use the former as a proxy to enhance compression performance. Our experimental results show that these approaches save between 1 to 3% of the rate for the same quality across various pre-trained methods. In addition, the entropy gradient based solution improves traditional codec performance significantly as well.

The past decade has witnessed the huge success of deep learning in well-known artificial intelligence applications such as face recognition, autonomous driving, and large language model like ChatGPT. Recently, the application of deep learning has been extended to a much wider range, with Neural Network-Based Video Coding (NNVC) being one of them. NNVC can be performed at two different levels: embedding neural network-based (NN-based) coding tools into a classical video compression framework or building the entire compression framework upon neural networks. This article elaborates our studies in response to the recent exploration efforts in JVET (Joint Video Experts Team of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC29) in the name of NNVC, falling in the former category. Specifically, in this article, we propose two advanced NN-based video coding technologies, i.e., NN-based intra prediction and NN-based in-loop filtering, which have been investigated for several meeting cycles in JVET and then adopted into the reference software, i.e., NNVC. In addition, we further propose a Small Ad-hoc Deep-Learning Library (SADL), which provides integer-based inference capabilities for neural networks to ensure interoperability across different systems. SADL has been adopted as the inference platform of all neural networks in NNVC. Extensive experiments on top of the NNVC have been conducted to evaluate the effectiveness of the proposed techniques. Compared with VTM-11.0_nnvc, the proposed two NN-based coding tools jointly achieve {11.94%, 21.86%, 22.59%}, {9.18%, 19.76%, 20.92%}, and {10.63%, 21.56%, 23.02%} BD-rate reductions on average for {Y, Cb, Cr} under random-access, low-delay, and all-intra configurations, respectively.

The last standard Versatile Video Codec (VVC) aims to improve the compression efficiency by saving around 50% of bitrate at the same quality compared to its predecessor High Efficiency Video Codec (HEVC). However, this comes with higher encoding complexity mainly due to a much larger number of block splits to be tested on the encoder side. This paper proposes an acceleration of the VVC partitioning based on a multi-output regression model that predicts a suitable split mode for 32 × 32 Coding Unit (CU). Experimental results show that our approach improves complexity trade-offs flexibility while adding better complexity trade-offs points compared to the original encoder.

The film grain characteristics (FGC) Supplemental Enhancement Information (SEI) message was developed in 2003 and was incorporated into the H.264/AVC specification then ported into the HEVC, VVC and Versatile supplemental enhancement information (VSEI) specifications. The FGC SEI message is used to signal and convey to video decoders metadata characterizing a film grain signal, that can be synthetized and superposed to the decoded video as a post-decoding process. It is an efficient solution to mimic the native film grain present in the input video that may have been removed by the encoding, enabling artistic intent preservation. It is also considered as an efficient solution to bring perceptual quality improvements by mitigating visual artefacts resulting from the coding process. The FGC SEI message parameters relate to the entire picture and do not allow a fine local control of the film grain. This paper describes the Film Grain Regions characteristics (FGR) SEI message, submitted as a candidate new VSEI SEI message, that enhances the FGC SEI by adding region-based control parameters enabling to adjust the synthesized film grain per region. After introducing the syntax and semantics of the FGC SEI message, the paper describes the additional features brought by the FGR SEI message. The region-based grain adaptation is showcased on different use cases that illustrate the potential benefits of the technology.

Deep variational autoencoders for image and video compression have gained significant attraction in the recent years, due to their potential to offer competitive or better compression rates compared to the decades long traditional codecs such as AVC, HEVC or VVC. However, because of complexity and energy consumption, these approaches are still far away from practical usage in industry. More recently, implicit neural representation (INR) based codecs have emerged, and have lower complexity and energy usage to classical approaches at decoding. However, their performances are not in par at the moment with state-of-the-art methods. In this research, we first show that INR based image codec has a lower complexity than VAE based approaches, then we propose several improvements for INR-based image codec and outperformed baseline model by a large margin.

End-to-end image and video compression using auto-encoders (AE) offers new appealing perspectives in terms of rate-distortion gains and applications. While most complex models are on par with the latest compression standard like VVC/H.266 on objective metrics, practical implementation and complexity remain strong issues for real-world applications. We propose a practical implementation suitable for realistic applications. We demonstrate that some gains can be achieved on top low-complexity AE, even when using simpler implementation. The proposed implementation also allows a direct integration of such approaches on a variety of platforms and code is made available as a pure C++ standalone codec [1]:

End-to-end image/video codecs are getting competitive compared to traditional compression techniques that have been developed through decades of manual engineering efforts. These trainable codecs have many advantages over traditional techniques such as easy adaptation on perceptual distortion metrics and high performance on specific domains thanks to their learning ability. However, state of the art neural codecs does not take advantage of the existence of gradient of entropy in decoding device. In this paper, we theoretically show that gradient of entropy (available at decoder side) is correlated with the gradient of the reconstruction error (which is not available at decoder side). We then demonstrate experimentally that this gradient can be used on various compression methods, leading to a 1−2% rate savings for the same quality. Our method is orthogonal to other improvements and brings independent rate savings.

The next-generation Versatile Video Coding (VVC) standard introduces a new Multi-Type Tree (MTT) block partitioning structure that supports Binary-Tree (BT) and Ternary-Tree (TT) splits in both vertical and horizontal directions. This new approach leads to five possible splits at each block depth. It thereby improves the coding efficiency of VVC over that of the preceding High Efficiency Video Coding (HEVC) standard, which only supports Quad-Tree (QT) partitioning with a single split per block depth. However, MTT also has brought a considerable impact on encoder computational complexity. This paper proposes a two-stage learning-based technique to tackle the complexity overhead of MTT in VVC intra encoders. In our scheme, the input block is first processed by a Convolutional Neural Network (CNN) to predict its spatial features through a vector of probabilities describing the partition at each$4\times 4$edge. Subsequently, a Decision Tree (DT) model leverages this vector of spatial features to predict the most likely splits at each block. Finally, based on this prediction, only the$N$most likely splits are processed by the Rate-Distortion (RD) process of the encoder. In order to train our CNN and DT models on a wide range of image contents, we also propose a public VVC frame partitioning dataset based on existing image dataset encoded with the VVC reference software encoder. Our solution relying on the top-3 configuration reaches 47.4% complexity reduction for a negligible bitrate increase of 0.79%. A top-2 configuration enables a higher complexity reduction of 70.4% for 2.49% bitrate loss. These results emphasize a better trade-off between VTM intra-coding efficiency and complexity reduction compared to the state-of-the-art solutions. The source code of the proposed method and the training dataset are made publicly available at GitHub.

The Sample Adaptive Offset (SAO) filter has been introduced in HEVC to reduce general coding and banding artefacts in the reconstructed pictures, in complement to the De-Blocking Filter (DBF) which reduces artifacts at block boundaries specifically. The new video compression standard Versatile Video Coding (VVC) reduces the BD-rate by about 36% at the same reconstruction quality compared to HEVC. It implements an additional new in-loop Adaptive Loop Filter (ALF) on top of the DBF and the SAO filter, the latter remaining unchanged compared to HEVC. However, the relative performance of SAO in VVC has been lowered significantly. In this paper, it is proposed to revisit the SAO filter using Neural Networks (NN). The general principles of the SAO are kept, but the a-priori classification of SAO is replaced with a set of neural networks that determine which reconstructed samples should be corrected and in which proportion. Similarly to the original SAO, some parameters are determined at the encoder side and encoded per CTU. The average BD-rate gain of the proposed SAO improves VVC by at least 2.3% in Random Access while the overall complexity is kept relatively small compared to other NN-based methods.

The interactions between different tools added successively to a block-based video codec are critical to its ratedistortion efficiency. In particular, when deep neural network-based intra prediction modes are inserted into a block-based video codec, as the neural network-based prediction function cannot be easily characterized, the adaptation of the transform selection process to the new modes can hardly be performed manually. That is why this paper presents a combined neural network-based intra prediction and transform selection for a block-based video codec. When putting a single neural network-based intra prediction mode and the learned prediction of the selected LFNST pair index into VTM-8.0, -3.71%, -3.17%, and -3.37% of mean BD-rate reduction in all-intra is obtained.

This paper presents a learning-based method to improve bi-prediction in video coding. In conventional video coding solutions, the motion compensation of blocks from already decoded reference pictures stands out as the principal tool used to predict the current frame. Especially, the bi-prediction, in which a block is obtained by averaging two different motion-compensated prediction blocks, significantly improves the final temporal prediction accuracy. In this context, we introduce a simple neural network that further improves the blending operation. A complexity balance, both in terms of network size and encoder mode selection, is carried out. Extensive tests on top of the recently standardized VVC codec are performed and show a BD-rate improvement of −1.4% in random access configuration for a network size of fewer than 10k parameters. We also propose a simple CPU-based implementation and direct network quantization to assess the complexity/gains tradeoff in a conventional codec framework.

This paper presents an iterative training of neural networks for intra prediction in a block-based image and video codec. First, the neural networks are trained on blocks arising from the codec partitioning of images, each paired with its context. Then, iteratively, blocks are collected from the partitioning of images via the codec including the neural networks trained at the previous iteration, each paired with its context, and the neural networks are retrained on the new pairs. Thanks to this training, the neural networks can learn intra prediction functions that both stand out from those already in the initial codec and boost the codec in terms of rate-distortion. Moreover, the iterative process allows the design of training data cleansings essential for the neural network training. When the iteratively trained neural networks are put into H.265 (HM-16.15), -4.2% of mean BD-rate reduction is obtained, i.e. -1.8% above the state-of-the-art. By moving them into H.266 (VTM-5.0), the mean BD-rate reduction reaches -1.9%.

The Joint Video Expert Team (JVET) is currently developing the next-generation MPEG/ITU video coding standard called Versatile Video Coding (VVC) and their ultimate goal is to double the coding efficiency over the state-of-the-art HEVC standard.The latest version of the VVC reference encoder, VTM6.1, is able to improve the intra coding efficiency by 24 % over the HEVC reference encoder HM16.20, but at the expense of 27 times the encoding time. The complexity overhead of VVC primarily stems from its novel block partitioning scheme that complements Quad-Tree (QT) split with Multi-Type Tree (MTT) partitioning in order to better fit the local variations of the video signal. This work reduces the block partitioning complexity of VTM6.1 through the use of Convolutional Neural Networks (CNNs). For each 64 × 64 Coding Unit (CU), the CNN is trained to predict a probability vector that speeds up coding block partitioning in encoding. Our solution is shown to decrease the intra encoding complexity of VTM6.1 by 51.5% with a bitrate increase of only 1.45%.

This article describes the main video coding technologies included in a joint proposal submitted by Qualcomm and Technicolor, in response to a Call for Proposals (CfP) issued by ITU-T SG16 WP3 Q.6 (VCEG) and ISO/IEC JTC1/SC29/WG11 (MPEG) in Oct. 2017. The proposal contains the majority of the tools that have been adopted into the Joint Exploration Model (JEM), developed in the exploratory phase that preceded the CfP. A flexible multi-tree type (MTT) block-partitioning scheme is proposed to extend the quadtree and binary tree (QTBT) based partitioning in JEM by including triple tree (TT) and asymmetric binary tree (ABT) partitions. In addition, several JEM tools in intra and inter prediction, transforms and arithmetic coding are modified, and new tools such as sign prediction and motion compensated padding are proposed. Objective standard dynamic range (SDR) gains of 43.1% and 15.5% in terms of average luma BD-rate improvement have been achieved for the CfP constraint set 1 (random-access configuration) relative to HEVC/H.265 (HM) and JEM anchors, respectively. For the CfP constraint set 2 (low-delay configuration), the average luma BD-rate improvements are 33.7% relative to the HM anchor and 12.7% relative to the JEM anchor. The proposed codec scored highly in both subjective evaluations and objective metrics and was among the best-performing CfP proposals.

Deep neural networks have been recently proposed to solve video interpolation tasks. Given a past and future frame, such networks can be trained to successfully predict the intermediate frame(s). In the context of video compression, these architectures could be useful as an additional inter-prediction mode. Current inter-prediction methods rely on block-matching techniques to estimate the motion between consecutive frames. This approach has severe limitations for handling complex non-translational motions, and is still limited to block-based motion vectors. This paper presents a deep frame interpolation network for video compression aiming at solving the previous limitations, i.e. able to cope with all types of geometrical deformations by providing a dense motion compensation. Experiments with the classical bi-directional hierarchical video coding structure demonstrate the efficiency of the proposed approach over the traditional tools of the HEVC codec.

This paper provides a technical overview of a deep-learning-based encoder method aiming at optimizing next generation hybrid video encoders for driving the block partitioning in intra slices. An encoding approach based on Convolutional Neural Networks is explored to partly substitute classical heuristics-based encoder speed-ups by a systematic and automatic process. The solution allows controlling the trade-off between complexity and coding gains, in intra slices, with one single parameter. This algorithm was proposed at the Call for Proposals of the Joint Video Exploration Team (JVET) on video compression with capability beyond HEVC. In All Intra configuration, for a given allowed topology of splits, a speed-up of ×2 is obtained without BD-rate loss, or a speed-up above ×4 with a loss below 1% in BD-rate.

This paper describes the coding block structure used in the joint Qualcomm/Technicolor responses to the JVET call for proposal on video compression with capability beyond HEVC. The proposed block structure relies on the known quadtree plus binary tree (QTBT) block structure and brings some higher degree of flexibility in it. Two sets of partitioning modes extend QTBT: the ternary tree (TT) and the asymmetric binary tree (ABT). The added split modes are used with a normative non-redundancy policy, which ensures any block topology can be reached through a unique series of split modes. Finally, some flexible binary and ternary partitioning is allowed near the bottom and right picture borders. Overall, the proposed split modes, non-redundancy and picture border strategies lead to 19.6% bitrate reduction over HEVC.

Video codecs are primarily designed assuming that rigid, block-based, two-dimensional displacements are suitable models to describe the motion taking place in a scene. However, translational models are not sufficient to handle real world motion types such as camera zoom, shake, pan, shearing or changes in aspect ratio. We present here a region-based interprediction scheme to compensate such motion. The proposed mode is able to estimate multiple homography models in order to predict complex scene motion. We also introduce an affine photometric correction to each geometric model. Experiments on targeted sequences with complex motion demonstrate the efficiency of the proposed approach compared to the state-of-the-art HEVC video codec.

This paper presents an adaptive clipping technique with optimized syntax in the video coding Joint Exploratory Model (JEM), which exploits the signal characteristics of the video sequence. The component-wise clipping bounds are coded for each slice. Two encoding methods leveraging the efficiency of the proposed technique are then described. The first one consists in modeling the errors induced by the clipping process in the Rate Distortion Optimization. The second one aims at reducing the cost of transform coefficients by smoothing the residuals. Finally, experimental results are provided and several variants are discussed.

A new video compression scheme for High Dynamic Range (HDR) contents is presented. It is assumed that both Standard Dynamic Range (SDR) and HDR devices coexist in the ecosystem, and the scheme is constructed to ensure backward compatibility such that a single video stream carries both the original HDR video and an automatically generated SDR version of this video without noticeable overhead relatively to the distribution of the HDR video only. Compression performance are shown to be solidly improved compared to the conservative and non-backward-compatible approach using the HEVC Main10 codec and the Perceptual Quantizer (PQ) Electro-Optical Transfer Function (OETF) as non-linearity.

We address the topic of intelligent vehicle systems and especially systems aiming at high level scene understanding. Our goal is to build an on-line Behavior Map of surrounding environment. In this case, the road structure becomes important in order to understand entities behavior. From a general view-point, this paper demonstrates two important concepts: 1) the smooth integration of global, absolute but partial information (the Navi Map) in a local and relative map (the Behavior Map). This is demonstrated by the building of the Behavior Map composed of the roads structure and pedestrian/vehicle's position and trajectory. 2) the use of both global and local information to improve the understanding of the environment. This is demonstrated by the proposed “Pedestrian Warning System”. From a technical viewpoint, the main contributions are: 1) the dual ground plane/image plane approach which avoid introduction of errors in both the cost function evaluation and the constraints for minimization, 2) the smooth and efficient integration of prior data in a realistic road model computation when they are available.

In this paper, we present a new approach for scene understanding by an autonomous robot, using imaging sensors only. Our goal is to build in real-time a 3D representation of the scene which is suitable for further 3D interaction. We assume the scene is composed of a mix of known and unknown objects. We present a strategy to automatically build online a consistent 3D representation from a set computer vision algorithms results. We especially focus on 2 important aspects: the constraints and choices of each parts of the system, as well as the global layout of these parts, and the final step where all information are merged together to build a unique 3D representation. Results on real scene and system performance are finally showed to validate the proposed approach

We present a method for fully automatic three-dimensional (3D) reconstruction from a pair of weakly calibrated images in order to deal with the modeling of complex rigid scenes. A two-dimensional (2D) triangular mesh model of the scene is calculated using a two-step algorithm mixing sparse matching and dense motion estimation approaches. The 2D mesh is iteratively refined to fit any arbitrary 3D surface. At convergence, each triangular patch corresponds to the projection of a 3D plane. The proposed algorithm relies first on a dense disparity field. The dense field estimation modelized within a robust framework is constrained by the epipolar geometry. The resulting field is then segmented according to homographic models using iterative Delaunay triangulation. In association with a weak calibration and camera motion estimation algorithm, this 2D planar model is used to obtain a VRML-compatible 3D model of the scene.

We present a global scheme for encoding/decoding natural video sequences with partial 3D models. This technique is based on a robust estimation of a constrained dense field and on the estimation of projection matrices for reference images. Then a sequential encoding of the sequence is performed in order to produce streamed 3D models. The paper presents the global scheme and focuses on the most original steps of the encoding method. Some results on real video sequences are given to validate this approach.