The film grain characteristics (FGC) Supplemental Enhancement Information (SEI) message was developed in 2003 and was incorporated into the H.264/AVC specification then ported into the HEVC, VVC and Versatile supplemental enhancement information (VSEI) specifications. The FGC SEI message is used to signal and convey to video decoders metadata characterizing a film grain signal, that can be synthetized and superposed to the decoded video as a post-decoding process. It is an efficient solution to mimic the native film grain present in the input video that may have been removed by the encoding, enabling artistic intent preservation. It is also considered as an efficient solution to bring perceptual quality improvements by mitigating visual artefacts resulting from the coding process. The FGC SEI message parameters relate to the entire picture and do not allow a fine local control of the film grain. This paper describes the Film Grain Regions characteristics (FGR) SEI message, submitted as a candidate new VSEI SEI message, that enhances the FGC SEI by adding region-based control parameters enabling to adjust the synthesized film grain per region. After introducing the syntax and semantics of the FGC SEI message, the paper describes the additional features brought by the FGR SEI message. The region-based grain adaptation is showcased on different use cases that illustrate the potential benefits of the technology.

Film grain which used to be a by-product of the chemical processing in the analog film stock, is a desirable feature in the era of digital cameras. Besides participating to the artistic intent during content creation, film grain has also interesting properties in the video compression chain such as its ability to mask compression artifacts. In this paper, we use a deep learning-based framework for film grain analysis, generation and synthesis. Our framework consists of three modules: a style encoder performing film grain style analysis, a mapping network responsible for film grain style generation, and a synthesis network that generates and blends a specific grain style to a given content in a content-adaptive manner. All modules are trained jointly, thanks to dedicated loss functions, on a new large and diverse dataset of pairs of grain-free and grainy images that we made publicly available to the community1. Quantitative and qualitative evaluations show that fidelity to the reference grain, diversity of grain styles as well as a perceptually pleasant grain synthesis are achieved, demonstrating that each module outperforms the state-of-the-art in the task it was designed for.

In this paper, deep learning-based techniques for film grain removal and synthesis that can be applied in video coding are proposed. Film grain is inherent in analog film content because of the physical process of capturing images and video on film. It can also be present in digital content where it is purposely added to reflect the era of analog film and to evoke certain emotions in the viewer or enhance the perceived quality. In the context of video coding, the random nature of film grain makes it both difficult to preserve and very expensive to compress. To better preserve it while compressing the content efficiently, film grain is removed and modeled before video encoding and then restored after video decoding. In this paper, a film grain removal model based on an encoder-decoder architecture and a film grain synthesis model based on a conditional generative adversarial network (cGAN) are proposed. Both models are trained on a large dataset of pairs of clean (grain-free) and grainy images. Quantitative and qualitative evaluations of the developed solutions were conducted and showed that the proposed film grain removal model is effective in filtering film grain at different intensity levels using two configurations: 1) a non-blind configuration where the film grain level of the grainy input is known and provided as input; and 2) a blind configuration where the film grain level is unknown. As for the film grain synthesis task, the experimental results show that the proposed model is able to reproduce realistic film grain with a controllable intensity level specified as input.

This paper presents experimental results related to adaptive video content mapping used as a compression tool for HDR-PQ content. The purpose of adaptive video content mapping is to adapt the video signal dynamically depending on its statistical properties in order to better exploit the signal codewords range. Adaptive video content mapping has been investigated during the Versatile Video Coding (VVC) standard development with two main implementation designs: in-loop mapping, and out-of-loop mapping. In VVC, the in-loop implementation was finally adopted with a tool named “Luma Mapping with Chroma Scaling” (LMCS). Thanks to its in-loop implementation, LMCS offers high flexibility (high temporal adaptivity, content type adaptivity) and implementation benefits (no need to manage multiple conformance points, one after the decoder, and one after a post-decoder inverse mapping). Nevertheless, it was also observed that for HDR-PQ content, out-of-loop mapping can bring additional coding gains. The paper investigates potential synergies between the in-loop LMCS and out-of-loop mapping. Out-of-loop mapping is implemented as a pre-encoding process, with inverse mapping applied as post-decoding inverse process. A Supplemental Enhancement Information (SEI) message is used to convey the out-of-loop mapping data aside from the video bitstream. Performance results are reported for HDR-PQ content, with out-of-loop mapping only, and with out-of-loop mapping combined with LMCS.

This paper presents an overview of the technologies for in-loop processing and filtering in the Versatile Video Coding (VVC) standard. These processes comprise luma mapping with chroma scaling, deblocking filter, sample adaptive offset, adaptive loop filter and cross-component adaptive loop filter. They are qualified as “in-loop” because they are applied inside the encoding and decoding loops, before storing the pictures in the decoded picture buffer. The filters are complementary and address different purposes. Luma mapping with chroma scaling aims at adaptively modifying the coded samples distribution for improved coding efficiency. The deblocking filter aims at reducing blocking discontinuities. Sample adaptive offset mostly aims at reducing artifacts resulting from the quantization of transform coefficients. Adaptive loop filter and cross-component adaptive loop filter are adaptive filters enabling to enhance the reconstructed signal, using for instance Wiener-filter encoding approaches. The paper provides an overview of the in-loop filtering process and a detailed description of the filtering algorithms. Objective compression efficiency results are provided for each filter, with indication of cumulative coding gains. Subjective benefits are illustrated. Implementation issues considered during the design of the VVC in-loop filters are also discussed.

This paper describes a new video coding tool in the Versatile Video Coding standard (VVC) named as luma mapping with chroma scaling (LMCS). Experimental compression performance results for LMCS and non-normative examples for deriving LMCS parameter values are also provided. LMCS has two main components: 1) a process for mapping input luma code values to a new set of code values for use inside the coding loop; and 2) a luma-dependent process for scaling chroma residue values. The first process, luma mapping, aims at improving the coding efficiency for standard and high dynamic range video signals by making better use of the range of luma code values allowed at a specified bit depth. The second process, chroma scaling, manages relative compression efficiency for the luma and chroma components of the video signal. The luma mapping process of LMCS is applied at the pixel sample level, and is implemented using a piecewise linear model. The chroma scaling process is applied at the chroma block level, and is implemented using a scaling factor derived from reconstructed neighboring luma samples of the chroma block.

The ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group issued a Call for Proposals (CfP) on video compression with capability beyond HEVC in October 2017. The CfP considered three categories of content - Standard Dynamic Range, High Dynamic Range and Wide Colour Gamut (HDR/WCG), and 360° Omni-directional video. As a result of the CfP process, the development of a new video coding standard, named Versatile Video Coding (VVC), was initiated. The goal of this paper is to provide an overview of the CfP responses for the HDR/WCG category. The paper includes a summary of work leading to the development of the CfP, a presentation of the CfP results for the HDR/WCG category, and a description of the specific HDR/WCG technologies submitted to the CfP.

This paper provides a technical overview of a deep-learning-based encoder method aiming at optimizing next generation hybrid video encoders for driving the block partitioning in intra slices. An encoding approach based on Convolutional Neural Networks is explored to partly substitute classical heuristics-based encoder speed-ups by a systematic and automatic process. The solution allows controlling the trade-off between complexity and coding gains, in intra slices, with one single parameter. This algorithm was proposed at the Call for Proposals of the Joint Video Exploration Team (JVET) on video compression with capability beyond HEVC. In All Intra configuration, for a given allowed topology of splits, a speed-up of ×2 is obtained without BD-rate loss, or a speed-up above ×4 with a loss below 1% in BD-rate.

This paper describes the coding block structure used in the joint Qualcomm/Technicolor responses to the JVET call for proposal on video compression with capability beyond HEVC. The proposed block structure relies on the known quadtree plus binary tree (QTBT) block structure and brings some higher degree of flexibility in it. Two sets of partitioning modes extend QTBT: the ternary tree (TT) and the asymmetric binary tree (ABT). The added split modes are used with a normative non-redundancy policy, which ensures any block topology can be reached through a unique series of split modes. Finally, some flexible binary and ternary partitioning is allowed near the bottom and right picture borders. Overall, the proposed split modes, non-redundancy and picture border strategies lead to 19.6% bitrate reduction over HEVC.

As the video industry begins deployment of ultrahigh-definition TV in both professional and consumer markets, including support for higher dynamic range and wider color gamut services is considered essential within the industry. Higher dynamic range and wider color gamut offer end users a significantly enhanced viewing experience by supporting intensity ranges and colors unattainable in existing distribution ecosystems. In response to this trend, several standardization organizations have launched efforts to better enable these features in both short term and midterm. In this paper, we provide a survey of these standardization activities, with the specific goal of providing a summary of the underlying technologies. Our emphasis is on both existing and potential extensions to the High Efficiency Video Coding standard.

In this paper, we investigate a new inter-channel coding mode called LM mode proposed for the next generation video coding standard called high efficiency video coding. This mode exploits inter-channel correlation using reconstructed luma to predict chroma linearly with parameters derived from neighboring reconstructed luma and chroma pixels at both encoder and decoder to avoid overhead signaling. In this paper, we analyze the LM mode and prove that the LM parameters for predicting original chroma and reconstructed chroma are statistically the same. We also analyze the error sensitivity of the LM parameters. We identify some LM mode problematic situations and propose three novel LM-like modes called LMA, LML, and LMO to address the situations. To limit the increase in complexity due to the LM-like modes, we propose some fast algorithms with the help of some new cost functions. We further identify some potentially-problematic conditions in the parameter estimation (including regression dilution problem) and introduce a novel model correction technique to detect and correct those conditions. Simulation results suggest that considerable BD-rate reduction can be achieved by the proposed LM-like modes and model correction technique. In addition, the performance gain of the two techniques appears to be essentially additive when combined.

A new inter-channel coding mode named LM mode has been intensively explored in the HEVC standardization project. This mode predicts the chroma signal from the luma signal using a linear model whose parameters are inferred from neighboring reconstructed luma and chroma samples. Although this mode presents very good coding efficiency, it is observed that ill cases in the linear parameters calculation can be detected and fixed. This paper gives an overview of the LM mode and presents a novel model correction scheme to detect and correct those ill cases. Simulation results show significant bit-rate savings by the proposed correction scheme with limited added complexity.

After having issued the version 1 of the new video coding standard HEVC, ISO-MPEG and ITU-T VCEG groups are specifying its scalable extension. The candidate schemes are based on a multi-layer multi-loop coding framework, exploiting inter-layer texture and motion prediction and full base layer picture decoding. Several inter-layer prediction tools have been explored, implemented either using high-level syntax or block-level core HEVC design changes. One of these tools, Generalized Residual Prediction (GRP), has been extensively studied during several meeting cycles. It is based on second order residual prediction, exploiting motion compensation prediction residual in the base layer. This paper is focused on this new mode. The principle of GRP is described with an analysis of several implementation variants completed by a complexity analysis. Performance of these different implementations is provided, showing that noticeable gains can be obtained without significant complexity increase compared to a simple scalable design comprising only texture and motion inter-layer prediction.

State-of-the-art video compression technologies, such as MPEG-4 AVC/H.264 or the new HEVC standard being developed by ISO MPEG and ITU-T VCEG, make use of tree-structured block partitioning for motion compensation. Such motion partitioning only captures horizontal and vertical motion boundaries. To better match actual motion frontiers, Geometry adaptive block partitioning (GEO) has been explored for several years. GEO enables splitting a block using non-horizontal or non-vertical line. Although noticeable coding efficiency gains can be obtained, GEO involves a significant increase of the number of modes to be tested with therefore a high impact on encoding complexity. This paper presents fast algorithms aiming at controlling the complexity while saving the coding efficiency gains of GEO. Experimental results are provided on top of the HEVC standard, demonstrating first the efficiency of the GEO tool. Simplified versions, offering noticeable complexity reduction with limited rate-distortion performance loss, are also demonstrated and compared.

This paper presents an adaptive scheme for synthesizing missing textured regions. In synthesis-based compression approaches, large textures are removed at encoder side and filled in at decoder side. This work proposes a synthesizer in which both complementary pixel-based and patch-based approaches are used. According to results shown by synthesis algorithms, patch-based and pixel-based approaches are efficient with different kinds of texture. Two algorithms are adapted to the region synthesis context: the sample patch is built from surrounding texture and potential anchor blocks inside the removed region; the scan order depends on a confidence map in order to take advantage of the designed patch; DCT-descriptors provide the minimum size of matching window which is a required parameter for both synthesizers. Finally an "a posteriori" gradient based assessor enables the synthesizer to switch between algorithms.

In this paper, we propose a classified quadtree-based adaptive loop filter (CQALF) in video coding. Pixels in a picture are classified into two categories by considering the impact of the deblocking filter, the pixels that are modified and the pixels that are not modified by the deblocking filter. A wiener filter is carefully designed for each category and the filter coefficients are transmitted to decoder. For the pixels that are modified by the deblocking filter, the filter is estimated at encoder by minimizing the mean square error between the original input frame and a combined frame which is a weighted average of the reconstructed frames before and after the deblocking filter. For pixels that the deblocking filter does not modify, the filter is estimated by minimizing the mean square error between the original frame and the reconstructed frame. The proposed algorithm is implemented on top of KTA software and compatible with the quadtree-based adaptive loop filter. Compared with kta2.6rl anchor, the proposed CQALF achieves 10.05%, 7.55%, and 6.19% BD bitrate reduction in average for intra only, IPPP, and HB coding structures respectively.

In this paper, we propose a new intra prediction scheme based on hard thresholding applied on shift-DCT coefficients. The explored intra prediction method, validated in an H.264/AVC framework, aims at better predicting complex 2D patterns that cannot be properly extrapolated using H.264/AVC directional intra prediction modes. The basic explored algorithm makes use of neighbor pixels of the current block to predict the current signal using a shift-DCT based extrapolation. Different versions of the algorithm, corresponding to different hard thresholding parameter derivation processes, are described. Experimental results show that the proposed intra coding prediction method can significantly improve the coding efficiency compared to the directional predictive modes of H.264/AVC, with an average BD-rate gain of 6.2% observed on a various pictures test set.

This paper describes a new video coding scheme based on a simplified block structure that significantly outperforms the coding efficiency of the ISO/IEC 14496-10 ITU-T H.264 advanced video coding (AVC) standard. Its conceptual design is similar to a typical block-based hybrid coder applying prediction and subsequent prediction error coding. The basic coding unit is an 8 × 8 block for inter, and an 8 × 8 or a 16 × 16 block for intra, instead of the usual 16 × 16 macroblock. No larger block sizes are considered for prediction and transform. Based on this simplified block structure, the coding scheme uses simple and fundamental coding tools with optimized encoding algorithms. In particular, the motion representation is based on a minimum partitioning with blocks sharing motion borders. In addition, compared to AVC, the new and improved coding techniques include: block-based intensity compensation, motion vector competition, adaptive motion vector resolution, adaptive interpolation filters, edge-based intra prediction and enhanced chrominance prediction, intra template matching, larger trans forms and adaptive switchable transforms selection for intra and inter blocks, and nonlinear and frame-adaptive de-noising loop filters. Finally, the entropy coder uses a generic flexible zero tree representation applied to both motion and texture data. Attention has also been given to algorithm designs that facilitate parallelization. Compared to AVC, the new coding scheme offers clear benefits in terms of subjective video quality at the same bit rate. Objective quality improvements are equally significant. At the same quality, an average bit-rate reduction of 31% compared to AVC is reported.

The scalable extension of AVC scalable video coding (SVC) is a current standardization project of the joint video team (JVT) of the ITU-T video coding experts group (VCEG) and the ISO/IEC moving picture experts group (MPEG). SVC has been initially designed for progressive video. However, even if progressive material is becoming the favorite format for production, broadcasting and consumers equipments, interlaced material is still widely used in the video world and will not disappear in the next few years. The specification of SVC should be finalized in 2007. Thanks to its AVC-based design and its AVC base-layer compatibility, migration from AVC to SVC equipments can be considered in a near future. In this context, interlaced support becomes an important requirement for SVC. This paper presents the main concepts for supporting interlaced coding in SVC. After a brief description of the basics of SVC, the generalizations of AVC interlaced tools are first described. Then main issues related to interlaced video scalable encoding are identified and the new mechanisms introduced in the SVC specification for raising these issues are presented. The paper also discusses related applications side and identifies several use cases illustrating the interest of interlaced support in SVC.

Nowadays standard technologies for spatially scalable video coding use Gaussian pyramidal approaches, that naturally lead to redundant descriptions after the temporal analysis. However solutions have been proposed to preserve the critical sampling criterion within a multi-resolution framework. That research area is highly worth the interest since the redundancy suppression potentially improves compression efficiency. We focus here on two different solutions, with a special focus on the spectral composition of the transmitted information. Last some results are presented and discussed.

Spatial scalability is mainly considered for addressing various devices and displays resolutions. This feature is of high interest for many applications, such as TV distribution over DSL networks, video in wireless home network or content production. In its basic design, SVC is defined for dyadic spatial scalability. This contribution relates to extended spatial scalability (ESS), a generalization of SVC spatial scalability, which introduces new scalability modes such as cropping and non-dyadic scaling. It describes the ESS inter-layer motion and texture prediction processes. Performance comparisons with alternate solutions, illustrating the efficiency of the proposed tools, are provided. Finally complexity and profiling issues are discussed.

For a scalable video coder to remain efficient over a wide range of bit-rates, covering e.g. both mobile video streaming and TV broadcasting, some form of scalability must exist in the motion information. In this paper we propose a new (1+2D) wavelet-based spatio-SNR-temporal-scalable video codec, coupled with an accuracy-scalable motion codec. It allows to decode a reduced amount of motion information at subresolutions, taking advantage that motion compensation requires less and less accuracy at lower spatial resolutions. This new motion codec proves its efficiency in our full-scalable framework, by improving significantly video quality at subresolutions without inducing any noticeable penalty at high bit-rates.

The tool presented in this paper performs an automatic segmentation of stereoscopic image sequences, based on the modeling of distance maps obtained by image processing. Two three-dimensional (3-D) image analysis algorithms are combined: (i) estimation of dense depth maps from stereoscopic image sequences and (ii) depth-based object segmentation. A Markovian statistical approach for such a segmentation of a dense depth map into arbitrarily shaped and oriented planar surfaces is described in detail. The simulation results on sequences "Fun fair" and "Tunnel", provided on video tape to the MPEG-4 tests of November 1995, are discussed.

This paper describes a region-based coding algorithm developed by Thomson Multimedia and submitted to MPEG-4 tests of November 1995 and January 1996. In this algorithm, the segmentation into regions is based on an advanced motion analysis, successively achieving a multiresolution motion field estimation and a segmentation based on a Markovian statistical approach, which insures a good temporal coherence of segmentation maps and an identification of covered areas. Moreover, the accurate motion description allows intermediate frames interpolation which can increase the displayed frame rate. Simulation results and MPEG-4 tests of January 1996 have shown that the algorithm is as efficient as block-based coding algorithms like MPEG-1 or H.263 for the compression functionality, while offering temporal scalability by higher frame rates. Moreover, description of the scene in coherent motion regions may be seen as an intermediate step toward object-based functionalities.

The authors deal with analysis of the dynamic content of a scene from an image sequence whatever the camera situation is (static or mobile). This problem involves a motion-based segmentation step. A method which ensures stable motion-based partitions owing to a statistical regularization approach is presented. It does not require the explicit estimation of optic flow fields and manages to link those partitions in time. The identification of the kinematical components of the scene relies on an intermediate layer accomplishing a generic qualitative motion labeling. This is achieved considering jointly several successive images. No 3-D measurements are required. Results obtained on several real image sequences corresponding to complex outdoor situations are reported.>