For bandwidth-constrained multimedia applications, simultaneously achieving ultra-low bitrate human video compression and accurate vertex prediction remains a critical challenge, as it demands the harmonization of dynamic motion modeling, detailed appearance synthesis, and geometric consistency. To address this challenge, we propose Sparse2Dense, a keypoint-driven generative framework that leverages extremely sparse 3D keypoints as compact transmitted symbols to enable ultra-low bitrate human video compression and precise human vertex prediction. The key innovation is the multi-task learning-based and keypointaware deep generative model, which could encode complex human motion via compact 3D keypoints and leverage these sparse keypoints to estimate dense motion for video synthesis with temporal coherence and realistic textures. Additionally, a vertex predictor is integrated to learn human vertex geometry through joint optimization with video generation, ensuring alignment between visual content and geometric structure. Extensive experiments demonstrate that the proposed Sparse2Dense framework achieves competitive compression performance for human video over traditional/generative video codecs, whilst enabling precise human vertex prediction for downstream geometry applications. As such, Sparse2Dense is expected to facilitate bandwidth-efficient human-centric media transmission, such as realtime motion analysis, virtual human animation, and immersive entertainment.

In the latest international video coding standard, Versatile Video Coding, merge mode utilizes spatial or temporal adjacent motion information as motion vector predictors to reduce the signaling overhead. However, only one merge candidate's motion information is selected from previously coded blocks for motion compensation and prediction. To improve the prediction efficiency, a dual merge mode which inherits two candidates' motion information from previously coded blocks is proposed. In the proposed method, motion compensation is executed utilizing the motion information from both merge candidates to generate two predictions, and the two predictions are ultimately combined with equal weight to derive the final predicted results. Both of the merge candidates can be uni-prediction or bi-prediction, requiring up to four motion compensations for the proposed dual merge mode. Experimental results show that the proposed dual merge mode achieves an average coding gain of 0.03 % over ECM-14.0 with negligible complexity increase, and the extended ablation studies verify the effectiveness of the proposed method.

The Versatile Video Coding (VVC) standard incorporates affine motion compensation to represent complex motions including rotation and zoom. However, affine advanced motion vector prediction (AMVP) mode incurs significant signaling overhead, particularly in biprediction where motion vector differences (MVDs) must be signaled for both reference pictures. This paper extends the Symmetric Motion Vector Difference (SMVD) [1] method to affine motion fields. The proposed approach signals MVDs only from one reference picture, deriving the other according to POC distances.

In search for more efficient video compression capability than versatile video coding (VVC), joint video experts team (JVET) launched exploration work on new video coding technologies beyond VVC in 2021. In 2023, audio video coding standard (AVS) workgroup started to explore the new video coding technologies beyond AVS3 and released a software platform named exploration video model (EVM). In EVM, the candidates of merge mode are reordered according to the template matching (TM) cost to reduce the candidate index signaling overhead. After reordering, the TM-based refinement is applied to the candidate to improve the accuracy of the motion derived in merge mode. However, it is found that the bi-prediction candidates are usually more effective than uni-prediction candidates despite the TM cost in non-low-delay pictures, while the current reordering method only relies on the TM cost. And when performing TM-based refinement, the motions of the two reference picture lists (RPLs) are processed independently without joint optimization. To solve these issues, an adaptive motion candidate reordering method and an improved TM motion refinement method which is based on the joint motion search are proposed in this paper. In the proposed candidate reordering method, the TM cost of uni-prediction candidate is weighted by a factor greater than 1 to prioritize the bi-prediction candidates. And in the proposed motion refinement method, the reference template of the bi-predicted block is obtained by averaging the templates of two predicted blocks, such that the motions of the two RPLs are jointly optimized. The proposed methods were implemented on top of EVM-0.9, it is reported that$\{-0.18 \%(\mathrm{Y}),-0.43 \%(\mathrm{U}),-0.08 \%(~\mathrm{V})\}$and$\{-0.03 \%(\mathrm{Y}),- 0.06 \%(\mathrm{U}), 0.18 \%(~\mathrm{V})\}$BD-rate reductions are achieved under random access and low-delay B configurations, respectively, with negligible runtime increase. Due to the good trade-off, the proposed methods have been adopted to EVM platform.

This paper proposes an efficient 3D avatar coding framework that leverages compact human priors and canonical-to-target transformation to enable high-quality 3D human avatar video compression at ultra-low bit rates. The framework begins by training a canonical Gaussian avatar using articulated splatting in a network-free manner, which serves as the foundation for avatar appearance modeling. Simultaneously, a human-prior template is employed to capture temporal body movements through compact parametric representations. This decomposition of appearance and temporal evolution minimizes redundancy, enabling efficient compression: the canonical avatar is shared across the sequence, requiring compression only once, while the temporal parameters, consisting of just 94 parameters per frame, are transmitted with minimal bit-rate. For each frame, the target human avatar is generated by deforming canonical avatar via Linear Blend Skinning transformation, facilitating temporalcoherent video reconstruction and novel view synthesis. Experimental results demonstrate that the proposed method significantly outperforms conventional 2D/3D codecs and existing learnable dynamic 3D Gaussian splatting compression method in terms of rate-distortion performance on mainstream multi-view human video datasets, paving the way for seamless immersive multimedia experiences in meta-verse applications.

To meet the growing demand for advanced video coding, the Joint Video Experts Team (JVET) launched the Enhanced Compression Model (ECM) in April 2021 to explore technologies beyond Versatile Video Coding (VVC). Following this trend, the Audio Video Coding Standard (AVS) Working Group began exploring video coding technologies beyond AVS3 in 2023 and released the Exploration Video Model (EVM) as a development platform. In AVS3, the motion vector prediction (MVP) candidate list for the affine inter mode contains only one 4 -parameter spatial affine candidate, which severely limits its prediction accuracy. Furthermore, the MVD coding approach does not take advantage of the correlation between control points, which introduces redundancy in bitstream representation. To address these issues, this paper proposes an adaptive enhanced method. First, the number of MVP candidates for affine inter mode is extended from a single candidate to multiple ones, with an additional independent 6 -parameter candidate list. Inherited, constructed, and historical motion information are integrated to provide a richer set of prediction candidates. Second, a new adaptive coding method for MVD is designed to reduce bit overhead. Experimental results show that under the random access (RA) and low-delay B-picture (LDB) configurations, the proposed method achieves overall Bjøntegaard Delta Rate (BD-rate) reductions of$\{0.33 \%(\mathrm{Y}), 0.04 \%(\mathrm{U}), 0.34 \%(~\mathrm{V})\}$and$\{0.26 \%(\mathrm{Y}),-0.05 \% (\mathrm{U}), 0.40 \%(~\mathrm{V})\}$, respectively. Due to the excellent performance and acceptable computational complexity of specific components of the proposed scheme, these components have been integrated into the EVM platform as promising coding tools for the next generation AVS standard.

Audio video coding standard (AVS) workgroup started to explore the latest video coding technologies beyond AVS3 standard and released a software platform named exploration video model (EVM) in March 2023. In EVM-0.7, the construction of the motion vector (MV) candidate list for angle weighted prediction (AWP) only considers temporal and spatial neighboring motion information, ignoring non-adjacent candidates. And those motion candidates in the list are directly borrowed from previously coded blocks and thus may not match well with the current coding block. In this paper, we first propose an improved AWP MV candidate list construction method that incorporates more non-adjacent motion information. Second, we introduce an angle-adaptive refinement method to refine the motion candidate in the MV candidate list of AWP. The proposed method was implemented on top of EVM-0.7. And the experimental results show that it overall achieves$\{0.11 \%(\mathrm{Y}), 0.16 \%(\mathrm{U}),\ 0.14 \%(~\mathrm{V})$\} and$\{0.13 \%(\mathrm{Y}), 0.02 \%(\mathrm{U}), 0.13 \%(~\mathrm{V})\}$BD-rate gain on random access (RA) and low delay B (LDB) configurations, respectively, by applying the improved MV candidate list construction method, and$\{0.31 \%(\mathrm{Y}), 0.32 \%(\mathrm{U}),\ 0.43 \%(~\mathrm{V}))$and$\{0.27 \%(\mathrm{Y}), 0.05 \%(\mathrm{U}), 0.37 \%(~\mathrm{V})\}$BD-rate gain on RA and LDB configurations, respectively, by applying both the proposed MV candidate list construction method and the angle adaptive refinement method. Due to the attractive trade-off between performance and complexity, the improved MV candidate list construction was adopted into EVM software as the potential coding tool for the next generation of AVS standard.

Geometric Partitioning Mode (GPM) is an effective coding tool for inter prediction that splits a block into two partitions and blends their predictions. This paper presents a new coding mode, Regression-based Geometric Partitioning Mode (RGPM), which derives a sample-based blending for bi-predictions using a reconstructed template. The RGPM can enhance flexibility in splitting and blending methods compared to GPM. Moreover, two extensions of RGPM scheme are investigated: 1) extending RGPM with template matching (TM) and merge with motion vector difference (MMVD) methods; 2) extending RGPM principle to Spatial Geometric Partitioning Mode (SGPM) for intra prediction. Experimental results show that RGPM with extensions provide 0.12%, 0.24% and 0.23% average luma BD-rate savings on top of enhanced compression model (ECM) in all intra, random access and low delay configurations, respectively. The proposed RGPM is currently adopted in ECM and its two extensions are under study in exploration experiments for future ECM developments.

The subblock merge mode, in which the current coding block is split into multiple subblocks for motion compensation but still inherits the motion at the coding block level, improves the accuracy of the inter prediction and reduces the signaling overhead of motion information at the same time. And thus, it was adopted into versatile video coding (VVC) due to its high efficiency and continually improved in the enhanced compression model (ECM). However, the motion used in subblock merge mode was inherited from the previously coded blocks and may not match well with the current coding block. Thus, to improve the accuracy of the motion for the subblock merge mode, it is proposed to apply template matching (TM) based motion refinement. For subblock temporal motion vector predictor (SbTMVP) candidates, it is proposed to refine both the motion displacement and subblock motion vectors (MVs) based on TM; for affine motion candidates, it is proposed to refine the affine model, including base MV and non-translation parameters, based on TM. The proposed method was implemented on top of ECM, and the experiment results show that by applying the proposed method, it achieves {−0.23%(Y), −0.23%(U), −0.20%(V)} and {−0.09%(Y), −0.36%(U), −0.01%(V)} BD-rate reduction on random access (RA) and low delay B (LDB) configurations, respectively. Due to the good trade-off between performance and complexity, the proposed method was adopted into ECM.

Recently, deep generative models have greatly advanced the progress of face video coding towards promising rate-distortion performance and diverse application functionalities. Beyond traditional hybrid video coding paradigms, Generative Face Video Compression (GFVC) relying on the strong capabilities of deep generative models and the philosophy of early Model-Based Coding (MBC) can facilitate the compact representation and realistic reconstruction of visual face signal, thus achieving ultra-low bitrate face video communication. However, these GFVC algorithms are sometimes faced with unstable reconstruction quality and limited bitrate ranges. To address these problems, this paper proposes a novel Progressive Face Video Compression framework, namely PFVC, that utilizes adaptive visual tokens to realize exceptional trade-offs between reconstruction robustness and bandwidth intelligence. In particular, the encoder of the proposed PFVC projects the high-dimensional face signal into adaptive visual tokens in a progressive manner, whilst the decoder can further reconstruct these adaptive visual tokens for motion estimation and signal synthesis with different granularity levels. Experimental results demonstrate that the proposed PFVC framework can achieve better coding flexibility and superior rate-distortion performance in comparison with the latest Versatile Video Coding (VVC) codec and the state-of-the-art GFVC algorithms. The project page can be found at https://github.com/Berlin0610/PFVC.

This paper proposes to learn generative priors from the motion patterns instead of video contents for generative video compression. The priors are derived from small motion dynamics in common scenes such as swinging trees in the wind and floating boat on the sea. Utilizing such compact motion priors, a novel generative scene dynamics compression framework is built to realize ultra-low bit-rate communication and high-quality reconstruction for diverse scene contents. At the encoder side, motion priors are characterized into compact representations in a dense-to-sparse manner. At the decoder side, the decoded motion priors serve as the trajectory hints for scene dynamics reconstruction via a diffusion based flow-driven generator. The experimental results illustrate that the proposed method can achieve superior rate-distortion performance and outperform the state-of-the-art conventional video codec Versatile Video Coding (VVC) on scene dynamics sequences.

Generative Face Video Coding (GFVC) techniques can exploit the compact representation of facial priors and the strong inference capability of deep generative models, achieving high-quality face video communication in ultra-low bandwidth scenarios. This paper conducts a comprehensive survey on the recent advances of the GFVC techniques and standardization efforts, which could be applicable to ultra low bitrate communication, user-specified animation/filtering and metaverse-related functionalities. In particular, we generalize GFVC systems within one coding framework and summarize different GFVC algorithms with their corresponding visual representations. Moreover, we review the GFVC standardization activities that are specified with supplemental enhancement information messages. Finally, we discuss fundamental challenges and broad applications on GFVC techniques and their standardization potentials, as well as envision their future trends. The project page can be found at https://github.com/Berlin0610/Awesome-Generative-Face-Video-Coding.

Temporal motion vector predictor (TMVP) is a well-known coding technology that has been included in recent video compression standards. The basic concept of TMVP is to utilize the motion continuity between temporal pictures. It obtains motion vector predictor from temporal collocated block by performing temporal motion vector scaling to align the reference pictures of the collocated block to that of to-be-coded block. To exploit the coding performance of TMVP, subblock-based temporal motion vector prediction (SbTMVP) is introduced in Versatile Video Coding (VVC) and is further improved in Enhanced Compression Model (ECM), the software platform for exploring compression capability beyond that of VVC. SbTMVP shares a similar concept to that of TMVP, but obtains motion vectors at the subblock level with a motion displacement which is derived from neighboring blocks. In both VVC and ECM design, the SbTMVP is treated as one of subblock-based merge candidates. In this paper, it is proposed to extend SbTMVP to advanced motion vector predictor (AMVP) for coding motion explicitly. To allow more flexibility, the motion displacement is directly signaled in the bitstream in the proposed mode. Experimental results show that on top of ECM, the proposed mode provides {0.11% (Y), 0.04% (U), 0.05% (V)} and {0.36% (Y), 0.25% (U), 0.33% (V)} BD-rate reduction in random access and low delay B configurations, respectively. The proposed mode is currently being studied in exploration experiments due to its promising compression efficiency gain.

Intra template matching prediction (Intra TMP) is a promising intra prediction tool which generates the prediction block by copying from a reconstructed block of the current frame. The position of the reconstructed block is derived by template matching at both encoder and decoder. Intra TMP has been adopted in enhanced compression model (ECM) for both screen content and natural content due to its outstanding trade-off between coding efficiency and complexity. This paper describes an advanced intra TMP algorithm with fusion techniques to further improve the coding efficiency. The proposed intra TMP fusion scheme includes the following aspects: 1) extended template matching search range, 2) improved search procedure with multiple candidates, 3) adaptive fusion method with template updating, and 4) support fractional-pel precision in intra TMP. Experimental results show that the proposed intra TMP fusion scheme provides 0.76% average luma Bjøntegaard delta rate (BD-rate) reduction with negligible runtime increase over ECM-8.0 in all-intra configuration.

Generative face video coding (GFVC) can achieve high-quality visual face communication at ultra-low bit-rate ranges via strong facial prior learning and realistic generation. However, different kinds of feature representations hinder the interoperability of GFVC, as the bitstream generated from one type of feature representation can only be correctly understood by the corresponding decoder. In this paper, we make the first attempt to propose a face feature transcoding framework that enables translatability in GFVC. By integrating a face feature transcoder at the decoder side, received face features can be translated to decoder-specific ones for subsequent face reconstruction. Furthermore, the translation between different types of face features can be achieved using a unified transcoding framework, facilitating seamless interoperability between different facial representations and their associated decoders. Experimental results demonstrate that three main-stream GFVC codecs, each utilizing different face features, can be effectively adapted to one another while retaining promising coding performance, largely extending the generality of the GFVC system. The project page can be found at https://github.com/xyzysz/GFVC_Software-Decoder_Interoperability.

In Versatile Video Coding (VVC), Cross-component Linear Model (CCLM) predicts chroma samples by assuming a linear relationship between luma and chroma components. In performing CCLM for video in YUV 4:2:0 chroma format, collocated luma samples are firstly downsampled by a low-pass filter to match luma resolution with chroma, and one linear model of luma-chroma sample pairs is applied on the reconstructed luma samples to generate the predicted chroma samples. However, the low-pass downsampling procedure ignores relative spatial variations among luma samples in proximity, such as edge and gradient information. To solve this issue, a new coding technique, namely gradient linear model (GLM), is proposed for further compression efficiency exploration beyond VVC. Instead of using a low-pass filter in CCLM, the GLM utilizes high-pass gradient filters to generate the downsampled luma values. In this paper, two GLM schemes are provided with different trade-offs between coding gain and complexity, including: 1) a 2-parameter scheme that shares the CCLM module framework but replaces the downsampling filter with high-pass gradient filters; 2) a 3-parameter scheme that further combines the luma gradients with the low-pass downsampled luma values. Based on the enhanced compression model (ECM-5.0) software from the joint video experts team (JVET), simulation results show that the 2-parameter GLM achieves average Bjontegaard delta-rate (BD-rate) savings of {1.01%, 1.66%, 1.81%} and {0.69%, 0.95%, 1.12%} for {Y, U, V} components under the All Intra and Random Access configurations, respectively, and the 3-parameter GLM provides {1.28%, 3.23%, 3.28%} and {0.92%, 2.19%, 2.26%} BD-rate savings for {Y, U, V} components under the All Intra and Random Access configurations, respectively. Both of the proposed GLM schemes have been adopted to the ECM software platform.

Decoder-side intra mode derivation (DIMD) is a promising coding tool in the enhanced compression model (ECM) developed by the joint video experts team (JVET). In DIMD, the intra prediction mode of a luma block is derived based on the gradient information of the adjacent luma samples at both encoder and decoder, rather than being explicitly signaled in the bitstream. Inspired by DIMD, a decoder-side chroma intra mode derivation (DCIMD) method is proposed in this paper to improve the coding efficiency of chroma intra prediction. In the proposed DCIMD, the gradient information of both adjacent luma samples and chroma samples is utilized to derive an angular intra mode to predict the current chroma block. There are two advantages to this proposed scheme: 1). the combination of luma information and chroma information can ensure that the derived angular intra mode is closely matched with the texture features of the current chroma block; 2). the number of chroma intra prediction modes can be effectively extended by introducing DCIMD as a new mode without incurring additional expensive bit overhead. Moreover, to further improve the coding efficiency of DCIMD, a chroma intra mode fusion (CIMF) method is proposed. In CIMF, predictions from the DCIMD chroma prediction mode and a cross-component correlation-based chroma prediction mode can be fused together with adaptive weights. Simulation results show that on top of ECM-4.0 the BD-rate savings of 0.07%, 1.17%, and 1.02% on average for Y, Cb, and Cr components, respectively, are achieved in all intra configuration with the proposed two methods. Both the proposed DCIMD and CIMF methods have been adopted into the ECM-5.0 by JVET.

In the development of video coding standard, decoder-side motion derivation technology has been proven to provide promising coding efficiency. With this type of technology, the motion information is derived at the decoder instead of being signaled in the bitstream by the encoder, and thus, the number of bits to be sent are reduced. A typical decoder-side derivation technology is template matching, which refines the motion by finding the closest match between neighboring reconstructed samples and corresponding reference samples in the reference pictures. In this paper, the template matching method is extended to temporal motion vector predictor, bi-prediction with CU-level weight and geometric partition modes in order to fully utilize its benefit. Specifically, template matching is used to determine the prediction direction and reference picture of temporal motion vector predictors, and to decide the bi-predicted weight of bi-prediction merge blocks. In addition, the motion of two geometric partitions are individually refined by the template matching mechanism. Simulation results show that on top of Enhanced Compression Model (ECM), which is the software platform for exploring activities beyond versatile video coding established by the joint video experts team (JVET), the three proposed methods achieve 0.20% luma BD-rate savings in random access configuration and 0.33% luma BD-rate saving in low delay B configuration with negligible encoding and decoding runtime impact. It is worth noting that all three proposed methods have been adopted to the ECM software platform.

We propose a talking face video compression framework by implicitly transforming the temporal evolution into compact feature representation. More specifically, the temporal evolution of faces, which is complex, non-linear and difficult to extrapolate, is modelled in an end-to-end inference framework based upon very compact features. This enables the high-quality rendering of the face videos, which benefits from the learning of dense motion map with compact feature representation. Therefore, the proposed framework can accommodate ultra-low bandwidth video communication and maintain the quality of the reconstructed videos. Experimental results demonstrate that compared with the state-of-the-art video coding standard Versatile Video Coding (VVC) as well as the latest generative compression scheme Face Video-to-Video Synthesis (Face_vid2vid), the proposed scheme is superior in terms of both objective and subjective quality assessment methods.

Recently, there has been a tremendous demand for high-efficiency face video communications, coinciding with the popularization of the digital human character in numerous applications. This paper demonstrates a new communication paradigm of 3D human digital characters in ultra low-bit-rate application scenarios. The paradigm is grounded on the mild assumption of the consistency and persistence of human ap-pearance, such that only the compact features that determine the pose and expression of the 3D character need to be transmitted. The proposed is also expected to benefit virtual-physical world interaction in Metaverse.

Among the recent deep image compression frameworks, transform coding together with a context-adaptive entropy model is the most representative approach to achieve the best coding performance. For entropy model, 2D mask convolution is widely utilized to capture the spatial context, which omits the correlations along channel dimension. To complement to the spatial context, a cross channel context model is proposed. For transform, if given more network layers to improve its representation ability, how to allocate these network layers in forward and inverse transform is investigated. After analyzing the scheme of deep image compression connected with loop filter, we find this investigation can be regarded as a more generalized loop filter. The proposed cross channel context model and generalized loop filter (CCCMGLF) are integrated into the deep image compression framework and jointly optimized to improve the coding performance. Experimental results demonstrate that, using PSNR as distortion metric, the proposed CCCMGLF outperforms VTM-11.0 by 1.20%, 10.82% and 5.38% in terms of BD-rate reductions for Y, U and V components, respectively, for the Kodak dataset. For the JVET CTC sequences, the proposed method outperforms VTM-11.0 by 1.44% for Y but has a coding performance loss of 24.74% and 11.91% for U and V, respectively. Over the baseline deep compression framework, the proposed method provides 7.80%, 12.66% and 11.15% performance improvement for Y, U, and V, respectively, for the Kodak dataset; 9.10%, 12.27%, and 12.68% performance improvement for Y, U and V, respectively, for the JVET CTC sequences. The proposed approaches are applicable in both image compression and intra coding in video compression.

As the latest video coding standard, Versatile Video Coding (VVC) achieves up to 40% Bjontegaard delta bit-rate (BD-rate) reduction compared with High Efficiency Video Coding (HEVC). Recently, Convolutional Neural Network (CNN) has attracted tremendous attention and shows great potential in video coding. In this paper, we design a Multi-Density Convolutional Neural Network (MDCNN) as an integrated in-loop filter to improve the quality of the reconstructed frames. The core of our approach is the multi-density block (MDB), which contains two branches: (a) the basic branch maintaining full resolution for capturing spatially-precise representations, (b) the density branch for learning rich spatial correlation with larger receptive field through down-sampling and upsampling. The feature maps of two branches will be fused into one stream repeatedly. Benefiting from this architecture, spatially-precise representations and density correlations in larger receptive field are utilized to improve the model performance and promote the model's robustness to different input resolutions. Experimental results show that, in terms of BD-rate savings for the (Y, U, V) components compared to the state-of-the-art VVC standard, the proposed MDCNN filter can achieve (5.06%, 13.86%, 13.76%) and (4.36%, 10.85%, 10.91 %) coding gain for the Random Access (RA) configuration and the All Intra (AI) configuration, respectively.

Block-based partitioning is one of the fundamental techniques in video coding. Geometric-based block partitioning is a well-studied method to enable better spatial adaptation to the signal properties. This paper introduces the most recent proposal of advanced geometric-based inter prediction (GIP) made to the state-of-the-art are video coding standard - Versatile Video Coding (VVC). Implemented in the latest test model VTM-6.0 to generalize the existing triangle partition mode (TPM) and evaluated with the Joint Video Experts Team (JVET) Common Test Conditions (CTC) sequences, the proposed advanced GIP scheme provides luma BD-rate reduction of 0.56% for random access (RA) and 1.37% for low-delay (LB) test cases with 2% encoder runtime increase and negligible decoder runtime increase. Furthermore, BD-rate reductions up to 2.92% and 3.49% for RA and LB test cases can be achieved in the absence of multiple related VVC inter prediction tools.

This paper describes a new video coding tool in the Versatile Video Coding standard (VVC) named as luma mapping with chroma scaling (LMCS). Experimental compression performance results for LMCS and non-normative examples for deriving LMCS parameter values are also provided. LMCS has two main components: 1) a process for mapping input luma code values to a new set of code values for use inside the coding loop; and 2) a luma-dependent process for scaling chroma residue values. The first process, luma mapping, aims at improving the coding efficiency for standard and high dynamic range video signals by making better use of the range of luma code values allowed at a specified bit depth. The second process, chroma scaling, manages relative compression efficiency for the luma and chroma components of the video signal. The luma mapping process of LMCS is applied at the pixel sample level, and is implemented using a piecewise linear model. The chroma scaling process is applied at the chroma block level, and is implemented using a scaling factor derived from reconstructed neighboring luma samples of the chroma block.

This paper describes a video coding scheme submitted in response to the joint call for proposals (CfP) on video compression for capability beyond HEVC issued by ITU-T SG16 Q.6 (VCEG) and ISO/IEC JTC1/SC29/WG11(MPEG) in October 2017. It includes video coding techniques for the standard dynamic range (SDR) and high dynamic range (HDR) categories. Design of the core SDR codec in the response is based on the joint exploration model (JEM) reference software developed by the joint video exploration team (JVET). Some of key coding tools in the JEM are significantly simplified to reduce both average and worst-case complexity for hardware design with negligible coding performance loss. Furthermore, two additional coding technologies, namely multi-type tree (MTT) and decoder-side intra mode derivation (DIMD), are used to further improve coding efficiency. For the HDR category, besides the tools used in SDR category, two additional coding tools: an in-loop reshaper and a luma-based QP prediction method are used to further improve HDR coding efficiency. Simulation results demonstrate the high coding efficiency achieved by the proposed video codec at the expense of moderate coding complexity over HEVC. For random access configuration, it achieves average bit rate savings of 35.7% and 4.00% over the HM and JEM anchors with decoding time of 263% and 33%, respectively, for the SDR sequences. For the HDR sequences, the proposed in-loop reshaper is configured to maximize HDR objective metrics, it achieves average bit rate savings of 31.3% and 4.6% over the HM and JEM for wPSNRY metrics for the HDR PQ content.

360-degree video has become popular in recent years with the advances in virtual reality (VR) and augmented reality (AR) technologies and has been rapidly commercialized. To provide viewers with an immersive experience, 360-degree video requires higher resolution and much higher bandwidth compared with conventional 2D video. In a typical 360-degree video compression and delivery framework, the stitched input 360-degree videos, represented in a native projection format, e.g., equirectangular (ERP), are converted into another projection format, e.g., cubemap (CMP), octahedron (OHP), etc. and frame packed before being fed into existing video codecs. The intermediate projection format is important and would potentially improve the representation efficiency and coding performance. Among all the projection solutions, CMP is very popular and has been widely used in the computer graphics community. The intrinsic rectilinear properties of the CMP format are advantageous for the translational motion model in the modern codec architecture. However, in the CMP representation, the samples on the sphere are not evenly distributed within the faces, resulting in a higher density near the face boundaries and a lower density near the face center. Such non-uniform sampling scheme penalizes the video representation efficiency and degrades the coding performance. Adjusted cubemap projection (ACP) was proposed to address such non-uniform sampling by introducing transform functions to improve the sampling uniformity. However, the transform function parameters in ACP are fixed regardless of the content inside each cube face. In this paper, a generalized hybrid cubemap projection (HCP) is proposed to improve the 360-degree video coding efficiency beyond ACP. HCP is defined by a pair of forward transform and inverse transform functions with a pair of horizontal and vertical transform parameters per cube face. The encoder can choose the optimal sampling for each face by adjusting the parameters in the horizontal and vertical directions based on the 360-degree video content characteristics inside each cube face. In order to maintain the boundary continuities between two neighboring faces, in a 3x2 packing layout, vertical parameter constraints are imposed such that faces in each face-row have the same vertical parameters. The HCP parameters are chosen to minimize the end-to-end weighted conversion error and determined using iterative search between the horizontal and the vertical directions. Significant changes in HCP parameter values can cause drastic change in sampling distribution, and may affect the inter-picture coding efficiency. Therefore, an efficient HCP parameter estimation algorithm is proposed to achieve a better trade-off between the temporal sampling adaptation and the inter-picture prediction efficiency by reducing the temporal variation of HCP parameters. The proposed HCP parameter search algorithm reduces the computational complexity by 5x compared to the exhaustive search method. The HCP parameters are selected by the encoder using the first picture of each Intra Random-Access Point (IRAP) and signalled once per IRAP. In SPS, projection format, frame packing parameters including number of faces in horizontal and vertical directions and each face's position and orientation are signalled. In PPS, the horizontal and vertical HCP parameters in 6-bit precision are encapsulated. The proposed HCP solution is implemented upon JEM-6.0 and 360Lib-3.0 software. Simulation results are reported using the test conditions specified in the JVET Call-for-Evidence (CfE) document. Compared with the CMP and ACP formats, the proposed HCP format demonstrates average 3.0 dB (up to 3.6 dB) and 0.2dB (up to 0.4 dB) End-to-End WS-PSNR improvement for the luma (Y) component, respectively, and average luma (Y) BD-rate reductions of 11.5% (up to 23.0%) and 0.5% (up to 1.0%), respectively.

360 Video has become popular in recent years, as commercial interests in deploying Virtual Reality (VR) applications rise. This type of video is usually captured using multi-camera arrays, such as the GoPro Omni camera rig. After separate video streams are captured from multiple cameras, image stitching is applied to obtain a spherical representation of the scene, which spans 360 degrees horizontally and 180 degrees vertically, hence the name 360 video. In the existing workflow of 360 spherical video coding, the 360 video is projected onto the 2D plane with a projection format, such as equirectangular (ERP), cubemap (CMP), equal-area (EAP), octahedron (OHP), etc. Most, if not all, of the currently available 360 video content are provided in ERP format defined in longitude and latitude. Projection format conversion may be performed to convert the native ERP format to another format before coding is applied. Some projection formats contain more than one face, for example, CMP projects the sphere onto a cube of six faces or OHP projects the sphere onto an octahedron of eight faces. For these multi-face projection formats, the faces are packed onto a 2D rectangular picture with a frame packing method. For example, the six faces of CMP can be packed with 4x3 configuration, or 3x2 configuration. Finally, the frame packed picture is coded as a 2D conventional video. Existing video codecs are designed only considering conventional 2D video captured on a plane. When motion compensated prediction uses any samples outside of a reference picture's padding will be performed by simply copying the sample values from the picture boundaries. This repetitive padding method is referred as conventional 2D padding method, which is widely used in video coding standards such as H.264, High Efficiency Video Coding (HEVC). However, a 360 video encompasses video information on the whole sphere, and thus intrinsically has a cyclic property. When considering this cyclic property, the reference pictures of a 360 video no longer have boundaries, as the information they contain is all wrapped around a sphere. This cyclic property holds regardless of which projection format or which frame packing is used to represent the 360 video on a 2D plane. The paper presents a new geometry padding method for motion compensated prediction in 360 video coding. Unlike the conventional padding method for 2D video coding, the proposed geometry padding method extends samples outside of a 2D picture's boundaries using neighboring samples on the sphere. The geometry projection format is considered when performing padding. The corresponding sample outside of a face's boundary (which may come from another side in the same face or from another face), is derived with rectilinear projection. Each face is extended with geometry padding separately. When visualized, the extended faces using geometry padding show continuous texture representing natural extension of the texture inside the face. The proposed geometry padding method is implemented in the HEVC reference software HM-16.12 for the ERP and CMP projection formats. In the simulation, a total of sixteen 4K ERP video and eight 8K ERP video are used. For 8K ground truth 8K video, they are converted to 4K video in ERP and CMP projection formats, coded, and converted back to reconstructed 8K video in ERP format. For 4K ground truth video, they are directly coded in 4K ERP, or converted to CMP consisting of 75% of effective samples, coded, and converted back to reconstructed 4K video in ERP format. Then, the end-to-end spherical PSNR (S-PSNR) is calculated between the original 8K or 4K and the reconstructed 8K or 4K ERP video. BD-rate is calculated between the reference unmodified HEVC, which uses the conventional padding method, and HEVC modified with the proposed geometry padding method. Simulation results showed that geometry padding performs better. For 8K sequences, the proposed geometry padding gives on average luma (Y) BD-rate reduction of 0.3% for ERP and 0.8% for CMP, for 4K sequences, the proposed geometry padding gives on average Y BD-rate reduction of 0.2% for ERP and 1.0% for CMP. Comparing the gains in ERP format with the gains in CMP format, the improvement for CMP is larger. This is because CMP has six faces, therefore the improvement from geometry padding method affects more out-of-boundary samples. The proposed geometry padding method is also especially effective for sequences with fast motion. For example, it achieves BD rate reductions of 4.3%, 2.7%, 1.9%, and 2.5% for Glacier, Chairlift, Sb_in_lot, and Driving, respectively. These four sequences are all captured using moving cameras and have fast moving objects. As a result, the sequences contain a lot of across-the-face-boundary motion which can benefit from improved padding method. Detailed simulation results can be found in JVET contribution JVET-D0075 available at http://phenix.int-evry.fr/jvet/doc_end_user/documents/4_Chengdu/wg11/JVET-D0075-v3.zip.

Recent broadcast standards support Layered Division Multiplexing (LDM) to achieve graceful degradation as signal quality degrades at the receiver. LDM is accomplished by using different constellations within the same Radio Frequency (RF) spectrum. LDM thus enables delivering multiple service tiers in a single broadcast channel. LDM when used in conjunction with scalable source coding codecs such as the Scalable extension of High Efficiency Video Coding (SHVC), further helps improve overall spectrum utilization and efficiency. In this paper we investigate a 2-tier broadcast LDM based service with one service tier aimed at lower video resolution such as 540p, 720p, 1080p for a mobile receiver (smaller/indoor antenna) and the other service tier targeting twice the video resolution of the lower tier, for stationary receivers (larger/outdoor antenna). The primary contribution of this paper is to identify 2-tier transmission configurations of interest to broadcasters and compare the spectrum and bitrate coding efficiency gains of an SHVC-based multi-tier service versus a simulcast (single layer) based multi-tier service for an Advanced Television Systems Committee (ATSC) 3.0 transmission system. Bitrate savings ranging from 38% and 57% is observed for the SHVC based layered system. For large coverage and pedestrian with a receiver test scenarios, channel utilization savings ranging from 23% to 46% is observed. For mobile and tablet in bedroom scenarios a smaller broadcast bandwidth savings ranging from 6% to 9% is observed.

The paper presents the joint proposal by Arris, Dolby and InterDigital as a response to the Call-for-Evidence of the High Dynamic Range and Wide Color Gamut (HDR/WCG) video compression in MPEG. The joint proposal introduces a set of new HDR coding technologies, including the IPT-PQ color space, the adaptive reshaping process, the color enhancement filters, and the adaptive transfer function. These new coding technologies are applied to the decoded output of an HEVC decoder. Hence, no changes to the lower level logics of the HEVC decoder are required to implement the proposal. Formal subjective tests conducted by MPEG confirmed that the proposal could achieve significant subjective quality improvements over the HEVC Main10 anchors at similar bit rates for HDR/WCG video content.

SHVC is the scalable extension of the latest video coding standard High Efficiency Video Coding (HEVC). Color Gamut Scalability (CGS) refers to a scalable use case in which base layer and enhancement layer have different color gamuts. In this case, special inter-layer prediction is needed to improve coding efficiency in SHVC. In this paper, a solution based on asymmetric 3D lookup table is presented for color gamut scalability. Compared to SHVC without CGS coding tool, the proposed solution provides 9.6% - 16.1% overall luma BD-rate reduction in different test cases.

Summary form only given. Upon the completion of the single-layer H.265/HEVC, scalable extensions of the H.265/HEVC standard, called Scalable High Efficiency Video Coding (SHVC), are currently under development. Compared to the simulcast solution that simply compresses each layer separately, SHVC offers higher coding efficiency by means of inter-layer prediction which is implemented by inserting inter-layer reference (ILR) pictures generated from reconstructed base layer (BL) pictures into the enhancement layer (EL) decoded picture buffer (DPB) for motion-compensated prediction of the collocated pictures in the EL. If the EL has a higher resolution than that of the BL, the reconstructed BL pictures need to be up-sampled to form the ILR pictures. Given that the ILR picture is generated based on the reconstructed BL picture, its suitability for an efficient inter-layer prediction may be limited due to the following reasons. Firstly, quantization is usually applied when coding the BL pictures. Quantization causes the BL reconstructed texture to contain undesired coding artifacts, such as blocking artifacts, ringing artifacts, and color artifacts. Secondly, in case of spatial scalability, a down-sampling process is used to create the BL pictures. To reduce aliasing, the high frequency information in the video signal is typically removed by the down-sampling process. As a result, the texture information in the ILR picture lacks certain high frequency information. In contrast to the ILR picture, the EL temporal reference pictures contain plentiful high frequency information, which could be extracted to enhance the quality of the ILR picture. To further improve the efficiency of inter-layer prediction, a low pass filter may be applied to the ILR picture to alleviate the quantization noise introduced by the BL coding process. In this paper, an ILR enhancement method is proposed to improve the quality of the ILR picture by combining the high frequency information extracted from the EL temporal reference pictures together with the low frequency information extracted from the ILR picture. Experimental results show that the proposed method can significantly increase the ILR efficiency for EL coding, under the Common Test Condition of SHVC, which defines a number of temporal prediction structures called Random Access (RA), Low-delay B (LD-B) and Low-delay P (LD-P), on average the proposed method provides {Y, U, V} BD-rate (BL+EL) gains of {2.0%, 7.1%, 8.2%}, {2.2%, 6.7%, 7.6%} and {4.0%, 7.4%, 8.4%} for RA, LD-B, and LD-P, respectively, in comparison to the performance of the SHVC reference software SHM-2.0.