Neural Video Compression has emerged in recent years, with condition-based frameworks outperforming traditional codecs. However, most existing methods rely solely on the previous frame's features to predict temporal context, leading to two critical issues. First, the short reference window misses long-term dependencies and fine texture details. Second, propagating only feature-level information accumulates errors over frames, causing prediction inaccuracies and loss of subtle textures. To address these, we propose the Long-term Spatio-Temporal Enhanced Context (L-STEC) method. We first extend the reference chain with LSTM to capture long-term dependencies. We then incorporate warped spatial context from the pixel domain, fusing spatio-temporal information through a multi-receptive field network to better preserve reference details. Experimental results show that L-STEC significantly improves compression by enriching contextual information, achieving 37.01% bitrate savings in PSNR and 31.65% in MS-SSIM compared to DCVC-TCM, outperforming both VTM-17.0 and DCVC-FM and establishing new state-of-the-art performance.

This study addresses the challenge of domain adaptation in learned image compression, focusing on shifting the model from natural images to user-generated content (UGC) domain. We propose a novel entropy adapter framework augmented with knowledge distillation techniques to improve performance. Unlike existing adapter-based methods that primarily enhance transformation modules, we identify the mismatch between the adapter-based transformation and the fixed entropy network. To resolve this, we introduce adapters within the hypernet and entropy model. Specifically, our decoupled entropy adapter features a deeper residual structure with two independent branches, enabling a separate refinement of mean and scale components. This design improves the accuracy of probability estimation and overall compression efficiency. To further enhance the effectiveness of the adapters, we incorporate a knowledge distillation (KD) strategy with a progressive loss function. It facilitates a smooth transition from KD loss to a rate-distortion (RD) loss in the training process, effectively transferring knowledge from a directly fine-tuned model to the student model. Consequently, this strengthens the adapter's learning capability and improves compression performance. Experimental results show that the proposed method achieves a significant 11.5% bitrate savings compared to the baseline model. Additionally, it demonstrates robust adaptability across diverse network architectures.

Neural video codecs (NVCs), leveraging the power of end-to-end learning, have demonstrated remarkable coding efficiency improvements over traditional video codecs. Recent research has begun to pay attention to the quality structures in NVCs, optimizing them by introducing explicit hierarchical designs. However, less attention has been paid to the reference structure design, which fundamentally should be aligned with the hierarchical quality structure. In addition, there is still significant room for further optimization of the hierarchical quality structure. To address these challenges in NVCs, we propose EHVC, an efficient hierarchical neural video codec featuring three key innovations: (1) a hierarchical multi-reference scheme that draws on traditional video codec design to align reference and quality structures, thereby addressing the reference-quality mismatch; (2) a lookahead strategy to utilize an encoder-side context from future frames to enhance the quality structure; (3) a layer-wise quality scale with random quality training strategy to stabilize quality structures during inference. With these improvements, EHVC achieves significantly superior performance to the state-of-the-art NVCs. Code will be released in: https://github.com/bytedance/NEVC.

In the pursuit of achieving further coding gains beyond the versatile video coding (VVC) standard, the enhanced compression model (ECM) has been initiated by the Joint Video Exploration Team (JVET) with the aim of developing next generation video coding techniques. In ECM, novel coding tools are studied to improve the coding efficiency for both camera-captured content and screen content. Intra block copy (IBC) has been included as a fundamental coding tool in both VVC and ECM, yielding significant improvement in compression efficiency for screen content. This paper presents a method of reconstruction reordered IBC (RR-IBC) to further improve the compression efficiency for screen content, by taking advantage of the symmetry property inherent in screen content sequences. The reconstruction block is flipped horizontally or vertically to restore the characteristics of samples in the original block. A flip-aware adjustment is performed to regulate block vector candidates of the RR-IBC block according to the types of symmetry. Similarly, the reference template of the template-based reordering method for the RR-IBC block is adjusted accordingly to accommodate the geometry property. A motion constraint is applied to restrict the block vector of an RR-IBC coded block to a single direction displacement perpendicular to the flip axis. An RR-IBC flip mode index is signalled to specify how to flip the reconstruction block. Experimental results show that the proposed RR-IBC can provide an average Bjontegaard delta rate (BD-rate) saving of 1.61%/1.79%/1.76% and 3.90%/3.63%/3.63% on Y/Cb/Cr components for class F and class TGM sequences, respectively, with a negligible change on the runtime, compared with ECM-5.0 in all intra configurations. RR-IBC has been adopted into ECM.

As an incubator of next generation video coding techniques beyond versatile video coding (VVC) capability, enhanced compression model (ECM) has been initiated by the Joint Video Exploration Team (JVET). This paper presents an Inter CCP merge mode to improve the coding performance for chroma inter coding. Experimental results show that Inter CCP merge mode provides an average Bjontegaard delta rate (BD-rate) change of 0.01%/-0.69%/-0.76% and -0.03%/-2.31%/-2.38% on Y/Cb/Cr components, respectively, compared with ECM-10.0 in RA/LDB configurations under the common test condition, with a negligible running time change.

Geometric partitioning mode (GPM) splits a coding block into two partitions, which can be non-rectangular, separated by a straight splitting line. Two uni inter-predictions generated by translational motion compensation (TMC) for the two GPM partitions are blended to obtain the final prediction. With a promising coding gain, GPM has been adopted in versatile video coding (VVC). Beyond VVC, enhanced compression model (ECM) introduces several extensions on GPM, but GPM still cannot deal with affine motions well. This paper presents a method of GPM with affine prediction (GPM-affine). A GPM partition can be predicted by affine motion compensation (AMC) or TMC, indicated by a flag. A GPM partition predicted by AMC can be blended with the other GPM partition predicted by AMC, TMC, or intra-prediction. Experimental results show that GPM-affine provides an average luma BD-rate saving of 0.19% compared to ECM-10.0 in random access configurations under the common test condition, with a negligible running time change. On sequences with rich affine motions, 1% coding gain in average is observed. Currently, GPM-affine is under study in exploration experiments (EE) for ECM in JVET.

In the versatile video coding (VVC) standard, cross-component prediction (CCP) is introduced to utilize the correlation between luma and chroma components. To further exploit the cross-component redundancy, a bundle of novel CCP methods with more sophisticated models are studied in enhanced compression model (ECM), which is a platform targeting at the next generation video coding standard, developed by JVET. This paper presents two methods, known as local-boosting CCP (LB-CCP) and non-local CCP (NL-CCP) to improve CCP with local and non-local information. With LB-CCP, prediction samples of CCP can be filtered with neighbouring samples. Besides, neighbouring template costs are calculated to determine the range of training samples, as well as the cross-component prediction method used in the chroma fusion mode. With NL-CCP, a CCP model can be derived with samples non-adjacent to the current block. As an equivalent but simpler implementation, the CCP model can be inherited from a CCP-coded neighbouring block as a spatial CCP candidate. Moreover, the CCP model of a previous CCP-coded block can be stored in a history-based table, which can be fetched and used by the current block as a history-based CCP candidate. A NL-CCP candidate list is built with the two kinds of candidates. The encoder can select the optimal candidate and send an index to the decoder. Experimental results show that LB-CCP together with NL-CCP can provide an average Bjontegaard delta rate (BD-rate) reduction of 0.27%, 2.31%, 2.44% on Y, Cb, Cr components, respectively, with a negligible change in the running time, compared with ECM-8.0 in all intra configurations. Both LB-CCP and NL-CCP have been adopted into ECM.

In versatile video coding (VVC) and enhanced compression model (ECM), which is developed by JVET to explore novel techniques beyond VVC, cross-component prediction (CCP) methods derive CCP models with local samples. This paper presents a method of non-local cross-component prediction (NL-CCP), where the cross-component relationship can be estimated with non-local information. First, the CCP model can be inherited from a CCP-coded neighbouring block. Second, the CCP model of a previous CCP-coded block is stored in a history-based table, which can be fetched and used by the current block. Finally, a candidate list is built with these two kinds of NL-CCP candidates. The encoder can select the optimal candidate and send an index to the decoder. Experimental results show that NL-CCP provides an average BD-rate saving of 0.19%/2.12%/2.30% on Y/Cb/Cr components with a negligible change on the running time, compared with ECM-8.0 in all intra configurations. NL-CCP has been adopted into ECM.

Geometric partitioning mode (GPM) is employed in the latest video coding standard known as versatile video coding (VVC). GPM provides a capability to partition an inter-coded block into two geometric regions with different motion vectors (MVs). However, MVs of the two partitions of a GPM coding unit (CU) are restricted to be selected from a MV candidate list, lacking the flexibility to refine the MVs. This paper presents a method named geometric partitioning with motion vector difference (GMVD). With the proposed GMVD mode, the MV of each GPM partition, partitioned by GPM, is generated as a sum of a base MV selected from a MV candidate list and a motion vector difference, signaled in a similar way as merge motion vector difference (MMVD) in VVC. The presented GMVD has been adopted into the enhanced compression model (ECM), developed by JVET to explore novel coding tools beyond VVC. Experimental results show that the presented GMVD provides an additional average luma BD-rate gain of 0.27% and 0.42% in RA and LB configurations over ECM-1.0 under the JVET CTC, with decent trade-off between compression efficiency and encoder complexity. Moreover, the decoder complexity of GMVD is negligible.

Block partitioning methods play a critical role in hybrid video coding standards such as HEVC and VVC. A quaternary tree (QT) / binary tree (BT) / ternary tree (TT) partitioning structure has been adopted in VVC, which improves the coding performance significantly compared to HEVC. In this paper, we explore two advanced block partitioning methods beyond VVC. Different from block symmetric partitioning methods such as QT and BT, the two proposed advanced partitioning methods are unsymmetric, namely unsymmetric quaternary tree (UQT) and unsymmetric binary tree (UBT). UQT and UBT split a block into four and two partitions unsymmetrically, respectively. Several modifications are proposed to make coding tools compatible with a block holding a width or a height not in a form of 2$^{\mathrm{n}}$. Besides, fast algorithms are presented to accelerate the encoder. Experimental results show that the proposed method provides an average BD-rate saving about 1.5% with a negligible decoding time change, compared with VVC in random access configurations under the JVET common test conditions.

This paper presents a self-aware filter estimation cross-component linear model(SAFE-CCLM) intra-prediction method, in which a set of luma down-sampling filter candidates are predefined, and the selected down-sampling filter is adaptively determined at both encoder and decoder sides in a consistent way, without signalling a filter index. To derive the selected filter of a chroma block, the luma neighbouring samples are tried to be down sampled by each filter candidate. With the down-sampled luma neighbouring samples, a linear model is derived for each filter candidate, and prediction values of chroma neighbouring samples are calculated with the model. Then a sum of absolute difference (SAD) cost between prediction values and reconstruction values of chroma neighbouring samples is obtained. Finally, the filter candidate with the lowest SAD cost is determined to be the selected down-sampling filter, used to generate prediction values of the current chroma block. Experimental results show that SAFE-CCLM provides an average bitrate saving of 0.80% on the luma component for All Intra configurations under the JVET common test conditions, with negligible runtime changes.

In video coding, motion information consisting of motion vectors and reference index is typically involved in motion compensation. Motion list is widely used to efficiently compress the motion information, in which a motion index indicating the motion information is signaled. And the compression efficiency can be improved by template matching based motion list reordering. Besides, the motion information is further refined before being used in motion compensation by motion refinement process such as decoder-side motion vector refinement and template matching. However, the original motion information of the motion list rather than the refined motion information is used in the motion list reordering, which limits the coding performance. Therefore, an enhanced motion list reordering (EMLR) approach is proposed in this paper, in which the refined motion information is used in the motion list reordering. To derive the refined motion information, a dedicated motion refinement with a simplified version of motion refinement process is proposed. Furthermore, a simplified version of EMLR with two fast algorithms (EMLR-S) is proposed. Experimental results demonstrate that EMLR can achieve 0.19% BD-rate saving on average, and EMLR-S can achieve 0.1 % BD-rate saving with negligible coding complexity change compared to ECM-4.0 under random access configuration.

In VVC, affine motion compensation (AMC) is a powerful coding tool to address non-translational motion, while history-based motion vector prediction (HMVP) is an efficient approach to compress motion vectors. However, HMVP was designed for translational motion vectors, without considering control point motion vectors (CPMV) for AMC. This paper presents a method of history-parameter-based affine model inheritance (HAMI), to utilize history information to represent CPMV more efficiently. With HAMI, affine parameters of previously affine-coded block are stored in a first history-parameter table (HPT). New affine-merge, affine motion vector prediction candidates and regular-merge candidates can be constructed with affine parameters fetched from the first HPT and base MVs fetched from neighbouring blocks in a “base-parameter-decoupled” way. New affine merge candidates can also be generated in a “base-parameter-coupled” way from a second HPT, which stores base MV information together with corresponding affine parameters. Besides, pair-wised affine merge candidates are generated by two existing affine merge candidates. Experimental results show that HAMI provides an average BD-rate saving about 0.34 % with a negligible change on the running time, compared with ECM-3.1 in random access configurations. HAMI has been adopted into ECM.

This paper presents a decoder derived cross-component linear model (DD-CCLM) intra-prediction method, in which one or more linear models can be used to exploit the similarities between luma and chroma sample values, and the number of linear models used for a specific coding unit is adaptively determined at both encoder and decoder sides in a consistent way, without signalling a syntax element. The neighbouring samples are classified into two or three groups based on a K-means algorithm. Moreover, DDCCLM can be combined with normal intra-prediction modes such as DM mode. The proposed method can be well incorporated with the state-of-the-art CCLM intra-prediction in the Versatile Video Coding standard. Experimental results show that the proposed method provides an overall average bitrate saving of 0.52% for All Intra configurations under the JVET common test conditions, with negligible runtime change. On sequences with rich chroma information, the coding gain is up to 2.07%.

Versatile Video Coding (VVC) is the latest video coding standard, finalized in July 2020. Dependent random access point (DRAP) is supported in HEVC and VVC, through a supplemental enhancement information (SEI) message, for improved coding efficiency in the case of random access. A DRAP picture is an inter-coded picture that can only refer to the most recent intra random access point (IRAP) picture earlier in decoding order. A DRAP picture can be used as a random access point in the bitstream given that the associated IRAP picture is available. Although the DRAP picture increases coding efficiency for random access, the performance is limited since only the associated IRAP picture is allowed to be used as the reference picture for DRAP pictures. In this paper, an extended dependent random access point (EDRAP) picture is proposed to further improve the coding efficiency for random access. Specifically, a particular EDRAP picture only relies on some pictures in a set of pictures consisting of the associated IRAP picture and certain EDRAP pictures between the associated IRAP picture and the particular EDRAP picture in decoding order. EDRAP pictures can be used as random access points as long as the few dependent IRAP or EDRAP pictures are provided. Simulation results demonstrate that the proposed method can achieve 10.2% Bj$\phi$ntegaard-Delta rate saving on average compared to VTM-11.0 under random access configuration. It is noted that the support of EDRAP for VVC, through another SEI message, was adopted by the JVET at its 21stmeeting in January 2021.

In the emerging next generation video coding standard Versatile Video Coding (VVC) developed by the Joint Video Exploration Team (JVET), sub-block-based inter-prediction plays a key role in promising coding tools such as Affine Motion Compensation (AMC) and sub-block-based Temporal Motion Vector Prediction (sbTMVP). With sub-block-based inter-prediction, a coding block is divided into sub-blocks, and the motion information of each sub-block is derived individually. Although sub-block-based inter-prediction can provide a higher quality prediction benefiting from a finer motion granularity, it still suffers two problems: uneven prediction quality and boundary discontinuity. In this paper, we present a method of interweaved prediction to further improve sub-block-based inter-prediction. With interweaved prediction, a coding block with AMC or sbTMVP mode is divided into sub-blocks with two different dividing patterns, so that a corner position of a sub-block in one dividing pattern coincides with the central position of a sub-block in the other dividing pattern. Then two auxiliary predictions are generated by AMC or sbTMVP with the two dividing patterns, independently. The final prediction is calculated as a weighted-sum of the two auxiliary predictions. Theoretical analysis and statistical data prove that interweaved prediction can significantly mitigate the two problems in sub-block-based inter-prediction. Simulation results show that the proposed methods can achieve 0.64% BD-rate saving on average with the random access configurations. On sequences with rich affine motions, the average BD-rate saving can be up to 2.54%.

360-degree video is emerging as a new way of offering immersive visual experience. The quality evaluation of 360- degree video is more difficult compared to the quality evaluation of conventional video. However, to ensure successful development of 360-degree video coding technologies, it is essential to precisely measure both objective and subjective quality. In this paper, an overview of the 360-degree video quality evaluation framework established by the joint video exploration team (JVET) of ITU-T VCEG and ISO/IEC MPEG is provided. This framework aims at reproducing the different processes in the 360-degree video processing workflow that are related to coding. The results of different experiments conducted using the JVET framework are reported to illustrate the impact on objective and subjective quality with different projection formats and codecs.

The Alliance for Open Media (AOMedia) aims at generating a new, royalty-free video codec specification named AV1 that is scheduled for completion in the 2nd half of 2017. To facilitate AV1 market acceptance and rapid deployment, attention has been paid toward development of hardware friendly coding tools. This paper presents the following techniques to facilitate hardware implementation of AV1 inter prediction: 1) a simplified motion vector prediction scheme to reduce memory footprint / bandwidth; 2) methods to limit / remove the dependency between the computation of probability context model and the reconstruction of motion vectors to improve hardware throughput. Experimental results on a wide range of test clips show negligible quality impact for 1) and minor quality impact for 2) while the motion vectors line buffer memory footprint is reduced by 50%, motion vectors local memory is reduced by 33%, and the throughput of the arithmetic decoder remains unchanged.

This paper describes the algorithm and processing flow of the in-loop deblocking filter in the VP9 coding standard, one of the most computationally intensive toolsets of the VP9 codec. Due to its inherent data dependency, it is a great challenge to efficiently implement the algorithm on massively parallel computing architectures, such as a General Purpose Graphical Processing Unit (GPGPU). In this paper, we describe the challenges involved in a GPGPU implementation of the VP9 Deblocking filter and introduce an innovative thread dispatching approach to address the parallelization challenges. This approach has been successfully implemented and productized in the VP9 decoder and encoder solutions on Intel GPUs. In order to further improve the parallelism of the deblocking algorithm itself, an improved in-loop deblocking algorithm and process flow is jointly proposed by Intel and Microsoft for the upcoming AV1 codec standard, developed by the Alliance for Open Media (AOM). A description of the algorithm and evaluation of the quality impact of this algorithm is presented with respect to the current state of the art AV1 reference codec.

In this paper, a fast algorithm is proposed to speed up the motion and disparity estimation in stereoscopic video coding. Based on the stereo-motion consistency constraint, an iterative search strategy is suggested to get the motion and disparity vectors simultaneously. A credible base vector selection scheme and an adaptive search range adjustment technique are designed to further strengthen the iterative search. Results show that the complexity can be significantly reduced compared to the JMVM full search with a negligible quality drop.