The in-loop filter tools in video coding, such as De-Blocking Filter (DBF) and Adaptive Loop Filter (ALF), can significantly improve the video compression performance of the VVC standard. In this paper, the coding-information based improvements for DBF and ALF are proposed to further increase the coding efficiency beyond VVC. The coding-information can be referred to the partitioning information which is not fully utilized by in-loop filters in the VVC standard. Two improvements are included in this paper. The first improvement is the coding-information based classification for ALF. In the current design of ALF, the partitioning information has not been considered for the classification of both online and offline filters. To further increase the classification diversity and efficiency, the coding-information is introduced into classification of both online and offline filters through an always applied rule. The second improvement is the boundary-aware offset refinement. In current DBF and ALF design, the offset generated by DBF and ALF cannot be further adjusted after the filtering process. To provide more possibilities for the filtering offset, an offset refinement process is proposed. The offset can be refined by a non-linear function with the help of the coding-information. The proposed two improvements are implemented on top of ECM-14.0 software which is published by JVET. Experimental results show that the classification method can achieve -0.08% and -0.12% while the offset refinement method can achieve -0.05% and -0.11% BD-rate reduction on the luma component in Random Access and Low-Delay B configurations respectively. Both proposed improvements have been adopted into ECM.

We introduce a novel and challenging task within the AIGC era: coding for AIGI (AI-generated images). Specifically, we propose the first AIGI dataset called PKU-AIGI-500K, which is meticulously constructed based on five major foundation models with diverse prompts. Furthermore, We conduct extensive and systematic analysis of the essential characteristics of AIGC images. We thoroughly benchmark the rate-distortion performance and runtime complexity analysis of conventional and learned image coding solutions that are openly available, revealing new insights for emerging studies in AIGI compression. The main contributions of this paper can be summarized as follows: (i) we contribute and build the first AIGI dataset, PKU-AIGI-500K, containing 105+ prompts and 528k+ images based on five generative models. Additionally, we analyze the image features that may be beneficial for image compression, such as quality, texture, color, etc., providing novel insights into available solutions and paving the way for future research. (ii) Building upon the proposed PKU-AIGI-500K, we evaluate the compression efficiency using some popular traditional codecs and learning-based codecs to form a strong benchmark. We also observe that the learning-based models trained on natural images cannot achieve competitive performance on the AIGIs without fine-tuning, emphasizing the necessity of a sufficiently large dataset to advance the research of AIGI coding. (iii) The evaluation and benchmarking are accomplished as an AIGIs’ compression project.

Multi-pass Decoder Side Motion Vector Refinement (DMVR) based on a Bilateral Matching (BM) concept is a very important tool being used in video codecs to improve the Motion Vector (MV) accuracy on the decoder side. In VVC, a single stage DMVR is applied on each 16x16 subblock to refine its MV. Exploring Coding Model (ECM), which is being developed for beyond VVC activity, adopts multi-pass DMVR. First, a BM approach is used to adjust the MV for each Prediction Unit (PU). Next, another BM approach is used to adjust the MVs for each 16x16 subblock. Finally, a Bi-directional Optical Flow (BDOF) based approach is used to adjust the MVs for each 8x8 subblock. In this paper, an iterative BDOF to improve multi-pass DMVR is proposed, where an additional pass of BDOF is added as the 4th stage of the multi-pass DMVR. To balance the gain and the complexity, the granularity of subblocks is adjusted adaptively in the third and forth stages of multi-pass DMVR. Finally, a regularization factor is added to avoid unnecessary over-tuning of BDOF adjustment. The proposed method is implemented on top of ECM-8.0. Experimental results show that the proposed method provides more than -1% bitrate saving for some sequences compared to ECM-8.0 under random access configurations. This method is adopted into ECM-10.0.

AVS3 is an emerging video coding standard, and screen content coding is a very important feature of AVS3. This paper presents a method of Implicit-Selected Transform Skip (ISTS) to further improve the screen content coding performance. With ISTS, transform skip mode is introduced as an optional substitution to transform-coding on residual signals of blocks with intra-prediction. The indication of whether to apply transform skip is hidden into the Parity of the Number of Non-zero Coefficients (PNNC) of a residual block, instead of being signaled to the decoder. Moreover, the coefficients of an intra-coded block are reordered to make the coefficients more compact. Experimental results show that the proposed method can achieve 12.04%, 8.15% and 10.19% BD-rate savings on average under All Intra (AI), Low Delay (LD) and Random Access (RA) configurations, respectively, with the encoding time reduced by 5% to 8%. ISTS has been adopted into AVS3.

Intra prediction is typically used to exploit the spatial redundancy in video coding. In the latest video coding standard Versatile Video Coding (VVC), 67 intra prediction modes are adopted in intra prediction. The encoder selects the best one from 67 modes and signals it to the decoder. Bits consuming of signaling the selected mode may limit the coding efficiency. To reduce the overhead of signaling the intra prediction mode, a probability-based decoder-side intra mode derivation (P-DIMD) is proposed in this paper. Specifically, an intra prediction mode candidate set is constructed based on the probabilities of intra prediction modes. The probability of an intra prediction mode is mainly estimated in two ways. First, the textures are typically continuous within a local region and intra prediction modes of neighboring blocks are similar to each other. Second, some intra prediction modes are preferable to be used than others. For each intra prediction mode in the constructed candidate set, intra prediction is processed on a template to calculate a cost. The intra prediction mode with the minimum cost is determined as the optimal mode and used in the intra prediction of the current block. Experimental results demonstrate that P-DIMD can achieve 0.56% BD-rate saving on average compared to VTM-11.0 under all intra configuration.

This paper describes an adaptive self-guided loop filter for video coding. The filter acts as a loop filter to reduce the artifacts caused by quantization. The proposed filter is applied on the reconstruction samples of a Transform Unit (TU). The unfiltered input block is regarded as a guidance for self-guided edge-preserving smooth filtering. The filtering strength for each sample inside the input block is decided according to the TU size, corresponding QP, and statistical information generated from a designed filtering window adaptively. The proposed filter could be performed as a prediction-loop-filter in which the filtered samples can be referenced by intra prediction. Alternatively, the proposed filter can be moved to the post-loop-filter stage that after the deblocking filter for a low decoding dependency. The proposed method has been implemented and tested according to the common test conditions in Versatile Video Coding (VVC) test model version 11.0 (VTM 11.0). The post-loop-filter version can achieve 0.46% and 0.41% BD-rate reduction in All Intra and Random-Access configurations respectively.

This paper overviews the motion vector coding and block merging techniques in the Versatile Video Coding (VVC) standard developed by the Joint Video Experts Team (JVET). In general, inter-prediction techniques in VVC can be classified into two major groups: “whole block-based inter prediction” and “subblock-based inter prediction”. In this paper, we focus on techniques for whole block-based inter prediction. As in its predecessor, High Efficiency Video Coding (HEVC), whole block-based inter prediction in VVC is represented by adaptive motion vector prediction (AMVP) mode or merge mode. Newly introduced features purely for AMVP mode include symmetric motion vector difference and adaptive motion vector resolution. The features purely for merge mode include pairwise average merge, merge with motion vector difference, combined inter-intra prediction and geometric partitioning mode. Coding tools such as history-based motion vector prediction and bidirectional prediction with coding unit weights can be applied on both AMVP mode and merge mode. This paper discusses the design principles and the implementation of the new inter-prediction methods. Using objective metrics, simulation results show that the methods overviewed in the paper can jointly achieve 6.2% and 4.7% BD-rate savings on average with the random access and low-delay configurations, respectively. Significant subjective picture quality improvements of some tools are also reported when comparing the resulting pictures at same bitrates.

In this paper, we focus on computationally modeling of the local texture correlations, in an effort to better explore the coding modes with higher priorities in the rate-distortion optimized intra coding. In particular, strong correlations and continuities of local texture with neighboring blocks have been revealed in our analysis, and empirical justifications provide us inspirations on the joint optimization of rate-distortion-complexity when angular modes become finer to adapt the local textures. We examine the philosophy with extensive experiments conducted for refining the intra full-RD list. The results show that better coding performance with on average 0.72% and 3.00% BD-Rate savings for the natural scene and screen content sequences can be achieved in AVS3 test model HPM-5.0 under all intra configuration, with negligible encoding and decoding time variations.

There has been a consensus regarding the intra prediction technique in video coding that the spatial redundancy can be efficiently removed by the locally accessible reference samples with certain projections and interpolations. In this paper, we revisit the short and long range correlations of the image content in the context of video coding, and it is interesting to find that the natural scene videos exhibit substantially different characteristics from screen content videos. This motivates us to redesign the intra mode coding method based on both short and long range correlations, as the existing approaches based on local content correlations cannot always effectively capture the most probable mode. One key feature of the proposed method is that it achieves unified content adaptive coding and is applicable across different video content. Experimental results on the versatile video coding (VVC) platform VTM-3.0 show the effectiveness of the proposed approach, leading to 3.73% bit rate savings for screen content videos and 0.10% bit rate savings for natural scene videos under all intra configuration.

The flexible partitioning structure such as quad-tree plus binary-tree (QTBT) plays important roles in the next generation video coding standard. This paper investigates new coding unit (CU) partitioning methods to further improve the compression efficiency. In particular, we propose the extended quad-tree partitioning and asymmetric ternary-tree partitioning to further develop the principle behind the existing partitioning structure. First, a central extended quad-tree (CENTRAL-EQT) partitioning is introduced, which extends the traditional QT partitioning with central pattern and generates four elaborately designed sub-CUs with different sizes. Second, we propose a parallel extended quad-tree (PARALLEL-EQT) partitioning that allows the CU to split along a single direction, leading to four identical size sub-blocks. Third, we present an asymmetric ternary-tree (ATT) method, which splits the CU asymmetrically into three sub-blocks. The proposed new partitioning methods allow the interleaving with binary-tree partitioning for enhanced adaptability, and they can be jointly enabled to capture different characteristics of the local content. Simulation results on the JEM7-QTBT-Only platform show that averagely 4.92%, 4.81% and 5.08% BD-Rate savings are achieved for the luma component by the proposed methods under random access (RA), low-delay P (LDP) and low-delay B (LDB) configurations, respectively, with around 725% encoding time increase and negligible decoding time increase. Furthermore, experimental results also reveal that the proposed partitioning schemes are effective when cooperating with the upcoming Versatile Video Coding standard.

In the emerging next generation video coding standard Versatile Video Coding (VVC) developed by the Joint Video Exploration Team (JVET), sub-block-based inter-prediction plays a key role in promising coding tools such as Affine Motion Compensation (AMC) and sub-block-based Temporal Motion Vector Prediction (sbTMVP). With sub-block-based inter-prediction, a coding block is divided into sub-blocks, and the motion information of each sub-block is derived individually. Although sub-block-based inter-prediction can provide a higher quality prediction benefiting from a finer motion granularity, it still suffers two problems: uneven prediction quality and boundary discontinuity. In this paper, we present a method of interweaved prediction to further improve sub-block-based inter-prediction. With interweaved prediction, a coding block with AMC or sbTMVP mode is divided into sub-blocks with two different dividing patterns, so that a corner position of a sub-block in one dividing pattern coincides with the central position of a sub-block in the other dividing pattern. Then two auxiliary predictions are generated by AMC or sbTMVP with the two dividing patterns, independently. The final prediction is calculated as a weighted-sum of the two auxiliary predictions. Theoretical analysis and statistical data prove that interweaved prediction can significantly mitigate the two problems in sub-block-based inter-prediction. Simulation results show that the proposed methods can achieve 0.64% BD-rate saving on average with the random access configurations. On sequences with rich affine motions, the average BD-rate saving can be up to 2.54%.

The quad-tree plus binary-tree (QTBT) coding unit (CU) partitioning structure, which has been adopted to the next generation video coding standard, shows promising coding performance when compared with the conventional quad-tree structure in HEVC. In this paper, we propose the Extended Quad-tree (EQT) partitioning, which further extends the QTBT scheme and increases the partitioning exibility. More specifcally, EQT splits a parent CU into four sub-CUs of dierent sizes, which can adequately model the local image content that cannot be elaborately characterized with QTBT. Meanwhile, EQT partitioning allows the interleaving with BT partitioning for enhanced adaptability. Experimental results on the JEM7-QTBT-Only platform show that EQT brings better coding performance with 3.17%, 3.20% and 3.06% BD-Rate gains under random access, low-delay P and low-delay B configurations, respectively.

In this paper, History-based Motion Vector Prediction (HMVP) is presented for video coding. With the proposed method, a table of HMVP candidates is maintained and updated on-the-fly. After decoding one inter-coded block, the table is updated by appending the associated motion information to the table as a new HMVP candidate. A First-In-First-Out (FIFO) rule is applied to manage the table. The HMVP candidates could be added to the Advanced Motion Vector Prediction (AMVP) candidate list as additional motion vector predictors. And they could also be added to the merge candidate list as additional merge candidates. With the proposed method, the motion information of previously coded blocks even not adjacent to the current block can be utilized for more efficient motion vector prediction. Simulation results have validated the efficiency of HMVP, wherein up to 4% BD rate saving could be achieved. The proposed method has been adopted by the next generation video coding standard, named Versatile Video Coding (VVC) developed by Joint Video Exploration Team (JVET).

With affine motion compensation (AMC) in the emerging next generation video coding standard Versatile Video Coding (VVC), a coding-block is divided into sub-blocks, and each sub-block is assigned with an individual motion vector derived by the affine model. The sub-block-based design for AMC faces a dilemma. With smaller sub-blocks, AMC can achieve a better coding performance but suffers a higher complexity burden. In this contribution, an interweaved prediction approach is proposed for AMC to address the dilemma. With the interweaved prediction, a coding block is divided into sub-blocks with two different dividing patterns. Then two auxiliary predictions are generated by AMC with the two dividing patterns respectively. The final prediction is calculated as a weighted-sum of the two auxiliary predictions. The interweaved prediction is only applied on the luma-component for affine-coded blocks with uni-prediction. Simulation results show 0.53% Bjøntegaard-Delta rate savings in average can be achieved compared to VTM-3.0 under random access configurations. The coding gain on sequences with affine motions is up to 3.3%.

Bi-directional optical flow (BDOF) is an efficient coding tool that has been recently adopted into Versatile Video Coding (VVC) standard. With BDOF, bi-predictive prediction samples of one coding block are enhanced via higher-precision motion vectors (MVs) derived from its two reference blocks. In this way, the energy of prediction error could be reduced, resulting in better coding performance. In VVC, the derived motion information is only used to enhance prediction samples. In this paper, it is proposed to use the derived motion information to also refine decoded MVs. The refined MVs may be used as spatial motion vector prediction (MVP) for the following coding units (CUs), as the temporal MVP for the subsequent pictures, and in the deblocking filtering process. Furthermore, the refined MVs can be used to perform motion compensation (MC) again to further improve the quality of the prediction samples. Simulation results show that the proposed methods can achieve -1.18% BD-rate saving in average under the random access configuration on top of the existing BDOF design in VVC.

The third generation of the audio video coding standard (AVS3) is the latest standardization developed by Audio Video Coding Standard Working Group of China on the basis of IEEE 1857.4. In this paper, we propose a History-based Motion Vector Prediction (HMVP) method for inter coding, which fully explores the history motion information of previous coded blocks. More specifically, an HMVP table, which records HMVP candidates, is elaborately maintained with limited capacity. In particular, the motion information of the previous coded blocks updates the HMVP table on-the-fly by appending the candidate to the last entry of the table after encoding or decoding an inter block. As such, the HMVP candidates act as additional modes for skip or direct modes to enhance coding efficiency. Besides, HMVP occupies limited memory to store motion info, which is crucial in real applications. Experimental results show that HMVP brings better coding performance with up to 1.38% and 2.33% BD-Rate reduction for random access and low-delay P configurations, respectively, on the latest TAVS3-2.2 platform. Moreover, nearly no encoding and decoding burden is introduced by HMVP. The proposed HMVP has been adopted by AVS3.

Audio and Video coding Standard-3 (AVS-3) is an emerging next-generation video coding standard beyond AVS-2. The Quad-Tree/Binary-Tree (QTBT) plus Extend Quad-Tree (EQT) partitioning structure has been adopted in AVS3. In this paper, Unsymmetrical Quad-Tree (UQT) partitioning is proposed to explore the potential coding efficiency of partitioning further. Particularly, UQT partitioning splits a parent block into four sub-blocks asymmetrically with 1:4:2:1 or 1:2:4:1 ratio. Not only UQT partitioning can complement the current QTBT plus EQT partition structure but also can characterize the pattern of multifarious video content more effectively. Experiments are conducted on the HPM3.2 platform and the simulation results demonstrate that the proposed UQT partitioning can achieve 0.66%, 0.77% and 0.75% BD-bitrate savings for Y, Cb and Cr components on average under the random-access configuration, respectively.

Affine Motion Model (AMM) based inter prediction, which can represent complex motions such as zooming, rotation or shearing, has been adopted into the Versatile Video Coding (VVC) standard. AMM is defined by Control Point Motion Vectors (CPMVs) in VVC. On the other hand, Adaptive Motion Vector Resolution (AMVR) has also been adopted into VVC standard due to a favorable trade-off between the Motion Vector (MV) precision and the bit consumption on MV Differences (MVDs). However, AMVR is only applied to the Translational Motion Model (TMM), and AMM cannot benefit from it. In this paper, it is proposed to extend AMVR to AMM. Specifically, 1-pixel, 1/4-pixel and 1/16-pixel MV precisions are allowed and can be selected adaptively by each affine-inter mode coded Coding Unit (CU). Simulation results reportedly show that the proposed method can achieve 0.32% BD-rate saving on average under the Random Access configuration.

AVS3 adopts quad-tree (QT) with nested binary tree (BT) and extended quad-tree (EQT) partitioning, which shows promising compression performance when compared to the conventional QT partitioning in AVS2 and HEVC. However, the interleaved and recursive splitting manner significantly increases the computational complexity of the encoder, which may impede the real applications of AVS3. This paper proposes fast coding unit splitting decision methods for QT, BT and EQT partitioning. In particular, the Bayesian decision rule is employed with Skip states, and BT and EQT partitioning can be early terminated. Additionally, the average splitting depth that is produced by BT partitioning, is incorporated as the prior information for terminating the EQT and QT splittings. Moreover, we explore the splitting settings for CUs that are predicted with Skip mode, with the aim of eliminating unnecessary partition attempts. Experimental results show that the proposed fast algorithms are effective and provide a good trade-off between computational complexity and coding performance. In particular, 69% encoding time reduction is achieved with only 0.55% increase in terms of BD-Rate on average, which greatly benefits the practical implementations of the AVS3 in real applications. The proposed methods have been adopted into TAVS3 reference software.

The Joint Video Exploration Team (JVET) has been developing an emerging standard Versatile Video Coding (VVC), which includes screen contents as one of its requirements. Intra block copy (IBC) and palette coding are the two powerful coding tools for screen content coding. In this paper, a compound palette mode is proposed to exploit the advantages of both IBC and palette coding, which allows samples to be reconstructed by either IBC predictions or palette entries. The proposed method is evaluated with VVC reference software VTM4 on typical sequences containing "text and graphics with motion". Experimental results report significant coding gain that the proposed scheme can achieve 7.80% and 1.03% BD-rate savings under AI conditions on average when compared with VTM4 and the existing palette scheme.

Frame rate up-conversion (FRUC) improves the viewing experience of a video because the motion in a FRUC-constructed high frame-rate video looks more smooth and continuous. This paper proposes a multiple hypotheses Bayesian FRUC scheme for estimating the intermediate frame with maximum a posteriori probability, in which both temporal motion model and spatial image model are incorporated into the optimization criterion. The image model describes the spatial structure of neighboring pixels while the motion model describes the temporal correlation of pixels along motion trajectories. Instead of employing a single uniquely optimal motion, multiple “optimal” motion trajectories are utilized to form a group of motion hypotheses. To obtain accurate estimation for the pixels in missing intermediate frames, the motion-compensated interpolations generated by all these motion hypotheses are adaptively fused according to the reliability of each hypothesis. We revealed by numerical analysis that this reliability (i.e., the variance of interpolation errors along the hypothesized motion trajectory) can be measured by the variation of reference pixels along the motion trajectory. To obtain the multiple motion fields, a set of block-matching sizes is used and the motion fields are estimated by progressively reducing the size of matching block. Experimental results show that the proposed method can significantly improve both the objective and the subjective quality of the constructed high frame rate video.

For video frame rate up-conversion, new frames need to be interpolated along motion trajectories and inserted between the original adjacent frames. This paper proposes a Bayesian frame interpolation strategy, which considers both the spatial (i.e. intra-frame) correlation model and the temporal (i.e. inter-frame) correlation model after motion compensation. Different from the conventional schemes that perform interpolation using only one estimated motion vector field (MVF), the proposed strategy estimates multiple MVFs. To cope with different scales of frame contents, we generate the multiple MVFs through variable block size motion estimation. A criterion is adopted to estimate reliability of the motion compensated prediction frames from different motion fields. Experimental results demonstrate that the proposed strategy improves the interpolation performance remarkably.

Distributed video coding (DVC) was proposed to meet the low complexity encoding requirement, and it was verified to work more efficiently than H.264/AVC intra coding on video sequences with low motion. This makes DVC suitable for surveillance application. This paper presents a background aided surveillance-oriented distributed video coding system. A high quality background frame is encoded for each group of pictures (GOP), which can provides high quality SI for the background parts of the Wyner-Ziv (WZ) frames. Consequently, bit rate for the WZ frames can be reduced. Experimental results demonstrate that the proposed system can decrease the bit rate by up to 67.4% when compared with traditional DVC codec.

This paper proposes a multi-hypothesis based Wyner-Ziv (WZ) decoder for the multi-view distributed video coding (MDVC). Two hypotheses, the intra-view SI and the interview SI, are fed together into the WZ decoder in the proposed scheme. A multi-hypothesis based correlation model (MHBCM) is presented to fully exploit the redundancy between these two SI frames and the original frame. The MHBCM is also applied on the optimal minimum mean-square error reconstruction of the quantized samples. The simulation results show that the proposed algorithms are able to significantly improve the coding efficiency of the MDVC system.

In this paper, we propose a novel two-pass reconstruction algorithm for the Wyner-Ziv (WZ) frames in distributed video coding (DVC), in which the traditional reconstructed WZ frame is utilized to perform motion estimation to obtain a more accurate motion field. During the motion estimation, the block, as well as its neighboring pixels are concerned. An overlapped block motion compensation is subsequently performed with the help of the motion field, consequently, an enhanced prediction for the WZ frame can be obtained, based on which an improved reconstruction can be achieved. Simulation results show that both the objective and subjective quality of WZ frames can be improved significantly.

This paper proposes an image partition based approach to enhance side information quality in low delay distributed video coding (DVC). The proposed method employs a checkerboard pattern to group blocks of the Wyner-Ziv frame into two sets, where one set is DPCM encoded and the other set is DVC encoded. These two sets are encoded independently and decoded successively. At decoder, DPCM set will be first reconstructed. Then the temporal concealment tool, such as boundary matching algorithm, is performed to conceal blocks in the DVC set. An improved side information is subsequently obtained for DVC set, based on which a higher compression can be achieved. Simulation results indicate that a more promising performance can be achieved when compared with existing motion extrapolated approach.

Motivated by theoretical analysis of the curve fitting problem based on equivalent kernel, in this paper we propose a local adaptive learning and fusion model for side information interpolation in distributed video coding. In the proposed model, each pixel in the interpolated frame is approximated as the linear combination of samples within a local spatio-temporal window using kernel parameters as weight. The size of training window can be adaptive to the motion characteristic of video, from samples in which the kernel parameters can be locally learned. In order to further improve the quality of interpolated frames, we introduce a belief-projection based fusion strategy with adaptive weights for multiple interpolated results which are with the same time index. Experimental results demonstrate that the proposed learning and fusion model is effective in performance for side information interpolation in distributed video coding.