Template-generated videos (TGVs), created by applying animation templates to static images, have become increasingly prevalent, producing massive user-generated content with highly consistent motion patterns. However, existing video compression schemes are designed to eliminate motion redundancy within individual videos, while overlooking the shared motion patterns widespread across TGVs. To address this limitation, we propose a novel compression scheme that effectively leverages inter-video motion priors to enhance the compression efficiency of TGVs. Specifically, the proposed scheme operates as a two-stage pipeline. In the first stage, high-quality motion priors are identified from a representative TGV based on spatial texture and prediction error. In the second stage, these motion priors are intelligently integrated to expand the motion representation space beyond the local candidate lists in Merge and AMVP modes, thereby enabling the codec to remove inter-video redundancy. Experimental results on the versatile video coding test model (VTM-23.0) demonstrate consistent coding gains across various compression scenarios for TGVs, achieving average BD-rate savings of$1.07 \%, 1.38 {\%}$, and 1.18% under low-delay P (LDP), low-delay B (LDB), and random access (RA) configurations, respectively.

The rapid growth of video-based social applications has intensified the demand for efficient transmission and personalized cartoon-style customization. Existing solutions typically apply talking face video and text codecs, followed by cartoon-style control algorithms, which often result in unsatisfactory compression performance and high inference latency. In this paper, we propose CartoonCodec, an efficient generative framework that unifies face video coding and cartoon-style control into an end-to-end process. Our framework encodes facial video sequences into compact motion feature representations, which are transmitted together with compressed text prompts. To integrate control into the coding process while ensuring efficient compression, a text-guided adaptive layer selection mechanism relying on the compact representations is introduced to dynamically select and optimize the most influential layers in the generators. Furthermore, to facilitate stylization over the decoupled spaces, we propose a self-supervised domain stylization training strategy that constructs both multimodal and unimodal data pairs, enabling the use of diverse loss functions. Extensive experiments demonstrate that CartoonCodec outperforms baselines in compression efficiency for video reconstruction and cartoon-style control tasks while maintaining competitive inference efficiency. CartoonCodec provides key insights for advancing face video communication with cartoon-style customization. The project page can be found at https://github.com/xiaonae/CartoonCodec/tree/main .

Traditional video compression methods perform well at high bitrates but struggle to preserve fine-grained semantic information at low bitrates. Recently, with the blossoming of Multimodal Large Language Models (MLLMs), Cross-modal compression techniques offer prospective solutions for improving video compression under low-bitrate conditions. In this paper, we propose a unified Cross-Modality Video Compression (CMVC) framework that integrates multimodal representations and video generative models. The encoder disentangles video into spatial and temporal components, which are mapped to compact cross modal representations using MLLMs. During decoding, different encoding-decoding modes are employed to acquire various video reconstruction qualities, including Text-Text-to-Video (TT2V) for semantic preservation and Image-Text-to-Video (IT2V) for perceptual consistency. Additionally, we elaborate on an efficient frame interpolation model using Low-Rank Adaptation (LoRA) to improve the perceptual quality. Experimental results demon strate that TT2V achieves effective semantic reconstruction, while IT2V ensures competitive perceptual consistency. These findings suggest the potential of leveraging multimodal priors to improve video compression, offering promising future research directions.

Learned reference picture resampling control (LRPRC) adaptively adjusts the coding scale for each frame using an offline-trained neural network. It demonstrates promising promising rate-distortion (R-D) performance improvements over traditional methods, particularly in high-resolution, low-bit-rate video coding scenarios. However, existing LRPRC methods rely exclusively on locally optimal decision labels derived from greedy strategies for network training, leading to suboptimal control performance. To address this limitation, we introduce a novel data-centric solution that substantially improves training label quality, thereby enhancing overall LRPRC performance. Specifically, our key contribution is a parallelized beam search-based coding scale labeling algorithm, which captures decision dependencies across coding steps and produces higher-quality training labels with enhanced R-D performance. By fully exploiting the intra-trellis and inter-trellis parallelism of beam search and hierarchical coding, our proposed labeling algorithm achieves logarithmic-squared time complexity, making it highly suitable for large-scale cluster computing. We validate this simple yet effective data-centric LRPRC approach in the Versatile Video Encoder (VVenC) using 4K video sequences. Experimental results demonstrate that merely upgrading the beam search labels (without any neural architecture re-designs) consistently outperforms the state-of-the-art LRPRC method, achieving BD-rate reductions of 5.09%, 3.98%, and 3.59% under thefast,medium, andslowpresets, respectively.

The popularity of template-generated videos has recently experienced a significant increase on social media platforms. In general, videos from the same template share similar temporal characteristics, which are unfortunately ignored in the current compression schemes. In view of this, we aim to examine how such temporal priors from templates can be effectively utilized during the compression process for template-generated videos. First, a comprehensive statistical analysis is conducted, revealing that the coding decisions, including the merge, non-affine, and motion information, across template-generated videos are strongly correlated. Subsequently, leveraging such correlations as prior knowledge, a simple yet effective prior-driven compression scheme for template-generated videos is proposed. In particular, a mode decision pruning algorithm is devised to dynamically skip unnecessarily advanced motion vector prediction (AMVP) or affine AMVP decisions. Moreover, an improved AMVP motion estimation algorithm is applied to further accelerate reference frame selection and the motion estimation process. Experimental results on the versatile video coding (VVC) platform VTM-23.0 demonstrate that the proposed scheme achieves moderate time reductions of 14.31% and 14.99% under the Low-Delay P (LDP) and Low-Delay B (LDB) configurations, respectively, while maintaining negligible increases in Bjøntegaard Delta Rate (BD-Rate) of 0.15% and 0.18%, respectively.

Although lambda-domain-based rate control is widely used in video encoders, developing an efficient rate control scheme for Coding Tree Units (CTUs) under the rate-distortion (R-D) principle remains a significant challenge. In this paper, we propose a spatial-temporal correlation information-based rate control scheme for Versatile Video Coding (VVC), aiming to improve coding performance. We introduce a weight estimation network to establish a CTU-level bit allocation strategy that fully exploits spatial-temporal contextual information. Moreover, the CTU-level coding parameter λ is adaptively optimized based on a dependency factor derived from distortion dependency information in both the spatial and temporal domains. Experimental results demonstrate that, compared to the default VVC rate control, the proposed scheme achieves BD-Rate savings of 6.48%, 17.33% and 13.75% in terms of the Peak Signal-to-Noise Ratio (PSNR), the Multi-Scale Structural Similarity Index (MS-SSIM) and the Video Multimethod Assessment Fusion (VMAF), respectively, under the Low Delay_P (LDP) configuration in the VVC Test Model (VTM) 19.0. Furthermore, the proposed method outperforms other state-of-the-art rate control schemes.

Efficient talking face video coding and control are crucial in modern video communication, reshaping how individuals connect, collaborate, and interact. Coding seeks to reduce transmission costs, while control enables the realization of user-customizable facial expressions and head poses in the transmitted videos. However, the compression efficiency of the common par-adigm of applying control algorithms before video coding is not satisfactory. In this paper, we propose an efficient, Controllable Generative Talking Face Video Coding (CoFaCo) framework, wh-ich seamlessly integrates control into the coding process. Specific-ally, CoFaCo projects talking face videos into ultra-compact and semantic feature representations that can be customized by users before compression. To enable independent controls of pose and expression, we design a set of sophisticated losses to accurately de-couple the pose and expression direction codes. Once the decoupled direction codes and the semantic face representations are obtained, the pose and expression control modules can be effectively learned to generate decoupled, controlled pose and expression direction codes. The controlled direction codes are subsequently smoothed to enhance temporal consistency in the controlled video output by the generators. Experimental results demonstrate that CoFaCo achieves competitive compression efficiency in ultra-low bit rate video reconstruction and control tasks, providing valuab-le insights for advancing face video communication with diverse control capabilities.

In the past few years, 360° video has started to infiltrate various aspects of daily life. Although there have been significant developments in 360° video coding technology, understanding of the human visual gaze mechanism has been somewhat overlooked. In this paper, we propose a rate control scheme for 360° Versatile Video Coding (VVC) based on a human visual gaze mechanism, targeting at improving the coding performance and bitrate accuracy. More specifically, based on the Equi-rectangular Projection (ERP) format, latitude information is systematically analyzed and a stripe-level bit allocation scheme is established, to better mitigate the projection distortion. Subsequently, the Lagrange parameter λ is further optimized with distortion dependency and identification of the visual gaze guided key Coding Tree Units (CTUs). The proposed rate control scheme is implemented on the VVC Test Model for 360° video. Experimental results show that the proposed rate control scheme can achieve BD-rate savings in terms of Weighted to Spherically uniform-Peak Signal-to-Noise Ratio (WS-PSNR) and Sphere-Peak Signal-to-Noise Ratio (S-PSNR) under the various configurations, respectively. Meanwhile, a healthier buffer status and better visual quality can be observed, further demonstrating the advantages of the proposed scheme.

This paper proposes a highly efficient sample adaptive offset (SAO) mode decision algorithm. By leveraging both the directional correlations between the SAO and intra-prediction decisions, and the SAO decisions' spatial correlations, the SAO mode candidates are effectively pruned during the rate-distortion optimization process, accelerating the SAO encoding process with negligible BD-rate loss.

Lossy image compression networks aim to minimize the latent entropy of images while adhering to specific distortion constraints. However, optimizing the neural network can be challenging due to its nature of learning quantized latent representations. In this paper, our key finding is that minimizing the latent entropy is, to some extent, equivalent to maximizing the conditional source entropy, an insight that is deeply rooted in information-theoretic equalities. Building on this insight, we propose a novel structural regularization method for the neural image compression task by incorporating the negative conditional source entropy into the training objective, such that both the optimization efficacy and the model's generalization ability can be promoted. The proposed information-theoretic regularizer is interpretable, plug-and-play, and imposes no inference overheads. Extensive experiments demonstrate its superiority in regularizing the models and further squeezing bits from the latent representation across various compression structures and unseen domains.

Understanding the human visual gaze mechanism is crucial for enhancing 360° video coding technology. This paper presents a Coding Tree Unit (CTU)-level rate control scheme with λ optimization strategy for 360° Versatile Video Coding (VVC), with the aim of enhancing rate-distortion performance and bitrate accuracy. Specifically, the Lagrange parameter λ is optimized with consideration of distortion dependency and the identification of key CTUs guided by visual gaze, which are derived from a 360° video path generation network, thoroughly integrating the characteristics of the human visual gaze. Experimental results show that the proposed scheme achieves BD-rate savings in terms of Weighted to Spherically uniform-Peak Signal-to-Noise Ratio (WS-PSNR) and Sphere-Peak Signal-to-Noise Ratio (S-PSNR) across various coding configurations.

We propose a spatio-temporal adaptive deep video compression scheme, which is capable of intelligently adjusting the spatial resolution and temporal frame rate for content adaptive compression, with the aim of pursuing enhanced rate-distortion performance. In particular, a neural network-based spatio-temporal adaptation network is integrated into the deep video coding paradigm, enabling the adaptive determination of the optimal rescaling ratios for compression, leading to the further reduction of spatial and temporal redundancies. Moreover, learning-based modules for rescaling parameter determination are incorporated into the spatio-temporal adaptation network. The proposed scheme can be easily plugged into, and seamlessly collaborate with the existing deep video coding frameworks. Experimental results demonstrate that, compared to the original neural video codecs, the proposed method achieves significant bitrate savings in terms of both PSNR and MS-SSIM.

Digital images in real world applications typically undergo a wide variety of quality degradations before compression or re-compression. Existing learning based codecs are typically data-driven, relying on the predefined compression pipeline with pristine or high quality images as the input. However, the images in the wild may exhibit the substantially different characteristics compared to the high quality images, casting major challenges to the learning based image coding. In this paper, we propose a robust noisy image compression framework with the blind assumption on the specific noise type and level. The specifically designed encoder decomposes the representation of visual content into two types of features, including the Features that represent the Intrinsic Content (FIC) and the Features that account for Additive Degradation (FAD). As such, beyond the philosophy of faithfully reconstructing the given image with high fidelity, only FIC needs to be compactly represented and conveyed. The principled disentanglement strategy facilitates the removal of the redundancy from multiple perspectives (e.g., spatial, channel and content), ensuring the handling of a wide variety of noisy images in the wild. Extensive experimental results show that our model can achieve superior performance in terms of the ultimate quality and exhibit the strong generalizability across images degraded by a variety of means. The proposed scheme also points out a new research avenue on learning based compression for images in the wild, which is technically challenging but desirable in practice.

Recent advancements in learned rate-distortion optimization (RDO) showcase that by making the intra coding decisions based on a learned measure, the encoding can be significantly accelerated without incurring much coding loss. Despite great progress in complexity reduction, the dependency issue has been largely neglected in the current learned RDO research. In this study, aiming to tap the full potential of dependent learned RDO, we first derive a probabilistic RDO framework for theoretical analysis, under which the classic and the learned RDO problems are equivalent to the maximum a posteriori (MAP) inference and the distribution imitation, respectively. Subsequently, we probabilistically revisit dependency considerations in the intra RDO research. Our key finding is that the existing learned RDO scheme can only produce a measure that indicates the local “goodness” of coding decisions. We therefore further discuss the opportunities for learning a dependent measure that is more optimal in the long run. Finally, as learning an accurate measure for the full decision space could be extremely challenging, taking the High Efficiency Video Coding (HEVC) intra coding as a case study, we experimentally identify that the prediction decision accounts for the majority of the dependent optimization gain and is of the utmost value to be learned, paving the way for future research on dependent learned RDO.

Compressing high-resolution videos under low bitrate constraints is a challenging task. Resampling-based compression, which reduces the resolution before encoding and restores it after decoding, has great potential to improve the rate-distortion performance in such scenarios. In this paper, we propose a learning-based frame-level coding scale control scheme that enhances the coding performance by adjusting the coding scale for each frame. The scheme cooperates with the Reference Picture Resampling of the latest video coding standard Versatile Video Coding (VVC), which allows coding scale variations on each frame. More specifically, a dataset with 5200 videos is created by a greedy rate-distortion optimization algorithm employed to select the optimal coding scale for each frame. A neural network-based decision model is further incorporated into VVC, learning to predict the coding scale for each frame in one pass. The scheme is implemented into the Fraunhofer Versatile Video Encoder (VVenC), a fast and efficient VVC encoder, and evaluated on 4 K contents. Experimental results show that the proposed scheme outperforms GOP-based coding scale adaptation methods, achieving average bitrate savings of 3.06% and 4.14% in terms of PSNR and MS-SSIM.

Action Quality Assessment (AQA) aims to evaluate and score human actions in videos accurately. Existing approaches involve extracting features from the input video and implementing regression based on those features. However, representations derived from a single branch often lack the necessary diversity and flexibility to capture the complexity of human actions effectively. This work addresses these limitations by introducing a multi-branch architecture designed to capture a broad spectrum of video dynamics at varying levels of granularity. Specifically, we enhance video representation in the flow-guided branch by integrating optical flow with video features. This combination of multimodal features offers a more comprehensive context of global motion. Meanwhile, the moment-focused branch is tailored to extract frame-specific features, constructing two distinct quality-based representations with different focuses on moments, which achieves adaptive clues aggregation. Furthermore, the detail-aware branch leverages multiscale deep embeddings from a hierarchy convolutional neural network to capture fine-grained spatial information, which is useful when objects have complex spatial changes. Finally, a post-fusion strategy is employed to merge outputs from all branches, contributing to the comprehensive action quality assessment. Experimental evaluations on three benchmark datasets, FineDiving, MTL-AQA, and AQA-7, demonstrate the superiority of our model in providing reliable assessments of action quality.

Image set compression (ISC) refers to compressing the sets of semantically similar images. Traditional ISC methods typically aim to eliminate redundancy among images at either signal or frequency domain, but often struggle to handle complex geometric deformations across different images effectively. Here, we propose a new Hybrid Neural Representation for ISC (HNR-ISC), including an implicit neural representation for Semantically Common content Compression (SCC) and an explicit neural representation for Semantically Unique content Compression (SUC). Specifically, SCC enables the conversion of semantically common contents into a small-and-sweet neural representation, along with embeddings that can be conveyed as a bitstream. SUC is composed of invertible modules for removing intra-image redundancies. The feature level combination from SCC and SUC naturally forms the final image set. Experimental results demonstrate the robustness and generalization capability of HNR-ISC in terms of signal and perceptual quality for reconstruction and accuracy for the downstream analysis task.

Recent advances in generative compression methods have demonstrated remarkable progress in enhancing the perceptual quality of compressed data, especially in scenarios with low bitrates. However, their efficacy and applicability to achieve extreme compression ratios (< 0.05 bpp) remain constrained. In this work, we propose a simple yet effective coding framework by introducing vector quantization (VQ)–based generative models into the image compression domain. The main insight is that the codebook learned by the VQGAN model yields a strong expressive capacity, facilitating efficient compression of continuous information in the latent space while maintaining reconstruction quality. Specifically, an image can be represented as VQ-indices by finding the nearest codeword, which can be encoded using lossless compression methods into bitstreams. We propose clustering a pre-trained large-scale codebook into smaller codebooks through the K-means algorithm, yielding variable bitrates and different levels of reconstruction quality within the coding framework. Furthermore, we introduce a transformer to predict lost indices and restore images in unstable environments. Extensive qualitative and quantitative experiments on various benchmark datasets demonstrate that the proposed framework outperforms state-of-the-art codecs in terms of perceptual quality-oriented metrics and human perception at extremely low bitrates (≤ 0.04 bpp). Remarkably, even with the loss of up to 20% of indices, the images can be effectively restored with minimal perceptual loss.

In this paper, we present a Conv-Attention, a decoder-friendly attention mechanism, in an effort to advancing the practical application of the artificial intelligence-based image coding. More specifically, the proposed method is tailored for JPEG AI, which is the latest advanced neural-network based image coding standard. By identifying the obstacles by profiling the decoding complexity of JPEG AI, the attention module accounts for a significant proportion, which mainly attributes to the intricate network structure and involvement of less efficient operations. Conv-Attention model is composed with plain convolution and activation computations, equipping with sub-scaling and up-scaling design, such that the non-adjacent features can be well captured, leading to the reduction of decoding complexity and maintenance of the synthesis and attentive capability. Simulation results verify the effectiveness of the proposed method with JPEG AI reference software, wherein the decoding complexity is reduced by 80% with negligible coding performance loss. The proposed method was adopted in the 100th JPEG meeting.

We extend the beam-search based joint rate-distortion optimization (BSJRDO) [1] to High Efficiency Video Coding (HEVC) and investigate the impact of different decisions on it. In BSJRDO, unlike the greedy search, which only maintains one locally optimal path at each decision stage, multiple paths are kept as candidates for future referencing.

With the rapid development of video conferencing and online education applications, screen content image (SCI) compression has become increasingly crucial. Recently, deep learning techniques have made significant progress in compressing natural images, surpassing the performance of traditional standards like versatile video coding. However, directly applying these methods to SCIs is challenging due to the unique characteristics of SCIs. In this paper, we propose a synergistic approach to preserve structural fidelity and text integrity for SCIs. Firstly, external prior guidance is proposed to enhance structural fidelity and text integrity by providing global spatial attention. Then, a structural enhancement module is proposed to improve the preservation of structural information by enhanced spatial feature transform. Finally, the loss function is optimized for better compression efficiency in text regions by weighted mean square error. Experimental results show that the proposed method achieves 13.3% BD-Rate saving compared to the baseline window attention convolutional neural networks (WACNN) on the JPEGAI, SIQAD, SCID, and MLSCID datasets on average. Our code is available at https://github.com/vpaHduGroup/SFTIP_SCC.

Assessing the quality of a player's performance, such as in diving events, requires precise measurement of subtle action details and overall fluidity. Existing methods primarily utilize appearance information from RGB frames, often neglecting crucial motion information that could contribute to a more comprehensive assessment. In response to this limitation, this paper introduces a novel Multi-Modality Network for Action Quality Assessment (AQA). The proposed method first employs a self-attention based module to foster interaction between optical flow and appearance clues, facilitating the extraction of discriminative features from each modality. Subsequently, a pairwise cross-attention mechanism is designed to comprehensively capture subtle differences via both intra-modality and inter-modality relationships between the query and exemplar video. Finally, to enhance the robustness and achieve accurate score prediction, an adaptive clip aggregation module is introduced to weigh the reliability of each patch based on multi-modal difference features. Experimental results on two benchmarks, FineDiving and MTL-AQA, validate the effectiveness of the proposed model.

The intra block copy (IBC) mode introduced can significantly improve the coding efficiency for screen content. However, with the mixed content, the coding performance improvement of IBC is limited and while also introducing the increases of the coding complexity. This paper proposes an early determination scheme for IBC prediction to reduce the encoding complexity over mixed content sequences containing captured content regions. Our method is based on a refined CTU-level screen content detection method, which can better distinguish screen content in mixed scenes. Experimental results show that the proposed method reduces encoding complexity with moderate coding performance loss on mixed content sequences.

In this paper, we propose a Learning-based gEnome Codec (LEC), which is designed for high efficiency and enhanced flexibility. The LEC integrates several advanced technologies, including Group of Bases (GoB) compression, multi-stride coding and bidirectional prediction, all of which are aimed at optimizing the balance between coding complexity and performance in lossless compression. The model applied in our proposed codec is data-driven, based on deep neural networks to infer probabilities for each symbol, enabling fully parallel encoding and decoding with configured complexity for diverse applications. Based upon a set of configurations on compression ratios and inference speed, experimental results show that the proposed method is very efficient in terms of compression performance and provides improved flexibility in real-world applications.

Traditional end-to-end video coding is typically featured with sparsely distributed operational rate-distortion (R-D) points. This creates daunting challenges to rate control which is typically regarded as the indispensable coding optimization module. To tackle this problem, this paper proposes high efficiency rate control for end-to-end scale-adaptive video coding which enables the conversion from sparsely to densely distributed R-D points. The proposed scheme does not increase the number of models in the sparse-to-dense conversion and provides more flexibility in end-to-end video coding thereby leading to better coding performance. More specifically, R-D analyses for scale-adaptive coding are first conducted, shedding light on the design of the rate control algorithm. Subsequently, generalized R-D models are presented, based on which high efficiency rate control is achieved. Extensive experimental results provide evidence of the efficiency of the proposed method in terms of R-D performance, control accuracy and computational complexity..

Recent years have witnessed a variety of applications of gaming video coding, while how to improve the coding efficiency has been relatively under-explored. The state-of-the-art video coding standard, Versatile Video Coding (VVC), adopts the Reference Picture Resampling (RPR) which allows the variation of the frame resolutions in encoding/decoding. The great flexibility supported by RPR motivates us to develop a bidirectional quality impulse based guidance scheme, in an effort to fully exploit the potential of RPR in gaming video coding. The design philosophy involves reducing the data volume on the encoder side through selective downsampling, and enhancing reconstruction by harnessing quality conveyance from neighboring frames. More specifically, a new RPR structure is developed based on the underlying philosophy that the periodic quality impulse could promisingly boost the quality of the whole sequence. On top of the developed structure, we propose a bidirectional guidance model that faithfully enhances video quality by resorting to frames with quality impulse. Experimental results exhibit the proposed scheme can achieve significant bit-rate savings for gaming videos.

The Versatile Video Coding (VVC) standard adopts a series of new coding tools in transform and quantization, including multiple transform selection, low-frequency non-separable transform, and trellis quantization. These new technologies, which bring significant coding gain, create daunting challenges to optimizing the VVC codec. In this work, we propose a new all-zero block (AZB) detection scheme tailored for VVC, with the collaboration of genuine all-zero block (GAZB) and pseudo all-zero block (PAZB) detection. First, to accommodate the multiple transform sizes in VVC, we develop a GAZB detection method that is apt for square and non-square residual blocks. Meanwhile, a theoretical upper bound is derived to locate the last significant coefficient and detect the potential frequency domain GAZB. Subsequently, a method tailored for trellis-coded quantization in VVC is devised for detecting PAZB. Finally, the GAZB and PAZB detection methods are collaboratively employed for AZB detection in VVC. The proposed method is implemented on the VVC codec Versatile Video Encoder (VVenC), and extensive experimental results show that the proposed method achieves promising time savings for test sequences of different resolutions with negligible rate-distortion performance loss.

The emergence of digital avatars has prompted an exponential increase in the demand for human point clouds with realistic and intricate details. The compression of such data becomes challenging due to massive amounts of data comprising millions of points. Herein, we leverage the human geometric prior in the geometry redundancy removal of point clouds to greatly promote compression performance. More specifically, the prior provides topological constraints as geometry initialization, allowing adaptive adjustments with a compact parameter set that can be represented with only a few bits. Therefore, we propose representing high-resolution human point clouds as a combination of a geometric prior and structural deviations. The prior is first derived with an aligned point cloud. Subsequently, the difference in features is compressed into a compact latent code. The proposed framework can operate in a plug-and-play fashion with existing learning-based point cloud compression methods. Extensive experimental results show that our approach significantly improves the compression performance without deteriorating the quality, demonstrating its promise in serving a variety of applications.

Motion Compensated Temporal Filter (MCTF) has been repeatedly proven to be an effective pre-processing tool that improves the coding performance. The philosophy is that by smoothing with temporal filter, the noise of the to-be-coded image can be reduced, thereby shrinking the prediction residuals and improving the rate-distortion (RD) performance. While abundant efforts have been devoted to the design of the MCTF filter weights, how motion vector variance and texture complexity influence MCTF has been relatively under-explored. In this work, we propose an enhanced MCTF method (EMCTF) based on multi-hypothesis reference, motion vector variance, and texture complexity. We take initial steps towards the incorporation of motion vector variance and texture complexity in the filtering weights design. Motion compensation blocks based on multi-hypothesis reference, can be efficiently aggregated in an effort to obtain the final inference. The proposed method is implemented on the top of Versatile Video Encoder (VVenC). Experimental results show that for faster preset, fast preset, medium preset and slow preset, the proposed EMCTF achieves 0.85%, 0.91%, 0.85% and 0.82% Bjøntegaard delta rate (BD-rate) savings, respectively. Moreover, the EMCTF introduces little additional encoding complexity increase, facilitating its future applications in real-world scenarios.

The autoregressive model has been widely used in learning-based image compression due to its superior context modeling capability. However, its sequential processing nature also undermines the ability of decoding in parallel and hinders the deployment in real applications. In this paper, we propose a decoupled framework to resolve this issue. With the decoupled architecture, the entropy decoding process is independent of the latent sample reconstruction process. The entropy decoding process thus can be finished before the latent sample prediction process begins, which leads to significant decoding time savings by enabling the two processes to be conducted in parallel. To further reduce the decoding time, we introduce wavefront processing, where multiple rows can be processed simultaneously when reconstructing the latent samples. On top of that, we design a series of coding tools to improve the rate-distortion efficiency and reduce the decoding complexity. Device interoperability is also supported by the proposed solution, where the same bitstream can be successfully decoded on different CPU/GPU devices. Comprehensive experiments are conducted to validate the effectiveness of the proposed method. Using objective evaluation metrics required by JPEG AI Call for Proposals (CfP), the proposed method achieves an average of -29.6% BD-rate changes with 2.44 times faster decoding speed compared to VVC image coding. When compared to the commonly used benchmark learning-based methods, the proposed method achieves -30.5% BD-rate changes and 101 times faster decoding speed over cheng2020attn. The proposed solution has been proposed to JPEG AI and IEEE 1857.11 as a response to CfP and the core techniques have been adopted to build the verification model. The software and the instructions can be accessed at https://github.com/bytedance/BEE.

Residual coding has gained prevalence in lossless compression, where a lossy layer is initially employed and the reconstruction errors (i.e., residues) are then losslessly compressed. The underlying principle of the residual coding revolves around the exploration of priors based on context modeling. Herein, we propose a residual coding framework for 3D medical images, involving the off-the-shelf video codec as the lossy layer and a Bilateral Context Modeling based Network (BCM-Net) as the residual layer. The BCM-Net is proposed to achieve efficient lossless compression of residues through exploring intra-slice and inter-slice bilateral contexts. In particular, a symmetry-based intra-slice context extraction (SICE) module is proposed to mine bilateral intra-slice correlations rooted in the inherent anatomical symmetry of 3D medical images. Moreover, a bi-directional inter-slice context extraction (BICE) module is designed to explore bilateral inter-slice correlations from bi-directional references, thereby yielding representative inter-slice context. Experiments on popular 3D medical image datasets demonstrate that the proposed method can outperform existing state-of-the-art methods owing to efficient redundancy reduction. Our code will be available on GitHub for future research.

The accelerated proliferation of visual content and the rapid development of machine vision technologies bring significant challenges in delivering visual data on a gigantic scale, which shall be effectively represented to satisfy both human and machine requirements. In this work, we investigate how hierarchical representations derived from the advanced generative prior facilitate constructing an efficient scalable coding paradigm for human-machine collaborative vision. Our key insight is that by exploiting the StyleGAN prior, we can learn three-layered representations encoding hierarchical semantics, which are elaborately designed into the basic, middle, and enhanced layers, supporting machine intelligence and human visual perception in a progressive fashion. With the aim of achieving efficient compression, we propose the layer-wise scalable entropy transformer to reduce the redundancy between layers. Based on the multi-task scalable rate-distortion objective, the proposed scheme is jointly optimized to achieve optimal machine analysis performance, human perception experience, and compression ratio. We validate the proposed paradigm's feasibility in face image compression. Extensive qualitative and quantitative experimental results demonstrate the superiority of the proposed paradigm over the latest compression standard Versatile Video Coding (VVC) in terms of both machine analysis as well as human perception at extremely low bitrates (< 0.01 bpp), offering new insights for human-machine collaborative compression.

Recent years have witnessed an exponential increase in the demand for face video compression, and the success of artificial intelligence has expanded the boundaries beyond traditional hybrid video coding. Generative coding approaches have been identified as promising alternatives with reasonable perceptual rate-distortion trade-offs, leveraging the statistical priors of face videos. However, the great diversity of distortion types in spatial and temporal domains, ranging from the traditional hybrid coding frameworks to generative models, present grand challenges in compressed face video quality assessment (VQA) that plays a crucial role in the whole delivery chain for quality monitoring and optimization. In this paper, we introduce the large-scale Compressed Face Video Quality Assessment (CFVQA) database, which is the first attempt to systematically understand the perceptual quality and diversified compression distortions in face videos. The database contains 3,240 compressed face video clips in multiple compression levels, which are derived from 135 source videos with diversified content using six representative video codecs, including two traditional methods based on hybrid coding frameworks, two end-to-end methods, and two generative methods. The unique characteristics of CFVQA, including large-scale, fine-grained, great content diversity, and cross-compression distortion types, make the benchmarking for existing image quality assessment (IQA) and VQA feasible and practical. The results reveal the weakness of existing IQA and VQA models, which challenge real-world face video applications. In addition, a FAce VideO IntegeRity (FAVOR) index for face video compression was developed to measure the perceptual quality, considering the distinct content characteristics and temporal priors of the face videos. Experimental results exhibit its superior performance on the proposed CFVQA dataset. The benchmark is now made publicly available at:https://github.com/Yixuan423/Compressed-Face-Videos-Quality-Assessment.

Video transcoding, which is a long-standing problem in video coding, can be used for a variety of applications including bit rate conversion, format conversion and video resolution conversion. In this paper, we present an effective transcoding solution from the popular previous generation video coding standard, High Efficiency Video Coding standard (HEVC), to Versatile Video Coding (VVC) which is the latest standard developed by JVET. VVC enjoys much better coding efficiency compared to its ancestors at the expense of higher encoding complexity. To facilitate the fast transcoding, we propose a low- complexity HEVC to VVC transcoding scheme. More specifically, the partition and motion information, which implies the intrinsic spatial and temporal characteristics of the video sequence, is inherited from HEVC for VVC encoding. The proposed scheme is tailored for HEVC to VVC conversion, and experimental results show the proposed method can achieve 31% encoding time saving with only 0.85%, 0.70% and 0.74% Bjøntegaard delta rate (BD- rate) loss on average for Y, Cb and Cr components, respectively, under random access configuration.

In this paper, a rate control scheme for screen content video coding is proposed for the Versatile Video Coding (VVC) standard. In view of the critical challenges arising from the spatial and temporal unnaturalness of screen content sequences, the proposed method relies on the specifically designed pre-analysis such that the content information regarding the scene complexity can be obtained. As such, the estimated residual complexity is then incorporated into the proposed complexity-aware rate models and distortion models, leading to the optimal bit allocations for each frame and coding tree unit (CTU). In particular, the optimization problem can be analytically solved with the proposed models, and the coding parameters such as Lagrangian multiplier$\lambda $and quantization parameter of each frame and CTU could be delicately derived according to the allocated bits through the proposed analytical models. Extensive experiments have been conducted to evaluate the effectiveness of the proposed method. Compared to the default hierarchical$\lambda $-domain rate control and other screen content rate control algorithms, the proposed method could achieve obvious RD performance gain, and the bit-rate accuracy could be improved.

In this work, we propose a neural network based rate control algorithm for Versatile Video Coding (VVC). The proposed method relies on the modeling of the Rate-Quantization (R-Q) and Distortion-Quantization (D-Q) relationships in a data driven manner based upon the characteristics of prediction residuals. In particular, a pre-analysis framework is adopted, in an effort to obtain the prediction residuals which govern the Rate-Distortion (R-D) behaviors. By inferring from the prediction residuals with deep neural networks, the Coding Tree Unit (CTU) level R-Q and D-Q model parameters are derived, which could efficiently guide the optimal bit allocation. Subsequently, the coding parameters, including Quantization Parameter (QP) and$\lambda $, at both frame and CTU levels, are obtained according to allocated bit-rates. We implement the proposed rate control algorithm on VVC Test Model (VTM-13.0). Experimental results exhibit that the proposed rate control algorithm achieves 0.77% BD-Rate savings under Low Delay B (LDB) configurations when compared to the default rate control algorithm used in VTM-13.0. For Random Access (RA) configurations, 1.77% BD-Rate savings can be observed. Furthermore, with better bit-rate estimation, more stable buffer status can be observed, further demonstrating the advantages of the proposed rate control method.

This article proposes the scalable cross-modality compression (SCMC) paradigm, in which the image compression problem is further cast into a representation task by hierarchically sketching the image with different modalities. Herein, we adopt the conceptual organization philosophy to model the overwhelmingly complicated visual patterns, based upon the semantic, structure, and signal level representation accounting for different tasks. The SCMC paradigm that incorporates the representation at different granularities supports diverse application scenarios, such as high-level semantic communication and low-level image reconstruction. The decoder, which enables the recovery of the visual information, benefits from the scalable coding based upon the semantic, structure, and signal layers. Qualitative and quantitative results demonstrate that the SCMC can convey accurate semantic and perceptual information of images, especially at low bitrates, and promising rate-distortion performance has been achieved compared to state-of-the-art methods. The code will be available onlinehttps://github.com/ppingzhang/SCMC.

In this paper, we present the first attempt at determining where the achievable rate-distortion (R-D) performance bound in versatile video coding (VVC) intra coding is when considering the mutual dependency in the rate-distortion optimization (RDO) process. In particular, the abundant search space of encoding parameters in VVC intra coding is practically explored with a beam search-based joint rate-distortion optimization (BSJRDO) scheme. As such, the partitioning, prediction and transform decisions are jointly optimized across different coding units (CUs) with a customized search subset instead of the full space. To make the beam search process implementation-friendly for VVC, the dependencies among the CUs are truncated at different depths. To facilitate finer computational scalability, the beam size is flexibly adjusted based on the characteristics of the CUs, such that the operational points that satisfy different complexity demands for diverse applications can be practically obtained. The proposed BSJRDO approach, which fully conforms to the VVC decoding syntax, can serve as both the way toward the optimal RDO bound and a practical performance-boosting solution. BSJRDO is further implemented on a VVC coding platform (VVC Test model (VTM) 12.0), and extensive experiments show that BSJRDO can achieve 1.30% and 3.22% bit rate savings compared to the VTM anchor under the common test condition and low-bit-rate coding scenarios, respectively. Moreover, the performance gain can also be flexibly customized with different computational overheads.

Recently, we have observed an exponential increase of user-generated content (UGC) videos. The distinguished characteristic of UGC videos originates from the video production and delivery chain, as they are usually acquired and processed by non-professional users before uploading to the hosting platforms for sharing. As such, these videos usually undergo multiple distortion stages that may affect visual quality before ultimately being viewed. Inspired by the increasing consensus that the optimization of the video coding and processing shall be fully driven by the perceptual quality, in this paper, we propose to study the quality of the UGC videos from both objective and subjective perspectives. We first construct a UGC video quality assessment (VQA) database, aiming to provide useful guidance for the UGC video coding and processing in the hosting platform. The database contains source UGC videos uploaded to the platform and their transcoded versions that are ultimately enjoyed by end-users, along with their subjective scores. Furthermore, we develop an objective quality assessment algorithm that automatically evaluates the quality of the transcoded videos based on the corrupted reference, which is in accordance with the application scenarios of UGC video sharing in the hosting platforms. The information from the corrupted reference is well leveraged and the quality is predicted based on the inferred quality maps with deep neural networks (DNN). Experimental results show that the proposed method yields superior performance. Both subjective and objective evaluations of the UGC videos also shed lights on the design of perceptual UGC video coding.

Screen content has become one of the prominent mediums in the increasingly connected world. With the prevalence of remote collaboration and communication such as virtual conferences and online education, recent years have witnessed a dramatic increase in the data volume of the screen content. Screen content compression serves as the fundamental technology in fostering the storage, transmission, and exhibition of screen content. In this article, we target the super-resolution of the compressed screen content images, intending to tackle the real-world challenge problems. A dataset is proposed for the super-resolution of the screen contents contaminated with different compression distortion levels. Subsequently, we introduce the principle of the multi-hypothesis into the super resolution and propose a new paradigm for the restoration of the compressed screen content images. The luminance and sharpness similarity metric is adopted in the network learning to better adapt to the screen content characteristic and ensure perceptual fidelity. Experimental results verify the superiority and effectiveness of the proposed method.

Flexible coding unit partitioning such as quad-tree nested binary-tree and ternary-tree adopted by the emerging enhanced compression model (ECM) brings promising coding performance improvement. Meanwhile, the computational complexity increases dramatically, which may block the exploration and validation of new coding tools. This paper investigates a partition mode early pruning scheme via a fully connected network to reduce the encoding complexity for the ECM. In particular, we carefully select features and devise the fully connected network, which could seamlessly cooperate with the encoder, revealing promising learning and inference capability. Experimental results demonstrate that the proposed method achieves 15%~50% encoding time savings with moderate bit-rate increasing on the ECM, and the extra complexity regarding the fully connected network and feature extraction is negligible.

Existing end-to-end image compression mainly concentrates on the coding of natural scene images. However, few works have been dedicated to the end-to-end compression of screen contents. In this paper, we propose an end-to-end compression scheme for screen content images inspired by the ideology of transform skip, with the goal of improving the compression performance for screen content images. In particular, the proposed model takes full consideration of the characteristics of screen contents and involves transform skip branches to the analyses and synthesis process. The transform skip branch equips with coarse feature extraction and reconstruction. As such, the visual signals could be more briefly interpreted at the encoder-side and recovered at the decoder-side. Experimental results show that the proposed method outperforms the existing hyperprior-based model for screen content compression, achieving 10.16% BD-Rate savings in high bit-rate coding scenario and 5.38% BD-Rate savings in low bit-rate coding scenario.

In this paper, we propose a deep video compression method for P-frame in sub-sampled color spaces regarding the YUV420, which has been widely adopted in many state-of-art hybrid video compression standards, in an effort to achieve high compression performance. We adopt motion estimation and motion compression to facilitate the inter prediction of the videos with YUV420 color format, shrinking the total data volume of motion information. Moreover, the motion compensation module on YUV420 is cooperated to enhance the quality of the compensated frame with the consideration of the resolution alignment in the sub-sampled color spaces. To explore the cross-component correlation, the residual encoder-decoder is accompanied with two head-branches and color information fusion. Additionally, a weighted loss emphasizing more on the Y component is utilized to enhance the compression efficiency. Experimental results show that the proposed method can realize 19.82% bit rate reductions on average compared to the deep video compression (DVC) method in terms of the combined PSNR and predominant gains on the Y component.

In this paper, we propose an end-to-end image compression framework, which cooperates with the swin-transformer modules to capture the localized and non-localized similarities in image compression. In particular, the swin-transformer modules are deployed in the analysis and synthesis stages, interleaving with convolution layers. The transformer layers are expected to perceive more flexible receptive fields, such that the spatially localized and non-localized redundancies could be more effectively eliminated. The proposed method reveals the excellent capability of signal conjunction and prediction, leading to the improvement of the rate and distortion performance. Experimental results show that the proposed method is superior to the existing methods on both natural scene and screen content images, where 22.46% BD-Rate savings are achieved when compared with the BPG. Over 30% BD-Rate gains could be observed with screen content images when compared with the classical hyper-prior end-to-end coding method.

Interpolation filters for motion compensation play an important role in video coding. Until recently, interpolation filters for luma motion compensation (MC) draw most of the attention while interpolation filters for chroma motion compen-sation are less noticeable. This paper comprehensively analyzes different types of interpolation filters and presents 6-tap DCT based chroma interpolation filters for chroma MC. The proposed interpolation filters are designed considering multiple factors, comprising the characteristics of the chroma component, the frequency response of interpolation filter and the expenses of computational complexity. The experimental results show the proposed method can achieve 0.05%, 1.23% and 1.25% BD-rate reductions on average for Y, Cb and Cr components, respectively, with only 2% decoding time increasing under Random Access configuration, compared with ECM-4.0, which is a reference software developed by JVET, targeting at future video coding technologies beyond VVC. Owing to the good trade-off between coding performance and complexity, the proposed method has been adopted in ECM.

In this work, we propose a novel rate control algorithm for Versatile Video Coding (VVC) standard based on its distinct rate-distortion characteristics. By modelling the transform coefficients with the composite Cauchy distribution, higher accuracy compared with traditional distributions has been achieved. Based on the transform coefficient modelling, the theoretically derived R-Q and D-Q models which have been shown to deliver higher accuracy in characterizing RD characteristics for sequences with different content are incorporated into the rate control process. Furthermore, to establish an adaptive bit allocation scheme, the dependency between different levels of frames is modelled by a dependency factor to describe relationship between the reference and to-be-coded frames. Given the derived R-Q and D-Q relationships, as well as the dependency factor, an adaptive bit allocation scheme is developed for optimal bits allocation. We implement the proposed algorithm on VVC Test Model (VTM) 3.0. Experiments show that due to proper bit allocation, for low delay configuration the proposed algorithm can achieve 1.03% BD-Rate saving compared with the default rate control algorithm and 2.96% BD-Rate saving compared with fixed-QP scheme. Moreover, 1.29% BD-Rate saving and higher control accuracy have also been observed under the random access configuration.

Flexible block partition and newly adopted intra coding tools bring significant performance improvement for next generation video coding, and meanwhile introduce non-negligible encoding complexity increment. This paper presents a scalable intra coding optimization (SICO) scheme for the third generation of audio video coding standard (AVS3). The complexity distribution and inheriting relationship among different partitioning, as well as intra prediction modes, are systematically analyzed. Subsequently, low-complexity algorithms are proposed, which could early exclude unlikely coding modes safely. In particular, a data-driven binary classifier is elegantly trained for the determination of coding unit partitioning. Moreover, the preliminary coding information can be exhaustively utilized for the mode decision in a low-cost manner. The proposed method provides scalable intra coding optimizing solutions, which is eligible to cater to various application scenarios. Experimental results show that the proposed method could achieve a wide range of encoding complexity reduction from 18% to 76% with moderate compression performance loss. One implementation of the proposed method has been adopted to the AVS3 reference software.

In this paper, we concentrate on the super-resolution (SR) of compressed screen content video, in an effort to address the real-world challenges by considering the underlying characteristics of screen content. Firstly, we propose a new dataset for the SR of screen content video with different distortion levels. Meanwhile, we design an efficient SR structure that could capture the characteristics of compressed screen content video and manipulate the inner-connections in consecutive compressed low-resolution frames, facilitating the high-quality recovery of the high-resolution counter-part. Moreover, we design a new loss function for network training to better remedy the compression distortion and perceptual distortion. Experimental results demonstrate the effectiveness and superiority of the proposed method.

In this paper, we propose the complexity configurable learning-based genome data compression method, in an effort to achieve a good balance between coding complexity and performance in lossless DNA compression. In particular, we first introduce the concept of Group of Bases (GoB), which serves as the foundation and enables the parallel implementation of the learning-based genome data compression. Subsequently, the Markov model is introduced for modeling the initial content, and the learning-based inference is achieved for the remaining base data. The compression is finally achieved with efficient arithmetic coding, and based upon a set of configurations on compression ratios and inference speed, the proposed method is shown to be more efficient and provide more flexibility in real-world applications.

Real-world video processing algorithms are often faced with the great challenges of processing the compressed videos instead of pristine videos. Despite the tremendous successes achieved in deep-learning based video super-resolution (SR), much less work has been dedicated to the SR of compressed videos. Herein, we propose a novel approach for compressed domain deep video SR by jointly leveraging the coding priors and deep priors. By exploiting the diverse and ready-made spatial and temporal coding priors (e.g., partition maps and motion vectors) extracted directly from the video bitstream in an effortless way, the video SR in the compressed domain allows us to accurately reconstruct the high resolution video with high flexibility and substantially economized computational complexity. More specifically, to incorporate the spatial coding prior, the Guided Spatial Feature Transform (GSFT) layer is proposed to modulate features of the prior with the guidance of the video information, making the prior features more fine-grained and content-adaptive. To incorporate the temporal coding prior, a guided soft alignment scheme is designed to generate local attention off-sets to compensate for decoded motion vectors. Our soft alignment scheme combines the merits of explicit and implicit motion modeling methods, rendering the alignment of features more effective for SR in terms of the computational complexity and robustness to inaccurate motion fields. Furthermore, to fully make use of the deep priors, the multi-scale fused features are generated from a scale-wise convolution reconstruction network for final SR video reconstruction. To promote the compressed domain video SR research, we build an extensive Compressed Videos with Coding Prior (CVCP) dataset, including compressed videos of diverse content and various coding priors extracted from the bitstream. Extensive experimental results show the effectiveness of coding priors in compressed domain video SR.

Cross-component linear model (CCLM) prediction has been repeatedly proven to be effective in reducing the inter-channel redundancies in video compression. Essentially speaking, the linear model is identically trained by employing accessible luma and chroma reference samples at both encoder and decoder, elevating the level of operational complexity due to the least square regression or max-min based model parameter derivation. In this paper, we investigate the capability of the linear model in the context of sub-sampled based cross-component correlation mining, as a means of significantly releasing the operation burden and facilitating the hardware and software design for both encoder and decoder. In particular, the sub-sampling ratios and positions are elaborately designed by exploiting the spatial correlation and the inter-channel correlation. Extensive experiments verify that the proposed method is characterized by its simplicity in operation and robustness in terms of rate-distortion performance, leading to the adoption by Versatile Video Coding (VVC) standard and the third generation of Audio Video Coding Standard (AVS3).

The forthcoming Versatile Video Coding (VVC) standard adopts the trellis-coded quantization, which leverages the delicate trellis graph to map the quantization candidates within one block into the optimal path. Despite the high compression efficiency, the complex trellis search with soft-decision quantization may hinder the applications due to high complexity and low throughput capacity. To reduce the complexity, in this paper, we propose a low complexity trellis-coded quantization scheme in a scientifically sound way with theoretical modeling of the rate and distortion. As such, the trellis departure point can be adaptively adjusted, and unnecessarily visited branches are accordingly pruned, leading to the shrink of total trellis stages and simplification of transition branches. Extensive experimental results on the VVC test model show that the proposed scheme is effective in reducing the encoding complexity by 11% and 5% with all intra and random access configurations, respectively, at the cost of only 0.11% and 0.05% BD-Rate increase. Meanwhile, on average 24% and 27% quantization time savings can be achieved under all intra and random access configurations. Due to the excellent performance, the VVC test model has adopted one implementation of the proposed scheme.

Cross-component prediction, which takes advantage of inter-channel correlations, predicts the chroma block with the luma reconstructed block according to associated linear model. Instead of involving all available reference samples in building the linear model, in this paper, we propose a sub-sampled approach that utilizes at most four neighboring chroma samples and their corresponding down-sampled luma samples, leading to significantly reduced operations in the derivation of model parameters at both encoder and decoder. The proposed scheme is hardware friendly in terms of the overheads of memory access and clock cycles, and greatly benefits the practical implementations of the emerging video coding standard in real applications. Extensive experiments reveal that the proposed sub-sampled method provides simple operations and robust coding performance, leading to the adoption by Versatile Video Coding (VVC) Standard and the third generation Audio Video Coding Standard (AVS3).

In this paper, we focus on computationally modeling of the local texture correlations, in an effort to better explore the coding modes with higher priorities in the rate-distortion optimized intra coding. In particular, strong correlations and continuities of local texture with neighboring blocks have been revealed in our analysis, and empirical justifications provide us inspirations on the joint optimization of rate-distortion-complexity when angular modes become finer to adapt the local textures. We examine the philosophy with extensive experiments conducted for refining the intra full-RD list. The results show that better coding performance with on average 0.72% and 3.00% BD-Rate savings for the natural scene and screen content sequences can be achieved in AVS3 test model HPM-5.0 under all intra configuration, with negligible encoding and decoding time variations.

There has been a consensus regarding the intra prediction technique in video coding that the spatial redundancy can be efficiently removed by the locally accessible reference samples with certain projections and interpolations. In this paper, we revisit the short and long range correlations of the image content in the context of video coding, and it is interesting to find that the natural scene videos exhibit substantially different characteristics from screen content videos. This motivates us to redesign the intra mode coding method based on both short and long range correlations, as the existing approaches based on local content correlations cannot always effectively capture the most probable mode. One key feature of the proposed method is that it achieves unified content adaptive coding and is applicable across different video content. Experimental results on the versatile video coding (VVC) platform VTM-3.0 show the effectiveness of the proposed approach, leading to 3.73% bit rate savings for screen content videos and 0.10% bit rate savings for natural scene videos under all intra configuration.

The flexible partitioning structure such as quad-tree plus binary-tree (QTBT) plays important roles in the next generation video coding standard. This paper investigates new coding unit (CU) partitioning methods to further improve the compression efficiency. In particular, we propose the extended quad-tree partitioning and asymmetric ternary-tree partitioning to further develop the principle behind the existing partitioning structure. First, a central extended quad-tree (CENTRAL-EQT) partitioning is introduced, which extends the traditional QT partitioning with central pattern and generates four elaborately designed sub-CUs with different sizes. Second, we propose a parallel extended quad-tree (PARALLEL-EQT) partitioning that allows the CU to split along a single direction, leading to four identical size sub-blocks. Third, we present an asymmetric ternary-tree (ATT) method, which splits the CU asymmetrically into three sub-blocks. The proposed new partitioning methods allow the interleaving with binary-tree partitioning for enhanced adaptability, and they can be jointly enabled to capture different characteristics of the local content. Simulation results on the JEM7-QTBT-Only platform show that averagely 4.92%, 4.81% and 5.08% BD-Rate savings are achieved for the luma component by the proposed methods under random access (RA), low-delay P (LDP) and low-delay B (LDB) configurations, respectively, with around 725% encoding time increase and negligible decoding time increase. Furthermore, experimental results also reveal that the proposed partitioning schemes are effective when cooperating with the upcoming Versatile Video Coding standard.

The quad-tree plus binary-tree (QTBT) coding unit (CU) partitioning structure, which has been adopted to the next generation video coding standard, shows promising coding performance when compared with the conventional quad-tree structure in HEVC. In this paper, we propose the Extended Quad-tree (EQT) partitioning, which further extends the QTBT scheme and increases the partitioning exibility. More specifcally, EQT splits a parent CU into four sub-CUs of dierent sizes, which can adequately model the local image content that cannot be elaborately characterized with QTBT. Meanwhile, EQT partitioning allows the interleaving with BT partitioning for enhanced adaptability. Experimental results on the JEM7-QTBT-Only platform show that EQT brings better coding performance with 3.17%, 3.20% and 3.06% BD-Rate gains under random access, low-delay P and low-delay B configurations, respectively.

The third generation of the audio video coding standard (AVS3) is the latest standardization developed by Audio Video Coding Standard Working Group of China on the basis of IEEE 1857.4. In this paper, we propose a History-based Motion Vector Prediction (HMVP) method for inter coding, which fully explores the history motion information of previous coded blocks. More specifically, an HMVP table, which records HMVP candidates, is elaborately maintained with limited capacity. In particular, the motion information of the previous coded blocks updates the HMVP table on-the-fly by appending the candidate to the last entry of the table after encoding or decoding an inter block. As such, the HMVP candidates act as additional modes for skip or direct modes to enhance coding efficiency. Besides, HMVP occupies limited memory to store motion info, which is crucial in real applications. Experimental results show that HMVP brings better coding performance with up to 1.38% and 2.33% BD-Rate reduction for random access and low-delay P configurations, respectively, on the latest TAVS3-2.2 platform. Moreover, nearly no encoding and decoding burden is introduced by HMVP. The proposed HMVP has been adopted by AVS3.

AVS3 adopts quad-tree (QT) with nested binary tree (BT) and extended quad-tree (EQT) partitioning, which shows promising compression performance when compared to the conventional QT partitioning in AVS2 and HEVC. However, the interleaved and recursive splitting manner significantly increases the computational complexity of the encoder, which may impede the real applications of AVS3. This paper proposes fast coding unit splitting decision methods for QT, BT and EQT partitioning. In particular, the Bayesian decision rule is employed with Skip states, and BT and EQT partitioning can be early terminated. Additionally, the average splitting depth that is produced by BT partitioning, is incorporated as the prior information for terminating the EQT and QT splittings. Moreover, we explore the splitting settings for CUs that are predicted with Skip mode, with the aim of eliminating unnecessary partition attempts. Experimental results show that the proposed fast algorithms are effective and provide a good trade-off between computational complexity and coding performance. In particular, 69% encoding time reduction is achieved with only 0.55% increase in terms of BD-Rate on average, which greatly benefits the practical implementations of the AVS3 in real applications. The proposed methods have been adopted into TAVS3 reference software.

Rate-Distortion Optimized Quantization (RDOQ) brings significant improvement of coding performance in High Efficiency Video Coding (HEVC). However, it results in high computational complexity to determine the best quantization levels for each transform coefficient when applying rate and distortion optimization operation. In this paper, we firstly proposed an efficient way to skip the all zero coding units. Then we further propose a fast RDOQ scheme by skipping the optimization step of All Quantized Zero Blocks except DC Coefficient (AQZB-DC). Moreover, a rate difference model is established to select optimal quantization level for AQZB-DC blocks. Experiments are carried out on reference software HM 16.0 and the results show that the proposed method achieves 42.00% and 39.63% quantization time saving under Random Access (RA) and Low Delay (LD) configuration on average, while the BD-Rate loss is only 0.03% and 0.06%, respectively.