Omnidirectional videos (ODVs) provide an immersive visual experience by capturing the 360° scene. With the rapid advancements in virtual/augmented reality, metaverse, and generative artificial intelligence, the demand for high-quality ODVs is surging. However, ODVs often suffer from low resolution due to their wide field of view and limitations in capturing devices and transmission bandwidth. Although video super-resolution (SR) is a capable video quality enhancement technique, the performance ceiling and practical generalization of existing methods are limited when applied to ODVs due to their unique attributes. To alleviate spatial projection distortions and temporal flickering of ODVs, we propose a Spatio-Temporal Distortion Aware Network (STDAN) with joint spatio-temporal alignment and reconstruction. Specifically, we incorporate a spatio-temporal continuous alignment (STCA) to mitigate discrete geometric artifacts in parallel with temporal alignment. Subsequently, we introduce an interlaced multi-frame reconstruction (IMFR) to enhance temporal consistency. Furthermore, we employ latitude-saliency adaptive (LSA) weights to focus on regions with higher texture complexity and human-watching interest. By exploring a spatio-temporal jointly framework and real-world viewing strategies, STDAN effectively reinforces spatio-temporal coherence on a novel ODV-SR dataset and ensures affordable computational costs. Extensive experimental results demonstrate that STDAN outperforms state-of-the-art methods in improving visual fidelity and dynamic smoothness of ODVs.

Recent advances in AI-generated content (AIGC) have led to the emergence of powerful text-to-video generation models. Despite these successes, evaluating the quality of AIGC-generated videos remains challenging due to limited generalization, lack of temporal awareness, heavy reliance on large-scale annotated datasets, and the lack of effective interaction with generation models. Most current approaches rely on supervised fine-tuning of vision-language models (VLMs), which often require large-scale annotated datasets and tend to decouple understanding and generation. To address these shortcomings, we propose VQ-Insight, a novel reasoning-style VLM framework for AIGC video quality assessment. Our approach features: (1) a progressive video quality learning scheme that combines image quality warm-up, general task-specific temporal learning, and joint optimization with the video generation model; (2) the design of multi-dimension scoring rewards, preference comparison rewards, and temporal modeling rewards to enhance both generalization and specialization in video quality evaluation. Extensive experiments demonstrate that VQ-Insight consistently outperforms state-of-the-art baselines in preference comparison, multi-dimension scoring, and natural video scoring, bringing significant improvements for video generation tasks.

Preprocessing is a well-established technique for optimizing compression, yet existing methods are predominantly Rate-Distortion (R-D) optimized and constrained by pixel-level fidelity. This work pioneers a shift towards Rate-Perception (R-P) optimization by, for the first time, adapting a large-scale pre-trained diffusion model for compression preprocessing. We propose a two-stage framework: first, we distill the multi-step Stable Diffusion 2.1 into a compact, one-step image-to-image model using Consistent Score Identity Distillation (CiD). Second, we perform a parameter-efficient fine-tuning of the distilled model's attention modules, guided by a Rate-Perception loss and a differentiable codec surrogate. Our method seamlessly integrates with standard codecs without any modification and leverages the model's powerful generative priors to enhance texture and mitigate artifacts. Experiments show substantial R-P gains, achieving up to a 30.13% BD-rate reduction in DISTS on the Kodak dataset and delivering superior subjective visual quality.

Generative face video coding (GFVC) is vital for modern applications like video conferencing, yet existing methods primarily focus on video motion while neglecting the significant bitrate contribution of audio. Despite the well-established correlation between audio and lip movements, this cross-modal coherence has not been systematically exploited for compression. To address this, we propose an Audio-Visual Cross-Modal Compression (AVCC) framework that jointly compresses audio and video streams. Our framework extracts motion information from video and tokenizes audio features, then aligns them through a unified audio-video diffusion process. This allows synchronized reconstruction of both modalities from a shared representation. In extremely low-rate scenarios, AVCC can even reconstruct one modality from the other. Experiments show that AVCC significantly outperforms the Versatile Video Coding (VVC) standard and state-of-the-art GFVC schemes in rate-distortion performance, paving the way for more efficient multimodal communication systems.

JPEG AI is an emerging learning-based image coding standard developed by Joint Photographic Experts Group (JPEG). The scope of the JPEG AI is the creation of a practical learning-based image coding standard offering a single-stream, compact compressed domain representation, targeting both human visualization and machine consumption. Scheduled for completion in early 2025, the first version of JPEG AI focuses on human vision tasks, demonstrating significant BD-rate reductions compared to existing standards, in terms of MS-SSIM, FSIM, VIF, VMAF, PSNR-HVS, IW-SSIM and NLPD quality metrics. Designed to ensure broad interoperability, JPEG AI incorporates various design features to support deployment across diverse devices and applications. This paper provides an overview of the technical features and characteristics of the JPEG AI standard.

Autoencoder-based neural compression methods leverage expressive models to fit large datasets but often incur considerable decoding complexity. Recently, overfitted codecs with reduced decoding complexity have gained attention as an alternative. However, they usually require access to entire videos or multiple frames simultaneously for encoding, resulting in substantial system delays. To address these limitations, we propose CNVC, a compact neural video codec that employs instance-level adaptation for efficient and flexible video compression. CNVC is fully overfitted (each frame is optimized independently using up to 45k iterations for maximum performance in this paper), building on the COOL-CHIC video model with substantial architectural and training enhancements. At a decoding complexity of just 1300 MACs per pixel, CNVC provides a more compact solution than previous autoencoder-based and overfitted codecs. Additionally, CNVC inherits the frame-wise overfitting mechanism of COOL-CHIC video, enabling flexible encoding configurations (e.g., low-delay1). In terms of compression efficiency, CNVC achieves significant bitrate reductions on HEVC and UVG datasets compared to COOL-CHIC video. To our knowledge, CNVC is the first compact neural codec to match HEVC (x265 slow setting) performance at such a decoding complexity level.

Implicit Neural Representations (INRs) have demonstrated significant potential in video compression by representing videos as neural networks. However, as the number of frames increases, the memory consumption for training and inference increases substantially, posing challenges in resource-constrained scenarios. Inspired by the success of traditional video compression frameworks, which process video frame by frame and can efficiently compress long videos, we adopt this modeling strategy for INRs to decrease memory consumption, while aiming to unify the frameworks from the perspective of timeline-based autoregressive modeling. In this work, we present a novel understanding of INR models from an autoregressive (AR) perspective and introduce a Unified AutoRegressive Framework for memory-efficient Neural Video Compression (UAR-NVC). UAR-NVC integrates timeline-based and INR-based neural video compression under a unified autoregressive paradigm. It partitions videos into several clips and processes each clip using a different INR model instance, leveraging the advantages of both compression frameworks while allowing seamless adaptation to either in form. To further reduce temporal redundancy between clips, we treat the corresponding model parameters as proxies for these clips, and design two modules to optimize the initialization, training, and compression of these model parameters. In special, the Residual Quantization and Entropy Constraint (RQEC) module dynamically balances the reconstruction quality of the current clip and the newly introduced bitrate cost using the previously optimized parameters as conditioning. In addition, the Interpolation-based Initialization (II) module flexibly adjusts the degree of reference used during the initialization of neighboring video clips, based on their correlation. UAR-NVC supports adjustable latencies by varying the clip length. Extensive experimental results demonstrate that UAR-NVC, with its flexible video clip setting, can adapt to resource-constrained environments and significantly improve performance compared to different baseline models. The project page: https://wj-inf.github.io/UAR-NVC-page/.

Point cloud sequences play a pivotal role in immersive applications as they provide 6 degree of freedom experience into dynamic 3D environments. However, the dynamic point cloud sequences have an overwhelming amount of data, which poses challenges for storage and transmission. Therefore, effective point cloud sequence compression is crucial for immersive applications. In this paper, we propose a dynamic point cloud compression framework, which enhances the compression efficiency of point cloud sequences from the perspectives of representation and prediction. To improve the representation and reconstruction ability of a single frame, we propose a specialized pre-training mechanism for compression, which enables a well-designed autoencoder to acquire substantial prior knowledge. The pre-training mechanism is based on the proposed mask operation and is trained on a large amount of data of multiple categories. This enables the pre-trained encoder to identify and extract features with strong representational capabilities, and allows the pre-trained decoder to accurately reconstruct the complete point cloud. For precise prediction, we develop tailored methods for different frame types. For intra-frames (I frames), we introduce a dual-branch spatial intra-prediction network by leveraging coordinate and occupancy information from downsampled sparse point clouds to predict representative features. For inter-frames (P frames), we propose a multi-scale context-based spatio-temporal inter-prediction network. Specifically, for each scale, we utilize the proposed Transformer-style spatio-temporal modeling module to analyze and model the temporal characteristics among multiple reference frames, providing rich context information. Integrating three-scale contexts, we achieve comprehensive feature prediction, significantly improving accuracy and eliminating spatio-temporal redundancy. Experimental results show that our method outperforms existing compression methods in both I-frame compression and P-frame compression.

Recent advances in video compression introduce implicit neural representation (INR) based methods, which effectively capture global dependencies and characteristics of entire video sequences. Unlike traditional and deep learning based approaches, INR-based methods optimize network parameters from a global perspective, resulting in superior compression potential. However, most current INR methods utilize a fixed and uniform network architecture across all frames, limiting their adaptability to dynamic variations within and between video sequences. This often leads to suboptimal compression outcomes as these methods struggle to capture the distinct nuances and transitions in video content. To overcome these challenges, we propose Hierarchically Adaptive Neural Representation for Video Compression (HANeRV), an innovative INR-based video compression network that adaptively conducts structure optimisation based on the specific content of each video sequence. To better capture dynamic information across video sequences, we propose a dynamic architecture-level adjustment (DAA). Furthermore, to enhance the capture of dynamics between frames within a sequence, we implement a dynamic frame-level adjustment (DFA). Finally, to effectively capture spatial structural information within video frames, thereby enhancing the detail restoration capabilities of HANeRV, we devise a structure level hierarchical structural adaptation (HSA). Experimental results show that HANeRV achieves state-of-the-art performance among INR-based video compression methods and surpasses the H.266/VVC (x266, medium preset) anchor on diverse datasets.

Neural video compression (NVC) has emerged as a promising paradigm for improving rate-distortion performance. However, existing neural video codecs predominantly rely on convolutional neural networks (CNNs) with limited local receptive fields to generate the latent representations, often neglecting global–local spatial correlations. This leads to suboptimal feature modeling and redundancy in the latent space. To address this limitation, we propose a novel Dual-Scale Transformer (DST) block specifically tailored for NVC, which effectively enhances coding efficiency. The DST block incorporates a Global–Local (Shifted) Window-based Self-Attention (GL(S)WSA) mechanism to jointly capture global structure information and local texture details. Moreover, we design a Cross-Gated Feed-Forward Network (CGFFN) to adaptively modulate complementary components, producing more compact and expressive latent representations. Furthermore, to overcome the drawbacks of traditional asynchronous training and further boost rate-distortion performance, we introduce a Variable Bitrate Synchronization (VBRS) strategy that leverages multi-GPU parallel training, with each GPU dedicated to a specific bitrate and synchronized via gradient backpropagation for joint optimization. Experimental results demonstrate that our proposed method achieves the higher coding performance compared to the previous state-of-the-art (SOTA) methods and significantly outperforms H.266/VVC (VTM-13.2) under various low delay B (LDB) coding configurations.

Resampling-based video coding is a well-established strategy for compressing high-resolution video under limited bandwidth. Recently, super resolution (SR) technologies has been explored for upsampling the low-resolution reconstructed video in the decoder side. Despite the promising rate-distortion improvements these SR model can offer, the significant complexity of neural network makes them impractical for real-world applications. In this paper, we propose a low complexity SR model for resampling-based video coding by applying CP decomposition to vanilla convolution to reduce complexity. To enhance the performance of the proposed low complexity SR, we utilize multiple decoded information generated during compression as additional auxiliary input of SR. This information provides valuable insights of the compressed picture, such as the texture and directional details. Additionally, to minimize the number of parameters in the neural network, a single model with split branches is designed to process luma and chroma components separately. Experimental results demonstrate that the proposed method achieves 3.72% and 3.65% BD-rate savings under random access and all-intra configurations compared to the latest video coding standard VVC/H.266, respectively. Compared to existing works, the proposed SR model exhibits a superior trade-off between complexity and coding performance.

Lossy image compression networks aim to minimize the latent entropy of images while adhering to specific distortion constraints. However, optimizing the neural network can be challenging due to its nature of learning quantized latent representations. In this paper, our key finding is that minimizing the latent entropy is, to some extent, equivalent to maximizing the conditional source entropy, an insight that is deeply rooted in information-theoretic equalities. Building on this insight, we propose a novel structural regularization method for the neural image compression task by incorporating the negative conditional source entropy into the training objective, such that both the optimization efficacy and the model's generalization ability can be promoted. The proposed information-theoretic regularizer is interpretable, plug-and-play, and imposes no inference overheads. Extensive experiments demonstrate its superiority in regularizing the models and further squeezing bits from the latent representation across various compression structures and unseen domains.

The in-loop filter tools in video coding, such as De-Blocking Filter (DBF) and Adaptive Loop Filter (ALF), can significantly improve the video compression performance of the VVC standard. In this paper, the coding-information based improvements for DBF and ALF are proposed to further increase the coding efficiency beyond VVC. The coding-information can be referred to the partitioning information which is not fully utilized by in-loop filters in the VVC standard. Two improvements are included in this paper. The first improvement is the coding-information based classification for ALF. In the current design of ALF, the partitioning information has not been considered for the classification of both online and offline filters. To further increase the classification diversity and efficiency, the coding-information is introduced into classification of both online and offline filters through an always applied rule. The second improvement is the boundary-aware offset refinement. In current DBF and ALF design, the offset generated by DBF and ALF cannot be further adjusted after the filtering process. To provide more possibilities for the filtering offset, an offset refinement process is proposed. The offset can be refined by a non-linear function with the help of the coding-information. The proposed two improvements are implemented on top of ECM-14.0 software which is published by JVET. Experimental results show that the classification method can achieve -0.08% and -0.12% while the offset refinement method can achieve -0.05% and -0.11% BD-rate reduction on the luma component in Random Access and Low-Delay B configurations respectively. Both proposed improvements have been adopted into ECM.

This paper explores the application of enhancement filtering techniques in neural video compression. Specifically, we categorize these techniques into in-loop contextual filtering and out-of-loop reconstruction enhancement based on whether the enhanced representation affects the subsequent coding loop. In-loop contextual filtering refines the temporal context by mitigating error propagation during frame-by-frame encoding. However, its influence on both the current and subsequent frames poses challenges in adaptively applying filtering throughout the sequence. To address this, we introduce an adaptive coding decision strategy that dynamically determines filtering application during encoding. Additionally, out-of-loop reconstruction enhancement is employed to refine the quality of reconstructed frames, providing a simple yet effective improvement in coding efficiency. To the best of our knowledge, this work presents the first systematic study of enhancement filtering in the context of conditional-based neural video compression. Extensive experiments demonstrate a 7.71% reduction in bit rate compared to state-of-the-art neural video codecs, validating the effectiveness of the proposed approach.

Recent advancements in learned rate-distortion optimization (RDO) showcase that by making the intra coding decisions based on a learned measure, the encoding can be significantly accelerated without incurring much coding loss. Despite great progress in complexity reduction, the dependency issue has been largely neglected in the current learned RDO research. In this study, aiming to tap the full potential of dependent learned RDO, we first derive a probabilistic RDO framework for theoretical analysis, under which the classic and the learned RDO problems are equivalent to the maximum a posteriori (MAP) inference and the distribution imitation, respectively. Subsequently, we probabilistically revisit dependency considerations in the intra RDO research. Our key finding is that the existing learned RDO scheme can only produce a measure that indicates the local “goodness” of coding decisions. We therefore further discuss the opportunities for learning a dependent measure that is more optimal in the long run. Finally, as learning an accurate measure for the full decision space could be extremely challenging, taking the High Efficiency Video Coding (HEVC) intra coding as a case study, we experimentally identify that the prediction decision accounts for the majority of the dependent optimization gain and is of the utmost value to be learned, paving the way for future research on dependent learned RDO.

In the pursuit of achieving further coding gains beyond the versatile video coding (VVC) standard, the enhanced compression model (ECM) has been initiated by the Joint Video Exploration Team (JVET) with the aim of developing next generation video coding techniques. In ECM, novel coding tools are studied to improve the coding efficiency for both camera-captured content and screen content. Intra block copy (IBC) has been included as a fundamental coding tool in both VVC and ECM, yielding significant improvement in compression efficiency for screen content. This paper presents a method of reconstruction reordered IBC (RR-IBC) to further improve the compression efficiency for screen content, by taking advantage of the symmetry property inherent in screen content sequences. The reconstruction block is flipped horizontally or vertically to restore the characteristics of samples in the original block. A flip-aware adjustment is performed to regulate block vector candidates of the RR-IBC block according to the types of symmetry. Similarly, the reference template of the template-based reordering method for the RR-IBC block is adjusted accordingly to accommodate the geometry property. A motion constraint is applied to restrict the block vector of an RR-IBC coded block to a single direction displacement perpendicular to the flip axis. An RR-IBC flip mode index is signalled to specify how to flip the reconstruction block. Experimental results show that the proposed RR-IBC can provide an average Bjontegaard delta rate (BD-rate) saving of 1.61%/1.79%/1.76% and 3.90%/3.63%/3.63% on Y/Cb/Cr components for class F and class TGM sequences, respectively, with a negligible change on the runtime, compared with ECM-5.0 in all intra configurations. RR-IBC has been adopted into ECM.

Point cloud compression is critical for the success of immersive multimedia applications. For attribute compression in geometric point cloud compression (G-PCC), Region Adaptive Hierarchical Transform (RAHT) is the preferred coding method. This paper presents several advances to predictive coding with RAHT: 1) Sample Domain Prediction: Prediction in RAHT is done in transform domain. This introduces undesirable distortion to the prediction signal because of fixed-point computations and leads to increased decoding complexity. We address this by naturally applying prediction in sample domain. The method opens door to skip the transform stage altogether when all residues are quantized to zero, leading to a significantly light decoder. 2) Reference Node Resampling: Inter-prediction signal derived in RAHT could have a different occupancy and weight distribution compared to the current block, causing a mismatch. To address this, we resample the reference node and align the occupancy and weight distribution. 3) Temporal Filtering: During inter-prediction, the reference node is simply copied as the prediction signal. This assumes a correlation coefficient of unity, which is barely true. We introduce a temporal filtering mechanism conditioned on the sub-band, that emulates a low-pass filtering and achieves improved prediction. 4) Inter-Eligibility: During AC inter-prediction, both encoder and decoder have access to the DC of the current and the reference nodes. We use this information to derive an inter-eligibility criterion. Experimental results show considerable gains and reduced complexity that demonstrate the utility of the proposed methods. All the presented methods have been adopted to the second version of G-PCC.

In recent years, there has been a rapid growth in applications that rely on point clouds to represent the 3D world, driven by the increasing demand for immersive and other related scenarios. However, compressing the large and high-precision point cloud data efficiently while maintaining high perceptual quality for human vision remains a challenge. To solve the problem, we propose a new structure-aware generative point cloud compression framework for human vision. In the encoder, we focus on information that is more sensitive to the human vision and obtain this type of information from different scale. This allows us to capture structural importance information from global scale and local scale, which are more difficult to reconstruct. For the decoder, we introduce a progressive generative reconstruction approach that utilizes acquired information from the encoder to guide the generation of point cloud surfaces. Moreover, we propose a novel probability cloud-based discriminator. Instead of directly assessing the authenticity of the generated point clouds, our discriminator evaluates the probability distribution of the existence of points within the generated point cloud. This approach reduces the difficulty of discrimination while effectively improving the accuracy of the generator in generating probability distributions. According to the correct probability, we can obtain a high accuracy point cloud by pruning the points with low probability. Through comprehensive experiments, we demonstrate the effectiveness and superiority of our proposed framework in terms of encoding efficiency, high perceptual quality, and generation quality.

The past decade has witnessed the huge success of deep learning in well-known artificial intelligence applications such as face recognition, autonomous driving, and large language model like ChatGPT. Recently, the application of deep learning has been extended to a much wider range, with Neural Network-Based Video Coding (NNVC) being one of them. NNVC can be performed at two different levels: embedding neural network-based (NN-based) coding tools into a classical video compression framework or building the entire compression framework upon neural networks. This article elaborates our studies in response to the recent exploration efforts in JVET (Joint Video Experts Team of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC29) in the name of NNVC, falling in the former category. Specifically, in this article, we propose two advanced NN-based video coding technologies, i.e., NN-based intra prediction and NN-based in-loop filtering, which have been investigated for several meeting cycles in JVET and then adopted into the reference software, i.e., NNVC. In addition, we further propose a Small Ad-hoc Deep-Learning Library (SADL), which provides integer-based inference capabilities for neural networks to ensure interoperability across different systems. SADL has been adopted as the inference platform of all neural networks in NNVC. Extensive experiments on top of the NNVC have been conducted to evaluate the effectiveness of the proposed techniques. Compared with VTM-11.0_nnvc, the proposed two NN-based coding tools jointly achieve {11.94%, 21.86%, 22.59%}, {9.18%, 19.76%, 20.92%}, and {10.63%, 21.56%, 23.02%} BD-rate reductions on average for {Y, Cb, Cr} under random-access, low-delay, and all-intra configurations, respectively.

Recently, many works have applied deep learning techniques to video compression tasks, achieving promising results and advancing the field of Deep Learning-Based Video Compression (DLVC). However, the architecture design of the existing DLVC is rigid and limited in terms of flexibility. Specifically, different networks must be designed for different scenarios, such as delay-constrained scenario or non-delay-constrained scenario. Frequent switching between networks would reduce the speed of modern deep learning platforms and increase the maintenance costs. To address this problem, we propose a Unified Video Compression (UVC) framework that can be freely switched to different application scenarios without changing the network architecture. Our proposed UVC framework is based on the explicit-compression and implicit-generation perspective, which contains two sub-networks—the Explicit Reference Frame Compression Network (ERFCN) and the Implicit Reference Frame Generation Network (IRFGN). The aim of ERFCN is to compress the current frame with the help of the reference frame. To improve the performance of ERFCN, we first introduce the Transformer in this network, which can fully remove the spatial redundancy of the input image and is beneficial for the following inter-prediction process. We also develop a novel long-range motion estimation module for inter-prediction to generate motion vectors based on global motion information between two frames, which can handle long-range complex motion relations. The aim of IRFGN is to capture the temporal relationship between forward and backward reconstructed frames and synthesize a high-quality implicit reference frame for the current frame. To achieve this, we design the split spatial-temporal attention and multi-scale prediction module. We conduct extensive experiments on three widely used video compression databases (HEVC, UVG, and MCL-JVC), and the results demonstrate the superiority of our approach over other related DLVC methods.

To satisfy the rapidly increasing demands on the large image (2K-8K) super-resolution (SR), prevailing methods follow two independent tracks: 1) accelerate existing networks by content-aware routing, and 2) design better super-resolution networks via token mixer refining. Despite directness, they encounter unavoidable defects (e.g., inflexible route or non-discriminative processing) limiting further improvements of quality-complexity trade-off. To erase the drawbacks, we integrate these schemes by proposing a content-aware mixer (CAMixer), which assigns convolution for simple contexts and additional deformable window-attention for sparse textures. Specifically, the CAMixer uses a learnable predictor to generate multiple bootstraps, including offsets for windows warping, a mask for classifying windows, and convolutional attentions for endowing convolution with the dynamic property, which modulates attention to include more useful textures self-adaptively and improves the representation capability of convolution. We further introduce a global classification loss to improve the accuracy of predictors. By simply stacking CAMixers, we obtain CAMixerSR which achieves superior performance on large-image SR, lightweight SR, and omnidirectional-image SR.

Adaptive video streaming requires efficient bitrate ladder construction to meet heterogeneous network conditions and end-user demands. Per-title encoding optimization typically traverses numerous encoding parameters to search the Pareto-optimal operating points for each video. Recently, researchers have attempted to predict the content-optimized bitrate ladder for pre-encoding overhead reduction [1] . However, as shown in Fig. 1 , current methods actually estimate the optimal encoding parameters that lie on the Pareto front and thus still require subsequent pre-encodings.

We introduce a novel and challenging task within the AIGC era: coding for AIGI (AI-generated images). Specifically, we propose the first AIGI dataset called PKU-AIGI-500K, which is meticulously constructed based on five major foundation models with diverse prompts. Furthermore, We conduct extensive and systematic analysis of the essential characteristics of AIGC images. We thoroughly benchmark the rate-distortion performance and runtime complexity analysis of conventional and learned image coding solutions that are openly available, revealing new insights for emerging studies in AIGI compression. The main contributions of this paper can be summarized as follows: (i) we contribute and build the first AIGI dataset, PKU-AIGI-500K, containing 105+ prompts and 528k+ images based on five generative models. Additionally, we analyze the image features that may be beneficial for image compression, such as quality, texture, color, etc., providing novel insights into available solutions and paving the way for future research. (ii) Building upon the proposed PKU-AIGI-500K, we evaluate the compression efficiency using some popular traditional codecs and learning-based codecs to form a strong benchmark. We also observe that the learning-based models trained on natural images cannot achieve competitive performance on the AIGIs without fine-tuning, emphasizing the necessity of a sufficiently large dataset to advance the research of AIGI coding. (iii) The evaluation and benchmarking are accomplished as an AIGIs’ compression project.

In this paper, we present a Conv-Attention, a decoder-friendly attention mechanism, in an effort to advancing the practical application of the artificial intelligence-based image coding. More specifically, the proposed method is tailored for JPEG AI, which is the latest advanced neural-network based image coding standard. By identifying the obstacles by profiling the decoding complexity of JPEG AI, the attention module accounts for a significant proportion, which mainly attributes to the intricate network structure and involvement of less efficient operations. Conv-Attention model is composed with plain convolution and activation computations, equipping with sub-scaling and up-scaling design, such that the non-adjacent features can be well captured, leading to the reduction of decoding complexity and maintenance of the synthesis and attentive capability. Simulation results verify the effectiveness of the proposed method with JPEG AI reference software, wherein the decoding complexity is reduced by 80% with negligible coding performance loss. The proposed method was adopted in the 100th JPEG meeting.

There has been a growing trend in compressing and transmitting videos from terminals for machine vision tasks. Nevertheless, most video coding optimization method focus on minimizing distortion according to human perceptual metrics, overlooking the heightened demands posed by machine vision systems. In this paper, we propose a video preprocessing framework tailored for machine vision tasks to address this challenge. The proposed method incorporates a neural preprocessor which retaining crucial information for subsequent tasks, resulting in the boosting of rate-accuracy performance. We further introduce a differentiable virtual codec to provide constraints on rate and distortion during the training stage. We directly apply widely used standard codecs for testing. Therefore, our solution can be easily applied to real-world scenarios. We conducted extensive experiments evaluating our compression method on two typical downstream tasks with various backbone networks. The experimental results indicate that our approach can save over 15% of bitrate compared to using only the standard codec anchor version.

In this paper, a hybrid video compression framework is proposed that serves as a demonstrative showcase of deep learning-based approaches extending beyond the confines of traditional coding methodologies. The proposed hybrid framework is founded upon the Enhanced Compression Model (ECM), which is a further enhancement of the Versatile Video Coding (VVC) standard. We have augmented the latest ECM reference software with well-designed coding techniques, including block partitioning, deep learning-based loop filter, and the activation of block importance mapping (BIM) which was integrated but previously inactive within ECM, further enhancing coding performance. We evaluate the coding performance of the proposed framework with extensive experiments on the JVET dataset compared with ECM10.0 and VTM-11.0. Due to the testing environment and the coding complexity of the ECM, we did not conduct testing on Class A. The QPs are set as 22, 27, 32, 37, and 42. Compared with ECM-10.0, our method achieves 6.26%, 13.33%, and 12.33% BD-rate savings for the Y, U, and V components under random access (RA) configuration. The traditional hybrid coding framework combined with the three coding tools can further improve compression efficiency and has great potential for performance improvement.

The essence of dynamic point cloud compression lies in the effective modeling of temporal context information, which poses significant challenges owing to the unstructured and sparse characteristics of point clouds. Existing dynamic compression methods exhibit a limited capacity to capture and leverage inter-frame information. Consequently, in this paper, we propose a Dynamic Point Cloud Compression framework with Spatio-Temporal Transformer-style Modeling (DPCC-STTM) to compress point cloud sequences within a latent space. To effectively extract and fully utilize temporal context, we introduce a spatio-temporal transformer-style modeling module, which performs effective modeling of the rich temporal information based on the correlation of temporal content. Furthermore, we introduce a multi-scale temporal processing module that captures temporal correlations across short and long ranges of multi-frame point clouds. This module also fuses modeled temporal information to enhance the prediction accuracy of potential features for the current frame. Extensive experiments demonstrate the superiority of our proposed framework, validated through both objective evaluation and subjective perception.

Sharpening is a widely adopted video enhancement technique. However, uniform sharpening intensity ignores texture variations, degrading video quality. Sharpening also increases bitrate, and there’s a lack of techniques to optimally allocate these additional bits across diverse regions. Thus, this paper proposes RPO-AdaSharp, an end-to-end region-adaptive video sharpening model for both perceptual enhancement and bitrate savings. We use the coding tree unit (CTU) partition mask as prior information to guide and constrain the allocation of increased bits. Experiments on benchmarks demonstrate the effectiveness of the proposed model qualitatively and quantitatively.

In recent years, user generated content (UGC) has become the dominant force in internet traffic. However, UGC videos exhibit a higher degree of variability and diverse characteristics compared to traditional encoding test videos. This variance challenges the effectiveness of data-driven machine learning algorithms for optimizing encoding in the broader context of UGC scenarios. To address this issue, we propose a Tri-Dynamic Preprocessing framework for UGC. Firstly, we employ an adaptive factor to regulate preprocessing intensity. Secondly, an adaptive quantization level is employed to fine-tune the codec simulator. Thirdly, we utilize an adaptive lambda tradeoff to adjust the rate-distortion loss function. Experimental results on large-scale test sets demonstrate that our method attains exceptional performance.

Recently, a decoupled framework for learning-based image compression has been proposed and adopted into the JPEG AI image coding standard developed by ISO/IEC WG1. The decoupled structure disentangles the sample reconstruction process and the entropy decoding process, making the decoding extremely fast. The corresponding techniques constitute the essential parts of the JPEG AI verification model software. However, its analysis transform and synthesis transform are relatively simple, which are built with stacked convolution layers, thereby may lack the capability to interpret data correlations. In this work, we enhance the transform networks by introducing the non-local attention mechanism, which has proven efficient in image compression tasks. The proposed framework thus shares the merits of the fast decoding from the decoupled architecture and the strong transform capabilities from the non-local attention, making it a stronger candidate for practical end-to-end image codec deployment. Experimental results on the Kodak test set and JPEG AI CfP test set show that our method achieves better BDRate performance compared to the original Decoupled-anchor and significantly faster decoding speed compared to NIC. The proposed solution has been adopted by the IEEE 1857.11 Working Subgroup (1857.11 WSG) in developing neural network-based image coding standards in the 10th Meeting.

Although ultra-high resolution point clouds have been acquired more easily, the huge amount of the data makes it more challenging to store and transmit, and also difficult to be applied in lightweight terminal. To address this challenge, we propose an Adaptive Downsampling and Spatial Upconversion framework for point cloud compression (ADSU). In the proposed adaptive downsampling, we introduce two key components, feature-aware augmented graph convolution (FAGC) and adaptive-sampling-based global graph aggregation module (ASGGA), to capture correlations between local and global features. For upsampling, we propose a high-frequency feature generation module (HFFG) to generate detailed information, which plays a crucial role in achieving precise reconstruction. Experimental results demonstrate that the combination of our proposed ADSU with popular point cloud compression methods can significantly improve compression performance.

With the advent of virtual reality technology, omnidirectional image (ODI) rescaling techniques are increasingly embraced to reduce transmitted and stored file sizes while preserving high image quality. Despite this progress, current ODI rescaling methods predominantly focus on enhancing the quality of images in equirectangular projection (ERP) format, which overlooks the fact that the content viewed on head-mounted displays (HMDs) is actually a rendered viewport instead of an ERP image. In this work, we emphasize that focusing solely on ERP quality results in inferior viewport visual experiences for users. Thus, we propose ResVR, which is the first comprehensive framework for the joint Rescaling and Viewport Rendering of ODIs. ResVR allows obtaining LR ERP images for transmission while rendering high-quality viewports for users to watch on HMDs. In our ResVR, a novel discrete pixel sampling strategy is developed to tackle the complex mapping between the viewport and ERP, enabling end-to-end training of the ResVR pipeline. Furthermore, a spherical pixel shape representation technique is innovatively derived from spherical differentiation to significantly improve the visual quality of rendered viewports. Extensive experiments demonstrate that our ResVR outperforms existing methods in viewport rendering tasks across different fields of view, resolutions, and view directions while keeping a low transmission overhead. Code is available at https://github.com/lwq20020127/ResVR.

Deep video compression has attracted increasing attention in recent years due to its end-to-end optimization ability. However, most existing neural video compression (NVC) models focus on incorporating sophisticated motion or residual coding networks for successive frames leveraging spatial-temporal redundancy removal, neglecting the efficient motion representation and essential structure of scaled prediction for motion dynamics. To resolve this problem, this paper proposed a novel model, named scaled hierarchical bi-directional prediction structure, which effectively captures temporal correlation among frames considering the quality variation when managing the reference frames. This paper first introduces parameter-shared motion codecs and efficient information fusion strategies to obtain predictive features more precisely. Subsequently, scaled motions from temporal contexts are learned as bi-directional prior for motion representation. Additionally, the concept of trustworthy motion modeling is proposed to represent the effectiveness of reference information, measuring the reliability of predictive accuracy in complex motions, camera rotations and occlusions. Extensive experimental results demonstrate that our approach offers significant advantages over state-of-the-art bi-directional NVC models in coding efficiency. The proposed method has been adopted as the latest reference model by Moving Picture, Audio, and Data Coding by Artificial Intelligence (MPAI) end-to-end video coding (EEV) standard. The code is available at: https://github.com/yefeng00/DVC_with_Scaled_Hierarchical_Bi_directional_Motion_Model.

Benefiting from the 360 • field of view (FoV) of the omnidirectional images (ODIs), users could enjoy an immersive experience with head-mounted devices or computers.High-resolution ODIs can provide pleasing visual experience and boost the performance of related visual tasks.Therefore, Super-resolution (SR) is an essential technique during the application of ODIs.However, traditional SR methods fail to enhance the most widely utilized equirectangular projection (ERP) format ODIs due to projection distortions.Existing ODI-SR methods take the latitude-related position information as a prior, but lack the adaptation to the ERP content distribution characteristics.To address this issue, we propose a novel Frequency Attention Transformer ODI-SR (FATO) network focusing on highfrequency details of ODIs.In particular, we transform an ODI into fine-grained patches in the frequency domain through Discrete Cosine Transform (DCT).After that, we design a frequency selfattention mechanism to capture the relationship between different frequency patches.Subsequently, we introduce a frequency loss function to further constrain the network.Extensive experimental results demonstrate that the proposed FATO achieves superior performance over state-of-the-art methods on ODIs.

Wavelet-like transform, based on convolutional neural network (CNN), is content-adaptive and has made remarkable achievements in end-to-end image compression. However, the subsequent sequential processing of each subband in the entropy module takes a relatively long decoding time, resulting in incon-venience for real-world applications. In this work, for lossy image compression, the wavelet-like transform is transplanted into the prevailing autoencoder structure to enhance the analysis and synthesis transform due to its excellent decomposition capability. The obtained subbands of different frequencies will undergo a hierarchical decorrelation architecture for subband fusion, also called cross fusing module. The specialized treatment will be applied to different subbands according to their spatial resolution to attain a more compact latent representation. In addition, the proposed solution features an architecture that decouples the arithmetic decoding process from the sample prediction process, which significantly reduces the decoding complexity. Experiments on the Kodak test set show that the proposed method achieves −3.04% BD-Rate compared to existing decoupled end-to-end structure in RGB Peak Signal-to-Noise Ratio (PSNR).

Subblock-based motion compensation (SMC) has shown re-markable effectiveness in handling non-uniform motions for inter video coding. However, existing strategies typically derive subblock motion rely solely on the inherited information from an already coded region, lacking content-adaptive optimization and potentially lead to deviations from the real motion. In this paper, we propose an overhead-free adaptive refinement scheme towards SMC, which leverages template matching (TM) to enhance the fidelity of subblock motion derivation for affine and subblock-based TMVP (SbTMVP) prediction. In particular, TM-based control point motion vector (CPMV) refinement is proposed to facilitate affine MC, where TM cost serves as a fidelity metric during the seeking of optimal translational affine motion offset. Additionally, the motion shift (MS) to locate SbTMVP is also refined by TM before specifying the subblock motion. The proposed method has been adopted into ECM-11.0. Experimental results on ECM-10.0 demonstrate overall 0.09% coding gains under random-access (RA) configuration, with negligible runtime increase.

Motion Compensated Temporal Filter (MCTF) has been repeatedly proven to be an effective pre-processing tool that improves the coding performance. The philosophy is that by smoothing with temporal filter, the noise of the to-be-coded image can be reduced, thereby shrinking the prediction residuals and improving the rate-distortion (RD) performance. While abundant efforts have been devoted to the design of the MCTF filter weights, how motion vector variance and texture complexity influence MCTF has been relatively under-explored. In this work, we propose an enhanced MCTF method (EMCTF) based on multi-hypothesis reference, motion vector variance, and texture complexity. We take initial steps towards the incorporation of motion vector variance and texture complexity in the filtering weights design. Motion compensation blocks based on multi-hypothesis reference, can be efficiently aggregated in an effort to obtain the final inference. The proposed method is implemented on the top of Versatile Video Encoder (VVenC). Experimental results show that for faster preset, fast preset, medium preset and slow preset, the proposed EMCTF achieves 0.85%, 0.91%, 0.85% and 0.82% Bjøntegaard delta rate (BD-rate) savings, respectively. Moreover, the EMCTF introduces little additional encoding complexity increase, facilitating its future applications in real-world scenarios.

The autoregressive model has been widely used in learning-based image compression due to its superior context modeling capability. However, its sequential processing nature also undermines the ability of decoding in parallel and hinders the deployment in real applications. In this paper, we propose a decoupled framework to resolve this issue. With the decoupled architecture, the entropy decoding process is independent of the latent sample reconstruction process. The entropy decoding process thus can be finished before the latent sample prediction process begins, which leads to significant decoding time savings by enabling the two processes to be conducted in parallel. To further reduce the decoding time, we introduce wavefront processing, where multiple rows can be processed simultaneously when reconstructing the latent samples. On top of that, we design a series of coding tools to improve the rate-distortion efficiency and reduce the decoding complexity. Device interoperability is also supported by the proposed solution, where the same bitstream can be successfully decoded on different CPU/GPU devices. Comprehensive experiments are conducted to validate the effectiveness of the proposed method. Using objective evaluation metrics required by JPEG AI Call for Proposals (CfP), the proposed method achieves an average of -29.6% BD-rate changes with 2.44 times faster decoding speed compared to VVC image coding. When compared to the commonly used benchmark learning-based methods, the proposed method achieves -30.5% BD-rate changes and 101 times faster decoding speed over cheng2020attn. The proposed solution has been proposed to JPEG AI and IEEE 1857.11 as a response to CfP and the core techniques have been adopted to build the verification model. The software and the instructions can be accessed at https://github.com/bytedance/BEE.

Cross-component prediction is an important intra-prediction tool in the modern video coders. Existing prediction methods to exploit cross-component correlation include cross-component linear model and its extension of multi-model linear model. These models are designed for camera captured content. For screen content coding, where videos exhibit different signal characteristics, a cross-component prediction model tailored to their characteristics is desirable. As a pioneering work, we propose a discrete-mapping based cross-component prediction model for screen content coding. Our model relies on the core observation that, screen content videos typically comprise of regions with a few distinct colors and luma value (almost always) uniquely conveys chroma value. Based on this, the proposed method learns a discrete-mapping function from available reconstructed luma-chroma pairs and uses this function to derive chroma prediction from the co-located luma samples. To achieve higher accuracy, a multi-filter approach is employed to derive co-located luma values. The proposed method achieves 2.61%, 3.51% and 3.92% Y, U and V bit-rate savings respectively over Enhanced Compression Model (ECM) 4.0, with negligible complexity, for text and graphics media under all-intra configuration.

In the versatile video coding (VVC) standard, cross-component prediction (CCP) is introduced to utilize the correlation between luma and chroma components. To further exploit the cross-component redundancy, a bundle of novel CCP methods with more sophisticated models are studied in enhanced compression model (ECM), which is a platform targeting at the next generation video coding standard, developed by JVET. This paper presents two methods, known as local-boosting CCP (LB-CCP) and non-local CCP (NL-CCP) to improve CCP with local and non-local information. With LB-CCP, prediction samples of CCP can be filtered with neighbouring samples. Besides, neighbouring template costs are calculated to determine the range of training samples, as well as the cross-component prediction method used in the chroma fusion mode. With NL-CCP, a CCP model can be derived with samples non-adjacent to the current block. As an equivalent but simpler implementation, the CCP model can be inherited from a CCP-coded neighbouring block as a spatial CCP candidate. Moreover, the CCP model of a previous CCP-coded block can be stored in a history-based table, which can be fetched and used by the current block as a history-based CCP candidate. A NL-CCP candidate list is built with the two kinds of candidates. The encoder can select the optimal candidate and send an index to the decoder. Experimental results show that LB-CCP together with NL-CCP can provide an average Bjontegaard delta rate (BD-rate) reduction of 0.27%, 2.31%, 2.44% on Y, Cb, Cr components, respectively, with a negligible change in the running time, compared with ECM-8.0 in all intra configurations. Both LB-CCP and NL-CCP have been adopted into ECM.

Video salient object detection (VSOD), as a fundamental computer vision problem, has been extensively discussed in the last decade. However, all existing works focus on addressing the VSOD problem in 2D scenarios. With the rapid development of VR devices, panoramic videos have been a promising alternative to 2D videos to provide immersive feelings of the real world. In this paper, we aim to tackle the video salient object detection problem for panoramic videos, with their corresponding ambisonic audios. A multimodal fusion module equipped with two pseudo-siamese audio-visual context fusion (ACF) blocks is proposed to effectively conduct audio-visual interaction. The ACF block equipped with spherical positional encoding enables the fusion in the 3D context to capture the spatial correspondence between pixels and sound sources from the equirectangular frames and ambisonic audios. Experimental results verify the effectiveness of our proposed components and demonstrate that our method achieves state-of-the-art performance on the ASOD60K dataset.

High-resolution (HR) images are usually downscaled to low-resolution (LR) ones for better display and afterward upscaled back to the original size to recover details. Recent work in image rescaling formulates downscaling and upscaling as a unified task and learns a bijective mapping between HR and LR via invertible networks. However, in real-world applications (e.g., social media), most images are compressed for transmission. Lossy compression will lead to irreversible information loss on LR images, hence damaging the inverse upscaling procedure and degrading the reconstruction accuracy. In this paper, we propose the Self-Asymmetric Invertible Network (SAIN) for compression-aware image rescaling. To tackle the distribution shift, we first develop an end-to-end asymmetric framework with two separate bijective mappings for high-quality and compressed LR images, respectively. Then, based on empirical analysis of this framework, we model the distribution of the lost information (including downscaling and compression) using isotropic Gaussian mixtures and propose the Enhanced Invertible Block to derive high-quality/compressed LR images in one forward pass. Besides, we design a set of losses to regularize the learned LR images and enhance the invertibility. Extensive experiments demonstrate the consistent improvements of SAIN across various image rescaling datasets in terms of both quantitative and qualitative evaluation under standard image compression formats (i.e., JPEG and WebP). Code is available at https://github.com/yang-jin-hai/SAIN.

We propose a practical downsampling compression scheme, as shown in Fig 1, to use a neural network as a pre-processing module for traditional codecs to improve compression performance. Specifically, we propose a neural network-based pre-processing method to improve video compression performance by downsampling high-resolution videos. Our method, called the rate-guided arbitrary rescaling network (RARN), uses a pre-trained entropy module [1] to estimate bitrate and guide the sampling process. We also use a transformer-based virtual codec (TVC) to simulate the performance of traditional codecs, using a swing-transformer-based invertible neural network to learn distortion from standard codecs, and the cyclic shift attention [2] in the network can approximate the prediction modes of HEVC. Our method, which is compatible with standard codecs (HEVC and VVC), performs well in various sampling ratios according to experimental results.

Point cloud compression has various purposes in different application scenarios, such as requiring visual fidelity in human vision tasks and pursuing semantic fidelity in machine vision tasks. To accommodate these diverse requirements, we propose a Human-Machine balanced point cloud geometry compression scheme (HM-PCGC) which considers the characteristics of various tasks. Our proposed scheme starts from a pre-trained, lightweight point cloud compression backbone and employs a Learned Semantic Mining module to aggregate multi-tasks features. By leveraging the aggregated features, HM-PCGC is able to retain the geometry and semantic properties of the point clouds during compression. To better balance between the signal distortion and semantic distortion, we integrated a multi-task learning mechanism during the training phase. Our approach is extensively evaluated and analyzed, and the results demonstrate its superiority over state-of-the-art traditional and deep learning based point cloud codecs for both signal reconstruction and machine vision tasks.

Bi-directional Optical Flow (BDOF), is a very effective tool, developed based on the optical flow concept, which assumes that the motion of an object is smooth and is in a straight line in a short period of time. BDOF as a prediction adjustment tool for each sample, is included in VVC SW, where it derives parameters for each 4×4 block and adjusts each of its samples individually. The reference SW which is being developed for beyond VVC activity, i.e. ECM, has 2 BDOFs: One as Motion Vector (MV) refinement tool for each 8 × 8 subblock, and another one as sample adjustment BDOF similar to VVC one, but deriving the parameters for each sample. In this paper, a High-Precision BDOF is introduced which consists of the following parts: Firstly, a more accurate BDOF formula is used to derive the parameters for BDOF MV refinement. Next, a position-dependent weight is used to increase the focus of the parameter derivation. Finally, higher granularity as small as 4×4 is used for each subblock. For BDOF sample adjustment, another position-dependent weighted sum is used. The proposed method is implemented on top of ECM-7.0. Experimental results show that the proposed method provides up to –0.92% bitrate saving for some sequences compared to the latest ECM SW under random access configuration. The method explained in this paper is adopted into ECM-9.0.

Learned image compression has exhibited promising compression performance, but variable bitrates over a wide range remain a challenge. State-of-the-art variable rate methods compromise the loss of model performance and require numerous additional parameters. In this paper, we present a Quantization-error-aware Variable Rate Framework (QVRF) that utilizes a univariate quantization regulator a to achieve wide-range variable rates within a single model. Specifically, QVRF defines a quantization regulator vector coupled with predefined Lagrange multipliers to control quantization error of all latent representation for discrete variable rates. Additionally, a reparameterization method makes QVRF compatible with round quantizer and integer entropy coding. Exhaustive experiments demonstrate that existing fixed-rate VAE-based methods equipped with QVRF can achieve wide-range continuous variable rates within a single model without significant performance degradation. Furthermore, QVRF outperforms contemporary variable-rate methods in rate-distortion performance with minimal additional parameters. The code is available at https://github.com/bytedance/QRAF.

Temporal motion information has been widely exploited in advanced video coding standards to facilitate the efficiency of inter coding. In prior-arts, temporal cues are derived from fixed locations in a single collocated frame without considering trajectory consistency. This paper proposes to enhance temporal motion derivation in two aspects. On one hand, two collocated frames are utilized to provide more abundant temporal motion information from both forward and backward directions. On the other hand, motion shifts to locate regular temporal motion vector prediction (TMVP) or sub-block based TMVP (SbTMVP) are adaptively determined from multiple locations according to template matching costs. Simulation results indicate the proposed methods can achieve 0.11% and 0.15% coding gains respectively for RA and LDB configurations on ECM-7.0, with negligible complexity increase. The proposed method has been adopted into ECM-8.0.

Sharpening is a widely adopted technique to improve video quality, which can effectively emphasize textures and alleviate blurring. However, increasing the sharpening level comes with a higher video bitrate, resulting in degraded Quality of Service (QoS). Furthermore, the video quality does not necessarily improve with increasing sharpening levels, leading to issues such as over-sharpening. Clearly, it is essential to figure out how to boost video quality with a proper sharpening level while also controlling bandwidth costs effectively. This paper thus proposes a novel Frequency-assisted Sharpening level Prediction model (FreqSP). We first label each video with the sharpening level correlating to the optimal bitrate and quality tradeoff as ground truth. Then taking uncompressed source videos as inputs, the proposed FreqSP leverages intricate CNN features and high-frequency components to estimate the optimal sharpening level. Extensive experiments demonstrate the effectiveness of our method.

To meet the growing demand for point cloud compression, MPEG is developing a point cloud compression standard called as G-PCC. In G-PCC, upsampled transform domain prediction (UTDP) is used to improve attribute coding performance. However, only the attributes in the previous level can be used to predict the attributes of transform sub-blocks in UTDP, which limits the efficiency of UTDP. To address this limitation, we propose a method called peer-UTDP to improve UTDP by using peer neighbors in this paper. With peer-UTDP, attributes of co-plane or co-line peer neighbors in the level same as that of the transform sub-block can be used as prediction in the upsampling process. Experimental results show that our method outperforms G-PCC with an average coding gain of -5.1%, -5.4%, -5.1% and -1.4% under C1 condition, and -5.1%, -5.6%, -5.6% and -1.7% under C2 condition for Y, Cb, Cr and reflectance, respectively. The proposed peer-UTDP has been adopted by G-PCC.

Video transcoding, which is a long-standing problem in video coding, can be used for a variety of applications including bit rate conversion, format conversion and video resolution conversion. In this paper, we present an effective transcoding solution from the popular previous generation video coding standard, High Efficiency Video Coding standard (HEVC), to Versatile Video Coding (VVC) which is the latest standard developed by JVET. VVC enjoys much better coding efficiency compared to its ancestors at the expense of higher encoding complexity. To facilitate the fast transcoding, we propose a low- complexity HEVC to VVC transcoding scheme. More specifically, the partition and motion information, which implies the intrinsic spatial and temporal characteristics of the video sequence, is inherited from HEVC for VVC encoding. The proposed scheme is tailored for HEVC to VVC conversion, and experimental results show the proposed method can achieve 31% encoding time saving with only 0.85%, 0.70% and 0.74% Bjøntegaard delta rate (BD- rate) loss on average for Y, Cb and Cr components, respectively, under random access configuration.

Rate control plays an essential role in real video applications due to the limitation of the bandwidth and storage budget. Supported by reference picture resampling (RPR) in Versatile Video Coding (VVC), which enables resolution changes without introducing an intra-coded frame, it is possible to adjust picture resolutions to adapt to the required bit-rate. However, few rate control algorithms with resolution changes have been explored for VVC. Therefore, we propose a$\lambda$-domain rate control algorithm with resolution changes in this paper. In the proposed rate control scheme, the coding resolution of one frame is determined based on the property of the frame. To make the rate control scheme adaptive to mixed resolutions, the rate control parameters of the full resolution and the low resolution are updated separately. Furthermore,$\lambda$and QP of the frames with different resolutions are unified to obtain appropriate coding parameters. Simulation results show that a BD-rate coding gain about 3.7% on average can be achieved with the reduction of computational complexity. Besides, negligible bitrate error can be observed at both sequence and frame level.

Intra block copy is an efficient coding tool for screen content video, in which the prediction is obtained from a reference block within the same picture indicated by a block vector. Since repetitive textures typically exist in the screen content video, the non-local information used in IBC can improve coding performance significantly. However, the local information thoroughly investigated in intra prediction has not been explored in IBC. In this paper, local-aware intra block copy (LA-IBC) is proposed to further improve the coding performance of IBC. LA-IBC is composed of three methods. First, combined IBC and intra prediction (IBC-CIIP) is proposed to fuse the two prediction signals generated by IBC and intra prediction. Second, IBC with geometry partitioning (IBC-GPM) is proposed, in which a block is divided into two sub-partitions geometrically and IBC and intra prediction are used to obtain their prediction signals, individually. Third, IBC with local illumination compensation (IBC-LIC) is proposed to explore the illumination variation between the local and non-local templates within the picture. LA-IBC is integrated in the latest exploration software of video coding (Enhanced Compression Model, ECM) published by JVET to evaluate its efficiency. The experimental results demonstrate that LA-IBC can achieve 1.7% and 0.88% Bjontegaard Delta rate (BD-rate) saving for screen content video compared to ECM-6.0 under all intra and random access configurations, respectively.

Multi-pass Decoder Side Motion Vector Refinement (DMVR) based on a Bilateral Matching (BM) concept is a very important tool being used in video codecs to improve the Motion Vector (MV) accuracy on the decoder side. In VVC, a single stage DMVR is applied on each 16x16 subblock to refine its MV. Exploring Coding Model (ECM), which is being developed for beyond VVC activity, adopts multi-pass DMVR. First, a BM approach is used to adjust the MV for each Prediction Unit (PU). Next, another BM approach is used to adjust the MVs for each 16x16 subblock. Finally, a Bi-directional Optical Flow (BDOF) based approach is used to adjust the MVs for each 8x8 subblock. In this paper, an iterative BDOF to improve multi-pass DMVR is proposed, where an additional pass of BDOF is added as the 4th stage of the multi-pass DMVR. To balance the gain and the complexity, the granularity of subblocks is adjusted adaptively in the third and forth stages of multi-pass DMVR. Finally, a regularization factor is added to avoid unnecessary over-tuning of BDOF adjustment. The proposed method is implemented on top of ECM-8.0. Experimental results show that the proposed method provides more than -1% bitrate saving for some sequences compared to ECM-8.0 under random access configurations. This method is adopted into ECM-10.0.

As a rapid development of neural-network-based machine learning algorithms, deep learning methods are being tentatively used in a much wider range than well-known artificial intelligence applications such as face recognition or auto-driving. Recently, deep learning models are investigated intensively to improve the compression efficiency for video coding, especially at the in-loop filtering stage. Although deep learning-based in-loop filtering methods in prior arts have already shown a remarkable potential capability in video coding, content propagation issue is still not well recognized and addressed yet. Content propagation is the fact that contents of reference frames are propagated to frames referring to them, which typically leads to over-filtering issues. In this article, we develop an iteratively trained deep in-loop filter with adaptive model selection (iDAM) to address the content propagation issue. First, we propose an iterative training scheme, which enables the network to gradually take into account the impacts of content propagation. Second, we propose a filter selection mechanism, i.e., allowing a block to select from a set of candidate filters with different filtering strengths. Besides, we propose a novel approach to design a conditional in-loop filtering method that can deal with multiple quality levels with a single model and serve the functionality of filter selection by modifying the input parameters. Extensive experiments on top of the latest video coding standard (Versatile Video Coding, VVC) have been conducted to evaluate the proposed techniques. Compared with VTM-11.0, our scheme achieves a new state-of-the-art, leading to {7.91%, 20.25%, 20.44%}, {11.64%, 26.40%, 26.50%}, and {10.97%, 26.63%, 26.77%} BD-rate reductions on average for {Y, Cb, Cr} under all-intra, random-access, and low-delay configurations, respectively. As far as we know, our proposed iDAM scheme provides the highest coding performance compared to all existing solutions. In addition, the syntax elements of the proposed scheme were adopted at the 76th meeting of Audio Video coding Standard (AVS) held this year.

Screen content has become one of the prominent mediums in the increasingly connected world. With the prevalence of remote collaboration and communication such as virtual conferences and online education, recent years have witnessed a dramatic increase in the data volume of the screen content. Screen content compression serves as the fundamental technology in fostering the storage, transmission, and exhibition of screen content. In this article, we target the super-resolution of the compressed screen content images, intending to tackle the real-world challenge problems. A dataset is proposed for the super-resolution of the screen contents contaminated with different compression distortion levels. Subsequently, we introduce the principle of the multi-hypothesis into the super resolution and propose a new paradigm for the restoration of the compressed screen content images. The luminance and sharpness similarity metric is adopted in the network learning to better adapt to the screen content characteristic and ensure perceptual fidelity. Experimental results verify the superiority and effectiveness of the proposed method.

Resampling-based coding, i.e. down-sampling before encoding and up-sampling after decoding, has been recognized to be an effective tool for compressing high-resolution videos at low bitrates. The newest video coding standard, Versatile Video Coding (VVC), supports resampling-based coding via a mechanism named Reference Picture Resampling (RPR), where the spatial resolution can be changed without inserting an intra frame. Intuitively, it is not wise to utilize a single resolution throughout the whole video, because frames with different contents may prefer different coding resolutions. In this paper, we propose a smart reference picture resampling approach, namely smart-RPR, where the coding-resolution of a frame is determined based on the property of the frame without multiple-pass encoding. Specifically, we first down- and up-sample a frame without considering compression and compare the up-sampled frame with the original frame to obtain the resampling distortion, which is then compared with a threshold to decide whether to code the frame in a resampling way. Then, we build up an exponential model to approximate the optimal threshold. In addition, we also study how to derive the coding parameters of the down-sampled frame to achieve better performance. Simulation results on the VTM-12.0 show that the proposed method could achieve 2.72%, 5.29%, and 10.82% BD-rate reductions for Y, Cb, and Cr components, respectively, with lower encoding and decoding complexity.

Existing deep learning based approaches for coding artifact removal mainly focus on the relationship between reconstruction and original signals. Although the CNN-based in-loop filtering reveals significant potential, the filtering procedure is separated from the rate distortion optimization (RDO) in the block-based hybrid framework. In this paper, we propose a joint rate distortion optimization scheme [1] with CNN-based in-loop filter for the Versatile Video Coding (VVC), with the goal of excavating the potential of in-loop filtering. In particular, a joint rate distortion optimization is presented, to consider the CNN-based in-loop filtering in the coding unit. More specifically, the CNN-based in-loop filter is involved in the partitioning mode selection, and meanwhile the prediction mode determination process remains unchanged. The proposed joint optimization is only applied to the CUs, whose height and width are no larger than 64. To keep the encoding complexity as low as possible, a light-weight version of CNN-based model is provided to facilitate the encoding optimization. As shown in Table 1, the proposed method brings 9.61% and 12.81% BD-Rate gains for luma component under all intra and random access configurations, respectively, on VTM-11.0-nnvc platform.

Geometric partitioning mode (GPM) is employed in the latest video coding standard known as versatile video coding (VVC). GPM provides a capability to partition an inter-coded block into two geometric regions with different motion vectors (MVs). However, MVs of the two partitions of a GPM coding unit (CU) are restricted to be selected from a MV candidate list, lacking the flexibility to refine the MVs. This paper presents a method named geometric partitioning with motion vector difference (GMVD). With the proposed GMVD mode, the MV of each GPM partition, partitioned by GPM, is generated as a sum of a base MV selected from a MV candidate list and a motion vector difference, signaled in a similar way as merge motion vector difference (MMVD) in VVC. The presented GMVD has been adopted into the enhanced compression model (ECM), developed by JVET to explore novel coding tools beyond VVC. Experimental results show that the presented GMVD provides an additional average luma BD-rate gain of 0.27% and 0.42% in RA and LB configurations over ECM-1.0 under the JVET CTC, with decent trade-off between compression efficiency and encoder complexity. Moreover, the decoder complexity of GMVD is negligible.

Existing end-to-end image compression mainly concentrates on the coding of natural scene images. However, few works have been dedicated to the end-to-end compression of screen contents. In this paper, we propose an end-to-end compression scheme for screen content images inspired by the ideology of transform skip, with the goal of improving the compression performance for screen content images. In particular, the proposed model takes full consideration of the characteristics of screen contents and involves transform skip branches to the analyses and synthesis process. The transform skip branch equips with coarse feature extraction and reconstruction. As such, the visual signals could be more briefly interpreted at the encoder-side and recovered at the decoder-side. Experimental results show that the proposed method outperforms the existing hyperprior-based model for screen content compression, achieving 10.16% BD-Rate savings in high bit-rate coding scenario and 5.38% BD-Rate savings in low bit-rate coding scenario.

Efficient 3D point cloud compression plays critical role in immersive multimedia presentation and autonomous driving. The temporal redundancy between consecutive point cloud frames is obvious, but the exploration of inter-prediction is insufficient in the geometry-based point cloud compression (G-PCC) framework. In inter exploration model (Inter-EM) of G-PCC, the reference information can only come from one previous frame and a consistent quantization parameter (QP) value is used for all frames. To perform a more efficient inter-prediction for geometry and attribute, a bidirectional inter-prediction (bi-prediction) scheme is proposed for G-PCC based on Inter-EM. With the bi-prediction scheme, the reference information can come from two reference frames. For attribute compression, neighboring search is started from one search center derived by a Morton code distance and a two-threshold method is applied to constrain inter-prediction. Meanwhile, the dependencies between frames are designed according to a hierarchical group of frames (GOF) structure with corresponding hierarchical QP values. Experimental results demonstrate that our method outperforms G-PCC and Inter-EM for both lossless and lossy compression. More specifically, the average coding gain over G-PCC is 14.1% on attribute and 11.8% on geometry under lossy condition. The designs on attribute compression have been adopted to the latest Inter-EM.

The learning-based end-to-end video compression exhibits a fast development with continuous improvements. In previous works, a key frame in random-access scenarios is typically compressed by an image compressor, and the remaining frames are reconstructed by interpolations. But solely using image compression fails to leverage the temporal correlations, which is critical to achieve a substantial gain in video compression. To exploit temporal correlations among key frames, we introduce a learning-based end-to-end spatial-temporal adaptive (e2e-STA) compression solution to offer flexible options for the key frames. First of all, we design an extrapolation-based key frame compression scheme. Given a key frame, e2e-STA can switch between an image compressor and an extrapolative compressor. A key frame is thereby able to select the optimal solution adaptively according to the rate-distortion optimization criteria and the optimal selection is sent to the decoder. The proposed approach is optimized end-to-end with all the networks. The experimental results validate the effectiveness of the proposed mechanism. The proposed method outperforms existing learning-based video compression methods by a noticeable margin and provides promising performance compared to traditional benchmark video codecs in terms of PSNR and MS-SSIM.

High-resolution (HR) images are typically downscaled by the bicubic method to fit the different resolutions of target devices and save the transmission bandwidth. However, the bicubic downsampling may cause the loss of high-frequency information (HFI), raising great challenges for recovering the original high-resolution images. Inspired by image steganography, wherein certain information needs to be written into an image as steganographic information and recovered later, in this work, we propose an invertible steganography rescaling network (ISRN) for single image rescaling. The proposed ISRN can preserve HFI that would otherwise be lost as stegano-graphic information in low-resolution (LR) images. Moreover, we develop an attentive steganography separate block (ASSB) to decompose the steganographic information into a case-aware part that can be embedded into the downscaled image and a case-agnostic part that can be captured using a prior distribution. Extensive experiments demonstrate that the proposed ISRN outperforms prior arts in terms of both quantitative and qualitative evaluations.

Versatile Video Coding (VVC) is capable of achieving approximately 25% bitrate reduction compared with High Efficiency Video Coding (HEVC) at the same objective quality under all intra configuration. Meanwhile, VVC sacrifices the encoding complexity by 26 times the encoding time of HEVC, which makes it impractical to use VVC without optimization. In view of the fact that most of complexity is due to the novel block partitioning structure in VVC, this paper focuses on predicting the partitioning structure with convolutional neural networks. Specifically, we first formulate the partitioning prediction problem into two alternatives: bottom-up-based where the split type of subblock boundaries is first predicted and then used to infer the partitioning structure of each coding unit, top-down-based where the probability distribution in the ensemble partitioning space is first derived and then used to decide the partitioning structure of each coding unit. Then, we address both formulations using convolutional neural networks. When evaluating on top of VTM7.0, proposed schemes perform favorably against state-of-the-art works. In particular, the bottom-up-based method can bring on average 52.3% encoding complexity reduction with 0.46% BD-rate increase while the top-down-based method could provide on average 39.5% encoding complexity reduction with 0.28% BD-rate increase. In addition, the proposed schemes can offer scalability in terms of coding efficiency and complexity trade-off.

Merge with motion vector difference (MMVD) is a simple yet efficient coding tool adopted in VVC. However, the efficiency of MMVD drops a lot in the reference software beyond VVC, known as ECM, this paper proposes a template matching (TM) based extension for MMVD (TME-MMVD) for inter prediction. Firstly, extra refinement positions are introduced along diagonal angles. Secondly, based on a template matching cost calculated as a distortion between samples of a template and their reference samples, the refinement positions are reordered. Finally, the preferable ones with the lowest costs are selected as available positions and consequently for index coding. This method is also extended to the affine MMVD mode. The proposed method is implemented on top of the ECM2. Experimental results show that the proposed method provides 0.69% and 0.64% bitrate saving compared to latest VVC SW and 0.24% and 0.31% bitrate saving compared to ECM2 under the JVET common test conditions for random access and low delay configurations, respectively. This tool is adopted into the current ECM SW.

The optimized bit allocation among frames has been intensively explored and improved the compression performance significantly in conventional video coding. However, the optimized bit allocation is still in its infant stage for learning-based video coding. Most existing learning-based video compression methods either use uniform bit allocation or empirically determined bit allocation weights among frames. In this paper, we develop an optimized bit allocation scheme for learning-based end-to-end video compression. In particular, we realize a hierarchical quality-control mechanism based on the importance of different frames under random-access scenarios. Considering the varying importance of frames on different temporal layers, we propose an efficient yet simple scheme, in which a set of optimized bit allocation weights are introduced to the rate-distortion (R-D) loss function. Experimental results demonstrate the effectiveness of the proposed scheme. In addition, the proposed scheme can be easily applied to most existing learning-based video compression frameworks under random-access scenarios.

Block partitioning methods play a critical role in hybrid video coding standards such as HEVC and VVC. A quaternary tree (QT) / binary tree (BT) / ternary tree (TT) partitioning structure has been adopted in VVC, which improves the coding performance significantly compared to HEVC. In this paper, we explore two advanced block partitioning methods beyond VVC. Different from block symmetric partitioning methods such as QT and BT, the two proposed advanced partitioning methods are unsymmetric, namely unsymmetric quaternary tree (UQT) and unsymmetric binary tree (UBT). UQT and UBT split a block into four and two partitions unsymmetrically, respectively. Several modifications are proposed to make coding tools compatible with a block holding a width or a height not in a form of 2$^{\mathrm{n}}$. Besides, fast algorithms are presented to accelerate the encoder. Experimental results show that the proposed method provides an average BD-rate saving about 1.5% with a negligible decoding time change, compared with VVC in random access configurations under the JVET common test conditions.

Efficient construction of merge candidate list is a critical issue in terms of the advanced inter coding technologies. Existing merge candidate list construction methods are typically realized by traversing the potential candidates in a predefined order, without considering different qualities of the potential candidates. In this paper, we propose a quality-aware merge candidate list construction method based on template matching (TM) cost sorting, termed as QAMC. Instead of checking and appending potential candidates based on a predefined traversing order, QAMC is proposed to select and rank potential merge candidates according to the quality, where the quality is measured by the distortion between a reconstructed template region and its reference region. Simulation results on ECM-2.0 demonstrate that QAMC provides 0.27% and 0.32% coding gains respectively for RA and LDB configurations, with reasonable complexity increase.

Recently years have witnessed an increasing interest in the neural network-based video coding, including end2end and hybrid schemes. To foster the research in this emerging field and provide a benchmark, we propose the Grand Challenge on Neural Network-based Video Coding (GC-NNVC) in the ISCAS 2022. In this paper, we review the grand challenge results and share interesting observations resulted from the challenge. This challenge includes two tracks, the hybrid-based and end2endbased. Different neural network-based coding schemes are evaluated according to their coding efficiency and innovations in methodologies in each track. To facilitate a solid comparison with conventional video coding techniques, the decoded sequences are evaluated in YUV 4:2:0 color format and PSNR is adopted as the distortion metric. Compared with the reference software of HEVC, submissions in hybrid track typically exhibit promising BD-rate reduction while submissions from the end2end track perform worse.

Current video coding schemes such as VVC and ECM employ separate palette coding for luma and chroma components under dual-tree structure, ignoring cross-component correlations. Although there are linear and multi-modal linear models to capture cross-component correlations, such models are not tailored for screen content sequences. To address this, we propose a novel cross-component prediction model for screen content sequences. The proposed method builds on the core observation that, regions of screen content sequences comprise of few distinct colors and luma value (almost always) uniquely conveys chroma values. In the light of this observation, the proposed method derives chroma prediction based on a discrete mapping function between luma and chroma values. Specifically, the method simply remembers the reconstructed luma values and their corresponding chroma values in a look-up table and employs this look-up table for cross-component prediction for the current chroma block. To achieve higher accuracy, a multi-filter approach is employed to derive co-located luma values. For an example configuration, the proposed method achieves 1.37%, 1.08% and 1.68% Y, U and V bit-rate savings respectively over ECM 3.1, for text and graphics media under all-intra configuration, demonstrating its efficacy.

This paper presents a self-aware filter estimation cross-component linear model(SAFE-CCLM) intra-prediction method, in which a set of luma down-sampling filter candidates are predefined, and the selected down-sampling filter is adaptively determined at both encoder and decoder sides in a consistent way, without signalling a filter index. To derive the selected filter of a chroma block, the luma neighbouring samples are tried to be down sampled by each filter candidate. With the down-sampled luma neighbouring samples, a linear model is derived for each filter candidate, and prediction values of chroma neighbouring samples are calculated with the model. Then a sum of absolute difference (SAD) cost between prediction values and reconstruction values of chroma neighbouring samples is obtained. Finally, the filter candidate with the lowest SAD cost is determined to be the selected down-sampling filter, used to generate prediction values of the current chroma block. Experimental results show that SAFE-CCLM provides an average bitrate saving of 0.80% on the luma component for All Intra configurations under the JVET common test conditions, with negligible runtime changes.

In video coding, motion information consisting of motion vectors and reference index is typically involved in motion compensation. Motion list is widely used to efficiently compress the motion information, in which a motion index indicating the motion information is signaled. And the compression efficiency can be improved by template matching based motion list reordering. Besides, the motion information is further refined before being used in motion compensation by motion refinement process such as decoder-side motion vector refinement and template matching. However, the original motion information of the motion list rather than the refined motion information is used in the motion list reordering, which limits the coding performance. Therefore, an enhanced motion list reordering (EMLR) approach is proposed in this paper, in which the refined motion information is used in the motion list reordering. To derive the refined motion information, a dedicated motion refinement with a simplified version of motion refinement process is proposed. Furthermore, a simplified version of EMLR with two fast algorithms (EMLR-S) is proposed. Experimental results demonstrate that EMLR can achieve 0.19% BD-rate saving on average, and EMLR-S can achieve 0.1 % BD-rate saving with negligible coding complexity change compared to ECM-4.0 under random access configuration.

In this paper, we propose an end-to-end image compression framework, which cooperates with the swin-transformer modules to capture the localized and non-localized similarities in image compression. In particular, the swin-transformer modules are deployed in the analysis and synthesis stages, interleaving with convolution layers. The transformer layers are expected to perceive more flexible receptive fields, such that the spatially localized and non-localized redundancies could be more effectively eliminated. The proposed method reveals the excellent capability of signal conjunction and prediction, leading to the improvement of the rate and distortion performance. Experimental results show that the proposed method is superior to the existing methods on both natural scene and screen content images, where 22.46% BD-Rate savings are achieved when compared with the BPG. Over 30% BD-Rate gains could be observed with screen content images when compared with the classical hyper-prior end-to-end coding method.

The rapid development of intelligent tasks, e.g., segmentation, detection, and classification, etc, has brought an urgent need for semantic compression, which aims to reduce the compression cost while maintaining the original semantic information. However, it is impractical to directly integrate the semantic metric into the traditional codecs since they cannot be optimized in an end-to-end manner. To solve this problem, some pioneering works have applied reinforcement learning to implement image-wise semantic compression. Nevertheless, the video semantic compression has not been explored since its complex reference architectures and compression modes. In this paper, we take a step forward to video semantic compression and propose the Hierarchical Reinforcement Learning based task-driven Video Semantic Coding, named as HRLVSC. Specifically, to simplify the complex mode decision of video semantic coding, we divided the action space into frame-level and CTU-level spaces in a hierarchical manner, and then explore the best mode selection for them progressively with the cooperation of frame-level and CTU-level agents. Moreover, since the modes of video semantic coding will exponentially increase with the number of frames in a Group of Pictures (GOP), we carefully investigate the effects of different mode selections for video semantic coding, and design a simple but effective mode simplification strategy for it. We have validated our HRLVSC on video segmentation task with HEVC reference software HM16.19. Extensive experimental results demonstrated that our HRLVSC can achieve over 39% BD-rate saving for video semantic coding under the Low Delay P configuration.

Interpolation filters for motion compensation play an important role in video coding. Until recently, interpolation filters for luma motion compensation (MC) draw most of the attention while interpolation filters for chroma motion compen-sation are less noticeable. This paper comprehensively analyzes different types of interpolation filters and presents 6-tap DCT based chroma interpolation filters for chroma MC. The proposed interpolation filters are designed considering multiple factors, comprising the characteristics of the chroma component, the frequency response of interpolation filter and the expenses of computational complexity. The experimental results show the proposed method can achieve 0.05%, 1.23% and 1.25% BD-rate reductions on average for Y, Cb and Cr components, respectively, with only 2% decoding time increasing under Random Access configuration, compared with ECM-4.0, which is a reference software developed by JVET, targeting at future video coding technologies beyond VVC. Owing to the good trade-off between coding performance and complexity, the proposed method has been adopted in ECM.

In VVC, affine motion compensation (AMC) is a powerful coding tool to address non-translational motion, while history-based motion vector prediction (HMVP) is an efficient approach to compress motion vectors. However, HMVP was designed for translational motion vectors, without considering control point motion vectors (CPMV) for AMC. This paper presents a method of history-parameter-based affine model inheritance (HAMI), to utilize history information to represent CPMV more efficiently. With HAMI, affine parameters of previously affine-coded block are stored in a first history-parameter table (HPT). New affine-merge, affine motion vector prediction candidates and regular-merge candidates can be constructed with affine parameters fetched from the first HPT and base MVs fetched from neighbouring blocks in a “base-parameter-decoupled” way. New affine merge candidates can also be generated in a “base-parameter-coupled” way from a second HPT, which stores base MV information together with corresponding affine parameters. Besides, pair-wised affine merge candidates are generated by two existing affine merge candidates. Experimental results show that HAMI provides an average BD-rate saving about 0.34 % with a negligible change on the running time, compared with ECM-3.1 in random access configurations. HAMI has been adopted into ECM.

In Versatile Video Coding (VVC), reference picture resampling (RPR) is supported to change resolution without inserting an instantaneous decoder refresh (IDR) picture or intra random access picture (IRAP). The introduction of RPR enables VVC to adapt to unstable network conditions, which benefits real-time application scenarios a lot. However, the existing bitrate leap and the low quality of the upsampled frames are the bottlenecks of promoting its application. In this paper, we propose an efficient framework to alleviate bitrate surges and further improve the coding performance. The contributions of this paper mainly include the following three aspects. Firstly, it is proved theoretically and statistically that the temporal long-distance high-resolution (HR) reference pictures can provide better prediction than temporal short-distance low-resolution (LR) reference pictures when the temporal distance between them is less than a threshold. Secondly, motivated by the theoretical analysis, existing HR reference pictures in the reference pictures list (RPL) are used to replace LR reference pictures and an adaptive decision on the number of HR reference pictures kept in RPL is applied. Thirdly, an adaptive upsampling filter (AUF) is proposed to restore pictures with the original resolution more accurately. Simulation results show 15.09% BD-rate savings on average can be achieved for the 2:1 scaling ratio under RPR configurations. For the 1.5:1 scaling ratio, the BD-rate saving is 8.14%. Besides, the bitrate surge in RPR is reduced greatly. About 25% bits reductions on average at the leaping point can be observed and the bits decrease rate is up to 50% for sequences with little and slow motion.

Transform coding removes redundancy by de-correlating the residual data, which plays a crucial role in video coding. Since different transform cores can adapt to different video content and residual data, multiple cores transform has been proposed to improve the coding performance. However, the overhead for representing the transform type is inevitable. This paper intensively studies the statistical characteristics of transform coefficients. Theoretical analysis shows that the implicit method in the Implicitly Selected Transform (IST) is superior to the explicit signalling method. As such, we propose the parity adjustment scheme which seamlessly cooperates with the rate-distortion optimization quantization for IST. Furthermore, we propose two combination methods, size-based and number-based, to optimize IST. Moreover, a restriction region of IST is applied to reduce the complexity. Experimental results on HPM-6.0, which is the reference software of the AVS3 video coding standard, show that our proposed method can achieve 1.76% and 0.76% BD-Rate savings on average under AI and RA configurations, respectively, along with negligible decoding time variations. The comparison results between the proposed method and explicit signalling method illustrate that the proposed method can achieve better coding gain than the latter. Our method has been adopted in AVS3.

Versatile Video Coding (VVC) achieves promising compression performances compared with High Efficiency Video Coding (HEVC) while sacrificing the encoding speed. This paper focuses on predicting the partitioning structures with convolutional neural networks to speedup the VVC encoder. Specifically, we formulate the partitioning prediction problem into two alternatives: implicit partitioning prediction based on the split type of subblock boundaries and explicit partitioning prediction from the ensemble partitioning space. Then, we address both formulations using convolutional neural networks.

In this paper, we concentrate on the super-resolution (SR) of compressed screen content video, in an effort to address the real-world challenges by considering the underlying characteristics of screen content. Firstly, we propose a new dataset for the SR of screen content video with different distortion levels. Meanwhile, we design an efficient SR structure that could capture the characteristics of compressed screen content video and manipulate the inner-connections in consecutive compressed low-resolution frames, facilitating the high-quality recovery of the high-resolution counter-part. Moreover, we design a new loss function for network training to better remedy the compression distortion and perceptual distortion. Experimental results demonstrate the effectiveness and superiority of the proposed method.

The quad-tree plus binary-tree (QTBT) partition structures was adopted into the versatile video coding (VVC) standard. In the QTBT partition structure, a dual tree structure can be applied on intra slices where the partitioning structures for luminance and chrominance components are separate. The dual tree structure is efficient for camera captured videos. However, it may be not the case for screen content videos. Therefore, adaptive dual tree structure is proposed in this paper wherein the coding structure of each coding tree unit is switched between separate and joint coding structure to adapt the textures adaptively. The proposed scheme is implemented into the reference software of the VVC standard-VTM. Simulation results report that up to 4.5% BD bitrate saving can achieve on the typical screen content videos when compared to VTM5.

In the emerging Versatile Video Coding (VVC) standard, there are three in-loop filters known as deblocking, sample adaptive offset (SAO), and adaptive loop filter (ALF) for suppressing compression artifacts as well as reducing distortion. However, those handcrafted filters are insufficient to deal with the complicated compression artifacts. Deep - filtering has demonstrated overwhelming successes in the field of image restoration. In this paper, we propose a convolutional neural network-based filter for enhancing the quality of VVC intra coded frames. The proposed filter takes auxiliary information including partitioning and prediction information as input. For chroma, auxiliary information further includes luma samples. Regarding the training, we find that data augmentation and loss function selection are not trivial. We conduct extensive experiments to evaluate the effectiveness of each design. When testing on top of VTM-10.0 under all intra configuration, the proposed filter achieves state-of-the-art performance, i.e. on average 7.57%, 13.18%, and 12.50% BD-rate reductions for Y, Cb, and Cr, respectively. The proposed filter ranks first among neural network-based in-loop filters at the 20thmeeting of the Joint Video Exploration Team (JVET).

This paper presents a decoder derived cross-component linear model (DD-CCLM) intra-prediction method, in which one or more linear models can be used to exploit the similarities between luma and chroma sample values, and the number of linear models used for a specific coding unit is adaptively determined at both encoder and decoder sides in a consistent way, without signalling a syntax element. The neighbouring samples are classified into two or three groups based on a K-means algorithm. Moreover, DDCCLM can be combined with normal intra-prediction modes such as DM mode. The proposed method can be well incorporated with the state-of-the-art CCLM intra-prediction in the Versatile Video Coding standard. Experimental results show that the proposed method provides an overall average bitrate saving of 0.52% for All Intra configurations under the JVET common test conditions, with negligible runtime change. On sequences with rich chroma information, the coding gain is up to 2.07%.

Reference picture resampling (RPR) is an important feature for real-time communication in Versatile Video Coding, which supports adaptive resolution change without introducing an instantaneous decoder refresh (IDR) or intra random access picture (IRAP). With RPR, considerable coding gain is achieved while bitrate leap still exists. In this paper, an enhanced reference structure for RPR is proposed. First, the high-resolution (HR) references are used to replace the low-resolution (LR) references to provide more accurate prediction. Furthermore, an adaptive decision on the number of HR references kept in RPL is applied to achieve more performance, which utilizes the history information in the current period. Simulation results show about 9% and 5% BD-rate savings on average can be achieved for scaling ratio of 2:1 and 1.5:1 compared to VTM-10.2 under RPR configurations. For sequences with slight motions, the coding gain is up to 20%. Besides, the bitrate explosion in RPR is reduced generally, where over 50% reductions of bitrates at explosion point is observed.

Versatile Video Coding (VVC) is the latest video coding standard, finalized in July 2020. Dependent random access point (DRAP) is supported in HEVC and VVC, through a supplemental enhancement information (SEI) message, for improved coding efficiency in the case of random access. A DRAP picture is an inter-coded picture that can only refer to the most recent intra random access point (IRAP) picture earlier in decoding order. A DRAP picture can be used as a random access point in the bitstream given that the associated IRAP picture is available. Although the DRAP picture increases coding efficiency for random access, the performance is limited since only the associated IRAP picture is allowed to be used as the reference picture for DRAP pictures. In this paper, an extended dependent random access point (EDRAP) picture is proposed to further improve the coding efficiency for random access. Specifically, a particular EDRAP picture only relies on some pictures in a set of pictures consisting of the associated IRAP picture and certain EDRAP pictures between the associated IRAP picture and the particular EDRAP picture in decoding order. EDRAP pictures can be used as random access points as long as the few dependent IRAP or EDRAP pictures are provided. Simulation results demonstrate that the proposed method can achieve 10.2% Bj$\phi$ntegaard-Delta rate saving on average compared to VTM-11.0 under random access configuration. It is noted that the support of EDRAP for VVC, through another SEI message, was adopted by the JVET at its 21stmeeting in January 2021.

AVS3 is an emerging video coding standard, and screen content coding is a very important feature of AVS3. This paper presents a method of Implicit-Selected Transform Skip (ISTS) to further improve the screen content coding performance. With ISTS, transform skip mode is introduced as an optional substitution to transform-coding on residual signals of blocks with intra-prediction. The indication of whether to apply transform skip is hidden into the Parity of the Number of Non-zero Coefficients (PNNC) of a residual block, instead of being signaled to the decoder. Moreover, the coefficients of an intra-coded block are reordered to make the coefficients more compact. Experimental results show that the proposed method can achieve 12.04%, 8.15% and 10.19% BD-rate savings on average under All Intra (AI), Low Delay (LD) and Random Access (RA) configurations, respectively, with the encoding time reduced by 5% to 8%. ISTS has been adopted into AVS3.

In the emerging next-generation video coding standard AVS3, interweaved prediction was proposed to further improve affine motion compensation (AMC). With interweaved prediction, a coding block with AMC mode is divided into subblocks with two different dividing patterns to generate two auxiliary predictions, and the final prediction is calculated as a weighted-sum of the two auxiliary predictions. Although interweaved prediction can provide a significant coding gain, it is not very friendly to the hardware design. In this paper, we present a hardware friendly interweaved prediction method, in which the off-chip memory bandwidth is reduced by limiting the reference sample number per pixel and the on-chip cache bandwidth is decreased by sharing reference samples between subblocks in pattern0 and pattern1 of interweaved prediction. Simulation results show that the proposed methods can achieve 0.47% BD-rate saving on average with the Random Access (RA) configurations while reducing 52.3% off-chip memory bandwidth and 62% on-chip memory bandwidth.

Down-sampling followed by an up-sampling is a well-known strategy to compress high-resolution pictures given a limited bandwidth in image as well as video coding. Recently, inspired by the latest advances of image super resolution (SR) technologies using convolutional neural network (CNN), CNN-based SR has been explored for resampling-based coding. However, the side information generated during the compression process is not utilized efficiently by the SR network in prior arts. In this paper, we propose a CNN-based SR method for video coding, where more side information is leveraged as a supplement to reconstruction samples. Specifically, we introduce prediction samples to be the auxiliary information, as it can provide the texture and directional information about the original picture. Considering the different characteristics, we design different networks for the luma and chroma components. When designing the chroma up-sampling CNN, the luma reconstruction is used as the auxiliary information of the chroma network, which exploits the cross-component correlation. Experimental results show that the proposed method achieves 11.07% BD-rate savings in all-intra configuration compared with VTM-11.0. Further experiments validate the effectiveness of using luma information to aid the chroma up-sampling process.

Intra prediction is typically used to exploit the spatial redundancy in video coding. In the latest video coding standard Versatile Video Coding (VVC), 67 intra prediction modes are adopted in intra prediction. The encoder selects the best one from 67 modes and signals it to the decoder. Bits consuming of signaling the selected mode may limit the coding efficiency. To reduce the overhead of signaling the intra prediction mode, a probability-based decoder-side intra mode derivation (P-DIMD) is proposed in this paper. Specifically, an intra prediction mode candidate set is constructed based on the probabilities of intra prediction modes. The probability of an intra prediction mode is mainly estimated in two ways. First, the textures are typically continuous within a local region and intra prediction modes of neighboring blocks are similar to each other. Second, some intra prediction modes are preferable to be used than others. For each intra prediction mode in the constructed candidate set, intra prediction is processed on a template to calculate a cost. The intra prediction mode with the minimum cost is determined as the optimal mode and used in the intra prediction of the current block. Experimental results demonstrate that P-DIMD can achieve 0.56% BD-rate saving on average compared to VTM-11.0 under all intra configuration.

This paper describes an adaptive self-guided loop filter for video coding. The filter acts as a loop filter to reduce the artifacts caused by quantization. The proposed filter is applied on the reconstruction samples of a Transform Unit (TU). The unfiltered input block is regarded as a guidance for self-guided edge-preserving smooth filtering. The filtering strength for each sample inside the input block is decided according to the TU size, corresponding QP, and statistical information generated from a designed filtering window adaptively. The proposed filter could be performed as a prediction-loop-filter in which the filtered samples can be referenced by intra prediction. Alternatively, the proposed filter can be moved to the post-loop-filter stage that after the deblocking filter for a low decoding dependency. The proposed method has been implemented and tested according to the common test conditions in Versatile Video Coding (VVC) test model version 11.0 (VTM 11.0). The post-loop-filter version can achieve 0.46% and 0.41% BD-rate reduction in All Intra and Random-Access configurations respectively.

This paper overviews the motion vector coding and block merging techniques in the Versatile Video Coding (VVC) standard developed by the Joint Video Experts Team (JVET). In general, inter-prediction techniques in VVC can be classified into two major groups: “whole block-based inter prediction” and “subblock-based inter prediction”. In this paper, we focus on techniques for whole block-based inter prediction. As in its predecessor, High Efficiency Video Coding (HEVC), whole block-based inter prediction in VVC is represented by adaptive motion vector prediction (AMVP) mode or merge mode. Newly introduced features purely for AMVP mode include symmetric motion vector difference and adaptive motion vector resolution. The features purely for merge mode include pairwise average merge, merge with motion vector difference, combined inter-intra prediction and geometric partitioning mode. Coding tools such as history-based motion vector prediction and bidirectional prediction with coding unit weights can be applied on both AMVP mode and merge mode. This paper discusses the design principles and the implementation of the new inter-prediction methods. Using objective metrics, simulation results show that the methods overviewed in the paper can jointly achieve 6.2% and 4.7% BD-rate savings on average with the random access and low-delay configurations, respectively. Significant subjective picture quality improvements of some tools are also reported when comparing the resulting pictures at same bitrates.

Cross-component linear model (CCLM) prediction has been repeatedly proven to be effective in reducing the inter-channel redundancies in video compression. Essentially speaking, the linear model is identically trained by employing accessible luma and chroma reference samples at both encoder and decoder, elevating the level of operational complexity due to the least square regression or max-min based model parameter derivation. In this paper, we investigate the capability of the linear model in the context of sub-sampled based cross-component correlation mining, as a means of significantly releasing the operation burden and facilitating the hardware and software design for both encoder and decoder. In particular, the sub-sampling ratios and positions are elaborately designed by exploiting the spatial correlation and the inter-channel correlation. Extensive experiments verify that the proposed method is characterized by its simplicity in operation and robustness in terms of rate-distortion performance, leading to the adoption by Versatile Video Coding (VVC) standard and the third generation of Audio Video Coding Standard (AVS3).

The forthcoming Versatile Video Coding (VVC) standard adopts the trellis-coded quantization, which leverages the delicate trellis graph to map the quantization candidates within one block into the optimal path. Despite the high compression efficiency, the complex trellis search with soft-decision quantization may hinder the applications due to high complexity and low throughput capacity. To reduce the complexity, in this paper, we propose a low complexity trellis-coded quantization scheme in a scientifically sound way with theoretical modeling of the rate and distortion. As such, the trellis departure point can be adaptively adjusted, and unnecessarily visited branches are accordingly pruned, leading to the shrink of total trellis stages and simplification of transition branches. Extensive experimental results on the VVC test model show that the proposed scheme is effective in reducing the encoding complexity by 11% and 5% with all intra and random access configurations, respectively, at the cost of only 0.11% and 0.05% BD-Rate increase. Meanwhile, on average 24% and 27% quantization time savings can be achieved under all intra and random access configurations. Due to the excellent performance, the VVC test model has adopted one implementation of the proposed scheme.

Recently, neural style transfer has drawn many attentions and significant progresses have been made, especially for image style transfer. However, flexible and consistent style transfer for videos remains a challenging problem. Existing training strategies, either using a significant amount of video data with optical flows or introducing single-frame regularizers, have limited performance on real videos. In this paper, we propose a novel interpretation of temporal consistency, based on which we analyze the drawbacks of existing training strategies; and then derive a new compound regularization. Experimental results show that the proposed regularization can better balance the spatial and temporal performance, which supports our modeling. Combining with the new cost formula, we design a zero-shot video style transfer framework. Moreover, for better feature migration, we introduce a new module to dynamically adjust inter-channel distributions. Quantitative and qualitative results demonstrate the superiority of our method over other state-of-the-art style transfer methods. Our project is publicly available at: https://daooshee.github.io/CompoundVST/.

Cross-component prediction, which takes advantage of inter-channel correlations, predicts the chroma block with the luma reconstructed block according to associated linear model. Instead of involving all available reference samples in building the linear model, in this paper, we propose a sub-sampled approach that utilizes at most four neighboring chroma samples and their corresponding down-sampled luma samples, leading to significantly reduced operations in the derivation of model parameters at both encoder and decoder. The proposed scheme is hardware friendly in terms of the overheads of memory access and clock cycles, and greatly benefits the practical implementations of the emerging video coding standard in real applications. Extensive experiments reveal that the proposed sub-sampled method provides simple operations and robust coding performance, leading to the adoption by Versatile Video Coding (VVC) Standard and the third generation Audio Video Coding Standard (AVS3).

In this paper, we focus on computationally modeling of the local texture correlations, in an effort to better explore the coding modes with higher priorities in the rate-distortion optimized intra coding. In particular, strong correlations and continuities of local texture with neighboring blocks have been revealed in our analysis, and empirical justifications provide us inspirations on the joint optimization of rate-distortion-complexity when angular modes become finer to adapt the local textures. We examine the philosophy with extensive experiments conducted for refining the intra full-RD list. The results show that better coding performance with on average 0.72% and 3.00% BD-Rate savings for the natural scene and screen content sequences can be achieved in AVS3 test model HPM-5.0 under all intra configuration, with negligible encoding and decoding time variations.

Palette mode is a powerful tool for screen content coding in the upcoming versatile video coding (VVC) standard. In the palette mode, escape samples are employed to handle an outlier case. In this paper, a quantization parameter (QP) dependent fixed-length binarization is proposed for escape samples coding to 1) simplify the design and 2) improve coding efficiency. The length is calculated according to the QP for the current block and correspondingly, the dequantization process can be also implemented by only left shifting. The proposed method is evaluated with VVC reference software VTM-6.0 on typical sequences containing "text and graphics with motion". Experimental results report that the proposed scheme can achieve up to 2.0% BD-rate savings compared to VTM-6.0 with a much simpler design.

Due to the widespread deployment of fingerprint/face/speaker recognition systems, attacking deep learning based biometric systems has drawn more and more attention. Previous research mainly studied the attack to the vision-based system, such as fingerprint and face recognition. While the attack for speaker recognition has not been investigated yet, although it has been widely used in our daily life. In this paper, we attempt to fool the state-of-the-art speaker recognition model and present speaker recognition attacker, a lightweight model to fool the deep speaker recognition model by adding imperceptible perturbations onto the raw speech waveform. We find that the speaker recognition system is also vulnerable to the attack, and we achieve a high success rate on the non-targeted attack. Besides, we also present an effective method to optimize the speaker recognition attacker to obtain a trade-off between the attack success rate with the perceptual quality. Experiments on the TIMIT dataset show that we can achieve a sentence error rate of 99.2% with an average SNR 57.2dB and PESQ 4.2 with speed rather faster than real-time.

Attacking deep learning based biometric systems has drawn more and more attention with the wide deployment of fingerprint/face/speaker recognition systems, given the fact that the neural networks are vulnerable to the adversarial examples, which have been intentionally perturbed to remain almost imperceptible for human. In this paper, we demonstrated the existence of the universal adversarial perturbations (UAPs) for the speaker recognition systems. We proposed a generative network to learn the mapping from the low-dimensional normal distribution to the UAPs subspace, then synthesize the UAPs to perturbe any input signals to spoof the well-trained speaker recognition model with high probability. Experimental results on TIMIT and LibriSpeech datasets demonstrate the effectiveness of our model.

There has been a consensus regarding the intra prediction technique in video coding that the spatial redundancy can be efficiently removed by the locally accessible reference samples with certain projections and interpolations. In this paper, we revisit the short and long range correlations of the image content in the context of video coding, and it is interesting to find that the natural scene videos exhibit substantially different characteristics from screen content videos. This motivates us to redesign the intra mode coding method based on both short and long range correlations, as the existing approaches based on local content correlations cannot always effectively capture the most probable mode. One key feature of the proposed method is that it achieves unified content adaptive coding and is applicable across different video content. Experimental results on the versatile video coding (VVC) platform VTM-3.0 show the effectiveness of the proposed approach, leading to 3.73% bit rate savings for screen content videos and 0.10% bit rate savings for natural scene videos under all intra configuration.

The flexible partitioning structure such as quad-tree plus binary-tree (QTBT) plays important roles in the next generation video coding standard. This paper investigates new coding unit (CU) partitioning methods to further improve the compression efficiency. In particular, we propose the extended quad-tree partitioning and asymmetric ternary-tree partitioning to further develop the principle behind the existing partitioning structure. First, a central extended quad-tree (CENTRAL-EQT) partitioning is introduced, which extends the traditional QT partitioning with central pattern and generates four elaborately designed sub-CUs with different sizes. Second, we propose a parallel extended quad-tree (PARALLEL-EQT) partitioning that allows the CU to split along a single direction, leading to four identical size sub-blocks. Third, we present an asymmetric ternary-tree (ATT) method, which splits the CU asymmetrically into three sub-blocks. The proposed new partitioning methods allow the interleaving with binary-tree partitioning for enhanced adaptability, and they can be jointly enabled to capture different characteristics of the local content. Simulation results on the JEM7-QTBT-Only platform show that averagely 4.92%, 4.81% and 5.08% BD-Rate savings are achieved for the luma component by the proposed methods under random access (RA), low-delay P (LDP) and low-delay B (LDB) configurations, respectively, with around 725% encoding time increase and negligible decoding time increase. Furthermore, experimental results also reveal that the proposed partitioning schemes are effective when cooperating with the upcoming Versatile Video Coding standard.

In the emerging next generation video coding standard Versatile Video Coding (VVC) developed by the Joint Video Exploration Team (JVET), sub-block-based inter-prediction plays a key role in promising coding tools such as Affine Motion Compensation (AMC) and sub-block-based Temporal Motion Vector Prediction (sbTMVP). With sub-block-based inter-prediction, a coding block is divided into sub-blocks, and the motion information of each sub-block is derived individually. Although sub-block-based inter-prediction can provide a higher quality prediction benefiting from a finer motion granularity, it still suffers two problems: uneven prediction quality and boundary discontinuity. In this paper, we present a method of interweaved prediction to further improve sub-block-based inter-prediction. With interweaved prediction, a coding block with AMC or sbTMVP mode is divided into sub-blocks with two different dividing patterns, so that a corner position of a sub-block in one dividing pattern coincides with the central position of a sub-block in the other dividing pattern. Then two auxiliary predictions are generated by AMC or sbTMVP with the two dividing patterns, independently. The final prediction is calculated as a weighted-sum of the two auxiliary predictions. Theoretical analysis and statistical data prove that interweaved prediction can significantly mitigate the two problems in sub-block-based inter-prediction. Simulation results show that the proposed methods can achieve 0.64% BD-rate saving on average with the random access configurations. On sequences with rich affine motions, the average BD-rate saving can be up to 2.54%.

The quad-tree plus binary-tree (QTBT) coding unit (CU) partitioning structure, which has been adopted to the next generation video coding standard, shows promising coding performance when compared with the conventional quad-tree structure in HEVC. In this paper, we propose the Extended Quad-tree (EQT) partitioning, which further extends the QTBT scheme and increases the partitioning exibility. More specifcally, EQT splits a parent CU into four sub-CUs of dierent sizes, which can adequately model the local image content that cannot be elaborately characterized with QTBT. Meanwhile, EQT partitioning allows the interleaving with BT partitioning for enhanced adaptability. Experimental results on the JEM7-QTBT-Only platform show that EQT brings better coding performance with 3.17%, 3.20% and 3.06% BD-Rate gains under random access, low-delay P and low-delay B configurations, respectively.

In this paper, History-based Motion Vector Prediction (HMVP) is presented for video coding. With the proposed method, a table of HMVP candidates is maintained and updated on-the-fly. After decoding one inter-coded block, the table is updated by appending the associated motion information to the table as a new HMVP candidate. A First-In-First-Out (FIFO) rule is applied to manage the table. The HMVP candidates could be added to the Advanced Motion Vector Prediction (AMVP) candidate list as additional motion vector predictors. And they could also be added to the merge candidate list as additional merge candidates. With the proposed method, the motion information of previously coded blocks even not adjacent to the current block can be utilized for more efficient motion vector prediction. Simulation results have validated the efficiency of HMVP, wherein up to 4% BD rate saving could be achieved. The proposed method has been adopted by the next generation video coding standard, named Versatile Video Coding (VVC) developed by Joint Video Exploration Team (JVET).

With affine motion compensation (AMC) in the emerging next generation video coding standard Versatile Video Coding (VVC), a coding-block is divided into sub-blocks, and each sub-block is assigned with an individual motion vector derived by the affine model. The sub-block-based design for AMC faces a dilemma. With smaller sub-blocks, AMC can achieve a better coding performance but suffers a higher complexity burden. In this contribution, an interweaved prediction approach is proposed for AMC to address the dilemma. With the interweaved prediction, a coding block is divided into sub-blocks with two different dividing patterns. Then two auxiliary predictions are generated by AMC with the two dividing patterns respectively. The final prediction is calculated as a weighted-sum of the two auxiliary predictions. The interweaved prediction is only applied on the luma-component for affine-coded blocks with uni-prediction. Simulation results show 0.53% Bjøntegaard-Delta rate savings in average can be achieved compared to VTM-3.0 under random access configurations. The coding gain on sequences with affine motions is up to 3.3%.

Bi-directional optical flow (BDOF) is an efficient coding tool that has been recently adopted into Versatile Video Coding (VVC) standard. With BDOF, bi-predictive prediction samples of one coding block are enhanced via higher-precision motion vectors (MVs) derived from its two reference blocks. In this way, the energy of prediction error could be reduced, resulting in better coding performance. In VVC, the derived motion information is only used to enhance prediction samples. In this paper, it is proposed to use the derived motion information to also refine decoded MVs. The refined MVs may be used as spatial motion vector prediction (MVP) for the following coding units (CUs), as the temporal MVP for the subsequent pictures, and in the deblocking filtering process. Furthermore, the refined MVs can be used to perform motion compensation (MC) again to further improve the quality of the prediction samples. Simulation results show that the proposed methods can achieve -1.18% BD-rate saving in average under the random access configuration on top of the existing BDOF design in VVC.

The third generation of the audio video coding standard (AVS3) is the latest standardization developed by Audio Video Coding Standard Working Group of China on the basis of IEEE 1857.4. In this paper, we propose a History-based Motion Vector Prediction (HMVP) method for inter coding, which fully explores the history motion information of previous coded blocks. More specifically, an HMVP table, which records HMVP candidates, is elaborately maintained with limited capacity. In particular, the motion information of the previous coded blocks updates the HMVP table on-the-fly by appending the candidate to the last entry of the table after encoding or decoding an inter block. As such, the HMVP candidates act as additional modes for skip or direct modes to enhance coding efficiency. Besides, HMVP occupies limited memory to store motion info, which is crucial in real applications. Experimental results show that HMVP brings better coding performance with up to 1.38% and 2.33% BD-Rate reduction for random access and low-delay P configurations, respectively, on the latest TAVS3-2.2 platform. Moreover, nearly no encoding and decoding burden is introduced by HMVP. The proposed HMVP has been adopted by AVS3.

Affine Motion Model (AMM) based inter prediction, which can represent complex motions such as zooming, rotation or shearing, has been adopted into the Versatile Video Coding (VVC) standard. AMM is defined by Control Point Motion Vectors (CPMVs) in VVC. On the other hand, Adaptive Motion Vector Resolution (AMVR) has also been adopted into VVC standard due to a favorable trade-off between the Motion Vector (MV) precision and the bit consumption on MV Differences (MVDs). However, AMVR is only applied to the Translational Motion Model (TMM), and AMM cannot benefit from it. In this paper, it is proposed to extend AMVR to AMM. Specifically, 1-pixel, 1/4-pixel and 1/16-pixel MV precisions are allowed and can be selected adaptively by each affine-inter mode coded Coding Unit (CU). Simulation results reportedly show that the proposed method can achieve 0.32% BD-rate saving on average under the Random Access configuration.

AVS3 adopts quad-tree (QT) with nested binary tree (BT) and extended quad-tree (EQT) partitioning, which shows promising compression performance when compared to the conventional QT partitioning in AVS2 and HEVC. However, the interleaved and recursive splitting manner significantly increases the computational complexity of the encoder, which may impede the real applications of AVS3. This paper proposes fast coding unit splitting decision methods for QT, BT and EQT partitioning. In particular, the Bayesian decision rule is employed with Skip states, and BT and EQT partitioning can be early terminated. Additionally, the average splitting depth that is produced by BT partitioning, is incorporated as the prior information for terminating the EQT and QT splittings. Moreover, we explore the splitting settings for CUs that are predicted with Skip mode, with the aim of eliminating unnecessary partition attempts. Experimental results show that the proposed fast algorithms are effective and provide a good trade-off between computational complexity and coding performance. In particular, 69% encoding time reduction is achieved with only 0.55% increase in terms of BD-Rate on average, which greatly benefits the practical implementations of the AVS3 in real applications. The proposed methods have been adopted into TAVS3 reference software.

The Joint Video Exploration Team (JVET) has been developing an emerging standard Versatile Video Coding (VVC), which includes screen contents as one of its requirements. Intra block copy (IBC) and palette coding are the two powerful coding tools for screen content coding. In this paper, a compound palette mode is proposed to exploit the advantages of both IBC and palette coding, which allows samples to be reconstructed by either IBC predictions or palette entries. The proposed method is evaluated with VVC reference software VTM4 on typical sequences containing "text and graphics with motion". Experimental results report significant coding gain that the proposed scheme can achieve 7.80% and 1.03% BD-rate savings under AI conditions on average when compared with VTM4 and the existing palette scheme.

Affine Motion Compensation (AMC) is a promising coding tool in Joint Exploration Model (JEM) developed by the Joint Video Exploration Team (JVET). AMC in JEM employs a 4-parameter affine model between the current block and its reference block. With this model, Motion Vectors (MV) of each sub-block can be derived from the MVs at two control points. In this paper, we present a practical framework to further improve the AMC in JEM. First, we introduce a multi-model AMC approach, which allows the encoder to select either the 4-parameter affine model or the 6-paramter affine model adaptively. Second, we improve the affine inter-mode in two aspects. For the normative part, we present an efficient affine motion coding method, which replaces the affine MV Prediction (MVP) candidates in JEM with more accurate but simpler ones, and employs a second-order MVP. For the non-normative part, we enhance the motion estimation process for AMC, by regulating the optimization algorithm. Finally, we propose to unify the affine merge-mode and the normal merge-mode into a unified merge-mode, which combine affine merge candidates and normal merge candidates in a single merge candidate list. Partial of these methods have been adopted into the next generation video coding standard named Versatile Video Coding (VVC). Simulation results show that the proposed methods can achieve 1.67% BD rate savings in average for the random access configurations.

Cross-Component Linear Model (CCLM) for chroma intra-prediction is a promising coding tool in Joint Exploration Model (JEM) developed by the Joint Video Exploration Team (JVET). CCLM assumes a linear correlation between the luma and chroma components in a coding block. With this assumption, the chroma components can be predicted by the Linear Model (LM) mode, which utilizes the reconstructed neighbouring samples to derive parameters of a linear model by linear regression. This paper presents three new methods to further improve the coding efficiency of CCLM. First, we introduce a multi-model CCLM (MM-CCLM) approach, which applies more than one linear models to a coding block. With MM-CCLM, reconstructed neighbouring luma and chroma samples of the current block are classified into several groups, and a particular set of linear model parameters is derived for each group. The reconstructed luma samples of the current block are also classified to predict the associated chroma samples with the corresponding linear model. Second, we propose a multi-filter CCLM (MF-CCLM) technique, which allows the encoder to select the optimal down-sampling filter for the luma component with the 4:2:0 colour format. Third, we present a LM-angular prediction (LAP) method, which synthesizes the angular intra-prediction and the MM-CCLM intra-prediction into a new chroma intra coding mode. Simulation results show that 0.55%, 4.66% and 5.08% BD rate savings in average on Y, Cb and Cr components respectively, are achieved for All Intra (AI) configurations with the proposed three methods. MM-CCLM and MF-CCLM have been adopted into the JEM by JVET.

In this paper, a multiple direct mode (MDM) method is presented for chroma intra coding. The main contributions of the proposed MDM method include two aspects: selection of multiple luma intra prediction modes from co-located luma blocks, and the derivation of chroma intra prediction modes from spatial neighbouring blocks. With the proposed method, both the cross-component correlation and spatial correlation of intra prediction modes can be better utilized for more efficient chroma intra coding. Simulation results have validated the efficiency of MDM especially under the decoupled luma-chroma partition trees. The proposed method has been adopted in the Joint Exploration Model (JEM) which is the test platform for future video coding technology exploration in Joint Video Exploration Team (JVET).

Cross-component Linear Model (CCLM) chroma intra prediction assumes a linear correlation between the luma and chroma components in a coding block. With this assumption, the chroma components can be predicted by LM mode, which utilizes the reconstructed neighbouring samples to derive parameters of the linear model by linear regression. This paper presents a multi-model CCLM (MM-CCLM) approach, which applies more than one linear models in a coding block. With MM-CCLM, reconstructed neighbouring luma and chroma samples of the current block are classified into several groups and each group is used as a training set to derive its own linear model. The reconstructed luma samples of the current block are also classified to use corresponding linear model to predict the associated chroma samples. Simulation results show that 0.26%, 1.89% and 1.96% BD rate savings on Y, Cb and Cr components are achieved for All Intra (AI) configurations in average. The proposed method has been adopted in the Joint Exploration Model (JEM) by Joint Video Exploration Team (JVET).

The Discrete Cosine Transform (DCT), and in particular the DCT type II, has been widely used for image and video compression. Although DCT efficiently approximates the optimal Karhunen–Loève transform under first-order Markov conditions with low complexity, the energy packing efficiency is still limited since a fixed transform cannot always capture the highly dynamic statistics of natural video content. In this paper, to further improve the transform efficiency, an Enhanced Multiple Transform (EMT) scheme is proposed. In the proposed EMT, a few sinusoidal transforms, other than DCT, have also been utilized for coding both Intra and Inter prediction residuals. The best transform, as selected from a pre-defined transform subset specified by prediction mode, is explicitly signaled in a joint coding block level manner. Moreover, to accelerate encoding process, fast methods have also been proposed by skipping unnecessary transform rate-distortion evaluations using previously encoding statistics. The proposed method has been implemented on top of High-Efficiency Video Coding (HEVC) reference software, and significant coding gain has been verified.

Recently, adaptive in-loop filter (ALF) for image/video coding has attracted increasing attention by its proven capability in improving coding performance. ALF is aiming to minimize the mean square error between original samples and decoded samples by using Wiener-based adaptive filter. Samples in a picture are classified into multiple categories and the samples in each category are then filtered with their associated adaptive filter. The filter coefficients may be signaled or inherited to optimize the tradeoff between the mean square error and the overhead. In this paper, a Geometry transformation-based ALF (GALF) scheme is proposed to further improve the performance of ALF, which introduces geometric transformations, such as rotation, diagonal and vertical flip, to be applied to the samples in filter support region depending on the orientation of the gradient of the reconstructed samples before ALF. With the introduction of geometric transformations, more spatial adaptation is supported without excessive signaling of filter coefficients. The experimental results show that GALF outperforms the existing ALF techniques and it has been adopted by the JEM reference software used as the test platform for future video coding technology exploration in JVET.

Compression of the statistical redundancy among different viewpoints, i.e., inter-view redundancy, is a fundamental and critical problem in multiview and three-dimensional (3D) video coding. To exploit the inter-view redundancy, disparity vectors are required to identify pixels of the same objects within two different views; in this way, the enhancement coding tools can be efficiently employed as new modes in block-based video codecs to achieve higher compression efficiency. Although disparity can be converted from depth, it is not possible in multiview video coding since depth information is not considered. Even when depth information is coded, it breaks the so-called multiview compatibility wherein texture views can be decoded without depth information. To resolve this problem, in this paper, a neighboring block-based disparity vector derivation (NBDV) method is proposed. The basic concept of NBDV is to derive a disparity vector (DV) of a current block by utilizing the motion information of spatially and temporally neighboring blocks predicted from another view. Through extensive experiments and analysis, it is shown that the proposed NBDV method achieves efficient DV derivation in the state-of-art video codecs, and it keeps the multiview compatibility with a relatively lower complexity. The proposed method has become an essential part of the 3D video standard extensions of H.264/AVC and HEVC.

This paper presents an in-loop adaptive color-space transform for the HEVC Screen Content Coding extension. In the proposed method, the prediction residual is adaptively converted into a different color space to reduce the cross-component redundancy. After the ACT, the signal is coded following the existing HEVC framework. To keep the complexity as low as possible, fixed color-space transforms that are easily implemented with shift and add operations are utilized. Significant coding gains are achieved by this method in the current HEVC Screen Content Coding reference software with no increase of decoding runtime. The proposed method has been adopted to the HEVC Screen Content Coding extension.

In 3D-HEVC, bi-partition based modes, i.e., depth modeling modes (DMM), are applied for depth intra coding. With DMM, a depth prediction unit (PU) is partitioned into two parts using a Wedgelet pattern selected from a predefined large Wedgelet set, and the Wedgelet set is generated during both encoder and decoder initialization for each block size ranging from 4 ×4 to 32×32. The generation of Wedgelet sets involves relatively high complexity in terms of both storage and computation, especially for large block sizes. To simplify the Wedgelet generation, in this paper, we propose a unified Wedgelet generation method which derives large Wedgelet patterns by extending the primitive 4×4 Wedgelet pattern along its partition boundary line, and the storage of large Wedgelet patterns can be saved. Experimental results show almost no coding efficiency degradation using the proposed method, and the complexity of Wedgelet generation process is largely reduced in a unified manner.

This paper presents an inter-view motion prediction technique for efficient compression of motion vectors of the depth views in 3D-HEVC. 3D-HEVC is an extension of HEVC standard for coding the multi-view video plus depth content, known as MVD. In MVD format, the motion characteristics of the adjacent views in the depth video are highly correlated. In this paper, we take benefit of this correlation and propose inter-view motion prediction technique, where motion information of the dependent depth views is predicted from the already coded motion information in a reference depth view. In addition, a novel method for deriving disparity vectors based on neighboring pixels is proposed in order to establish correspondences between the blocks in different depth views. Experimental results show that proposed inter-view motion prediction method provides an average bit-rate savings of 1.5% for the synthesized views when the motion information of the depth views is predicted without using the texture information.

3D-HEVC incorporates advanced inter-view prediction techniques based on more accurately derived disparity vector to better exploit the correlation between objects in different views. The efficient disparity vector derivation method, namely, Neighboring Block based Disparity Vector Derivation (NBDV) provides disparity without accessing any depth information. The NBDV has been developed as a part of the 3D-HEVC in Joint Collaborative Team on 3D Video Coding (JCT-3V) for several meeting cycles, and adopted as a common coding tool used for all the inter-view prediction techniques for high efficient coding of texture views. This paper presents a low complexity NBDV, which is the state-of-the-art disparity vector derivation method in 3D-HEVC.

Advanced residual prediction (ARP) is an efficient tool for 3D video coding by exploiting the residual correlation between views. In ARP, the residual predictor could be efficiently produced by aligning the motion information at the current view for motion compensation in the reference view. On the other hand, such on-the-fly residual predictor derivation process increases the complexity significantly due to the increased motion compensation steps. In this paper, a low-complexity ARP scheme is proposed. Experimental results demonstrate that the proposed scheme significantly reduces the decoding complexity of the original design in terms of both memory access and computational complexity while keeping comparable coding performance.

This paper presents a novel inter-view motion prediction technique used in 3D-HEVC which provides efficient compression for motion vectors. The 3D extension of HEVC (High Efficiency Video Coding) namely 3D-HEVC is under development in JCT-3V for coding multi-view video and depth data. Inter-view motion prediction takes benefit of the inter-view correlation between views by inferring the motion information of a view from the already coded motion information in another view as well as the local disparity between two views. Experimental results show that inter-view motion prediction in 3D-HEVC provides an average bit-rate savings of 15% for the dependent views.

In the 3D video extension of H.264/AVC, namely 3D-AVC, Neighboring Based Disparity Vector (NBDV) derivation has been proposed to support multiview/stereo compatibility, therefore texture views can be decoded independently to depth views. NBDV generates a disparity vector for the current macroblock (MB) using the motion information of neighboring blocks, especially those coded with motion vectors pointing to inter-view reference pictures. In 3D-AVC, NBDV has been utilized to access minimum number of spatial and temporal neighboring blocks, therefore there is a high probability that NBDV does not derive an efficient disparity vector. This paper introduces a derived disparity vector scheme, wherein only one disparity vector derived from NBDV is maintained for the whole slice and it is used as the disparity vector of the current MB if NBDV does not derive one from neighboring blocks. Simulation results show that the proposed method provides 3.6% bit rate reduction for multiview coding.

MV-HEVC refers to the multiview extension of High Efficiency Video Coding (HEVC). At the time of writing, MV-HEVC was being developed by the Joint Collaborative Team on 3D Video Coding Extension Development (JCT-3V) of International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) Moving Picture Experts Group and ITU-T VCEG. Before HEVC itself was technically finalized in January 2013, the development of MV-HEVC had already started and it was decided that MV-HEVC would only contain high-level syntax changes compared with HEVC, i.e., no changes to block-level processes, to enable the reuse of the first-generation HEVC decoder hardware as is for constructing an MV-HEVC decoder with only firmware changes corresponding to the high-level syntax part of the codec. Consequently, any block-level process that is not necessary for HEVC itself but on the other hand is useful for MV-HEVC can only be enabled through so-called hooks. Motion hooks refer to techniques that do not have a significant impact on the HEVC single-view version 1 codec and can mainly improve MV-HEVC. This paper presents techniques for efficient MV-HEVC coding by introducing hooks into the HEVC design to accommodate inter-view prediction in MV-HEVC. These hooks relate to motion prediction, hence named motion hooks. Some of the motion hooks developed by the authors have been adopted into HEVC during its finalization. Simulation results show that the proposed motion hooks provide on average 4% of bitrate reduction for the views coded with inter-view prediction.

Inter-view residual prediction (IVRP) is employed t o efficiently code non-base texture views by exploiting the correlation of residues between two views in 3D video coding extension of HEVC(3D-HEVC). To further improve the performance of texture coding, advanced residual prediction (ARP) is proposed in this paper. In IVRP, residues in a non-base view are predicted from decoded residues in base view. In contrast, ARP makes the residual predictor for the non-base view block based on newly generated base-view residues by applying motion vector at the non-base view to the baseview. Moreover, an adaptive weighting factor is applied to reduce prediction error. Comprehensive simulations show that up to 6.0% luma BD-rate reduction was obtained over IVRP in the reference software of 3D-HEVC.

In 3D-HEVC, depth modeling modes (DMM) are applied for efficient intra depth coding. With DMM modes, a prediction unit is partitioned into two parts, and each part is predicted by a single value. In one DMM mode, the partition pattern is implicitly derived at the decoder by searching all pre-defined Wedgelet patterns on a Co-located Texture Luma Block (CTLB), which increases the decoding complexity drastically. To simplify the design of this DMM mode, in this paper, we propose to utilize the Intra Prediction Mode (IPM) of CTLB to largely skip unnecessary Wedgelet searches at the decoder. To achieve this goal, for each IPM, only a limited number of Wedgelet patterns are selected as candidates in this DMM mode. Experimental results demonstrate that, with almost no coding performance degradation, the proposed method significantly reduces the decoding complexity by skipping 90% of Wedgelet searches. The proposed method has been adopted by 3D-HEVC.

The specialized QP determination and reference picture set (RPS) mechanism in High Efficiency Video Coding (HEVC) significantly improves the coding efficiency. In this paper, we propose a frame layer rate control algorithm for HEVC based on this specialized mechanism. Firstly, we design a RPS based hierarchical partition structure. Then an efficient hierarchical bit allocation scheme is proposed. Moreover, a novel header bits ratio prediction method is also provided to further improve the accuracy of bit allocation. Finally, an improved mean of absolute difference (MAD) prediction scheme is provided for quadratic R-Q model to calculate an accurate quantization parameter. Experimental results demonstrate the proposed algorithm has much better R-D performance. Compared with the state-of-the-art rate control scheme for HEVC, the BD-Rate gain can be up to 30% and BD-PSNR gain can be 0.6dB on average in Low_Delay configuration. Moreover, our proposed algorithm also provides smaller mismatch between the target bit rate and the actual bit rate.

Coding multiview video content often benefits significantly from disparity compensation. More advanced inter-view predictions, to predict e.g., motion vectors and residues among views require disparity vector to identify blocks in different views corresponding to the same objects. Block-level disparity vectors in typically approaches are either transmitted thus with additional overhead, or predicted with sophisticated algorithms. This paper describes a new method to derive disparity vectors, based on which advanced inter-view predictions can be better supported. The proposed method derives a disparity vector of a block only from spatial and temporal neighboring blocks. It was implemented and adopted into the 3D-HEVC codec, which is currently developed by Joint Collaborative Team on 3D Video coding development (JCT-3V). Experimental results show that with the proposed method, advanced inter-view prediction in 3D-HEVC provides in average 5.2% bitrate saving of the overall bitrate, for multiview video coding.

3D-AVC, being developed under Joint Collaborative Team on 3D Video Coding (JCT-3V), significantly outperforms the Multiview Video Coding plus Depth (MVC+D) which has no new macroblock level coding tools compared to Multiview video coding extension of H.264/AVC (MVC). However, for multiview compatible configuration, i.e., when texture views are decoded without accessing depth information, the performance of the current 3D-AVC is only marginally better than MVC+D. The problem is caused by the lack of disparity vectors which can be obtained only from the coded depth views in 3D-AVC. In this paper, a disparity vector derivation method is proposed by using the motion information of neighboring blocks and applied along with existing coding tools in 3D-AVC. The proposed method improves 3D-AVC in the multiview compatible mode substantially, resulting in about 20% bitrate reduction for texture coding. When enabling the so-called view synthesis prediction to further refine the disparity vectors, the performance of the proposed method is 31% better than MVC+D and even better than 3D-AVC under the best performing 3D-AVC configuration.

Block-based discrete cosine transform (DCT) has been successfully adopted into several international image/video coding standards, e.g., MPEG-2, H.264/AVC, as it can achieve a good tradeoff between performance and complexity. Although DCT theoretically approximates the optimum Karhunen–Loève transform under first-order Markov conditions, one fixed set of transform basis functions (TBF) cannot handle all the cases efficiently due to the non-stationary nature of video contents. To further improve the performance of block-based transform coding, in this paper, we present the design of rate-distortion optimized transform (RDOT) which contributes to both intraframe and interframe coding. The most important property which makes a difference between RDOT and the conventional DCT is that, in the proposed method, transform is implemented with multiple TBF candidates which are obtained from off-line training. With this feature, for coding each residual block, the encoder is capable to select the optimal set of TBF in terms of rate-distortion performance, and better energy compaction is achieved in the transform domain. To obtain an optimum group of candidate TBF, we have developed a two-step iterative optimization technique for the off-line training, with which the TBF candidates are refined at each iteration until the training process becomes converged. Moreover, analysis on the optimal group of candidate TBF is also presented in this paper, with a detailed description of a practical implementation for the proposed algorithm on the latest VCEG key technical area software platform. Extensive experimental results show that, compared with the conventional DCT-based transform scheme adopted into the state-of-the-art H.264/AVC video coding standard, significant improvement of coding performance has been achieved for both intraframe and interframe coding with our proposed method.

In this paper, a novel intra prediction algorithm, named enhanced line-based intra prediction (ELIP), is proposed to improve the traditional intra prediction methods for image/intra-frame coding. Different from the existing schemes where the interpolation filtering only depends on the intra prediction mode, in the proposed ELIP, the linear filtering coefficients are further refined by imposing both mode and position dependencies. For each intra prediction mode and each position within one block, a set of linear filtering coefficients are obtained from off-line training using the least square method. The proposed algorithm has been implemented in the latest ITU-T VCEG KTA software. Experimental results demonstrate that, compared with the intra coding scheme of KTA with new intra coding tool enabled, up to 0.44dB additional coding gain is achieved by the proposed method, while keeping applicable computational complexity for practical video codecs.

The direct mode in H.264/AVC can efficiently improve the coding performance of B pictures, since it exploits the spatial or temporal correlation by deriving its motion vector from previously encoded information. Therefore, it does not require any additional motion information and could save many bits. Considering the spatial and temporal correlation has not been fully exploited in the current direct mode, in this paper, we propose an advanced Spatial and Temporal Direct Mode (STDM) for B picture coding. The motion vector is selected from a set of spatial-temporal neighboring motion vectors, and the selection criterion is to minimize a spatial-temporal cost function. The framework of Decoder-side Motion Vector Derivation (DMVD) is utilized, where encoder and decoder use the same derivation process to obtain motion vectors, thus no index for the chosen motion vectors need to be coded and transmitted. Simulation results show that the proposed method significantly outperforms the current SDM and TDM in H.264/AVC.

As the next generation standard of video coding, the High Efficiency Video Coding (HEVC) is intended to provide significantly better coding efficiency than all existing video coding standards. To improve the coding efficiency of intra frame coding, up to 34 intra prediction modes are defined in HEVC. The best mode among these pre-defined intra prediction modes is selected by rate-distortion optimization (RDO) for each block. If all directions are tested in the RDO process, it will be very time-consuming. To alleviate the encoder computation load, this paper proposes a new method to reduce the candidates in RDO process. In addition, the direction information of the neighboring blocks is made full use of to speed up intra mode decision. Experimental results show that the proposed scheme provides 20% and 28% time savings in intra high efficiency and low complexity cases on average compared to the default encoding scheme in HM 1.0 with almost the same coding efficiency. This algorithm has been proposed to HEVC standard and partially adopted into the HEVC test model.

Intra prediction is one effective method to remove the spatial redundancies in intra frame coding. Better intra prediction will result in the residual with less energy, which will decrease the number of bits needed to reconstruct the signal at decoder. To improve the accuracy of intra prediction, a context-adaptive pixel based prediction (CAPBP) algorithm is proposed in this paper. For each pixel within the target block to be encoded, the prediction is calculated as the linear weighted summation of the reconstructed pixels within the left column and the above row. Based on the assumption that pixels having the same coordinates within one block own the same prediction weights, we calculate the corresponding weights for each pixel within the target block by the least square method. The same processing is also performed at decoder; hence the prediction weights do not need to be sent to the decoder. Experimental results verify that the proposed algorithm is able to improve the efficiency of intra frame coding up to 0.5dB.

In this paper, a novel algorithm is proposed for intra-frame coding, named as rate-distortion optimized transform (RDOT). Unlike existing intra-frame coding schemes where the transform matrices are either fixed or mode dependent, in the proposed algorithm, transform is implemented with multiple candidate transform matrices. With this flexibility, for coding each residual block, the encoder is endowed with the power to select the optimal transform matrix in terms of rate-distortion tradeoff. The proposed algorithm has been implemented in the latest ITU-T VCEG-KTA software. Experimental results show that, over a wide range of test set, the proposed method achieves average 0.43dB coding gain compared with the recent Mode-Dependent Directional Transform (MDDT). The improvement is more significant at high bit-rates, and up to 1dB coding gain can be achieved.

Intra prediction has been efficiently employed in block based image/video coding to remove the spatial redundancy. In this paper, we propose a position dependent linear intra prediction (PDLIP) approach to further improve the accuracy of prediction. Specifically, for each prediction direction, the prediction of each sample within a target block is calculated as a linear weighted summation of the surrounding spatial neighboring samples in a fixed window. And the prediction coefficients are adaptively derived in the encoding process using the least square estimation method. The derivation process utilizes the original data of the previous image with the same prediction direction after rate-distortion selection to adaptively adjust to the video contents and local context within one block. To alleviate the computation burden of decoding, these prediction coefficients are sent to decoder as side information. Experimental results demonstrate that the proposed PDLIP can achieve a significant coding gain with a slight increase in computational complexity at encoder.

In our previous work, the rate-distortion optimized transform (RDOT) is introduced for Intra coding, which is featured by the usage of multiple offline-trained transform matrix candidates. The proposed RDOT achieves remarkable coding gain for KTA Intra coding, while maintaining almost the same computational complexity at the decoder. However, at the encoder, the computational complexity is increased drastically by the expensive ratedistortion (R-D) optimized selection of transform matrix. To resolve this problem, in this paper, we propose a fast RDOT scheme using macroblock- and block-level R-D cost thresholding. With the proposed method, unnecessary mode trials and R-D evaluations of transform matrices can be efficiently skipped from the mode decision process. Extensive experimental results show that, with negligible coding performance degradation, about 88.9% of the total encoding time is saved by the proposed method.

Super-Resolution (SR) is a technique to construct a high-resolution (HR) frame by fusing a group of low-resolution (LR) frames describing the same scene. The effectiveness of the conventional super-resolution techniques, when applied on video sequences, strongly relies on the efficiency of motion alignment achieved by image registration. Unfortunately, such efficiency is limited by the motion complexity in the video and the capability of adopted motion model. In image regions with severe registration errors, annoying artifacts usually appear in the produced super-resolution video. This paper proposes a robust video super-resolution technique that adapts itself to the spatially-varying registration efficiency. The reliability of each reference pixel is measured by the corresponding registration error and incorporated into the optimization objective function of SR reconstruction. This makes the SR reconstruction highly immune to the registration errors, as outliers with higher registration errors are assigned lower weights in the objective function. In particular, we carefully design a mechanism to assign weights according to registration errors. The proposed superresolution scheme has been tested with various video sequences and experimental results clearly demonstrate the effectiveness of the proposed method.

This paper proposed a novel Region of Interest (ROI) based rate control algorithm for video communication systems, in which the subjective quality of ROI can be adjusted according to users' requirements. In the proposed scheme, in order to analyze the relationship between subjective quality and encoding parameters, a structural similarity index map - quantization parameter (SSIM-QP) model is established. Through this relation, the possible visual quality range of ROI is defined according to the range of ROI QP, which is predicted by rate control algorithm. Then, with interest levels being identified within the visual quality range, resources allocated to ROI is determined. Finally, considering both the quality of ROI and the entire frame, resource allocation is slightly adjusted. Experimental results demonstrate that the subjective visual quality of ROI in our proposed scheme can be adjustable improved compared to the existing rate control algorithms.

In inter-picture coding, block-based frequency transform is usually carried out on the predicted errors for each interblock to remove the spatial correlation among them. However, it can not always do well since the predicted errors in some inter-blocks have marginal or diagonal correlation. A good solution is to omit transform operations for the predicted errors of those inter-blocks with low correlation before quantization operation. The same phenomenon also can be observed in Fine Grain Scalability (FGS) layer coding. In this paper, an adaptive prediction error coding method in spatial and frequency domain with lower complexity is considered for FGS layer coding. Transform operation is only needed when there are non-zero reconstructed coefficients in spatially co-located block in base layer. The experimental results show that compared with FGS coding in JSVM, higher coding efficiency can be achieved with lower computational complexity at decoder since inverse transform is no longer needed for those predicted errors coded in spatial domain at encoder.

Discrete cosine transform (DCT) has been widely used in image/video coding systems, where zigzag scan is usually employed for DCT coefficient organization. However, due to local diversity of prediction errors, the fixed scan pattern, such as the traditional zigzag scan, is not efficient all the time for organizing the DCT transformed prediction errors. In this paper, the statistical distribution of prediction residuals is studied first, and two other scan patterns are introduced to complete the set of possible patterns. Then, a new adaptive scheme that adaptively chooses for each macroblock the best scan pattern to arrange the quantized DCT coefficients into arrays is presented. The best scan pattern is selected for each macroblock according to its own residual characteristics and hence leads to improved coding efficiency. Experimental results show that this macroblock-level adaptive scan scheme (MLASS) can always outperform the traditional zigzag one and the peak signal to noise ratio (PSNR) gain can be up to 0.65dB. Moreover, the extra two scan patterns do not increase memory requirement much. Additionally, a fast scan pattern decision algorithm is given for complexity reduction. This fast version of MLASS can keep almost the same coding efficiency while with much lower complexity.

This paper presents a new context modeling technique for arithmetic coding of DCT coefficients in video compression. A key feature of the new technique is the inclusion of all previously coded coefficient magnitudes in a DCT block in context modeling. This enables adaptive arithmetic coding to exploit the redundancy of the high-order Markov process in the DCT domain with a few conditioning states. In addition, a context weighting technique is used to further improve the coding efficiency. The complexity of the new arithmetic coding scheme is slightly lower than that of context-based adaptive binary arithmetic coding (CABAC) of H.264. Moreover, the scheme is made compatible to the AVS baseline profile. It achieves on average 13% improvement in compression ratio over context-based two dimension variable length coding (C2DVLC) designed for the DCT domain, and a similar coding efficiency as the CABAC technique in H.264.

Video encoding now is being implemented in various computing platforms with different computing capability, the requirement on the encoding complexity is also different according to different applications. As the most computation-intensive part of video encoding, the ME (motion estimation) should have a scalable complexity. This paper proposes a ME algorithm with fine-granular scalable complexity, a more important feature of the proposed algorithm is that it seeks for the complexity-distortion optimization. The given computation budget will be allocated to each MB (macroblock) in one frame. Each MB will consume its allocated computation by a hybrid search pattern. Experimental results show that the proposed algorithm can get a better computation-distortion performance than the existing ME algorithms

Owing to the great computation complexity of the ME (motion estimation) in video coding, a lot of fast ME algorithms have been proposed in literature. Most of them are designed based on a given hypothesis about the character of the video’s motion field. But due to the inherent variety of the real-world video, these algorithms are not generally efficient, especially for the variable block size ME in H. 264, only using one single ME strategy is difficult to get satisfactory result. This paper firstly will give two different fast ME algorithms, which aim at video sequences with different character. An adaptive strategy is proposed to combine the two algorithms together for H. 264. Experimental result shows that this strategy can keep coding efficiency in all kinds of video sequences, meanwhile the computation complexity is reduced adaptively according to the video’s content.