RGB-IR(RGB-Infrared) image pairs are frequently applied simultaneously in various applications like intelligent surveillance. However, as the number of modalities increases, the required data storage and transmission costs also double. Therefore, efficient RGB-IR data compression is essential. This work proposes a joint compression framework for RGB-IR image pair. Specifically, to fully utilize cross-modality prior information for accurate context probability modeling within and between modalities, we propose a Channel-wise Cross-modality Entropy Model (CCEM). Among CCEM, a Low-frequency Context Extraction Block (LCEB) and a Low-frequency Context Fusion Block (LCFB) are designed for extracting and aggregating the global low-frequency information from both modalities, which assist the model in predicting entropy parameters more accurately. Experimental results demonstrate that our approach outperforms existing RGB-IR image pair and single-modality compression methods on LLVIP and KAIST datasets. For instance, the proposed framework achieves a 23.1% bit rate saving on LLVIP dataset compared to the state-of-the-art RGB-IR image codec presented at CVPR 2022.

High-efficient image compression is a critical requirement. In several scenarios where multiple modalities of data are captured by different sensors, the auxiliary information from other modalities are not fully leveraged by existing image-only codecs, leading to suboptimal compression efficiency. In this paper, we increase image compression performance with the assistance of point cloud, which is widely adopted in the area of autonomous driving. As depicted in Figure 1 (a), we have unified the digital representation of image and point cloud, and propose the point cloud-assisted neural image codec (PCA-NIC) to enhance the preservation of image texture and structure by utilizing the high-dimensional point cloud information. As depicted in Figure 1 (b), we further introduce a multi-modal feature fusion transform module (MMFFT) to capture more representative image features, remove redundant information between channels and modalities that are not relevant to the image content.

Quality metrics play a crucial role in guiding the development of image enhancement algorithms, which have consistently sought effective quality assessment methodologies and comprehensive datasets. To address this need, we first built a large-scale dataset for low-light enhanced image quality assessment and gathered the corresponding subjective evaluation scores. Recently, vision-language pre-training models have demonstrated considerable potential in the realm of quality assessment. However, its efficacy is limited by the fine-grained perception in low-level quality assessment. As such, we further propose a novel quality assessment framework using contrastive prompt learning, which harnesses the robust priors of vision-language pre-training models to improve the perceptual capacity of deep networks for low-level quality features. Experiments on the proposed RSLE dataset show that our method outperforms existing SOTA image quality assessment methods. Our database and the source code will be made publicly available.

In this paper, we propose a novel framework for Interactive Face Video Coding (IFVC), which allows humans to interact with the intrinsic visual representations instead of the signals. The proposed solution enjoys several distinct advantages, including ultra-compact representation, low delay interaction, and vivid expression/headpose animation. In particular, we propose the Internal Dimension Increase (IDI) based representation, greatly enhancing the fidelity and flexibility in rendering the appearance while maintaining reasonable representation cost. By leveraging strong statistical regularities, the visual signals can be effectively projected into controllable semantics in the three dimensional space (e.g., mouth motion, eye blinking, head rotation, head translation and head location), which are compressed and transmitted. The editable bitstream, which naturally supports the interactivity at the semantic level, can synthesize the face frames via the strong inference ability of the deep generative model. Experimental results have demonstrated the performance superiority and application prospects of our proposed IFVC scheme. In particular, the proposed scheme not only outperforms the state-of-the-art video coding standard Versatile Video Coding (VVC) and the latest generative compression schemes in terms of rate-distortion performance for face videos, but also enables the interactive coding without introducing additional manipulation processes. Furthermore, the proposed framework is expected to shed lights on the future design of the digital human communication in the metaverse. The project page can be found at https://github.com/Berlin0610/Interactive_Face_Video_Coding.

Recent advancements in neural image compression have shown great potential in outperforming conventional standard codecs in terms of both rate-distortion and rate-analysis performance. However, there is an issue of divergent preferences in information preservation or reconstruction in the process of compression for humans and machines, respectively. Compression for humans tends to retain the signal fidelity or perceptual quality of visual appearance while compression for machines requires preserving critical semantic information, resulting in the limitation of the bitstream supporting only a single requirement during the compression. To bridge this gap, we propose a dynamic adaptation approach that generates a single bitstream serving both humans and machines. This approach aims to mitigate the domain gap among tasks, which facilitates maintaining the performance of out-of-scope tasks. Specifically, the proposed method concentrates on learning a dynamic adaptation process, i.e., optimizing the latent representation in the compressed domain in an end-to-end manner while adhering to the rate-performance constraint. Extensive results reveal that our paradigm significantly reduces the domain gap, surpassing existing codecs.

With the rapid development of video conferencing and online education applications, screen content image (SCI) compression has become increasingly crucial. Recently, deep learning techniques have made significant progress in compressing natural images, surpassing the performance of traditional standards like versatile video coding. However, directly applying these methods to SCIs is challenging due to the unique characteristics of SCIs. In this paper, we propose a synergistic approach to preserve structural fidelity and text integrity for SCIs. Firstly, external prior guidance is proposed to enhance structural fidelity and text integrity by providing global spatial attention. Then, a structural enhancement module is proposed to improve the preservation of structural information by enhanced spatial feature transform. Finally, the loss function is optimized for better compression efficiency in text regions by weighted mean square error. Experimental results show that the proposed method achieves 13.3% BD-Rate saving compared to the baseline window attention convolutional neural networks (WACNN) on the JPEGAI, SIQAD, SCID, and MLSCID datasets on average. Our code is available at https://github.com/vpaHduGroup/SFTIP_SCC.

Existing quantization parameter (QP) adjustment methods in video coding often focus solely on coding efficiency and ignore the impact of bit-rate fluctuations on video transmission and bandwidth waste. This is mainly because intra pictures, in a hierarchical coding structure, are allocated smaller QP and thus consume more bits. To address this issue, we propose an adaptive block-level QP adjustment method. Specifically, intra picture importance (IPI) is first introduced to evaluate the adjustability of intra picture QP. For intra pictures whose QP can be adjusted, we further propose block importance (BI) to determine their optimal block-level QP adjustment. Experimental results show that our proposed method reduce the bit-rate fluctuations while basically maintaining the coding performance. Notably, significant improvements can be observed in high-resolution videos, with a reduction of approximately 11% in bit-rate fluctuations.

Video Coding for Machines (VCM) aims to compress visual signals for machine analysis. However, existing methods only consider a few machines, neglecting the majority. Moreover, the machine's perceptual characteristics are not leveraged effectively, resulting in suboptimal compression efficiency. To overcome these limitations, this paper introduces Satisfied Machine Ratio (SMR), a metric that statistically evaluates the perceptual quality of compressed images and videos for machines by aggregating satisfaction scores from them. Each score is derived from machine perceptual differences between original and compressed images. Targeting image classification and object detection tasks, we build two representative machine libraries for SMR annotation and create a large-scale SMR dataset to facilitate SMR studies. We then propose an SMR prediction model based on the correlation between deep feature differences and SMR. Furthermore, we introduce an auxiliary task to increase the prediction accuracy by predicting the SMR difference between two images in different quality. Extensive experiments demonstrate that SMR models significantly improve compression performance for machines and exhibit robust generalizability on unseen machines, codecs, datasets, and frame types.

The rate-distortion optimized quantization (RDOQ) provides significant coding gain in the third generation of Audio Video coding Standard (AVS3). However, the high computational complexity and strong data dependency in RDOQ impede the hardware implementation. To address these issues, we propose a zig-zag scanline-level parallelized RDOQ algorithm and its fully pipelined hardware architecture for AVS3 video coding. For algorithm optimization, we update the run-level context for rate estimation in the inner zig-zag scanline and propose an efficient RD cost calculation form in the optimal coefficient level (OCL) decision step. In the last significant coefficient (LSC) position decision step, a greedy strategy based algorithm is proposed to optimize the determination process in parallel. Moreover, the proposed parallelized RDOQ algorithm is accelerated by single instruction multiple data (SIMD) on the Intel X86 platform. For hardware architecture design, a fully pipelined hardware architecture is proposed with nine pipeline stages. This design can process multiple transform units in parallel when the height is less than 32. Experimental results show that the proposed algorithm achieves 31.37%, 28.58%, and 28.53% time-saving by 0.25%, 0.26%, and 0.27% Bjøntegaard delta rate (BD-Rate) increase on average under all intra (AI), random access (RA), and low delay B (LDB) configurations, respectively. The hardware implementation achieves 32 coefficients per cycle, and the area consumption is 1223.2K logic gates when working at 471.2MHz. It is proven that the proposed algorithm and hardware architecture design achieve a good trade-off between coding efficiency and hardware throughput.

Recent years have witnessed strong demands for video composition in online video communications, enabling a series of new functionalities for video conferencing including virtual conference rooms, virtual reunions, and virtual backgrounds. In video composition, typically the foreground videos including the human bodies and faces are subject to compression due to the constrained bandwidth, whereas the virtual background is uncompressed and in pristine quality. The disharmony caused by the incoherent quality of foreground and background, which may worsen the quality of experience, has not been extensively studied. In this paper, we focus on this particular problem and present an image quality harmonization framework. Our principle is to align the quality of the background with that of the foreground such that they share similar levels of distortion. This is achieved by inferring the quantization parameter for background compression based on the foreground information. In particular, we aim to learn the quality and compression parameters in a self-supervised manner without laborious human annotation. Furthermore, a large dataset is constructed to provide sufficient training samples and testing scenarios for validation. The composite videos show superior harmonized quality in both quantitative and qualitative comparisons, demonstrating the effectiveness of the proposed framework.

The metaverse, a 3D virtual world, requires efficient interactive avatar communication. To achieve this goal, we envision a new metaverse paradigm for virtual avatar faces and develop semantic face compression with compact 3D facial descriptors. The paradigm comprises a compression framework that transmits 3D face descriptors for semantic compression and applications based on the semantic descriptors. The fundamental principle is that the communication of virtual avatar faces primarily emphasizes the conveyance of semantic information. In light of this, the proposed scheme offers the advantages of being highly flexible, efficient, and semantically meaningful. The promise of the proposed paradigm is also demonstrated by performance comparisons with the state-of-the-art video coding standard, Versatile Video Coding. A significant improvement in terms of rate-accuracy performance has been achieved. The proposed scheme is expected to enable numerous applications especially for real-time communication in the metaverse, such as digital human communication based on machine analysis, and to form the cornerstone of interactions.

How to compress face video is a crucial problem for a series of online applications, such as video chat/conference, live broadcasting and remote education. Compared to other natural videos, these face-centric videos owning abundant structural information can be compactly represented and high-quality reconstructed via deep generative models, such that the promising compression performance can be achieved. However, the existing generative face video compression schemes are faced with the inconsistency between the 3D facial motion in the physical world and the face content evolution in the 2D view. To solve this drawback, we propose a 3D-Keypoint-and-2D-Motion based generative method for Face Video Compression, namely FVC-3K2M, which can well ensure perceptual compensation and visual consistency between motion description and face reconstruction. In particular, the temporal evolution of face video can be characterized into separate 3D keypoints from the global and local perspectives, entailing great coding flexibility and accurate motion representation. Moreover, a cascade motion conversion mechanism is further proposed to internally convert 3D keypoints to 2D dense motion, enforcing the face video reconstruction to be perceptually realistic. Finally, an adaptive reference frame selection scheme is developed to enhance the adaptation of various temporal movements. Experimental results show that the proposed scheme can realize reliable video communication in the extremely limited bandwidth, e.g., 2 kbps. Compared to the state-of-the-art video coding standards and the latest face video compression methods, extensive comparisons demonstrate that our proposed scheme achieves superior compression performance in terms of multiple quality evaluations.

In this paper, a novel Diffusion-based 3D Pose estimation (D3DP) method with Joint-wise reProjection-based Multi-hypothesis Aggregation (JPMA) is proposed for probabilistic 3D human pose estimation. On the one hand, D3DP generates multiple possible 3D pose hypotheses for a single 2D observation. It gradually diffuses the ground truth 3D poses to a random distribution, and learns a denoiser conditioned on 2D keypoints to recover the uncontaminated 3D poses. The proposed D3DP is compatible with existing 3D pose estimators and supports users to balance efficiency and accuracy during inference through two customizable parameters. On the other hand, JPMA is proposed to assemble multiple hypotheses generated by D3DP into a single 3D pose for practical use. It reprojects 3D pose hypotheses to the 2D camera plane, selects the best hypothesis joint-by-joint based on the reprojection errors, and combines the selected joints into the final pose. The proposed JPMA conducts aggregation at the joint level and makes use of the 2D prior information, both of which have been overlooked by previous approaches. Extensive experiments on Human3.6M and MPI-INF-3DHP datasets show that our method outperforms the state-of-the-art deterministic and probabilistic approaches by 1.5% and 8.9%, respectively. Code is available at https://github.com/paTRICK-swk/D3DP.

In this paper, we propose to compactly represent the nonlinear dynamics along the temporal trajectories for talking face video compression. By projecting the frames into a high dimensional space, the temporal trajectories of talking face frames, which are complex, non-linear and difficult to extrapolate, are implicitly modelled in an end-to-end inference framework based upon very compact feature representation. As such, the proposed framework is suitable for ultra-low bandwidth video communication and can guarantee the quality of the reconstructed video in such applications. The proposed compression scheme is also robust against large head-pose motions, due to the delicately designed dynamic reference refresh and temporal stabilization mechanisms. Experimental results demonstrate that compared to the state-of-the-art video coding standard Versatile Video Coding (VVC) as well as the latest generative compression schemes, our proposed scheme is superior in terms of both objective and subjective quality at the same bitrate. The project page can be found athttps://github.com/Berlin0610/CTTR.

There has been an increasing consensus that the machine vision is gradually replacing human vision in numerous tasks, with the demonstrated success of artificial intelligence. In this paper, we propose a deep image compression scheme towards machine vision, with the principle of “begin with the end in mind”. In particular, a unified optimization scheme for end-to-end image compression towards machine vision is proposed, accompanied with the dedicated variable bitrate coding and generalized rate-accuracy optimization. The presented framework, which jointly optimizes the compression and the machine vision networks, exploits the utmost potential of robust machine vision for compressed images. The variable bitrate modules towards machine vision, which effectively shrink the storage space for model parameters, are further developed to accommodate to the real-world applications. Moreover, an iterative algorithm is presented to achieve the optimality in terms of the generalized rate-accuracy towards machine vision. Experimental results show that the proposed framework achieves the state-of-the-art object detection performance among the end-to-end image compression methods: in the exploration of Video Coding for Machines (VCM) in Moving Picture Experts Group (MPEG), and the proposed framework achieves 31.69% and 23.96% BD-rate gains compared with the VCM official test datasets, the Open Images dataset and the TVD dataset respectively, which are generated using the state-of-the-art standard Versatile Video Coding (VVC) standard. The generalization capability of the proposed framework is also verified with instance segmentation under various scenarios.

We propose a talking face video compression framework by implicitly transforming the temporal evolution into compact feature representation. More specifically, the temporal evolution of faces, which is complex, non-linear and difficult to extrapolate, is modelled in an end-to-end inference framework based upon very compact features. This enables the high-quality rendering of the face videos, which benefits from the learning of dense motion map with compact feature representation. Therefore, the proposed framework can accommodate ultra-low bandwidth video communication and maintain the quality of the reconstructed videos. Experimental results demonstrate that compared with the state-of-the-art video coding standard Versatile Video Coding (VVC) as well as the latest generative compression scheme Face Video-to-Video Synthesis (Face_vid2vid), the proposed scheme is superior in terms of both objective and subjective quality assessment methods.

Recently, there has been a tremendous demand for high-efficiency face video communications, coinciding with the popularization of the digital human character in numerous applications. This paper demonstrates a new communication paradigm of 3D human digital characters in ultra low-bit-rate application scenarios. The paradigm is grounded on the mild assumption of the consistency and persistence of human ap-pearance, such that only the compact features that determine the pose and expression of the 3D character need to be transmitted. The proposed is also expected to benefit virtual-physical world interaction in Metaverse.

Among the recent deep image compression frameworks, transform coding together with a context-adaptive entropy model is the most representative approach to achieve the best coding performance. For entropy model, 2D mask convolution is widely utilized to capture the spatial context, which omits the correlations along channel dimension. To complement to the spatial context, a cross channel context model is proposed. For transform, if given more network layers to improve its representation ability, how to allocate these network layers in forward and inverse transform is investigated. After analyzing the scheme of deep image compression connected with loop filter, we find this investigation can be regarded as a more generalized loop filter. The proposed cross channel context model and generalized loop filter (CCCMGLF) are integrated into the deep image compression framework and jointly optimized to improve the coding performance. Experimental results demonstrate that, using PSNR as distortion metric, the proposed CCCMGLF outperforms VTM-11.0 by 1.20%, 10.82% and 5.38% in terms of BD-rate reductions for Y, U and V components, respectively, for the Kodak dataset. For the JVET CTC sequences, the proposed method outperforms VTM-11.0 by 1.44% for Y but has a coding performance loss of 24.74% and 11.91% for U and V, respectively. Over the baseline deep compression framework, the proposed method provides 7.80%, 12.66% and 11.15% performance improvement for Y, U, and V, respectively, for the Kodak dataset; 9.10%, 12.27%, and 12.68% performance improvement for Y, U and V, respectively, for the JVET CTC sequences. The proposed approaches are applicable in both image compression and intra coding in video compression.

Face videos own abundant structured information and prior knowledge which can be utilized by generative neural networks to achieve ultra-low bitrate compression. However, generative neural network based face video compression suffers from large head motion which may easily result in deformed images. In this paper, the dynamic multi-reference prediction method is proposed for generative face video compression. Specifically, key map is extracted as the compact latent to represent the face image. The key maps of the current frame and multiple reference frames are used together to estimate multiple dense motion maps. The multiple motion maps are further applied to the corresponding reference frames to generate the final prediction of the current frame. Moreover, the reference frame can be dynamically refreshed during encoding to convert large head motion to relatively small motion. Experimental results show that the proposed method achieves superior compression performance compared to the state-of-the-art VVC standard as well as the latest generative face compression frameworks.

As the latest video coding standard, Versatile Video Coding (VVC) achieves up to 40% Bjontegaard delta bit-rate (BD-rate) reduction compared with High Efficiency Video Coding (HEVC). Recently, Convolutional Neural Network (CNN) has attracted tremendous attention and shows great potential in video coding. In this paper, we design a Multi-Density Convolutional Neural Network (MDCNN) as an integrated in-loop filter to improve the quality of the reconstructed frames. The core of our approach is the multi-density block (MDB), which contains two branches: (a) the basic branch maintaining full resolution for capturing spatially-precise representations, (b) the density branch for learning rich spatial correlation with larger receptive field through down-sampling and upsampling. The feature maps of two branches will be fused into one stream repeatedly. Benefiting from this architecture, spatially-precise representations and density correlations in larger receptive field are utilized to improve the model performance and promote the model's robustness to different input resolutions. Experimental results show that, in terms of BD-rate savings for the (Y, U, V) components compared to the state-of-the-art VVC standard, the proposed MDCNN filter can achieve (5.06%, 13.86%, 13.76%) and (4.36%, 10.85%, 10.91 %) coding gain for the Random Access (RA) configuration and the All Intra (AI) configuration, respectively.

Video chat becomes more and more popular in our daily life. However, how to provide a high-quality video chat with the limited bandwidth is a key challenging task. In this paper, beyond the state-of-the-art video compression system, we propose an encoder-decoder joint enhancement algorithm for the video chat. In particular, the sparse map of the original frame is extracted at the encoder side and signaled to the decoder, which is utilized together with the sparse map of the decoded frame to obtain the boundary transformation map. In this manner, the boundary transformation map represents the key difference between the original frame and the decoded frame and hence can be used to enhance the decoded frame. Experimental results show that the proposed algorithm brings clear subjective and objective quality improvements. At the same quality, the proposed algorithm can achieve 35% bitrate savings compared to the VVC.

Motion compensation has been widely employed for removing temporal redundancies in typical hybrid video coding framework. The popular video compression standards, such as H.264/AVC and HEVC, adopt the block based partitioning model to describe the motion filed due to its high compression efficiency and relatively low computational complexity. However, block based motion compensation may not align with the actual object motion boundaries, potentially limiting the compression efficiency. In view of this, we propose a three-zone segmentation based motion compensation scheme to improve the description accuracy of motion field as well as the coding efficiency. In particular, the segmentation information is implied in the reference frame instead of being explicitly signalled. Based on the segmentation information, three motion compensation zones can be identified, including one edge, one foreground, and one background zone. The foreground zone is motion compensated by the signalled motion vector of the block, and the background zone is motion compensated by the motion information implicitly derived from the local motion field. Regarding the edge zone, it is viewed as an overlapped area and the weighted compensation strategy is adopted. The proposed algorithm is implemented into the reference software VTM-1.0 of Versatile Video Coding (VVC), and simulation results show that the algorithm can achieve 1.14% and 1.06% bitrate savings for random access and lowdelay configurations, respectively.

Motion compensation plays a key role in high efficiency video coding. The popular video compression standards, such as H.264/AVC and HEVC, adopt block based motion compensation technique due to its high compression efficiency and relatively low computational complexity. However, block based motion compensation may not be in accordance with the actual object boundary, potentially leading to low prediction accuracy especially in the high-texture areas. In this paper, we propose a locally refined motion compensation method to address this issue. In particular, the image segmentation is applied on the prediction block indicated by a motion vector rather than the original block to avoid explicit signaling. Furthermore, the local content is analyzed to select one segmented region and subsequently the prediction of this region is generated based on the local motion filed. Experimental results show that the proposed algorithm can achieve 0.8%, 1.1% and 1.7% bitrate savings for Random Access, Lowdelay-B and Lowdelay-P configurations respectively without introducing noticeable computational complexity.

Traditional video coding standards H.264/AVC and HEVC adopt inter-frame prediction with single prediction (P frame) and bi-directional prediction (B frame), in which one and two hypotheses are utilized to generate the final prediction. Though more hypotheses have been proved to significantly improve the prediction accuracy, the additional bits used for signaling the motion information may decrease the overall rate-distortion performance conversely. In this paper, we propose a multi-hypothesis prediction scheme based on implicit derivation of motion vector. In the proposed scheme, the first prediction block is generated identically with the method in HEVC. Subsequently, motion estimation is applied on each reference frame by taking the first prediction block as template, and the search results serve as the additional hypothesis candidates. In this manner, the overheads of motion vector can be reduced. Simulation results show that the proposed algorithm can achieve average 1.1% coding gain compared to HEVC.

In the latest Joint Video Exploration Team (JVET) development, the quadtree plus binary tree (QTBT) block partition structure is proposed for more flexible block partitioning. Compared to the quadtree partitioning in HEVC, QTBT can achieve better compression performance at the expense of significantly increased encoding complexity. To address this issue, we propose a convolution neural network (CNN) oriented fast QTBT partitioning decision algorithm for inter coding. We analyze the QTBT in a statistical way, which effectively guides us to design the architecture of the CNN. Furthermore, the false prediction risk is controlled based on temporal correlation to improve the robustness of the scheme. Experimental results show that the proposed algorithm can speed up QTBT block partition structure by reducing 35% encoding time on average with only 0.55% increase in bit rate, which enables its applications in practical scenarios.

In the latest Joint Video Exploration Team development, the quadtree plus binary tree (QTBT) block partitioning structure has been proposed for future video coding. Compared to the traditional quadtree structure of High Efficiency Video coding (HEVC) standard, QTBT provides more flexible patterns for splitting the blocks, which results in dramatically increased combinations of block partitions and high computational complexity. In view of this, a confidence interval based early termination (CIET) scheme is proposed for QTBT to identify the unnecessary partition modes in the sense of rate-distortion (RD) optimization. In particular, a RD model is established to predict the RD cost of each partition pattern without the full encoding process. Subsequently, the mode decision problem is casted into a probabilistic framework to select the final partition based on the confidence interval decision strategy. Experimental results show that the proposed CIET algorithm can speed up QTBT block partitioning structure by reducing 54.7% encoding time with only 1.12% increase in terms of bit rate. Moreover, the proposed scheme performs consistently well for the high resolution sequences, of which the video coding efficiency is crucial in real applications.

Block partition structure has been recognized as a crucial module in video coding scheme. Recently, a quadtree plus binary tree (QTBT) block partition structure has been proposed in the Joint Video Exploration Team (JVET) development. Compared to the quadtree structure in HEVC, QTBT can achieve better coding performance with hugely increased encoding complexity. Here, we propose an effective QTBT partition decision algorithm to achieve a good trade-off between computational complexity and coding performance. In particular, at the Coding Tree Unit level, the partition parameters of QTBT are dynamically derived to adapt to the local characteristics without transmitting any overhead. Subsequently, at the Coding Unit level, a joint-classifier decision tree structure is designed to eliminate unnecessary iterations and meanwhile control the risk of false prediction. Experimental results show that the proposed algorithm can achieve 64% encoding time reduction on average with only 1.26% increase in terms of bit rate. This greatly benefits the practical implementations of QTBT in real application scenarios.

In the state-of-the-art H.265/HEVC video coding standard, the motion vector (MV) resolution is fixed to be 1/4-pel for the entire video sequence, while the inherent video characteristics, e.g., texture complexities and motion activities, have been largely ignored. Obviously such strategy may not suffice the demand of high-accuracy motion compensation. In this paper, we propose a specially designed rate-distortion model in terms of the MV resolution by taking the video characteristics into consideration. In particular, the MV resolution selection is formulated as a rate-distortion optimization problem by analyzing the rate-distortion cost of each MV resolution candidate. To further improve the coding performance, the progressive MV resolution strategy is employed, where the optimal progressive MV resolution is determined by decision trees constructed with the rate-distortion model. In this manner, a novel adaptive progressive motion vector resolution selection scheme can be realized and the MV resolution can be adaptively adjusted based on the properties of local content. Extensive experiments and comparisons show that the proposed algorithm significantly improves the coding performance, and 1.8% BD-rate gain on average has been achieved without introducing any noticeable computational complexity.

In the state-of-the-art H.265/HEVC video coding standard, the motion vector is always fixed to be 1/4-pixel resolution for the entire video sequence regardless of the different video contents, which is not efficient for prediction coding. In this paper, we propose a frame level adaptive motion vector resolution selection scheme based on a rate-distortion model in terms of motion vector resolution. In the proposed rate-distortion model, the relationship between the distortion and the motion vector resolution is approximated with a linear model. And a rate model of motion vector is built, which reflects the relationship between the coding bits of motion vector and its value. With the proposed rate-distortion model, an optimal motion vector resolution minimizing the total rate-distortion cost will be selected for each frame. Experimental results show that the proposed scheme can achieve 1.5%, 1.3% and 2.5% BD-rate gain on average for Random Access, Lowdelay-B and Lowdelay-P configurations without complexity increment.

In the latest Joint Video Exploration Team (JVET) development, a quadtree plus binary tree (QTBT) block partition structure is proposed for enhanced video coding. Compared to the quadtree block partition in HEVC, QTBT can achieve much better compression performance at the expense of largely increased encoding computational complexity. In this paper, a fast QTBT block partition algorithm based on the local constraint is proposed. Given the decoded information of the previous frame, the partition parameters can be dynamically derived for each coding tree unit (CTU) without any overhead. Furthermore, a constrained binary tree partition algorithm is proposed to reduce the redundancy between the quadtree partition and the binary tree partition. Experimental results have shown that the proposed scheme can speed up QTBT block partition structure by 30% on average without obvious performance decrease, which enables its practical implementations in real application scenarios.

In the classical block-based video coding, motion vector is derived for each coding block to remove the inter-frame redundancy. However, the motion vector resolution is usually restricted to be identical, typically 1/4-pixel resolution, regardless of the different video contents. In this paper, we propose an algorithm that can adaptively select the optimal motion vector resolution at frame level according to the characteristics of the video contents. We first derived a residual energy model, and the major factors that may impact the motion vector resolution are considered, including the texture complexity, motion scale, inter-frame noise and quantization parameter. Experimental results have shown that the proposed scheme can achieve 1.8% BD-rate gain on average without complexity increment.