Nowadays, high-quality images are pursued by both humans for better viewing experience and by machines for more accurate visual analysis. However, images are usually compressed before being consumed, decreasing their quality. It is meaningful to predict the perceptual quality of compressed images for both humans and machines, which guides the optimization for compression. This issue remains underexplored. In this paper, we propose a unified approach to fill this gap. Specifically, we create a deep learning-based model to predict Satisfied User Ratio (SUR) and Satisfied Machine Ratio (SMR) of compressed images simultaneously. We first pre-train a feature extractor network on a large-scale SMR-annotated dataset with human perception-related quality labels generated by diverse image quality models, which simulates the acquisition of SUR labels. Then, we leverage and fuse the extracted multi-layer features to predict SUR and SMR with a more capable network architecture. We propose a Difference Feature Residual Learning (DFRL) module to learn more discriminative difference features. We also design a Multi-Head Attention Aggregation and Pooling (MHAAP) layer to aggregate difference features and reduce their redundancy. We further introduce an MLP-Mixer module to integrate global spatial and channel information for subsequent SUR and SMR regression. Experimental results indicate that the proposed model significantly outperforms state-of-the-art SUR and SMR prediction methods by up to 48% on several challenging datasets. Moreover, our joint learning scheme of human and machine perceptual quality prediction tasks is effective at improving the performance of both.

This paper elaborates the binary bin distribution of each syntax element in AVS3. The overall syntax elements in AVS3 can be divided into Context-Modeled(CM) Bin, Bypass(BPS) Bin, Weighted Context-Modeled(WCM) Bin. A comprehensive and quantitative analysis on these bitstreams are conducted and the result is shown in Fig. 1. Moreover, Fig. 2 illustrates the distribution of CM Bin, BPS Bin, and WCM Bin across various resolutions and QPs. The analysis reveals insightful correlations between the types of binary symbols and syntax elements within the AVS3 bitstream. Through this analysis, we aim to contribute to the implementation of AVS3 codecs.

This paper propose a Image Coding for Machine (ICM) framework with Visual-Language Mimic Feature Learning (VLM-ICM). VLM-ICM decouples the position and semantic information into language modality and extracts universal features from the input image. Language, inherently more semantically compact, helps reduce the bitrate. Meanwhile, the universal features in VLM-ICM, guided by the language at the decoder side, allow for flexible domain adaptation, thereby enhancing versatility and practicality.

Logarithmic binary arithmetic coding (LBAC) is a entropy coding method first introduced in AVS2 and now used in the third generation of audio video coding standard (AVS3). While LBAC provides high coding efficiency, the data dependencies in AVS3 make it challenging to parallelize parsing syntax element. To enhance the throughput of AVS3 LBAC entropy decoding engine, a sophisticated hardware implementation architecture is meticulously designed in the paper. The proposed method decouples the bitstream decoding process into four distinct modules. To ensure the decoding efficiency and performance, an advanced parallel pipeline is carefully designed. Furthermore, we develop sub-branch state prediction mechanism and grouped decoding method to further improve the decoding efficiency. The proposed AVS3 entropy decoder has been implemented on the S10 FPGA System Board, and experimental results show that the proposed method could achieve real-time and seamless decoding efficiency for 8K AVS3 bitstreams, even at bitrates as high as 207.1Mbps.

Existing learning-based methods often lack temporal coherence in low-light video enhancement due to rarely considering intrinsic temporal dependence. To address this issue, we propose the Long-short Temporal Filtering Network (TFNet) to learn the mapping from low-light videos to normal-light ones, utilizing the well-considered data-centric strategy and a refined architecture. From the data-centric temporal strategy, we incorporate both long-range and short-range temporal dependence into TFNet, effectively capturing the temporal information. From the model design perspective, the TFNet incorporates the Temporal-aware Attentional Filtering (TAF) module, which aims to estimate and adaptively combine filtering kernels for guided filtering towards features of the middle frame. To further refine the filtered features, the cascaded Grouped Attention (GA) blocks are presented in a grouped attention strategy. Experimental results on benchmark datasets have demonstrated the superiority of our TFNet against the state-of-the-art methods in terms of video frame quality and brightness consistency.

Navigation within Cave Automatic Virtual Environment (CAVE) systems often faces challenges due to limited physical space and the necessity for seamless user interaction. Traditional solutions typically rely on multi-view tracking systems or constrained locomotion techniques, which can interrupt immersion and hinder usability. In this paper, we introduce MoRLACS, a novel locomotion approach for CAVE systems that leverages a single RGBD camera. This hybrid framework integrates small-scale physical walking with controller-based large-scale exploration through a tailored guidance method. By accurately tracking the user's head position in the real world and synchronizing it with the virtual camera, MoRLACS enables natural walking within confined CAVE spaces and supports extended interaction in larger virtual environments. Preliminary user experiments demonstrate the approach's effectiveness, revealing improvements in usability and a heightened sense of presence. These findings underscore the potential of MoRLACS to enrich user experiences in immersive CAVE settings and offer valuable design insights for integrating 3D sensor data into multimedia interaction frameworks.

Video prediction aims to predict future frames by modeling the complex spatiotemporal dynamics in videos. However, most existing methods only model the temporal information and the spatial information for videos in an independent manner but have not fully explored the correlations between both terms. In this paper, we propose a SpatioTemporal-Aware Unit (STAU) for video prediction and beyond by exploring the significant spatiotemporal correlations in videos. On the one hand, the motion-aware attention weights are learned from the spatial states to help aggregate the temporal states in the temporal domain. On the other hand, the appearance-aware attention weights are learned from the temporal states to help aggregate the spatial states in the spatial domain. In this way, the temporal information and the spatial information can be greatly aware of each other in both domains, during which, the spatiotemporal receptive field can also be greatly broadened for more reliable spatiotemporal modeling. Experiments are not only conducted on video prediction tasks (deterministic and stochastic), but also another task beyond video prediction, the early action recognition task. Experimental results show that the proposed STAU can achieve satisfactory performance on all tasks compared with other methods.

Recent studies on Video Coding for Machine (VCM) have achieved remarkable results. However, in practical visual-analytic applications, except for transmitting the visual feature for machine analysis, video textures are also mandatory for human monitoring and decision-making. To this end, this paper proposes a human-machine friendly video compression scheme (HMFVC) which can satisfy both human viewing and machine analysis well. First, we propose a learned semantic representation (LSR) method to extract semantic information between temporal neighboring frames. LSR could be utilized in signal reconstruction for human viewing and visual analysis for machine understanding. Second, given the proposed LSR, we design an end-to-end optimized video compression framework to jointly optimize the visual quality for human perception, analysis accuracy for machines, and compression efficiency as well. Finally, an HMFVC codec is developed, which can achieve higher action recognition accuracy and better reconstruction quality than the traditional codecs and learned video compression approaches. Specifically, HMFVC saves 77% bitrate to achieve the same analysis performance with the original videos compared to x265. To our knowledge, HMFVC is the first end-to-end optimized video compression scheme to serve both humans and machines. It is a promising framework for human-machine friendly video compression approaches.

An ideal full-reference image quality (FR-IQA) model should exhibit both high separability for images with different quality and compactness for images with the same or indistinguishable quality. However, existing learning-based FR-IQA models that directly compare images in deep-feature space, usually overly emphasize the quality separability, neglecting to maintain the compactness when images are of similar quality. In our work, we identify that the perception bias mainly stems from an inappropriate subspace where images are projected and compared. For this issue, we propose a Debiased Mapping based quality Measure (DMM), leveraging orthonormal bases formed by singular value decomposition (SVD) in the deep features domain. The SVD effectively decomposes the quality variations into singular values and mapping bases, enabling quality inference with more reliable feature difference measures. Extensive experimental results reveal that our proposed measure could mitigate the perception bias effectively and demonstrates excellent quality prediction performance on various IQA datasets.

Screen Content Coding (SCC) is an indispensable tool for enabling distributed collaboration, such as video conferencing. Encoders in the latest video coding standards, particularly for SCC scenarios, employ a wider variety of partitioning tree splitting types, recursively traversing all branches, as well as a larger number of coding modes and submodes, to achieve higher coding efficiency compared to encoders in previous standards. This process leads to very high coding complexity, as each tree leaf node, called a coding unit (CU), for every partitioning size and location in the picture is repeatedly visited and evaluated multiple times during the optimal partitioning search. Additionally, each CU visit involves evaluating a vast number of coding options and their combinations to identify the best one. The complexity is further exacerbated in SCC due to the addition of many new CU coding modes and options. To significantly reduce SCC complexity without coding efficiency loss, this article proposes a new technique, Accelerated Revisit CU-coding (ARC), along with an SCC search space analysis for in-depth operation-level and run/platform-independent assessment of SCC complexity. ARC exploits the correlation between the first visit and subsequent revisits of a CU with the same location and size. By fully leveraging the correlation and information from the first visit, ARC significantly accelerates revisit CU-coding while maintaining the same high coding efficiency. ARC is implemented in HPM, the AVS3 reference software. Experiments demonstrate that ARC reduces encoding runtime by 29.74%, 47.78%, and 54.25% for 1,920 × 1,080 FHD, 4K UHD, and 8K UHD test sequences, respectively, in All Intra configuration, without coding efficiency loss. These runtime reductions align with corresponding search space reductions of 30.91%, 49.67%, and 54.41%, as obtained from the search space analysis.

Existing geometry-based Point Cloud Compression (PCC) frameworks are typically designed to code the geometric coordinates first, followed by compressing the attributes (e.g., colors, reflectances) according to the order derived from geometric structures, such as the Morton codes. Although geometry-based reordering methods can eliminate the redundancy of attributes, the errors caused by dramatic variations of attributes in the non-smooth areas potentially limit the efficiency of the point cloud attribute coding. To tackle this challenge, a novel joint structure-texture scan-order and coding scheme is proposed, which aims to explore a better attribute coding order from the viewpoint of improving the geometry-attribute consistency. Specifically, we formulate the attribute reordering problem as a geometry-attribute alignment task, and utilize the affine-transform model to find the optimal correspondence between geometry and attribute information by minimizing attributed prediction residuals. Then, the Morton codes-based point cloud reordering is conducted on the transformed point cloud. Note that our residual-based mode decision scheme implicitly embodies that the proposed reordering method further incorporates attribute textures based on the geometric structure. However, the exhaustive search for the optimal transformation space introduces the extremely high complexity to the encoder. Therefore, we also propose a fast pruning algorithm to narrow the search space for the approximate solution as an alternative. Experimental results conducted on the various benchmark datasets have illustrated that our proposed method outperforms the MPEG standard G-PCC with gains of up to 2%, 9%, and 7% in luma, chroma, and reflectance, respectively.

This paper proposes a fast four-parameter Affine Motion Compensation (AMC) algorithm. As shown in Fig. 1, the translation Motion Vector (MV) is derived by reusing the AMC sub-block MV derivation method firstly, which is used to conduct translation pre-transform. Secondly, a coordinate system whose coordinate origin is located on its top-left control point is established for the transformed block. Finally, the geometric relationship between two control point motion vectors (CPMVs) of the transformed block can be described as follows,\begin{equation*}\delta = \left| {\left({m{v_{0x}} - m{v_{1x}}}\right) \times H - \left({m{v_{0y}} + m{v_{1y}}}\right) \times W} \right| = 0\tag{1}\end{equation*}

End-to-end image compression has achieved satisfactory results in recent studies. However, existing methods suffer from high complexity of complicated neural network computation and cannot be directly deployed on mobile devices due to the limitations of computing ability and storage. Therefore, considering the resource and computing ability constrains of the mobile devices, we make a trade-off in this paper between rate-distortion (R-D) performance, inference time, and model complexity. Then we design a novel lightweight perceptual image compression framework to alleviate the storage and complexity burden of mobile devices. Moreover, we design a hardware-friendly deployment scheme to apply the proposed compression framework on high-end mobile devices, which can achieve efficient image compression. Based on the above structures, we propose the first mobile system that achieves image compression on mobile devices. The supplementary material of our system demo is on https://sigport.org/documents/extreme-low-bitrate-Image-compression-system-mobile-deployment.

In the fourth generation Audio Video coding Standard (AVS4), the Inter Prediction Filter (INTERPF) reduces discontinuities between prediction and adjacent reconstructed pixels in inter prediction. The paper proposes a low complexity learning-based inter prediction (LLIP) method to replace the traditional INTERPF. LLIP enhances the filtering process by leveraging a lightweight neural network model, where parameters can be exported for efficient inference. Specifically, we extract pixels and coordinates utilized by the traditional INTERPF to form the training dataset. Subsequently, we export the weights and biases of the trained neural network model and implement the inference process without any third-party dependency, enabling seamless integration into video codec without relying on Libtorch, thus achieving faster inference speed. Ultimately, we replace the traditional handcraft filtering parameters in INTERPF with the learned optimal filtering parameters. This practical solution makes the combination of deep learning encoding tools with traditional video encoding schemes more efficient. Experimental results show that our approach achieves 0.01%, 0.31%, and 0.25% coding gain for the Y, U, and V components under the random access (RA) configuration on average.

Video Coding for Machines (VCM) aims to compress visual signals for machine analysis. However, existing methods only consider a few machines, neglecting the majority. Moreover, the machine's perceptual characteristics are not leveraged effectively, resulting in suboptimal compression efficiency. To overcome these limitations, this paper introduces Satisfied Machine Ratio (SMR), a metric that statistically evaluates the perceptual quality of compressed images and videos for machines by aggregating satisfaction scores from them. Each score is derived from machine perceptual differences between original and compressed images. Targeting image classification and object detection tasks, we build two representative machine libraries for SMR annotation and create a large-scale SMR dataset to facilitate SMR studies. We then propose an SMR prediction model based on the correlation between deep feature differences and SMR. Furthermore, we introduce an auxiliary task to increase the prediction accuracy by predicting the SMR difference between two images in different quality. Extensive experiments demonstrate that SMR models significantly improve compression performance for machines and exhibit robust generalizability on unseen machines, codecs, datasets, and frame types.

Recent years have witnessed a variety of applications of gaming video coding, while how to improve the coding efficiency has been relatively under-explored. The state-of-the-art video coding standard, Versatile Video Coding (VVC), adopts the Reference Picture Resampling (RPR) which allows the variation of the frame resolutions in encoding/decoding. The great flexibility supported by RPR motivates us to develop a bidirectional quality impulse based guidance scheme, in an effort to fully exploit the potential of RPR in gaming video coding. The design philosophy involves reducing the data volume on the encoder side through selective downsampling, and enhancing reconstruction by harnessing quality conveyance from neighboring frames. More specifically, a new RPR structure is developed based on the underlying philosophy that the periodic quality impulse could promisingly boost the quality of the whole sequence. On top of the developed structure, we propose a bidirectional guidance model that faithfully enhances video quality by resorting to frames with quality impulse. Experimental results exhibit the proposed scheme can achieve significant bit-rate savings for gaming videos.

Cross-modal compression (CMC) aims to compress highly redundant visual data into compact, common, and human-comprehensible domains, such as text, to preserve semantic fidelity. However, CMC is limited by a constant level of semantic fidelity and constrained semantic fidelity due to a single compression domain (plain text). To address these issues, we propose a new approach called Multiple-domains rate-distortion optimized CMC (M-CMC). Specifically, our method divides the image into two complementary representations: (1) a structure representation with an edge map, and (2) a texture representation with dense captions, which include numerous region-caption pairs instead of plain text. In this way, we expand the single domain to multiple domains, namely, edge maps, regions, and text. To achieve diverse levels of semantic fidelity, we suggest a rate-distortion reward function, where the distortion measures the semantic fidelity of reconstructed images and the rate measures the information content of the text. We also propose Multiple-stage Self-Critical Sequence Training (MSCST) to optimize the reward function. Extensive experimental results demonstrate that the proposed method achieves diverse levels of semantic translation more effectively than other CMC-based methods, achieves higher semantic compression performance compared to traditional block-based and learning-based image compression frameworks with 97,000-500 times compression ratio, and provides a simple yet effective way for image editing.

Perceptual Lossless Compression (PLC) is a novel compression standard directed by the Audio Video coding Standard (AVS) work group. It defines a lightweight, low-latency, and visually lossless image compression framework, which offers an alternative mezzanine codec for most user-agnostic on-chip compression scenarios, alleviating the tension between growing transmission demands and expensive integration upgrades. In this paper, the technical designs in the development process of PLC will be fully introduced. The balance between feature modeling and ASIC implementation costs will be present throughout. A high throughput and low hardware complexity implementation will be detailed and evaluated namely HIM. Hopefully, the design of the PLC standard and HIM framework will bring new inspiration for the emerging low-latency interaction systems.

To combine the advantages of deterministic and probabilistic 3D human pose estimation methods, we decompose pose estimation into two processes: hypotheses generation and hypotheses aggregation. For hypotheses generation, we propose a novel Diffusion-based 3D Pose generation (D3DP) method. D3DP generates a diversified group of plausible 3D pose hypotheses from a single 2D keypoint observation. Utilizing a diffusion process, it gradually transforms ground-truth 3D poses towards a random distribution, subsequently employing a conditioned denoiser guided by the observed keypoints to recover the uncorrupted 3D poses. Moreover, D3DP is compatible with existing deterministic 3D pose estimators and allows users to optimize the trade-off between computational efficiency and pose accuracy via two adjustable parameters. For hypotheses aggregation, we propose two alternative approaches: a Reprojection-Based Selection (RBS) method and a Hypotheses Selection Network (HSN). These methods adopt the joint-level strategy to assemble multiple hypotheses generated by D3DP into a single 3D pose for practical use. Specifically, RBS reprojects 3D pose hypotheses to the 2D camera plane, and selects the best hypothesis based on the reprojection errors. HSN evaluates each hypothesis and selects the hypothesis with the highest confidence score as the output. Then these selected joints are combined into the final pose. The proposed methods implement a joint-by-joint aggregation strategy that capitalizes on the 2D prior and temporal information, both of which have been ignored by previous pose-level methods. Extensive experiments on two benchmarks highlight that the proposed method outperforms the state-of-the-art deterministic and probabilistic approaches.

Pixel-level road crack detection has always been a challenging task in intelligent transportation systems. Due to the external environments, such as weather, light, and other factors, pavement cracks often present low contrast, poor continuity, and different sizes in length and width. However, most of the existing studies pay less attention to crack data under different situations. Meanwhile, recent algorithms based on deep convolutional neural networks (DCNNs) have promoted the development of cutting-edge models for crack detection. Nevertheless, they usually focus on complex models for good performance, but ignore detection efficiency in practical applications. In this article, to address the first issue, we collected two new databases (i.e. Rain365 and Sun520) captured on rainy and sunny days respectively, which enrich the data of the open source community. For the second issue, we reconsider how to improve detection efficiency with excellent performance, and then propose our lightweight encoder-decoder architecture termed CarNet. Specifically, we introduce a novel olive-shaped structure for the encoder network, a lightweight multi-scale block and a new up-sampling method in the decoder network. Numerous experiments show that our model can better balance detection performance and efficiency compared with previous models. Especially, on the Sun520 dataset, our CarNet significantly advances the state-of-the-art performance with ODS F-score from 0.488 to 0.514. Meanwhile, it does so with an improved detection speed (104 frames per second) which is orders of magnitude faster than some recent DCNNs-based algorithms specially designed for crack detection.

Traditional image/video compression compresses the highly redundant visual data while preserving signal fidelity. Recently, cross modal compression (CMC) is proposed to compress the data into a compact, human-comprehensible domain (such as text) with an ultra-high compression ratio while preserving semantic fidelity for machine analysis and semantic monitoring. CMC is with a constant rate because the CMC encoder can only represent the data with a fixed grain. But in practice, variable rate is necessary due to the complicated and dynamic transmission conditions, the different storage mediums, and the diverse levels of application requirements. To deal with this problem, in this paper, we propose variable rate cross modal compression (VR-CMC), where we introduce variable rate prompt to represent the data with different grains. Variable rate prompt is composed of three strategies. Specifically, 1) target length prompt (TLP) introduces the target length into the language prompt to guide the generation of the text representation; 2) decaying EOS probability (DEP) exponentially decays the probability of the EOS token with regard to the decoding step and target length, where the EOS (end-of-sequence) token is a special token indicating the end of the text; 3) text augmentation (TA) enriches the training data and makes the text representation length more balanced when training. Experimental results show that our proposed VR-CMC can effectively control the rate in the CMC framework and achieve state-of-the-art performance on MSCOCO and IM2P datasets.

Advanced compression technologies have succeeded in exploring compact representations of intricate video contents, but their dramatic increase in computational complexity causes severe challenges when deploying new-generation video codecs. In this article, we revisited the essential compression characteristics of IEEE1857.10/AVS3 from the perspective of reducing the computational cost of different application scenarios as well as its merits in terms of accommodating parallel implementation in high-performance multicore computing platforms by leveraging the scalable video technology (SVT) architecture. A hybrid acceleration scheme is constructed to extract texture and contextual information for pruning massive encoding candidates, while the visual quality of the reconstructed images is well considered in terms of adaptive coding budgets. Furthermore, we have carefully studied the compression performance of existing AVS3 coding tools within different technical parameters and rate-distortion granularity, where the parametric encapsulation and resource scheduling derived a series of preset configurations capable of offering satisfactory trade-offs for various applications. Therefore, we propose the first and fastest AVS3 standard-compliant encoder with the capability of real-time processing video signals up to 8 K resolution, which may hopefully further benefit the emerging 8K-UHD video industry.

As a highly ill-posed issue, single-image super-resolution (SISR) has been widely investigated in recent years. The main task of SISR is to recover the information loss caused by the degradation procedure. According to the Nyquist sampling theory, the degradation leads to the aliasing effect and makes it hard to restore the correct textures from low-resolution (LR) images. In practice, there are correlations and self-similarities among the adjacent patches in the natural images. This article considers the self-similarity and proposes a hierarchical image super-resolution network (HSRNet) to suppress the influence of aliasing. We consider the SISR issue in the optimization perspective and propose an iterative solution pattern based on the half-quadratic splitting (HQS) method. To explore the texture with local image prior, we design a hierarchical exploration block (HEB) and progressive increase the receptive field. Furthermore, multilevel spatial attention (MSA) is devised to obtain the relations of adjacent feature and enhance the high-frequency information, which acts as a crucial role for visual experience. The experimental result shows that HSRNet achieves better quantitative and visual performance than other works and remits the aliasing more effectively.

Recently, cross modal compression (CMC) is proposed to compress highly redundant visual data into a compact, common, human-comprehensible domain (such as text) to preserve semantic fidelity for semantic-related applications. However, CMC only achieves a certain level of semantic fidelity at a constant rate, and the model aims to optimize the probability of the ground truth text but not directly semantic fidelity. To tackle the problems, we propose a novel scheme named rate-distortion optimized CMC (RDO-CMC). Specifically, we model the text generation process as a Markov decision process and propose rate-distortion reward which is used in reinforcement learning to optimize text generation. In rate-distortion reward, the distortion measures both the semantic fidelity and naturalness of the encoded text. The rate for the text is estimated by the sum of the amount of information of all the tokens in the text since the amount of information of each token is a lower bound of coding bits. Experimentally, RDO-CMC effectively controls the rate in the CMC framework and achieves competitive performance on MSCOCO dataset.

Lightweight video compression has effectively alleviated the tension between growing transmission demands and expensive integration upgrades. Effective rate control algorithms are believed to be the crucial bottleneck for quality improvement during those ultra-high throughput coding processes. This paper proposes a novel rate control (RC) scheme that constructs a contextual adaptive bit estimation model through clustering historical compression information into block-gradient complexity categories. A buffer-aware tuning method and a flexible quantization parameter (QP) mapping algorithm are designed to determine the Luma/Chroma QP distribution where a simplified Lagrangian multiplier is further defined to preserve the stability of the overall compression process. As a result, the constant-bitrate compression towards low-latency interoperable ASICs is implemented with a promising RC performance.

This paper proposes a novel quantization parameter (QP) derivation module that constructs several spatiotemporal characteristics into key-frame QP determination through an efficient pre-analysis progress. A series of simplified prediction modes and a histogram statistic are employed to model the reference quality that key-frames provide to subsequent frames. An adaptive delta-QP value is generated to address the conflict between the low compression efficiency of intra-only frames and the critical predictive basis of temporal-underlying frames. The experimental result shows that the proposed method reduces 41.91% of peak-to-valley bitrate difference while leading to a 0.02% BDBR performance change, indicates that high-quality key-frames may not be indispensable in nowadays video compression frameworks. The proposed method has shown that the pre-analysis based QP optimization for intra-only frames is promising for the enhancement of transmission bandwidth utilization, which may hopefully provide new inspiration for bit allocation and rate control designs.

In this paper, a novel Diffusion-based 3D Pose estimation (D3DP) method with Joint-wise reProjection-based Multi-hypothesis Aggregation (JPMA) is proposed for probabilistic 3D human pose estimation. On the one hand, D3DP generates multiple possible 3D pose hypotheses for a single 2D observation. It gradually diffuses the ground truth 3D poses to a random distribution, and learns a denoiser conditioned on 2D keypoints to recover the uncontaminated 3D poses. The proposed D3DP is compatible with existing 3D pose estimators and supports users to balance efficiency and accuracy during inference through two customizable parameters. On the other hand, JPMA is proposed to assemble multiple hypotheses generated by D3DP into a single 3D pose for practical use. It reprojects 3D pose hypotheses to the 2D camera plane, selects the best hypothesis joint-by-joint based on the reprojection errors, and combines the selected joints into the final pose. The proposed JPMA conducts aggregation at the joint level and makes use of the 2D prior information, both of which have been overlooked by previous approaches. Extensive experiments on Human3.6M and MPI-INF-3DHP datasets show that our method outperforms the state-of-the-art deterministic and probabilistic approaches by 1.5% and 8.9%, respectively. Code is available at https://github.com/paTRICK-swk/D3DP.

Rate control plays an essential role in real video applications due to the limitation of the bandwidth and storage budget. Supported by reference picture resampling (RPR) in Versatile Video Coding (VVC), which enables resolution changes without introducing an intra-coded frame, it is possible to adjust picture resolutions to adapt to the required bit-rate. However, few rate control algorithms with resolution changes have been explored for VVC. Therefore, we propose a$\lambda$-domain rate control algorithm with resolution changes in this paper. In the proposed rate control scheme, the coding resolution of one frame is determined based on the property of the frame. To make the rate control scheme adaptive to mixed resolutions, the rate control parameters of the full resolution and the low resolution are updated separately. Furthermore,$\lambda$and QP of the frames with different resolutions are unified to obtain appropriate coding parameters. Simulation results show that a BD-rate coding gain about 3.7% on average can be achieved with the reduction of computational complexity. Besides, negligible bitrate error can be observed at both sequence and frame level.

Audio Video Coding Standard (AVS) Intelligent Coding Group mainly studies video coding tools based on neural network technology and its potential benefit for next generation video coding. Extensive efforts have been dedicated to the research on neural network (NN) based coding tools. In this paper, we present a novel NN based video coding framework by leveraging the supervised trained NN models for multiple modules in the hybrid coding framework, from the predictive coding to the in-loop filtering. Specifically, NN based intra prediction models the non-linear mapping from contextual pixels to the predictions. The inter prediction efficiency is enhanced by introducing a virtual reference frame (VRF) network. The convolutional neural network based loop filtering (CNNLF) with discriminative model selection exploits the texture adaptivity. The experimental results show that the CNNLF, NN Intra, and VRF models can bring 8.60%, 1.02%, and 2.26% luma BD-rate reduction under random access (RA) configuration compared with AVS reference software HPM13.0. Additional experiments with the combined three NN coding tools reveal that around 13% YUV BD-rate reduction could be obtained. The proposed framework opens novel sights for next generation video coding from the intelligent coding perspective.

Intra prediction, which aims to remove the redundancies within a frame, has shown promising performance by simply projecting and interpolating samples along multiple angular directions. Recently, with numerous approaches devoted to learning nonlinear predictors with deep neural networks (DNN) based on local correlations, much less work has been dedicated to exploring non-local self-similarities in intra prediction. In this paper, we propose a unified prediction model that exploits both local and non-local correlations for intra prediction. The proposed model not only supports the nonlinear prediction using local reference samples as input, but also aggregates useful non-local information from a large reconstructed region with a Patch-level Non-local Attention Network (PNA-Net). More specifically, PNA-Net incorporates template matching with attention mechanism in feature domain to obtain the responses of all non-local features to the content to be predicted, leading to the prediction produced with weighted non-local patches. Finally, the predictions in the local and non-local manners are blended adaptively with a trainable network, ensuring the capability to handle a variety of contents. Experimental results on Versatile Video Coding (VVC) software VTM-11.0 show that the proposed model achieves on average 4.69% bit rate savings for natural scene sequences, and 4.24% bit rate savings for screen content sequences under the all intra configuration.

Low-latency live video streaming proposes more challenges on designing Adaptive Bit Rate (ABR) algorithms than video-on-demand streaming. In this paper, we present Fleet, a solution that improves user quality of experience (QoE) in terms of video quality, video switches, video freezes, and latency. Fleet is designed to meet two operational challenges at the same time: 1) optimizing QoE under dynamic mobile network conditions and 2) ensuring a low latency experience with minimal visual quality degradation. The core of Fleet is a stochastic model predictive controller that incorporates network conditions and client states for bitrate adaptation. The idle period problem presents great challenges for bandwidth measurement and client state prediction. Fleet consists of an HTTP chunk level bandwidth measurement algorithm, and a practical live video streaming evolution model for client state prediction. Besides, a throughput probability predictor is trained to capture the mobile network’s uncertainty. And, a triple threshold playback speed controller is designed for latency management. Fleet is practically implemented indash.jsand evaluated over both synthetic and real-world mobile network conditions. Our study shows that compared with the state-of-the-art solution, Fleet achieves an average QoE improvement of 39.7% and 37.9%, respectively. Moreover, Fleet has a good generalization, outperforming existing algorithms even on new video traces and network conditions.

Inter prediction is the critical component in hybrid coding framework to deal with the temporal redundancy. Most of the neural video coding methods typically follow the motion compensation based inter coding scheme, establishing motion vector (MV) as the central role. In this paper, we innovatively propose an efficient motion modeling approach by inherently decomposing it into two components, the intrinsic motion and the compensatory motion. The intrinsic motion originates from the implicit spatiotemporal context hidden in the historical sequence, which can be intuitively captured free of bits. On the top of it, the compensatory motion acts a role of structural refinement and texture enhancement as a form of side information. In particular, the inter prediction is performed in the feature space as a manner of progressive temporal transition, conditioned on the decomposed motion. By the motion decomposition paradigm, we innovatively answer the question of motion representation, compensation and coding in the learned video compression framework. With the temporal prediction, the remaining pixel residue is signaled to obtain the reconstruction. Extensive experimental results demonstrate that the proposed method achieves state-of-the-art coding performance on par with other end-to-end coding methods, and outperforms versatile video coding (VVC) under low-delay P (LDP) configuration in terms of MS-SSIM metric.

Learned image compression methods have achieved satisfactory results in recent years. However, existing methods are typically designed for RGB format, which are not suitable for YUV420 format due to the variance of different formats. In this paper, we propose an information-guided compression framework using cross-component attention mechanism, which can achieve efficient image compression in YUV420 format. Specifically, we design a dual-branch advanced information-preserving module (AIPM) based on the information-guided unit (IGU) and attention mechanism. On the one hand, the dual-branch architecture can prevent changes in original data distribution and avoid information disturbance between different components. The feature attention block (FAB) can preserve the important information. On the other hand, IGU can efficiently utilize the correlations between Y and UV components, which can further preserve the information of UV by the guidance of Y. Furthermore, we design an adaptive cross-channel enhancement module (ACEM) to reconstruct the details by utilizing the relations from different components, which makes use of the reconstructed Y as the textural and structural guidance for UV components. Extensive experiments show that the proposed framework can achieve the state-of-the-art performance in image compression for YUV420 format. More importantly, the proposed framework outperforms Versatile Video Coding (VVC) with 8.37% BD-rate reduction on common test conditions (CTC) sequences on average. In addition, we propose a quantization scheme for context model without model retraining, which can overcome the cross-platform decoding error caused by the floating-point operations in context model and provide a reference approach for the application of neural codec on different platforms.

Video prediction has always been a very challenging problem in video representation learning due to the complexity in spatial structure and temporal variation. However, existing methods mainly predict videos by employing language-based memory structures from the traditional Long Short-Term Memories (LSTMs) or Gated Recurrent Units (GRUs), which may not be powerful enough to model the long-term dependencies in videos, consisting of much more complex spatiotemporal dynamics than sentences. In this paper, we propose a SpatioTemporal Attention based Memory (STAM), which can efficiently improve the long-term spatiotemporal memorizing capacity by incorporating the global spatiotemporal information in videos. In the temporal domain, the proposed STAM aims to observe temporal states from a wider temporal receptive field to capture accurate global motion information. In the spatial domain, the proposed STAM aims to jointly utilize both the high-level semantic spatial state and the low-level texture spatial states to model a more reliable global spatial representation for videos. In particular, the global spatiotemporal information is extracted with the help of an Efficient SpatioTemporal Attention Gate (ESTAG), which can adaptively apply different levels of attention scores to different spatiotemporal states according to their importance. Moreover, the proposed STAM are built with 3D convolutional layers due to their advantages in modeling spatiotemporal dynamics for videos. Experimental results show that the proposed STAM can achieve state-of-the-art performance on widely used datasets by leveraging the proposed spatiotemporal representations for videos.

Low bit-width quantization can effectively reduce the storage and computational costs of deep neural networks. Existing quantization methods are commonly designed for single model compression. For multi-model compression scenarios, multiple models for the same task or similar tasks need to be compressed simultaneously in multimedia tasks, such as compressing image super-resolution models for different scales and transferring of different models in multimedia. However, single model quantization methods do not consider the correlations among the weights of different models, which limits the further compression for the above multi-model compression scenarios. To sufficiently excavate the potential of compression on multi-model, we propose a novel quantization scheme for multi-model compression, namely differential weight quantization (DWQ), which focuses on the weights increment between the target model and the reference model. Specifically, DWQ is achieved by increment computation, increment quantization and fine-tuning, which utilizes the reference model to guide the subsequent quantization on the target model. Due to the correlations between the weights of different models, the distribution of weights increment is more centralized compared with original weights, which can achieve a higher compression ratio by lower bit representation on weights increment. Moreover, the progressive training method is proposed to accelerate the convergence and reduce quantization loss on the DWQ framework. Extensive experiments validate the effectiveness of DWQ based on weight-sharing and parameterized clipping activation (PACT) quantization technologies on multiple tasks. The proposed framework can achieve 2× compression improvement and reduce 30% computational complexity with comparable performance in the popular multimedia tasks.

In this paper, we propose a novel low-complexity in-loop filtering approach named textural and directional information based offset (TDIO) for the video coding standard AVS3. Different from conventional offset-based filtering methods which partially use contextual samples, the key contribution of TDIO is that it fully utilizes the textural and edge directional features of each sample to comprehensively determine which type of texture characteristics each sample belongs to. The corresponding offsets are generated and signaled to decoder such that sample-level distortion is reduced. Specifically, the multi-directionality and sample-intensity pattern based classifiers are first proposed to extract the directional and textural features, respectively. The classification results are obtained by incorporating these features, and the optimal offset values for each class are derived based on rate-distortion optimization. Since sample-level offset signalling may cause heavy burden to the overhead of TDIO, we subsequently propose a filtering offset sharing mechanism based on historical information between available temporal-adjacent compressed frames. In addition, an iteration-based filter adaptation method is designed to improve the local adaptivity of TDIO for better compression efficiency. Experimental results show that the proposed TDIO achieves 0.64%, 1.29%, 1.86%, and 2.20% bit rate savings for all intra, random access, low delay B, and low delay P configurations, respectively. Moreover, TDIO is helpful to improve subjective quality by leveraging the fine-grained local texture characteristics. It can be observed that the blurring and ringing artifacts could be significantly suppressed by using the proposed method, yielding higher subjective quality.

Recent years have witnessed the explosion of virtual reality (VR) videos and applications. This new form of media grants us the precious freedom we never had before to look at any directions of the video content. With such privilege, our desires for higher video resolution and frame-rate, better visual quality, and smoother watching experience have remarkably risen. However, VR videos occupy an astronomical amount of data, which poses unprecedented challenges to computation efficiency, deployment cost, and user experience of the system. In this paper, we propose RealVR, an end-to-end tile-based VR video system that faces and tackles such challenges. With all essential procedures from the initial capturing to the ultimate rendering included, the system is designed, implemented, and configured specifically to achieve the best balance among efficiency, economy, and quality-of-experience (QoE). It leverages the promising international VR standard, MPEG OMAF, to process and deliver 8K 60 fps VR video, but is also improved for much better performance and user experience than the pristine standard, especially for reducing the motion-to-high-quality (MtHQ) latency. It does not rely on additional expensive edge servers to offer an immersive user experience, which makes it more economical than many other works. Through extensive experiments, it is proven that RealVR can significantly improve the MtHQ experience and save bandwidth consumption without compromising on encoding efficiency or application cost, maximizing the user‘s freedom.

Multi-scale design has been considered in recent image super-resolution (SR) works to explore the hierarchical feature information. Existing multi-scale networks aim at building elaborate blocks or progressive architecture for restoration. In general, larger scale features concentrate more on structural and high-level information, while smaller scale features contain plentiful details and textured information. In this point of view, information from larger scale features can be derived from smaller ones. Based on the observation, in this article, we build a sequential hierarchical learning super-resolution network (SHSR) for effective image SR. Specially, we consider the inter-scale correlations of features, and devise a sequential multi-scale block (SMB) to progressively explore the hierarchical information. SMB is designed in a recursive way based on the linearity of convolution with restricted parameters. Besides the sequential hierarchical learning, we also investigate the correlations among the feature maps and devise a distribution transformation block (DTB). Different from attention-based methods, DTB regards the transformation in a normalization manner, and jointly considers the spatial and channel-wise correlations with scaling and bias factors. Experiment results show SHSR achieves superior quantitative performance and visual quality to state-of-the-art methods with near 34% parameters and 50% MACs off when scaling factor is × 4. To boost the performance without further training, the extension model SHSR + with self-ensemble achieves competitive performance than larger networks with near 92% parameters and 42% MACs off with scaling factor ×4.

Although many video prediction methods have obtained good performance in low-resolution (64∼128) videos, predictive models for high-resolution (512∼4K) videos have not been fully explored yet, which are more meaningful due to the increasing demand for high-quality videos. Compared with low-resolution videos, high-resolution videos contain richer appearance (spatial) information and more complex motion (temporal) information. In this paper, we propose a Spatiotemporal Residual Predictive Model (STRPM) for high-resolution video prediction. On the one hand, we propose a Spatiotemporal Encoding-Decoding Scheme to preserve more spatiotemporal information for high-resolution videos. In this way, the appearance details for each frame can be greatly preserved. On the other hand, we design a Residual Predictive Memory (RPM) which focuses on modeling the spatiotemporal residual features (STRF) between previous and future frames instead of the whole frame, which can greatly help capture the complex motion information in high-resolution videos. In addition, the proposed RPM can supervise the spatial encoder and temporal encoder to extract different features in the spatial domain and the temporal domain, respectively. Moreover, the proposed model is trained using generative adversarial networks (GANs) with a learned perceptual loss (LP-loss) to improve the perceptual quality of the predictions. Experimental results show that STRPM can generate more satisfactory results compared with various existing methods.

Resampling-based coding, i.e. down-sampling before encoding and up-sampling after decoding, has been recognized to be an effective tool for compressing high-resolution videos at low bitrates. The newest video coding standard, Versatile Video Coding (VVC), supports resampling-based coding via a mechanism named Reference Picture Resampling (RPR), where the spatial resolution can be changed without inserting an intra frame. Intuitively, it is not wise to utilize a single resolution throughout the whole video, because frames with different contents may prefer different coding resolutions. In this paper, we propose a smart reference picture resampling approach, namely smart-RPR, where the coding-resolution of a frame is determined based on the property of the frame without multiple-pass encoding. Specifically, we first down- and up-sample a frame without considering compression and compare the up-sampled frame with the original frame to obtain the resampling distortion, which is then compared with a threshold to decide whether to code the frame in a resampling way. Then, we build up an exponential model to approximate the optimal threshold. In addition, we also study how to derive the coding parameters of the down-sampled frame to achieve better performance. Simulation results on the VTM-12.0 show that the proposed method could achieve 2.72%, 5.29%, and 10.82% BD-rate reductions for Y, Cb, and Cr components, respectively, with lower encoding and decoding complexity.

End-to-end optimized neural image compression (NIC) has obtained superior lossy compression performance recently. In this paper, we consider the problem of rate-distortion (R-D) characteristic analysis and modeling for NIC. We make efforts to formulate the essential mathematical functions to describe the R-D behavior of NIC using deep networks. Thus arbitrary bit-rate points could be elegantly realized by leveraging such model via a single trained network. We propose a plugin-in module to learn the relationship between the target bit-rate and the binary representation for the latent variable of auto-encoder. The proposed scheme resolves the problem of training distinct models to reach different points in the R-D space. Furthermore, we model the rate and distortion characteristic of NIC as a function of the coding parameter$\lambda$respectively. Our experiments show our proposed method is easy to adopt and realizes state-of-the-art continuous bit-rate coding performance, which implies that our approach would benefit the practical deployment of NIC.

Recently many efforts have been devoted to learning non-linear predictions from neighboring samples with deep neural networks. However, existing methods mainly generate predictions with local reference samples, regardless of nonlocal self-similarity. In this paper, we aim to incorporate local and nonlocal correlations for intra prediction and propose a two-stage coarse-to-fine network (CTFN), which is integrated into VVC codec as an optional intra prediction mode. The prediction process of CTFN is decomposed into two stages. In the first stage, we train a set of networks to generate a coarse result with local reference samples. In the second stage, we extract sufficient features from nonlocal region using the coarse result as priors and transform the features into a fine prediction result. In particular, a patch-wise attention layer (PAL) is designed in the second stage that can fully explore nonlocal correlations in feature domain and assign weights to each nonlocal feature adaptively, as shown in Fig. 1. As such, the proposed CTFN can not only learn a non-linear mapping from local context, but also explicitly borrow similar features from nonlocal region in a weighted form. Different from image inpainting tasks, the patch synthesis problem is converted to patch matching problem with the CTFN, yielding more reliable predictions. More-over, we construct a classified dataset based on Pearson Correlation Coefficient for network training to better handle contents that are highly correlated. Experiments on VTM-11.0 show that the proposed network achieves 1.77% ED-rate reductions under all intra configuration, which outperforms the state-of-the-art methods.

In-loop filter has been comprehensively explored during the development of video coding standards to suppress compression artifacts. However, the existing in-loop filters in Versatile Video Coding (VVC) mainly take advantage of the image local similarity. Although some non-local based in-loop filters can make up for this short-coming, the unsupervised parameter selection scheme, which is widely used by non-local filters, limits the content adaptability. Given this, we propose a parametric non-local in-loop filter (PNLF) that fully considers the non-local characteristics and trains the filter coefficients based on the video content. In the filtering process, the reference samples based on the non-local similarity are first derived for each to-be-filtered sample. Then to-be-filtered samples are grouped into specific classes based on multiple features. For each class, filter coefficients are online trained in the encoder and transmitted to the decoder. Finally, the filtering process is conducted using the online-selected coefficients. Simulation results reveal that the proposed approach achieves 0.70%, 1.43%, and 2.09% bit-rate savings on average compared to VTM-11.0 under All Intra (AI), Random Access (RA), and Low-Delay B (LDB) configurations, respectively. The sequences used in the experiment include Class AI, A2, B, C, D, E, F, and SCC. Compared to the non-local structure-based filter (NLSF) [1], our proposed PNLF with fast block matching scheme [2] applied on B-frames and P-frames can achieve better performance gain with lower software and hardware complexity under RA and LDB configurations.

Flexible coding unit partitioning such as quad-tree nested binary-tree and ternary-tree adopted by the emerging enhanced compression model (ECM) brings promising coding performance improvement. Meanwhile, the computational complexity increases dramatically, which may block the exploration and validation of new coding tools. This paper investigates a partition mode early pruning scheme via a fully connected network to reduce the encoding complexity for the ECM. In particular, we carefully select features and devise the fully connected network, which could seamlessly cooperate with the encoder, revealing promising learning and inference capability. Experimental results demonstrate that the proposed method achieves 15%~50% encoding time savings with moderate bit-rate increasing on the ECM, and the extra complexity regarding the fully connected network and feature extraction is negligible.

In recent years, medical report generation has received increasing research interest with the goal of automatically generating long and coherent descriptive paragraphs that can de-scribe in detail the observations of normal and abnormal regions in the input medical images. Unlike general image captioning tasks, medical report generation is more challenging for data-driven neural models. This is mainly due to severe visual and textual data biases. To address these problems, we propose an Adaptive Image-Tag Merging Network (AIMNet) that first predicts the tags of diseases from the input image, and then adaptively merges the visual information of the input image and disease information from the disease tags to learn the disease-oriented visual features that can better represent abnormal regions of the input image, and thus can be used to alleviate data bias problems. The experiments and analyses on the public MIMIC-CXR and IU-Xray datasets show that our proposed AIMNet achieves the state-of-the-art results under all metrics and significantly outperforms previous models on the MIMIC-CXR and IU-Xray datasets with relatively 15.1% and 6.5% margins in terms of BLEU-4 score.

As 3D scanning devices and depth sensors advance, point clouds have attracted increasing attention as a format for 3D representation. Nevertheless, the tremendous amount of data in point clouds significantly burden transmission and storage. To address these problems, we propose a multi-scale end-to-end framework for point cloud geometry compression. Firstly, point transformer is used to extract the global feature of geometry information, embedding the geometry information and the relation among points. Secondly, the multi-scale neighbor embedding strategy is used to extract the level of details within multi-scale and multi-resolution feature of point clouds. Finally, to reconstruct the point cloud with high quality in the decoder, the local spatial information is restored via graph spatial extension based on local down-sampled features and global features. Experimental results show that we achieve around 34% bit rate reduction on average over competitive point cloud geometry compression methods.

In this paper, a model selection based multi-scale convolutional neural network (CNN) model for in-loop filtering is proposed for Versatile Video Coding (VVC). We propose a novel network to jointly filter the luminance component and chrominance components simultaneously in the coding loop to improve the quality of the reconstructed frames. The proposed model contains two main branches with different scales as well as the global identity connection. Each branch contains several residual blocks. We fuse the residual features of the two branches by using a convolutional block attention module (CBAM) [1] at the end of the model. In addition, we adopt a model selection strategy that can select the best coding tree unit (CTU) level model in terms of R-D cost at the encoder. Compared with VTM-14.0 our method achieves 5.17%, 11.05% and 10.89% BD-rate reduction under all intra (AI) configuration, and 6.56%, 13.14%, 11.98% BD-rate reduction under random access (RA) configuration. Moreover, under RA configuration, the proposed model selection strategy brings extra 0.34% BD-rate reduction in the luminance component.

Hardware-oriented neural video coding (NVC) has been an emerging research topic recently. In this work, we present a P-frame based NVC system by leveraging the parallel neural inference capability of resource-constrained embed platforms, such as field programmable gate array (FPGA). The proposed system, namely FPX-NVC, consists of multiple resolution video capture, reconfigurable and integerized neural video codec and display. Our neural codec enjoys the learned unidirectional inter-predictive coding by jointly optimizing the implicit motion representation (IMR), feature domain motion compensation, and the corresponding prediction error compression, realizing end-to-end on-board encoding and decoding. Our system supports end-to-end video data operation flow from capturing to neural coding and display. To our knowledge, the FPX-NVC is the first inter-predictive coding (P-frame) based hardware NVC system.

In Versatile Video Coding (VVC), reference picture resampling (RPR) is supported to change resolution without inserting an instantaneous decoder refresh (IDR) picture or intra random access picture (IRAP). The introduction of RPR enables VVC to adapt to unstable network conditions, which benefits real-time application scenarios a lot. However, the existing bitrate leap and the low quality of the upsampled frames are the bottlenecks of promoting its application. In this paper, we propose an efficient framework to alleviate bitrate surges and further improve the coding performance. The contributions of this paper mainly include the following three aspects. Firstly, it is proved theoretically and statistically that the temporal long-distance high-resolution (HR) reference pictures can provide better prediction than temporal short-distance low-resolution (LR) reference pictures when the temporal distance between them is less than a threshold. Secondly, motivated by the theoretical analysis, existing HR reference pictures in the reference pictures list (RPL) are used to replace LR reference pictures and an adaptive decision on the number of HR reference pictures kept in RPL is applied. Thirdly, an adaptive upsampling filter (AUF) is proposed to restore pictures with the original resolution more accurately. Simulation results show 15.09% BD-rate savings on average can be achieved for the 2:1 scaling ratio under RPR configurations. For the 1.5:1 scaling ratio, the BD-rate saving is 8.14%. Besides, the bitrate surge in RPR is reduced greatly. About 25% bits reductions on average at the leaping point can be observed and the bits decrease rate is up to 50% for sequences with little and slow motion.

Recent years have witnessed the exponential increase in the demand for ultra high definition (UHD) video compression. The third generation of Audio Video Coding Standard (AVS3), which is also known as IEEE Standard 1857.10, is the latest audio and video coding standard developed by the China AVS working group. In AVS3, targeting for UHD videos, a series of efficient coding tools have been introduced, leading to the dramatical increase of computational burden. In this scenario, real-time decoding of UHD videos becomes extremely challenging. This paper presents an improved hybrid CPU + GPU accelerated framework for AVS3 decoding. In particular, the motion vector (MV) derivation process is extracted from entropy decoding threads on the CPU. Therefore, the dependency between threads is removed and the entropy decoding can be performed by multiple threads efficiently. Regarding the GPU, we design compact data structures for transform, prediction, and in-loop filtering to reduce the burden of data transmission. A flexible information buffer supporting multi-thread random writing is further created to coordinate the computation between CPU and GPU. Through asynchronous operations on the buffer, the computation of different computing units and the data transmission between them could be performed in parallel. With NVIDIA GeForce RTX 2080Ti GPU and Intel Core i7 8700K CPU, the proposed decoder achieves 151 frames per second (fps) for 4K videos and 55 fps for 8K videos in all intra configuration. In random access configuration, 218 fps and 74 fps are obtained for 4K and 8K videos, respectively.

The recent trend in neural image compression (NIC) research could be generally grounded into two categories: analysis-synthesis transform network improvements and entropy estimation optimization. They promote the compression efficiency of NIC by leveraging more expressive network structures and advanced entropy models respectively. From a different but more systematic viewpoint, we extend the horizon of NIC from software- to hardware-based lossy compression using more resource-constrained platforms, such as field programmable gate array (FPGA) or deep-learning processor unit (DPU). In this paper, we propose a novel hardware-oriented NIC system for real-time edge-computing video services. We for the first time present FPX-NIC, an FPGA-accelerated NIC framework designed for hardware encoding, which consists of a novel NIC scheme and an energy-efficient neural network (NN) deployment method. The former contribution is a block-based adaptive NIC approach based on local content characteristics. Essential side-information is also signalled to realize adaptive patch representation. The critical advantage of our latter contribution lies in the network-reconfigurable framework plus fixed-precision weights quantization method that takes advantage of quantization-aware post training procedure to compensate the performance degradation caused by quantization error. Therefore it is able to improve both processing speed and energy efficiency. We finally establish an intelligent video coding system using the proposed scheme, enabling visual capturing, neural encoding, decoding, and display, realizing 4K ultra-high-definition (UHD) all intra neural video coding on edge-computing devices.

It is challenging to restore low-resolution (LR) images to super-resolution (SR) images with correct and clear details. Existing deep learning works almost neglect the inherent structural information of images, which acts as an important role for visual perception of SR results. In this paper, we design a hierarchical feature exploitation network to probe and preserve structural information in a multi-scale feature fusion manner. First, we propose a cross convolution upon traditional edge detectors to localize and represent edge features. Then, cross convolution blocks (CCBs) are designed with feature normalization and channel attention to consider the inherent correlations of features. Finally, we leverage multi-scale feature fusion group (MFFG) to embed the cross convolution blocks and develop the relations of structural features in different scales hierarchically, invoking a lightweight structure-preserving network named as Cross-SRN. Experimental results demonstrate the Cross-SRN achieves competitive or superior restoration performances against the state-of-the-art methods with accurate and clear structural details. Moreover, we set a criterion to select images with rich structural textures. The proposed Cross-SRN outperforms the state-of-the-art methods on the selected benchmark, which demonstrates that our network has a significant advantage in preserving edges.

Dynamic textures are time-varying motion patterns that exhibit certain temporal stationarity. The motion of such patterns is usually anisotropic, which is a great challenge for video coding. In this paper, we firstly analyze the predictive and temporal characteristics of dynamic textures and, based on the analysis, a pixel-level segmentation-synthesis framework for dynamic texture compression is proposed to improve predictive-coding efficiency. This framework consists of three sub-modules: dynamic texture segmentation, dynamic texture synthesis and fusion process. A deep-learning-based segmentation method is introduced to obtain a pixel-level mask. Subsequently, optical flow fields are used to generate flowlines and help analyze the motion periodicity of a given sequence. The flowlines and period values are utilized as inputs for flow-based dynamic texture synthesis. The segmented dynamic texture area is then reconstructed using the synthesized results at the decoder side at pixel-level, such that the overhead for transmitting these contents could be saved. The segmented mask is transmitted through an All-Zero-Run-Length coding algorithm and an effective fusion module is further proposed to reduce edge artifacts that occur at the boundary of dynamic texture areas. Coding blocks have been classified into three types and a weighting reconstruction scheme is performed. The proposed framework has been integrated into VTM-10.0. Experimental results on a collected dynamic texture test set demonstrate that 15.1% and 16.1% MOS BD-rate savings can be achieved on average for LDB and LDP configurations, respectively.

Flexible block partition and newly adopted intra coding tools bring significant performance improvement for next generation video coding, and meanwhile introduce non-negligible encoding complexity increment. This paper presents a scalable intra coding optimization (SICO) scheme for the third generation of audio video coding standard (AVS3). The complexity distribution and inheriting relationship among different partitioning, as well as intra prediction modes, are systematically analyzed. Subsequently, low-complexity algorithms are proposed, which could early exclude unlikely coding modes safely. In particular, a data-driven binary classifier is elegantly trained for the determination of coding unit partitioning. Moreover, the preliminary coding information can be exhaustively utilized for the mode decision in a low-cost manner. The proposed method provides scalable intra coding optimizing solutions, which is eligible to cater to various application scenarios. Experimental results show that the proposed method could achieve a wide range of encoding complexity reduction from 18% to 76% with moderate compression performance loss. One implementation of the proposed method has been adopted to the AVS3 reference software.

In this paper, we improve the inter coding performance of surveillance videos by simultaneously investigating the distinct characteristics of background and foreground redundancy, and introduce two novel reference frames in a complementary manner. On one hand, a block level background reference frame (BRF) is proposed to reduce the background redundancy. The proposed scheme incorporates semantic information into the compression process, and makes use of instance segmentation to facilitate the background block decision, making the generated BRF free from foreground pollution. On the other hand, in order to handle foreground redundancy, a foreground reference frame (FRF) is generated based on Surveillance Prediction Generative Adversarial Network (SP-GAN), which utilizes previous reconstructed frames, optical flow based prediction, as well as BRF to infer the foreground objects of the to-be-coded frame. We integrate the proposed scheme into HM-16.6 software and append BRF and FRF to the reference pictures list (RPS). Simulation results demonstrate considerable superiority of the proposed scheme. In particular, by adding the proposed BRF to RPS, 3% coding gains are observed compared with the state-of-the-art BRF method. When both BRF and FRF are incorporated into RPS, 5.8% gains are achieved for surveillance video coding.

In the latest video coding standard, namely Versatile Video Coding (VVC), more directional intra modes and reference lines have been utilized to improve prediction efficiency. However, complex content still cannot be predicted well with only the adjacent reference samples. Although nonlocal prediction has been proposed to further improve the prediction efficiency in existing algorithms, explicit signalling or matching error potentially limits the coding efficiency. To address these issues, we propose a joint local and nonlocal progressive prediction scheme, targeting at improving nonlocal prediction accuracy without additional signalling. Specifically, template matching based prediction (TMP) is conducted firstly to derive an initial nonlocal predictor. Based on the first prediction and previously decoded reconstruction information, a local template, including inner textures and neighboring reconstruction, is carefully designed. With the local template involved in nonlocal matching process, a more accurate nonlocal predictor can be found progressively in the second prediction. Finally, the coefficients from the two predictions are fused and transmitted in bitstreams. In this way, more accurate nonlocal predictor can be derived implicitly with local information instead of being explicitly signalled. Experimental results on the reference software VTM-9.0 of VVC show that the method achieves 1.02% BD-Rate reduction for natural sequences and 2.31% BD-Rate reduction for screen content videos under all intra (AI) configuration.

Geometric partitioning has attracted increasing attention by its remarkable motion field description capability in the hybrid video coding framework. However, the existing geometric partitioning (GEO) scheme in Versatile Video Coding (VVC) causes a non-negligible burden for signaling the side information. Consequently, the coding efficiency is limited. In view of this, we propose a spatio-temporal correlation guided geometric partitioning (STGEO) scheme to efficiently describe the object information in the motion field of video coding. The proposed method can economize the bits consumed for side information signaling, including the partitioning mode and motion information. We firstly analyze the characteristics of partitioning mode decision and motion vector selection in a statistically-sound way. Based on the observed spatio-temporal correlation, we design a mode prediction and coding method to reduce the overhead for representing the above mentioned side information. The main idea is to predict the STGEO modes and motion candidates that have higher selection possibilities, which can guide the entropy coding, i.e., representing the predicted high-probability modes and motion candidates with fewer bits. In particular, the high-probability STGEO modes are predicted based on the edge information and history modes of adjacent STGEO-coded blocks. The corresponding motion information is represented by the index in a merge candidate list, which is adaptively inferred based on the off-line trained merge candidate selection probability. Simulation results show that the proposed approach achieves 0.95% and 1.98% bit-rate savings on average compared to VTM-8.0 without GEO for Random Access and Low-Delay B configurations, respectively.

In-loop filters have attracted increasing attention due to the remarkable noise-reduction capability in the hybrid video coding framework. However, the existing in-loop filters in Versatile Video Coding (VVC) mainly take advantage of the image local similarity. Although some non-local based in-loop filters can make up for this shortcoming, the widely-used unsupervised parameter estimation method by non-local filters limits the performance. In view of this, we propose a deformable Wiener Filter (DWF). It combines the local and non-local characteristics and supervisedly trains the filter coefficients based on the Wiener Filter theory. In the filtering process, local adjacent samples and non-local similar samples are first derived for each sample of interest. Then the to-be-filtered samples are classified into specific groups based on the patch-level noise and sample-level characteristics. Samples in each group share the same filter coefficients. After that, the local and non-local reference samples are adaptively fused based on the classification results. Finally, the filtering operation with outlier data constraints is conducted for each to-be-filtered sample. Moreover, the performance of the proposed DWF is analyzed with different reference sample derivation schemes in detail. Simulation results show that the proposed approach achieves 1.16%, 1.92%, and 2.67% bit-rate savings on average compared to the VTM-11.0 for All Intra, Random Access, and Low-Delay B configurations, respectively.

Transform coding removes redundancy by de-correlating the residual data, which plays a crucial role in video coding. Since different transform cores can adapt to different video content and residual data, multiple cores transform has been proposed to improve the coding performance. However, the overhead for representing the transform type is inevitable. This paper intensively studies the statistical characteristics of transform coefficients. Theoretical analysis shows that the implicit method in the Implicitly Selected Transform (IST) is superior to the explicit signalling method. As such, we propose the parity adjustment scheme which seamlessly cooperates with the rate-distortion optimization quantization for IST. Furthermore, we propose two combination methods, size-based and number-based, to optimize IST. Moreover, a restriction region of IST is applied to reduce the complexity. Experimental results on HPM-6.0, which is the reference software of the AVS3 video coding standard, show that our proposed method can achieve 1.76% and 0.76% BD-Rate savings on average under AI and RA configurations, respectively, along with negligible decoding time variations. The comparison results between the proposed method and explicit signalling method illustrate that the proposed method can achieve better coding gain than the latter. Our method has been adopted in AVS3.

Single image super-resolution (SISR), as a traditional ill-conditioned inverse problem, has been greatly revitalized by the recent development of convolutional neural networks (CNN). These CNN-based methods generally map a low-resolution image to its corresponding high-resolution version with sophisticated network structures and loss functions, showing impressive performances. This paper provides a new insight on conventional SISR algorithm, and proposes a substantially different approach relying on the iterative optimization. A novel iterative super-resolution network (ISRN) is proposed on top of the iterative optimization. We first analyze the observation model of image SR problem, inspiring a feasible solution by mimicking and fusing each iteration in a more general and efficient manner. Considering the drawbacks of batch normalization, we propose a feature normalization (F-Norm, FN) method to regulate the features in network. Furthermore, a novel block with FN is developed to improve the network representation, termed as FNB. Residual-in-residual structure is proposed to form a very deep network, which groups FNBs with a long skip connection for better information delivery and stabling the training phase. Extensive experimental results on testing benchmarks with bicubic (BI) degradation show our ISRN can not only recover more structural information, but also achieve competitive or better PSNR/SSIM results with much fewer parameters compared to other works. Besides BI, we simulate the real-world degradation with blur-downscale (BD) and downscale-noise (DN). ISRN and its extension ISRN+ both achieve better performance than others with BD and DN degradation models.

Compactly representing visual information plays a fundamental role in optimizing the ultimate utility of myriad visual data-centered applications. Numerous approaches have been proposed to efficiently compress the texture and visual features for human visual perception and machine intelligence, respectively; however, much less work has been dedicated to studying the interactions between them. Here, we investigate the integration of feature and texture compression and show that a universal and collaborative visual information representation can be achieved in a hierarchical way. In particular, we study feature and texture compression in a scalable coding framework, where the base layer serves as the deep learning feature and the enhancement layer targets to perfectly reconstruct the texture. Based on the strong generative capability of deep neural networks, the gap between the base feature layer and enhancement layer is further filled with feature-level texture reconstruction, with the goal of further constructing texture representations from features. As such, the residuals between the original and reconstructed texture could be further conveyed in the enhancement layer. To improve the efficiency of the proposed framework, the base layer neural network is trained in a multitask manner such that the learned features enjoy both high-quality reconstruction and high-accuracy analysis. The framework and optimization strategies are further applied in face image compression, and promising coding performance has been achieved in terms of both rate-fidelity and rate-accuracy evaluations.

Recently, deep learning for video coding, such as deep predictive coding, deep transform coding, and deep in-loop filtering, has been an emerging research area. The coding gain of hybrid coding framework could be extensively promoted by the data-driven models. However, previous deep coding tools especially deep in-loop filtering mainly consider the performance improvement while pay less attention to the reliability, usability, and adaptivity of the networks. In this article, a nested-residual guided convolutional neural network (NR-CNN) structure with cascaded global shortcut and configurable residual blocks is proposed for in-loop filtering. By taking advantage of the correlation between different color components, we further extend the NR-CNN by utilizing luminance as textural and structural guidance for chrominance filtering, which significantly improves the filtering performance. To fully exploit the proposed network into codec integration, we subsequently introduce an efficient and adaptive framework consisting of an adaptive granularity optimization and a parallel inference pipeline for deep learning based filtering. The former contributes to the coding performance improvement through an adaptive decision-making based on rate-distortion analysis at various granularities. The latter reduces the running time of network inference. The extensive experimental results show the superiority of the proposed method, achieving 8.2%, 14.9%, and 13.2% BD-rate savings on average under random access (RA) configuration. Meanwhile, the proposed method also obtains better subjective quality.

In video coding, block partition segments images into non-overlap blocks for individual coding, the structure of which is becoming more and more flexible along with the development of video coding standards. Multiple types of tree structures have been proposed recently, which extensively improved the complexity of the encoding process due to the recursive rate-distortion search for the optimal partition. In this paper, a two-stage Convolutional Neural Network (CNN) based partition structure prediction method is proposed to bypass the decision process of the block size in intra frame coding. Specifically, the Coding Unit (CU) partition is first represented in sub-block granularity and predicted by the end-to-end trained CNNs. Then, the final partition structure compatible with the coding standard is derived from the prediction results directly. Experimental results show that the proposed CNN based partition method achieves about 56 times speedup (97% time-saving) with 9% BD-rate degradation against the reference software of the latest AVS3 coding standard (IEEE Standard 1857.10).

In-loop filter is a crucial module in video coding, which can improve both subjective and object quality of reconstructed videos. In this paper, a new sample-based classification method is first proposed using features extracted from different stages of the existing in-loop filter process. Based on this method, an adaptive three-layer Quad-tree Based Sample Refinement Filter (QSRF) algorithm is designed to further improve the coding efficiency. Experimental results show that the proposed QSRF algorithm achieves 0.39%, 0.77% and 0.70% BD-rate savings for random access, lowdelay B and lowdelay P configurations compared to AVS3 reference software, respectively. Moreover, the proposed method can also improve visual quality of reconstructed videos significantly.

In the Versatile Video Coding (VVC) standard, adaptive loop filter (ALF), including Geometry transformation-based Adaptive Loop Filter (GALF) and Cross Component Adaptive Loop Filter (CCALF), plays an essential role in reducing compression artifacts. However, it also has high coding complexity and requires many picture buffer accesses in the encoder that will increase external memory access and is unfriendly to the software and hardware design. Therefore, we propose an optimized ALF framework, including the parallel design of GALF and CCALF, the adaptive parameter decision of GALF, and one-pass CCALF scheme by effectively estimating the CCALF filtering distortion without conducting filter operation. Compared to VTM-8.0, the proposed method can reduce the picture buffer access from 152 to 1 and achieve roughly 25% time-savings of the ALF module with negligible coding performance change under RA configuration. Some of the proposed methods have been adopted in the VVC reference software.

The basic ingredients of modern video coding standards are block-based prediction and transforms. However, when dealing with video contents containing dynamic textures (DT), the existing prediction schemes usually failed due to temporal variability and randomness of DT, which results in more bit cost on residual coding compared with other contents. In view of this point, a novel video compression scheme for DT is proposed in this work. In particular, wavelet-based analysis on motion characteristics of DT is firstly presented and based on the analysis, we introduce a flow-grounded texture synthesis method for video compression. Instead of conventional inter prediction, synthesized DT contents are used for reconstruction at the decoder. The proposed scheme has been fully integrated into the test model of Versatile Video Coding standard, VTM-10.0, for validation and a subjective test has also been carried out. Experimental results show that bitrate savings can be achieved by 40% on average at comparable visual quality.

Quantization is an effective way for reducing the memory and computation costs of deep neural networks. Most of existing methods exploit the fixed-precision quantization approach, e.g., weights and activations (i.e., output features) are represented as 8-bit values. Although mixed-precision quantization provides us a greater possibility to efficiently allocate computation resources and maintain the network performance, it is difficult to accurately solve the optimal bit-width of each layer. In this paper, we develop a novel evolutionary based method to automatically determine the bit-widths of weights and activations in each convolutional layer, namely, Evolutionary Mixed-Precision Quantization (EMQ). Specifically, the quantization intervals of weights and activations of all layers in the given network will be simultaneously encoded as an individual. The fitness of each individual is calculated as the performance of the corresponding quantized network. The optimal quantization result will be updated and elected during the evolutionary search. Extensive experiments conducted on benchmark datasets and models demonstrate the effectiveness of the proposed method over the state-of-the-art network quantization algorithms.

In this paper, we propose a novel end-to-end feature compression scheme by leveraging the representation and learning capability of deep neural networks, towards intelligent front-end equipped analysis with promising accuracy and efficiency. In particular, the extracted features are compactly coded in an end-to-end manner by optimizing the rate- distortion cost to achieve feature-in-feature representation. The multi-granularity constraint is further imposed, serving as the optimization objective to make the feature compression more "healthier" from the perspective of ultimate utility. More specifically, the analysis accuracy is considered in the coarse granularity level constraint, ensuring the capability of facial analysis with the reconstructed feature. Furthermore, at the fine granularity level the feature fidelity is involved to preserve the original feature quality. Moreover, a latent code level teacher-student enhancement model is proposed to efficiently transfer the low bit-rate representation into a high bit- rate one. Such a strategy further allows us to adaptively shift the representation cost to decoding computations, leading to more flexible feature compression with enhanced decoding capability. We verify the effectiveness of the proposed model with the facial feature, and experimental results reveal better compression performance in terms of rate-accuracy compared with existing models.

Videos are typically encoded to low-dimensional features to save computation resources for video prediction models. However, the unacceptable information loss while encoding is restricting the performance of the predictive models. To solve this problem, in this paper, we propose an Information-Preserving Spatiotemporal Predictive Model for video prediction, denoted as IPRNN. In our method, we apply multiple skip-connections between the corresponding layers between the encoders and decoders. In this way, more useful information from the encoders can be recalled by the decoders to achieve a more satisfactory performance. Moreover, to further save the computation resources for predictive models, we design a spatiotemporal gated recurrent unit (STGRU), which can efficiently capture the spatial appearance information and temporal motion information for videos. Experimental results show that the proposed method can obtain better performance compared with other state-of-the-art methods.

Reference picture resampling (RPR) is an important feature for real-time communication in Versatile Video Coding, which supports adaptive resolution change without introducing an instantaneous decoder refresh (IDR) or intra random access picture (IRAP). With RPR, considerable coding gain is achieved while bitrate leap still exists. In this paper, an enhanced reference structure for RPR is proposed. First, the high-resolution (HR) references are used to replace the low-resolution (LR) references to provide more accurate prediction. Furthermore, an adaptive decision on the number of HR references kept in RPL is applied to achieve more performance, which utilizes the history information in the current period. Simulation results show about 9% and 5% BD-rate savings on average can be achieved for scaling ratio of 2:1 and 1.5:1 compared to VTM-10.2 under RPR configurations. For sequences with slight motions, the coding gain is up to 20%. Besides, the bitrate explosion in RPR is reduced generally, where over 50% reductions of bitrates at explosion point is observed.

HTTP adaptive streaming with chunked transfer encoding can provide low latency live video streaming experience. However, providing high and smooth quality video streaming experience under low latency conditions poses a great challenge for Adaptive BitRate (ABR) algorithm design due to the shorter time to react to bandwidth fluctuations. To address the problem, we design a Fuzzy logic controller based ABR for low latency live video streaming with Chunked Transfer Encoding (FCTE). We take player buffer size, throughput mean, and throughput standard deviation as inputs to make bitrate decisions. By making full use of network information and dealing with uncertainties using fuzzy language, FCTE can make quick and robust bitrate decisions in the complicated network environment. We evaluate FCTE over five network scenarios released by Twitch. FCTE achieves average QoE improvement by 129%, 338%, and 127% compared with three representative algorithms: L2A, LoL, and Stallion, respectively.

Predicting high-resolution videos (≥ 256) is always a very difficult task in video prediction domain. To predict high-quality frames for high-resolution videos, both the challenging spatiotemporal representations and the computation resources are needed to be carefully considered. In this paper, we propose a SpatioTemporal Auto-Encoder for High-Resolution Video Prediction, which is named STAE. In our method, we first jointly utilize the spatial and temporal encoders to extract low-dimensional spatial and temporal features from the high-resolution video input, which can preserve the spatiotemporal information from the input and significantly reduce the computation load for the following modules. In addition, we design a SpatioTemporal Attention based Memory (STAM) to predict the spatiotemporal features for future frames using the encoded low-dimensional features. Then the predicted spatial and temporal features are decoded back to the high-dimensional data space using the spatial and temporal decoders. Finally, the predicted high-dimensional spatial and temporal representations are jointly utilized to predict the future frames. All modules in STAE are built on the basis of 3D neural networks to improve the local perception to videos. Experimental results show the proposed method outperforms diverse state-of-the-arts on widely used datasets and the computation load is relative low.

Video prediction has always been a challenging task in video representation learning due to the diversity of spatial-temporal evolution in videos. In this paper, we propose an Attention based SpatioTemporal Model for Video Prediction based on 3D Convolutional Neural Networks and Long Short-Term Memory (LSTM), which is named ASTM. In our method, we leverage both multi-term and short-term inter-frame dependencies in temporal domain to capture reliable motion information for videos. In particular, we design an Efficient Inter-Frame Attention Gate (EIFAG) to efficiently aggregate the multi-term inter-frame dependencies and integrate 3D convolutional operations into the proposed model to further improve the local perception to videos by capturing more accurate short-term temporal dependency. In addition, we make use of the multilayer Spatiotemporal LSTM (PredRNN) structure to preserve more spatial appearance details for videos. To evaluate the adaptability of our model on more complex real scenes, we collect a multi-level spatiotemporal (MLST) dataset. Experimental results show that the proposed model can achieve state-of-the-art performance on both widely used datasets and the proposed MLST dataset.

Optimized for pixel fidelity metrics, images compressed by existing image codec are facing systematic challenges when used for visual analysis tasks, especially under low-bitrate coding. This paper proposes a visual analysis-motivated rate-distortion model for Versatile Video Coding (VVC) intra compression. The proposed model has two major contributions, a novel rate allocation strategy and a new distortion measurement model. We first propose the region of interest for machine (ROIM) to evaluate the degree of importance for each coding tree unit (CTU) in visual analysis. Then, a novel CTU-level bit allocation model is proposed based on ROIM and the local texture characteristics of each CTU. After an in-depth analysis of multiple distortion models, a visual analysis friendly distortion criteria is subsequently proposed by extracting deep feature of each coding unit (CU). To alleviate the problem of lacking spatial context information when calculating the distortion of each CU, we finally propose a multi-scale feature distortion (MSFD) metric using different neighboring pixels by weighting the extracted deep features in each scale. Extensive experimental results show that the proposed scheme could achieve up to 28.17% bitrate saving under the same analysis performance among several typical visual analysis tasks such as image classification, object detection, and semantic segmentation.

Deinterlacing is a classical issue in video processing area, which aims to generate the progressive video from the interlaced instance. Although numerous algorithms have been proposed in the past decades, their performances are still not satisfactory from both quality of experience and processing efficiency. This paper focuses on the spatial-temporal correlation in the given frame, and design a network for recovering the missing field. Intra-frame motion compensation is considered between the given fields for detail refinement. Furthermore, we address the inherent correlations among image features with channel attention for better exploration. Extensive experimental results on different video sequences show that our method outperforms state-of-the-art methods according to both objective and subjective evaluations satisfying the real-time requirement.

In this paper, a flexible and adaptive block-wise image compression architecture with restoration network is proposed to improve the rate-distortion (R-D) performance of learned image compression. In contrast to existing learned compression algorithms which perform the image based transform coding, our approach splits the input image into non-overlapped blocks and compresses each block adaptively based on their local content characteristics. To reduce the artifacts caused by the block partitioning, block fusion network is subsequently proposed to overcome the drawbacks of block-wise independent coding. Further, quantization fine-tuning training process is proposed and utilized to reduce the difference of quantization operations between the training process and testing process. Experimental results show that the proposed network architecture has the ability to obtain significant coding gain compared with existing conventional compression standards including Versatile Video Coding (VVC) and achieves comparable R-D performance with state-of-the-art learned image codec using less parameters.

Recently, learning-based lossy image compression has achieved notable breakthroughs with their excellent modeling and representation learning capabilities. Comparing to traditional image codecs based on block partitioning and transform, these data-driven approaches with artificial-neural-network (ANN) structures bring significantly different distortion patterns. Efficient objective image quality assessment (IQA) measures play the key role in quantitative evaluation and optimization of image compression algorithms. In this paper, we construct a large-scale image database for quality assessment of compressed images. In the proposed database, 100 reference images are compressed to different quality levels by 10 codecs, involving both traditional and learning-based codecs. Based on this database, we present a benchmark for existing IQA methods and reveal the challenges of IQA on learning-based compression distortions. Furthermore, we develop an objective quality assessment framework in which a self-attention module is adopted to leverage multi-level features from reference and compressed images. Extensive experiments demonstrate the superiority of our method in terms of prediction accuracy. The subjective and objective study of various compressed images also shed lights on the optimization of image compression methods.

Most of the existing 3D human pose estimation approaches mainly focus on predicting 3D positional relationships between the root joint and other human joints (local motion) instead of the overall trajectory of the human body (global motion). Despite the great progress achieved by these approaches, they are not robust to global motion, and lack the ability to accurately predict local motion with a small movement range. To alleviate these two problems, we propose a relative information encoding method that yields positional and temporal enhanced representations. Firstly, we encode positional information by utilizing relative coordinates of 2D poses to enhance the consistency between the input and output distribution. The same posture with different absolute 2D positions can be mapped to a common representation. It is beneficial to resist the interference of global motion on the prediction results. Second, we encode temporal information by establishing the connection between the current pose and other poses of the same person within a period of time. More attention will be paid to the movement changes before and after the current pose, resulting in better prediction performance on local motion with a small movement range. The ablation studies validate the effectiveness of the proposed relative information encoding method. Besides, we introduce a multi-stage optimization method to the whole framework to further exploit the positional and temporal enhanced representations. Our method outperforms state-of-the-art methods on two public datasets. Code is available at https://github.com/paTRICK-swk/Pose3D-RIE.

Accurately predicting inter-frame motion information plays a key role in video prediction tasks. In this paper, we propose a Motion-Aware Unit (MAU) to capture reliable inter-frame motion information by broadening the temporal receptive field of the predictive units. The MAU consists of two modules, the attention module and the fusion module. The attention module aims to learn an attention map based on the correlations between the current spatial state and the historical spatial states. Based on the learned attention map, the historical temporal states are aggregated to an augmented motion information (AMI). In this way, the predictive unit can perceive more temporal dynamics from a wider receptive field. Then, the fusion module is utilized to further aggregate the augmented motion information (AMI) and current appearance information (current spatial state) to the final predicted frame. The computation load of MAU is relatively low and the proposed unit can be easily applied to other predictive models. Moreover, an information recalling scheme is employed into the encoders and decoders to help preserve the visual details of the predictions. We evaluate the MAU on both video prediction and early action recognition tasks. Experimental results show that the MAU outperforms the state-of-the-art methods on both tasks.

In the emerging next-generation video coding standard AVS3, interweaved prediction was proposed to further improve affine motion compensation (AMC). With interweaved prediction, a coding block with AMC mode is divided into subblocks with two different dividing patterns to generate two auxiliary predictions, and the final prediction is calculated as a weighted-sum of the two auxiliary predictions. Although interweaved prediction can provide a significant coding gain, it is not very friendly to the hardware design. In this paper, we present a hardware friendly interweaved prediction method, in which the off-chip memory bandwidth is reduced by limiting the reference sample number per pixel and the on-chip cache bandwidth is decreased by sharing reference samples between subblocks in pattern0 and pattern1 of interweaved prediction. Simulation results show that the proposed methods can achieve 0.47% BD-rate saving on average with the Random Access (RA) configurations while reducing 52.3% off-chip memory bandwidth and 62% on-chip memory bandwidth.

Efficient intelligent analysis and video compression are critical modules in terms of the advanced surveillance system. However, existing solutions always deal each task with independent strategies, leading to low-efficiency of the surveillance system. In this paper, we propose to handle these two tasks in a hybrid manner. In particular, a hybrid surveillance processing scheme towards efficient analysis and compression is presented, where the extracted semantic information can not only be utilized in intelligent analysis tasks, but also used to improve compression efficiency by facilitating the background reference frame (BRF) generation. Moreover, we propose to remove background redundancy of surveillance video by introducing the high quality BRF, where motion metric and semantic metric work in a complementary way to ensure the accurate detection of background blocks. Experimental results manifest considerable advantages of the proposed BRF. When the proposed BRF is integrated into reference picture set (RPS), 3% coding gains are obtained compared with state-of-the-art method.

Cross-component prediction has great potential for removing the redundancy of multi-components. Recently, cross-component sample adaptive offset (CCSAO) was adopted in the third generation of Audio Video coding Standard (AVS3), which utilizes the intensities of co-located luma samples to determine the offsets of chroma sample filters. However, the frame-level based offset is rough for various content, and the edge information of classified samples is ignored. In this paper, we propose an enhanced CCSAO (ECCSAO) method to further improve the coding performance. Firstly, four selectable 1-D directional patterns are added to make the mapping between luma and chroma components more effectively. Secondly, one four-layer quad-tree based structure is designed to improve the filtering flexibility of CCSAO. Experimental results show that the proposed approach achieves 1.51%, 2.33% and 2.68% BD-rate savings for All-Intra (AI), Random-Access (RA) and Low Delay B (LD) configurations compared to AVS3 reference software, respectively. A subset improvement of ECCSAO has been adopted by AVS3.

The ubiquitous camera networks in the city brain system grow at a rapid pace, creating massive amounts of images and videos at a range of spatial-temporal scales and thereby forming the “biggest” big data. However, the sensing system often lags behind the construction of the fast-growing city brain system, in the sense that such exponentially growing data far exceed today’s sensing capabilities. Therefore, critical issues arise regarding how to better leverage the existing city brain system and significantly improve the city-scale performance in intelligent applications. To tackle the unprecedented challenges, we articulate a vision towards a novel visual computing framework, termed asdigital retina, which aligns high-efficiency sensing models with the emerging Visual Coding for Machine (VCM) paradigm. In particular, digital retina may consist of video coding, feature coding, model coding, as well as their joint optimization. The digital retina is biologically-inspired, rooted on the widely accepted view that the retina encodes the visual information for human perception, and extracts features by the brain downstream areas to disentangle the visual objects. Within the digital retina framework, three streams, i.e., video stream, feature stream, and model stream, work collaboratively over the end-edge-cloud platform. In particular, the compressed video stream serves for human vision, the compact feature stream targets for machine vision, and the model stream incrementally updates deep learning models to improve the performance of human/machine vision tasks. We have developed a prototype to demonstrate the technical advantages of digital retina, and extensive experiments have been conducted to validate that it is able to effectively support the video big data analysis and retrieval in the intelligent city system. In particular, up to$7000\times $compression ratio could be realized for visual data compression while maintaining competitive performance with pristine signal in a series of visual analysis tasks.

As 3D scanning devices and depth sensors advance, dynamic point clouds have attracted increasing attention as a format for 3D objects in motion, with applications in various fields such as immersive telepresence, navigation for autonomous driving and gaming. Nevertheless, the tremendous amount of data in dynamic point clouds significantly burden transmission and storage. To this end, we propose a complete compression framework for attributes of 3D dynamic point clouds, focusing on optimal inter-coding. Firstly, we derive the optimal inter-prediction and predictive transform coding assuming the Gaussian Markov Random Field model with respect to a spatio-temporal graph underlying the attributes of dynamic point clouds. The optimal predictive transform proves to be the Generalized Graph Fourier Transform in terms of spatio-temporal decorrelation. Secondly, we propose refined motion estimation via efficient registration prior to inter-prediction, which searches the temporal correspondence between adjacent frames of irregular point clouds. Finally, we present a complete framework based on the optimal inter-coding and our previously proposed intra-coding, where we determine the optimal coding mode from rate-distortion optimization with the proposed offline-trained λ-Q model. Experimental results show that we achieve around 17% bit rate reduction on average over competitive dynamic point cloud compression methods.

Cross-component linear model (CCLM) prediction has been repeatedly proven to be effective in reducing the inter-channel redundancies in video compression. Essentially speaking, the linear model is identically trained by employing accessible luma and chroma reference samples at both encoder and decoder, elevating the level of operational complexity due to the least square regression or max-min based model parameter derivation. In this paper, we investigate the capability of the linear model in the context of sub-sampled based cross-component correlation mining, as a means of significantly releasing the operation burden and facilitating the hardware and software design for both encoder and decoder. In particular, the sub-sampling ratios and positions are elaborately designed by exploiting the spatial correlation and the inter-channel correlation. Extensive experiments verify that the proposed method is characterized by its simplicity in operation and robustness in terms of rate-distortion performance, leading to the adoption by Versatile Video Coding (VVC) standard and the third generation of Audio Video Coding Standard (AVS3).

Human pose transfer (HPT) is an emerging research topic with huge potential in fashion design, media production, online advertising and virtual reality. For these applications, the visual realism of fine-grained appearance details is crucial for production quality and user engagement. However, existing HPT methods often suffer from three fundamental issues: detail deficiency, content ambiguity and style inconsistency, which severely degrade the visual quality and realism of generated images. Aiming towards real-world applications, we develop a more challenging yet practical HPT setting, termed as Fine-grained Human Pose Transfer (FHPT), with a higher focus on semantic fidelity and detail replenishment. Concretely, we analyze the potential design flaws of existing methods via an illustrative example, and establish the core FHPT methodology by combing the idea of content synthesis and feature transfer together in a mutually-guided fashion. Thereafter, we substantiate the proposed methodology with a Detail Replenishing Network (DRN) and a corresponding coarse-to-fine model training scheme. Moreover, we build up a complete suite of fine-grained evaluation protocols to address the challenges of FHPT in a comprehensive manner, including semantic analysis, structural detection and perceptual quality assessment. Extensive experiments on the DeepFashion benchmark dataset have verified the power of proposed benchmark against start-of-the-art works, with 12%-14% gain on top-10 retrieval recall, 5% higher joint localization accuracy, and near 40% gain on face identity preservation. Our codes, models and evaluation tools will be released at https://github.com/Lotayou/RATE.

The non-local adaptive in-loop filter (NALF) for video coding has achieved significant coding gain by exploiting image non-local self-similarity (NSS) to efficiently reduce the compression artifacts. However, the intensive computation of NALF hinders its practical deployment in video standardizations. In this paper, we propose a fast NALF optimization algorithm in parallel-computing framework by leveraging the massive parallel execution resources of GPU. First, the computational complexity of original NALF is analyzed in depth, then the pipelines of computational-intensive modules are re-designed to adapt to the general-purpose GPU with more parallel-friendly consideration. Specifically, we speed up the NALF by optimizing thread allocation to maximize the parallelism degree and elaborately designing the GPU block dimension to avoid access conflict. The group-level and pixel-level parallelization for collaboratively filtering and patch matching modules are designed respectively. To reduce the cost in data transmission, the whole filtering process is implemented on GPU by taking the advantage of low data dependency in NALF. Extensive experimental results show that the proposed fast NALF optimization using GPU architecture achieves high-speeed processing while maintaining the significant coding performance of original NALF, which shows the potential of NALF in the future video coding standard.

Cross-component prediction, which takes advantage of inter-channel correlations, predicts the chroma block with the luma reconstructed block according to associated linear model. Instead of involving all available reference samples in building the linear model, in this paper, we propose a sub-sampled approach that utilizes at most four neighboring chroma samples and their corresponding down-sampled luma samples, leading to significantly reduced operations in the derivation of model parameters at both encoder and decoder. The proposed scheme is hardware friendly in terms of the overheads of memory access and clock cycles, and greatly benefits the practical implementations of the emerging video coding standard in real applications. Extensive experiments reveal that the proposed sub-sampled method provides simple operations and robust coding performance, leading to the adoption by Versatile Video Coding (VVC) Standard and the third generation Audio Video Coding Standard (AVS3).

To achieve higher compression efficiency, the new developing video coding standard Versatile Video Coding(VVC) introduced a large amount of new coding technologies, which increases the computational complexity of the decoder significantly. Among these technologies, the inter prediction methods, including affine motion compensation and decoder side motion vector refinement(DMVR), make inter prediction become the most time consuming module and bring new challenges for real-time decoding. In this paper, we proposed an efficient GPU-based motion compensation scheme to speedup the decoding. Through re-partition of coding unit(CU) according to the data dependency and different thread organization methods for different situation, the computational resources of GPU are utilized efficiently. Experiments on NVIDIA GeForce RTX 2080Ti GPU showed the motion compensation can be done in 5ms for Ultra HD 4K, which means the decoding speed is accelerated by 16 times compared to the VVC reference software on CPU.

In traditional intra prediction, nearest reference samples are utilized to generate the prediction block. Although more directional intra modes and reference lines have been utilized, encoders could not predict complex content with only the 10-cal reference samples efficiently. To address this issue, a twostep progressive prediction method combining local and nonlocal information is proposed. The non-local information can be obtained through template matching based prediction, and the local information can be derived by the high frequency coefficients from the first prediction step. Experimental results show that the proposed method can achieve 0.87% BD-rate reduction in VTM-7.0. In particular, the method is of significant advantages over prediction schemes using only non-local information.

HTTP based video streaming is widely adopted in video service and its adaptive bitrate algorithms have attracted a lot of studies in recent years. However, most of the algorithms are designed for on-demand video service and are not suitable for live video streaming service, which is sensitive to end-to-end latency. In this paper, we propose a live video streaming framework based on HTTP/2 which enables frame dropping for low latency environment. Firstly, we formulate live video streaming model and QoE model considering frame dropping. Then, an optimization problem is formulated aiming at high quality of video streaming. Furthermore, to solve this problem, we propose a live video streaming adaptation algorithm with frame dropping based on Model Predictive Control (MPC). Finally, extensive experiments are conducted to evaluate the proposed method over realistic traces with general Adaptive Bitrate Algorithms (ABR). Compared with the optimal solution, the proposed method achieves comparable performance with only 8.06% quality loss.

In recent years, large amount of ground truth data is typically required to feed into the supervised depth estimation models to produce satisfactory performance, while it is usually costly and impracticable to acquire depth ground truth. In this paper, we propose a self-supervised joint deep learning pipeline for depth and pose estimation of monocular video sequences, which uses the cycle generation adversarial network structure to extend the existing reconstruction loss function based on photometric consistency. The generation function of the algorithm learns to synthesize the adjacent image to predict the depth map and the relative target pose, and the discriminant function learns the dispersion of the monocular images to correctly classify the realism of the composite image. At the same time, a reconstruction loss function based on pose consistency is used to assist the generator function in training. Extensive experimental results on the KITTI dataset show superior performance of the proposed method.

To improve the coding performance, geometric partition (GEO) was proposed for the upcoming VVC standard. GEO provides 140 partition candidates. The index of optimal GEO mode needs to be signaled explicitly. Considering different structural characteristics of different CUs and the correlation between spatial adjacent blocks and temporal collocated blocks, we propose a GEO mode prediction strategy by constructing a Most Probable Mode (MPM) list to reduce the overhead of GEO index and improve coding efficiency. Based on the observation of the high correlation between the partition mode and object boundaries, an edge-directed geometric partition scheme is proposed to construct the MPM list according to spatio-temporal edge information. The proposed method provides an objective BD-rate gain of 0.58% and 1.00% on average for RA and LDB configurations compared to VTM-6.0. Besides, it also promotes the visual quality of object boundaries.

The ability to produce convincing textural details is essential for the fidelity of synthesized person images. Existing methods typically follow a “warping-based” strategy that propagates appearance features through the same pathway used for pose transfer. However, most fine-grained features would be lost during down-sampling, leading to over-smoothed clothes and missing details in the output images. In this paper we presents RATE-Net, a novel framework for synthesizing person images with sharp texture details. The proposed framework leverages an additional texture enhancing module to extract appearance information from the source image and estimate a fine-grained residual texture map, which helps to refine the coarse estimation from the pose transfer module. In addition, we design an effective alternate updating strategy to promote mutual guidance between two modules for better shape and appearance consistency. Experiments conducted on DeepFashion benchmark dataset have demonstrated the superiority of our framework compared with existing networks.

Existing face restoration researches typically rely on either the image degradation prior or explicit guidance labels for training, which often lead to limited generalization ability over real-world images with heterogeneous degradation and rich background contents. In this paper, we investigate a more challenging and practical "dual-blind" version of the problem by lifting the requirements on both types of prior, termed as "Face Renovation"(FR). Specifically, we formulate FR as a semantic-guided generation problem and tackle it with a collaborative suppression and replenishment (CSR) approach. This leads to HiFaceGAN, a multi-stage framework containing several nested CSR units that progressively replenish facial details based on the hierarchical semantic guidance extracted from the front-end content-adaptive suppression modules. Extensive experiments on both synthetic and real face images have verified the superior performance of our HiFaceGAN over a wide range of challenging restoration subtasks, demonstrating its versatility, robustness and generalization ability towards real-world face processing applications. Code is available at https://github.com/Lotayou/Face-Renovation.

AVS3 is the newest video coding standard developed by AVS (Audio Video coding Standard) group. AVS3 adopted QTBT(Quad-tree and Binary-tree) plus EQT(Extended quad-tree) block partition scheme, which makes the split process more flexible. The CU split structure is determined by a brute-force rate-distortion optimization (RDO) search. After the whole RDO search, the CU partition with minimum RD cost is selected. The flexible block partition and thorough RDO search bring promising coding gain while extremely complicate the encoder. To reduce the computational complexity of the CU split decision process in AVS3, this paper proposed a spatial information based fast split decision algorithm. In the proposed algorithm, the predicted value of split complexity was calculated firstly according to the information of spatial neighboring blocks. Then the predicted value was used to decide whether to split current CU or not. The experimental results show that the proposed algorithm resulted in average 31.03% encoding time saving with average 0.54% BD-BR loss for Random Access (RA) configuration. The proposed algorithm can greatly reduce the computational complexity of the CU split decision process with negligible performance loss.

Recent years have witnessed an explosion of machine visual intelligence. While impressive performance on visual analysis has been achieved by powerful Deep-Learning-based models, the texture and feature distortion caused by image and video coding is becoming a challenge in practical situations. In this paper, a new rate control scheme is proposed to improve visual analysis performance on coded video frames. Firstly, a new kind of visual analysis distortion is introduced to build a Rate-Joint-Distortion model. Secondly, the Rate-Joint-Distortion Optimization problem is solved by using Lagrange multiplier method, and the relationship between rate and Lagrange multiplier λ is described by a hyperbolic model. Thirdly, a logarithmic λ - QP model is established to achieve minimum Rate-Joint-Distortion cost for given λs. The experimental results show that the proposed scheme can improve visual analysis performance with stable bits used for coding.

In recent years, the image and video coding technologies have advanced by leaps and bounds. However, due to the popularization of image and video acquisition devices, the growth rate of image and video data is far beyond the improvement of the compression ratio. In particular, it has been widely recognized that there are increasing challenges of pursuing further coding performance improvement within the traditional hybrid coding framework. Deep convolution neural network which makes the neural network resurge in recent years and has achieved great success in both artificial intelligent and signal processing fields, also provides a novel and promising solution for image and video compression. In this paper, we provide a systematic, comprehensive and up-to-date review of neural network-based image and video compression techniques. The evolution and development of neural network-based compression methodologies are introduced for images and video respectively. More specifically, the cutting-edge video coding techniques by leveraging deep learning and HEVC framework are presented and discussed, which promote the state-of-the-art video coding performance substantially. Moreover, the end-to-end image and video coding frameworks based on neural networks are also reviewed, revealing interesting explorations on next generation image and video coding frameworks/standards. The most significant research works on the image and video coding related topics using neural networks are highlighted, and future trends are also envisioned. In particular, the joint compression on semantic and visual information is tentatively explored to formulate high efficiency signal representation structure for both human vision and machine vision, which are the two dominant signal receptors in the age of artificial intelligence.

In recent years, supervised deep learning methods have shown a great promise in dense depth estimation. However, massive high-quality training data are expensive and impractical to acquire. Alternatively, self-supervised learning-based depth estimators can learn the latent transformation from monocular or binocular video sequences by minimizing the photometric warp error between consecutive frames, but they suffer from the scale ambiguity problem or have difficulty in estimating precise pose changes between frames. In this paper, we propose a joint self-supervised deep learning pipeline for depth and ego-motion estimation by employing the advantages of adversarial learning and joint optimization with spatial-temporal geometrical constraints. The stereo reconstruction error provides the spatial geometric constraint to estimate the absolute scale depth. Meanwhile, the depth map with an absolute scale and a pre-trained pose network serves as a good starting point for direct visual odometry (DVO). DVO optimization based on spatial geometric constraints can result in a fine-grained ego-motion estimation with the additional backpropagation signals provided to the depth estimation network. Finally, the spatial and temporal domain-based reconstructed views are concatenated, and the iterative coupling optimization process is implemented in combination with the adversarial learning for accurate depth and precise ego-motion estimation. The experimental results show superior performance compared with state-of-the-art methods for monocular depth and ego-motion estimation on the KITTI dataset and a great generalization ability of the proposed approach.

Existing methods for image captioning are usually trained by cross entropy loss, which leads to exposure bias and the inconsistency between the optimizing function and evaluation metrics. Recently it has been shown that these two issues can be addressed by incorporating techniques from reinforcement learning, where one of the popular techniques is the advantage actor-critic algorithm that calculates per-token advantage by estimating state value with a parametrized estimator at the cost of introducing estimation bias. In this paper, we estimate state value without using a parametrized value estimator. With the properties of image captioning, namely, the deterministic state transition function and the sparse reward, state value is equivalent to its preceding state-action value, and we reformulate advantage function by simply replacing the former with the latter. Moreover, the reformulated advantage is extended to n-step, which can generally increase the absolute value of the mean of reformulated advantage while lowering variance. Then two kinds of rollout are adopted to estimate state-action value, which we call self-critical n-step training. Empirically we find that our method can obtain better performance compared to the state-of-the-art methods that use the sequence level advantage and parametrized estimator respectively on the widely used MSCOCO benchmark.

To reduce the perceptual redundancy in the video coding process, human visual system (HVS)-based visual attention and visual sensitivity can be utilized due to their intrinsic natures. Just Noticeable Distortion (JND) is one of widely used models to simulate human visual sensitivity, while visual saliency map has been popular for years in image processing to describe the visual attention feature, which has been proved by the ability to enhance the visual sensitivity effect. In this paper, we proposed a perceptual video coding (PVC) scheme with visual saliency modulated JND model to suppress the DCT coefficient without resulting in noteworthy subjective quality degradation. The experimental results show that the PVC scheme with the proposed VS-JND model can save bit rates up to 35.58% in high bit rates case with the similar subjective quality compared with that of HEVC software reference code HM 16.12.

Owing to the ability of removing compression artifacts, extensive in-loop filters have been proposed for video coding standards. They are performed after the reconstruction of all coding units (CUs), however, none of them has been taken into account in the mode decision when coding each CU. To address this issue and make the rate-distortion optimization (RDO) more precise for each CU, we introduce a low-pass filter when checking the rate-distortion cost after the reconstruction of each CU. Specifically, based on Haar wavelet, the reconstructed block is transformed to the frequency domain, and then an adaptive wavelet domain filter (AWF) is proposed to suppress the quantization noises in coded blocks. To be adaptive, the filter strength varies from CU to CU according to the texture complexity and quantization parameters (QPs). Experimental results show that the proposed method can reduce the compression artifacts and improve both the objective and subjective quality.

Motivated by the insight of Marr on generative image representations, we propose a layered conceptual image compression scheme by integrating the advantages of both variational auto-encoders (VAEs) and generative adversarial networks (GANs). In particular, the image is represented by two layers: the low-dimensional codes of the stochastic textures encoded by the VAE and the geometric structures characterized by edge maps. Subsequently, the edge maps and latent codes are compressed individually such that the final bit streams are formed in a combined manner. At the decoder side, the GAN synthesizes the decoded images on the basis of the latent codes and the reconstructed edge maps. Experimental results demonstrate that our proposed scheme achieves better visual reconstruction quality than the traditional image compression algorithms such as JPEG, JPEG2000 and HEVC (intra coding) in the low bit rate coding scenarios.

Contrast distortion has a significant influence on the perceptual quality of an image, which may be generated in various image processing procedures. In this paper, we propose a no-reference image quality assessment (IQA) algorithm for contrast-distorted images based on hybrid features from information and appearance attributes. From information attribute aspect, we utilize the basic information feature to quantify the information of the visible part and the extended information feature further containing the invisible part of an image. From the appearance aspect, we propose an efficient perceptual contrast and colorfulness index to capture the direct visual changes. With the hybrid information attribute and appearance attribute features, support vector regression (SVR) is utilized to learn an IQA model to predict the quality of contrast distorted images. Extensive experimental results on CCID2014 and TID2013 databases further demonstrate the superior performance and robustness of the proposed method.

Lossy compression introduces complex compression artifacts such as the blocking, ringing and blurring artifacts, making decoded videos unpleasant for human visual system. In this paper, we propose a coding prior based high efficiency restoration algorithm to remove these compression artifacts. To improve the quality of the restored compressed videos, we take full advantage of side information from coding streams as coding prior which is ignored or not fully exploited by most existing post-processing methods. In particular, the unfiltered frames and the prediction frames are derived from coding streams which are utilized as coding priors and a high efficiency neural network is designed for these information to improve the overall quality of restored videos. Extensive experimental results on HEVC coding streams demonstrate that our proposed method can significantly improve both the objective and subjective quality of compressed videos.

In this paper, a composite long-term reference picture coding algorithm for Versatile Video Coding (VVC) standard is proposed that can be used for the scenarios with large motionless area. In order to generate a high quality composite reference, a new generation method based on combined cost is adopted to compose the picture for long-term reference. After the generation of a composite long-term reference, VVC / High Efficiency Video Coding(HEVC) reference management high-level syntax is exploited to encode the reference as a no-display picture. Experiments are conducted under the common test condition in VVC and the results demonstrate that the proposed algorithm can achieve 2.55% bit-rate reduction on average with only 10% encoding time increase and negligible decoding time increase. For the certain sequence with large background area, e.g. BasketballDrill, the coding gain can be up to 13.47%. The proposed scheme was adopted by Joint Video Experts Team as an encoder coding tool for VVC standard under low delay configuration.

Interpolation is a very important module in inter prediction for any video decoder, e.g. AVS2 [1] and HEVC [2], which occupies most of the time in the whole decoding process . Thus, the real-time decoder is largely limited by the speed of inter prediction. To solve this problem, we propose an efficient GPU-based interpolation framework for inter prediction. Through optimizing shared memory allocation and thread scheduling on the GPU side, GPU are utilized efficiently and inter prediction is accelerated effectively. The experimental results on AVS2 show that for all Ultra HD 4K, WQXGA and full HD video sequences tested, the inter prediction acceleration ratio is over 6 times, and the average processing time is up to 1.25ms, 0.75ms and 0.45ms, respectively, with the NVIDIA GeForce GTX 1080TI GPU.

In the emerging video coding standard, Versatile Video Coding (VVC), a quadtree with nested multi-type tree (MTT) using binary and ternary tree structure was proposed. MTT brings significant coding efficiency but increases the encoding complexity. In this paper, a look-ahead prediction based coding unit size pruning algorithm is proposed to cut down redundant MTT partitions. The proposed scheme aims to identify the unnecessary partition direction in advance and consists of two steps, i.e. SATD-based mode decision (SMD) for possible blocks and refined cost derivation based on rate-distortion optimization. Experimental results show that the proposed method can save 41% encoder time with only 0.84% increase in bit rate on average.

The generative deep learning models such as the generative adversarial networks (GAN) have been shown to efficiently generate visually appealing images by learning the natural scene statistics. However, the signal fidelity, instead of the visual quality, has been largely ignored in the generation process, especially for the highly structural regions. In this paper, we introduce a region-aware visual signal restoration scheme to achieve a good balance between visual quality and fidelity. As a specific example of this framework, we develop an enhanced decoding scheme with hybrid neural networks, such that the base fidelity layer and texture quality enhancement layer are combined adaptively to restore the compressed images. The efficiency of the proposed framework is demonstrated with extensive experimental results, which show favorable performance against the state-of-the-art methods.

In this paper, we propose a scalable image compression scheme, including the base layer for feature representation and enhancement layer for texture representation. More specifically, the base layer is designed as the deep learning feature for analysis purpose, and it can also be converted to the fine structure with deep feature reconstruction. The enhancement layer, which serves to compress the residuals between the input image and the signals generated from the base layer, aims to faithfully reconstruct the input texture. The proposed scheme can feasibly inherit the advantages of both compress-then-analyze and analyze-then-compress schemes in surveillance applications. The performance of this framework is validated with facial images, and the conducted experiments provide useful evidences to show that the proposed framework can achieve better rate-accuracy and rate-distortion performance over conventional image compression schemes.

In this paper, we propose an unsupervised joint deep learning pipeline for depth and ego-motion estimation that explicitly incorporated with traditional spatial-temporal geometric constraints. The stereo reconstruction error provides the spatial geometric constraint to estimate the absolute scale depth. Meanwhile, the depth map with absolute scale and a pre-trained pose network serve as a good starting point for direct visual odometry (DVO), resulting in a fine-grained ego-motion estimation with the additional back-propagation signals provided to the depth estimation network. The proposed joint training pipeline enables an iterative coupling optimization process for accurate depth and precise ego-motion estimation. The experimental results show the state-of-the-art performance for monocular depth and ego-motion estimation on the KITTI dataset and a great generalization ability of the proposed approach.

Audio and Video coding Standard-3 (AVS-3) is an emerging next-generation video coding standard beyond AVS-2. The Quad-Tree/Binary-Tree (QTBT) plus Extend Quad-Tree (EQT) partitioning structure has been adopted in AVS3. In this paper, Unsymmetrical Quad-Tree (UQT) partitioning is proposed to explore the potential coding efficiency of partitioning further. Particularly, UQT partitioning splits a parent block into four sub-blocks asymmetrically with 1:4:2:1 or 1:2:4:1 ratio. Not only UQT partitioning can complement the current QTBT plus EQT partition structure but also can characterize the pattern of multifarious video content more effectively. Experiments are conducted on the HPM3.2 platform and the simulation results demonstrate that the proposed UQT partitioning can achieve 0.66%, 0.77% and 0.75% BD-bitrate savings for Y, Cb and Cr components on average under the random-access configuration, respectively.

Deep learning based video coding tools development has been an emerging topic recently. In this paper, we propose a novel deep convolutional neural network (CNN) based in-loop filter algorithm for the third generation of Audio Video Coding Standard (AVS3). Specifically, we first introduce a residual block based CNN model with global identity connection for the luminance in-loop filter to replace conventional rule-based algorithms in AVS3. Subsequently, the reconstructed luminance channel is deployed as textural and structural guidance for chrominance filtering. The corresponding syntax elements are also designed for the CNN based in-loop filtering. In addition, we build a large scale database for the learning based in-loop filtering algorithm. Experimental results show that our method achieves on average 7.5%, 16.9% and 18.6% BD-rate reduction under all intra (AI) configuration on common test sequences. In particular, the performance for 4K videos is 6.4%, 15.5% and 17.5% respectively. Moreover, under random access (RA) configuration, the proposed method brings 3.3%, 14.4%, and 13.6% BD-rate reduction separately.

The paper presents a systematic and comprehensive overview of the third generation of Audio Video Standard (AVS) in China. The AVS standards continuously attract extensive attention both domestically and worldwide along with the industrialization of AVS2 standard and the broadcasting of China Central Television (CCTV) 4K channel. Currently, AVS3 has adopted many novel coding techniques including block partitioning structure, intra/inter and transform coding tools. The compression performance has been significantly improved that the latest version of AVS3 obtains about 24.3% and 26.88% bit-rate reduction against AVS2 and HEVC respectively under 4K resolution sequences.

In this paper, we provide a systematical overview and analysis on the joint feature and texture representation framework, which aims to smartly and coherently represent the visual information with the front-end intelligence in the scenario of video big data applications. In particular, we first demonstrate the advantages of the joint compression framework in terms of both reconstruction quality and analysis accuracy. Subsequently, the interactions between visual feature and texture in the compression process are further illustrated. Finally, the future joint coding scheme by incorporating the deep learning features is envisioned, and future challenges toward seamless and unified joint compression are discussed. The joint compression framework, which bridges the gap between visual analysis and signal-level representation, is expected to contribute to a series of applications, such as video surveillance and autonomous driving.

In this paper, we propose an enhanced bi-prediction scheme based on the convolutional neural network (CNN) to improve the rate-distortion performance in video compression. In contrast to the traditional bi-prediction strategy which computes the linear superposition as the predictive signals with pixel-to-pixel correspondence, the proposed scheme employs CNN to directly infer the predictive signals in a data-driven manner. As such, the predicted blocks are fused in a nonlinear fashion to improve the coding performance. Moreover, the patch-to-patch inference strategy with CNN also improves the prediction accuracy since the patch-level information for the prediction of each individual pixel can be exploited. The proposed enhanced bi-prediction scheme is further incorporated into the high-efficiency video coding standard, and the experimental results exhibit a significant performance improvement under different coding configurations.

Recently, convolutional neural network (CNN) has attracted tremendous attention and has achieved great success in many image processing tasks. In this paper, we focus on CNN technology combined with image restoration to facilitate video coding performance and propose the content-aware CNN based in-loop filtering for high-efficiency video coding (HEVC). In particular, we quantitatively analyze the structure of the proposed CNN model from multiple dimensions to make the model interpretable and optimal for CNN-based loop filtering. More specifically, each coding tree unit (CTU) is treated as an independent region for processing, such that the proposed content-aware multimodel filtering mechanism is realized by the restoration of different regions with different CNN models under the guidance of the discriminative network. To adapt the image content, the discriminative neural network is learned to analyze the content characteristics of each region for the adaptive selection of the deep learning model. The CTU level control is also enabled in the sense of rate-distortion optimization. To learn the CNN model, an iterative training method is proposed by simultaneously labeling filter categories at the CTU level and fine-tuning the CNN model parameters. The CNN based in-loop filter is implemented after sample adaptive offset in HEVC, and extensive experiments show that the proposed approach significantly improves the coding performance and achieves up to 10.0% bit-rate reduction. On average, 4.1%, 6.0%, 4.7%, and 6.0% bit-rate reduction can be obtained under all intra, low delay, low delay P, and random access configurations, respectively.

Motion compensation has been widely employed for removing temporal redundancies in typical hybrid video coding framework. The popular video compression standards, such as H.264/AVC and HEVC, adopt the block based partitioning model to describe the motion filed due to its high compression efficiency and relatively low computational complexity. However, block based motion compensation may not align with the actual object motion boundaries, potentially limiting the compression efficiency. In view of this, we propose a three-zone segmentation based motion compensation scheme to improve the description accuracy of motion field as well as the coding efficiency. In particular, the segmentation information is implied in the reference frame instead of being explicitly signalled. Based on the segmentation information, three motion compensation zones can be identified, including one edge, one foreground, and one background zone. The foreground zone is motion compensated by the signalled motion vector of the block, and the background zone is motion compensated by the motion information implicitly derived from the local motion field. Regarding the edge zone, it is viewed as an overlapped area and the weighted compensation strategy is adopted. The proposed algorithm is implemented into the reference software VTM-1.0 of Versatile Video Coding (VVC), and simulation results show that the algorithm can achieve 1.14% and 1.06% bitrate savings for random access and lowdelay configurations, respectively.

In this paper, we propose an efficient inter prediction scheme by introducing the deep virtual reference frame (VRF), which serves better reference in the temporal redundancy removal process of video coding. In particular, the high quality VRF is generated with the deep learning-based frame rate up conversion (FRUC) algorithm from two reconstructed bi-directional frames, which is subsequently incorporated into the reference list serving as the high quality reference. Moreover, to alleviate the compression artifacts of VRF, we develop a convolutional neural network (CNN)-based enhancement model to further improve its quality. To facilitate better utilization of the VRF, a CTU level coding mode termed as direct virtual reference frame (DVRF) is devised, which achieves better trade-off between compression performance and complexity. The proposed scheme is integrated into HM-16.6 and JEM-7.1 software platforms, and the simulation results under random access (RA) configuration demonstrate significant superiority of the proposed method. When adding VRF to RPS, more than 6% average BD-rate gain is achieved for HEVC test sequences on HM-16.6, and 0.8% BD-rate gain is observed based on JEM-7.1 software. Regarding the DVRF mode, 3.6% bitrate saving is achieved on HM-16.6 with the computational complexity effectively reduced.

Motion estimation (ME) plays a crucial role in removing the temporal redundancy for video compression. However, during the encoding process a substantial computational burden is imposed by ME due to the exhaustive evaluations of possible candidates within the searching window. In view of the increasing computing capacity of GPU, we propose a GPU-based low delay parallel ME scheme for high efficiency video coding (HEVC). In particular, considering the quadtree coding structure of HEVC, we achieve the parallelization in a hierarchical way by optimizing the ME process in a coding tree unit (CTU), prediction unit (PU), and motion vector (MV) layers. Specifically, in the CTU layer, a novel motion vector predictor determination scheme is proposed to alleviate the side effects of inaccurate MV prediction due to the removal of the CTU-level dependency. In the PU layer, a novel indexing table is particularly designed to realize an efficient cost derivation strategy. As such, the cost of each PU can be computed in a convenient and efficient manner. In an MV layer, we propose a compact descriptor to represent MV and its corresponding cost as a whole, such that the redundant branches can be further avoided in the searching process. With such an optimization strategy, the proposed scheme can completely save the encoding time for ME on CPU. Experimental results demonstrate that the proposed scheme can achieve 41% encoding time savings with the ME acceleration up to 12.7 times, and the incurred BD-BR loss is only 0.52% on average. Moreover, further experimental results show that the proposed GPU-based ME can achieve up to 200 times acceleration compared to the full search ME on CPU.

Motion compensation plays a key role in high efficiency video coding. The popular video compression standards, such as H.264/AVC and HEVC, adopt block based motion compensation technique due to its high compression efficiency and relatively low computational complexity. However, block based motion compensation may not be in accordance with the actual object boundary, potentially leading to low prediction accuracy especially in the high-texture areas. In this paper, we propose a locally refined motion compensation method to address this issue. In particular, the image segmentation is applied on the prediction block indicated by a motion vector rather than the original block to avoid explicit signaling. Furthermore, the local content is analyzed to select one segmented region and subsequently the prediction of this region is generated based on the local motion filed. Experimental results show that the proposed algorithm can achieve 0.8%, 1.1% and 1.7% bitrate savings for Random Access, Lowdelay-B and Lowdelay-P configurations respectively without introducing noticeable computational complexity.

Point cloud has attracted more and more attention in 3D object representation, especially in free-view rendering. However, it is challenging to efficiently deploy the point cloud due to its huge data amount with multiple attributes including coordinates, normal and color. In order to represent point clouds more compactly, we propose a novel point cloud compression method for attributes, based on geometric clustering and Normal Weighted Graph Fourier Transform (NWGFT). Firstly, we divide the entire point cloud into different sub-clouds via K-means based on the geometry to acquire sub-clouds with more uniform structures, which enables efficient representation with less cost. Secondly, for the purpose of reducing the redundancy further, we apply NWGFT to each sub-cloud, in which graph edge weights are derived from the similarity in normal. Finally, extensive experimental results show that, compared with traditional transform based point cloud compression, the proposed approach achieves about 34.34% bit rate reduction on average for Y components of color.

Surface Light Fields (SLF) have previously been proposed for representing 3D scenes under complex lighting conditions' enabling immersive viewing experiences from arbitrary observation directions. In this work, we present a new approach for SLF representation and a framework for SLF compression. Specifically, the SLF is compactly represented in a B-Spline wavelet basis. This representation is capable of modeling diverse surface materials and complex lighting conditions. The coefficients of the B-Spline wavelet are then compressed by removing the spatial redundancy over surface points. Compared with image based light field compression, the proposed scheme is functionally advanced because it enables rendering objects from arbitrary viewpoints with both good quality and high efficiency. In terms of bitrate and distortion, experimental results have shown that the proposed method can achieve competitive performance but with much lower decoder computational complexity, indicating its potential in practical virtual and augmented reality applications.

Traditional video coding standards H.264/AVC and HEVC adopt inter-frame prediction with single prediction (P frame) and bi-directional prediction (B frame), in which one and two hypotheses are utilized to generate the final prediction. Though more hypotheses have been proved to significantly improve the prediction accuracy, the additional bits used for signaling the motion information may decrease the overall rate-distortion performance conversely. In this paper, we propose a multi-hypothesis prediction scheme based on implicit derivation of motion vector. In the proposed scheme, the first prediction block is generated identically with the method in HEVC. Subsequently, motion estimation is applied on each reference frame by taking the first prediction block as template, and the search results serve as the additional hypothesis candidates. In this manner, the overheads of motion vector can be reduced. Simulation results show that the proposed algorithm can achieve average 1.1% coding gain compared to HEVC.

In the latest Joint Video Exploration Team (JVET) development, the quadtree plus binary tree (QTBT) block partition structure is proposed for more flexible block partitioning. Compared to the quadtree partitioning in HEVC, QTBT can achieve better compression performance at the expense of significantly increased encoding complexity. To address this issue, we propose a convolution neural network (CNN) oriented fast QTBT partitioning decision algorithm for inter coding. We analyze the QTBT in a statistical way, which effectively guides us to design the architecture of the CNN. Furthermore, the false prediction risk is controlled based on temporal correlation to improve the robustness of the scheme. Experimental results show that the proposed algorithm can speed up QTBT block partition structure by reducing 35% encoding time on average with only 0.55% increase in bit rate, which enables its applications in practical scenarios.

Inter prediction serves as the foundation of prediction based hybrid video coding framework. The state-of-the-art video coding standards employ the reconstructed frames as the references, and the motion vectors which convey the relative position shift between the current block and the prediction block are explicitly signalled in the bitstream. In this paper, we propose a high efficient inter prediction scheme by introducing a new methodology based on virtual reference frame, which is effectively generated with the deep neural network such that the motion data does not need to be explicitly signalled. In particular, the high quality virtual reference frame is generated with the deep learning based frame rate up-conversion (FRUC) algorithm from two reconstructed bi-prediction frames. Subsequently, a novel CTU level coding mode termed as direct virtual reference frame (DVRF) mode, is proposed to adaptively compensate for the current to-be-coded block in the sense of rate-distortion optimization (RDO). The proposed scheme is integrated into the HM-16.6 software, and experimental results demonstrate significant superiority of the proposed method, which provides more than 3% coding gains on average for HEVC test sequences.

Lossy image compression usually introduces undesired compression artifacts, such as blocking, ringing and blurry effect{###} S, especially in low bit rate coding scenarios. Although many algorithms have been proposed to reduce these compression artifacts, most of them are based on image local smoothness prior, which usually leads to over-smoothing around the areas with distinct structures, e.g., edges and textures. In this paper, we propose a novel framework to enhance the perceptual quality of decoded images by well preserving the edge structures and predicting visually pleasing textures. Firstly, we propose an edge-preserving generative adversarial network (EP-GAN) to achieve edge restoration and texture generation simultaneously. Then, we elaborately design an edge fidelity regularization term to guide our network, which jointly utilizes the signal fidelity, feature fidelity and adversarial constraint to reconstruct high quality decoded images. Experimental results demonstrate that the proposed EP-GAN is able to efficiently enhance decoded images at low bit rate and reconstruct more perceptually pleasing images with abundant textures and sharp edges.

In this paper, we propose a novel light field image compression scheme by exploiting the intrinsic similarity of light field images with deep learning. In particular, instead of conveying all LF sub-views, only sparsely sampled LF sub-views are compressed and the remaining sub-views are reconstructed from the coded sub-views in the neighbourhood with convolutional neural network (CNN). To jointly suppress the artifacts induced in compression and reconstruct the un-coded views with high geometric accuracy, a multi-view joint enhancement network is introduced to improve the coding performance. Extensive experiments show the superior compression performance of our scheme compared with the state-of-the-art methods.

In this paper, we propose a frame level rate control scheme for screen content video (SCV) coding. Firstly, inter frame correlation (IFC) is computed to describe the similarity of successive frames for SCV. Based on the IFC, the frames in SCV are further classified into key frames (KFs) and non-key frames (NKFs). Secondly, an efficient bit rate allocation scheme is proposed for KFs and NKFs based on the IFC and Hypothetical Reference Decoder (HRD), which can well ensure the R-D performance. Finally, rate quantization (R-Q) models for KFs and NKFs are established by considering the characteristics of the two types of frames, respectively. As such, accurate coding bits allocation at frame level can be achieved by the proposed rate control algorithm. The proposed rate control scheme introduces low coding complexity burden as well as low coding delay, enabling its application in real screen sharing scenarios. Compared with the recommended rate control scheme in HEVC, experimental results demonstrate that the proposed rate control scheme can achieve better performance in terms of both rate control accuracy and R-D performance. In particular, the average bit rate mismatch is within 1.4%, and the average R-D performance improvements are over 19%.

The state-of-the-art High Efficiency Video Coding (HEVC) standard adopts the bi-prediction to improve the coding efficiency for B frame. However, the underlying assumption of this technique is that the motion field is characterized by the block-wise translational motion model, which may not be efficient in the challenging scenarios such as rotation and deformation. Inspired by the excellent signal level prediction capability of deep learning, we propose a bi-directional motion compensation algorithm with convolutional neural network, which is further incorporated into the video coding pipeline to improve the performance of video compression. Our network consists of six convolutional layers and a skip connection, which integrates the prediction error detection and non-linear signal prediction into an end-to-end framework. Experimental results show that by incorporating the proposed scheme into HEVC, up to 10.5% BD-rate savings and 3.1% BD-rate savings on average for random access (RA) configuration have been observed.

In order to compensate the shortcomings of existing in-loop filters only based on local correlation in video coding standards, many non-local based loop filters with high coding performance and computational complexity are proposed. In this paper, we propose a fast block matching algorithm, adaptive two-step block matching algorithm, based on our previous work, structure-driven adaptive non-local filter (SANF) which is computationally intensive because of the high complexity of block matching and singular value decomposition (SVD). Our proposed algorithm based on image spatial statistical characteristics utilizes fixed template to select adaptive number of similar blocks according to image content, which can reduce up to 75.2% search candidates compared to exhaustive search in SANF and the adaptive determination strategy can remove blocks with less relation to reference block in similar block group which have little help for compression performance, and the remove of them can reduce the computational complexity of SVD. Our proposed optimization algorithm can save encoding and decoding time significantly with negligible performance loss, which achieves 70.7%, 84.4%, 80.82% and 81.95% decoding time saving with only 0.13%, 0.05%, 0.13% and 0.15% increases of BD-rate for AI, RA, LDB and LDP configurations, respectively compared to original SANF in JEM-7.0.

Transform and quantization account for a considerable amount of computation time in video encoding process. However, there are a large number of discrete cosine transform coefficients which are finally quantized into zeros. In essence, blocks with all zero quantized coefficients do not transmit any information, but still occupy substantial unnecessary computational resources. As such, detecting all-zero block (AZB) before transform and quantization has been recognized to be an efficient approach to speed up the encoding process. Instead of considering the hard-decision quantization (HDQ) only, in this paper, we incorporate the properties of soft-decision quantization into the AZB detection. In particular, we categorize the AZB blocks into genuine AZBs (G-AZB) and pseudo AZBs (P-AZBs) to distinguish their origins. For G-AZBs directly generated from HDQ, the sum of absolute transformed difference-based approach is adopted for early termination. Regarding the classification of P-AZBs which are generated in the sense of rate-distortion optimization, the rate-distortion models established based on transform coefficients together with the adaptive searching of the maximum transform coefficient are jointly employed for the discrimination. Experimental results show that our algorithm can achieve up to 24.16% transform and quantization time-savings with less than 0.06% RD performance loss. The total encoder time saving is about 5.18% on average with the maximum value up to 9.12%. Moreover, the detection accuracy of larger TU sizes, such as and can reach to 95% on average.

In the latest Joint Video Exploration Team development, the quadtree plus binary tree (QTBT) block partitioning structure has been proposed for future video coding. Compared to the traditional quadtree structure of High Efficiency Video coding (HEVC) standard, QTBT provides more flexible patterns for splitting the blocks, which results in dramatically increased combinations of block partitions and high computational complexity. In view of this, a confidence interval based early termination (CIET) scheme is proposed for QTBT to identify the unnecessary partition modes in the sense of rate-distortion (RD) optimization. In particular, a RD model is established to predict the RD cost of each partition pattern without the full encoding process. Subsequently, the mode decision problem is casted into a probabilistic framework to select the final partition based on the confidence interval decision strategy. Experimental results show that the proposed CIET algorithm can speed up QTBT block partitioning structure by reducing 54.7% encoding time with only 1.12% increase in terms of bit rate. Moreover, the proposed scheme performs consistently well for the high resolution sequences, of which the video coding efficiency is crucial in real applications.

Block partition structure has been recognized as a crucial module in video coding scheme. Recently, a quadtree plus binary tree (QTBT) block partition structure has been proposed in the Joint Video Exploration Team (JVET) development. Compared to the quadtree structure in HEVC, QTBT can achieve better coding performance with hugely increased encoding complexity. Here, we propose an effective QTBT partition decision algorithm to achieve a good trade-off between computational complexity and coding performance. In particular, at the Coding Tree Unit level, the partition parameters of QTBT are dynamically derived to adapt to the local characteristics without transmitting any overhead. Subsequently, at the Coding Unit level, a joint-classifier decision tree structure is designed to eliminate unnecessary iterations and meanwhile control the risk of false prediction. Experimental results show that the proposed algorithm can achieve 64% encoding time reduction on average with only 1.26% increase in terms of bit rate. This greatly benefits the practical implementations of QTBT in real application scenarios.

In this paper, we explore the structure of light field (LF) and efficiently improve the performance of the pseudo sequence based lenslet image compression, by optimized sub-view rearrangement, enhanced illumination compensation and adaptive reconstruction filtering. First, the decomposed sub-view images are rearranged into a pseudo sequence according to our optimized scan order based on sub-view correlation. Second, the generated pseudo sequence is compressed with JEM codec in which we enhance the illumination compensation by adaptively selecting reference pixels used in parameter derivation. Finally, to reduce the distortions in lenslet image decomposition and reconstruction, we propose an lenslet reconstruction method by applying adaptive filters to the reconstructed lenslet images to compensate the reconstruction errors. Each filter is derived by minimizing the distortions between the original and reconstructed pixels with same geolocation in the lenslet image. Extensive experimental results show that the proposed method achieves up to 53.7% bit rate reduction over HEVC intra coding and 34.8% over JEM intra coding in terms of BDBR.

The fast growth of surveillance video big data presents great challenges to the video coding technology. Most existing video coding techniques target for visual quality optimization, while the ultimate utility of surveillance videos mainly lies in intelligent analyses, e.g., pedestrian detection and vehicle tracking. In view of this, we aim at proposing an efficient, standard-compatible and simultaneously analysis-friendly coding framework for intelligent surveillance videos. In particular, the foreground objects are first extracted from the video sequence by accurate background modeling. Subsequently, the foregrounds can be constructed as a sequence and compressed in higher quality while very few background pictures are required to signal in lower quality. At the decoder side, the foreground objects can be directly used for efficient analysis tasks and the surveillance videos can be also reconstructed by synthesizing background and foreground frames. The effectiveness and potential of the proposed framework have been demonstrated in the pedestrian detection application, where the coding bits can be greatly saved with the detection accuracy being well maintained.

To improve the detection accuracy of SAD or SATD based threshold and save the time cost of RDO determination, we proposed an All Zero Block (AZB) detection method by adaptively searching the maximum transform coefficient amplitude in low frequency of TU after conventional SATD detection was failed. The experimental results show that our algorithm can achieve around 39% transform and quantization time-saving with only 0.1% on average RD performance reduction. The detection accuracy of larger TU size, i.e. 16×16 and 32 × 32, can reach up to about 95% on average.

In this paper, a fast intra Coding Unit (CU) size decision framework based on keypoint detection on Graphic Processing Unit (GPU) is proposed. In this framework, firstly the original frames are sent to GPU and then keypoint detection is conducted with numerous threads, which is able to avoid bringing in additional computational complexity even in realtime systems. Then, based on the keypoint distribution, whether to split the CU to the next coding depth is efficiently predicted. Experiments show that the proposed algorithm can achieve over 25% time saving under all intra (AI) configuration with ignorable performance loss.

Rate-Distortion Optimized Quantization (RDOQ) brings significant improvement of coding performance in High Efficiency Video Coding (HEVC). However, it results in high computational complexity to determine the best quantization levels for each transform coefficient when applying rate and distortion optimization operation. In this paper, we firstly proposed an efficient way to skip the all zero coding units. Then we further propose a fast RDOQ scheme by skipping the optimization step of All Quantized Zero Blocks except DC Coefficient (AQZB-DC). Moreover, a rate difference model is established to select optimal quantization level for AQZB-DC blocks. Experiments are carried out on reference software HM 16.0 and the results show that the proposed method achieves 42.00% and 39.63% quantization time saving under Random Access (RA) and Low Delay (LD) configuration on average, while the BD-Rate loss is only 0.03% and 0.06%, respectively.

Deep learning has demonstrated tremendous break through in the area of image/video processing. In this paper, a spatial-temporal residue network (STResNet) based in-loop filter is proposed to suppress visual artifacts such as blocking, ringing in video coding. Specifically, the spatial and temporal information is jointly exploited by taking both current block and co-located block in reference frame into consideration during the processing of in-loop filter. The architecture of STResNet only consists of four convolution layers which shows hospitality to memory and coding complexity. Moreover, to fully adapt the input content and improve the performance of the proposed in-loop filter, coding tree unit (CTU) level control flag is applied in the sense of rate-distortion optimization. Extensive experimental results show that our scheme provides up to 5.1% bit-rate reduction compared to the state-of-the-art video coding standard.

In this paper, we propose a novel reduced-reference (RR) image quality assessment (IQA) algorithm based on the internal generative mechanism, which suggests that the human visual system (HVS) can actively predict the primary visual information and avoid the uncertainty. Specifically, the explanation of the visual scene is formulated as the process of sparse representation. In particular, the entropy of primitive accounts for the primary visual information and the discrepancy between the image signal and its best sparse description is regarded as the uncertainty in perception. As such, the combined feature that can summarize the primary visual information and uncertainty in sparse domain is required to be transmitted in the RR-IQA framework. Comparative studies of the proposed reduced reference metric is conduced on both single and multiple distortion databases, and experimental results demonstrate that the proposed metric can achieve high correlation with the human perception by only sending ignorable additional information.

We propose an adaptive scanning scheme to improve color attribute compression performance of static point cloud. To better exploit the intrinsic correlations of point cloud color data, multiple scanning modes are dedicated to projecting the color attributes into a series of texture blocks before compression. The best mode is subsequently selected in the sense of rate-distortion optimization, such that the compression flexibility and performance can be greatly improved. To achieve rate-distortion optimized scan, the Lagrange multiplier that balances the rate and distortion is off-line derived based on the statistics of color attribute coding. The proposed algorithm is implemented in reference software for point cloud compression (PCC-MP3DG), which is introduced by 3D Graphics (3DG) group of Motion Picture Experts Group (MPEG). Experimental results show that proposed method performs significantly better than the PCC-MP3DG reference software, and on average 6.51% and 9.36% bit-rate savings can be achieved for high and low bit rate codings, respectively.

We propose an inter-view dependency-based rate control (RC) algorithm for 3D extension of high efficiency video coding (HEVC). First, considering the rate-distortion (R-D) dependency between the synthesized views and the input views (including texture videos and depth maps), a synthesized view distortion model is derived. Second, a novel distortion model for dependent views is proposed by investigating the inter-view dependency between the base and dependent views. Based on these two distortion models, a joint optimal bit allocation strategy (including texture/depth level, view level, and frame level) is developed to allocate target bits for both texture videos and depth maps of different views. Furthermore, an effective initial quantization parameter decision scheme considering the characteristics of the input video content is presented. Extensive experimental results exhibit that the proposed scheme achieves a higher rate control accuracy and provides a better R-D performance than the state-of-the-art RC algorithms.

Rate distortion optimized quantization (RDOQ) is an efficient encoder optimization method that plays an important role in improving the rate-distortion (RD) performance of the high-efficiency video coding (HEVC) codecs. However, the superior performance of RDOQ is achieved at the expense of high computational complexity cost in two stages RD minimization, including the determination of optimal quantized level among available candidates for each transformed coefficient and the determination of best quantized coefficients for transform units with the minimum total cost, to softly optimize the quantized coefficients. To reduce the computational cost of the RDOQ algorithm in HEVC, we propose a low-complexity RDOQ scheme by modeling the statistics of the transform coefficients with hybrid Laplace distribution. In this manner, specifically designed block level rate and distortion models are established based on the coefficient distribution. Therefore, the optimal quantization levels can be directly determined by optimizing the RD performance of the whole block, while the complicated RD cost calculations can be eventually avoided. Extensive experimental results show that with about 0.3%-0.4% RD performance degradation, the proposed low-complexity RDOQ algorithm is able to reduce around 70% quantization time with up to 17% total encoding time reduction compared with the original RDOQ implementation in HEVC on average.

Visual feature descriptors have been successfully deployed in a wide range of applications, e.g. visual retrieval and analysis. To transmit these descriptors over bandwidth-limited networks, a high efficiency feature coding technique is highly desired to maximize compression capability and achieve compact feature representations. In this paper, a hybrid visual feature descriptor compression framework is presented and implemented in the encoding and decoding loops of texture videos. In particular, the multiple-hypothesis prediction is employed to effectively remove redundancies originated not only from spatial and temporal similarities, but also from reconstructed video frames. As the ultimate purpose of the transmitted descriptors is retrieval, the rate-accuracy optimization (RAO) technique is proposed to obtain the best tradeoff between the rate and retrieval performance. Such paradigm enables the conventional video stream to achieve high efficient retrieval/analysis with very low bitrate consumption. Moreover, we also demonstrate that texture video compression can also benefit from the additional information provided by the transmitted descriptors, leading to significantly improvement of coding efficiency on top of the high efficiency video coding (HEVC) standard. Extensive simulations have shown that the proposed method can offer significant bitrate reduction in representing both the descriptors and texture video frames, and meanwhile providing desirable retrieval performance.

In this paper, a graphics processing unit (GPU) based sample adaptive offset (SAO) parameters decision scheme is proposed for High Efficiency Video Coding (HEVC). Then, in order to further improve the performance of SAO, a perceptual based optimization scheme is provided according to the adjustment of Lagrange multiplier aiming to improve the subjective performance of SAO. Experimental results demonstrate that the proposed GPU based SAO parameter decision scheme can achieve average 0.76% and 0.78% BD-rate gain in terms of PSNR (Peak Signal to Noise Ratio) and SSIM (Structure Similarity) respectively. Combined with the perceptual optimization scheme, the maximum BD-rate gain in terms of PSNR and SSIM can be up to 1.77% and 3.3% with the average as 1.23% and 1.37%. Moreover, much computation complexity of SAO can be distributed to GPU.

In the 3D extension of Audio Video Coding Standard (AVS), i.e. 3D-AVS, the Global Disparity Vector (GDV) derivation technique has been proposed to provide an estimation for Disparity Vector (DV) in inter-view prediction, where the GDV is generated by averaging all Disparity Vectors (DVs) in the latest previously coded frame. The prediction accuracy of GDV may be however limited by the lack of local adaptivity. In this paper, we introduce a novel Local Disparity Vector (LDV) derivation scheme. Specifically, the DV of the current block is calculated from the DVs within a neighboring region, whose size can be adaptively expanded to increase the robustness and accuracy. Experimental results show that the proposed LDV derivation method can provide around 2.12% and 1.37% bitrate reductions for compressed views and synthesized views compared with the GDV scheme, respectively.

In this paper, a new mode decision scheme is proposed for depth map coding in 3D-AVS. The novelty of the paper mainly contains the following two points. Firstly, an improved distortion estimation model of synthesized views is proposed. Secondly, for the mode decision of depth map coding, the distortion is represented to be the weighted sum of depth distortion and estimated distortion of the synthesized view. We proposed a new scheme to derive the weighting factors adaptively based on the disparity. Then the distortion is utilized to calculate the rate distortion cost for mode decision. Experimental results demonstrate that the proposed scheme achieves remarkable performance improvement in 3D-AVS. The average BD-rate gain is about 12%.

High dynamic range (HDR) imaging techniques have been widely advocated that could shape next generation of digital photography. However, the popularity of HDR contents is hindered by the lack of displaying devices for rendering HDR images which could be very expensive. To tackle this, extensive tone-mapping operators (TMOs) have been proposed in order for transforming HDR images to viewable low dynamic range (LDR), and also applied in the backward-compatibility based HDR compression. However, how to efficiently improve the compression performance based on the perceptual evaluation is seldom addressed. In this work, we first propose a quality evaluation index for measuring the quality of the LDR image with the access of pristine HDR image. Then a sparse coding framework for efficiently compressing the LDR image, which is generated from its HDR version using TMO, is presented. Finally the compression efficiency could be improved by jointly optimize the sparse coding process in terms of the proposed quality metric based on the divisive normalization mechanism. Extensive experiments have shown that the proposed scheme can improve the perceptual quality of the compressed LDR image.

In-loop filtering is an important technique in modern video coding standards. In this paper, we propose a transform-domain in-loop filter to further improve the compression performance of high efficiency video coding (HEVC) standard. The proposed method estimates block transform coefficients by adaptively fusing two prediction sources according to their uncertainties respectively. The first prediction is the block transform coefficients of compressed video frames, the uncertainty of which is related to quantization parameters. The second prediction is the weighted average of transform blocks in a neighborhood, and the weights are designed according to block similarity. Its uncertainty is estimated based on the coefficient variance. To optimize the filtering performance, the parameters utilized in the proposed in-loop filter are learned from compressed videos for each quantization parameter offline, and frame level flags are utilized to switch the proposed in-loop filter according to rate-distortion cost. Extensive experimental results show that the proposed in-loop filter can further improves the compression efficiency of HEVC.

High Efficiency Video Coding (HEVC) adopts hybrid transform coding scheme to improve the transform performance. However, it does not consider the influence of the prediction unit boundary. This paper proposes an adaptive boundary dependent transform optimization scheme for HEVC. Based on the transform unit boundary type identification by checking whether it lies at a prediction unit boundary or not, some additional transform kernels, including DCT-IV, are incorporated for transform. Moreover, the additional adopted transform kernels are generated by re-using the existing kernels in HEVC. Therefore, the butterfly structure for transform implementation can be preserved to facilitate parallel computing, and computational complexity increasing can be avoided. Experimental results show that the coding performance improvements can be up to 0.96% and 0.64% for low delay P and random access testing configurations respectively.

In this paper, we analyze the binary arithmetic coding of High Efficiency Video Coding (HEVC) and the second generation of audio and video coding standard (AVS2). Then an optimized probability estimation scheme is proposed for arithmetic coder. The proposed scheme is incorporated into the HEVC reference software (HM 16.0) and AVS2 reference software (RD 10.1). Experimental results demonstrate that the proposed scheme can efficiently improve the coding efficiency of entropy coding. The rate-distortion (R-D) performance gain can be up to 0.21% for AVS2 and 0.30% for HEVC respectively.

In High Efficiency Video Coding (HEVC), the intra coding performance is significantly improved due to the recursive splitting structure and up to 35 intra prediction modes. However, the computational complexity of intra coding increases largely as well. In this paper, a fast intra coding scheme is proposed based on CPU and GPU cooperation. Firstly, the intra prediction of variable blocks is performed in parallel on multi-cores GPU. Secondly, the intra prediction mode with minimum Sum of Absolute Difference (SAD) cost is selected and transmitted to the host CPU. Instead of exhaustively searching all the intra modes in Rough Mode Decision (RMD) process, the mode returned by the GPU is directly selected. Lastly, the texture gradient of each coding unit (CU) is assessed during parallel intra prediction, then used by the CPU for fast CU size decision. Experiment results show that the proposed parallel intra coding method achieves up to 62% complexity reduction with acceptable coding performance loss.

In the high efficiency video coding (HEVC) encoder, motion estimation (ME) takes up more than 50% encoding time. To reduce the complexity of the ME module in HEVC, this paper proposes a flexible coding tree unit (CTU)-level parallel ME method through CPU and GPU pipeline collaboration. Firstly a highly scalable CTU-level parallel motion search scheme on GPU is provided, in which, the parallel CTU group can be configured to be any size to adapt to the variable sequence resolution and hardware configurations. Then, the motion search range can be adaptively adjusted based on the motion intensity. Therefore, the unnecessary GPU time wasting can be further avoided for slow-moving scenes, while high performance kept for fast-moving scenes. Moreover, the ME information returned from GPU can be used by CPU for fast mode decision. Experimental results show that the proposed method achieves up to 73% complexity reduction than HM10.0 anchor using CPU only with acceptable coding performance loss, providing higher performance than the state-of-the-art scheme.

In this paper, we propose a new frame level rate control algorithm for the high efficiency video coding (HEVC) based 3D video (3DV) compression standard. In the proposed scheme, a new initial quantization parameter (QP) decision scheme is provided, and the bit allocation for each view is investigated to smooth the bitrate fluctuation and reach accurate rate control. Meanwhile, a simplified complexity estimation method for the extended view is introduced to reduce the computational complexity while improves the coding performance. The experimental results on 3DV test sequences demonstrate that the proposed algorithm can achieve better R-D performance and more accurate rate control compared to the benchmark algorithms in HTM10.0. The maximum performance improvement can be up to 12.4% and the average BD-rate gain for each view is 5.2%, 6.5% and 6.6% respectively.

High Efficiency Video Coding (HEVC) has drastically improved the coding efficiency, and can provide more than 23% bit rate reduction compared to its predecessor H264/AVC for all intra (AI) configuration. However, the improvement on coding efficiency is obtained at the expense of much more intensive computation complexity. In this paper, a fast intra optimization scheme is proposed. Firstly, a shrinkage scheme for the number of mode prediction is proposed based on gradient variance. Then a novel estimation model for rate distortion cost is provided based on Hadamard transform and quantization to better determine the mode prediction for RQT. Experimental results demonstrate that, compared with the original HEVC reference encoder implementation in AI configuration, the proposed algorithms can achieve more than 26% coding complexity reduction on average with ignorable coding performance degradation.

The emerging High Efficiency Video Coding (HEVC) standard has improved the coding efficiency drastically, and can provide equivalent subjective quality with more than 50% bit rate reduction compared to its predecessor H.264/AVC. As expected, the improvement on coding efficiency is obtained at the expense of more intensive computation complexity. In this paper, based on an overall analysis of computation complexity in HEVC encoder, a low complexity rate distortion optimization (RDO) coding scheme is proposed by reducing the number of available candidates for evaluation in terms of the intra prediction mode decision, reference frame selection and CU splitting. With the proposed scheme, the RDO technique of HEVC can be implemented in a low-complexity way for complexity-constrained encoders. Experimental results demonstrate that, compared with the original HEVC reference encoder implementation, the proposed algorithms can achieve about 30% reduced encoding time on average with ignorable coding performance degradation (0.8%).

In this paper, a novel quadratic ρ-domain frame layer rate control algorithm is proposed for Low Delay (LD) configuration in High Efficiency Video Coding (HEVC). Firstly, we propose a quadratic ρ-domain rate quantization (R-Q) model to establish the relationship between bit rate and quantization parameter. Subsequently, inspired by the specialized QP determination and reference picture set (RPS) mechanism, we design a RPS based hierarchical partition structure for rate control. Based on the hierarchical partition, an efficient hierarchical bit allocation scheme is provided. Finally, the quantization parameter (QP) is calculated via the proposed R-Q model after the bit allocation to meet the target bits. Experimental results demonstrate the proposed algorithm can significantly improve the R-D performance compared to the state-of-the-art rate control scheme for HEVC. Moreover, our proposed algorithm also provides smaller mismatch between the target bit rate and the actual bit rate.

In this paper, a fast multi reference frame motion estimation algorithm is proposed to reduce the computational complexity for high efficiency video coding (HEVC). Firstly, according to the reference frame distribution, we define the motion complexity for a frame. Based on the motion complexity, an initial reference frame set (RFS) is constructed to reduce the number of the reference frame for motion estimation. Then for each prediction unit (PU), the average distortion per pixel together with the motion vector difference (MVD) in the first reference frame is employed to shrink the RFS in order to early terminate the motion estimation process. In addition, to enhance the robustness of the proposed scheme, an expansion method for RFS is proposed to guarantee the video quality. Experimental results demonstrate that the proposed scheme can significantly save the encoding time. For Low Delay testing configuration, over 30% of time saving can be achieved on average with ignorable performance loss.

The specialized QP determination and reference picture set (RPS) mechanism in High Efficiency Video Coding (HEVC) significantly improves the coding efficiency. In this paper, we propose a frame layer rate control algorithm for HEVC based on this specialized mechanism. Firstly, we design a RPS based hierarchical partition structure. Then an efficient hierarchical bit allocation scheme is proposed. Moreover, a novel header bits ratio prediction method is also provided to further improve the accuracy of bit allocation. Finally, an improved mean of absolute difference (MAD) prediction scheme is provided for quadratic R-Q model to calculate an accurate quantization parameter. Experimental results demonstrate the proposed algorithm has much better R-D performance. Compared with the state-of-the-art rate control scheme for HEVC, the BD-Rate gain can be up to 30% and BD-PSNR gain can be 0.6dB on average in Low_Delay configuration. Moreover, our proposed algorithm also provides smaller mismatch between the target bit rate and the actual bit rate.

This paper gives a study on the rate distortion modeling for High Efficiency Video Coding (HEVC) and new rate distortion models are proposed by considering the coding framework change of HEVC. Although HEVC is also characterized by the traditional prediction/transform hybrid coding framework with more improved coding tools, e.g. large size coding unit (CU), prediction unit (PU) and transform unit (TU), two new coding modules are added as sample adaptive offset(SAO) and adaptive loop filter(ALF). SAO and ALF are added after the motion compensation reconstruction, which would reduce the distortion of the reconstructed image. With considerations on these changes, new rate distortion models are proposed. For rate modeling, this paper proposes a separate model for header bits and coefficient bits respectively. For distortion modeling, the quantization distortion and the distortion reduction induced by SAO/ALF are modeled jointly. The experimental results show that the coding rate and distortion can be estimated by the proposed model accurately. These models can be used for rate distortion cost estimation for mode decision or rate control in the future.