The optimization of block-level quantization parameters (QP) is critical to improving the performance of practical block-based video compression encoders, but the extremely large optimization space makes it challenging to solve. Existing solutions, e.g. HEVC encoder x265, usually add some optimization constraints of the block-independent assumption and linear distortion propagation model, which limits compression efficiency improvement to a certain extent. To address this problem, a deep learning-based encoder-only adaptive quantization method (DAQ) is proposed in this paper, where a deep network is designed to adaptively model the joint temporal propagation relationship of quantization among blocks. Specifically, DAQ consists of two phases: in the training phase, considering the heavy searching cost of the traditional codec, we introduce a well-designed end-to-end learned block-based video compression network as an effective training proxy tool for the deep encoder-side network. While in the deployment phase, the trained deep network is applied to jointly predict all block QPs in a frame for the traditional encoder. Besides, our network deploys only on the encoder side without changing the standard decoder and has very low inference complexity, making it able to apply in practice. At last, we deploy DAQ in HEVC and VVC encoder for performance comparison, and the experimental results demonstrate that DAQ significantly outperforms practically used x265 with on average 15.0%, 10.9% BD-rate reduction under the SSIM and PSNR, and also achieves 12.5%, 5.0% coding gain than VTM. Moreover, for deploying deep video codec in practice, this work provides a new insight for optimizing the encoder parameters with a large space.

In this work, we propose a deep learning-based adaptive quantization method to promote video coding performance. Due to inter-prediction and reference mechanism, the block-level quantization parameter (QP) not only influences current block distortion but also has complex temporal propagation effects on subsequent coding frames. Our idea is to utilize a deep network to model the complex temporal propagation relationship of quantization. As shown in Fig. 1, the deep network directly predicts all block-level QPs of the frame for the traditional encoder without changing the standard decoder. Since our network deploys only on the encoder side and has low inference complexity, it can be easily applied in practice. In addition, we use a learned coding network as a proxy of the traditional codec to train our network.

Neural video compression (NVC) has made significant progress in recent years, while neural B-frame video compression (NBVC) remains underexplored compared to P-frame compression. NBVC can adopt bi-directional reference frames for better compression performance. However, NBVC's hierarchical coding may complicate continuous temporal prediction, especially at some hierarchical levels with a large frame span, which could cause the contribution of the two reference frames to be unbalanced. To optimize reference information utilization, we propose a novel NBVC method, termed Bi-directional Reference Harmonization Video Compression (BRHVC), with the proposed Bi-directional Motion Converge (BMC) and Bi-directional Contextual Fusion (BCF). BMC converges multiple optical flows in motion compression, leading to more accurate motion compensation on a larger scale. Then BCF explicitly models the weights of reference contexts under the guidance of motion compensation accuracy. With more efficient motions and contexts, BRHVC can effectively harmonize bi-directional references. Experimental results indicate that our BRHVC outperforms previous state-of-the-art NVC methods, even surpassing the traditional coding, VTM-RA (under random access configuration), on the HEVC datasets. The source code will be released. The source code is released at https://github.com/kwai/NVC.

Existing end-to-end image coding for machines (ICM) methods generally use joint training strategies to promote the compression efficiency for machine vision without considering the influence of different regions in the image. To encourage the image compression network to focus on the regions that are critical to the subsequent visual task, this paper proposes a saliency map-guided image compression network (SMIC-Net) for ICM. Specifically, a saliency map-guided transform module (SMTM) is proposed to improve the representation ability of image features for object detection task by exploring the semantic and structural information of the detected object. Besides, a saliency map-guided mean square error (SM-MSE) loss is designed to place more emphasis on the detected object regions. Experimental results demonstrate that the proposed SMIC-Net effectively promotes the compression efficiency for machine vision.

In contrast to image compression, the key of video compression is to efficiently exploit the temporal context for reducing the inter-frame redundancy. Existing learned video compression methods generally rely on utilizing short-term temporal correlations or image-oriented codecs, which prevents further improvement of the coding performance. This paper proposed a novel temporal context-based video compression network (TCVC-Net) for improving the performance of learned video compression. Specifically, a global temporal reference aggregation (GTRA) module is proposed to obtain an accurate temporal reference for motion-compensated prediction by aggregating long-term temporal context. Furthermore, in order to efficiently compress the motion vector and residue, a temporal conditional codec (TCC) is proposed to preserve structural and detailed information by exploiting the multi-frequency components in temporal context. Experimental results show that the proposed TCVC-Net outperforms public state-of-the-art methods in terms of both PSNR and MS-SSIM metrics.

Existing learning-based stereo compression methods usually adopt a unidirectional approach to encoding one image independently and the other image conditioned upon the first. This paper proposes a novel bidirectional coding-based end-to-end stereo image compression network (BCSIC-Net). BCSIC-Net consists of a novel bidirectional contextual transform module which performs nonlinear transform conditioned upon the inter-view context in a latent space to reduce inter-view redundancy, and a bidirectional conditional entropy model that employs interview correspondence as a conditional prior to improve coding efficiency. Experimental results on the InStereo2K and KITTI datasets demonstrate that the proposed BCSIC-Net can effectively reduce the inter-view redundancy and out-performs state-of-the-art methods.

End-to-end compression methods designed for the texture image have achieved excellent coding performances. Due to the characteristic differences between the depth map and the texture image, the texture-oriented methods have limitations in depth map compression. To address this problem, this paper proposes a texture-guided end-to-end depth map compression network (TDMC-Net). Specifically, the proposed TDMC-Net is mainly composed of the texture-guided transform module (TTM) which performs the nonlinear transform with providing the textual context to reduce the redundancy in depth feature, and a texture-guided conditional entropy model (TCEM) which is designed to improve the entropy model by introducing the texture conditional prior. Experimental results show that the proposed TDMC-Net boosts the depth coding efficiency by utilizing the texture information and achieves superior performance.

In video coding, it is a challenge to deal with scenes with complex motions, such as rotation and zooming. Although affine motion compensation (AMC) is employed in Versatile Video Coding (VVC), it is still difficult to handle non-translational motions due to the adopted hand-craft block-based motion compensation. In this paper, we propose a deep affine motion compensation network (DAMC-Net) for inter prediction in video coding to effectively improve the prediction accuracy. To the best of our knowledge, our work is the first attempt to deal with the deformable motion compensation based on CNN in VVC. Specifically, a deformable motion-compensated prediction (DMCP) module is proposed to compensate the current encoding block through a learnable way to estimate accurate motion fields. Meanwhile, the spatial neighboring information and the temporal reference block as well as the initial motion field are fully exploited. By effectively fusing the multi-channel feature maps from DMCP, an attention-based fusion and reconstruction (AFR) module is designed to reconstruct the output block. The proposed DAMC-Net is integrated into VVC and the experimental results demonstrate that the proposed method considerably enhances the coding performance.