The ever-growing multimedia traffic has underscored the importance of effective multimedia codecs. Among them, the up-to-date lossy video coding standard, Versatile Video Coding (VVC), has been attracting attentions of video coding community. However, the gain of VVC is achieved at the cost of significant encoding complexity, which brings the need to realize fast encoder with comparable Rate Distortion (RD) performance. In this paper, we propose to optimize the VVC complexity at intra-frame prediction, with a two-stage framework of deep feature fusion and probability estimation. At the first stage, we employ the deep convolutional network to extract the spatial-temporal neighboring coding features. Then we fuse all reference features obtained by different convolutional kernels to determine an optimal intra coding depth. At the second stage, we employ a probability-based model and the spatial-temporal coherence to select the candidate partition modes within the optimal coding depth. Finally, these selected depths and partitions are executed whilst unnecessary computations are excluded. Experimental results on standard database demonstrate the superiority of proposed method, especially for High Definition (HD) and Ultra-HD (UHD) video sequences.

End-to-end learning-based video coding has attracted substantial attentions by compressing video signals as stacked visual features. This paper proposes an end-to-end deep video codec with jointly optimized compression and enhancement modules (JCEVC). First, we propose a dual-path generative adversarial network (DPEG) to reconstruct video details after compression. An α-path and a β-path concurrently reconstruct the structure information and local textures. Second, we reuse the DPEG network in both motion compensation and quality enhancement modules, which are further combined with other necessary modules to formulate our JCEVC framework. Third, we employ a joint training of deep video compression and enhancement that further improves the rate-distortion (RD) performance of compression. Compared with x265 LDP very fast mode, our JCEVC reduces the average bit-per-pixel (bpp) by 39.39%/54.92% at the same PSNR/MS-SSIM, which outperforms the state-of-the-art deep video codecs by a considerable margin. Sourcecode is available at: https://github.com/fwz1021/JCEVC.

The latest standard, Versatile Video Coding (VVC), doubles the coding efficiency over the previous generation standard. However, better performance is at the cost of a sharp increase in coding complexity. In order to reduce the complexity of VVC intra coding, this paper proposes a multi-stage block partition decision framework based on deep learning. First, we propose a three-stage redundant modes removal framework that decreases the number of modes checked in the brute-force process. Then, we build a lightweight CNN to complete the classification task of each stage. To reduce the burden of CNN and adapt to different Coding Unit (CU) sizes, we pre-process the luminance component of CU and use the results as input of the network. Finally, the multi-threshold adjusting scheme is proposed for trading off complexity reduction with the bit-rate increase. The experimental results shows our method can reduce the encoding time ranging from 16.93% to 69.40% with the bit-rate increase ranging from 0.31% to 3.59%. Such results demonstrate that our method has superior performance with a wide range of adjustments compared with other state-of-the-art methods.

This paper proposes a fast VVC coding unit partition algorithm based on ensemble convolutional neural network (CNN) by investigating and bagging spatial-temporal adjacent coding features. First, we propose an ensemble CNN framework to aggregate the reference features to predict the depths of uncoded CUs. The proposed model consists of three lightweight CNNs, which can compromise prediction accuracy with overhead. Then a majority voting mechanism is used to unify the predicted depth. By extracting the majority prediction of base learners, the outputs of three CNNs are integrated to obtain the final prediction. To avoid Rate Distortion (RD) loss caused by a small probability of prediction failure, we introduce the optimal depth strategy. During the encoding process, the optimal depth is used for the decision-making of coding unit partition, thus avoiding redundant rate distortion optimization process. Compared with the original encoder, the proposed algorithm saves 21.56% encoding time on average, with a BDBR loss of 0.39%. The performance is even superior in High-Definition (HD) and Ultra HD (UHD) sequences, up to 59.52%. This approach has a great efficiency of time reduction compared with state-of-the-arts with negligible RD performance loss.