As the successor of H.265/HEVC, the new versatile video coding standard (H.266/VVC) can provide up to 50% bitrate saving with the same subjective quality, at the cost of increased decoding complexity. To accelerate the application of the new coding standard, a real-time H.266/VVC software decoder that can support various platforms is implemented, where SIMD technologies, parallelism optimization, and the acceleration strategies based on the characteristics of each coding tool are applied. As the mobile devices have become an essential carrier for video services nowadays, the mentioned optimization efforts are not only implemented for the x86 platform, but more importantly utilized to highly optimize the decoding performance on the ARM platform in this work. The experimental results show that when running on the Apple A14 SoC (iPhone 12pro), the average single-thread decoding speed of the present implementation can achieve 53fps (RA and LB) for full HD (1080p) bitstreams generated by VTM-11.0 reference software using 8bit Common Test Conditions (CTC). When multi-threading is enabled, an average of 32 fps (RA) can be achieved when decoding the 4K bitstreams.

In this article, a new CTU-level bit allocation scheme aimed at subjectively optimized video coding for video conferencing applications is presented, in which the non-cooperative Stackelberg game is used for formulating and solving the bit allocation problem during the encoding process. Videos are divided into the Region of interests (ROI) which attracts people more and the non-ROI. The two regions are defined as the players in the game, where the ROI is the leader who takes the priority in strategy making and the non-ROI follows the leader's strategy. Based on the formulated game, the bit allocation problem can be expressed as a utility optimization problem. By solving the corresponding utility optimization problem, the bit allocation strategy between the ROI and the non-ROI will be established. Then the bits will be allocated to each CTU by a Newton-method-based algorithm for encoding, in which a trade-off between the ROI's quality and the overall quality can be achieved. Both the objective and subjective experimental results show that our proposed bit allocation method can improve the quality of ROI significantly with an acceptable overall quality degradation, leading to a better visual experience.

Rate control plays an important role in practical video coding. In this paper, we present a rate control method for Versatile Video Coding (VVC). We conduct some statistical analysis to investigate the influence of the skip block upon the rate-distortion parameter estimation and quality dependency among frames. Based on the analysis, we propose the rate-distortion parameter updating strategy and establish a quality dependency factor based frame-level bit allocation scheme. Experimental results show that the proposed method can achieve up to 7.31%/5.44% BD-rate improvements in Low-Delay/Random-Access configuration upon previous rate control method in VVC respectively. The presented rate control algorithm has already been adopted into VVC VTM platform by Joint Video Experts Team (JVET).

Screen content video applications require efficient coding of computer-generated materials. The new screen content coding tools such as intra block copy (IBC) and palette mode (PLT) have addressed this requirement. However, the added computational complexity on top of the existing sophisticated video encoders is also challenging. In this paper, we focus on the fast and efficient encoder implementation of these screen content coding tools. Improvements on hash-based IBC search, PLT optimization, mode decision between PLT and intra mode, and other general encoder accelerations towards screen content applications are studied and discussed. Experimental results show that with these methods added, the encoder can achieve some faster runtime performance than before while the compression efficiency is almost doubled with screen content coding tools.

With the development of applications such as wireless displays and remote desktop, screen content coding (SCC) has been a topic of increasing interest. The state-of-the-art partition modes in AVS3 restrict a parent coding unit (CU) to partition into no more than four sub-CUs in each partition process. To obtain more than four smaller sub-CUs, a parent CU needs to partition recursively. In this paper, to simplify the recursive partition processes and improve SCC efficiency, a novel partition mode which allows a parent CU to be partitioned into several pre-defined smaller sub-CUs with the same shape and size directly is proposed. We make some pre-experiments to verify the validity of the proposed partition mode, and limit the application scope to four rectangular CU sizes based on the pre-experimental results. As demonstrated in the experimental results, the proposed partition mode can obviously improve SCC efficiency.

Efficient compression of omnidirectional video is important for emerging virtual reality applications. To compress this kind of video, each frame is projected to a 2D plane [e.g., equirectangular projection (ERP) map] first, adapting to the input format of existing video coding systems. At the display side, an inverse projection is applied to the reconstructed video to restore signals in spherical domain. Such a projection, however, makes presentation and encoding in different domains. Thus, an encoder agnostic to the projection performs inefficiently. In this paper, we analyze how a projection influences the distortion measurements in different domains. Based on the analysis, we propose a scheme to optimize the encoding process based on signals' distortion in spherical domain. With the proposed optimization, an average 4.31% (up to 9.67%) luma BD-rate reduction is achieved for ERP in random access configuration. The corresponding bit saving is averagely 10.84% (up to 34.44%) when considering the viewing field being π/2 . The proposed method also benefits other projections and viewport settings, with a marginal complexity increase.

Objective quality assessment plays a crucial role in the evaluation and optimization processes of Virtual Reality (VR) technologies, for which state-of-the-art objective quality evaluation metrics for omnidirectional video, i.e., 360 degree video, are typically derived from traditional MSE (or PSNR). Here we propose an objective omnidirectional video quality assessment method based on structural similarity (SSIM) in the spherical domain. Adopting the relationship of the structural similarity between the 2-D plane and sphere, the interference brought by the projection between the two domains can be well handled in the assessment process. The performance of the proposed spherical structural similarity (S-SSIM) index is evaluated with a subjective omnidirectional video quality assessment database. As demonstrated in the experimental results, the proposed S-SSIM outperforms state-of-the-art objective quality assessment metrics in omnidirectional video quality assessment.

Region of Interests (ROI) coding has shown advantages in subjectively optimized video coding. In this paper, we propose a new CTU-Ievel rate allocation scheme based on the Stackelberg Game model to enhance the visual quality of ROI which formulates the rate allocation process as a noncooperative Stackelberg Game between the ROI and the non-ROI. In this game, ROI is the leader who takes the priority. Based on the formulated game, the rate allocation problem can be expressed as a utility optimization problem. By solving the corresponding utility optimization problem, a novel CTU-level rate allocation strategy is established, in which a trade-off between the ROI's quality and the overall quality can be achieved. The experimental results show that our proposed scheme can improve the quality of ROI significantly with a negligible overall quality degradation.

Achieving constant video quality is an essential optimization task for video coding applications. In this paper, we propose a two-pass rate control algorithm for High Efficiency Video Coding (HEVC) to reduce the fluctuation of video quality. First, we present a GOP-level rate allocation scheme which aims at minimizing distortion variance among GOPs. To derive an appropriate quantization parameter (QP) for each frame, we establish the distortion-quantizer (D-Q) and the rate-quantizer (R-Q) model based on ν-support vector regression. Further, an R-D tradeoff is designed to adjust the frame QP considering the constant quality control and rate contraints. Extensive experimental results show that the proposed algorithm achieves more constant video quality.

In-loop filtering is a key technique in the state of art video coding standard that plays a significant role in suppressing compression artifacts. Motivated by the latest advances of deep learning, in this paper, we design a dense residual convolutional neural network (DRN) based in-loop filter for High Efficiency Video Coding (HEVC). Taking advantage of both dense shortcuts and residual learning, DRN efficiently exploits the multi-level features to restore the high quality image from the degraded one. Bottleneck layers are employed in DRN in order to adaptively fuse the hierarchical features and saving the computational resources at the same time. Experimental results show that the proposed DRN based in-loop filter can further boost the coding performance, which provides 6.9% BD-rate reduction on average compared to the HEVC baseline. In addition, the proposed DRN outperforms previous CNN based in-loop filters.

Emerging virtual reality (VR) applications bring much challenge to video coding for 360-degree videos. To compress this kind of video, each picture should be projected to a 2D plane (e.g. equirctangular projection map) first, adapting to the input of existing video coding systems. At the display side, an inverse projection is performed before viewport rendering. However, such a project introduces much different levels of distortions depending on the location, which makes the rate-distortion optimization process in video coding much inefficient. In this paper, we consider the distortion in spherical domain and analyse its influence to the rate-distortion optimization process. Then we derive the optimal rate-distortion relationship in spherical domain and present its optimal solution based on HEVC/H.265. Experimental results show that the proposed method can bring up to 11.5% bit-saving compared with the current HEVC/H.265 anchor for 360-degree video coding.

360-degree video has been gaining its popularity as virtual reality (VR) technology develops in recent years. To compress these videos using standard video encoders, most of them are converted to a 2D image planar format with equirectangular projection (ERP). Compared with other encoders, High Efficiency Video Coding (HEVC) achieves significant improvements in coding efficiency. However, exhaustive computational complexity of HEVC makes it too time-consuming to compress 360-degree videos of high resolution and high frame rate for VR application. In this paper, a fast intra prediction algorithm is proposed according to the characteristics of ERP format video. Compared with original reference software HM 16.15, experimental results show that the proposed algorithm can achieve about 24.5% encoder time saving on average in All-Intra configuration with negligible quality loss.

In this paper, we propose a convolutional neural network (CNN)-based rate-distortion (R-D) modeling method for H.265/HEVC. A fully convolutional neural network (CNN) is designed to learn end-to-end, pixels-to-pixels mappings from the original images to the structural similarity (SSIM) maps indicating distortion. The rate information is predicted through a CNN with fully connected layers as well. When compared to traditional CNN methods, the proposed mappings to the distortion or rate information. The experiments demonstrate the feasibility of our CNN-based framework for rate-distortion modeling.