It has been recognized that videos have to be encoded in a rate-distortion optimized manner for high coding performance. Therefore, operational coding methods have been developed for conventional distortion metrics such as Sum of Squared Error (SSE). Nowadays, with the rapid development of machine learning, the state-of-the-art learning based metric Video Multimethod Assessment Fusion (VMAF) has been proven to outperform conventional ones in terms of the correlation with human perception, and thus deserves integration into the coding framework. However, unlike conventional metrics, VMAF has no specific computational formulas and may be frequently updated by new training data, which invalidates the existing coding methods and makes it highly desired to develop a rate-distortion optimized method for VMAF. Moreover, VMAF is designed to operate at the frame level, which leads to further difficulties in its application to today's block based coding. In this paper, we propose a VMAF oriented perceptual coding method based on piecewise metric coupling. Firstly, we explore the correlation between VMAF and SSE in the neighborhood of a benchmark distortion. Then a rate-distortion optimization model is formulated based on the correlation, and an optimized block based coding method is presented for VMAF. Experimental results show that 3.61% and 2.67% bit saving on average can be achieved for VMAF under the low_delay_p and the random_access_main configurations of HEVC coding respectively.

In the light of low costs and automatic assessment, objective visual quality metrics enjoy many important applications such as perceptual coding. Recently multiple metrics obtain further improvement by means of machine learning. However, due to the absence of specific formulas, it's often hard to incorporate learning based metrics into video coding. In this paper, taking the state-of-the-art learning based metric VMAF for example, we propose a method of perceptual coding in an inferential manner for learning based metrics. The rate distortion optimization is adapted during coding as well. Experimental results show that compared with conventional methods, the proposed method can achieve obvious bitrate saving under HEVC coding.

Being developed by Joint Video Exploration Team (JVET), Future Video Coding (FVC) aims at higher resolutions and higher compression performance than the state-of-the-art HEVC standard, undoubtedly at the cost of further computing increases. As an efficient computing platform, Graphics Processing Unit (GPU) is often used to accelerate encoding. But with the adoption of instruction set acceleration in the reference software of FVC, previous methods often become less efficient or even lead to a lower speed. In this paper, based on the comparative analysis of the time consumption between HEVC and FVC, we propose a GPU based acceleration method for the most computation-intensive step - frame interpolation of FVC, where frame caching strategy and a multi-stream mechanism is designed to make the best of GPU resources. Experimental results show that compared with the instruction set accelerated reference software of FVC, our method could achieve average 67.12% speed-up gains on the interpolation module and average 6.35% speed-up gains on overall encoding with exactly the same performance as before.

As an open-source and computationally efficient High Efficiency Video Coding (HEVC) encoder, x265 has been gaining increasing popularity in video applications. x265 provides numerous encoding parameters in view of flexibility. However, proper and efficient setting of parameters often becomes a great challenge in practice. In this paper, we deeply investigate the influence of x265 parameters based on the Slow preset and pick out important parameters in terms of efficiency and complexity. Then a Markov Chain Monte Carlo (MCMC) based algorithm is proposed for efficient parameter adaptation at the target encoding time. This paper shows that carefully selected low-complexity encoding configurations can achieve the coding efficiency comparable to that of high-complexity ones. Specifically, average 26.72% encoding time reduction can be achieved while maintaining similar Rate Distortion (RD) performance to x265 presets using the proposed algorithm.

This paper presents a masking effects based rate control scheme for high efficiency video coding (HEVC). Rate control is regarded as a very effective tool to improve the performance of video coding under the limited bandwidth. However, the state-of-the-art rate control algorithm based on R-X model ignores the characteristics of human visual system (HVS), which leads to poor performance in subjective quality. Moreover, some structural similarity (SSIM) or saliency based perceptual rate control algorithms only consider spatial characteristics. Since spatial and temporal visual masking effects can better reflect the characteristics of HVS, in this paper masking effects based perceptual factor for coding tree unit (CTU) is proposed, which takes both texture complexity and motion information into account. Then the proposed perceptual factor is utilized to guide bit allocation in CTU-level rate control. Experimental results show that the proposed scheme can effectively improve the coding performance compared with the R-λ algorithm.

Video encoding is based on the DPCM framework where temporal prediction coding introduces Rate-Distortion (RD) dependence. The RD operating point of the current unit depends on the particular choices of RD points of its reference units. Unfortunately, common Lagrangian optimization method based Rate-Distortion Optimization (RDO) for video coding is based on an independence assumption which omits the RD dependences, and thus compromises the RD performance. In this paper, we revisit the Lagrangian optimization method based RDO for dependent video coding. A theoretical RD dependence decoupling method based on independent distortion decomposition is firstly presented. After the discussion of reasonability of the theoretical decoupling method, the practical One Step Ahead Decoupling Strategy (OSADS) is proposed. After implemented on the HEVC encoder, the strategy achieves average 2.1% BD-rate saving compared with the HM encoder under the same low-delay P configuration.

This paper describes a luminance based rate control (RC) model for high dynamic range (HDR) video. A novel mathematical relationship between luminance and bit allocation of a coding tree unit (CTU) is presented. By adjusting the existing RC algorithm through the proposed model, a better balance between dark and bright areas can be achieved and -4.4% gains can be obtained in terms of average BD-Rate (tPSNR-XZY). Moreover, subjective assessment also shows that, compared with the existing RC model, the proposed method can convey a wider range of perceptible shadow and highlight more details.

Motion compensated prediction (MCP) is an effective technology for video coding to improve compression efficiency. Fractional sample precision prediction is utilized in HEVC to further remove temporal redundancy, and finite impulse response (FIR) filters designed using decomposition of the discrete cosine transform are applied to generate samples that do not fall on the integer positions. However, the coefficients of these DCT-based interpolation filters are fixed, which may not be able to adapt to varied video content. Inspired by the remarkable success of convolutional neural network (CNN) in the single image super-resolution task, we propose to learn a convolutional neural network for fractional interpolation in HEVC inter prediction. Compared with super-resolution, there is one big difference in fractional interpolation - fractional interpolation needs to maintain samples at integer positions while super-resolution generates a whole high-resolution image. Another difference is no real ground truth is available in fractional interpolation process. To overcome these two challenges, we introduce a constraint strategy to the training phase of the original super-resolution network as well as a specially designed preprocessing step which reuses the DCTIF interpolation process. Unlike other previous work, our proposed approach simultaneously generating the fractional positions from one network and experimental results show our proposed approach achieves 0.45% BD-Rate reduction under the low-delay-P configuration on average.

Entropy coding, which acts as one of the most important compression tools in video coding standard, had been improved step by step for HEVC. There are also several advanced methods which provide better performance than current solutions of HEVC proposed during the standardization of HEVC. However, these methods are all tested in different conditions. Comprehensive evaluation of these advanced methods under a common scenario is desired to indicate where the potential improvement of entropy coding may come from for next generation video codec. In this paper, we first introduce several advanced entropy coding methods for DCT transform coefficients, which aim to improve CABAC performance from two aspects - context modeling and probability updating. Then some modifications based on these original ones are presented. Comprehensive comparison of these methods is conducted under common test conditions. Besides, some combined methods of these two aspects are also tested. Experimental results show that all individual approaches can achieve coding gain and two new combined methods can reduce the BD-Rate up to 1.7%, 1.2% and 1.0% on common test sequences and 1.4%, 1.0% and 1.1% on 4K sequences under all intra, random access and low delay configurations, respectively.

The field of video coding has been exploring the compact representation of video data, where perceptual redundancies in addition to signal redundancies are removed for higher compression. Many research efforts have been dedicated to modeling the human visual system's characteristics. The resulting models have been integrated into video coding frameworks in different ways. Among them, coding enhancements with the just noticeable distortion (JND) model have drawn much attention in recent years due to its significant gains. A common application of the JND model is the adjustment of quantization by a multiplying factor corresponding to the JND threshold. In this paper, we propose an alternative perceptual video coding method to improve upon the current H.264/advanced video control (AVC) framework based on an independent JND-directed suppression tool. This new tool is capable of finely tuning the quantization using a JND-normalized error model. To make full use of this new rate distortion adjustment component the Lagrange multiplier for rate distortion optimization is derived in terms of the equivalent distortion. Because the H.264/AVC integer discrete cosine transform (DCT) is different from classic DCT, on which state-of-the-art JND models are computed, we analytically derive a JND mapping formula between the integer DCT domain and the classic DCT domain which permits us to reuse the JND models in a more natural way. In addition, the JND threshold can be refined by adopting a saliency algorithm in the coding framework and we reduce the complexity of the JND computation by reusing the motion estimation of the encoder. Another benefit of the proposed scheme is that it remains fully compliant with the existing H.264/AVC standard. Subjective experimental results show that significant bit saving can be obtained using our method while maintaining a similar visual quality to the traditional H.264/AVC coded video.

Raptor codes are state-of-the-art forward error correction (FEC) solutions for multimedia transmission, which have been applied to unequal error protection (UEP) of multi-layered media such as scalable video coding. In this paper, we address the problem of UEP for single-layered video over packet erasure channels. By exploiting the different priorities of video packets inside a group of pictures (GOP) and making full use of the good characteristics of standardized Raptor codes at large block length, we propose an optimized UEP framework for single-layered video and develop an efficient algorithm to solve it. Simulation results show that significant gains can be obtained by our method in case of packet losses.

Due to the best effort feature of many existing transmission channels, video streams often suffer from inevitable transmission errors. In this paper, we propose a scheme of robust video transmission based on the state-of-the-art Raptor codes, whose applications are in full swing now. And considering Region of Interest (ROI) often draws much attention in images, the scheme adopts a nested protection framework to show partialities to ROI areas for better protection. Different from many existing Raptor codes based UEP methods, our scheme is developed based on the easy-to-use standardized Raptor codes. Experimental results show that significant robustness can be obtained for the video streams, especially for the ROI areas.