Object quality assessment for compressed video is critical to various video compression systems that are essential in the video delivery and storage. Although mean squared error (MSE) is computationally simple, it may not be accurate to reflect the perceptual quality of compressed videos, which are also affected dramatically by the characteristics of the human visual system (HVS), such as contrast sensitivity, visual attention, and masking effect. In this paper, a video quality metric is proposed based on perceptually weighted MSE. A low-pass filter is designed to model the contrast sensitivity of the HVS with the consideration of visual attention. The imperceptible distortion is adaptively removed in the salient and nonsalient regions. To quantitatively measure the masking effect, the randomness of video content is proposed in both the spatial and temporal domains. Since the masking effect highly depends on the regularity of structure and motion in the spatial and temporal directions, the video signal is modeled as a linear dynamic system, and the prediction error of future frames from previous frames is used as randomness to measure the significance of masking. The relation is investigated between MSE and perceptual quality scores across various contents, and a masking modulation model is proposed to compensate the impact of the masking effect on the MSE. The performance of the proposed quality metric is validated on three video databases with various compression distortions. The experimental results demonstrate that the proposed algorithm outperforms other benchmark quality metrics.

Image quality assessment (IQA) is a fundamental yet constantly developing task for computer vision and image processing. Most IQA evaluation mechanisms are based on the pertinence of subjective and objective estimation. Each image distortion type has its own property correlated with human perception. However, this intrinsic property may not be fully exploited by existing IQA methods. In this paper, we make two main contributions to the IQA field. First, a novel IQA method is developed based on a local linear model that examines the distortion between the reference and the distorted images for better alignment with human visual experience. Second, a distortion-specific compensation strategy is proposed to offset the negative effect on IQA modeling caused by different image distortion types. These score offsets are learned from several known distortion types. Furthermore, for an image with an unknown distortion type, a convolutional neural network-based method is proposed to compute the score offset automatically. Finally, an integrated IQA metric is proposed by combining the aforementioned two ideas. Extensive experiments are performed to verify the proposed IQA metric, which demonstrate that the local linear model is useful in human perception modeling, especially for individual image distortion, and the overall IQA method outperforms several state-of-the-art IQA approaches.

Based on the notion of just noticeable differences (JND), a stair quality function (SQF) was recently proposed to model human perception on JPEG images. Furthermore, a k-means clustering algorithm was adopted to aggregate JND data collected from multiple subjects to generate a single SQF. In this work, we propose a new method to derive the SQF using the Gaussian Mixture Model (GMM). The newly derived SQF can be interpreted as a way to characterize the mean viewer experience. Furthermore, it has a lower information criterion (BIC) value than the previous one, indicating that it offers a better model. A specific example is given to demonstrate the advantages of the new approach.

A compressed video quality assessment dataset based on the just noticeable difference (JND) model, called MCL-JCV, is recently constructed and released. In this work, we explain its design objectives, selected video content and subject test procedures. Then, we conduct statistical analysis on collected JND data. We compute the difference between every two adjacent JND points and propose an outlier detection algorithm to remove unreliable data. We also show that each JND difference group can be well approximated by a normal distribution so that we can adopt the Gaussian mixture model (GMM) to characterize the distribution of multiple JND points. Finally, it is demonstrated by experimental results that the proposed JND analysis performed in the difference domain, called the D-method, achieves a lower BIC (Bayesian information criteria) value than the previously proposed G-method.

Objective quality assessment for compressed images is critical to various image compression systems that are essential in image delivery and storage. Although the mean squared error (MSE) is computationally simple, it may not be accurate to reflect the perceptual quality of compressed images, which is also affected dramatically by the characteristics of human visual system (HVS), such as masking effect. In this paper, an image quality metric (IQM) is proposed based on perceptually weighted distortion in terms of the MSE. To capture the characteristics of HVS, a randomness map is proposed to measure the masking effect and a preprocessing scheme is proposed to simulate the processing that occurs in the initial part of HVS. Since the masking effect highly depends on the structural randomness, the prediction error from neighborhood with a statistical model is used to measure the significance of masking. Meanwhile, the imperceptible signal with high frequency could be removed by preprocessing with low-pass filters. The relation is investigated between the distortions before and after masking effect, and a masking modulation model is proposed to simulate the masking effect after preprocessing. The performance of the proposed IQM is validated on six image databases with various compression distortions. The experimental results show that the proposed algorithm outperforms other benchmark IQMs.

The quality assessment for synthesized video with texture/depth compression distortion is important for the design, optimization, and evaluation of the multi-view video plus depth (MVD)-based 3D video system. In this paper, the subjective and objective studies for synthesized view assessment are both conducted. First, a synthesized video quality database with texture/depth compression distortion is presented with subjective scores given by 56 subjects. The 140 videos are synthesized from ten MVD sequences with different texture/depth quantization combinations. Second, a full reference objective video quality assessment (VQA) method is proposed concerning about the annoying temporal flicker distortion and the change of spatio-temporal activity in the synthesized video. The proposed VQA algorithm has a good performance evaluated on the entire synthesized video quality database, and is particularly prominent on the subsets which have significant temporal flicker distortion induced by depth compression and view synthesis process.

Depth video is used as the geometrical information of 3D world scenes in 3D view synthesis. Due to the mismatch between the number of depth levels and disparity levels in the view synthesis, the relationship between depth distortion and rendering position error can be modeled as a many-to-one mapping function, in which different depth distortion values might be projected to the same geometrical distortion in the synthesized virtual view image. Based on this property, we present an allowable depth distortion (ADD) model for 3D depth map coding. Then, an ADD-based rate-distortion model is proposed for mode decision and motion/disparity estimation modules aiming at minimizing view synthesis distortion at a given bit rate constraint. In addition, an ADD-based depth bit reduction algorithm is proposed to further reduce the depth bit rate while maintaining the qualities of the synthesized images. Experimental results in intra depth coding show that the proposed overall algorithm achieves Bjontegaard delta peak signal-to-noise ratio gains of 1.58 and 2.68 dB on average for half and integer-pixel rendering precisions, respectively. In addition, the proposed algorithms are also highly efficient for inter depth coding when evaluated with different metrics.

In this work, a new coding tool called Edge Mode is proposed for HEVC intra coding, aimed at improving coding efficiency for screen content video. A set of edge modes that correspond to edge positions are identified based upon intra prediction directions. Then, a simplified scheme is developed to select the best edge mode. To avoid applying a transform over strong edges, directional 2D separable transforms are applied to blocks partitioned using these edge modes. Experimental results show that HEVC with edge modes (HEVC/EM) can achieve up to an 18% reduction in bit-rate as compared to unmodified HEVC, with an average reduction of 10.4% for screen content video sequences.

A new video quality index, called the distortion/content-dependent video quality index (DCVQI), is proposed in this paper. The distortion and content effects on video quality assessment are taken into account explicitly in the DCVQI. First, we classify distortions into two types (local and global distortions) with a distortion classification scheme. Then, for a given distortion type, we observe a linear relation model between the perceptual quality and SSIM for a given video clip, and the parameters of the linear model are determined by video contents. These parameters can be obtained through a machine learning process. It is shown by experimental results the proposed DCVQI offers a PLCC (Pearson Linear Correlation Coefficient) value of 0.869 when tested on the LIVE database, which outperforms all state-of-the-art video quality assessment methods, including MOVIE and STMAD.

In this paper, a novel rate control scheme with optimized bits allocation for the 3-D video coding is proposed. First, we investigate the R-D characteristics of the texture and depth map of the coded view, as well as the quality dependency between the virtual view and the coded view. Second, an optimal bit allocation scheme is developed to allocate target bits for both the texture and depth maps of different views. Meanwhile, a simplified model parameter estimation scheme is adopted to speed up the coding process. Finally, the experimental results on various 3-D video sequences demonstrate that the proposed algorithm achieves excellent R-D efficiency and bit rate accuracy compared to benchmark algorithms.

In this paper, we propose a view synthesis distortion model (VSDM) that establishes the relationship between depth distortion and view synthesis distortion for the regions with different characteristics: color texture area corresponding depth (CTAD) region and color smooth area corresponding depth (CSAD), respectively. With this VSDM, we propose regional bit allocation (RBA) and rate distortion optimization (RDO) algorithms for multiview depth video coding (MDVC) by allocating more bits on CTAD for rendering quality and fewer bits on CSAD for compression efficiency. Experimental results show that the proposed VSDM based RBA and RDO can improve the coding efficiency significantly for the test sequences. In addition, for the proposed overall MDVC algorithm that integrates VSDM based RBA and RDO, it achieves 9.99% and 14.51% bit rate reduction on average for the high and low bit rate, respectively. It can improve virtual view image quality 0.22 and 0.24 dB on average at the high and low bit rate, respectively, when compared with the original joint multiview video coding model. The RD performance comparisons using five different metrics also validate the effectiveness of the proposed overall algorithm. In addition, the proposed algorithms can be applied to both INTRA and INTER frames.

In this paper, we investigate the issues over the smooth quality and the smooth bit rate during rate control (RC) in H.264/AVC. An adaptive quantization-parameter (Q(p)) clip scheme is proposed to optimize the quality smoothness while keeping the bit-rate fluctuation at an acceptable level. First, the frame complexity variation is studied by defining a complexity ratio between two nearby frames. Second, the range of the generated bits is analyzed to prevent the encoder buffer from overflow and underflow. Third, based on the safe range of the generated bits, an optimal Q(p) clip range is developed to reduce the quality fluctuation. Experimental results demonstrate that the proposed Q(p) clip scheme can achieve excellent performance in quality smoothness and buffer regulation.

A novel frame-level rate control (RC) algorithm is presented in this paper for temporal scalability of scalable video coding. First, by introducing a linear quality dependency model, the quality dependency between a coding frame and its references is investigated for the hierarchical B-picture prediction structure. Second, linear rate-quantization (R-Q) and distortion-quantization (D-Q) models are introduced based on different characteristics of temporal layers. Third, according to the proposed quality dependency model and R-Q and D-Q models for each temporal layer, adaptive weighting factors are derived to allocate bits efficiently among temporal layers. Experimental results on not only traditional quarter common intermediate format/common intermediate format but also standard definition and high definition sequences demonstrate that the proposed algorithm achieves excellent coding efficiency as compared to other benchmark RC schemes.

Fast mode decision (FMD) algorithm aims to reduce the computational complexity of video encoders, especially for H.264/AVC, and meanwhile keep the coding efficiency. It is necessary for coding high resolution sequences, such as standard definition (SD) or high definition (HD) sequences. In this paper, an efficient algorithm is proposed to speed up inter-mode decision for H.264/AVC. Firstly, for a macroblock (MB) being coded, with modes of neighboring blocks, a probability model is utilized to build up an optimal encoding mode list; after that, rate-distortion (RD) cost-based early termination is employed to skip unnecessary inter-modes. Simulation results demonstrate that, the proposed algorithm could achieve significant encoding time saving for QCIF/CIF and SD/HD sequences, meanwhile with almost the same RD performance as the original encoder.

In this paper, a frame level rate control algorithm is proposed with a novel Rate-Quantization (R-Q) model for H.264/AVC. Firstly, a two-stage rate control scheme is adopted to decouple the inter-dependency between Rate Distortion Optimization (RDO) and rate control. Secondly, in order to predict the frame complexity accurately, instead of the Mean Absolute Difference (MAD) of the residual signal, bits information in the RDO-based mode decision process is employed to predict the frame complexity. Thirdly, a self-adaptive exponential R-Q model is proposed for rate control. Experimental results reveal that the proposed R-Q model can estimate the actual output bits very well, and the novel rate control scheme has excellent performance both in bit rate accuracy and coding efficiency as compared to JVT-W043 and the FixedQp tool in the Joint Scalable Video Model reference software.