In practical media distribution systems, visual content often undergoes multiple stages of quality degradations along the delivery chain between the source and destination. By contrast, current image quality assessment (IQA) models are typically validated on image databases with a single distortion stage. In this work, we construct two large-scale image databases that are composed of more than 2 million images undergoing multiple stages of distortions and examine how state-of-the-art IQA algorithms behave over distortion stages. Our results suggest that the performance of existing IQA models degrades rapidly with distortion stages, especially when the distortion types of different stages vary. We also find that full-reference and no-reference frameworks, though both readily applicable, have major drawbacks at predicting the quality of images at middle distortion stages. However, when the quality level of the previous stage is accessible, significantly improved quality prediction performance may be achieved. This study points out a new avenue of degraded-reference IQA research that is both practically desirable and technically challenging.

We propose a structural similarity (SSIM)motivated two-pass variable bit rate control algorithm for High Efficiency Video Coding. Given a bit rate budget, the available bits are optimally allocated at group of pictures (GoP), frame, and coding unit (CU) levels by hierarchically constructing a perceptually uniform space with an SSIM-inspired divisive normalization mechanism. The Lagrange multiplier λ, which controls the tradeoff between perceptual distortion and bit rate, is adopted as the GoP level complexity measure. To derive λ, Laplacian distribution-based rate and perceptual distortion models are established after the first pass encoding, and the target bits are dynamically allocated by maintaining a uniform Lagrange multiplier level for each GoP through λ equalization. Within each GoP, rate control is further performed at frame and CU levels based on SSIM-inspired divisive normalization, aiming to transform the prediction residuals into a perceptually uniform space. Experiments show that the proposed scheme achieves high accuracy rate control and superior rate-SSIM performance, which is further verified by subjective visual testing.

With the rapid growth of streaming media applications, there has been a strong demand of objective models that can predict end users' quality-of-experience (QoE) when watching the video being streamed to their display devices. Existing methods typically use bitrate and global statistics of stalling events as the QoE indicators. This is problematic for two reasons. First, using the same bitrate to encode different video content could result in drastically different presentation QoE. Second, the interactions between presentation visual quality and playback stalling are not accounted for. Here we propose a novel QoE prediction approach that takes into consideration the instantaneous quality degradation due to perceptual video presentation impairment, the playback stalling events caused by imperfect network delivery, and the instantaneous interactions between presentation quality and playback stalling. The proposed algorithm demonstrates strong promise when tested using a subject-rated video streaming QoE database.

High frame rate video has been a hot topic in the past few years driven by a strong need in the entertainment and gaming industry. Nevertheless, progress on perceptual quality assessment of high frame rate video remains limited, making it difficult to evaluate the exact perceptual gain by switching from low to high frame rates. In this work, we first conduct a subjective quality assessment experiment on a database that contains videos compressed at different frame rates, quantization levels and spatial resolutions. We then carry out a series of analysis on the subjective data to investigate the impact of frame rate on perceived video quality and its interplay with quantization level, spatial resolution, spatial complexity, and motion complexity. We observe that perceived video quality generally increases with frame rate, but the gain saturates at high rates. Such gain also depends on the interactions between quantization level, spatial resolution, and spatial and motion complexities.

We propose a perceptual two-pass rate control scheme for High Efficiency Video Coding (HEVC). The target bits are optimally allocated by hierarchically constructing a perceptual uniform space derived based on an SSIM-inspired divisive normalization mechanism for each group of pictures (GoP), each frame, and each coding unit (CU). The Lagrange multiplier λ, which controls the trade-off between perceptual distortion and bit rate, is adopted as the GoP level complexity measure. After the first pass compression, Laplacian based rate and perceptual distortion models are established to adaptively derive λ, and the target bits are dynamically allocated by maintaining an uniform Lagrange multiplier level through λ equalization. Within each GoP, rate control is further performed at frame and CU levels in the perceptually uniform space. Extensive simulations verify that, the proposed scheme can achieve high accuracy rate control and superior rate-SSIM performance.

Objective quality assessment of distorted stereoscopic images is a challenging problem, especially when the distortions in the left and right views are asymmetric. Existing studies suggest that simply averaging the quality of the left and right views well predicts the quality of symmetrically distorted stereoscopic images, but generates substantial prediction bias when applied to asymmetrically distorted stereoscopic images. In this paper, we first build a database that contains both single-view and symmetrically and asymmetrically distorted stereoscopic images. We then carry out a subjective test, where we find that the quality prediction bias of the asymmetrically distorted images could lean toward opposite directions (overestimate or underestimate), depending on the distortion types and levels. Our subjective test also suggests that eye dominance effect does not have strong impact on the visual quality decisions of stereoscopic images. Furthermore, we develop an information content and divisive normalization-based pooling scheme that improves upon structural similarity in estimating the quality of single-view images. Finally, we propose a binocular rivalry-inspired multi-scale model to predict the quality of stereoscopic images from that of the single-view images. Our results show that the proposed model, without explicitly identifying image distortion types, successfully eliminates the prediction bias, leading to significantly improved quality prediction of the stereoscopic images.

We propose a perceptual video coding framework based on the divisive normalization scheme, which is found to be an effective approach to model the perceptual sensitivity of biological vision, but has not been fully exploited in the context of video coding. At the macroblock (MB) level, we derive the normalization factors based on the structural similarity (SSIM) index as an attempt to transform the discrete cosine transform domain frame residuals to a perceptually uniform space. We further develop an MB level perceptual mode selection scheme and a frame level global quantization matrix optimization method. Extensive simulations and subjective tests verify that, compared with the H.264/AVC video coding standard, the proposed method can achieve significant gain in terms of rate-SSIM performance and provide better visual quality.

Recent advances in video capturing and display technologies, along with the exponentially increasing demand of video services, challenge the video coding research community to design new algorithms able to significantly improve the compression performance of the current H.264/AVC standard. This target is currently gaining evidence with the standardization activities in the High Efficiency Video Coding (HEVC) project. The distortion models used in HEVC are mean squared error (MSE) and sum of absolute difference (SAD). However, they are widely criticized for not correlating well with perceptual image quality. The structural similarity (SSIM) index has been found to be a good indicator of perceived image quality. Meanwhile, it is computationally simple compared with other state-of-the-art perceptual quality measures and has a number of desirable mathematical properties for optimization tasks. We propose a perceptual video coding method to improve upon the current HEVC based on an SSIM-inspired divisive normalization scheme as an attempt to transform the DCT domain frame prediction residuals to a perceptually uniform space before encoding. Based on the residual divisive normalization process, we define a distortion model for mode selection and show that such a divisive normalization strategy largely simplifies the subsequent perceptual rate-distortion optimization procedure. We further adjust the divisive normalization factors based on local content of the video frame. Experiments show that the proposed scheme can achieve significant gain in terms of rate-SSIM performance when compared with HEVC.

We propose a rate-distortion optimization (RDO) scheme based on the structural similarity (SSIM) index, which was found to be a better indicator of perceived image quality than mean-squared error, but has not been fully exploited in the context of image and video coding. At the frame level, an adaptive Lagrange multiplier selection method is proposed based on a novel reduced-reference statistical SSIM estimation algorithm and a rate model that combines the side information with the entropy of the transformed residuals. At the macroblock level, the Lagrange multiplier is further adjusted based on an information theoretical approach that takes into account both the motion information content and perceptual uncertainty of visual speed perception. Finally, the mode for H.264/AVC coding is selected by the SSIM index and the adjusted Lagrange multiplier. Extensive experiments show that the proposed scheme can achieve significantly better rate-SSIM performance and provide better visual quality than conventional RDO coding schemes.

Reduced-reference image quality assessment (RR-IQA) provides a practical solution for automatic image quality evaluations in various applications where only partial information about the original reference image is accessible. Here we propose an RR-IQA method by estimating the structural similarity (SSIM) index, which is a widely used full-reference (FR) image quality measure shown to be a good indicator of perceptual image quality. Specifically, we extract statistical features from a multi-scale, multi-orientation divisive normalization transform and develop a distortion measure by following the philosophy in the construction of SSIM. We found an interesting linear relationship between the FR SSIM measure and our RR estimate when the image distortion type is fixed. A regression-bydiscretization method is then applied to normalize our measure across image distortion types. We use six publiclyavailable subject-rated databases to test the proposed RR-SSIM method, which shows strong correlations with both SSIM and subjective quality evaluations. Finally, we introduce the novel idea of partially repairing an image using RR features and use deblurring as an example to demonstrate its application.

Perceptual image quality assessment (IQA) and sparse signal representation have recently emerged as high-impact research topics in the field of image processing. Here we make one of the first attempts to incorporate the structural similarity (SSIM) index, a promising IQA measure, into the framework of optimal sparse signal representation and approximation. In particular, we introduce a novel image denoising scheme where a modified orthogonal matching pursuit algorithm is proposed for finding the best sparse coefficient vector in maximum-SSIM sense for a given set of linearly independent atoms. Furthermore, a gradient descent algorithm is developed to achieve SSIM-optimal compromise in combining the input and sparse dictionary reconstructed images. Our experimental results show that the proposed method achieves better SSIM performance and provide better visual quality than least square optimal denoising methods.

The structural similarity (SSIM) index has been found to be a good indicator of perceived image quality. In this paper, we propose a rate-SSIM optimization scheme for mode se lection in H.264/AVC video coding. To derive the Lagrange multiplier based on the properties of input sequences, a novel reduced-reference statistical SSIM model and a source-side information combined rate model are established. The proposed method is fully standard-compatible. Experimental results demonstrate that, compared with conventional rate distortion optimization coding schemes, the proposed scheme can achieve better rate-SSIM performance and provide better visual quality.

Complex wavelet structural similarity (CW-SSIM) index has been proposed as a promising image similarity measure that is robust to small geometric distortions such as translation, scaling and rotation of images, but how to make the best use of it in image classification problems has not been deeply investigated. In this paper, we propose a novel “feature-extraction free” image classification algorithm based on CW-SSIM and use handwritten digit recognition as an example to demonstrate it. First, a CW-SSIM based unsupervised clustering method is used to divide the training images into clusters and to pick a representative image for each cluster. A supervised learning method based on support vector machines is then employed to maximize the classification accuracy based on CW-SSIM values between an input image and the representative images. Our experiments show that such a conceptually simple image classification method, which does not involve any registration, intensity normalization or sophisticated feature extraction processes, and does not rely on any modeling of the image patterns or distortion processes, achieves competitive performance with reduced computational complexity.

Perceptually inspired image processing has been an emerging field of study in recent years. Here we make one of the first efforts to incorporate the structural similarity (SSIM) index, a successful perceptual image quality assessment measure, into the framework of non-local means (NLM) image denoising, which is a state-of-the-art method that delivers superior desnoising performance. Specifically, a denoised image patch is obtained by weighted averaging of neighboring patches, where the similarity between patches as well as the weights assigned to the patches are determined based on an estimation of SSIM. A two-stage approach is proposed for robust SSIM estimation in the presence of noise. Moreover, motivated by the ideas behind SSIM, we adjust the contrast and mean of each patch before feeding it into the weighted averaging process. Our experimental results show that the proposed SSIM-based NLM algorithm achieves better SSIM and PSNR performance and provides better visual quality than least square based NLM method.

We propose a perceptual video coding framework based on an SSIM-inspired divisive normalization scheme as an attempt to transform the DCT domain frame prediction residuals to a perceptually uniform space before coding. Based on the residual divisive normalization process, we define a distortion model for mode selection and show that such a divisive normalization strategy largely simplifies the subsequent perceptual rate-distortion optimization procedure. Experiments demonstrate that the proposed scheme can achieve significant gain in terms of rate-SSIM performance in comparison with H.264/AVC.

The structural similarity (SSIM) index has been shown to be a good perceptual image quality predictor. In many real-world applications such as network visual communications, however, SSIM is not applicable because its computation requires full access to the original image. Here we propose a reduced-reference approach that estimates SSIM with only partial information about the original image. Specifically, we extract statistical features from a multi-scale, multi-orientation divisive normalization transform and develop a distortion measure by following the philosophy analogous to that in the construction of SSIM. We found an interesting linear relationship between our reduced-reference SSIM estimate and full-reference SSIM when the image distortion type is fixed. A regression-by-discretization method is then applied to normalize our measure between image distortion types. We use the LIVE database to test the proposed distortion measure, which shows strong correlations with both SSIM and subjective evaluations. We also demonstrate how our reduced-reference features may be employed to partially repair a distorted image.

Distributed video coding is typically treated as a channel coding problem among others. The encoder generates parity bits (or syndrome bits) for the source and transmits part of them to the decoder for a certain target quality. The decoder tries to reconstruct the source using the received parity bits along with the side information available. In this paper, we aim to estimate the critical number of parity bits to transmit. Having observed the uncertainty of the critical rate, we model it as a random variable, use its distribution to calculate the decoding failure probability and formulate the expected distortion. We allocate a certain bit budget among different bit-planes such that the expected distortion is minimized. Moreover, we introduce fast decoding at the encoder, which helps us to estimate the critical rate far more accurately. Eventually, we achieve up to 1.5 dB gain in rate-distortion performance at high bit rates.