Over-the-top (OTT) video streaming services have been growing rapidly in the last decade, and thus increasing Quality-of-Experience (QoE) of end users is of great interest in emerging services. Although, numerous subjective studies have been conducted to investigate the impact of video presentation quality and playback interruptions, understanding the interactions between impairment types is still an open problem. In this work, we develop a streaming video dataset that contains compressed videos with different distortion levels as well as playback stalling events. Then, a subjective user study is performed to measure the QoE of these videos. The results of our experiment reveal strong dependency between video presentation quality and playback interruptions that provides useful insight for designing QoE models in video streaming.

With the fast advances in video acquisition, computational imaging, and display technologies, there has been a growing interest in high dynamic range (HDR) videos. Tone mapping operators (TMOs) that convert HDR content to low dynamic range (LDR) ones provide a practically useful solution for the visualization of HDR videos on standard LDR displays, where the user experience highly depends on the performance of the TMOs being used. Without an appropriate perceptual quality measure, different TMOs cannot be compared. Subjective experiments may be a reliable solution, but is time consuming, expensive, and difficult to be embedded into optimization processes. Here we make one of the first attempts to develop an objective quality assessment model for tone-mapped videos that incorporates structural fidelity, statistical naturalness and memory effect. Validation using subject-rated tone-mapped videos show that the proposed method is well-correlated with subjective scores.

Network streaming video services have been growing explosively in the past decade, but how to measure and assure the video quality-of-experience (QoE) of end consumers is still an open problem. Poor presentation quality and playback stalling have been identified as the most dominant factors that degrade user QoE. Although both factors have been studied individually, little is known about the interactions between them. In this work, we first construct a streaming video database that contains compressed videos at different distortion levels and with different stalling patterns. We then carry out a subjective test to evaluate the QoE of the videos. The results reveal some interesting dependency between presentation quality and playback stalling. Specifically, playback stalling always causes QoE degradation, but the strength of such degradation depends on the presentation quality when the stalling event occurs.

Advances in the field of medical imaging have led to an immense increase in the volume of images being acquired. A fast growing application is to enable physicians to access image data remotely from any viewing device. This casts new challenges for data rate compression. Meanwhile medical images typically have High Dynamic Range (HDR), which needs to be transformed to Low Dynamic Range (LDR) through a so-called “windowing” operation in order for them to be viewed on standard displays to best visualize specific types of content such as tissues or bone structures. This leads to a basic question: Should data compression be performed before windowing or vice versa? Answering this question needs domain knowledge and also requires comparing HDR and LDR images in terms of objective measures, which has only recently become possible. In this paper, we compare the two alternative schemes by using a recently proposed structural fidelity measure. Our study suggests that data compression followed by windowing delivers better performance than the other alternative.

Contrast is a fundamental attribute of images that plays an important role in human visual perception of image quality. With numerous approaches proposed to enhance image contrast, much less work has been dedicated to automatic quality assessment of contrast changed images. Existing approaches rely on global statistics to estimate contrast quality. Here we propose a novel local patch-based objective quality assessment method using an adaptive representation of local patch structure, which allows us to decompose any image patch into its mean intensity, signal strength and signal structure components and then evaluate their perceptual distortions in different ways. A unique feature that differentiates the proposed method from previous contrast quality models is the capability to produce a local contrast quality map, which predicts local quality variations over space and may be employed to guide contrast enhancement algorithms. Validations based on four publicly available databases show that the proposed patch-based contrast quality index (PCQI) method provides accurate predictions on the human perception of contrast variations.

Tone mapping operators (TMOs) aim to compress high dynamic range (HDR) images to low dynamic range (LDR) ones so as to visualize HDR images on standard displays. Most existing TMOs were demonstrated on specific examples without being thoroughly evaluated using well-designed and subject-validated image quality assessment models. A recently proposed tone mapped image quality index (TMQI) made one of the first attempts on objective quality assessment of tone mapped images. Here, we propose a substantially different approach to design TMO. Instead of using any predefined systematic computational structure for tone mapping (such as analytic image transformations and/or explicit contrast/edge enhancement), we directly navigate in the space of all images, searching for the image that optimizes an improved TMQI. In particular, we first improve the two building blocks in TMQI—structural fidelity and statistical naturalness components—leading to a TMQI-II metric. We then propose an iterative algorithm that alternatively improves the structural fidelity and statistical naturalness of the resulting image. Numerical and subjective experiments demonstrate that the proposed algorithm consistently produces better quality tone mapped images even when the initial images of the iteration are created by the most competitive TMOs. Meanwhile, these results also validate the superiority of TMQI-II over TMQI.

Image interpolation techniques that create high-resolution images from low-resolution (LR) images are widely used in real world applications, but how to evaluate the quality of interpolated images is not a well-resolved issue. Subjective assessment methods are useful and reliable, but are also slow and expensive. Here, we propose an objective method to assess the quality of an interpolated natural image using the available LR image as a reference. Our method adopts a natural scene statistics (NSS) framework, where image quality degradation is gauged by the deviation of its statistical features from the NSS models trained upon high-quality natural images. Two distortion measures are proposed, namely, interpolated natural image distortion (IND) and weighted IND. Validations by subjective tests show that the proposed approach performs statistically equivalent or sometimes better than an average human subject. Moreover, we demonstrate the potential application of the proposed method in parameter tuning of image interpolation algorithms.

There has been a growing recent interest of applying Kol-mogorov complexity and its related normalized information distance (NID) measures in real-world problems, but their application in the field of medical image processing remains limited. In this work we attempt to incorporate NID in the design of windowing operators for optimal visualization of high dynamic range (HDR) medical images, where predefined intensity interval of interest needs to be mapped to match the low dynamic range (LDR) of standard displays. By approximating NID using a Shannon entropy based method, we are able to optimize parametric windowing operators to maximize the information similarity between the HDR image and the LDR image after mapping. Experimental results demonstrate promising performance of the proposed approach.

The vast majority of today's internet video services are consumed over-the-top (OTT) via reliable streaming (HTTP via TCP), where the primary noticeable delivery-related impairments are startup delay and stalling. In this paper we introduce an objective model called the delivery quality score (DQS) model, to predict user's QoE in the presence of such impairments. We describe a large subjective study that we carried out to tune and validate this model. Our experiments demonstrate that the DQS model correlates highly with the subjective data and that it outperforms other emerging models.

An active research topic in recent years is to design tone mapping operators (TMOs) that convert high dynamic range (H-DR) to low dynamic range (LDR) images, so that HDR images can be visualized on standard displays. Nevertheless, most existing work has been done in the absence of a well-established and subject-validated image quality assessment (IQA) model, without which fair comparisons and further improvement are difficult. Recently, a tone mapped image quality index (TMQI) was proposed, which has shown to have good correlation with subjective evaluations of tone mapped images. Here we propose a substantially different approach to design TMO, where instead of using any pre-defined systematic computational structure (such as image transformation or contrast/edge enhancement) for tone mapping, we navigate in the space of all images, searching for the image that optimizes TMQI. The navigation involves an iterative process that alternately improves the structural fidelity and statistical naturalness of the resulting image, which are the two fundamental building blocks in TMQI. Experiments demonstrate the superior performance of the proposed method.

Tone mapping operators (TMOs) that convert high dynamic range (HDR) images to standard low dynamic range (LDR) images are highly desirable for the visualization of these images on standard displays. Although many existing TMOs produce visually appealing images, it is until recently validated objective measures that can assess their quality have been proposed. Without such objective measures, the design of traditional TMOs can only be based on intuitive ideas, lacking clear goals for further improvement. In this paper, we propose a substantially different tone mapping approach, where instead of explicitly designing a new computational structure for TMO, we search in the space of images to find better quality images in terms of a recent objective measure that can assess the structural fidelity between two images of different dynamic ranges. Specifically, starting from any initial image, the proposed algorithm moves the image along the gradient ascent direction and stops until it converges to a maximal point. Our experiments show that the proposed algorithm reliably produces better quality images upon a number of state-of-the-art TMOs.

Tone-mapping operators (TMOs) that convert high dynamic range (HDR) to low dynamic range (LDR) images provide practically useful tools for the visualization of HDR images on standard LDR displays. Different TMOs create different tone-mapped images, and a natural question is which one has the best quality. Without an appropriate quality measure, different TMOs cannot be compared, and further improvement is directionless. Subjective rating may be a reliable evaluation method, but it is expensive and time consuming, and more importantly, is difficult to be embedded into optimization frameworks. Here we propose an objective quality assessment algorithm for tone-mapped images by combining: 1) a multiscale signal fidelity measure on the basis of a modified structural similarity index and 2) a naturalness measure on the basis of intensity statistics of natural images. Validations using independent subject-rated image databases show good correlations between subjective ranking score and the proposed tone-mapped image quality index (TMQI). Furthermore, we demonstrate the extended applications of TMQI using two examples-parameter tuning for TMOs and adaptive fusion of multiple tone-mapped images.

There has been an increasing number of image super-resolution (SR) algorithms proposed recently to create images with higher spatial resolution from low-resolution (LR) images. Nevertheless, how to evaluate the performance of such SR and interpolation algorithms remains an open problem. Subjective assessment methods are useful and reliable, but are expensive, time-consuming, and difficult to be embedded into the design and optimization procedures of SR and interpolation algorithms. Here we make one of the first attempts to develop an objective quality assessment method of a given resolution-enhanced image using the available LR image as a reference. Our algorithm follows the philosophy behind the natural scene statistics (NSS) approach. Specifically, we build statistical models of frequency energy falloff and spatial continuity based on high quality natural images and use the departures from such models to quantify image quality degradations. Subjective experiments have been carried out that verify the effectiveness of the proposed approach.

There has been a growing interest in recent years to develop tone mapping algorithms that can convert high dynamic range (HDR) to low dynamic range (LDR) images, so that they can be visualized on standard displays. With a number of tone mapping algorithms proposed, a natural question is which one gives the best performance. Although subjective assessment methods provide useful references, they are expensive and time-consuming, and are difficult to be embedded into the design stage of tone mapping algorithms for optimization and parameter tuning purposes. This paper focuses on objective assessment of tone mapping operators. Inspired by the success of the structural similarity index method for image quality assessment, we propose a new objective assessment algorithm that creates multi-scale similarity maps between HDR and LDR images. Our experiments show that the proposed method correlates well with subjective rankings of existing tone mapping operators. Furthermore, we demonstrate how the proposed algorithm can be employed in an existing tone mapping algorithm for optimal parameter tuning.