Blind video quality assessment (BVQA) plays a pivotal role in evaluating and improving the viewing experience of end-users across a wide range of video-based platforms and services. Contemporary deep learning-based models primarily analyze video content in its aggressively subsampled format, while being blind to the impact of the actual spatial resolution and frame rate on video quality. In this paper, we propose a modular BVQA model and a method of training it to improve its modularity. Our model comprises a base quality predictor, a spatial rectifier, and a temporal rectifier, responding to the visual content and distortion, spatial resolution, and frame rate changes on video quality, respectively. During training, spatial and temporal rectifiers are dropped out with some probabilities to render the base quality predictor a standalone BVQA model, which should work better with the rectifiers. Extensive experiments on both professionally-generated content and user-generated content video databases show that our quality model achieves superior or comparable performance to current methods. Additionally, the modularity of our model offers an opportunity to analyze existing video quality databases in terms of their spatial and temporal complexity.

Investigating how people perceive virtual reality (VR) videos in the wild (i.e., those captured by everyday users) is a crucial and challenging task in VR-related applications due to complexauthenticdistortionslocalizedinspaceandtime.Existingpanoramic video databases only consider synthetic distortions, assume fixed viewing conditions, and are limited in size. To overcome these shortcomings, we construct the VR Video Quality in the Wild (VRVQW) database, containing 502 user-generated videos with diverse content and distortion characteristics. Based on VRVQW, we conduct a formal psychophysical experiment to record the scanpaths and perceived quality scores from 139 participants under two different viewing conditions. We provide a thorough statistical analysis of the recordeddata, observing significantimpact of viewing conditions on both human scanpaths and perceived quality. Moreover, we develop an objective quality assessment model for VR videos based on pseudocylindrical representation and convolution. Results on the proposed VRVQW show that our method is superior to existing video quality assessment models.We have made the database and code available at https://github.com/ limuhit/VR-Video-Quality-in-the-Wild.

Quality of experience (QoE) that serves as a direct evaluation of viewing experience from the end users is of vital importance for network optimization, and should be constantly monitored. Unlike existing video-on-demand streaming services, real-time interactivity is critical to the mobile live broadcasting experience for both broadcasters and their audiences. While existing QoE metrics that are validated on limited video contents and synthetic stall patterns have shown effectiveness in their trained QoE benchmarks, a common caveat is that they often encounter challenges in practical live broadcasting scenarios, where one needs to accurately understand the activity in the video with fluctuating QoE and figure out what is going to happen to support the real-time feedback to the broadcaster. In this paper, we propose a temporal relational reasoning guided QoE evaluation approach for mobile live video broadcasting, namely TRR-QoE, which explicitly attends to the temporal relationships between consecutive frames to achieve a more comprehensive understanding of the distortion-aware variation. In our design, video frames are first processed by deep neural network (DNN) to extract quality-indicative features. Afterwards, besides explicitly integrating features of individual frames to account for the spatial distortion information, multi-scale temporal relational information corresponding to diverse temporal resolutions are made full use of to capture temporal-distortion-aware variation. As a result, the overall QoE prediction could be derived by combining both aspects. The results of experiments conducted on a number of benchmark databases demonstrate the superiority of TRR-QoE over the representative state-of-the-art metrics.

Three-dimensional structure-based localization aims to estimate the six-DOF camera pose of a query image by means of feature matches against a 3D Structure-from-Motion (SfM) point cloud. For city-scale SfM point clouds with tens of millions of points, it becomes more and more difficult to disambiguate matches. Therefore, a 3D structure-based localization method, which can efficiently handle matches with very large outlier ratios, is needed. We propose a two-stage outlier filtering framework for city-scale localization that leverages both visibility and geometry intrinsics of the SfM point clouds. First, we propose a visibility-based outlier filter, which is based on a bipartite visibility graph, to filter outliers on a coarse level. Second, we apply a geometry-based outlier filter to generate a set of fine-grained matches with a novel data-driven geometrical constraint for efficient inlier evaluation. The proposed two-stage outlier filtering framework only relies on the intrinsic information of the SfM point cloud. It is thus widely applicable to be embedded into the existing localization approaches. The experimental results on two real-world datasets demonstrate the effectiveness of the proposed two-stage outlier filtering framework for city-scale localization.

Image set compression has recently emerged as an active research topic due to the rapidly increasing demand in cloud storage. In this paper, we propose a novel framework for image set compression based on the rate-distortion optimized sparse coding. Specifically, given a set of similar images, one representative image is first identified according to the similarity among these images, and a dictionary can be learned subsequently in wavelet domain from the training samples collected from the representative image. In order to improve coding efficiency, the dictionary atoms are reordered according to their use frequencies when representing the representative image. As such, the remaining images can be efficiently compressed with sparse coding based on the reordered dictionary that is highly adaptive to the content of the image set. To further improve the efficiency of sparse coding, the number of dictionary atoms for image patches is further optimized in a rate-distortion sense. Experimental results show that the proposed method can significantly improve the image compression performance compared with JPEG, JPEG2000, and the state-of-the-art dictionary learning-based methods.

Objective image retargeting quality assessment aims to use computational models to predict the retargeted image quality consistent with subjective perception. In this paper, we propose a multiple-level feature (MLF)-based quality measure to predict the perceptual quality of retargeted images. We first provide an in-depth analysis on the low-level aspect ratio similarity feature, and then propose a mid-level edge group similarity feature, to better address the shape/structure related distortion. Furthermore, a high-level face block similarity feature is designed to deal with sensitive region deformation. The multiple-level features are complementary as they quantify different aspects of quality degradation in the retargeted image, and the MLF measure learned by regression is used to predict the perceptual quality of retargeted images. Extensive experimental results performed on two public benchmark databases demonstrate that the proposed MLF measure achieves higher quality prediction accuracy than the existing relevant state-of-the-art quality measures.

Wiener filtering, which has been widely used in the field of image restoration, is statistically optimal in the sense of mean square error. The adaptive loop filter in video coding inherits the design of Wiener filters, and has been proved to achieve significant improvement on compression performance by reducing coding artifacts and providing high-quality references for subsequent frames. To further improve the compression performance via filtering technique, we explore the factors that may hinder the potential performance of Wiener-based filters, and propose a near-optimal filter learning scheme for high-efficiency image coding. Based on the analyses, we observe that the foremost factor affecting the performance of Wiener-based filters is the divergence of statistical characteristics of training samples, instead of the filter taps or shapes. In view of this, we propose an iterative training method to derive the near-optimal Wiener filter parameters by simultaneously labeling sample categories at the pixel level. These parameters are compressed and transmitted to the decoder side to improve the quality of decoded images by reducing the coding artifacts. Experimental results show that the proposed scheme achieves significant bitrate savings compared with high-efficiency video coding in high-bitrate intra coding scenario.

During the past few years, there have been various kinds of content-aware image retargeting methods proposed for image resizing. However, the lack of effective objective retargeting quality metric limits the further development of image retargeting. Different from the traditional image quality assessment, the quality degradation of the retargeted images is mainly caused by the geometric changes due to retargeting. In this paper, we propose a practical approach to reveal the geometric changes during image retargeting, and design an Aspect Ratio Similarity (ARS) metric to predict the visual quality of the retargeted image. The experimental results on the widely used dataset show that the proposed metric outperforms the state of the arts.

In this paper, we propose a novel reduced-reference quality assessment metric for image super-resolution (RRIQA-SR) based on the low-resolution (LR) image information. First, we use the Markov Random Field (MRF) to model the pixel correspondence between LR and high-resolution (HR) images. Based on the pixel correspondence, we predict the perceptual similarity between image patches of LR and HR images by two components: the energy change and texture variation. The overall quality of HR images is estimated by the perceptual similarity between local image patches of LR and HR images. Experimental results demonstrate that the proposed method can obtain better performance of quality prediction for HR images than other existing ones, even including some full-reference (FR) metrics.

In this paper, we propose an effective no-reference image quality assessment (IQA) method based on local region statistics (NRLRS). The proposed method is built on the hypothesis that image distortions may alter the local region statistics which can be well characterized by the inter-pixel relationship. Hence, by extracting perceptual features that describe the inter-pixel patterns of a distorted image, we can effectively quantify the impact of image degradation. For this purpose, the perceptual gray-level differences between neighboring pixels are extracted and a Gaussian Mixture Model (GMM) codebook is constructed as the generative model of extracted features. The Fisher vector representation is then derived to describe image as their derivations from the GMM model. Finally, partial least square regression is used to map the Fisher encodings to quality scores. Experimental results indicate that the proposed method achieves better performance in quality prediction as compared to relevant full-reference and no-reference IQA methods.

Distortions cause structural changes in digital images, leading to degraded visual quality. Dictionary-based sparse representation has been widely studied recently due to its ability to extract inherent image structures. Meantime, it can extract image features with slightly higher level semantics. Intuitively, sparse representation can be used for image quality assessment, because visible distortions can cause significant changes to the sparse features. In this paper, a new sparse representation-based image quality assessment model is proposed based on the construction of adaptive sub-dictionaries. An overcomplete dictionary trained from natural images is employed to capture the structure changes between the reference and distorted images by sparse feature extraction via adaptive sub-dictionary selection. Based on the observation that image sparse features are invariant to weak degradations and the perceived image quality is generally influenced by diverse issues, three auxiliary quality features are added, including gradient, color, and luminance information. The proposed method is not sensitive to training images, so a universal dictionary can be adopted for quality evaluation. Extensive experiments on five public image quality databases demonstrate that the proposed method produces the state-of-the-art results, and it delivers consistently well performances when tested in different image quality databases.

During the past few years, there have been various kinds of content-aware image retargeting operators proposed for image resizing. However, the lack of effective objective retargeting quality assessment metrics limits the further development of image retargeting techniques. Different from traditional image quality assessment (IQA) metrics, the quality degradation during image retargeting is caused by artificial retargeting modifications, and the difficulty for image retargeting quality assessment (IRQA) lies in the alternation of the image resolution and content, which makes it impossible to directly evaluate the quality degradation like traditional IQA. In this paper, we interpret the image retargeting in a unified framework of resampling grid generation and forward resampling. We show that the geometric change estimation is an efficient way to clarify the relationship between the images. We formulate the geometric change estimation as a backward registration problem with Markov random field and provide an effective solution. The geometric change aims to provide the evidence about how the original image is resized into the target image. Under the guidance of the geometric change, we develop a novel aspect ratio similarity (ARS) metric to evaluate the visual quality of retargeted images by exploiting the local block changes with a visual importance pooling strategy. Experimental results on the publicly available MIT RetargetMe and CUHK data sets demonstrate that the proposed ARS can predict more accurate visual quality of retargeted images compared with the state-of-the-art IRQA metrics.

In this paper, we propose a novel dense correspondence based prediction approach to reduce the inter-image redundancy for image set compression. Unlike previous methods, we manage to utilize the dense correspondence to predict and parameterize the inter-image relation and then reconstruct a new reference for the subsequent HEVC inter-prediction and encoding. Comparing to relevant state-of-the-art feature-based methods, our method is able to locally approximate the inter-image relation and thus more robust to complex local variations. Experimental results show that our proposed approach achieves better coding gains when the local variations are dominant.

Image set coding improves the compression efficiency by reducing both intra- and inter-image redundancy. The key of success is to select representative image(s) to predict set of similar images. This paper proposes an inter-image redundancy measure for representative image selection in image set compression. In the proposed method, the inter-image redundancy is measured jointly by the extent of similar content (EOS) and the correlation of similar content (COS) shared in two images. We take the covered area of matched SIFT points to measure the EOS, and take the distance of the matched SIFT descriptors to measure the COS. The image with largest redundancy for the set is selected as the representative one to predict other images. Experimental results show that the proposed method can select better representative image, and achieve bitrate saving up to 9.2% and 20.8% compared with state-of-the-art image set compression method and HEVC inter coding method.