Banding is a visually unpleasing artifact appearing in flat areas of encoded content that no video standard has fully addressed. We propose a normative debanding filter to tackle banding artifacts and have tested it as an inloop and post-loop filter in AVM. Debanding is achieved by introducing dithering on a frame level to the luma component. The proposed filter shows CAMBI gains for content with banding while not affecting other content. Although the added dithering has a minor negative impact on some objective metrics, subjective improvements in banding-prone content are (informally) observed. On the test set, encoding time increases on average by ~0.5%, while decoding time increases by around 0.5% for in-loop and 1.5% for post-loop.

Staircase-like contours introduced to a video by quantization in flat areas, commonly known as banding, have been a longstanding problem in both video processing and quality assessment communities. The fact that even a relatively small change of the original pixel values can result in a strong impact on perceived quality makes banding especially difficult to be detected by objective quality metrics. In this paper, we study how banding annoyance compares to more commonly studied scaling and compression artifacts with respect to the overall perceptual quality. We further propose a simple combination of VMAF and the recently developed banding index, CAMBI, into a banding-aware video quality metric showing improved correlation with overall perceived quality.

In recent years, screen content video is becoming increasingly popular in several major video applications, such as video recording and video conferencing. Due to the unique features of screen content videos that are not captured by camera sensors but produced artificially, dedicated coding tools have been developed for achieving significant compression efficiency gain. In recognition of the popularity of screen content applications, an open video dataset for screen content is proposed in this paper for the development of screen content coding technologies. The proposed video dataset consists of 12 typical screen content type video clips that are publicly available. In addition, to better understand the characteristics of the proposed video dataset, several major screen content coding tools in AOMedia Video 1 (AV1) have been evaluated on this dataset and analyzed in this paper.

In HTTP Adaptive Streaming, video content is conventionally encoded by adapting its spatial resolution and quantization level to best match the prevailing network state and display characteristics. It is well known that the traditional solution, of using a fixed bitrate ladder, does not result in the highest quality of experience for the user. Hence, in this paper, we introduce a content-driven approach for estimating the bitrate ladder, based on spatio-temporal features extracted from the uncompressed content. The method implements a content-driven interpolation. It uses the extracted features to train a machine learning model to infer the curvature points of the Rate-VMAF curves in order to guide a set of initial encodings. We employ the VMAF quality metric as a means of perceptually conditioning the estimation. When compared to the generation of a reference ladder using exhaustive encoding, 76.63% the estimated ladder's Rate-VMAF points are identical to those of the reference ladder. The proposed method benefits from a significant (77.4%) reduction in the number of encodes required with only a small (1.04%) average Bj⊘ntegaard Delta Rate increase.

Banding artifacts are artificially-introduced contours arising from the quantization of a smooth region in a video. Despite the advent of recent higher quality video systems with more efficient codecs, these artifacts remain conspicuous, especially on larger displays. In this work, a comprehensive subjective study is performed to understand the dependence of the banding visibility on encoding parameters and dithering. We subsequently develop a simple and intuitive no-reference banding index called CAMBI (Contrast-aware Multiscale Banding Index) which uses insights from Contrast Sensitivity Function in the Human Visual System to predict banding visibility. CAMBI correlates well with subjective perception of banding while using only a few visually-motivated hyperparameters.

In this paper we compare the performance of two state-of-the-art competing codecs, AV1 and HEVC, in the context of adaptive streaming. We specifically consider a Dynamic Optimizer (DO) methodology that is content-aware and selects the resolution of the video sequence after constructing the convex hull of the Rate-Quality curves of all considered resolutions. We start with an objective evaluation of the Dynamic Optimizer, based on both PSNR and VMAF quality metrics. The Rate-VMAF curves show an average of 6.3% BD-Rate gain of AV1 over HEVC, while the Rate-PSNR curves an show an average BD-Rate loss of 1.8%. We then report subjective tests which evaluate the perceived quality of the selected bitstreams generated by the two codecs. In this case it was found that, for most rate points, the difference in the perceived quality between HEVC and AV1 is not significant.

This paper presents a novel Convolutional Neural Network (CNN) based effective bit depth adaptation approach (EBDA-CNN) for video compression. It applies effective bit depth down-sampling before encoding and reconstructs the original bit depth using a deep CNN based up-sampling method at the decoder. The proposed approach has been integrated with the High Efficiency Video Coding reference software HM 16.20, and evaluated under the Joint Video Exploration Team Common Test Conditions using the Random Access configuration. The results show consistent coding gains on all tested sequences, with an average bitrate saving of 6.4%, based on Bjøntegaard Delta measurements using PSNR.

A video compression framework based on spatio-temporal resolution adaptation (ViSTRA) is proposed, which dynamically resamples the input video spatially and temporally during encoding, based on a quantisation-resolution decision, and reconstructs the full resolution video at the decoder. Temporal upsampling is performed using frame repetition, whereas a convolutional neural network super-resolution model is employed for spatial resolution upsampling. ViSTRA has been integrated into the high efficiency video coding reference software (HM 16.14). Experimental results verified via an international challenge show significant improvements, with BD-rate gains of 15% based on PSNR and an average MOS difference of 0.5 based on subjective visual quality tests.

In this paper we present the BVI-SR video database, which contains 24 unique video sequences at a range of spatial resolutions up to UHD-1 (3840p). These sequences were used as the basis for a large-scale subjective experiment exploring the relationship between visual quality and spatial resolution when using three distinct spatial adaptation filters (including a CNN-based super-resolution method). The results demonstrate that while spatial resolution has a significant impact on mean opinion scores (MOS), no significant reduction in visual quality between UHD-1 and HD resolutions for the super-resolution method is reported. A selection of image quality metrics were benchmarked on the subjective evaluations, and analysis indicates that VIF offers the best performance.

This paper presents a full reference objective video quality metric (SRQM), which characterises the relationship between variations in spatial resolution and visual quality in the context of adaptive video formats. SRQM uses wavelet decomposition, subband combination with perceptually inspired weights, and spatial pooling, to estimate the relative quality between the frames of a high resolution reference video, and one that has been spatially adapted through a combination of down and upsampling. The BVI-SR video database is used to benchmark SRQM against five commonly-used quality metrics. The database contains 24 diverse video sequences that span a range of spatial resolutions up to UHD-1 (3840×2160). An indepth analysis demonstrates that SRQM is statistically superior to the other quality metrics for all tested adaptation filters, and all with relatively low computational complexity.

In this paper, a novel spatial resolution adaptation approach for video compression is proposed. Its ability to dynamically apply downsampling to frames exhibiting low spatial detail delivers improved rate distortion performance, together with a reduction in computational complexity of the encoding process. This method is based on an experimental investigation of the dependence between the QP threshold, which determines when to encode lower resolution frames, and the distortion obtained after downsampling/upsampling. The proposed approach is integrated with the High Efficiency Video Coding (HEVC) reference codec for intra coding, and evaluated on 15 high-resolution test sequences with varying levels of spatial detail. The results show a promising average bitrate savings of approximately 4% (B-D measurements), and significant complexity reduction (29% on average).

Encoding spatio-temporally varying textures is challenging for standardised video encoders, with significantly more bits required for textured blocks compared to non-textured blocks. It is therefore beneficial to understand video textures in terms of both their spatio-temporal characteristics and their encoding statistics in order to optimize coding modes and performance. To this end, we examine the classification of video texture based on encoder performance. For this purpose, we employ spatio-temporal features and follow a two-step feature selection process by employing unsupervised machine learning approaches across the selected feature space. Finally, supervised machine learning approaches are applied on the set of the selected features that support classification prior to encoding with up to 95.1% accuracy. The results of this study offer the potential to underpin a new informed approach to a new informed approach to codec configuration and mode selection.

In this paper, an extensive study of different video texture properties based on encoding statistics extracted from the HEVC HM reference software is presented. Mode selection, partitioning, motion vectors and bitrate allocation are among the statistics obtained from the encoder. For this study, a new dataset of homogeneous static and dynamic video textures, HomTex, is proposed. A comprehensive investigation of the results reveals a significant variability of coding statistics within dynamic textures, suggesting that this category should be further split into two relevant subcategories, continuous dynamic textures and discrete dynamic textures. This case is supported by an unsupervised learning approach on the statistics extracted. Finally, following the results obtained, some suggestions of improvements in video texture coding are presented.

This work addresses the problem of predicting the compression efficiency of a video codec solely from features extracted from uncompressed content. Towards this goal, we have used a database of videos of homogeneous texture and extracted both spatial and frequency domain features. The videos are encoded using High Efficiency Video Coding (HEVC) reference codec at different quantization scales and their Rate-Distortion (RD) curves are modelled using linear regression. Using the extracted features and the fitted parameters of the RD model, a Support Vector Regression Model (SVRM) is trained to learn the relationship of the textural features with the RD curves. The SVRM is tested using iterative five-fold cross-validation. The presented experimental results demonstrate that RD curve characteristics can be predicted based on the textural features of the uncompressed videos, which offers potential benefits for encoder optimization.

This paper presents the results of an experimental study of the popular Bag-of-Features (BoF) model for the application of unsupervised learning of images, or image clustering.Although this method has been extensively applied for image classification and scene recognition, there has been few works which employ it in an unsupervised way.Also, due to the fact that the BoF model requires a great amount of steps, algorithms and parameter settings, we felt like there was a lack of detailed studies about the subject.We implemented testing routines in Python which we made publicly available in GitHub.In order to assess the performance of the model, three image datasets were used, namely, Coil-20 dataset, Natural and Urban dataset and Event dataset.The results obtained indicate that the BoF method provides a good representation of simple image collections for the purpose of clustering.However, it requires fine tunning of the parameters and algorithms for each dataset and obtains poor results for more complex scene datasets.We can therefore conclude that more advanced techniques are required in order to be able to effectively extract information from large image collections.