Video compression standard H.264/AVC was released in 2003, and has been dominating the industry for the past decade. Over the last few years, a number of next-generation standards/formats like VP9 (2012), H.265/HEVC (2013) and AV1 (2018) were introduced, all claiming significant improvement over H.264/AVC. In this paper, we present our evaluation of the performance of these compression standards. Our evaluation is conducted using open-source encoder implementations of these standards, x264 (for H.264/AVC), x265 (for H.265/HEVC), libvpx (for VP9) and aomenc (for AV1). The process is designed to evaluate the attainable compression efficiency for on-demand adaptive streaming applications. Results with two different quality metrics, PSNR and VMAF, are reported. Our results reveal that x265, libvpx and aomenc all achieve substantial compression efficiency improvement over x264.

HTTP adaptive streaming is being increasingly deployed by network content providers, such as Netflix and YouTube. By dividing video content into data chunks encoded at different bitrates, a client is able to request the appropriate bitrate for the segment to be played next based on the estimated network conditions. However, this can introduce a number of impairments, including compression artifacts and rebuffering events, which can severely impact an end-user's quality of experience (QoE). We have recently created a new video quality database, which simulates a typical video streaming application, using long video sequences and interesting Netflix content. Going beyond previous efforts, the new database contains highly diverse and contemporary content, and it includes the subjective opinions of a sizable number of human subjects regarding the effects on QoE of both rebuffering and compression distortions. We observed that rebuffering is always obvious and unpleasant to subjects, while bitrate changes may be less obvious due to content-related dependencies. Transient bitrate drops were preferable over rebuffering only on low complexity video content, while consistently low bitrates were poorly tolerated. We evaluated different objective video quality assessment algorithms on our database and found that objective video quality models are unreliable for QoE prediction on videos suffering from both rebuffering events and bitrate changes. This implies the need for more general QoE models that take into account objective quality models, rebuffering-aware information, and memory. The publicly available video content as well as metadata for all of the videos in the new database can be found at http://live.ece.utexas.edu/research/LIVE_NFLXStudy/nflx_index.html.

A cloud-based encoding pipeline which generates streams for video-on-demand distribution typically processes a wide diversity of content that exhibit varying signal characteristics. To produce the best quality video streams, the system needs to adapt the encoding to each piece of content, in an automated and scalable way. In this paper, we describe two algorithm optimizations for a distributed cloud-based encoding pipeline: (i) per-title complexity analysis for bitrate-resolution selection; and (ii) per-chunk bitrate control for consistent-quality encoding. These improvements result in a number of advantages over a simple “one-size-fits-all” encoding system, including more efficient bandwidth usage and more consistent video quality.

A compressed video quality assessment dataset based on the just noticeable difference (JND) model, called MCL-JCV, is recently constructed and released. In this work, we explain its design objectives, selected video content and subject test procedures. Then, we conduct statistical analysis on collected JND data. We compute the difference between every two adjacent JND points and propose an outlier detection algorithm to remove unreliable data. We also show that each JND difference group can be well approximated by a normal distribution so that we can adopt the Gaussian mixture model (GMM) to characterize the distribution of multiple JND points. Finally, it is demonstrated by experimental results that the proposed JND analysis performed in the difference domain, called the D-method, achieves a lower BIC (Bayesian information criteria) value than the previously proposed G-method.

Parallel encoding systems face the problem of distributed coding decisions, an example of which is rate allocation between different encoding chunks. Distributed encoding is traditionally performed by assigning a fixed bitrate or quality setting to each chunk, after which the chunk uses rate control to achieve that particular target. In this paper, we discuss the use of constant-slope encoding, resulting in chunks which all operate at the same quality-rate trade-off. The (Lagrangian) optimization can be performed independent of the chosen codec or distortion/quality metric. In this paper, we report results for mobile resolutions for VP9 encodes, on both open-source and Netflix catalog original and licensed content titles. BD-rate gains are obtained of 6.8 % on average, and a more constant quality across titles is observed.

Natural scene statistics are well studied in the context of picture quality assessment and have been used in a wide variety of top-performing picture quality prediction models. Upscaling artifacts have been measured with regards to quality impairment using these kinds of models. However, the assessment and classification of subtle, less discriminable upscaling artifacts remains an unsolved problem. The nearly imperceptible artifacts pertaining to the extent and type of upscaling have not been predicted using natural scene statistics (NSS)-based models. We develop an accurate model for predicting the upscaling ratio applied to any natural image. By decomposing an input image frame using an orthogonal filter bank and locally normalizing the resulting responses, we show that the local energy terms can be used to predict the upscaling ratio. In fact, a simple linear regressor can be trained on these energy measurements; hence, no hyperparameter tuning is necessary. We compare the proposed model with other no-reference models using real-world data contained in the Netflix collection.

The Netflix ingest and encoding pipeline is a cloud-based platform that generates video encodes for the Netflix streaming service. Due to the large throughput of the system, automated video quality assessment of the source videos and the generated encodes is essential in ensuring the quality of experience of viewers. This paper discusses the motivations for integrating video quality assessment in the production pipeline, outlines currently deployed solutions and presents the technical challenges in improving the system.

This paper proposes systematic lossy error protection of video waveforms using multiple embedded Wyner-Ziv video descriptions. A video signal transmitted over an error-prone channel, without channel coding, constitutes the systematic portion of the transmission. Error protection is achieved by additionally transmitting two or more bitstreams generated by Wyner-Ziv coding of the video signal. The Wyner-Ziv bitstreams contain multiple embedded coarsely quantized descriptions of the original video signal. In the event of channel errors, the Wyner-Ziv descriptions are decoded, thereby limiting the maximum distortion that can occur. The available Wyner-Ziv bit-rate is allocated among the multiple embedded descriptions, using the observation that the visibility of slice losses and quantization artifacts depends on the employed coding structure. Experimental results show that the trade-off between error-resilience and residual quantization distortion can be better exploited using multiple embedded Wyner-Ziv descriptions, compared to earlier results with a single Wyner-Ziv bitstream. The system delivers gracefully deteriorating video quality without requiring a layered video compression scheme in the systematic portion of the transmission.

Distributed coding is a new paradigm for video compression, based on Slepian and Wolf's and Wyner and Ziv's information-theoretic results from the 1970s. This paper reviews the recent development of practical distributed video coding schemes. Wyner-Ziv coding, i.e., lossy compression with receiver side information, enables low-complexity video encoding where the bulk of the computation is shifted to the decoder. Since the interframe dependence of the video sequence is exploited only at the decoder, an intraframe encoder can be combined with an interframe decoder. The rate-distortion performance is superior to conventional intraframe coding, but there is still a gap relative to conventional motion-compensated interframe coding. Wyner-Ziv coding is naturally robust against transmission errors and can be used for joint source-channel coding. A Wyner-Ziv MPEG encoder that protects the video waveform rather than the compressed bit stream achieves graceful degradation under deteriorating channel conditions without a layered signal representation.

In the current interframe video compression systems, the encoder performs predictive coding to exploit the similarities of successive frames. The Wyner-Ziv theorem on source coding with side information available only at the decoder suggests that an asymmetric video codec, where individual frames are encoded separately, but decoded conditionally (given temporally adjacent frames) could achieve similar efficiency. In the previous work we propose a Wyner-Ziv coding scheme for motion video that uses intraframe encoding instead of interframe decoding. In this paper we improve on our Wyner-Ziv video codec by sending hash codewords of the current frame to aid the decoder in accurately estimating the motion. This allows us to implement a low-delay system where only the previous reconstructed frame is used to generate the side information of a current frame. Simulation results show significant gains above conventional DCT-based intraframe coding. The Wyner-Ziv video codec with hash-based motion compensation at the receiver enables low-complexity encoding while achieving high compression efficiency.

We present a practical scheme for error-resilient digital video broadcasting, using the Wyner-Ziv coding paradigm. We apply the general framework of systematic lossy source-channel coding to generate a supplementary bitstream that can correct transmission errors in the decoded video waveform up to a certain residual distortion. The systematic portion consists of a conventional MPEG-2 bitstream, which is transmitted over the error-prone channel without forward error correction. The supplementary bitstream is a low rate representation of the transmitted video sequence generated using Wyner-Ziv encoding. We use the received error-prone MPEG-2 prediction error signal as side information to decode the Wyner-Ziv bits. The decoder combines the error-prone side information and the Wyner-Ziv description to yield an improved decoded video signal. Wc describe a system that uses an embedded Wyner-Ziv codec to achieve graceful quality degradation without the need for a layered representation of the video waveform.

Image-based rendering data sets, such as light fields, require efficient compression due to their large data size, but also easy random access when rendering from the data set. Efficient compression usually depends upon prediction between images, which creates dependencies between them, conflicting with the requirement of having easy random access. In this paper we propose to eliminate prediction at the encoder by using Wyner-Ziv coding for compressing the light field images. The images are independently encoded by a Wyner-Ziv encoder. At the receiver, previously reconstructed images are used by the Wyner-Ziv decoder as side information to exploit similarities among images. Simulation results show significant compression performance gains compared to conventional independent image coding while maintaining random access capabilities.

In current interframe video compression systems, the encoder performs predictive coding to exploit the similarities of successive frames. The Wyner-Ziv Theorem on source coding with side information available only at the decoder suggests that an asymmetric video codec, where individual frames are encoded separately, but decoded conditionally (given temporally adjacent frames) could achieve similar efficiency. We propose a transformdomain Wyner-Ziv coding scheme for motion video that uses intraframe encoding, but interframe decoding. In this system, the transform coefficients of a Wyner-Ziv frame are encoded independently using a scalar quantizer and turbo coder. The decoder uses previously reconstructed frames to generate side information to conditionally decode the Wyner-Ziv frames. Simulation results show significant gains above DCT-based intraframe coding and improvements over the pixel-domain Wyner-Ziv video coder.

We present a novel scheme for error-resilient digital video broadcasting,using the Wyner-Ziv coding paradigm. We apply the general framework of systematic lossy source-channel coding to generate a supplementary bitstream that can correct transmission errors in the decoded video waveform up to a certain residual distortion. The systematic portion consists of a conventional MPEG-coded bitstream, which is transmitted over the error-prone channel without forward error correction.The supplementary bitstream is a low rate representation of the transmitted video sequence generated using Wyner-Ziv encoding. We use the conventionally decoded error-concealed MPEG video sequence as side information to decode the Wyner-Ziv bits. The decoder combines the error-prone side information and the Wyner-Ziv description to yield an improved decoded video signal. Our results indicate that, over a large range of channel error probabilities, this scheme yields superior video quality when compared with traditional forward error correction techniques employed in digital video broadcasting.

Two separate applications of Wyner-Ziv coding of motion video coding are considered. The Wyner-Ziv theorem on source coding with side information available only at the decoder suggests that an asymmetric video codec, where individual frames are encoded separately, but decoded conditionally could achieve efficiency comparable to current interframe video compression systems. First results on the Wyner-Ziv coding scheme for motion video are reported that use interframe encoding, but interframe decoding. Secondly, Wyner-Ziv coding for the transmission of compressed video over an error-prone channel is applied. This coding scheme is used to generate a supplementary bit stream, which contains a coarsely quantized representation of the transmitted video signal. Using the conventionally decoded, error-concealed video signal as side information to decode the Wyner-Ziv bits, the transmission errors in the decoded video waveform are corrected up to a certain residual distortion, significantly improving the visual quality of the decoded video.

A novel scheme for error-resilient digital video broadcasting, using Wyner-Ziv coding, is presented in this paper. We apply the general framework of systematic lossy source-channel coding to generate a supplementary bitstream that can correct transmission errors in the decoded video waveform up to a certain residual distortion. The systematic portion consists of a conventional MPEG-coded bitstream, which is transmitted over the error-prone channel without any forward error correction. The supplementary bitstream is a low rate representation of the transmitted video sequence generated using Wyner-Ziv encoding. We use the conventionally decoded error-concealed MPEG video sequence as side information to decode the Wyner-Ziv bits. The decoder combines the error-prone side information and the Wyner-Ziv description to yield an improved decoded video signal. We show how this scheme can be used to build a system which achieves graceful degradation without the need of a layered representation of the video waveform.

In current interframe video compression systems, the encoder performs predictive coding to exploit the similarities of successive frames. The Wyner-Ziv theorem on source coding with side information available only at the decoder suggests that an asymmetric video codec, where individual frames are encoded separately, but decoded conditionally (given temporally adjacent frames) achieves similar efficiency. We report results on a Wyner-Ziv coding scheme for motion video that uses intraframe encoding, but interframe decoding. In the proposed system, key frames are compressed by a conventional intraframe codec and in-between frames are encoded using a Wyner-Ziv intraframe coder. The decoder uses previously reconstructed frames to generate side information for interframe decoding of the Wyner-Ziv frames.

We show that turbo codes can come close to the Slepian-Wolf bound in lossless distributed source coding. In the asymmetric scenario considered, X and Y are statistically dependent signals and X is encoded with no knowledge of Y. However, Y is known as side information at the decoder. We use a system based on turbo codes to send X at a rate close to H(X|Y). We apply our system to binary sequences and simulations show performance close to the information-theoretic limit. For distributed source coding of Gaussian sequences, our results show significant improvement over previous work. The scheme also performs well for joint source-channel coding.