Video processing and compression are being re-envisioned in the age of AI. Traditional video codecs, which rely on rigid, pre-defined rules, are being augmented and, in some cases, replaced by AI-driven approaches. These new methods leverage machine learning to intelligently analyze video content, allowing for more adaptive and efficient compression. We are going to discuss AOM's new codec AV2, its low- and high-level features that enable significantly smaller file sizes with no perceptible loss in quality, a crucial development for streaming and storage. The shift to AI has also transformed how we evaluate video quality. Traditional metrics, while directionally useful, don't always align with human perception, especially for user generated content (UGC). They fail to capture what's most important for machine vision tasks. We will talk about new AI-based quality metrics that are being developed. They correlate better with a human's subjective experience and a machine's ability to perform tasks like object recognition. Along the way, we'll cover large scale industrial infrastructure challenges and the ways to achieve high reliability and accuracy.

Low-precision quantization is recognized for its efficacy in neural network optimization. Our analysis reveals that non-quantized elementwise operations which are prevalent in layers such as parameterized activation functions, batch normalization, and quantization scaling dominate the inference cost of low-precision models. These non-quantized elementwise operations are commonly overlooked in SOTA efficiency metrics such as Arithmetic Computation Effort (ACE) [46]. In this paper, we propose ACEv2- an extended version of ACE which offers a better alignment with the inference cost of quantized models and their energy consumption on ML hardware. Moreover, we introduce PikeLPN11Pike is a slim fast fish, LPN stands for Low-Precision Network., a model that addresses these efficiency issues by applying quantization to both elementwise operations and multiply-accumulate operations. In particular, we present a novel quantization technique for batch normalization layers named QuantNorm which allows for quantizing the batch normalization parameters without compromising the model performance. Additionally, we propose applying Double Quantization where the quantization scaling parameters are quantized. Furthermore, we recognize and resolve the issue of distribution mismatch in Separable Convolution layers by introducing Distribution-Heterogeneous Quantization which enables quantizing them to low-precision. PikeLPN achieves Pareto-optimality in efficiency-accuracy trade-off with up to 3× efficiency improvement compared to SOTA low-precision models.

We present a common framework for in-loop same-resolution and super-resolution restoration for incorporation into a next-generation video codec. Building on the in-loop filtering pipeline in the AV1 video codec from the Alliance for Open Media (AOM), we first enhance it to support symmetric spatial down-up scaling with better down and upscaling filters, followed by adding switchable frame-level CNNs to restore the reconstructed frames. Furthermore, the architectures for the CNNs used are constrained to be very simple with a relatively small number of parameters and multiply-add operations per decoded pixel to make them practically feasible in hardware. Preliminary results are presented on test sets being used by AOM for their ongoing next-generation video codec (AV2) development effort.

Video sharing (e.g., YouTube, Vimeo, Facebook, TikTok) accounts for the majority of internet traffic, and video processing is also foundational to several other key workloads (video conferencing, virtual/augmented reality, cloud gaming, video in Internet-of-Things devices, etc.). The importance of these workloads motivates larger video processing infrastructures and – with the slowing of Moore’s law – specialized hardware accelerators to deliver more computing at higher efficiencies. This paper describes the design and deployment, at scale, of a new accelerator targeted at warehouse-scale video transcoding. We present our hardware design including a new accelerator building block – the video coding unit (VCU) – and discuss key design trade-offs for balanced systems at data center scale and co-designing accelerators with large-scale distributed software systems. We evaluate these accelerators “in the wild" serving live data center jobs, demonstrating 20-33x improved efficiency over our prior well-tuned non-accelerated baseline. Our design also enables effective adaptation to changing bottlenecks and improved failure management, and new workload capabilities not otherwise possible with prior systems. To the best of our knowledge, this is the first work to discuss video acceleration at scale in large warehouse-scale environments.

This paper describes a video coding technology proposal submitted by Qualcomm in response to a joint call for proposals (CfP) issued by ITU-T SG16 Q.6 (VCEG) and ISO/IEC JTC1/SC29/WG11 (MPEG) in January 2010. The proposed video codec follows a hybrid coding approach based on temporal prediction, followed by transform, quantization, and entropy coding of the residual. Some of its key features are extended block sizes (up to 64$\,\times\,$64), single pass switched interpolation filters with offsets, mode-dependent directional transforms for intra-coding, luma and chroma high precision filtering, geometric motion partitions, adaptive motion vector resolution and efficient 16-point transforms. It also incorporates internal bit-depth increase and modified quadtree-based adaptive loop filtering. Simulation results are presented to demonstrate the high compression efficiency achieved by the proposed video codec at the expense of moderate increase in encoding and decoding complexity compared to the advanced video coding standard (AVC/H.264). For the random access and low delay configurations, it achieved average bit rate reductions of 30.9% and 33.0% for equivalent peak signal-to-noise ratio, respectively, compared to the corresponding AVC anchors. The proposed codec scored highly in both subjective evaluations and objective metrics and was among the best-performing CfP proposals.

In this paper, we apply dynamic voltage scaling (DVS) to the matching metric computation (MMC) used within motion estimation (ME) in typical video encoders. Our approach is based on "soft DSP" concepts. We analyze the effect of ME errors (due to DVS) in overall coding performance. We propose a model for the resulting rate increase (at a given fixed quantization parameter) as a function of input characteristics and input voltage, for given ME algorithm and MMC architecture. This model is validated using simulations. We then compare ME algorithms and MMC architectures, and propose a method for power saving of the ME process that depend on input characteristics and desired coding performance. As an illustration of the potential benefits of allowing computation errors, we show that allowing errors that lead to a small rate increase (about 3%) produces 37% power savings in the ME process, as compared to not using DVS. An essentially "error-free" DVS approach (no rate penalty) can achieve around 10% power savings.

In this paper, we propose power efficient motion estimation (ME) using multiple imprecise sum of absolute difference (SAD) metric computations. We extend the recent work of Varatkar et al (2006) by providing analytical solutions based on modelling of computation errors due to voltage overscaling (VOS) and sub-sampling (SS). Results show that our solutions provide significantly better performance in the sense of rate increase for fixed QP, e.g., less than 5% increase, while in Varatkar et al (2006) the rate increase could be as high as 20%. It allows us to apply lower voltage which leads to additional power saving. Our analysis also allows us to compare different ME algorithms (e.g., full search vs. a fast algorithm) and SAD computation architectures (parallel vs. serial) in terms of their robustness to imprecise metric computations and their power efficiency. Finally, we demonstrate that additional power savings can be achieved by removing redundancy between the various computations.

In this paper we study the computation error tolerance properties of motion estimation algorithms. We are motivated by two scenarios where hardware systems may introduce computation errors. First, we consider hardware faults such as those arising in a typical fabrication process. Second, we consider "soft" errors due to voltage scaling, which can arise when operating at a lower voltage than specified for the system. Current practice is to discard all faulty systems. However there is an increasing interest in tools that can identify faulty systems which provide acceptable performance. We show that motion estimation (ME) algorithms exhibit significant error tolerance in these two scenarios. We propose simple error models and use these to provide insights into what features in these ME algorithms lead to increased error tolerance. Our comparison of the full search ME and a state of the art fast ME approach in the context of H.264/AVC shows that while both techniques are error tolerant, the faster algorithm is in fact more robust to computation errors.