The rapid growth in video consumption has introduced significant challenges to modern streaming architectures. Over-the-Top (OTT) video delivery now predominantly relies on Adaptive Bitrate (ABR) streaming, which dynamically adjusts bitrate and resolution based on client-side constraints such as display capabilities and network bandwidth. This pipeline typically involves downsampling the original high-resolution content, encoding and transmitting it, followed by decoding and upsampling on the client side. Traditionally, these processing stages have been optimized in isolation, leading to suboptimal end-to-end rate-distortion (R-D) performance. The advent of deep learning has spurred interest in jointly optimizing the ABR pipeline using learned resampling methods. However, training such systems end-to-end remains challenging due to the non-differentiable nature of standard video codecs, which obstructs gradient-based optimization. Recent works have addressed this issue using differentiable proxy models, based either on deep neural networks or hybrid coding schemes with differentiable components such as soft quantization, to approximate the codec behavior. While differentiable proxy codecs have enabled progress in compression-aware learning, they remain approximations that may not fully capture the behavior of standard, non-differentiable codecs. To our knowledge, there is no prior evidence demonstrating the inefficiencies of using standard codecs during training. In this work, we introduce a novel framework that enables end-to-end training with real, non-differentiable codecs by leveraging data-driven surrogate gradients derived from actual compression errors. It facilitates the alignment between training objectives and deployment performance. Experimental results show a 5.19\% improvement in BD-BR (PSNR) compared to codec-agnostic training approaches, consistently across the entire rate-distortion convex hull spanning multiple downsampling ratios.

Generative Adversarial Networks (GANs) deliver significant perceptual quality improvements over super-resolution methods using simple reconstruction losses. By combining adversarial training with perceptual loss functions, they avoid the over-smoothing resulting from the convergence towards the average of all plausible solutions. However, the likelihood of mode collapse if generator and discriminator learning rates are not carefully tuned or the training data does not represent the entire solution space, often introduces unpleasant artifacts that can compromise the overall quality of the upscaled images. We propose a new GAN architecture trained on the statistical distribution of a residue resulting from subtracting a spatial interpolation of the low-resolution (LR) image from the high-resolution (HR) ground-truth, instead of directly learning the pixel distribution of the HR image conditioned to the LR input. Since the statistical distribution of this residue is easier to model, the probability of mode collapse and consequent introduction of artifacts are mitigated, resulting in an algorithm competitive with much more computationally complex diffusion or transformer models.

Tone Mapping Operators (TMO) designed for videos can be classified into two categories. In a first approach, TMOs are temporal filtered to reduce temporal artifacts and provide a Standard Dynamic Range (SDR) content with improved temporal consistency. This however does not improve the SDR coding Rate Distortion (RD) performances. A second approach is to design the TMO with the goal of optimizing the SDR coding rate-distortion performances. This second category of methods may lead to SDR videos altering the artistic intent compared with the produced HDR content. In this paper, we combine the benefits of the two approaches by introducing new Weighted Prediction (WP) methods inside the HEVC SDR codec. As a first step, we demonstrate the interest of the WP methods compared to TMO optimized for RD performances. Then we present the newly introduced WP algorithm and WP modes. The WP algorithm consists in performing a global motion compensation between frames using an optical flow, and the new modes are based on non linear functions in contrast with the literature using only linear functions. The contribution of each novelty is studied independently and in a second time they are all put in competition to maximize the RD performances. Tests were made for HDR backward compatible compression but also for SDR compression only. In both cases, the proposed WP methods improve the RD performances while maintaining the SDR temporal coherency.

Optimal adaptive quantization is one of the key points to optimize the coding efficiency of video encoders. The latest block-based video compression standards, such as high-efficiency video coding (HEVC), extensively use predictive coding techniques that create dependencies between blocks and increase the complexity of optimal block quantizers search. Specifically, the motion compensation is responsible for a dependency network connecting all blocks of the same GOP together. In this paper, this dependency network is estimated by a temporal distortion propagation model and an accurate estimation of Inter and Skip modes probabilities. Optimal quantizers are then designed per block in order to achieve global optimization in terms of rate-distortion efficiency. By implementing the algorithm into the HEVC reference model (HM), we report -16.51% PSNR-based and -26.26% SSIM-based average bitrate savings compared to no adaptive quantization. The proposed algorithm outperforms several related methods from the state-of-the-art. Moreover, along with the demonstration of an optimal quantizer solution, we propose an in-depth analysis of the algorithm behavior. This analysis includes, among others, the relative distribution of rates between frames and the control of quantizers dynamic range.

HEVC is the latest block-based video compression standard, outperforming H.264/AVC by 50% bitrate savings for the same perceptual quality. An HEVC encoder provides Rate-Distortion optimization coding tools for block-wise compression. Because of complexity limitations, Rate-Distortion Optimization (RDO) is usually performed independently for each block, assuming coding efficiency losses to be negligible. In this paper, we propose an acceleration solution for the Intra coding scheme named Dual-JRDO, which takes advantage of Inter-Block dependencies related to both predictive coding and CABAC. The Dual-JRDO improves Intra coding efficiency at the expense of higher computational complexity. The acceleration of the Dual-JRDO scheme includes adaptive use of the Dual-JRDO model based on source analysis, short-listing and early decisions strategies. The proposed Fast Dual-JRDO reduces the original model complexity by 89.54%, while providing tractable computation for average R-D gains of -0.45% (up to -0.82%) in the HM16.12 reference software model.

Video data traffic is moving from Broadcast to Internet (e.g. OTTTV) distribution. Related economic models are different. It requires to review the rate control paradigm moving from Constant Bitrate (CBR) model to Variable Bitrate (VBR) model using a capped bitrate constraint. In this paper, we propose a Constant Quality Control (CQC) algorithm which minimizes the bitrate under constraints of a target video quality level and a capped bitrate. CQC algorithm relies on a R-D model which considers the temporal distortion propagation along a video sequence. Introducing a D(Q) model, and a capped bitrate constraint, the optimal local quantizers are computed to fulfill a target level of distortion. Consistency and R-D performances of the proposed solution are verified in the context of HEVC. It meets the target level of quality with an average deviation of 6.9% while reducing bitrate of -7.6% in average compared to constant QP mode. CQC algorithm is particularly well adapted to the OTT-TV distribution, where the Adaptive Bitrate (ABR) mechanism is built on top of quality levels.

Hybrid video coding systems use spatial and temporal predictions in order to remove redundancies within the video source signal. These predictions create coding-scheme-related dependencies, often neglected for sake of simplicity. The R-D Spatio-Temporal Adaptive Quantization (RDSTQ) solution uses such dependencies to achieve better coding efficiency. It models the temporal distortion propagation by estimating the probability of a Coding Unit (CU) to be Inter coded. Uased on this probability, each CU is given a weight depending on its relative importance compared to other CUs. However, the initial approach roughly estimates the Inter probability and does not take into account the Skip mode characteristics in the propagation. It induces important Target uitrate Deviation (TBD) compared to the reference target rate. This paper provides undeniable improvements of the original RDSTQ model in using a more accurate estimation of the Inter probability. Then a new analytical solution for local quantizers is obtained by introducing the Skip probability of a CU into the temporal distortion propagation model. The proposed solution brings -2.05% BD-BR gain in average over the RDSTQ at low rate, which corresponds to -13.54% BD-BR gain in average against no local quantization. Moreover, the TBD is reduced from 38% to 14%.

Recent MPEG video compression standards are still block-based: blocks of pixels are sequentially coded using spatial or temporal prediction schemes. For each block, a vector of coding parameters has to be selected. In order to limit the complexity of this decision, independence between blocks is assumed, and coding parameters are locally optimized to maximize the coding efficiency. Few studies have investigated the benefits of inter-block dependencies consideration using Joint Rate-Distortion Optimization (JRDO), especially in Intra coding. To the best of our knowledge, maximum achievable gains of such approaches have never been exhibited. In this paper, we propose two JRDO models performing joint optimization of multiple blocks applied to intra prediction mode decision. The proposed models have been evaluated in both H.264/AVC and HEVC standards. These two models enables a bitrate saving with respect to the classical RDO model up to -3.10% and -2.31% in H.264/AVC and HEVC, respectively.

Nowadays, Rate-Distortion Optimization (RDO) is commonly used in hybrid video coding to maximize coding efficiency. Usually, the rate distortion tradeoff is explicitly computed in offline encoder implementations whereas R(D) model are used in live encoders to select the best decisions at a lower computational cost. For sake of simplicity, this (mathematical) modelling is often performed for each coding unit (CU) individually and independently, obliterating the spatial or temporal dependency between CUs. In this paper, we provide a new spatio-temporal algorithm to compute local quantizers, based on a theoretical framework able to describe the temporal distortion propagation from an R-D standpoint. In particular, we model the temporal distortion propagation making possible the retro accumulations of any (spatial) psycho-visually weighted distortion onto reference images. Using the R(D) Shannon bound, its high bitrate approximation, and a Lagrange optimization, analytical solutions are obtained for the local quantizers and the Lagrange multiplier. The proposed algorithm shows -4.4% BD-BR SSIM gains in average over state-of-the art algorithm in HEVC, using the same SSIM-based psycho-visual function.

This paper addresses the problem of designing a global tone mapping operator for rate distortion optimized backward compatible compression of high dynamic range (HDR) images. We address the problem of tone mapping design for two different use cases leading to two different minimization problems. The first problem considered is the minimization of the distortion on the reconstructed HDR signal under a rate constraint on the standard dynamic range (SDR) layer. The second problem remains the same minimization with an additional constraint to preserve a good quality for the SDR signal. Both the distortion and rate are expressed as a function of the spatial gradient in HDR images. Experiments show that the proposed rate and distortion models based on the HDR image gradient accurately predict the real image rate and distortion measures. Experimental results show that for the first minimization, the optimal rate-distortion performances are achieved, and that the second optimization yields the best tradeoff between rate-distortion performance and quality preservation of the SDR signal.

This paper addresses the problem of designing a global tone mapping operator for rate-distortion optimized backward compatible compression of HDR images. We consider a two layer coding scheme in which a base SDR layer is coded with HEVC, inverse tone mapped and subtracted from the input HDR signal to yield the enhancement HDR layer. The tone mapping curve design is formulated as the minimization of the distortion on the reconstructed HDR signal under the constraint of a total rate cost on both layers, while preserving a good quality for the SDR signal. We first demonstrate that the optimum tone mapping function only depends on the rate of the base SDR layer and that the minimization problem can be separated in two consecutive minimization steps. Experimental results show that the proposed tone mapping optimization yields the best trade-off between rate-distortion performance and quality preservation of the coded SDR.

Since the finalization of the H.264/AVC standard and in order to meet the target set by both ITU-T and MPEG to define a new standard that reaches 50% bit rate reduction compared to H.264/AVC, many tools have efficiently improved the texture coding and the motion compensation accuracy. These improvements have resulted in increasing the proportion of bit rate allocated to motion information. Thus, the bit rate reduction of this information becomes a key subject of research. This paper proposes a method for motion vector coding based on an adaptive redistribution of motion vector residuals before entropy coding. Motion information is gathered to forecast a list of motion vector residuals which are redistributed to unexpected residuals of lower coding cost. Compared to H.264/AVC, this scheme provides systematic gain on tested sequences, and 2.3% in average, reaching up to 4.9% for a given sequence.