459

This work investigates video encoding parallelization techniques based on the Versatile Video Coding (VVC) standard, using the open and optimized encoder software implementation VVenC. Modern multi-processor systems offer significant opportunities for accelerating video encoding. By employing a proposed combination of parallelization methods, the VVenC encoder achieves an acceleration factor of up to 22 compared to single-threaded mode on a 32-core system, with potential increases to 27× at higher bitrates. Building upon prior work on Inter-frame Parallelization (IFP), the study introduces frame region-based synchronization, enabling further acceleration of up to 10%. Beyond that, the study demonstrates extending frame parallelization beyond Group of Pictures (GOP) boundaries, which improves IFP speed up by 37% and 11% at high-definition (HD) and ultra-high-definition (UHD) resolutions, respectively. Additional combinations with other VVC parallelization tools, such as tiles and VVC Wavefront Parallel Processing (WPP), are also explored. The article provides a comprehensive analysis of parallelization challenges and highlights areas for further improvement.

In telecommunications, Autonomous Networks (ANs) automatically adjust configurations based on specific requirements (e.g., bandwidth) and available resources. These networks rely on continuous monitoring and intelligent mechanisms for self-optimization, self-repair, and self-protection, nowa-days enhanced by Neural Networks (NNs) to enable predictive modeling and pattern recognition.In this work, we propose a neural architecture called Mixed-Token Transformer (MTT) that can process and predict both numerical and textual tokens representing various mobile network features (e.g., ping, SNR or band frequency). We train the models using a Federated Learning (FL) approach, enabling multiple AN cells — each equipped with an MTT — to collaboratively learn from cellular data while preserving data privacy.Since FL involves frequent transmission of neural network updates, it requires an efficient, standardized compression strategy for reliable communication. To address this, we investigate NNCodec, our implementation of the ISO/IEC Neural Network Coding (NNC) standard. Our experimental results on the Berlin V2X dataset demonstrate that NNCodec achieves transparent compression (i.e., negligible performance loss) while reducing communication overhead to below 1%, showing the effectiveness of combining NNC with FL in collaboratively learned autonomous mobile networks. Our proposed MTT architecture speeds up model inference by ≈ 5×. Code is available at: github.com/Jostarndt/NNCodec-FL-MixedTokenTransformer.

With the emergence of the Versatile Video Coding standard (VVC), novel coding tools like matrix-based intra prediction and low-frequency non-separable transforms have been developed based on data-driven optimization methods. Motivated by the growing compression efficiency of learned nonlinear transforms in image coding, we incorporate a neural-network-based coding tool into the transform coding stage of VVC. First, a nonlinear update of the transform coefficients is applied which aims at decreasing the expected bitrate cost. Then, on the decoder side, a filter is applied before the synthesis transform to improve the reconstruction quality. These networks are jointly trained and have been added to the rate-distortion optimized quantization process. The integration of these approaches into the VTM software leads to bitrate savings between 0.94% and 2.12% in terms of the Bjøntegaard-Delta rate. We furthermore conducted multiple experiments on reducing the memory complexity of the networks by exploiting structural similarities of the intra prediction modes and block symmetries. As a result, we demonstrate that the number of distinct networks can be reduced without significantly diminishing the coding gain of the tool. With memory usage reduced by 90%, we achieve bitrate savings between 1.07% and 1.94%.

Machine learning-based intra-prediction methods for video coding have gained profound attention in recent years. However, most of such approaches use complex non-linear networks that increase the coding complexity significantly. In this paper, a data-driven recursive intra-prediction (DRIP) method is proposed which applies a linear network at the sub-block level for intra-prediction. Briefly, a given block is partitioned into sub-blocks, and by multiplying the reconstructed or predicted samples present at the top and left of a sub-block with a weight matrix (learned offline) its prediction is obtained. The sub-blocks are predicted recursively, using the same weight matrix, until the prediction of the block is complete. For further coding improvement, multiple such matrices are used resulting in multiple prediction modes for a block. The index of the chosen mode is transmitted to the decoder in the bit-stream. An example implementation of DRIP with 32 modes and 4 × 4 sub-blocks on top of the VVC Test Model (VTM version 23.6) is presented in this paper. It achieves an average Bjøntegaard-delta rate of 1.02% (luma channel) against the anchor using the all-intra− (AI) picture configuration.

Versatile Video Coding (VVC) offers substantial improvements in compression efficiency compared to its predecessor, High Efficiency Video Coding (HEVC). However, these gains come at the cost of significantly increased encoding complexity, particularly due to VVC's quadtree with nested multi-type tree (QTMT) partitioning structure. This complexity poses challenges for real-time and low-latency applications. This paper presents an enhanced split cost prediction framework designed to accelerate the VVC encoding process while preserving compression efficiency. Building on prior split cost prediction methods, our approach refines context modeling by incorporating Coding Unit (CU) size, prediction mode, and partitioning depth features. We construct a large-scale training dataset and optimize context-dependent predictors for multiple encoder presets. The proposed method is implemented in both Fraunhofer Versatile Video Encoder (VVenC) and the VVC Test Model (VTM), demonstrating encoding time reductions of up to 6% with negligible compression efficiency loss compared to the anchor using the benchmark method. Performance improvements are most significant in the medium and slower presets, where the partitioning search space is larger. Our results demonstrate that split cost prediction can be further optimized through feature-driven context modeling, eliminating the need for complex machine learning models, thereby making it practical for deployment in existing encoder pipelines.

This paper introduces a method for digitally signing and verifying elementary video bitstreams. The method verifies temporal consistency of the video while allowing random access into the bitstream and adaptation to temporal and spatial scalability including sub-bitstream extraction. It was adopted by the Joint Video Experts Team (JVET) into the Versatile supplemental enhancement information messages for coded video bitstreams (VSEI) specification version 4 by introducing three new Digitally Signed Content SEI messages.

The continuous improvements on image compression with variational autoencoders have lead to learned codecs competitive with conventional approaches in terms of rate-distortion efficiency. Nonetheless, taking the quantization into account during the training process remains a problem, since it produces zero derivatives almost everywhere and needs to be replaced with a differentiable approximation which allows end-to-end optimization. Though there are different methods for approximating the quantization, none of them model the quantization noise correctly and thus, result in suboptimal networks. Hence, we propose an additional finetuning training step: After conventional end-to-end training, parts of the network are retrained on quantized latents obtained at the inference stage. For entropy-constraint quantizers like Trellis-Coded Quantization, the impact of the quantizer is particularly difficult to approximate by rounding or adding noise as the quantized latents are interdependently chosen through a trellis search based on both the entropy model and a distortion measure. We show that retraining on correctly quantized data consistently yields additional coding gain for both uniform scalar and especially for entropy-constraint quantization, without increasing inference complexity. For the Kodak test set, we obtain average savings between $1 \%$ and $2 \%$, and for the TecNick test set up to $2.2 \%$ in terms of Bjøntegaard-Delta bitrate.

VVenC, an open Versatile Video Coding (VVC) encoder, has recently been equipped with rate capping functionality in its two-pass rate control modes, providing constrained variable bitrate coding governed by target rate and maximum rate parameters. This paper reports on implementations and evaluation results of straightforward extensions to VVenC which enable the use of the maximum rate parameter also in the single-pass fixed-QP modes, controlled by a base quantization parameter (QP) instead of a target rate. The rate capping in the fixed-QP mode is achieved, with sufficient accuracy, by evaluating only already calculated pre-processing statistics, thereby avoiding increases in encoder runtime. This encoding mode, given that it supports visual quality optimizations such as XPSNR based block-wise perceptual QP adaptation, can be considered a rate capped constant-quality mode, which was missing in VVenC and which is an interesting configuration for video streaming.

As adaptive streaming becomes crucial for delivering high-quality video content across diverse network conditions, accurate metrics to assess perceptual quality are essential. This paper explores using the eXtended Peak Signal-to-Noise Ratio (XPSNR) metric as an alternative to the popular Video Multimethod Assessment Fusion (VMAF) metric for determining optimized bitrate-resolution pairs in the context of Versatile Video Coding (VVC). Our study is rooted in the observation that XPSNR shows a superior correlation with subjective quality scores for VVC-coded Ultra-High Definition (UHD) content compared to VMAF. We predict the average XPSNR of VVC-coded bitstreams using spatiotemporal complexity features of the video and the target encoding configuration and then determine the convex-hull online. On average, the proposed convex-hull using XPSNR (VEXUS) achieves an overall quality improvement of 5.84 dB PSNR and 0.62 dB XPSNR while maintaining the same bitrate, compared to the default UHD encoding using the VVenC encoder, accompanied by an encoding time reduction of 44.43% and a decoding time reduction of 65.46%. This shift towards XPSNR as a guiding metric shall enhance the effectiveness of adaptive streaming algorithms, ensuring an optimal balance between bitrate efficiency and perceptual fidelity with advanced video coding standards.

The Versatile Video Coding (VVC) standard promises high compression efficiency for diverse content types. Based on VVenC, an open and optimized VVC software video encoder, this work presents an inter-frame parallelization (IFP) method designed to exploit the processing power of modern platforms featuring a high number of computing cores. Encoding an ultrahigh definition video on a 32-core machine with the VVenC's faster preset, the proposed method shows more than 20% increase in encoder speed while only a 1% decrease in compression efficiency compared to the default multi-threading mode. In comparison to single-threaded mode, it corresponds to a speedup factor of 18, up from 15x achievable with the previous parallelization scheme. Furthermore, the synergy of the developed inter-frame parallelization technique with other parallelization methods is explored, including tiles and VVC wavefront parallel processing (WPP). The combination of these approaches enables a notable speedup factor of 21, albeit with a trade-off in coding efficiency. With a focus on VVC, this research contributes to the ongoing discourse on video coding optimization, providing valuable insights into possible pitfalls and the potential gains achievable through efficient parallelization techniques on high-core platforms.

Traditional per-title encoding schemes aim to optimize encoding resolutions to deliver the highest perceptual quality for each representation. XPSNR is observed to correlate better with the subjective quality of VVC-coded bitstreams. Towards this realization, we predict the average XPSNR of VVC-coded bitstreams using spatiotemporal complexity features of the video and the target encoding configuration using an XGBoost-based model. Based on the predicted XPSNR scores, we introduce a Quality-Aware Dynamic Resolution Adaptation (QADRA) framework for adaptive video streaming applications, where we determine the convex-hull online. Furthermore, keeping the encoding and decoding times within an acceptable threshold is mandatory for smooth and energy-efficient streaming. Hence, QADRA determines the encoding resolution and quantization parameter (QP) for each target bitrate by maximizing XPSNR while constraining the maximum encoding and/ or decoding time below a threshold. QADRA implements a JND-based representation elimination algorithm to remove perceptually redundant representations from the bitrate ladder. QADRA is an open-source Python-based framework published under the GNU GPLv3 license.

Modern hybrid video codecs like Versatile Video Coding (VVC) heavily rely on transform coding tools. Given a prediction signal at the encoder, the residual is transformed using trigonometric transforms. Rate-distortion-optimized quantization (RDOQ) and entropy coding of the transformed residual is well-understood due to the orthogonality and the energy compaction of these transforms. Within this setting, there is considerable success in optimizing secondary orthogonal transforms. The most prominent example is the Low-Frequency Non-Separable Transform (LFNST) in VVC. However, training nonlinear transforms without re-designing the RDOQ and entropy coding stage is a hard problem. In learned image compression, variational autoencoders have shown impressive results, but they use their own entropy model, remain difficult to train for small blocks and RDOQ is nontrivial for them. This paper describes a novel design of a nonlinear transform network for block-based video coding. Given a transform block, a fully-connected neural network predict coefficients from previously reconstructed ones and the adherent block boundary, such that only the residual coefficients need to be transmitted. Furthermore, another neural network filters the entire transform block before the inverse transform is applied and the intra prediction signal is added. Against the Versatile Video Coding Test Model 14.2 (VTM-14.2), luma bit-rate savings of approximately 1.9 % are reported for the All-Intra configuration.

Rate control (RC), specifically two-pass, is the main operation mode in VVenC, an open and optimized Versatile Video Coding (VVC) encoder. VVC offers substantial bitrate savings over its predecessor, High Efficiency Video Coding (HEVC), at the price of increased complexity. This complexity increase is apparent in both encoding passes of VVenC. While the complexity redaction in the final pass has been discussed, this paper considers complexity reduction in the first pass, in addition to its already reduced search space. To reduce the overall runtime of a two-pass RC method, spatial and temporal sub-sampling of the first encoding pass is proposed. The experimental results show that the proposed first-pass sub-sampling in two-pass RC can speed up the encoding process of the default two-pass rate control algorithm in VVenC by 18%, with 0.48% loss in coding efficiency, when using the faster preset. Using temporal sub-sampling for the look-ahead, one-pass RC in VVenC can achieve time savings of 11% for bit-rate increases of 0.28%.

In this paper, we present a novel in-loop filter for video coding which is based on a convolutional neural network (CNN). For that, the adaptive loop filter (ALF) of Versatile Video Coding (VVC) is generalized to define the model architecture for a CNN-based in-loop filter which requires significantly lower computational complexity compared to other existing CNN-based in-loop filters. Experimental results show that, under the JVET common test conditions, BD-rate savings of 1.84% and 1.83% can be achieved compared to VVC for the all-intra and random-access configurations, respectively. At the same time, the computational complexity is at only about 455 multiplications per luma sample.

Conventional per-title encoding schemes strive to optimize encoding resolutions to deliver the utmost perceptual quality for each bitrate ladder representation. Nevertheless, maintaining encoding time within an acceptable threshold is equally imperative in online streaming applications. Further-more, modern client devices are equipped with the capability for fast deep-learning-based video super-resolution (VSR) techniques, enhancing the perceptual quality of the decoded bitstream. This suggests that opting for lower resolutions in representations during the encoding process can curtail the overall energy consumption without substantially compromising perceptual quality. In this context, this paper introduces a video super-resolution-based latency-aware optimized bitrate encoding scheme (ViSOR) designed for online adaptive streaming applications. ViSOR determines the encoding resolution for each target bitrate, ensuring the highest achievable perceptual quality after VSR within the bound of a maximum acceptable latency. Random forest-based prediction models are trained to predict the perceptual quality after VSR and the encoding time for each resolution using the spatiotemporal features extracted for each video segment. Experimental results show that ViSOR targeting fast super-resolution convolutional neural network (FSRCNN) achieves an overall average bitrate reduction of 24.65% and 32.70% to maintain the same PSNR and VMAF, compared to the HTTP Live Streaming (HLS) bitrate ladder encoding of 4s segments using the x265 encoder, when the maximum acceptable latency for each representation is set as two seconds. Considering a just noticeable difference (JND) of six VMAF points, the average cumulative storage consumption and encoding energy for each segment is reduced by 79.32% and 68.21%, respectively, contributing towards greener streaming.

In the last years, deep video coding has attracted a lot of research interest. Usually, it employs the concept of inter coding by transmitting features in a latent space that represent a motion field or a residual. However, in such a setting there are still redundancies between the features of consecutive frames. In previous approaches, these redundancies are exploited for compression by adding an additional input at the encoder and decoder. However, this often comes at the cost of changing the whole network architecture. In this paper, we present a conditional coding for motion features which utilizes already transmitted features for coding the features of the current picture in a more effective way. This concept can be applied on top of any existing coding framework. Our coding experiments, which were conducted for JVET test sequences, demonstrate that the proposed conditional motion feature coding can yield bit-rate savings of up to 9% relative to an independent coding of the motion features for individual pictures.

Pareto-front optimization is crucial for addressing the multi-objective challenges in video streaming, enabling the identification of optimal trade-offs between conflicting goals such as bitrate, video quality, and decoding complexity. This paper explores the construction of efficient bitrate ladders for adaptive Versatile Video Coding (VVC) streaming, focusing on optimizing these trade-offs. We investigate various ladder construction methods based on Pareto-front optimization, including exhaustive Rate-Quality and fixed ladder approaches. We propose a joint decoding time-rate-quality Pareto-front, providing a comprehensive framework to balance bitrate, decoding time, and video quality in video streaming. This allows streaming services to tailor their encoding strategies to meet specific requirements, prioritizing low decoding latency, bandwidth efficiency, or a balanced approach, thus enhancing the overall user experience. The experimental results confirm and demonstrate these opportunities for navigating the decoding time-rate-quality space to support various use cases. For example, when prioritizing low decoding latency, the proposed method achieves a decoding time reduction of 14.86 % while providing Bjøntegaard delta rate savings of 4.65 % and 0.32 dB improvement in the eXtended Peak Signal-to-Noise Ratio (XPSNR)-Rate domain over the traditional fixed ladder solution.

This paper presents a convolutional neural network (CNN)-based enhancement to inter prediction in Versatile Video Coding (VVC). Our approach aims at improving the prediction signal of inter blocks with a residual CNN that incorporates spatial and temporal reference samples. It is motivated by the theoretical consideration that neural network-based methods have a higher degree of signal adaptivity than conventional signal processing methods and that spatially neighboring reference samples have the potential to improve the prediction signal by adapting it to the reconstructed signal in its immediate vicinity. We show that adding a polyphase decomposition stage to the CNN results in a significantly better trade-off between computational complexity and coding performance. Incorporating spatial reference samples in the inter prediction process is challenging: The fact that the input of the CNN for one block may depend on the output of the CNN for preceding blocks prohibits parallel processing. We solve this by introducing a novel signal plane that contains specifically constrained reference samples, enabling parallel decoding while maintaining a high compression efficiency. Overall, experimental results show average bit rate savings of 4.07% and 3.47% for the random access (RA) and low-delay B (LB) configurations of the JVET common test conditions, respectively.

Distributed learning requires a frequent communication of neural network update data. For this, we present a set of new compression tools, jointly called differential neural network coding (dNNC). dNNC is specifically tailored to efficiently code incremental neural network updates and includes tools for federated BatchNorm folding (FedBNF), structured and unstructured sparsification, tensor row skipping, quantization optimization and temporal adaptation for improved context-adaptive binary arithmetic coding (CABAC). Furthermore, dNNC provides a new parameter update tree (PUT) mechanism, which allows to identify updates for different neural network parameter sub-sets and their relationship in synchronous and asynchronous neural network communication scenarios. Most of these tools have been included into the standardization process of the NNC standard (ISO/IEC 15938-17) edition 2. We benchmark dNNC in multiple federated and split learning scenarios using a variety of NN models and data including vision transformers and large-scale ImageNet experiments: It achieves compression efficiencies of 60% in comparison to the NNC standard edition 1 for transparent coding cases, i.e., without degrading the inference or training performance. This corresponds to a reduction in the size of the NN updates to less than 1% of their original size. Moreover, dNNC reduces the overall energy consumption required for communication in federated learning systems by up to 94%.

Data-driven optimization is employed to study alternative approaches [1] to the probability estimator of the the Enhanced Compression Model (ECM) (which includes additional coding tools on top of the Versatile Video Coding standard). In ECM, each context model uses a weighted sum of two hypotheses for probability estimation with different associated adaptation rates. Four alternative approaches are studied:

In August 2022, ISO/IEC MPEG published the first international standard on compression of neural networks, namely Neural Network Coding (NNC, MPEG-7 part 17). It compresses neural networks to about 5% to 15% in size at virtually no performance loss. In NNC, the model weights are usually quantized and then encoded into the bitstream using DeepCABAC entropy coding. In order to improve the coding efficiency, this paper presents new training strategies for optimized model weights considering the quantization and entropy coding process of NNC, by making the training process bitrate- and quantization-aware. With this bitrate-performance optimized training the bitrate can be further reduced by more than 25% on average for state-of-the-art image classification models.

For improved performance, practical video encoders integrate algorithms for screen content detection and rate control. This paper outlines recently implemented optimizations to both the screen content classifier (SCC) and two-pass rate control (RC) of VVenC, an open Versatile Video Coding (VVC) compliant encoder. The improvements, confirmed by evaluation experiments in random-access configurations using an extended set of test videos, are mainly achieved by leveraging motion error statistics acquired during motion compensated temporal pre-filtering (MCTPF), carried out in VVenC’s pre-analysis stage. All three aspects – pre-analysis stage, SCC, and RC – are revisited herein, and the exploitation of MCTPF data is described.

Versatile Video Coding (VVC) allows for large compression efficiency gains over its predecessor, High Efficiency Video Coding (HEVC). The added efficiency comes at the cost of increased runtime complexity, especially for encoding. It is thus highly relevant to explore all available runtime reduction options. This paper proposes a novel first pass for two-pass rate control in all-intra configuration, using low-complexity video analysis and a Random Forest (RF)-based machine learning model to derive the data required for driving the second pass. The proposed method is validated using VVenC, an open and optimized VVC encoder. Compared to the default two-pass rate control algorithm in VVenC, the proposed method achieves around 32% reduction in encoding time for the preset faster, while on average only causing 2% BD-rate increase and achieving similar rate control accuracy.

The research on deep-learned end-to-end video compression has attracted a lot of attention over the course of recent years. A central component of many approaches is to perform motion-compensated prediction by using convolutional neural networks (CNN) which determine a compressed representation of the motion field as features. Often, this task is divided into searching motion vectors by one network and efficiently representing them by another one. However, these networks may find motion fields far from optimal because the search radius of CNNs is mainly determined by their depth and kernel size. In this paper, we apply motion estimation techniques from classical block-based hybrid video compression to search a motion field which is then fed into a variational autoencoder. These strategies include different distortion measures, different block partitions and an improved approximation of the residual bitrate. With our modifications, bitrate savings of up to 13% over the underlying end-to-end based video codec can be obtained.

Rate control (RC) schemes allow audio and video encoders to produce bitstreams according to specific overall bitrate constraints. However, when no rate capping is enforced, the instantaneous bitrate may vary strongly and may exceed the target rate by an order of magnitude, potentially causing playback stutter especially in video streaming scenarios. This paper introduces a rate capping extension for the two RC modes in VVenC, an open Versatile Video Coding (VVC) compliant encoder implementation. After a revisit of VVenC’s two-pass RC approach, the algorithmic details of the rate capping model are described. The paper concludes with an objective evaluation of the performance of the RC extension in a random-access configuration.

Copy prediction is a renowned category of prediction techniques in video coding where the current block is predicted by copying the samples from a similar block that is present somewhere in the already decoded stream of samples. Motion-compensated prediction, intra block copy, template matching prediction etc. are examples. While the displacement information of the similar block is transmitted to the decoder in the bit-stream in the first two approaches, it is derived at the decoder in the last one by repeating the same search algorithm which was carried out at the encoder. Region-based template matching is a recently developed prediction algorithm that is an advanced form of standard template matching. In this method, the reference area is partitioned into multiple regions and the region to be searched for the similar block(s) is conveyed to the decoder in the bit-stream. Further, its final prediction signal is a linear combination of already decoded similar blocks from the given region. It was demonstrated in previous publications that region-based template matching is capable of achieving coding efficiency improvements for intra as well as inter-picture coding with considerably less decoder complexity than conventional template matching. In this paper, a theoretical justification for region-based template matching prediction subject to experimental data is presented. Additionally, the test results of the aforementioned method on the latest H.266/Versatile Video Coding (VVC) test model (version VTM-14.0) yield an average Bjøntegaard-Delta (BD) bit-rate savings of $-0.75\%$ using all intra (AI) configuration with 130% encoder run-time and 104% decoder run-time for a particular parameter selection.

The optimization of the energy demand is crucial for modern video codecs. Previous studies show that the energy demand of VVC decoders can be improved by more than 50% if specific coding tools are disabled in the encoder. However, those approaches increase the bit rate by over 20% if the concept is applied to practical encoder implementations such as VVenC. Therefore, in this work, we investigate VVenC and study possibilities to reduce the additional bit rate, while still achieving low-energy decoding at reasonable encoding times. We show that encoding using our proposed coding tool profiles, the decoding energy efficiency is improved by over 25% with a bit rate increase of less than 5% with respect to standard encoding. Furthermore, we propose a second coding tool profile targeting maximum energy savings, which achieves 34% of energy savings at bitrate increases below 15%.

In this paper, a data-driven generalization of the adaptive loop filter (ALF) of Versatile Video Coding (VVC) is presented. It is shown how the conventional ALF process of classification and FIR filtering can be generalized to define a natural model architecture for convolutional neural network (CNN) based in-loop filters. Experimental results show that over VVC, average bit-rate savings of 3.85%/4.75% and 4.39%/4.33% can be achieved for the all intra and random access configurations in the low- and high-QP settings.

This paper presents a convolutional neural network (CNN) based solution for inter prediction in Versatile Video Coding (VVC). Our approach aims at improving the motion-compensated prediction signal of inter blocks with a residual CNN that incorporates spatial and temporal reference samples, i.e., intra-inter prediction. It is motivated by two considerations. Firstly, incorporating intra reference samples has the potential to improve the prediction signal by adapting it to the reconstructed signal in its immediate vicinity. Secondly, neural network-based methods offer a higher degree of signal adaptivity than conventional signal processing methods. The proposed network is integrated as a post-processing module for inter predicted blocks in VVC. Experimental results show that average bit rate savings of 3.2% to 4.0% can be achieved for the random access configuration.

The performance of variational auto-encoders (VAE) for image compression has steadily grown in recent years, thus becoming competitive with advanced visual data compression technologies. These neural networks transform the source image into a latent space with a channel-wise representation. In most works, the latents are scalar quantized before being entropy coded. On the other hand, vector quantizers generally achieve denser packings of high-dimensional data regardless of the source distribution. Hence, low-complexity variants of these quantizers are implemented in the compression standards JPEG 2000 and Versatile Video Coding. In this paper we demonstrate coding gains by using trellis-coded quantization (TCQ) over scalar quantization. For the optimization of the networks with regard to TCQ, we employ a specific noisy representation of the features during the training stage. For variable-rate VAEs, we obtained 7.7% average BD-rate savings on the Kodak images by using TCQ over scalar quantization. When different networks per target bitrate are optimized, we report a relative coding gain of 2.4% due to TCQ.

This paper presents an improved probability estimation scheme for the entropy coder of Incremental Neural Network Coding (INNC), which is currently under standardization in ISO/IEC MPEG. More specifically, the paper first analyzes the compression performance of INNC and how the bitstream size relates to the neural network (NN) layers. For the layers requiring the most bits, it analyzes the coded NN weight updates and their temporal dependencies. Major finding is that the probability of a significant (i.e., non-zero) update for a weight can depend considerably on whether the weight has been updated before. Based on this finding, the paper proposes a new probability estimation scheme: Depending on whether a significant update has been received before (i.e., based on the weight’s history), the entropy coder models the probability for a current significant update differently. This scheme achieves a bitstream size reduction of about 2% and 1% in a transfer and a federated learning scenario, respectively, without any accuracy loss or significant complexity increase. Therefore, MPEG adopted our history dependent significance probability (HDSP) scheme to its emerging standard for INNC.

The Versatile Video Coding (VVC) standard has been developed to meet the ever-increasing demand for higher compression of digital video data. Compared to its predecessor, the High-Efficiency Video Coding (HEVC) standard, VVC reduces the bitrate by around 50% for the same perceived quality. This increase in compression efficiency is associated with an increase in computational complexity, mainly on the encoder side. As an open and optimized VVC software encoder implementation, VVenC integrates algorithmic optimizations for each coding tool in VVC. This allows to define a set of five presets from faster to slower as Pareto-optimal tradeoffs between runtime and efficiency. On top, multithreading allows to reduce the runtime and preserves most of the compression efficiency of each preset. This paper presents and analyses the different multi-threading strategies in VVenC. Using a combination of pre-processing, picture-level and in-picture parallelization, VVenC can achieve a parallelization speedup with a factor of 4 for 4 threads while reducing the compression efficiency by only 0.4%. For higher thread numbers, i.e. 16, the speedup depends on the video resolution and used encoder preset, ranging from 6-9 for high definition to 10-12 for ultrahigh definition video with similar loss of compression efficiency. Using additional wavefront and tiles in-picture parallelization, higher speedups can be achieved at the costs of decreased coding efficiency.

Contemporary motion picture content, consisting of scenes with different amounts of visual complexity or camera noise, represents demanding input for video encoders operating in rate control (RC) modes. This paper presents improvements to the 2-pass RC method integrated into VVenC, an open VVC encoder implementation, outlined in previous publications. We specifically introduce three extensions to our RC solution: first, frame type adaptation operating near scene cuts, along with an associated simple detector; second, rate stabilization means to allow for more reliable lookahead based 2-pass RC operation in on-the-fly encoding applications; and third, a low-complexity approach for estimating the instantaneous intensity of camera noise or film grain to avoid large variations in bit consumption when encoding individual frames in the final RC pass. Experimental evaluation confirms that these extensions significantly improve both the objective (BD rate) and subjective (visual) RC performance of VVenC especially on challenging video content.

Recently, convolutional neural network (CNN)-based in-loop filters have been introduced for video coding and they show huge coding gains. However, one of the main issues of this approach is the high computational complexity of these filters. In this paper, we present various settings for CNN-based in-loop filters targeting on the reduction of their decoder complexity and describe the corresponding gain-complexity trade-offs. For this, we introduce an effective complexity measure and show that it is possible to notably reduce this value for some CNN-based in-loop filters while keeping the compression gain over Versatile Video Coding (VVC).

Variational autoencoders have shown promising results for still image compression and have gained a lot of consideration in this field. Recently, noteworthy attempts were made to extend such end-to-end methods to the setting of video compression. Here, low-latency scenarios have been commonly investigated. In this paper, it is shown that the compression efficiency in this setting is improved by applying tools that are typically used in block-based hybrid coding such as rate-distortion optimized encoding of the features and advanced interpolation filters for computing samples at fractional positions. Additionally, a separate motion estimation network is trained to further increase the compression efficiency. Experimental results show that the rate-distortion performance benefits from including the aforementioned tools.

VVenC, an open and optimized VVC encoder implementation, employs a temporal filter from the literature as a pre-processing step. The filter effectively reduces camera noise from input video, thereby increasing the encoding gain for lossy encoding, at a price of fairly high complexity, further increased by the necessity of consistent application to many pictures. The filter represents one of the most runtime consuming processing steps for the fastest operating points of VVenC. In this paper, steps are described to reduce the complexity of the temporal filtering in VVenC, to allow its application with low-complexity presets. Overall, the filter runtime is reduced by a factor of around 17 compared to the state of the art, while slightly improving its performance. An additional 4 times speedup is achieved using vectorized implementation. In the proposed version, for the VVenC preset faster, the filter provides 7.36% BD-rate gain at only 2% runtime overhead.

This paper presents a CNN to reduce the encoding time of a VVC-based intra-picture encoder. For encoding a 32 × 32 block, the CNN estimates two partitioning parameters that restrict the allowed coding block width and height. To estimate them such that the encoder skips testing inefficient partitioning modes, we train the CNN as follows: First, we generate training data by encoding sequences without the CNN. While encoding, we test all combinations of the two parameters for each 32 × 32 block and store the resulting Lagrangian rate-distortion-time (RDT) cost. We use the recorded cost to derive the loss function when training the CNN. Consequently, the CNN is trained such that it minimizes the Lagrangian RDT cost. Our CNN reduces the encoding time by 50% with a bit rate increase of 0.9%, which outperforms existing CNN-based approaches. Our generic training approach could also be applied for other encoder parameters.

Given the capabilities of massive GPU hardware, there has been a surge of using artificial neural networks (ANN) for still image compression. These compression systems usually consist of convolutional layers and can be considered as non-linear transform coding. Notably, these ANNs are based on an end-to-end approach where the encoder determines a compressed version of the image as features. In contrast to this, existing image and video codecs employ a block-based architecture with signal-dependent encoder optimizations. A basic requirement for designing such optimizations is estimating the impact of the quantization error on the resulting bitrate and distortion. As for non-linear, multi-layered neural networks, this is a difficult problem. This paper presents a performant auto-encoder architecture for still image compression, which represents the compressed features at multiple scales. Then, we demonstrate how an algorithm, which tests multiple feature candidates, can reduce the Lagrangian cost and optimize compression efficiency. The algorithm avoids multiple network executions by pre-estimating the impact of the quantization on the distortion by a higher-order polynomial.

The novel Neural Network Compression and Representation Standard (NNR), recently issued by ISO/IEC MPEG, achieves very high coding gains, compressing neural networks to 5% in size without accuracy loss. The underlying NNR encoder technology includes parameter quantization, followed by efficient arithmetic coding, namely DeepCABAC. In addition, NNR also allows very flexible adaptations, such as signaling specific local scaling values, setting quantization parameters per tensor rather than per network and supporting specific parameter fusion operations. This paper presents our new approach for optimally deriving these parameters, namely the derivation of parameters for local scaling adaptation (LSA), inference-optimized quantization (IOQ), and batch-norm folding (BNF). By allowing inference and fine tuning within the encoding process, quantization errors are reduced and the NNR coding efficiency is further improved to a compressed bitstream size of only 3% in comparison to the original model size.

This paper presents two new methods for fast VVC intra-picture encoding. Both are based on an approach that uses a CNN for blockadaptive parameter estimation. The parameters restrict the multitype-tree (MTT) partitionings tested by the encoder. The methods aim for an improvement of the approach by further constraints with additional parameters. Adding parameters increases the time required for training data generation exponentially. This raises the question which parameters to add and how. To explore further partitioning restrictions, the first method adds parameters controlling the block sizes the MTT can start from. Although this leads to four parameters, we can exploit that some of their combinations are invalid. To investigate whether testing fewer prediction and transform modes is feasible, the second method adds a single parameter that restricts their number jointly. The paper evaluates hypothetical and actual encoding time reductions for VTM-10.2. The first method outperforms our other and other existing method: The encoding time decreases by 50% with a bit rate increase of 0.7%.

Versatile Video Coding (VVC) is the most recent international video coding standard jointly developed by ITU-T and ISO/IEC, which has been finalized in July 2020. VVC allows for significant bit-rate reductions around 50% for the same subjective video quality compared to its predecessor, High Efficiency Video Coding (HEVC). One year after finalization, VVC support in devices and chipsets is still under development, which is aligned with the typical development cycles of new video coding standards. This paper presents open-source software packages that allow building a complete VVC end-to-end toolchain already one year after its finalization. This includes the Fraunhofer HHI VVenC library for fast and efficient VVC encoding as well as HHI's VVdeC library for live decoding. An experimental integration of VVC in the GPAC software tools and FFmpeg media framework allows packaging VVC bitstreams, e.g. encoded with VVenC, in MP4 file format and using DASH for content creation and streaming. The integration of VVdeC allows playback on the receiver. Given these packages, step-by-step tutorials are provided for two possible application scenarios: VVC file encoding plus playback and adaptive streaming with DASH.

The Versatile Video Coding (VVC) standard was finalized in July 2020. It has since been studied in the literature with regards to both compression efficiency and complexity. Most of the experiments regarding VVC were performed based on the VTM reference software using its full configuration, usually compared to the test model of its predecessor, High Efficiency Video Coding (HEVC), HM. In this paper, we present a comprehensive and in-depth analysis of the VVC encoding complexity by using the fast and efficient open VVC encoder implementation VVenC. As a practical non-reference encoder, it offers alternative tradeoffs between compression efficiency and complexity. To find those, we approximate the Pareto set of the configuration space in an iterative process, encompassing search space extending encoding tools as well as search space pruning fast algorithms. While the VVenC Pareto set is derived based on HD and UHD video encoding results, it is validated and improved for alternative input material, namely for low-resolution and screen content video. Based on this optimization, five presets are defined constituting approximate Pareto-optimal encoding points for various content types as well as for multi-threaded operation. Without multi-threading, the presets span working points from faster at 4.7% HM runtime at 11% BD-rate gain through medium providing 38% BD-rate gain over HM at a quarter of the HM runtime, up to slower providing all of VTM’s gain at less than half of its runtime.

Practical open-source video encoders implementing different video compression standards often provide various operation points for different trade-offs between compression efficiency and encoding time. This paper presents an evaluation of the trade-offs for the freely available open-source VVenC encoder conforming to the open and public Versatile Video Coding (VVC) standard together with x265 conforming to the open and public High Efficiency Video Coding (HEVC) standard and aomenc conforming to the AOM Video 1 (AV1) specification developed within an industry consortium. The evaluation includes the HEVC and VVC reference encoder implementations to analyze the tradeoffs provided by the practical encoders when focusing on high-resolution 10-bit consumer applications that require random access. Compared to the HEVC reference implementation HM, VVenC provides averaged objective bit-rate savings using the Peak-Signal-to-Noise-Ratio (PSNR) metric ranging from more than 37% at less than 68% encoding time to more than 9% at less than 5% of the HM encoding time. Sampling the data for an encoding time next to but lower than that of the HM and using HM's outcome as the anchor, VVenC provides 37.6% averaged bit-rate savings, followed by aomenc with 11.5%. In contrast to VVenC that provides the same performance as the VVC reference implementation VTM at a lower encoding time, x265 generates an average of 35.2% bit-rate overhead relative to its reference implementation HM and does not achieve the HM performance even for higher encoding times.

The final version of the Versatile Video Coding (VVC) standard incorporates the Matrix-Based Intra Prediction (MIP) tool. It consists of additional intra prediction modes which were derived from a data-driven training. These modes, in general, are applied to the luma component only. This paper describes how to apply the MIP modes to the chroma components in certain cases. In the generic case, if a chroma block uses the intra direct mode (DM), its intra mode is derived from the co-located luma block. If this luma block uses MIP, the chroma block uses the planar mode, because a MIP mode is not applicable for all block shapes. However, if the chroma format is 4:4:4 and single-tree coding is used, luma and chroma blocks share the same partitioning. The paper investigates the impact of using the MIP mode of the luma component when the chroma mode is the DM in these cases. This extension also harmonizes MIP with the Adaptive Color-Space Transform (ACT) for RGB-content, because the chroma mode of blocks using the ACT is inferred to be the DM. For the extended MIP tool in the Versatile Video Coding Test Model 11.2 (VTM-11.2) that implements the final VVC standard, bit-rate savings of 1.51%/0.60%/0.62% in terms of the Bj⊘ntegaard-Delta bit rate (BD-rate) are reported compared to the VTM-11.2 with MIP disabled for the All-Intra configuration and natural content in the RGB 4:4:4 format.

This paper presents a new method for fast VVC intra-picture encoding using a CNN. The CNN operates on the original samples of$\mathbf{32}\times \mathbf{32}$blocks. Given a current block, it derives for each of the block's multi-type trees (MTTs), which are nested in quad-tree (QT) nodes, a parameter pair. The parameter pairs constrain the minimum width and height of the sub-blocks in their MTTs. This enables the CNN to control the number of tested MTT splits with fine granularity. To skip modes while maintaining the rate-distortion (RD) performance, we train the CNN considering the Lagrangian rate-distortion-time (RDT) cost caused by the derived parameters. First, we generate training data by encoding; when reaching a quad-tree node in a$\mathbf{32}\times \mathbf{32}$block, we encode the associated MTT with varying parameter pair values and record the resulting the RD and time cost. Then, when the CNN outputs parameters in training, we estimate the related RDT cost of the$\mathbf{32}\times \mathbf{32}$block using the recorded data. For this, we model the dependency between RDT cost and the parameters by emulating the encoder's RD optimization process. This way, we train the CNN while considering the RDT cost with an accuracy that is sufficient to outperform existing approaches. The approach achieves an encoding time reduction of 50% with a bit rate increase of only 0.7% for VTM-10.2.

In July 2020 the new video coding standard Versatile Video Coding (VVC) was released. VVC delivers bitrate savings of up to 50% relative to its predecessor High Efficiency Video Coding (HEVC) without compromising subjective quality. It has been designed for a broad range of applications and input material ranging from standard dynamic range to high dynamic range camera-captured content, screen content, and immersive applications. One of the key technologies included in the new standard is the flexible block partitioning using a quad-tree with multi-type tree structures (QT+MTT). Based on the VVC test model VTM, the VVenC project provides an open optimized VVC encoder. VVenC inherits and utilizes all of VTMs in-built partitioning search strategies. The QT+MTT partitioning search still poses a challenge for fast encoder implementations. After defining alternative low-depth partitioning configurations to VTM, it was observed that the initial search algorithm performs less optimal. We analyze and propose in this paper a set of additional fast search strategies for partitioning within VVenC, optimized for low-depth partitioning configurations. As a result, an average of 38% runtime reduction is achieved at the managable cost of less than 2% BD-rate loss.

The latest international Versatile Video Coding (VVC) standard was finalized in July 2020 by the Joint Video Experts Team (JVET) from ITU-T and ISO/IEC. Compared to the High Efficiency Video Coding (HEVC) standard, VVC offers up to 50% of bitrate savings. However, the encoder runtime of the reference software increases tenfold compared to the HEVC equivalent. This paper shows that VVC coding tools which do not consume much of the reference encoder time can have prohibitive computational cost for practical encoders, such as VVenC. VVenC is an optimized open-source real-world implementation with high relevance for practical applications. After analyzing the encoding tools in VTM and VVenC, Symmetric Motion Vector Difference (SMVD) was identified as one of the tools whose efficiency-complexity trade-off may be unfavorable for practical applications. For that reason, two complexity reduction methods for the SMVD search are proposed in this paper. The experimental evaluation on VVenC encoder at an optimized operation point confirms the SMVD encoding runtime reduction by over 72%, with minimal impact on the encoding efficiency.

In an increasingly connected world, consumer video experiences have diversified away from traditional broadcast video into new applications with increased use of non-camera-captured content such as computer screen desktop recordings or animations created by computer rendering, collectively referred to as screen content. There has also been increased use of graphics and character content that is rendered and mixed or overlaid together with camera-generated content. The emerging Versatile Video Coding (VVC) standard, in its first version, addresses this market change by the specification of low-level coding tools suitable for screen content. This is in contrast to its predecessor, the High Efficiency Video Coding (HEVC) standard, where highly efficient screen content support is only available in extension profiles of its version 4. This paper describes the screen content support and the five main low-level screen content coding tools in VVC: transform skip residual coding (TSRC), block-based differential pulse-code modulation (BDPCM), intra block copy (IBC), adaptive color transform (ACT), and the palette mode. The specification of these coding tools in the first version of VVC enables the VVC reference software implementation (VTM) to achieve average bit-rate savings of about 41% to 61% relative to the HEVC test model (HM) reference software implementation using the Main 10 profile for 4:2:0 screen content test sequences. Compared to the HM using the Screen-Extended Main 10 profile and the same 4:2:0 test sequences, the VTM provides about 19% to 25% bit-rate savings. The same comparison with 4:4:4 test sequences revealed bit-rate savings of about 13% to 27% for$Y'C_{B}C_{R}$and of about 6% to 14% for$R'G'B'$screen content. Relative to the HM without the HEVC version 4 screen content coding extensions, the bit-rate savings for 4:4:4 test sequences are about 33% to 64% for$Y'C_{B}C_{R}$and 43% to 66% for$R'G'B'$screen content.

In this paper we present a "State-Based Multi-Parameter Probability Estimation" (SBMP) for Context-Based Adaptive Binary Arithmetic Coding (CABAC) which employs a two hypotheses probability estimator based on exponentially weighted moving averages. It uses a logarithmic state representation and a single subsampled transition table with only 32 elements for the probability update. This reduces the memory requirements virtually without affecting the compression efficiency, compared to corresponding approaches that use a linear state representation and a computation-based probability update. The proposed scheme is based on simple operations like table look-ups and additions. Compared to the state-of-the-art probability estimator of the video compression standard H.265/HEVC, the compression efficiency is increased by up to 1 % Bjøntegaard-Delta bit rate (BD rate) when applied to draft 2 of the Versatile Video Coding (VVC) standard. Furthermore, SBMP was recently adopted to working draft 2 of the MPEG-7 part 17 standard for compression of neural networks for multimedia content description and analysis.

The support for screen content coding has received more attention with the latest development in video compression, the upcoming Versatile Video Coding (VVC) standard. Among the dedicated screen content coding tools, the transform skip mode (TSM) represents a promising approach for improving the coding efficiency at a low impact on implementation complexity. In this work, we present a dedicated residual coding for transform blocks coded in TSM. Due to the lack of the energy compaction of the transform, the quantization indexes for blocks coded in TSM have different statistical properties, which can be exploited in the entropy coding. Our coding experiments with screen content sequences yielded bit-rate savings of 3.9% for intra-only coding and 2.8% for typical random access configurations.

The objective PSNR metric is known to correlate quite poorly with subjective assessments of video coding quality. Thus, a number of alternative VQA measures such as (MS-)SSIM and VMAF have been proposed. These, however, are often algorithmically complex and difficult to use for visually motivated encoder optimization tasks, especially subjectively optimized bit allocation. In this paper we show that, by way of low-complexity enhancements of our previous work on a perceptually weighted PSNR (WPSNR) metric, addressing shortcomings with video and ultra high-definition content, the prediction of human judgments of video coding quality by the WPSNR can be improved. In fact, the resulting XPSNR seems to match the performance of the aforementioned state-of-the-art methods.

To reduce the residual energy of a video signal, motion compensated prediction with fractional-sample accuracy has been successfully employed in modern video coding technology. In contrast to the fixed quarter-sample motion vector resolution for the luma component in High Efficiency Video Coding standard, the current draft of a new Versatile Video Coding standard introduces a block-level adaptive motion vector resolution (AMVR) scheme. The AMVR allows coding of motion vector difference at different precisions. Nevertheless, the interpolation filters for each fractional sample position are fixed. In this paper, alternative half-luma-sample interpolation filters are proposed. Enabling the interpolation filter selection at a fine granularity allows to better adapt to the local image characteristics. Experimental results show that the proposed method can improve the average BD-rate of VTM 5.0 reference software by -0.40% for Random Access configuration and by -0.76% for Low Delay P configuration.

In the current version of the evolving Versatile Video Coding (VVC) standard, the Matrix-Based Intra Prediction (MIP) tool is included which is based on data-driven training techniques. This paper describes how an efficient fixed point implementation of the MIP prediction modes was designed. First, a method is presented that improves the integer quantization of the floating-point matrices resulting from the training by geometrically transforming the input and output. Second, a soft-clipping function can be used during the training stage to restrict the range of floating-point matrix entries. This enables the quantization of the trained coefficients with a fixed bit depth and precision. The clipping itself has no impact on the compression efficiency. The obtained predictors are incorporated into the Versatile Video Coding Test Model 7. All-Intra bit-rate savings of 0.6 % across different resolutions in terms of the Bjøntegaard-Delta bit rate (BD-rate) are reported.

Lossy compression is the main target of the upcoming video coding standard Versatile Video Coding (VVC). However, lossless coding is supported in VVC by utilizing a certain encoder configuration. Particularly, the Transform Skip Mode (TSM) is always selected at the block level to bypass the transform stage (together with a QP that results in the same output as input at the quantization stage). Consequently, the Intra Subpartition (ISP) coding mode cannot be used for lossless coding, considering that its combination with TSM is not supported in VVC because it does not provide a significant coding benefit for the lossy common test conditions. For this reason, it is proposed to enable such a combination for the benefit of lossless coding. Besides, the encoder search has been optimized to improve the trade-off between compression benefit and encoder run-time. Experimental results show a 0.71% coding gain with a corresponding encoder run-time of 111%.

Recent approaches to compression of deep neural networks, like the emerging standard on compression of neural networks for multimedia content description and analysis (MPEG-7 part 17), apply scalar quantization and entropy coding of the quantization indexes. In this paper we present an advanced method for quantization of neural network parameters, which applies dependent scalar quantization (DQ) or trellis-coded quantization (TCQ), and an improved context modeling for the entropy coding of the quantization indexes. We show that the proposed method achieves 5.778% bitrate reduction and virtually no loss (0.37%) of network performance in average, compared to the baseline methods of the second test model (NCTM) of MPEG-7 part 17 for relevant working points.

An increasing number of distributed machine learning applications require efficient communication of neural network parameterizations. DeepCABAC, an algorithm in the current working draft of the emerging MPEG-7 part 17 standard for compression of neural networks for multimedia content description and analysis, has demonstrated high compression gains for a variety of neural network models. In this paper we propose a method for employing DeepCABAC in a Federated Learning scenario for the exchange of intermediate differential parameterizations. Furthermore, we discuss the efficiency of DeepCABAC when compressing trained neural networks. Our experiments on large neural networks show that in both scenarios, DeepCABAC achieves competitive compression rates, without degrading the network accuracy.

Intra block copy is a prediction technique in intra coding, which has high compression performance for screen content or computer generated videos. Accordingly, this tool has become part of the upcoming video coding standard H.266/VVC (Versatile Video Coding). Intra block copy is analogous to motion-compensated prediction for the usual inter-picture case with the additional constraint that the reference picture is given by the current partially reconstructed picture. Moreover, the entropy coding of the displacement information of the reference block is carried out in the same way as in inter-picture motion compensation. This publication proposes a set of region-based predictors to improve the current intra block copy approach. The proposed method achieves up to -3.81% of Bjøntegaard-Delta bit-rate saving on top of the existing intra block copy for an all-intra picture configuration.

Versatile Video Coding (VVC) is the emerging video coding standard to be finalized by the Joint Video Experts Team in July 2020. Compared to its predecessor, the High Efficiency Video Coding (HEVC) standard, VVC provides 50% bit-rate reduction at comparable visual quality for natural video content in high-definition (HD) and ultra high-definition (UHD) resolution. To achieve this, the standard incorporates more advanced and generalized algorithms, leading to an increase in computational complexity. This includes for example additional in-loop filters, decoder-side motion refinement and search as well as an increased number of transforms, which creates a grand challenge for implementers to achieve live decoding on general-purpose CPUs. In this paper, the work on an efficient software decoder implementation for the upcoming VVC standard is described, including optimization of sample operations using single instruction multiple data (SIMD) instructions and parallelization approaches with multithreading. As a result, the presented decoder can perform live decoding of 10bit HD video at 60 frames per second (fps) and 10bit UHD video at 30fps on modern mobile consumer hardware, showcasing that VVC live decoding is possible already right before finalization of the standard.

Versatile Video Coding (VVC) is a new international video coding standard to be finalized in July 2020. It is designed to provide around 50% bit-rate saving at the same subjective visual quality over its predecessor, High Efficiency Video Coding (H.265/HEVC). During the standard development, objective bit-rate savings of around 40% have been reported for the VVC reference software (VTM) compared to the HEVC reference software (HM). The unoptimized VTM encoder is around 9x, and the decoder around 2x, slower than HM. This paper discusses the VVC encoder complexity in terms of soft-ware runtime. The modular design of the standard allows a VVC encoder to trade off bit-rate savings and encoder runtime. Based on a detailed tradeoff analysis, results for different operating points are reported. Additionally, initial work on software and algorithm optimization is presented. With the optimized software algorithms, an operating point with an over 22x faster single-threaded encoder runtime than VTM can be achieved, i.e. around 2.5x faster than HM, while still providing more than 30% bit-rate savings over HM. Finally, our experiments demonstrate the flexibility of VVC and its potential for optimized soft-ware encoder implementations.

In this paper we combine video compression and modern image processing methods. We construct novel iterative filter methods for prediction signals based on Partial Differential Equation (PDE) based methods. The mathematical framework of the employed diffusion filter class is given and some desirable properties are stated. In particular, two types of diffusion filters are constructed: a uniform diffusion filter using a fixed filter mask and a signal adaptive diffusion filter that incorporates the structures of the underlying prediction signal. The latter has the advantage of not attenuating existing edges while the uniform filter is less complex. The filters are embedded into a software based on HEVC with additional QTBT (Quadtree plus Binary Tree) and MTT (Multi-Type-Tree) block structure. In this setting, several measures to reduce the coding complexity of the tool are introduced, discussed and tested thoroughly. The coding complexity is reduced by up to 70% while maintaining over 80% of the gain. Overall, the diffusion filter method achieves average bitrate savings of 2.27% for Random Access having an average encoder runtime complexity of 119% and 117% decoder runtime complexity. For individual test sequences, results of 7.36% for Random Access are accomplished.

Video coding in the YCbCr color space has been widely used, since it is efficient for compression, but it can result in color distortion due to conversion error. Meanwhile, coding in the RGB color space maintains high color fidelity, having the drawback of a substantial bitrate increase with respect to YCbCr coding. Cross-component prediction (CCP) efficiently compresses video content by decorrelating color components while keeping high color fidelity. In this scheme, the chroma residual signal is predicted from the luma residual signal inside the coding loop. This paper gives a description of the CCP scheme from several points of view, from theoretical background to practical implementation. The proposed CCP scheme has been evaluated in standardization communities and adopted into H.265/High Efficiency Video Coding (HEVC) Range Extensions. The experimental results show significant coding performance improvements for both natural and screen content video, while the quality of all color components is maintained. The average coding gains for natural video are 17% and 5% bitrate reduction in the case of intra coding and 11% and 4% in the case of inter coding for RGB and YCbCr coding, respectively, while the average increment of encoding and decoding times in the HEVC reference software implementation are 10% and 4%, respectively.

In this paper, we describe a video coding design that enables a higher coding efficiency than the HEVC standard. The proposed video codec follows the design of block-based hybrid video coding, but includes a number of advanced coding tools. A part of the incorporated advanced concepts was developed by the Joint Video Exploration Team, while others are newly proposed. The key aspects of these newly proposed tools are the following. A video frame is subdivided into rectangles of variable size using a binary partitioning with variable split ratios. Three new approaches for generating spatial intra prediction signals are supported: A line-wise application of conventional intra prediction modes, coupled with a mode-dependent processing order, a region-based template matching prediction method and intra prediction modes based on neural networks. For motion-compensated prediction, a multi-hypothesis mode with more than two motion hypotheses can be used. In transform coding, mode dependent combinations of primary and secondary transforms are applied. Moreover, scalar quantization is replaced by trellis-coded quantization and the entropy coding of the quantized transform coefficients is improved. The intra and inter prediction signals can be filtered using an edge-preserving diffusion filter or a non-linear DCT-based thresholding operation. The video codec includes an adaptive in-loop filter for which one of three classifiers can be chosen on a picture basis. We also incorporated an optional encoder control, which adjusts the quantization parameters based on a perceptually motivated distortion measure. In a random access scenario, our proposed video codec achieves luma BD-rate savings between 32.5% for HDR HLG UHD and 39.6% for SDR UHD over the HEVC (HM software) anchor for different categories of test sequences.

In this paper we combine video compression and modern image processing methods. Iterative filter methods for prediction signals based on classic inpainting methods are introduced and extensive parameter tests are described. In order to construct an alternative prediction filter for video coding, techniques originally employed for inpainting are applied. Thereby, the structures of the underlying prediction were incorporated into the filter construction making it signal adaptive. The resulting optimization problem is solved using the so-called Alternating Direction Method of Multipliers (ADMM). The undertaken novel parameter tests are described and it is shown that they improve the coding efficiency of the tool. The suggested filter is embedded into a software based on HEVC with additional QTBT (Quadtree plus Binary Tree) and MTT (Multi-Type-Tree) block structure. Overall, the proposed filter method obtains average bitrate savings of 1.35% at an average encoder runtime increase of 28% and decoder runtime increase of 38%. UHD test sequences achieve bitrate savings of up to 3.66% for Random Access.

Intra-picture prediction as in HEVC exploits the nearest reference line adjacent to the current coding unit (CU) for prediction of samples. If this reference line represents a discontinuity, the reference samples in this reference line can differ to a large extent from the original samples and may lead to a large prediction error. We propose a multiple reference lines (MRLs) coding to allow not only the nearest reference line 0 but also reference lines 1 and 3 to be candidates for angular intra prediction as shown in Fig. 1. To reduce the complexity arising from additional lines to be checked at encoder side, we further propose to restrict the MRL to angular most probable modes (MPMs) only. The MRL coding signals the reference line index before the intra prediction mode. This allows to not signal the MPM flag of the current CU and implicitly derive it as true when a non-zero reference line index is signaled. Experimental results are provided to evaluate the performance of the proposed MRL coding on top of the VVC test model VTM-2.0.1. 26 test sequences in different categories, including 4k, 1080p, 720p, WVGA, WQVGA resolutions and screen contents are tested. Two coding structures are evaluated, all intra (AI) and random access (RA). The objective coding efficiency is measured in terms of Bjøntegaard Delta (BD) rate (%) computed using four rate/PSNR points that were generated by using quantization parameters 22, 27, 32 and 37. Lower (negative) BD-rate implies better compression rate. Table 1 shows that the presented MRL provides 0.46% bitrate savings for an all-intra and 0.2% for a random-access configuration on average. Furthermore, it provides 1.45% bitrate reduction for screen content test sequences, which are representing an increasingly important video application. Because of a fairly good trade-off between coding efficiency and complexity, the proposed MRL coding mode with MPM restriction was adopted into the current VVC draft standard.

Versatile Video Coding (VVC) is going to be the successor of High Efficiency Video Coding (HEVC). The new standard (currently under development) requires the adoption of tools that provide a good trade-off between gain and implementation complexity. In previous work we proposed a Line-Based Intra Prediction (LIP) algorithm to improve the coding performance of the upcoming standard. However, that contribution did not sufficiently take into account the hardware-related complexity of the method. For this reason, this publication addresses these problems and proposes several modifications of the algorithm to solve them. Experimental results show an average gain of 1.0% and 0.5% for the All Intra (AI) and Random Access (RA) configurations, respectively.

We train a neural network to perform intra picture prediction for block based video coding. Our network has multiple prediction modes which co-adapt during training to minimize a loss function. By applying the l1-norm and a sigmoid-function to the prediction residual in the DCT domain, our loss function reflects properties of the residual quantization and coding stages present in the typical hybrid video coding architecture. We simplify the resulting predictors by pruning them in the frequency domain, thus greatly reducing the number of multiplications otherwise needed for the dense matrix-vector multiplications. Also, by quantizing the network weights and using fixed point arithmetic, we allow for a hardware friendly implementation. We demonstrate significant coding gains over state of the art intra prediction.

It is well known that input-invariant quantization in perceptual image or video coding often leads to visually suboptimal results and that quantization parameter adaptation (QPA) based on a model of the human visual system can improve subjective coding quality. This paper introduces a simple low-complexity QPA algorithm, controlled using a block-wise perceptually weighted distortion measure representing a generalization of the PSNR metric. The weighting scheme of this WPSNR metric is based on a psychovisual model. It directly leads to a perceptually adapted scaling of the block-wise Lagrange parameter used in the bit-allocation process in the encoder and, consequently, to a block-wise QPA. Unlike prior QPA approaches, the proposal avoids classifications of picture regions and easily extends from still-image or grayscale to video or chromatic coding. The WPSNR metric also uses fewer algorithmic operations than e. g. the multiscale structural similarity measure (MS-SSIM). Due to the results of two formal subjective tests indicating its visual benefit, the QPA proposal has been adopted into VTM, the currently developed Versatile Video Coding (VVC) reference software.

In state-of-the-art video coding, the prediction error signals are transmitted using transform coding, which consists of an orthogonal transform, scalar quantization, and entropy coding of the quantization indexes. We show that the coding efficiency of transform coding can be improved by replacing scalar quantization with trellis-coded quantization (TCQ) and using advanced entropy coding techniques for coding the quantization indexes. The proposed approach was implemented into the first test model (VTM-1) of the new standardization project Versatile Video Coding (VVC). Our coding experiments yielded average bit-rate savings of 4.9% for intra-only coding and 3.3% for typical random access configurations, where bit-rate savings of 3.5% (intra-only) and 2.4% (random access) can be attributed to the usage of TCQ. These coding gains are obtained at a 5-10% increase in encoder run time and without any change in decoder run time.

In a template matching (TM) intra method, the neighboring samples of the current block are regarded as a template. The decoder searches for the best template match in the current reconstructed picture using an error minimizing metric like sum of squared differences (SSD). The prediction signal is generated by copying the samples of the adjacent block to the selected template match. The increased decoder complexity from the search algorithm makes it less attractive for modern applications like video calling. In a previous publication [1], we presented a region-based template matching (RTM) approach for intra coding. Compared to the conventional TM methods which searches for the template match in a complete search window, RTM searches in a region of the search window. Thus, RTM offers a better trade-off between coding efficiency and decoder complexity. Nevertheless, the memory requirements and number of computations to be carried at the decoder are still high, making RTM difficult for hardware realization. This paper aims to address these issues.

One of the innovations in H.265/HEVC is the quad-tree partitioning framework. It allows flexible block subdivision and mode allocation across the encoded picture. The increased flexibility comes at a cost of vast search space expansion, making exhaustive search algorithms inapplicable. We propose a novel early termination condition to skip the exhaustive search of whole tree-branches in the well-established top-down encoding approach. The condition is based on a simple and intuitive split cost prediction. It can be parametrized to control the trade-off between the speed-up and caused BD-rate loss. Data driven parameter estimation and parameter number reduction is presented. For random-access encoding, the method can achieve an average speed-up of 30% with a BD-rate loss of 0.03%. At another trade-off point, speed-up is increased to over 40% for a BD-rate loss below 0.5%.

Bit-allocation based on the MSE is computationally convenient in image and video compression, but leads to perceptually suboptimal compression results. Distortion sensitivity, modeled as a reference specific property, can be used to improve the accuracy of perceptual quality prediction based on the MSE. This paper shows how distortion sensitivity directly leads to computationally beneficial perceptual optimization of irrelevance reduction and, thereby, of bit-allocation in image and video compression. To this end distortion sensitivity is estimated using a deep convolutional neural network. The proposed method of distortion sensitive bit-allocation is evaluated experimentally using HEVC and on our testset shows average bit-rate reductions with regard to the MOS of 15.9% compared to constant QP-based bit-allocation and 7.3% compared to state-of-the-art perceptual bit-allocation schemes.

The upcoming Versatile Video Coding (VVC) Standard includes various new intra prediction tools that were not present in its predecessor High Efficiency Video Coding (HEVC), such as the wide angle intra prediction, the position dependent prediction combination or the multiple reference line intra prediction. In order to improve the intra prediction coding efficiency, this paper proposes the usage of the Intra Subpartition (ISP) algorithm. ISP is an updated version of the Line-Based Intra Prediction (LIP) mode that improves the trade-off between coding gain and complexity of the original method. The basic principle of ISP consists in subdividing an intra-predicted block into 2 or 4 subpartitions of at least 16 samples according to the original block dimensions. The method has been implemented on top of the VVC Test Model 3.0 (VTM-3.0), with the result of obtaining a gain of 0.57% with encoding and decoding run-times of 112% and 102% respectively for the All Intra configuration and a gain of 0.29% with encoding and decoding run-times of 102% and 101% respectively for the Random Access configuration.

The peak signal-to-noise ratio (PSNR) is the most used objective measure for assessing perceptual image quality when it comes to image and video compression tasks, despite the fact that it exhibits weak performance in reflecting human perception. To address this problem, many image quality assessment (IQA) methods were proposed, e. g. the structural similarity quality measure (SSIM) and its extension, the multi-scale SSIM (MS-SSIM). In this paper we revisit and evaluate a block-based perceptually weighted PSNR (WPSNR) which calculates weighting factors to capture visual sensitivity of local image regions. We further introduce a sample-based version of WPSNR which determines those sensitivity weights with higher spatial accuracy. These methods are computationally inexpensive compared to other similarity measures and are shown to outperform PSNR, SSIM and similar perceptual quality measures when it comes to approximate subjective ratings of JPEG or JPEG2000 compressed images.

This paper presents a novel method for a data-driven training of affine-linear predictors which perform intra prediction in state-of-the-art video coding. The main aspect of our training design is the use of subband decomposition of both the input and the output of the prediction. Due to this architecture, the same set of predictors can be shared across different block shapes leading to a very limited memory requirement. Also, the computational complexity of the resulting predictors can be limited such that it does not exceed the complexity of the conventional angular intra prediction. In the training itself, a loss function modelling the bit-rate of the DCT-transformed residuals is used. The obtained predictors are incorporated into the Versatile Video Coding Test Model 3 in addition to the conventional intra prediction modes. All-Intra bit-rate savings ranging from 0.8% to 1.4% across different resolutions have been measured in terms of the Bjøntegaard-Delta bit rate (BD-rate).

One key component of all block-based hybrid video codecs is transform coding of prediction residues, which consists of an orthogonal block transform, scalar quantization of transform coefficients, and entropy coding of the resulting quantization indexes. For improving coding efficiency relative to the state-of-the-art video coding standard HEVC, we propose the following modifications: (1) Replacing scalar quantization with trellis-coded quantization; and (2) utilizing additional statistical dependencies between quantization indexes for entropy coding. The proposed approach was integrated into the first test model VTM-1 for the new standardization project Versatile Video Coding (VVC). Our coding experiments showed average bit-rate savings of 4.9 % for intra-only, 3.4 % for random access, and 2.8 % for low-delay configurations.

Template matching is a texture synthesis method which has found applications in video coding. However, the increased decoder complexity from its underlying search algorithm makes it less suitable for practical applications. In our previous paper, we presented region-based template matching where the decoder complexity is considerably reduced. This publication describes further improvements of the aforementioned tool for intra prediction and decoder side motion vector derivation. When applied together, it achieves an average coding gain of -1.04% with 107% decoder run-time for all intra and -1.60% with 113% decoder run-time for random access configurations respectively. The proposed method needs considerably fewer computations and memory requirements compared to the previous version of the algorithm.

One of the key technologies in the upcoming Versatile Video Coding (VVC) standard is the flexible partitioning. Compared to HEVC, partitioning alone accounts for about 8.5% bitrate savings. The added flexibility comes at a cost of large search space expansion.This paper gives an overview of fast encoder strategies for selecting the block partitioning in the Test Model VTM-3.0 for VVC. In total, 13 approaches for speeding up the block partitioning selection in an encoder are described; 8 of these techniques were developed by the authors. 10 of the 13 methods are used in the common test conditions of the Joint Video Experts Team (JVET); these 10 methods achieve an overall speed-up factor of 7 at a bit rate increase of only 1.1%. By enabling 12 approaches, a speed up factor of over 8.5 is obtained at a bit rate increase of about 1.5%.

In-loop filtering is an important task in video coding, as it refines both the reconstructed signal for display and the pictures used for inter-prediction. At the current stage of the Versatile Video Coding (VVC) standardization, there are three in-loop filtering procedures consisting of deblocking filter (DBF), sample adaptive offset (SAO) and adaptive loop filter (ALF). Among them, SAO is the simplest in-loop filtering process and highly effective in removing coding artifacts. It simply modifies decoded samples by conditionally adding an offset value to each sample after the application of the DBF. For this, a classification is applied for each sample location, which gives a partition of the set of all sample locations. After that, an offset value is added to all samples associated with each class. Therefore, the performance of SAO essentially relies on how its classification behaves. In this paper, we introduce a novel classification method for SAO. Based on this, we derive an additional SAO filtering process which we call post sample adaptive offset (PSAO). Experimental results show the effectiveness of our proposed PSAO filtering process. On average, 0.42%, 0.31% and 0.33% additional coding gains can be achieved on top of VTM-5.0 for all intra (AI), random access (RA) and low delay with B pictures (LB) configurations, respectively.

The Versatile Video Coding (VVC) development has adopted the possibility to bypass the transform when the transform block size is equal to 4×4 from its predecessor High Efficiency Video Coding (HEVC). This so-called Transform Skip Mode (TSM) results in increased encoding time when extending it to transform block sizes up to 32×32 while the compression efficiency improvement is for screen content only. This paper presents the so-called Unified MTS scheme that makes TSM for luma transform block sizes up to 32×32 possible without the disadvantage of excessive encoding time increase by incorporating the TSM with the existing Multiple Transform Set (MTS) technique. The Unified MTS scheme achieves compression efficiency improvements, in terms of BD-rate, of about -5.0% in the All-Intra configuration and -5.4% in the Random-Access configuration, respectively, for the screen content sequences of the used test set. Compared to the straightforward extension of TSM to transform block sizes up to 32×32, the encoding time is about 25% less in the All-Intra configuration and about 21% less in the Random-Access configuration, respectively, whereas the compression efficiency improvements are only 0.04% less in the All-Intra configuration, and 0.09% less in the Random-Access configuration, respectively. Relative to the anchor using TSM for 4×4 transform blocks only, the encoding time is the same for natural content and 3% higher for screen content.

In this paper we compare two different methods to filter prediction signals in video coding: The anisotropic diffusion filter modeled using a (modified) heat equation and the ADMM (Alternating Direction Method of Multipliers) filter using directional total variation. The two methods are based on the same idea, namely using the structures in the underlying prediction signal to control the direction of the smoothing. It is shown here, that mathematically, in a continuous setting for a certain choice of parameters, the solutions of the two problem formulations are the same.Both filters are compared in the setting of a software based on HEVC (High Efficiency Video Coding). While the results for the diffusion filter method involve the testing of four different filters of uniform and signal adaptive type, the ADMM filter results feature only one type of filter configuration that can be en- or disabled in a block-wise manner. Despite the fact that both methods rely on the same idea, it is illustrated that the overall gains of the combination are higher than the RD gains of the individual tools. In particular, it is demonstrated that this is still the case if the filters are combined in an exclusive way. This proves that there is an additional value for each of the filter methods.

In natural digital images and videos, correlations between color components can be observed. These correlations can be exploited to achieve additional coding gain in modern block-based hybrid video coding. To this end, we propose the use of a block-wise, rotational inter-component transform (ICT) applied to the two residual chroma signals that result from conventional intra or inter-picture prediction. Different ICT parameterizations in terms of number and quantization of the rotational angles as well as resulting components signaled in the coded bitstream are investigated. An implementation into the currently developed Versatile Video Coding (VVC) reference software provides average bitrate savings of up to 0.7% (All Intra configuration) with negligible increases in implementation complexity and runtime. Our proposal has been adopted into the VVC draft specification text.

This paper presents a data-driven training of affine- linear predictors which perform intra-picture prediction for video coding. The trained predictors use a single line of reconstructed boundary samples as input like the conventional intra prediction modes. For large blocks, the presented predictors initially transform the input samples via Discrete Cosine Transform. This allows to omit high frequency coefficients and consequently reduce the input dimension. The output is the result of a single matrix-vector multiplication and offset addition. Here, the predictors only construct certain coefficients in the frequency domain. The final prediction signal is then obtained by inverse transform. The coefficients of the prediction modes need to be stored in advance, requiring 0.273 MB of memory. The training employs a recursive block partitioning, where the loss function targets to approximate the bit-rate of the DCT-transformed block residuals. The obtained predictors are incorporated into the Versatile Video Coding Test Model 4. The authors report All- Intra bit-rate savings ranging from 0.7% to 2.0% across different resolutions in terms of the Bjøntegaard-Delta bit rate (BD-rate).

In state-of-the-art video compression residual coding is done by transforming the prediction error signals into a less correlated representation and performing the quantization and entropy coding in the transform domain. For complexity reasons usually separable transforms are used. A more flexible transform structure is given by row-column transforms, which apply a separate transform to each row and each column of a signal block. This paper describes a method for training such structured transforms by maximizing the data likelihood under a parameterized probabilistic model with a compelled structure. An explicit model is derived for the case of row-column transforms and its efficiency is demonstrated in the application of video compression. It is shown that trained row-column transforms achieve almost the same coding gain as unconstrained KLTs when applied as secondary transforms, while the encoder and decoder runtime are the same as in the separable transform case.

Block partitioning is the basis of every modern hybrid video coding standard. It specifies how the video pictures can be subdivided into blocks for prediction and residual coding. In H.265/HEVC, quad-tree partitioning is one of the key technologies allowing for flexible mode allocation and providing a substantial part of the gains over H.264/AVC. The current draft of the upcoming standard Versatile Video Coding (VVC) provides over 30% bit-rate savings over HEVC and almost one third of the gain is achieved by using a more flexible partitioning scheme than the quad-tree partitioning used in HEVC. In this paper, we describe a partitioning concept that generalizes many of the ideas developed during the exploration and early standardization phase of VVC. In fact, our method includes the VVC partitioning as well as many other state-of-the-art methods. The proposed method can be implemented in a fully configurable design. For instance, it can be configured to match the performance of VTM-1.0 at much faster runtime (69%) or it can be configured to obtain additional bit-rate savings of up to 3% by exploiting additional degrees of freedom.

A key component of state-of-the art video coding is motion-compensated prediction, also called inter prediction. Current standards allow uni- and bi-prediction, i.e. linear superposition of up to two motion-compensated prediction signals. It is well-known that by a superposition of more than two prediction signals (or hypotheses), the energy of the prediction error can be further reduced. In this paper, it is shown that allowing the encoder to choose among different weights for the individual hypotheses is beneficial from a rate-distortion perspective. A practical multi-hypothesis inter prediction scheme based on the Versatile Video Coding Test Model (VTM) is presented. For VTM-1, in the Random Access configuration according to the JVET Common Test Conditions, the average luma BD bit rate is in the range of -1.6 % to -1.9 % for different settings using up to four prediction hypotheses. For VTM-2, the corresponding BD bit rate is -0.95 %. For higher bit rates (i.e., QP values 12, 17, 22, 27) the BD bit rates are -2.2 % for VTM-1 and -1.4 % for VTM-2.

Intra prediction in modern block-based hybrid video coding standards like H.265/High Efficiency Video Coding (HEVC) extrapolates the reconstructed values of neighboring reference samples of a block to generate the prediction signal. Due to the loss of correlation between samples with increasing distance, this procedure may lead to residual signals with high levels of energy concentrated in the most distant regions of the block relative to the neighboring reference samples. To circumvent this loss in prediction quality with increasing distance and improve the coding efficiency of intra prediction, this publication proposes an algorithm that divides blocks into line-based sub-partitions that are processed individually. This approach has been implemented in the context of HEVC and the new Joint Exploration Test Model (JEM), with the result of obtaining average bit-rate savings of 2.77% and 1.62% for a whole test set of video sequences according to the All Intra and Random Access configurations, respectively.

In this paper we combine state of the art video compression with inpainting methods. We introduce a new signal adaptive method to filter the predictions of a hybrid video codec using a constrained optimization approach. The filter is embedded into the framework of HEVC. The efficiency of the HEVC video codec is improved by up to -6.14% Bj Ø ntegaard delta (BD) rate for All Intra.

This paper presents a thresholding method for processing the predicted samples in the state-of-the-art High Efficiency Video Coding (HEVC) standard. The method applies an integer-based approximation of the discrete cosine transform to an extended prediction block and sets transform coefficients beneath a certain threshold to zero. Transforming back into the sample domain yields the improved prediction signal. The method is incorporated into a software implementation that is conforming to the HEVC standard and applies to both intra and inter predictions. Consequently, bit-rate savings ranging from 2.3% to 8.0% have been measured in terms of the Bjøntegaard-Delta bit rate (BD-rate).

Intra prediction plays a very important role in current video coding technologies like the H.265/High Efficiency Video Coding (HEVC) standard, the Joint Exploration Test Model (JEM) and the upcoming Versatile Video Coding (VVC) standard. In previous work we proposed a Line-Based Intra Prediction algorithm to improve the state-of-art coding performance of HEVC and JEM. This method divides (horizontally or vertically) a block into lines and then it codes each of them individually in a sequential manner. At the encoder side, however, it is necessary to select an optimal combination of intra mode and 1-D split type in a Rate-Distortion sense. Since testing for every block all possible combinations of these two parameters would imply a very significant increase in the encoder complexity, this paper proposes several fast algorithms to reduce the number of tests and improve the overall trade-off between complexity and gain. The experimental results show a reduction of the encoder run-time from 322% to 166% in exchange for a loss of 0.34% for the All Intra configuration and from 151% to 116% for a loss of 0.15% in the case of Random Access.

In video coding, adaptive loop filter (ALF) has attracted attention due to its increasing coding performances. Recently ALF has been further developed for its extension, which introduces geometry transformation-based adaptive loop filter (GALF) outperforming the existing ALF techniques. The main idea of ALF is to apply a classification to obtain multiple classes, which gives a partition of a set of all pixel locations. After that, a Wiener filter is applied for each class. Therefore, the performance of ALF essentially relies on how its classification behaves. In this paper, we introduce a novel classification method, Multiple feature-based Classifications ALF (MCALF) extending a classification in GALF and show that it increases coding efficiency while only marginally raising encoding complexity. The key idea is to apply more than one classifier at the encoder to group all reconstructed samples and then to select a classifier with the best RD-performance to carry out the classification process. Simulation results show that around 2% bit rate reduction can be achieved on top of GALF for some selected test sequences.

This paper presents and discusses an objective performance evaluation of two lines of development for the next generation of video coding schemes: AV1 of the Alliance for Open Media (AOM) and JEM of the Joint Video Exploration Team (JVET). Their already established predecessors, as given by VP9 for AV1 and High Efficiency Video Coding (HEVC) for JEM, serve as references for this evaluation process. A large testset of 28 video sequences with different content, various resolution, and different frame rate has been used as the common data basis. For each sequence and each test candidate, four different rate points associated with varying qualities of reconstruction have been generated. When evaluated in this way relative to their corresponding predecessors, AV1 and JEM show a quite significant coding-efficiency improvement in terms of averaged Bjøntegaard-Delta (BD) bit rate of about 23% and about 32%, respectively. Moreover, this performance study also reveals that there exist different trade-offs between compression efficiency and computational complexity in terms of encoder run time. While the JEM encoder requires a factor of about 8.5 in run time relative to HM, the corresponding run time factor for the AV1 encoder is about 55.8 relative to VP9. When comparing the next-generation coding schemes among each other, this study reveals that AV1 requires an average BD bit-rate overhead of about 49% at the same objective quality while consuming 2.47 times the run time of the JEM encoder.

In this paper we combine state of the art video compression and Partial Differential Equation (PDE) based image processing methods. We introduce a new signal adaptive method to filter the predictions of a hybrid video codec using a system of PDEs describing a diffusion process. The method can be applied to intra as well as inter predictions. The filter is embedded into the framework of HEVC. The efficiency of the HEVC video codec is improved by up to -2.76% for All Intra and -3.56% for Random Access measured in Bjøntegaard delta (BD) rate. Coding gains of up to -8.76% can be observed for individual test sequences.

In this paper, a region-based template matching method is proposed for decoder-side motion vector derivation. The proposed method is based on a template matching search algorithm, where the search window is partitioned into five regions and the index of the region to be searched is transmitted to the decoder. The final prediction signal is a linear combination of the blocks found through template matching from the given region. For a specific set of region sizes, the proposed inter method achieves an average Bjontegaard Delta bit rate gain of -3.15% for a random access configuration with 132% decoder and 201% encoder run-time respectively. The region sizes can be varied such that different trade-offs between coding gain and complexity can be obtained.

The H.265/MPEG-H High Efficiency Video Coding compliant encoding process faces the challenge of high computational complexity. Particularly, in the case of inter-picture prediction, most of the computational resources are allocated for the motion estimation (ME) process. In turn, ME and motion compensation enable improving coding efficiency by addressing the blocks of video frames as corresponding displacements from one or more reference blocks. These displacements do not necessarily have to be limited to integer sample positions, but may have an accuracy of half sample or quarter sample positions, which are identified during fractional sample refinement. In this paper, a context-based scheme for fractional sample refinement is proposed. The scheme takes the advantage of already obtained information in prior ME steps and provides significant flexibility in terms of parameterization. In this way, it adaptively achieves a desired tradeoff between computational complexity and coding efficiency. According to the experimental results obtained for an example algorithm utilizing the proposed framework, a significant decrease in the number of search points can be achieved. For instance, considering only 6 instead of 16 fractional sample positions results in a tradeoff of only 0.4% Bjøntegaard Delta-rate loss for high-definition video sequences compared with the conventional interpolation-and-search method.

For the entropy coding of independent and identically distributed (i.i.d.) binary sources, variable-to-variable length (V2V) codes are an interesting alternative to arithmetic coding. Such a V2V code translates variable length words of the source into variable length code words by employing two prefix-free codes. In this article, several properties of V2V codes are studied, and new concepts are developed. In particular, it is shown that the redundancy of a V2V code cannot be zero for a binary i.i.d. source {X} with 0 <pX(1) < 0.5. Furthermore, the concept of prime and composite V2V codes is proposed, and it is shown why composite V2V codes can be disregarded in the search for particular classes of minimum redundancy codes. Moreover, a canonical representation for V2V codes is proposed, which identifies V2V codes that have the same average code length function. It is shown how these concepts can be employed to greatly reduce the complexity of a search for minimum redundancy (size-limited) V2V codes.

Intra prediction is an important step in video coding, where spatial redundancy is exploited for coding efficiency. The state-of-the-art H.265/High Efficiency Video Coding (HEVC) standard provides 35 intra prediction modes including 33 angular modes and a DC as well as a Planar mode. Although these intra prediction methods are quite efficient, there is still room for improvements, especially when complex image structures are involved. Template matching was proposed previously for H.264/Advanced Video Coding (AVC) with up to 11% BD-rate gain. However substantial decoder complexity increase from the implied search algorithm makes it less attractive for practical applications. In this paper, we propose a fast template matching intra mode for HEVC. The proposed mode is scalable in the sense of providing different trade-offs between decoder complexity and BD-rate gains. For a specific choice of parameters, fast template matching mode can achieve an average BD-rate gain of 1.15% with a maximum BD-rate gain of up to 4.6% at a 33% increase in decoder run-time. Overall, fast template matching can provide the same coding gain as conventional template matching approaches, but with significantly less decoder complexity.

Transform coding tools in video coding have traditionally relied on the Discrete Cosine Transform Type II (DCT-II) to map residual signals to a new domain where quantization and entropy coding tools achieve a better coding efficiency than in the spatial domain. However, the DCT-II is not sufficient to model all different types of residual signals efficiently, especially in the intra-predicted blocks case. For this reason, the DST-VII was introduced in H.265/High Efficiency Video Coding (HEVC) in order to improve the compression performance of 4 × 4 intra-predicted blocks. In this paper we propose a multiple core transform approach, in which each transform is separable and generated by combining two one-dimensional transforms for the vertical and horizontal directions. The pair of 1-D transforms is selected from a set of three different types of Discrete Trigonometric Transforms and the Identity Transformation. Test results show that the proposed algorithm achieves bit rate reductions of 3% on average with respect to HEVC for intra-predicted residuals.

This work presents a coding efficiency evaluation of the recently published first release of the video coding scheme of the Alliance for Open Media (AOM), so called AOM/AV1, in comparison to the video coding standards H.264/MPEG-AVC (Advanced Video Coding) and H.265/MPEG-HEVC (High-Efficiency Video Coding). As representatives of the two last-mentioned video coding standards, the corresponding reference software encoders of JM and HM were selected, and for HEVC, in addition, the Fraunhofer HHI HEVC commercial software encoder and the open source software implementation x265 were used. According to the experimental results, which were obtained by using similar configurations for all examined representative encoders, the H.265/MPEG-HEVC reference software implementation provides significant average bit-rate savings of 38.4 % and 32.8 % compared to AOM/AV1 and H.264/MPEG-AVC, respectively. Particularly, when directly compared to H.264/MPEG-AVC High Profile, the AOM/AV1 encoder produces an average bit-rate overhead of 10.5% at the same objective quality. In addition, it was observed that the AOM/AV1 encoding times are quite similar to those of the full-fledged HM and JM reference software encoders. On the other hand, the typical encoding times of the HM encoder are in the range of 30-300 times higher on average than those measured for the configurable HHI HEVC encoder, depending on its chosen trade-off between encoding speed and coding efficiency.

The Range Extensions (RExt) of the High Efficiency Video Coding (HEVC) standard have recently been approved by both ITU-T and ISO/IEC. This set of extensions targets video coding applications in areas including content acquisition, postproduction, contribution, distribution, archiving, medical imaging, still imaging, and screen content. In addition to the functionality of HEVC Version 1, RExt provide support for monochrome, 4:2:2, and 4:4:4 chroma sampling formats as well as increased sample bit depths beyond 10 bits per sample. This extended functionality includes new coding tools with a view to provide additional coding efficiency, greater flexibility, and throughput at high bit depths/rates. Improved lossless, near-lossless, and very high bit-rate coding is also a part of the RExt scope. This paper presents the technical aspects of HEVC RExt, including a discussion of RExt profiles, tools, applications, and provides experimental results for a performance comparison with previous relevant coding technology. When compared with the High 4:4:4 Predictive Profile of H.264/Advanced Video Coding (AVC), the corresponding HEVC 4:4:4 RExt profile provides up to ~25$ %, ~32%, and ~36% average bit-rate reduction at the same PSNR quality level for intra, random access, and low delay configurations, respectively.

This paper presents an advanced depth intra-coding approach for 3D video coding based on the High Efficiency Video Coding (HEVC) standard and the multiview video plus depth (MVD) representation. This paper is motivated by the fact that depth signals have specific characteristics that differ from those of natural signals, i.e., camera-view video. Our approach replaces conventional intra-picture coding for the depth component, targeting a consistent and efficient support of 3D video applications that utilize depth maps or polygon meshes or both, with a high depth coding efficiency in terms of minimal artifacts in rendered views and meshes with a minimal number of triangles for a given bit rate. For this purpose, we introduce intra-picture prediction modes based on geometric primitives along with a residual coding method in the spatial domain, substituting conventional intra-prediction modes and transform coding, respectively. The results show that our solution achieves the same quality of rendered or synthesized views with about the same bit rate as MVD coding with the 3D video extension of HEVC (3D-HEVC) for high-quality depth maps and with about 8% less overall bit rate as with 3D-HEVC without related depth tools. At the same time, the combination of 3D video with 3D computer graphics content is substantially simplified, as the geometry-based depth intra signals can be represented as a surface mesh with about 85% less triangles, generated directly in the decoding process as an alternative decoder output.

The support for chroma formats other than 4;2;0 and bit depths higher than 10 bit has been included into the High Efficiency Video Coding (HEVC) standard by the publication of Version 2. In contrast to 4;2;2 profiles without additional dedicated coding tools relative to the Main profile, all 4;4;4 profiles specify a novel compression efficiency tool referred to as Cross-Component Prediction (CCP). This paper describes and analyses two additional extensions relative to the specified CCP variant in Version 2. In the first extension, an additional predictor is introduced, particularly, the first chroma component can also serve as the predictor for the second chroma component beside the luma component. The second extension is concerned with the binarization of the single prediction model parameter, which usually has a different distribution for R'G'B' than for Y'CbCrcontent. Experimental results show that the in this paper presented extensions can improve the compression efficiency by -0.7% on average for R'G'B' content, and by -1.1% for Y'CbCrcontent, respectively, both in terms of BD rate.

The first version of the High Efficiency Video Coding (HEVC) standard was approved by both ITU-T and ISO/IEC in 2013 and includes three profiles: Main and Main 10 for typical video data with 8 and 10 bits, respectively, as well as a profile referred to as Main Still Picture (MSP) profile. Apparently, the MSP profile extends the HEVC application space toward still images which, in turn, brings up the question of how this HEVC profile performs relative to existing still image coding technologies. This paper aims at addressing this question from a coding-efficiency point-of-view by presenting a rate-distortion performance analysis of the HEVC MSP profile in comparison to other popular still image and video compression schemes, including JPEG, JPEG 2000, JPEG XR, H.264/MPEG-4 AVC, VP8, VP9, and WebP. In summary, it can be stated that the HEVC MSP profile provides average bit-rate savings in the range from 10% to 44% relative to the whole set of competing video and still image compression schemes when averaged over a representative test set of photographic still images. Compared with Baseline JPEG alone, the average bit-rate saving for the HEVC MSP profile is 44%.

The work on support for higher bit depths and 4:2:2 as well as 4:4:4 chroma sampling formats for the High Efficiency Video Coding (HEVC) standard is currently being conducted under the term Range Extensions (RExt). A technique that exploits the correlation between residual color components in 4:4:4 chroma sampling format, also referred to as Cross-Component Prediction (CCP), has been adopted as part of the current RExt draft. In this paper, this relatively simple but yet effective CCP scheme is presented. Conceptually, CCP relies on the idea that an adaptively switched predictor based on a linear model is invoked for coding of the residuals of the second and third color component by using the residual of the first color component. Experimental results show that, depending on the underlying color space, average bit-rate savings in the range of 2-18% or 3-26% can be achieved by CCP for test sets of natural and screen content, respectively.

The paper describes a scalable video coding extension of the upcoming HEVC video coding standard for spatial and quality scalable coding. Besides coding tools known from scalable profiles of prior video coding standards, it includes new coding tools that further improve the enhancement layer coding efficiency. The effectiveness of the proposed scalable HEVC extension is demonstrated by comparing the coding efficiency to simulcast and single-layer coding for several test sequences and coding conditions.

This paper proposes an image interpolation algorithm exploiting sparse representation for natural images. It involves three steps: (a) obtaining an initial estimate of the high resolution image using linear methods like FIR filtering, (b) promoting sparsity in a selected dictionary through thresholding and (c) extracting high frequency information from the approximation and adding it to the initial estimate. For the sparse modeling, a shearlet dictionary is chosen to yield a multi-scale directional representation. The proposed algorithm is compared to several state-of-the-art methods to assess its objective as well as subjective performance. Compared to the cubic spline interpolation method, an average PSNR gain of around 0.7 dB is observed over a dataset of 200 images.

In this paper we present a new local and multiscale disparity estimation algorithm. Results show that the proposed method can preserve arbitrarily shaped depth discontinuities. Some results obtained from this method are shown as depth maps and synthetic views produced using the estimated disparity maps.

This work presents a performance comparison of the two latest video coding standards H.264/MPEG-AVC and H.265/MPEG-HEVC (High-Efficiency Video Coding) as well as the recently published proprietary video coding scheme VP9. According to the experimental results, which were obtained for a whole test set of video sequences by using similar encoding configurations for all three examined representative encoders, H.265/MPEG-HEVC provides significant average bit-rate savings of 43.3% and 39.3% relative to VP9 and H.264/MPEG-AVC, respectively. As a particular aspect of the conducted experiments, it turned out that the VP9 encoder produces an average bit-rate overhead of 8.4% at the same objective quality, when compared to an open H.264/MPEG-AVC encoder implementation - the x264 encoder. On the other hand, the typical encoding times of the VP9 encoder are more than 100 times higher than those measured for the x264 encoder. When compared to the full-fledged H.265/MPEG-HEVC reference software encoder implementation, the VP9 encoding times are lower by a factor of 7.35, on average.

This paper describes an extension of the high efficiency video coding (HEVC) standard for coding of multi-view video and depth data. In addition to the known concept of disparity-compensated prediction, inter-view motion parameter, and inter-view residual prediction for coding of the dependent video views are developed and integrated. Furthermore, for depth coding, new intra coding modes, a modified motion compensation and motion vector coding as well as the concept of motion parameter inheritance are part of the HEVC extension. A novel encoder control uses view synthesis optimization, which guarantees that high quality intermediate views can be generated based on the decoded data. The bitstream format supports the extraction of partial bitstreams, so that conventional 2D video, stereo video, and the full multi-view video plus depth format can be decoded from a single bitstream. Objective and subjective results are presented, demonstrating that the proposed approach provides 50% bit rate savings in comparison with HEVC simulcast and 20% in comparison with a straightforward multi-view extension of HEVC without the newly developed coding tools.

The state-of-the-art hybrid video coding standard H.264/AVC defines two entropy-coding schemes with different complexity-performance trade-offs. Supporting these two schemes within a single standard raises several problems ranging from higher efforts for product development to increased silicon costs for hardware implementations. To overcome these issues, this work proposes a unified and complexity-scalable entropy-coding framework that is based on PIPE/V2V. The proposed framework uses a single set of tools for all configurations and achieves the same complexity-performance trade-offs as the existing entropy-coding schemes through scalability.

This paper presents an approach for 3D video coding that uses a format in which a small number of views as well as associated depth maps are coded and transmitted. At the receiver side, additional views required for displaying the 3D video on an autostereoscopic display can be generated based on the corresponding decoded signals by using depth image based rendering (DIBR) techniques. In terms of coding technology, the proposed coding scheme represents an extension of High Efficiency Video Coding (HEVC), similar to the Multiview Coding (MVC) extension of H.264/AVC. Besides the well-known disparity-compensated prediction, advanced techniques for inter-view and inter-component prediction, the representation of depth blocks, and the encoder control for depth signals have been developed and integrated. In comparison to simulcasting the different signals using HEVC, the proposed approach provides about 40% and 50% average bit rate savings for a whole test set when configured to comply with a 2- and 3-view scenario, respectively. The proposed codec was submitted as response to a Call for Proposals on 3D Video Technology issued by the ISO/IEC Moving Picture Experts Group (MPEG) and it was ranked as the overall best performing HEVC-based proposal in the related subjective tests.

Adapting the resolution of motion compensated prediction in a video codec is considered in this paper. A new motion search and signaling scheme for increasing the accuracy of prediction without affecting the complexity of encoding is proposed. It involves using common information available to both encoder and decoder, e.g. current slice type, texture in reference pictures, etc as a cue to control motion vector accuracy and an efficient reuse of motion information for predicting subsequent blocks. An average bit rate reduction of around 2.5% for P-pictures and 0.5% for B-pictures is observed with no extra search compared to a fixed quarter-sample resolution.

This paper presents a new approach for 3D video coding, where the video and the depth component of an MVD representation are jointly coded in an integrated framework. This enables a new type of prediction for exploiting the correlations of video and depth signals in addition to existing methods for temporal and inter-view prediction. Our new method is referred to as inter-component prediction and we adopt it for predicting non-rectangular partitions in depth blocks. By dividing the block into two regions, each represented with a constant value, such block partitions are well-adapted to the characteristics of depth maps. The results show that this approach reduces the bit rate of the depth component by up to 11% and leads to an increased quality of rendered views.

Preliminary performance comparisons between the current test model encoder of the upcoming high efficiency video coding (HEVC) standard and an H.264/AVC High Profile conforming encoder report average bit-rate savings of about 40% and 25% for a random-access and an all-intra configuration, respectively. Even though H.264/AVC and HEVC are primarily designed for video coding applications, it might be interesting to find out how their corresponding intra-coding tools perform when applied to still images, especially when compared to current state-of-the-art still image coding standards. To this end, we evaluated the intra-only coding modes of HEVC and H.264/AVC together with popular image-coding standards like JPEG, JPEG 2000, and JPEG XR as well as the proprietary WebP scheme. By using a representative test set of photographic still images, we obtained a clear ranking in terms of objective coding performance with average bit-rate savings in the range of 17-44% for HEVC intra-only coding relative to all other competing coding schemes.

In hybrid video coding, an entropy coding scheme transmits the quantized transform coefficients, resulting from block-based transformation and quantization of the difference between the prediction signal and the original signal, and additional side information. The state-of-the-art hybrid video coding standard H.264/AVC defines two different entropy coding schemes with different complexity-performance tradeoff. As a result, the support for two different entropy coding schemes has to be maintained and introduces several problems. To overcome these issues, a unified solution is proposed, which is based on the PIPE/V2V coding concept. It achieves the same complexity-performance trade-offs as the existing entropy coding schemes by scalability. The advantage of the proposed scheme over the existing concept is the usage of the same set of tools for all configurations. Simulation results and complexity analysis on hardware show the efficiency of the proposed scheme.

The presented approach for 3D video coding uses the multiview video plus depth format, in which a small number of video views as well as associated depth maps are coded. Based on the coded signals, additional views required for displaying the 3D video on an autostereoscopic display can be generated by depth image based rendering techniques. The developed coding scheme represents an extension of HEVC, similar to the MVC extension of H.264/AVC. However, in addition to the well-known disparity-compensated prediction advanced techniques for inter-view and inter-component prediction, the representation of depth blocks, and the encoder control for depth signals have been integrated. In comparison to simulcasting the different signals using HEVC, the proposed approach provides about 40% and 50% bit rate savings for the tested configurations with 2 and 3 views, respectively. Bit rate reductions of about 20% have been obtained in comparison to a straightforward multiview extension of HEVC without the newly developed coding tools.

The joint development of the upcoming High Efficiency Video Coding (HEVC) standard by ITU-T Video Coding Experts Group and ISO/IEC Moving Picture Experts Group marks a new step in video compression capability. In technical terms, HEVC is a hybrid video-coding approach using quadtree-based block partitioning together with motion-compensated prediction. Even though a high degree of adaptability is achieved by quadtree-based block partitioning, this approach has certain intrinsic drawbacks, which may result in redundant sets of motion parameters being transmitted. Previous work has shown that those redundancies can effectively be removed by merging the leafs of a particular quadtree structure. Following this concept, a block merging algorithm for HEVC is now proposed. This algorithm generates a single motion parameter set for a whole region of contiguous motion-compensated blocks. In this paper, we describe the various components of the proposed block merging algorithm and, using experimental evidence, demonstrate their benefits in terms of coding efficiency.

The recently emerging probability interval partitioning entropy (PIPE) coding scheme offers high coding efficiency at a comparably low complexity level. In this paper, a new set of systematic variable-to-variable length (v2v) codes is proposed for use within the PIPE coding concept that allows the complexity requirements to be reduced even further. The proposed systematic v2v codes can be efficiently implemented by using simple counters instead of memory consuming tables. At the same time, the average number of operations per decoded binary symbol can be reduced by more than a factor of 2 relative to a fast multiplication-free binary arithmetic decoder. In terms of coding efficiency, experimental results show that in a typical video coding environment the average Bjontegaard delta (BD) rate increase is typically less than 0.5% when compared to the use of nearly optimal binary arithmetic codes.

In hybrid video coding, the difference between the intra or inter prediction signal and the original signal is transmitted using block-based transform coding. The state-of-the-art in coding the quantized transform coefficients is the approach specified in H.264/AVC for context-adaptive binary arithmetic coding. It has, however, been shown that the number of binary symbols that have to be arithmetically coded for the transform coefficients can become very large, making the concept less attractive for high rate applications. To overcome this issue, we propose a combination of simple variable-length codes and context-adaptive binary coding, which yields the same coding efficiency as the H.264/AVC transform coefficient coding at a lower complexity level and which has been adopted into the HEVC test model (HM).

Recently, ITU-T VCEG and ISO/IEC MPEG have started a new joint standardization activity on video coding, called High Efficiency Video Coding (HEVC). In this paper, the new transform coding techniques in the HEVC Test Model are de- scribed, including the residual quadtree (RQT) approach and coded block pattern signaling. Experimental results showing the advantage of using larger block size transforms, especially for high resolution video material, are presented. Furthermore, the trade-off between coding gain and encoder complexity in terms of the maximum RQT depth is discussed.

Quadtree-based block partitioning together with motion-compensated prediction has proven to be an efficient approach in video compression. However, when dealing with spatially neighboring blocks in uniformly displaced regions, quadtree-based partitioning may lead to redundant sets of transmitted motion parameters. This paper proposes and describes a simple but efficient block merging algorithm that aims at removing those redundancies by using only a single parameter set for a whole motion-compensated region of contiguous blocks. Simulation results show that our proposed merging technique works more efficiently than the conceptually similar direct mode as, e.g., specified in H.264/AVC. Due its efficiency and simplicity, our proposed merging approach has been adopted into the first test model of the high efficiency video coding (HEVC) standardization project, as currently pursued by ITU-T VCEG and ISO/IEC MPEG.

In this paper, we present a multipoint video conferencing system that adapts to heterogeneous members including mobiles. The system is built upon a low complexity scalable extension of our H.264 codec DAVC, and a congestion-aware dynamic adaptation layer. Our temporally scaled video codec DSVC has the same RD performance as the non-scaled version with comparable configuration. We achieve this by QP cascading, i.e., assigning gradually refining quantization parameters to the declining temporal layers. We present and analyse a mobile-compliant version of DSVC at reduced complexity that still admits comparable performance. Finally, we report on early work of dynamic layer tuning. Derived of delay variation measures, senders exploit scalable video layering to adapt the video transmission to varying network conditions. Initial results indicate that video performance remains close to optimal.

It is a well-known fact that, in order to overcome annoying blocking artifacts, transforms with block-overlapping basis functions have been proposed for image coding. Typically in transform coding, the encoder determines the transform coefficient values by applying the forward transform followed by scalar quantization. In this paper we present an approach, how rate-distortion optimized Lapped Biorthogonal Transform (LBT) coefficient values can be determined by solving ℓ1-regularized least squares problems. We compare a global version, where all the transform coefficients are obtained in one single optimization step, and a local version, where the optimization is done separately for each block, which results in losing optimality, but achieving highly reduced complexity. Our simulation results show gains of about 0.5 dB PSNR compared to ordinary forward transform and scalar quantization with only small losses (<; 0.1 dB) for the local variant.

This paper describes a novel video coding scheme that can be considered as a generalization of the block-based hybrid video coding approach of H.264/AVC. While the individual building blocks of our approach are kept simple similarly as in H.264/AVC, the flexibility of the block partitioning for prediction and transform coding has been substantially increased. This is achieved by the use of nested and pre-configurable quadtree structures, such that the block partitioning for temporal and spatial prediction as well as the space-frequency resolution of the corresponding prediction residual can be adapted to the given video signal in a highly flexible way. In addition, techniques for an improved motion representation as well as a novel entropy coding concept are included. The presented video codec was submitted to a Call for Proposals of ITU-T VCEG and ISO/IEC MPEG and was ranked among the five best performing proposals, both in terms of subjective and objective quality.

We present a novel approach to entropy coding, which provides the coding efficiency and simple probability modeling capability of arithmetic coding at the complexity level of Huffman coding. The key element of the proposed approach is a partitioning of the unit interval into a small set of probability intervals. An input sequence of discrete source symbols is mapped to a sequence of binary symbols and each of the binary symbols is assigned to one of the probability intervals. The binary symbols that are assigned to a particular probability interval are coded at a fixed probability using a simple code that maps a variable number of binary symbols to variable length codewords. The probability modeling is decoupled from the actual binary entropy coding. The coding efficiency of the probability interval partitioning entropy (PIPE) coding is comparable to that of arithmetic coding.

Recent investigations have shown that the support of extended block sizes for motion-compensated prediction and transform coding can significantly increase the coding efficiency for high-resolution video relative to H.264/AVC. In this paper, we present a new context-modeling scheme for the coding of transform coefficient levels that is particularly suitable for transform blocks greater than 8 × 8. While the basic concept for transform coefficient coding is similar to CABAC, the probability model selection has been optimized for larger block transforms. The proposed context modeling is compared to a straightforward extension of the CABAC context modeling; both schemes have been implemented in a hybrid video codec design that supports block sizes of up to 128 × 128 samples. In our simulations, we obtained overall bit rate reductions of up to 4%, with an average of 1.7% with the proposed context modeling scheme.

Abstract-A video coding architecture is described that is based on nested and pre-configurable quadtree structures for flexible and signal-adaptive picture partitioning. The primary goal of this partitioning concept is to provide a high degree of adaptability for both temporal and spatial prediction as well as for the purpose of space-frequency representation of prediction residuals. At the same time, a leaf merging mechanism is included in order to prevent excessive partitioning of a picture into prediction blocks and to reduce the amount of bits for signaling the prediction signal. For fractional-sample motion-compensated prediction, a fixed-point implementation of the maximal-order minimum-support algorithm is presented that uses a combination of infinite impulse response and FIR filtering. Entropy coding utilizes the concept of probability interval partitioning entropy codes that offers new ways for parallelization and enhanced throughput. The presented video coding scheme was submitted to a joint call for proposals of ITU-T Visual Coding Experts Group and ISO/IEC Moving Picture Experts Group and was ranked among the five best performing proposals, both in terms of subjective and objective quality.

For compression of 3-D dynamic meshes, the novel framework of so-called frame-based animated mesh compression (FAMC) has been introduced recently. In this context, we propose an efficient scheme for representation and statistical coding which is conceptually based on our previous work on context-based adaptive binary arithmetic coding (CABAC). After reviewing the basic principles of both CABAC and FAMC, we present suitable modifications and adaptations of both concepts in order to build an integrated solution with a high degree of coding efficiency. To this end, particular focus of our study has been put on the design of appropriate binarization and context modeling schemes. In our experiments, we obtained average bit-rate savings of more than 30% for a typical test set of dynamic meshes, when comparing the final design of our CABAC enriched FAMC scheme to the original version of FAMC using a conventional N-ary arithmetic coder. Our integrated approach has been adopted recently as part of the MPEG-4 Animated Framework extension (AFX).

Context-based adaptive binary arithmetic coding (CABAC) has proven to be an efficient technique in the area of video coding. This paper presents an approach for integrating CABAC into the framework of frame-based animated mesh compression (FAMC). It presents the specific modifications and adaptations that have been worked out to adapt CABAC to the specific requirements of FAMC in order to build a solution with a higher degree of coding efficiency. For a typical test set of animated meshes, average bit rate savings of 25% have been observed for the combination of CABAC and FAMC when compared to a previous version of FAMC using a conventional N-ary arithmetic coder. The presented approach has been recently adopted as part of the MPEG-4 AFX standard.

A rapidly growing number of carriers offer wireless video services to their customers, taking advantage of high quality video codecs implemented in dedicated hardware of selected mobile devices. In this paper we introduce a video conferencing software, which seamlessly integrates mobile with stationary users in a provider and device independent fashion. Innovations of this work are twofold. At first we report on a mobile realization of an H.264 video codec and its performance on a standard consumer Smartphone. Operating within the tight bounds of real-time compliance on mobiles, this software is an adapted version of a highly optimized H.264 codec. This DAVC codec, which we introduce along the line, significantly out-performs compatible H.264 realizations and allows for a scalable adaptation of its frame rate. In the second part we present a barrier-resistant peer-to-peer group communication scheme, which scales well for medium-size conferences and accounts for the heterogeneous nature of mobile and stationary participants. An outlook on mobility related group communication issues and future optimizations based on structured communication layers concludes the work.

Fidelity scalability involves the refinement of residual texture information. The entropy coding of texture refinement information in the scalable video coding (SVC) extension of H.264/AVC relies on a simple statistical model that is tuned to an encoder-specific way of quantization for generating a single fidelity enhancement layer on top of the backward compatible base layer. For fidelity enhancement layers above the first layer, we demonstrate how and why the current model fails to properly reflect the statistics of texture refinement information. By analyzing the specific properties of the typical quantization process in fidelity scalable coding of SVC, we are able to derive a generic modeling approach for coding of refinement symbols, independent of the specific choice of dead-zone parameters and classification rules. Experimental results for a broad range of quantization parameters show averaged bit-rate savings of around 5% (relative to the total bit rate) by using our proposed context modeling approach for a representative set of video sequences in a test scenario including up to three fidelity enhancement layers.

Scalable video coding (SVC) is an extension of H.264/AVC, which is currently being developed by the Joint Video Team. SVC supports any combination of temporal, spatial, and SNR scalability, but there is no special treatment for the case that two or more different bit-depths have to be supported by the scalable bit-stream. There is a number of applications (especially in the HD area) which would benefit from a design where an H.264/AVC conforming base layer with a sample bit-depth of typically 8 bit is extended in a backwards compatible way by an higher bit-depth enhancement layer. In this paper we present new techniques which can be used to achieve bit-depth scalability and show experimental results.

Coding of video sequences using hierarchical B pictures in the joint scalable video model (JSVM) for the scalability amendment of H.264/AVC has the benefit of improved rate distortion efficiency relative to other known temporal decomposition structures, besides providing temporal scalability. In the operational encoder control of the JSVM, the inter-picture dependencies within a hierarchical B picture structure are considered using a heuristic, where pictures that are more frequently used for motion compensation are coded with higher fidelity compared to pictures that are less often used for motion compensation. In this paper, we describe an approach where the dependencies introduced by motion compensation are also considered when selecting transform coefficient values. Our experimental results using an H.264/AVC conforming encoder show bit rate reductions of up to 10% compared to the quantization method used by JSVM.

With the introduction of the H.264/AVC video coding standard, significant improvements have recently been demonstrated in video compression capability. The Joint Video Team of the ITU-T VCEG and the ISO/IEC MPEG has now also standardized a Scalable Video Coding (SVC) extension of the H.264/AVC standard. SVC enables the transmission and decoding of partial bit streams to provide video services with lower temporal or spatial resolutions or reduced fidelity while retaining a reconstruction quality that is high relative to the rate of the partial bit streams. Hence, SVC provides functionalities such as graceful degradation in lossy transmission environments as well as bit rate, format, and power adaptation. These functionalities provide enhancements to transmission and storage applications. SVC has achieved significant improvements in coding efficiency with an increased degree of supported scalability relative to the scalable profiles of prior video coding standards. This paper provides an overview of the basic concepts for extending H.264/AVC towards SVC. Moreover, the basic tools for providing temporal, spatial, and quality scalability are described in detail and experimentally analyzed regarding their efficiency and complexity.

Block-based video-coding for 4:4:4 color sampling is extended by an adaptive color space transform. The presented technique enables an encoder to switch between several given color space representations in order to optimize rate-distortion performance. Simulations based on the current draft of the H.264/MPEG4-AVC 4:4:4 extensions demonstrate that our technique provides a rate-distortion performance equal or better than that obtained when using any of the individual color spaces only.

The scalable extension of H.264/MPEG4-AVC is a current standardization project of the joint video team of the ITU-T Video Coding Experts Group and the ISO/IEC moving picture experts group. This paper gives an overview of the design of the scalable H.264/MPEG4-AVC extension and describes the basic concepts for supporting temporal, spatial, and SNR scalability. The efficiency of the described concepts for providing spatial and SNR scalability is analyzed by means of simulation results and compared to H.264/MPEG4-AVC compliant single layer coding.

In this paper, an investigation of H.264/MPEG4-AVC conforming coding with hierarchical B pictures is presented. We analyze the coding delay and memory requirements, describe details of an improved encoder control, and compare the coding efficiency for different coding delays. Additionally, the coding efficiency of hierarchical B picture coding is compared to that of MCTF-based coding by using identical coding structures and a similar degree of encoder optimization. Our simulation results turned out that in comparison to the widely used IBBP...structure coding gains of more than 1 dB can be achieved at the expense of an increased coding delay. Further experiments have shown that the coding efficiency gains obtained by using the additional update steps in MCTF coding are generally smaller than the losses resulting from the required open-loop encoder control

The technical work on the first amendment of the H.264/MPEG4-AVC video coding standard has recently been completed. In these so-called fidelity range extensions (FRExt) a set of new coding tools is specified which is primarily targeted at providing significant improvements in coding efficiency for higher-fidelity video material. This paper presents an overview of the corresponding methods, briefly discusses some important aspects regarding profiles and applications, and finally provides experimental results for a performance comparison with existing coding technology.

The scalability extension of H.264/AVC uses an oversampled pyramid representation for spatial scalability, where for each spatial resolution a separate motion compensation or MCTF loop is deployed. When the reconstructed signal at a lower resolution is used to predict the next higher resolution, the motion compensation or MCTF loops including the deblocking filter operations of both resolutions have to be executed. This imposes a large complexity burden on the decoding of the higher resolution signals, especially when multiple spatial layers are utilized. In this paper, we investigate the approach to only allow prediction between spatial layers for parts of the lower resolution pictures that are intra-coded in order to avoid decoding that requires multiple motion compensation or MCTF loops. Experimental results evaluate the effectiveness of the proposed approach.

For the ongoing standardization activities for IP-based broadcast video and audio streaming services over DVB H, DMB or 3GPP’s MBMS the use of new forward error protection techniques is investigated, since these activities only propose the use of equal error protection schemes. Especially in distributing video hierarchical protection schemes can be used for more efficient content aware protection. Even older video coding standards allow simple quality reducing mechanism like temporal scalability. New scalable video coding schemes like the scalable extensions of H. 264/AVC can be used for a further enhancement of the protection schemes. In combination with an unequal erasure protection scheme the efficiency of network delivery can be enhanced significantly. We propose a scheme, which shows benefits compared to single layer stream protection schemes. The scheme allows protection over a larger range of error rates compared to single layer protection while showing “ graceful degradation” like behavior.

The scalability extension of H.264/AVC (H. Schwarz, et al., 2004) uses a layered approach for providing spatial, temporal, and SNR scalability. Due to this concept, only a restricted set of spatio-temporal-SNR points can be extracted and decoded from a global scalable bit-stream, and this set of points is defined by the chosen encoder configuration. In this paper, we present a new approach for providing flexible combined spatial, temporal, and SNR scalability. The increased flexibility is achieved by introducing NAL units that represent a refinement signal for a picture in a coarse-to-fine-description and can be truncated at any arbitrary point. The simulation results show that this approach is capable of providing flexible combined scalability while the coding efficiency is only slightly worse than that of the layered approach.

We present an SNR-scalable extension of the H.264/AVC video coding standard. To achieve an efficient SNR-scalable bit-stream representation of a video sequence, the temporal dependencies between pictures are exploited by using an open-loop subband approach. The related temporal analysis-synthesis filter bank structure is generalized to facilitate an adaptive block-based choice between the motion-compensated lifting representations of the Haar filter (uni-directional prediction) and the 5/3 filter (bi-directional prediction), both coupled with multiple-reference frame capabilities. As a remarkable feature of our approach, most components of H.264/AVC are used as specified in the standard, while only a few have been adjusted to the motion-compensated temporal filtering structure. Our proposed SNR-scalable extension was tested for a set of CIF sequences, and the results indicate that a coding efficiency comparable to that of the state-of-the-art H.264/AVC standard can be achieved.

A novel and highly efficient algorithm of multiplication-free binary arithmetic coding is proposed. Our proposed method relies on simple table lookups for performing the computationally critical operations of interval subdivision and probability estimation. Moreover, the underlying design principle provides a great flexibility for serving the different needs of all kind of coding applications where binary or binarized data have to be processed. A binary arithmetic coder of the type described in this paper has become part of the CABAC entropy coding scheme of the emerging H.264/AVC video coding standard. Experiments using this binary arithmetic coder in its native video coding environment demonstrate a superior coding efficiency as well as a significantly higher throughput rate in comparison to the MQ coder, which is currently being considered state-of-the-art in fast binary arithmetic coding.

A new algorithm for context modeling of binary sources with application to video compression is presented. Our proposed method is based on a tree rearrangement and tree selection process for an optimized modeling of binary context trees. We demonstrate its use for adaptive context-based coding of selected syntax elements in a video coder. For that purpose we apply our proposed technique to the H.264/AVC standard and evaluate its performance for different sources and different quantization parameters. Experimental results show that by using our proposed algorithm coding gains similar or superior to those obtained with the H.264/AVC CABAC algorithm is achieved.

Context-based adaptive binary arithmetic coding (CABAC) as a normative part of the new ITU-T/ISO/IEC standard H.264/AVC for video compression is presented. By combining an adaptive binary arithmetic coding technique with context modeling, a high degree of adaptation and redundancy reduction is achieved. The CABAC framework also includes a novel low-complexity method for binary arithmetic coding and probability estimation that is well suited for efficient hardware and software implementations. CABAC significantly outperforms the baseline entropy coding method of H.264/AVC for the typical area of envisaged target applications. For a set of test sequences representing typical material used in broadcast applications and for a range of acceptable video quality of about 30 to 38 dB, average bit-rate savings of 9%-14% are achieved.

In this paper a new adaptive entropy coding scheme for video compression is presented. It utilizes an adaptive arithmetic coding technique to better match the first order entropy of the coded symbols and to keep track of nonstationary symbol statistics. In addition, remaining symbol redundancies are exploited by context modeling to further reduce the bit-rate. A novel approach for coding of transform coefficients and a table look-up method for probability estimation and arithmetic coding is presented. Our new approach has been integrated in the current JVT test model (JM) to demonstrate the performance gain, and it was adopted as a part of the current JVT/H.26L draft.

We present a novel video coding approach which provides the functionality of fine granular bitstream scalability. The proposed progressive texture video coding (PTVC) scheme consists of two parts: the current ITU-T H.26L test model for coding macroblock mode and motion information only; a new technique for progressively coding all intra and inter texture information thus providing an embedded bitstream. It can still be decoded with moderate quality degradation in case of partial loss of the bitstream. It is therefore well suited for multicast or broadcast scenarios where time varying or different fixed bitrates are provided to the clients, as no transcoding or multiple coding process is required.

We present an efficient fine granular scalable video compression scheme which supports a fast bit rate adaptation independent of the encoder. The proposed scheme generates an embedded bitstream for each frame or, by appropriate multiplexing, for each group of picture. This rate-scalability is supported by an embedded bitstream which allows decoding at multiple rates, or to be more specific at virtually any rate. Drift removing techniques based on intra refresh or feedback mechanisms are presented. We show the potential of this video codec for variable bit rate channels as well as in combination with an unequal erasure protection scheme for error robust and efficient transmission over packet erasure channels. Possible enhancements of the presented coding schemes are discussed.

A new entropy coding scheme for video compression is presented. Context models are utilized for efficient prediction of the coding symbols. A novel binary adaptive arithmetic coding technique is employed to match the conditional entropy of the coding symbols given the context model estimates. The adaptation is also employed to keep track of non-stationary symbol statistics. Our new approach has been integrated into the current ITU-T H.26L test model (TML) to demonstrate the performance gain. By using our new entropy coding scheme instead of the variable length code approach of the current TML, large bit-rate savings up to 32% can be achieved. As a remarkable outcome of our experiments, we observed that high gains are reached not only at high bit-rates, but also at very low rates.

In this paper, we propose a wavelet-based image-coding scheme allowing lossless and lossy compression, simultaneously. Our two-layered approach utilizes the best of two worlds: it uses a highly performing wavelet-based or wavelet packet-based coding technique for lossy compression in the low bit range as a first stage. For the second (optional) stage, we extend the concept of reversible integer wavelet transforms to the more flexible class of adaptive reversible integer wavelet packet transforms which are based on the generation of a whole library of bases, from which the best representation for a given residue between the reconstructed lossy compressed image and the original image is chosen using a fast-search algorithm. We present experimental results demonstrating that our compression algorithm yields a rate-distortion performance similar or superior to the best currently published pure lossy still image-coding methods. At the same time, the lossless compression performance of our two-layered scheme is comparable to that of state-of-the-art pure lossless image-coding schemes. Compared to other combined lossy/lossless coding schemes such as the emerging JPEG-2000 still image-coding standard PSNR improvements up to 3 dB are achieved for a set of standard test images.

In this paper, we propose a very low bit-rate video coding scheme based on a discrete wavelet transform (DWT), block-matching motion estimation (BME), and overlapped block motion compensation (OBMC). Our approach reveals that the coding process works more efficiently if the quantized wavelet coefficients are preprocessed by a mechanism exploiting the redundancies in the wavelet subband structure. Thus, we introduce a new framework of preceding techniques based on the concepts of partitioning, aggregation, and conditional coding (PACC). Our experimental results show that our PACC coder outperforms the VM (Version 5.1) of MPEG4 both for the coding of intraframes (1-2 dB PSNR) and residual frames (up to 1.5 dB PSNR) of typical MPEG4 test sequences. The subjective quality of reconstructed video is, in general, superior to that obtained from the VM implementation. In addition, when restricted to the intraframe mode, the proposed coding algorithm produces results which are among the best reported for still image compression.

This paper describes a video coding algorithm that combines new ideas in motion estimation and wavelet-based coding techniques. A motion estimation technique using an image warping model is employed to reduce temporal redundancies in a given image sequence. The underlying continuous motion model is based on a bilinear warp of a grid of quadrangles. It has the advantage of dealing with more complex motion than simple block matching schemes. In addition, a block adaptive high-quality texture interpolation is used to further improve the quality of the prediction. The prediction error image is decorrelated using a wavelet transform coupled with a framework of highly efficient pre-coding techniques utilizing the concepts of partitioning, aggregation and conditional coding (PACC). This combined coding strategy has proved to be very efficient both with respect to subjective and objective measures. The improvements achieved with our proposed coder compared to the H.263 coder (with all options applied) range from 0.5 to 1.2 dB PSNR. Especially at very low bit-rates where reconstructed videos of block-based coders suffer from visually annoying blocking artifacts the proposed coding scheme produces a superior subjective quality.