This work investigates video encoding parallelization techniques based on the Versatile Video Coding (VVC) standard, using the open and optimized encoder software implementation VVenC. Modern multi-processor systems offer significant opportunities for accelerating video encoding. By employing a proposed combination of parallelization methods, the VVenC encoder achieves an acceleration factor of up to 22 compared to single-threaded mode on a 32-core system, with potential increases to 27× at higher bitrates. Building upon prior work on Inter-frame Parallelization (IFP), the study introduces frame region-based synchronization, enabling further acceleration of up to 10%. Beyond that, the study demonstrates extending frame parallelization beyond Group of Pictures (GOP) boundaries, which improves IFP speed up by 37% and 11% at high-definition (HD) and ultra-high-definition (UHD) resolutions, respectively. Additional combinations with other VVC parallelization tools, such as tiles and VVC Wavefront Parallel Processing (WPP), are also explored. The article provides a comprehensive analysis of parallelization challenges and highlights areas for further improvement.

Versatile Video Coding (VVC) offers substantial improvements in compression efficiency compared to its predecessor, High Efficiency Video Coding (HEVC). However, these gains come at the cost of significantly increased encoding complexity, particularly due to VVC's quadtree with nested multi-type tree (QTMT) partitioning structure. This complexity poses challenges for real-time and low-latency applications. This paper presents an enhanced split cost prediction framework designed to accelerate the VVC encoding process while preserving compression efficiency. Building on prior split cost prediction methods, our approach refines context modeling by incorporating Coding Unit (CU) size, prediction mode, and partitioning depth features. We construct a large-scale training dataset and optimize context-dependent predictors for multiple encoder presets. The proposed method is implemented in both Fraunhofer Versatile Video Encoder (VVenC) and the VVC Test Model (VTM), demonstrating encoding time reductions of up to 6% with negligible compression efficiency loss compared to the anchor using the benchmark method. Performance improvements are most significant in the medium and slower presets, where the partitioning search space is larger. Our results demonstrate that split cost prediction can be further optimized through feature-driven context modeling, eliminating the need for complex machine learning models, thereby making it practical for deployment in existing encoder pipelines.

VVenC, an open Versatile Video Coding (VVC) encoder, has recently been equipped with rate capping functionality in its two-pass rate control modes, providing constrained variable bitrate coding governed by target rate and maximum rate parameters. This paper reports on implementations and evaluation results of straightforward extensions to VVenC which enable the use of the maximum rate parameter also in the single-pass fixed-QP modes, controlled by a base quantization parameter (QP) instead of a target rate. The rate capping in the fixed-QP mode is achieved, with sufficient accuracy, by evaluating only already calculated pre-processing statistics, thereby avoiding increases in encoder runtime. This encoding mode, given that it supports visual quality optimizations such as XPSNR based block-wise perceptual QP adaptation, can be considered a rate capped constant-quality mode, which was missing in VVenC and which is an interesting configuration for video streaming.

As adaptive streaming becomes crucial for delivering high-quality video content across diverse network conditions, accurate metrics to assess perceptual quality are essential. This paper explores using the eXtended Peak Signal-to-Noise Ratio (XPSNR) metric as an alternative to the popular Video Multimethod Assessment Fusion (VMAF) metric for determining optimized bitrate-resolution pairs in the context of Versatile Video Coding (VVC). Our study is rooted in the observation that XPSNR shows a superior correlation with subjective quality scores for VVC-coded Ultra-High Definition (UHD) content compared to VMAF. We predict the average XPSNR of VVC-coded bitstreams using spatiotemporal complexity features of the video and the target encoding configuration and then determine the convex-hull online. On average, the proposed convex-hull using XPSNR (VEXUS) achieves an overall quality improvement of 5.84 dB PSNR and 0.62 dB XPSNR while maintaining the same bitrate, compared to the default UHD encoding using the VVenC encoder, accompanied by an encoding time reduction of 44.43% and a decoding time reduction of 65.46%. This shift towards XPSNR as a guiding metric shall enhance the effectiveness of adaptive streaming algorithms, ensuring an optimal balance between bitrate efficiency and perceptual fidelity with advanced video coding standards.

The Versatile Video Coding (VVC) standard promises high compression efficiency for diverse content types. Based on VVenC, an open and optimized VVC software video encoder, this work presents an inter-frame parallelization (IFP) method designed to exploit the processing power of modern platforms featuring a high number of computing cores. Encoding an ultrahigh definition video on a 32-core machine with the VVenC's faster preset, the proposed method shows more than 20% increase in encoder speed while only a 1% decrease in compression efficiency compared to the default multi-threading mode. In comparison to single-threaded mode, it corresponds to a speedup factor of 18, up from 15x achievable with the previous parallelization scheme. Furthermore, the synergy of the developed inter-frame parallelization technique with other parallelization methods is explored, including tiles and VVC wavefront parallel processing (WPP). The combination of these approaches enables a notable speedup factor of 21, albeit with a trade-off in coding efficiency. With a focus on VVC, this research contributes to the ongoing discourse on video coding optimization, providing valuable insights into possible pitfalls and the potential gains achievable through efficient parallelization techniques on high-core platforms.

Traditional per-title encoding schemes aim to optimize encoding resolutions to deliver the highest perceptual quality for each representation. XPSNR is observed to correlate better with the subjective quality of VVC-coded bitstreams. Towards this realization, we predict the average XPSNR of VVC-coded bitstreams using spatiotemporal complexity features of the video and the target encoding configuration using an XGBoost-based model. Based on the predicted XPSNR scores, we introduce a Quality-Aware Dynamic Resolution Adaptation (QADRA) framework for adaptive video streaming applications, where we determine the convex-hull online. Furthermore, keeping the encoding and decoding times within an acceptable threshold is mandatory for smooth and energy-efficient streaming. Hence, QADRA determines the encoding resolution and quantization parameter (QP) for each target bitrate by maximizing XPSNR while constraining the maximum encoding and/ or decoding time below a threshold. QADRA implements a JND-based representation elimination algorithm to remove perceptually redundant representations from the bitrate ladder. QADRA is an open-source Python-based framework published under the GNU GPLv3 license.

Rate control (RC), specifically two-pass, is the main operation mode in VVenC, an open and optimized Versatile Video Coding (VVC) encoder. VVC offers substantial bitrate savings over its predecessor, High Efficiency Video Coding (HEVC), at the price of increased complexity. This complexity increase is apparent in both encoding passes of VVenC. While the complexity redaction in the final pass has been discussed, this paper considers complexity reduction in the first pass, in addition to its already reduced search space. To reduce the overall runtime of a two-pass RC method, spatial and temporal sub-sampling of the first encoding pass is proposed. The experimental results show that the proposed first-pass sub-sampling in two-pass RC can speed up the encoding process of the default two-pass rate control algorithm in VVenC by 18%, with 0.48% loss in coding efficiency, when using the faster preset. Using temporal sub-sampling for the look-ahead, one-pass RC in VVenC can achieve time savings of 11% for bit-rate increases of 0.28%.

Pareto-front optimization is crucial for addressing the multi-objective challenges in video streaming, enabling the identification of optimal trade-offs between conflicting goals such as bitrate, video quality, and decoding complexity. This paper explores the construction of efficient bitrate ladders for adaptive Versatile Video Coding (VVC) streaming, focusing on optimizing these trade-offs. We investigate various ladder construction methods based on Pareto-front optimization, including exhaustive Rate-Quality and fixed ladder approaches. We propose a joint decoding time-rate-quality Pareto-front, providing a comprehensive framework to balance bitrate, decoding time, and video quality in video streaming. This allows streaming services to tailor their encoding strategies to meet specific requirements, prioritizing low decoding latency, bandwidth efficiency, or a balanced approach, thus enhancing the overall user experience. The experimental results confirm and demonstrate these opportunities for navigating the decoding time-rate-quality space to support various use cases. For example, when prioritizing low decoding latency, the proposed method achieves a decoding time reduction of 14.86 % while providing Bjøntegaard delta rate savings of 4.65 % and 0.32 dB improvement in the eXtended Peak Signal-to-Noise Ratio (XPSNR)-Rate domain over the traditional fixed ladder solution.

For improved performance, practical video encoders integrate algorithms for screen content detection and rate control. This paper outlines recently implemented optimizations to both the screen content classifier (SCC) and two-pass rate control (RC) of VVenC, an open Versatile Video Coding (VVC) compliant encoder. The improvements, confirmed by evaluation experiments in random-access configurations using an extended set of test videos, are mainly achieved by leveraging motion error statistics acquired during motion compensated temporal pre-filtering (MCTPF), carried out in VVenC’s pre-analysis stage. All three aspects – pre-analysis stage, SCC, and RC – are revisited herein, and the exploitation of MCTPF data is described.

Versatile Video Coding (VVC) allows for large compression efficiency gains over its predecessor, High Efficiency Video Coding (HEVC). The added efficiency comes at the cost of increased runtime complexity, especially for encoding. It is thus highly relevant to explore all available runtime reduction options. This paper proposes a novel first pass for two-pass rate control in all-intra configuration, using low-complexity video analysis and a Random Forest (RF)-based machine learning model to derive the data required for driving the second pass. The proposed method is validated using VVenC, an open and optimized VVC encoder. Compared to the default two-pass rate control algorithm in VVenC, the proposed method achieves around 32% reduction in encoding time for the preset faster, while on average only causing 2% BD-rate increase and achieving similar rate control accuracy.

Rate control (RC) schemes allow audio and video encoders to produce bitstreams according to specific overall bitrate constraints. However, when no rate capping is enforced, the instantaneous bitrate may vary strongly and may exceed the target rate by an order of magnitude, potentially causing playback stutter especially in video streaming scenarios. This paper introduces a rate capping extension for the two RC modes in VVenC, an open Versatile Video Coding (VVC) compliant encoder implementation. After a revisit of VVenC’s two-pass RC approach, the algorithmic details of the rate capping model are described. The paper concludes with an objective evaluation of the performance of the RC extension in a random-access configuration.

The optimization of the energy demand is crucial for modern video codecs. Previous studies show that the energy demand of VVC decoders can be improved by more than 50% if specific coding tools are disabled in the encoder. However, those approaches increase the bit rate by over 20% if the concept is applied to practical encoder implementations such as VVenC. Therefore, in this work, we investigate VVenC and study possibilities to reduce the additional bit rate, while still achieving low-energy decoding at reasonable encoding times. We show that encoding using our proposed coding tool profiles, the decoding energy efficiency is improved by over 25% with a bit rate increase of less than 5% with respect to standard encoding. Furthermore, we propose a second coding tool profile targeting maximum energy savings, which achieves 34% of energy savings at bitrate increases below 15%.

The Versatile Video Coding (VVC) standard has been developed to meet the ever-increasing demand for higher compression of digital video data. Compared to its predecessor, the High-Efficiency Video Coding (HEVC) standard, VVC reduces the bitrate by around 50% for the same perceived quality. This increase in compression efficiency is associated with an increase in computational complexity, mainly on the encoder side. As an open and optimized VVC software encoder implementation, VVenC integrates algorithmic optimizations for each coding tool in VVC. This allows to define a set of five presets from faster to slower as Pareto-optimal tradeoffs between runtime and efficiency. On top, multithreading allows to reduce the runtime and preserves most of the compression efficiency of each preset. This paper presents and analyses the different multi-threading strategies in VVenC. Using a combination of pre-processing, picture-level and in-picture parallelization, VVenC can achieve a parallelization speedup with a factor of 4 for 4 threads while reducing the compression efficiency by only 0.4%. For higher thread numbers, i.e. 16, the speedup depends on the video resolution and used encoder preset, ranging from 6-9 for high definition to 10-12 for ultrahigh definition video with similar loss of compression efficiency. Using additional wavefront and tiles in-picture parallelization, higher speedups can be achieved at the costs of decreased coding efficiency.

Contemporary motion picture content, consisting of scenes with different amounts of visual complexity or camera noise, represents demanding input for video encoders operating in rate control (RC) modes. This paper presents improvements to the 2-pass RC method integrated into VVenC, an open VVC encoder implementation, outlined in previous publications. We specifically introduce three extensions to our RC solution: first, frame type adaptation operating near scene cuts, along with an associated simple detector; second, rate stabilization means to allow for more reliable lookahead based 2-pass RC operation in on-the-fly encoding applications; and third, a low-complexity approach for estimating the instantaneous intensity of camera noise or film grain to avoid large variations in bit consumption when encoding individual frames in the final RC pass. Experimental evaluation confirms that these extensions significantly improve both the objective (BD rate) and subjective (visual) RC performance of VVenC especially on challenging video content.

VVenC, an open and optimized VVC encoder implementation, employs a temporal filter from the literature as a pre-processing step. The filter effectively reduces camera noise from input video, thereby increasing the encoding gain for lossy encoding, at a price of fairly high complexity, further increased by the necessity of consistent application to many pictures. The filter represents one of the most runtime consuming processing steps for the fastest operating points of VVenC. In this paper, steps are described to reduce the complexity of the temporal filtering in VVenC, to allow its application with low-complexity presets. Overall, the filter runtime is reduced by a factor of around 17 compared to the state of the art, while slightly improving its performance. An additional 4 times speedup is achieved using vectorized implementation. In the proposed version, for the VVenC preset faster, the filter provides 7.36% BD-rate gain at only 2% runtime overhead.

This paper presents a CNN to reduce the encoding time of a VVC-based intra-picture encoder. For encoding a 32 × 32 block, the CNN estimates two partitioning parameters that restrict the allowed coding block width and height. To estimate them such that the encoder skips testing inefficient partitioning modes, we train the CNN as follows: First, we generate training data by encoding sequences without the CNN. While encoding, we test all combinations of the two parameters for each 32 × 32 block and store the resulting Lagrangian rate-distortion-time (RDT) cost. We use the recorded cost to derive the loss function when training the CNN. Consequently, the CNN is trained such that it minimizes the Lagrangian RDT cost. Our CNN reduces the encoding time by 50% with a bit rate increase of 0.9%, which outperforms existing CNN-based approaches. Our generic training approach could also be applied for other encoder parameters.

This paper presents two new methods for fast VVC intra-picture encoding. Both are based on an approach that uses a CNN for blockadaptive parameter estimation. The parameters restrict the multitype-tree (MTT) partitionings tested by the encoder. The methods aim for an improvement of the approach by further constraints with additional parameters. Adding parameters increases the time required for training data generation exponentially. This raises the question which parameters to add and how. To explore further partitioning restrictions, the first method adds parameters controlling the block sizes the MTT can start from. Although this leads to four parameters, we can exploit that some of their combinations are invalid. To investigate whether testing fewer prediction and transform modes is feasible, the second method adds a single parameter that restricts their number jointly. The paper evaluates hypothetical and actual encoding time reductions for VTM-10.2. The first method outperforms our other and other existing method: The encoding time decreases by 50% with a bit rate increase of 0.7%.

Versatile Video Coding (VVC) is the most recent international video coding standard jointly developed by ITU-T and ISO/IEC, which has been finalized in July 2020. VVC allows for significant bit-rate reductions around 50% for the same subjective video quality compared to its predecessor, High Efficiency Video Coding (HEVC). One year after finalization, VVC support in devices and chipsets is still under development, which is aligned with the typical development cycles of new video coding standards. This paper presents open-source software packages that allow building a complete VVC end-to-end toolchain already one year after its finalization. This includes the Fraunhofer HHI VVenC library for fast and efficient VVC encoding as well as HHI's VVdeC library for live decoding. An experimental integration of VVC in the GPAC software tools and FFmpeg media framework allows packaging VVC bitstreams, e.g. encoded with VVenC, in MP4 file format and using DASH for content creation and streaming. The integration of VVdeC allows playback on the receiver. Given these packages, step-by-step tutorials are provided for two possible application scenarios: VVC file encoding plus playback and adaptive streaming with DASH.

The Versatile Video Coding (VVC) standard was finalized in July 2020. It has since been studied in the literature with regards to both compression efficiency and complexity. Most of the experiments regarding VVC were performed based on the VTM reference software using its full configuration, usually compared to the test model of its predecessor, High Efficiency Video Coding (HEVC), HM. In this paper, we present a comprehensive and in-depth analysis of the VVC encoding complexity by using the fast and efficient open VVC encoder implementation VVenC. As a practical non-reference encoder, it offers alternative tradeoffs between compression efficiency and complexity. To find those, we approximate the Pareto set of the configuration space in an iterative process, encompassing search space extending encoding tools as well as search space pruning fast algorithms. While the VVenC Pareto set is derived based on HD and UHD video encoding results, it is validated and improved for alternative input material, namely for low-resolution and screen content video. Based on this optimization, five presets are defined constituting approximate Pareto-optimal encoding points for various content types as well as for multi-threaded operation. Without multi-threading, the presets span working points from faster at 4.7% HM runtime at 11% BD-rate gain through medium providing 38% BD-rate gain over HM at a quarter of the HM runtime, up to slower providing all of VTM’s gain at less than half of its runtime.

Practical open-source video encoders implementing different video compression standards often provide various operation points for different trade-offs between compression efficiency and encoding time. This paper presents an evaluation of the trade-offs for the freely available open-source VVenC encoder conforming to the open and public Versatile Video Coding (VVC) standard together with x265 conforming to the open and public High Efficiency Video Coding (HEVC) standard and aomenc conforming to the AOM Video 1 (AV1) specification developed within an industry consortium. The evaluation includes the HEVC and VVC reference encoder implementations to analyze the tradeoffs provided by the practical encoders when focusing on high-resolution 10-bit consumer applications that require random access. Compared to the HEVC reference implementation HM, VVenC provides averaged objective bit-rate savings using the Peak-Signal-to-Noise-Ratio (PSNR) metric ranging from more than 37% at less than 68% encoding time to more than 9% at less than 5% of the HM encoding time. Sampling the data for an encoding time next to but lower than that of the HM and using HM's outcome as the anchor, VVenC provides 37.6% averaged bit-rate savings, followed by aomenc with 11.5%. In contrast to VVenC that provides the same performance as the VVC reference implementation VTM at a lower encoding time, x265 generates an average of 35.2% bit-rate overhead relative to its reference implementation HM and does not achieve the HM performance even for higher encoding times.

The user experience in adaptive HTTP streaming relies on offering bitrate ladders with suitable operation points for all users and typically involves multiple resolutions. While open GOP coding structures are generally known to provide substantial coding efficiency benefit, their use in HTTP streaming has been precluded through lacking support of reference picture resampling (RPR) in AVC and HEVC. The newly emerging Versatile Video Coding (VVC) standard supports RPR, but only conversational scenarios were primarily investigated during the design of VVC. This paper aims at enabling usage of RPR in HTTP streaming scenarios through analysing the drift potential of VVC coding tools and presenting a constrained encoding method that avoids severe drift artefacts in resolution switching with open GOP coding in VVC. In typical live streaming configurations, the presented method achieves -8.7% BD-rate reduction compared to closed GOP coding while in a typical Video on Demand configuration, -1.89% BD-rate reduction is reported. The constraints penalty compared to regular open GOP coding is 0.65% BD-rate in the worst case. The presented method was integrated into the publicly available open source VVC encoder VVenC v0.3.

This paper presents a new method for fast VVC intra-picture encoding using a CNN. The CNN operates on the original samples of$\mathbf{32}\times \mathbf{32}$blocks. Given a current block, it derives for each of the block's multi-type trees (MTTs), which are nested in quad-tree (QT) nodes, a parameter pair. The parameter pairs constrain the minimum width and height of the sub-blocks in their MTTs. This enables the CNN to control the number of tested MTT splits with fine granularity. To skip modes while maintaining the rate-distortion (RD) performance, we train the CNN considering the Lagrangian rate-distortion-time (RDT) cost caused by the derived parameters. First, we generate training data by encoding; when reaching a quad-tree node in a$\mathbf{32}\times \mathbf{32}$block, we encode the associated MTT with varying parameter pair values and record the resulting the RD and time cost. Then, when the CNN outputs parameters in training, we estimate the related RDT cost of the$\mathbf{32}\times \mathbf{32}$block using the recorded data. For this, we model the dependency between RDT cost and the parameters by emulating the encoder's RD optimization process. This way, we train the CNN while considering the RDT cost with an accuracy that is sufficient to outperform existing approaches. The approach achieves an encoding time reduction of 50% with a bit rate increase of only 0.7% for VTM-10.2.

In July 2020 the new video coding standard Versatile Video Coding (VVC) was released. VVC delivers bitrate savings of up to 50% relative to its predecessor High Efficiency Video Coding (HEVC) without compromising subjective quality. It has been designed for a broad range of applications and input material ranging from standard dynamic range to high dynamic range camera-captured content, screen content, and immersive applications. One of the key technologies included in the new standard is the flexible block partitioning using a quad-tree with multi-type tree structures (QT+MTT). Based on the VVC test model VTM, the VVenC project provides an open optimized VVC encoder. VVenC inherits and utilizes all of VTMs in-built partitioning search strategies. The QT+MTT partitioning search still poses a challenge for fast encoder implementations. After defining alternative low-depth partitioning configurations to VTM, it was observed that the initial search algorithm performs less optimal. We analyze and propose in this paper a set of additional fast search strategies for partitioning within VVenC, optimized for low-depth partitioning configurations. As a result, an average of 38% runtime reduction is achieved at the managable cost of less than 2% BD-rate loss.

Two-pass rate control (RC) schemes have proven useful for generating low-bitrate video-on-demand or streaming catalogs. Visually optimized encoding particularly using latest-generation coding standards like Versatile Video Coding (VVC), however, is still a subject of intensive study. This paper describes the two-pass RC method integrated into version 1 of VVenC, an open VVC encoding software. The RC design is based on a novel two-step rate-quantization parameter (R-QP) model to derive the second-pass coding parameters, and it uses the low-complexity XPSNR visual distortion measure to provide numerically as well as visually stable, perceptually R-D optimized encoding results. Random-access evaluation experiments confirm the improved objective as well as subjective performance of our RC solution.

A steady increase in available processing power continues to drive advances in video compression technology. The recently completed Versatile Video Coding (VVC) standard aims to double the compression efficiency of HEVC and deliver a same quality of video at half the bitrate. To achieve this goal, VVC includes several new methods that improve coding efficiency at the cost of increased complexity. This paper provides a complexity analysis of VVC and its VTM reference software. Whereas VVC is more complex than HEVC, it remains readily implementable in software on current generation processors. Performance of practical decoders are reported, showing that real-time decoding of 8K content is feasible. An encoder is also presented, showing that most of the compression gains of VVC over HEVC can be obtained at a small fraction of the resources needed by the VTM encoder under common test conditions.

Versatile Video Coding (VVC) is the emerging video coding standard to be finalized by the Joint Video Experts Team in July 2020. Compared to its predecessor, the High Efficiency Video Coding (HEVC) standard, VVC provides 50% bit-rate reduction at comparable visual quality for natural video content in high-definition (HD) and ultra high-definition (UHD) resolution. To achieve this, the standard incorporates more advanced and generalized algorithms, leading to an increase in computational complexity. This includes for example additional in-loop filters, decoder-side motion refinement and search as well as an increased number of transforms, which creates a grand challenge for implementers to achieve live decoding on general-purpose CPUs. In this paper, the work on an efficient software decoder implementation for the upcoming VVC standard is described, including optimization of sample operations using single instruction multiple data (SIMD) instructions and parallelization approaches with multithreading. As a result, the presented decoder can perform live decoding of 10bit HD video at 60 frames per second (fps) and 10bit UHD video at 30fps on modern mobile consumer hardware, showcasing that VVC live decoding is possible already right before finalization of the standard.

Versatile Video Coding (VVC) is a new international video coding standard to be finalized in July 2020. It is designed to provide around 50% bit-rate saving at the same subjective visual quality over its predecessor, High Efficiency Video Coding (H.265/HEVC). During the standard development, objective bit-rate savings of around 40% have been reported for the VVC reference software (VTM) compared to the HEVC reference software (HM). The unoptimized VTM encoder is around 9x, and the decoder around 2x, slower than HM. This paper discusses the VVC encoder complexity in terms of soft-ware runtime. The modular design of the standard allows a VVC encoder to trade off bit-rate savings and encoder runtime. Based on a detailed tradeoff analysis, results for different operating points are reported. Additionally, initial work on software and algorithm optimization is presented. With the optimized software algorithms, an operating point with an over 22x faster single-threaded encoder runtime than VTM can be achieved, i.e. around 2.5x faster than HM, while still providing more than 30% bit-rate savings over HM. Finally, our experiments demonstrate the flexibility of VVC and its potential for optimized soft-ware encoder implementations.

In this paper, we describe a video coding design that enables a higher coding efficiency than the HEVC standard. The proposed video codec follows the design of block-based hybrid video coding, but includes a number of advanced coding tools. A part of the incorporated advanced concepts was developed by the Joint Video Exploration Team, while others are newly proposed. The key aspects of these newly proposed tools are the following. A video frame is subdivided into rectangles of variable size using a binary partitioning with variable split ratios. Three new approaches for generating spatial intra prediction signals are supported: A line-wise application of conventional intra prediction modes, coupled with a mode-dependent processing order, a region-based template matching prediction method and intra prediction modes based on neural networks. For motion-compensated prediction, a multi-hypothesis mode with more than two motion hypotheses can be used. In transform coding, mode dependent combinations of primary and secondary transforms are applied. Moreover, scalar quantization is replaced by trellis-coded quantization and the entropy coding of the quantized transform coefficients is improved. The intra and inter prediction signals can be filtered using an edge-preserving diffusion filter or a non-linear DCT-based thresholding operation. The video codec includes an adaptive in-loop filter for which one of three classifiers can be chosen on a picture basis. We also incorporated an optional encoder control, which adjusts the quantization parameters based on a perceptually motivated distortion measure. In a random access scenario, our proposed video codec achieves luma BD-rate savings between 32.5% for HDR HLG UHD and 39.6% for SDR UHD over the HEVC (HM software) anchor for different categories of test sequences.

Versatile Video Coding (VVC) is going to be the successor of High Efficiency Video Coding (HEVC). The new standard (currently under development) requires the adoption of tools that provide a good trade-off between gain and implementation complexity. In previous work we proposed a Line-Based Intra Prediction (LIP) algorithm to improve the coding performance of the upcoming standard. However, that contribution did not sufficiently take into account the hardware-related complexity of the method. For this reason, this publication addresses these problems and proposes several modifications of the algorithm to solve them. Experimental results show an average gain of 1.0% and 0.5% for the All Intra (AI) and Random Access (RA) configurations, respectively.

One of the innovations in H.265/HEVC is the quad-tree partitioning framework. It allows flexible block subdivision and mode allocation across the encoded picture. The increased flexibility comes at a cost of vast search space expansion, making exhaustive search algorithms inapplicable. We propose a novel early termination condition to skip the exhaustive search of whole tree-branches in the well-established top-down encoding approach. The condition is based on a simple and intuitive split cost prediction. It can be parametrized to control the trade-off between the speed-up and caused BD-rate loss. Data driven parameter estimation and parameter number reduction is presented. For random-access encoding, the method can achieve an average speed-up of 30% with a BD-rate loss of 0.03%. At another trade-off point, speed-up is increased to over 40% for a BD-rate loss below 0.5%.

One of the key technologies in the upcoming Versatile Video Coding (VVC) standard is the flexible partitioning. Compared to HEVC, partitioning alone accounts for about 8.5% bitrate savings. The added flexibility comes at a cost of large search space expansion.This paper gives an overview of fast encoder strategies for selecting the block partitioning in the Test Model VTM-3.0 for VVC. In total, 13 approaches for speeding up the block partitioning selection in an encoder are described; 8 of these techniques were developed by the authors. 10 of the 13 methods are used in the common test conditions of the Joint Video Experts Team (JVET); these 10 methods achieve an overall speed-up factor of 7 at a bit rate increase of only 1.1%. By enabling 12 approaches, a speed up factor of over 8.5 is obtained at a bit rate increase of about 1.5%.

Block partitioning is the basis of every modern hybrid video coding standard. It specifies how the video pictures can be subdivided into blocks for prediction and residual coding. In H.265/HEVC, quad-tree partitioning is one of the key technologies allowing for flexible mode allocation and providing a substantial part of the gains over H.264/AVC. The current draft of the upcoming standard Versatile Video Coding (VVC) provides over 30% bit-rate savings over HEVC and almost one third of the gain is achieved by using a more flexible partitioning scheme than the quad-tree partitioning used in HEVC. In this paper, we describe a partitioning concept that generalizes many of the ideas developed during the exploration and early standardization phase of VVC. In fact, our method includes the VVC partitioning as well as many other state-of-the-art methods. The proposed method can be implemented in a fully configurable design. For instance, it can be configured to match the performance of VTM-1.0 at much faster runtime (69%) or it can be configured to obtain additional bit-rate savings of up to 3% by exploiting additional degrees of freedom.