This work investigates video encoding parallelization techniques based on the Versatile Video Coding (VVC) standard, using the open and optimized encoder software implementation VVenC. Modern multi-processor systems offer significant opportunities for accelerating video encoding. By employing a proposed combination of parallelization methods, the VVenC encoder achieves an acceleration factor of up to 22 compared to single-threaded mode on a 32-core system, with potential increases to 27× at higher bitrates. Building upon prior work on Inter-frame Parallelization (IFP), the study introduces frame region-based synchronization, enabling further acceleration of up to 10%. Beyond that, the study demonstrates extending frame parallelization beyond Group of Pictures (GOP) boundaries, which improves IFP speed up by 37% and 11% at high-definition (HD) and ultra-high-definition (UHD) resolutions, respectively. Additional combinations with other VVC parallelization tools, such as tiles and VVC Wavefront Parallel Processing (WPP), are also explored. The article provides a comprehensive analysis of parallelization challenges and highlights areas for further improvement.

Versatile Video Coding (VVC) offers substantial improvements in compression efficiency compared to its predecessor, High Efficiency Video Coding (HEVC). However, these gains come at the cost of significantly increased encoding complexity, particularly due to VVC's quadtree with nested multi-type tree (QTMT) partitioning structure. This complexity poses challenges for real-time and low-latency applications. This paper presents an enhanced split cost prediction framework designed to accelerate the VVC encoding process while preserving compression efficiency. Building on prior split cost prediction methods, our approach refines context modeling by incorporating Coding Unit (CU) size, prediction mode, and partitioning depth features. We construct a large-scale training dataset and optimize context-dependent predictors for multiple encoder presets. The proposed method is implemented in both Fraunhofer Versatile Video Encoder (VVenC) and the VVC Test Model (VTM), demonstrating encoding time reductions of up to 6% with negligible compression efficiency loss compared to the anchor using the benchmark method. Performance improvements are most significant in the medium and slower presets, where the partitioning search space is larger. Our results demonstrate that split cost prediction can be further optimized through feature-driven context modeling, eliminating the need for complex machine learning models, thereby making it practical for deployment in existing encoder pipelines.

VVenC, an open Versatile Video Coding (VVC) encoder, has recently been equipped with rate capping functionality in its two-pass rate control modes, providing constrained variable bitrate coding governed by target rate and maximum rate parameters. This paper reports on implementations and evaluation results of straightforward extensions to VVenC which enable the use of the maximum rate parameter also in the single-pass fixed-QP modes, controlled by a base quantization parameter (QP) instead of a target rate. The rate capping in the fixed-QP mode is achieved, with sufficient accuracy, by evaluating only already calculated pre-processing statistics, thereby avoiding increases in encoder runtime. This encoding mode, given that it supports visual quality optimizations such as XPSNR based block-wise perceptual QP adaptation, can be considered a rate capped constant-quality mode, which was missing in VVenC and which is an interesting configuration for video streaming.

As adaptive streaming becomes crucial for delivering high-quality video content across diverse network conditions, accurate metrics to assess perceptual quality are essential. This paper explores using the eXtended Peak Signal-to-Noise Ratio (XPSNR) metric as an alternative to the popular Video Multimethod Assessment Fusion (VMAF) metric for determining optimized bitrate-resolution pairs in the context of Versatile Video Coding (VVC). Our study is rooted in the observation that XPSNR shows a superior correlation with subjective quality scores for VVC-coded Ultra-High Definition (UHD) content compared to VMAF. We predict the average XPSNR of VVC-coded bitstreams using spatiotemporal complexity features of the video and the target encoding configuration and then determine the convex-hull online. On average, the proposed convex-hull using XPSNR (VEXUS) achieves an overall quality improvement of 5.84 dB PSNR and 0.62 dB XPSNR while maintaining the same bitrate, compared to the default UHD encoding using the VVenC encoder, accompanied by an encoding time reduction of 44.43% and a decoding time reduction of 65.46%. This shift towards XPSNR as a guiding metric shall enhance the effectiveness of adaptive streaming algorithms, ensuring an optimal balance between bitrate efficiency and perceptual fidelity with advanced video coding standards.

The Versatile Video Coding (VVC) standard promises high compression efficiency for diverse content types. Based on VVenC, an open and optimized VVC software video encoder, this work presents an inter-frame parallelization (IFP) method designed to exploit the processing power of modern platforms featuring a high number of computing cores. Encoding an ultrahigh definition video on a 32-core machine with the VVenC's faster preset, the proposed method shows more than 20% increase in encoder speed while only a 1% decrease in compression efficiency compared to the default multi-threading mode. In comparison to single-threaded mode, it corresponds to a speedup factor of 18, up from 15x achievable with the previous parallelization scheme. Furthermore, the synergy of the developed inter-frame parallelization technique with other parallelization methods is explored, including tiles and VVC wavefront parallel processing (WPP). The combination of these approaches enables a notable speedup factor of 21, albeit with a trade-off in coding efficiency. With a focus on VVC, this research contributes to the ongoing discourse on video coding optimization, providing valuable insights into possible pitfalls and the potential gains achievable through efficient parallelization techniques on high-core platforms.

Traditional per-title encoding schemes aim to optimize encoding resolutions to deliver the highest perceptual quality for each representation. XPSNR is observed to correlate better with the subjective quality of VVC-coded bitstreams. Towards this realization, we predict the average XPSNR of VVC-coded bitstreams using spatiotemporal complexity features of the video and the target encoding configuration using an XGBoost-based model. Based on the predicted XPSNR scores, we introduce a Quality-Aware Dynamic Resolution Adaptation (QADRA) framework for adaptive video streaming applications, where we determine the convex-hull online. Furthermore, keeping the encoding and decoding times within an acceptable threshold is mandatory for smooth and energy-efficient streaming. Hence, QADRA determines the encoding resolution and quantization parameter (QP) for each target bitrate by maximizing XPSNR while constraining the maximum encoding and/ or decoding time below a threshold. QADRA implements a JND-based representation elimination algorithm to remove perceptually redundant representations from the bitrate ladder. QADRA is an open-source Python-based framework published under the GNU GPLv3 license.

Rate control (RC), specifically two-pass, is the main operation mode in VVenC, an open and optimized Versatile Video Coding (VVC) encoder. VVC offers substantial bitrate savings over its predecessor, High Efficiency Video Coding (HEVC), at the price of increased complexity. This complexity increase is apparent in both encoding passes of VVenC. While the complexity redaction in the final pass has been discussed, this paper considers complexity reduction in the first pass, in addition to its already reduced search space. To reduce the overall runtime of a two-pass RC method, spatial and temporal sub-sampling of the first encoding pass is proposed. The experimental results show that the proposed first-pass sub-sampling in two-pass RC can speed up the encoding process of the default two-pass rate control algorithm in VVenC by 18%, with 0.48% loss in coding efficiency, when using the faster preset. Using temporal sub-sampling for the look-ahead, one-pass RC in VVenC can achieve time savings of 11% for bit-rate increases of 0.28%.

Conventional per-title encoding schemes strive to optimize encoding resolutions to deliver the utmost perceptual quality for each bitrate ladder representation. Nevertheless, maintaining encoding time within an acceptable threshold is equally imperative in online streaming applications. Further-more, modern client devices are equipped with the capability for fast deep-learning-based video super-resolution (VSR) techniques, enhancing the perceptual quality of the decoded bitstream. This suggests that opting for lower resolutions in representations during the encoding process can curtail the overall energy consumption without substantially compromising perceptual quality. In this context, this paper introduces a video super-resolution-based latency-aware optimized bitrate encoding scheme (ViSOR) designed for online adaptive streaming applications. ViSOR determines the encoding resolution for each target bitrate, ensuring the highest achievable perceptual quality after VSR within the bound of a maximum acceptable latency. Random forest-based prediction models are trained to predict the perceptual quality after VSR and the encoding time for each resolution using the spatiotemporal features extracted for each video segment. Experimental results show that ViSOR targeting fast super-resolution convolutional neural network (FSRCNN) achieves an overall average bitrate reduction of 24.65% and 32.70% to maintain the same PSNR and VMAF, compared to the HTTP Live Streaming (HLS) bitrate ladder encoding of 4s segments using the x265 encoder, when the maximum acceptable latency for each representation is set as two seconds. Considering a just noticeable difference (JND) of six VMAF points, the average cumulative storage consumption and encoding energy for each segment is reduced by 79.32% and 68.21%, respectively, contributing towards greener streaming.

Pareto-front optimization is crucial for addressing the multi-objective challenges in video streaming, enabling the identification of optimal trade-offs between conflicting goals such as bitrate, video quality, and decoding complexity. This paper explores the construction of efficient bitrate ladders for adaptive Versatile Video Coding (VVC) streaming, focusing on optimizing these trade-offs. We investigate various ladder construction methods based on Pareto-front optimization, including exhaustive Rate-Quality and fixed ladder approaches. We propose a joint decoding time-rate-quality Pareto-front, providing a comprehensive framework to balance bitrate, decoding time, and video quality in video streaming. This allows streaming services to tailor their encoding strategies to meet specific requirements, prioritizing low decoding latency, bandwidth efficiency, or a balanced approach, thus enhancing the overall user experience. The experimental results confirm and demonstrate these opportunities for navigating the decoding time-rate-quality space to support various use cases. For example, when prioritizing low decoding latency, the proposed method achieves a decoding time reduction of 14.86 % while providing Bjøntegaard delta rate savings of 4.65 % and 0.32 dB improvement in the eXtended Peak Signal-to-Noise Ratio (XPSNR)-Rate domain over the traditional fixed ladder solution.

For improved performance, practical video encoders integrate algorithms for screen content detection and rate control. This paper outlines recently implemented optimizations to both the screen content classifier (SCC) and two-pass rate control (RC) of VVenC, an open Versatile Video Coding (VVC) compliant encoder. The improvements, confirmed by evaluation experiments in random-access configurations using an extended set of test videos, are mainly achieved by leveraging motion error statistics acquired during motion compensated temporal pre-filtering (MCTPF), carried out in VVenC’s pre-analysis stage. All three aspects – pre-analysis stage, SCC, and RC – are revisited herein, and the exploitation of MCTPF data is described.

Versatile Video Coding (VVC) allows for large compression efficiency gains over its predecessor, High Efficiency Video Coding (HEVC). The added efficiency comes at the cost of increased runtime complexity, especially for encoding. It is thus highly relevant to explore all available runtime reduction options. This paper proposes a novel first pass for two-pass rate control in all-intra configuration, using low-complexity video analysis and a Random Forest (RF)-based machine learning model to derive the data required for driving the second pass. The proposed method is validated using VVenC, an open and optimized VVC encoder. Compared to the default two-pass rate control algorithm in VVenC, the proposed method achieves around 32% reduction in encoding time for the preset faster, while on average only causing 2% BD-rate increase and achieving similar rate control accuracy.

Rate control (RC) schemes allow audio and video encoders to produce bitstreams according to specific overall bitrate constraints. However, when no rate capping is enforced, the instantaneous bitrate may vary strongly and may exceed the target rate by an order of magnitude, potentially causing playback stutter especially in video streaming scenarios. This paper introduces a rate capping extension for the two RC modes in VVenC, an open Versatile Video Coding (VVC) compliant encoder implementation. After a revisit of VVenC’s two-pass RC approach, the algorithmic details of the rate capping model are described. The paper concludes with an objective evaluation of the performance of the RC extension in a random-access configuration.

The optimization of the energy demand is crucial for modern video codecs. Previous studies show that the energy demand of VVC decoders can be improved by more than 50% if specific coding tools are disabled in the encoder. However, those approaches increase the bit rate by over 20% if the concept is applied to practical encoder implementations such as VVenC. Therefore, in this work, we investigate VVenC and study possibilities to reduce the additional bit rate, while still achieving low-energy decoding at reasonable encoding times. We show that encoding using our proposed coding tool profiles, the decoding energy efficiency is improved by over 25% with a bit rate increase of less than 5% with respect to standard encoding. Furthermore, we propose a second coding tool profile targeting maximum energy savings, which achieves 34% of energy savings at bitrate increases below 15%.

The Versatile Video Coding (VVC) standard has been developed to meet the ever-increasing demand for higher compression of digital video data. Compared to its predecessor, the High-Efficiency Video Coding (HEVC) standard, VVC reduces the bitrate by around 50% for the same perceived quality. This increase in compression efficiency is associated with an increase in computational complexity, mainly on the encoder side. As an open and optimized VVC software encoder implementation, VVenC integrates algorithmic optimizations for each coding tool in VVC. This allows to define a set of five presets from faster to slower as Pareto-optimal tradeoffs between runtime and efficiency. On top, multithreading allows to reduce the runtime and preserves most of the compression efficiency of each preset. This paper presents and analyses the different multi-threading strategies in VVenC. Using a combination of pre-processing, picture-level and in-picture parallelization, VVenC can achieve a parallelization speedup with a factor of 4 for 4 threads while reducing the compression efficiency by only 0.4%. For higher thread numbers, i.e. 16, the speedup depends on the video resolution and used encoder preset, ranging from 6-9 for high definition to 10-12 for ultrahigh definition video with similar loss of compression efficiency. Using additional wavefront and tiles in-picture parallelization, higher speedups can be achieved at the costs of decreased coding efficiency.

Contemporary motion picture content, consisting of scenes with different amounts of visual complexity or camera noise, represents demanding input for video encoders operating in rate control (RC) modes. This paper presents improvements to the 2-pass RC method integrated into VVenC, an open VVC encoder implementation, outlined in previous publications. We specifically introduce three extensions to our RC solution: first, frame type adaptation operating near scene cuts, along with an associated simple detector; second, rate stabilization means to allow for more reliable lookahead based 2-pass RC operation in on-the-fly encoding applications; and third, a low-complexity approach for estimating the instantaneous intensity of camera noise or film grain to avoid large variations in bit consumption when encoding individual frames in the final RC pass. Experimental evaluation confirms that these extensions significantly improve both the objective (BD rate) and subjective (visual) RC performance of VVenC especially on challenging video content.

VVenC, an open and optimized VVC encoder implementation, employs a temporal filter from the literature as a pre-processing step. The filter effectively reduces camera noise from input video, thereby increasing the encoding gain for lossy encoding, at a price of fairly high complexity, further increased by the necessity of consistent application to many pictures. The filter represents one of the most runtime consuming processing steps for the fastest operating points of VVenC. In this paper, steps are described to reduce the complexity of the temporal filtering in VVenC, to allow its application with low-complexity presets. Overall, the filter runtime is reduced by a factor of around 17 compared to the state of the art, while slightly improving its performance. An additional 4 times speedup is achieved using vectorized implementation. In the proposed version, for the VVenC preset faster, the filter provides 7.36% BD-rate gain at only 2% runtime overhead.

Versatile Video Coding (VVC) is the most recent international video coding standard jointly developed by ITU-T and ISO/IEC, which has been finalized in July 2020. VVC allows for significant bit-rate reductions around 50% for the same subjective video quality compared to its predecessor, High Efficiency Video Coding (HEVC). One year after finalization, VVC support in devices and chipsets is still under development, which is aligned with the typical development cycles of new video coding standards. This paper presents open-source software packages that allow building a complete VVC end-to-end toolchain already one year after its finalization. This includes the Fraunhofer HHI VVenC library for fast and efficient VVC encoding as well as HHI's VVdeC library for live decoding. An experimental integration of VVC in the GPAC software tools and FFmpeg media framework allows packaging VVC bitstreams, e.g. encoded with VVenC, in MP4 file format and using DASH for content creation and streaming. The integration of VVdeC allows playback on the receiver. Given these packages, step-by-step tutorials are provided for two possible application scenarios: VVC file encoding plus playback and adaptive streaming with DASH.

The Versatile Video Coding (VVC) standard was finalized in July 2020. It has since been studied in the literature with regards to both compression efficiency and complexity. Most of the experiments regarding VVC were performed based on the VTM reference software using its full configuration, usually compared to the test model of its predecessor, High Efficiency Video Coding (HEVC), HM. In this paper, we present a comprehensive and in-depth analysis of the VVC encoding complexity by using the fast and efficient open VVC encoder implementation VVenC. As a practical non-reference encoder, it offers alternative tradeoffs between compression efficiency and complexity. To find those, we approximate the Pareto set of the configuration space in an iterative process, encompassing search space extending encoding tools as well as search space pruning fast algorithms. While the VVenC Pareto set is derived based on HD and UHD video encoding results, it is validated and improved for alternative input material, namely for low-resolution and screen content video. Based on this optimization, five presets are defined constituting approximate Pareto-optimal encoding points for various content types as well as for multi-threaded operation. Without multi-threading, the presets span working points from faster at 4.7% HM runtime at 11% BD-rate gain through medium providing 38% BD-rate gain over HM at a quarter of the HM runtime, up to slower providing all of VTM’s gain at less than half of its runtime.

Practical open-source video encoders implementing different video compression standards often provide various operation points for different trade-offs between compression efficiency and encoding time. This paper presents an evaluation of the trade-offs for the freely available open-source VVenC encoder conforming to the open and public Versatile Video Coding (VVC) standard together with x265 conforming to the open and public High Efficiency Video Coding (HEVC) standard and aomenc conforming to the AOM Video 1 (AV1) specification developed within an industry consortium. The evaluation includes the HEVC and VVC reference encoder implementations to analyze the tradeoffs provided by the practical encoders when focusing on high-resolution 10-bit consumer applications that require random access. Compared to the HEVC reference implementation HM, VVenC provides averaged objective bit-rate savings using the Peak-Signal-to-Noise-Ratio (PSNR) metric ranging from more than 37% at less than 68% encoding time to more than 9% at less than 5% of the HM encoding time. Sampling the data for an encoding time next to but lower than that of the HM and using HM's outcome as the anchor, VVenC provides 37.6% averaged bit-rate savings, followed by aomenc with 11.5%. In contrast to VVenC that provides the same performance as the VVC reference implementation VTM at a lower encoding time, x265 generates an average of 35.2% bit-rate overhead relative to its reference implementation HM and does not achieve the HM performance even for higher encoding times.

The user experience in adaptive HTTP streaming relies on offering bitrate ladders with suitable operation points for all users and typically involves multiple resolutions. While open GOP coding structures are generally known to provide substantial coding efficiency benefit, their use in HTTP streaming has been precluded through lacking support of reference picture resampling (RPR) in AVC and HEVC. The newly emerging Versatile Video Coding (VVC) standard supports RPR, but only conversational scenarios were primarily investigated during the design of VVC. This paper aims at enabling usage of RPR in HTTP streaming scenarios through analysing the drift potential of VVC coding tools and presenting a constrained encoding method that avoids severe drift artefacts in resolution switching with open GOP coding in VVC. In typical live streaming configurations, the presented method achieves -8.7% BD-rate reduction compared to closed GOP coding while in a typical Video on Demand configuration, -1.89% BD-rate reduction is reported. The constraints penalty compared to regular open GOP coding is 0.65% BD-rate in the worst case. The presented method was integrated into the publicly available open source VVC encoder VVenC v0.3.

In July 2020 the new video coding standard Versatile Video Coding (VVC) was released. VVC delivers bitrate savings of up to 50% relative to its predecessor High Efficiency Video Coding (HEVC) without compromising subjective quality. It has been designed for a broad range of applications and input material ranging from standard dynamic range to high dynamic range camera-captured content, screen content, and immersive applications. One of the key technologies included in the new standard is the flexible block partitioning using a quad-tree with multi-type tree structures (QT+MTT). Based on the VVC test model VTM, the VVenC project provides an open optimized VVC encoder. VVenC inherits and utilizes all of VTMs in-built partitioning search strategies. The QT+MTT partitioning search still poses a challenge for fast encoder implementations. After defining alternative low-depth partitioning configurations to VTM, it was observed that the initial search algorithm performs less optimal. We analyze and propose in this paper a set of additional fast search strategies for partitioning within VVenC, optimized for low-depth partitioning configurations. As a result, an average of 38% runtime reduction is achieved at the managable cost of less than 2% BD-rate loss.

The latest international Versatile Video Coding (VVC) standard was finalized in July 2020 by the Joint Video Experts Team (JVET) from ITU-T and ISO/IEC. Compared to the High Efficiency Video Coding (HEVC) standard, VVC offers up to 50% of bitrate savings. However, the encoder runtime of the reference software increases tenfold compared to the HEVC equivalent. This paper shows that VVC coding tools which do not consume much of the reference encoder time can have prohibitive computational cost for practical encoders, such as VVenC. VVenC is an optimized open-source real-world implementation with high relevance for practical applications. After analyzing the encoding tools in VTM and VVenC, Symmetric Motion Vector Difference (SMVD) was identified as one of the tools whose efficiency-complexity trade-off may be unfavorable for practical applications. For that reason, two complexity reduction methods for the SMVD search are proposed in this paper. The experimental evaluation on VVenC encoder at an optimized operation point confirms the SMVD encoding runtime reduction by over 72%, with minimal impact on the encoding efficiency.

Two-pass rate control (RC) schemes have proven useful for generating low-bitrate video-on-demand or streaming catalogs. Visually optimized encoding particularly using latest-generation coding standards like Versatile Video Coding (VVC), however, is still a subject of intensive study. This paper describes the two-pass RC method integrated into version 1 of VVenC, an open VVC encoding software. The RC design is based on a novel two-step rate-quantization parameter (R-QP) model to derive the second-pass coding parameters, and it uses the low-complexity XPSNR visual distortion measure to provide numerically as well as visually stable, perceptually R-D optimized encoding results. Random-access evaluation experiments confirm the improved objective as well as subjective performance of our RC solution.

In the last 17 years, since the finalization of the first version of the now-dominant H.264/Moving Picture Experts Group-4 (MPEG-4) Advanced Video Coding (AVC) standard in 2003, two major new generations of video coding standards have been developed. These include the standards known as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC). HEVC was finalized in 2013, repeating the ten-year cycle time set by its predecessor and providing about 50% bit-rate reduction over AVC. The cycle was shortened by three years for the VVC project, which was finalized in July 2020, yet again achieving about a 50% bit-rate reduction over its predecessor (HEVC). This article summarizes these developments in video coding standardization after AVC. It especially focuses on providing an overview of the first version of VVC, including comparisons against HEVC. Besides further advances in hybrid video compression, as in previous development cycles, the broad versatility of the application domain that is highlighted in the title of VVC is explained. Included in VVC is the support for a wide range of applications beyond the typical standard- and high-definition camera-captured content codings, including features to support computer-generated/screen content, high dynamic range content, multilayer and multiview coding, and support for immersive media such as 360° video.

Versatile Video Coding (VVC) was finalized in July 2020 as the most recent international video coding standard. It was developed by the Joint Video Experts Team (JVET) of the ITU-T Video Coding Experts Group (VCEG) and the ISO/IEC Moving Picture Experts Group (MPEG) to serve an ever-growing need for improved video compression as well as to support a wider variety of today’s media content and emerging applications. This paper provides an overview of the novel technical features for new applications and the core compression technologies for achieving significant bit rate reductions in the neighborhood of 50% over its predecessor for equal video quality, the High Efficiency Video Coding (HEVC) standard, and 75% over the currently most-used format, the Advanced Video Coding (AVC) standard. It is explained how these new features in VVC provide greater versatility for applications. Highlighted applications include video with resolutions beyond standard- and high-definition, video with high dynamic range and wide color gamut, adaptive streaming with resolution changes, computer-generated and screen-captured video, ultralow-delay streaming, 360° immersive video, and multilayer coding e.g., for scalability. Furthermore, early implementations are presented to show that the new VVC standard is implementable and ready for real-world deployment.

The support for screen content coding has received more attention with the latest development in video compression, the upcoming Versatile Video Coding (VVC) standard. Among the dedicated screen content coding tools, the transform skip mode (TSM) represents a promising approach for improving the coding efficiency at a low impact on implementation complexity. In this work, we present a dedicated residual coding for transform blocks coded in TSM. Due to the lack of the energy compaction of the transform, the quantization indexes for blocks coded in TSM have different statistical properties, which can be exploited in the entropy coding. Our coding experiments with screen content sequences yielded bit-rate savings of 3.9% for intra-only coding and 2.8% for typical random access configurations.

To reduce the residual energy of a video signal, motion compensated prediction with fractional-sample accuracy has been successfully employed in modern video coding technology. In contrast to the fixed quarter-sample motion vector resolution for the luma component in High Efficiency Video Coding standard, the current draft of a new Versatile Video Coding standard introduces a block-level adaptive motion vector resolution (AMVR) scheme. The AMVR allows coding of motion vector difference at different precisions. Nevertheless, the interpolation filters for each fractional sample position are fixed. In this paper, alternative half-luma-sample interpolation filters are proposed. Enabling the interpolation filter selection at a fine granularity allows to better adapt to the local image characteristics. Experimental results show that the proposed method can improve the average BD-rate of VTM 5.0 reference software by -0.40% for Random Access configuration and by -0.76% for Low Delay P configuration.

Versatile Video Coding (VVC) is the emerging video coding standard to be finalized by the Joint Video Experts Team in July 2020. Compared to its predecessor, the High Efficiency Video Coding (HEVC) standard, VVC provides 50% bit-rate reduction at comparable visual quality for natural video content in high-definition (HD) and ultra high-definition (UHD) resolution. To achieve this, the standard incorporates more advanced and generalized algorithms, leading to an increase in computational complexity. This includes for example additional in-loop filters, decoder-side motion refinement and search as well as an increased number of transforms, which creates a grand challenge for implementers to achieve live decoding on general-purpose CPUs. In this paper, the work on an efficient software decoder implementation for the upcoming VVC standard is described, including optimization of sample operations using single instruction multiple data (SIMD) instructions and parallelization approaches with multithreading. As a result, the presented decoder can perform live decoding of 10bit HD video at 60 frames per second (fps) and 10bit UHD video at 30fps on modern mobile consumer hardware, showcasing that VVC live decoding is possible already right before finalization of the standard.

Versatile Video Coding (VVC) is a new international video coding standard to be finalized in July 2020. It is designed to provide around 50% bit-rate saving at the same subjective visual quality over its predecessor, High Efficiency Video Coding (H.265/HEVC). During the standard development, objective bit-rate savings of around 40% have been reported for the VVC reference software (VTM) compared to the HEVC reference software (HM). The unoptimized VTM encoder is around 9x, and the decoder around 2x, slower than HM. This paper discusses the VVC encoder complexity in terms of soft-ware runtime. The modular design of the standard allows a VVC encoder to trade off bit-rate savings and encoder runtime. Based on a detailed tradeoff analysis, results for different operating points are reported. Additionally, initial work on software and algorithm optimization is presented. With the optimized software algorithms, an operating point with an over 22x faster single-threaded encoder runtime than VTM can be achieved, i.e. around 2.5x faster than HM, while still providing more than 30% bit-rate savings over HM. Finally, our experiments demonstrate the flexibility of VVC and its potential for optimized soft-ware encoder implementations.

The tutorial provides an overview on the latest emerging video coding standard VVC (Versatile Video Coding) to be jointly published by ITU-T and ISO/IEC. It has been developed by the Joint Video Experts Team (JVET), consisting of ITU-T Study Group 16 Question 6 (known as VCEG) and ISO/IEC JTC 1/SC 29/WG 11 (known as MPEG). VVC has been designed to achieve significantly improved compression capability compared to previous standards such as HEVC, and at the same time to be highly versatile for effective use in a broadened range of applications. Some key application areas for the use of VVC particularly include ultra-high-definition video (e.g. 4K or 8K resolution), video with a high dynamic range and wide colour gamut (e.g., with transfer characteristics specified in Rec. ITU-R BT.2100), and video for immersive media applications such as 360° omnidirectional video, in addition to the applications that have commonly been addressed by prior video coding standards. Important design criteria for VVC have been low computational complexity on the decoder side and friendliness for parallelization on various algorithmic levels. VVC is planned to be finalized by July 2020 and is expected to enter the market very soon.The tutorial details the video layer coding tools specified in VVC and develops the concepts behind the selected design choices. While many tools or variants thereof have been available before, the VVC design reveals many improvements compared to previous standards which result in compression gain and implementation friendliness. Furthermore, new tools such as the Adaptive Loop Filter, or Matrix-based Intra Prediction have been adopted which contribute significantly to the overall performance. The high-level syntax of VVC has been re-designed compared to previous standards such as HEVC, in order to enable dynamic sub-picture access as well as major scalability features already in version 1 of the specification.

After the development of the High-Efficiency Video Coding Standard (HEVC), ITU-T VCEG and ISO/IEC MPEG formed the Joint Video Exploration Team (JVET), which started exploring video coding technology with higher coding efficiency, including development of a Joint Exploration Model (JEM) algorithm and a corresponding software implementation. The technology explored in the last version of the JEM further increases the compression capabilities of the hybrid video coding approach by adding new tools, reaching up to 30% bit rate reduction compared to HEVC based on the Bjøntegaard delta bit rate (BD-rate) metric, and further improvement beyond that in terms of subjective visual quality. This provided enough evidence to issue a joint Call for Proposals (CfP) for a new standardization activity now known as Versatile Video Coding (VVC). All technology proposed in the responses to the CfP was based on the classic block-based hybrid video coding design, extending it by new elements of partitioning, intra- and inter-picture prediction, prediction signal filtering, transforms, quantization/scaling, entropy coding, and in-loop filtering. This article provides an overview of technology that was proposed in the responses to the CfP, with a focus on techniques that were not already explored in the JEM context.

In this paper, we describe a video coding design that enables a higher coding efficiency than the HEVC standard. The proposed video codec follows the design of block-based hybrid video coding, but includes a number of advanced coding tools. A part of the incorporated advanced concepts was developed by the Joint Video Exploration Team, while others are newly proposed. The key aspects of these newly proposed tools are the following. A video frame is subdivided into rectangles of variable size using a binary partitioning with variable split ratios. Three new approaches for generating spatial intra prediction signals are supported: A line-wise application of conventional intra prediction modes, coupled with a mode-dependent processing order, a region-based template matching prediction method and intra prediction modes based on neural networks. For motion-compensated prediction, a multi-hypothesis mode with more than two motion hypotheses can be used. In transform coding, mode dependent combinations of primary and secondary transforms are applied. Moreover, scalar quantization is replaced by trellis-coded quantization and the entropy coding of the quantized transform coefficients is improved. The intra and inter prediction signals can be filtered using an edge-preserving diffusion filter or a non-linear DCT-based thresholding operation. The video codec includes an adaptive in-loop filter for which one of three classifiers can be chosen on a picture basis. We also incorporated an optional encoder control, which adjusts the quantization parameters based on a perceptually motivated distortion measure. In a random access scenario, our proposed video codec achieves luma BD-rate savings between 32.5% for HDR HLG UHD and 39.6% for SDR UHD over the HEVC (HM software) anchor for different categories of test sequences.

Intra-picture prediction as in HEVC exploits the nearest reference line adjacent to the current coding unit (CU) for prediction of samples. If this reference line represents a discontinuity, the reference samples in this reference line can differ to a large extent from the original samples and may lead to a large prediction error. We propose a multiple reference lines (MRLs) coding to allow not only the nearest reference line 0 but also reference lines 1 and 3 to be candidates for angular intra prediction as shown in Fig. 1. To reduce the complexity arising from additional lines to be checked at encoder side, we further propose to restrict the MRL to angular most probable modes (MPMs) only. The MRL coding signals the reference line index before the intra prediction mode. This allows to not signal the MPM flag of the current CU and implicitly derive it as true when a non-zero reference line index is signaled. Experimental results are provided to evaluate the performance of the proposed MRL coding on top of the VVC test model VTM-2.0.1. 26 test sequences in different categories, including 4k, 1080p, 720p, WVGA, WQVGA resolutions and screen contents are tested. Two coding structures are evaluated, all intra (AI) and random access (RA). The objective coding efficiency is measured in terms of Bjøntegaard Delta (BD) rate (%) computed using four rate/PSNR points that were generated by using quantization parameters 22, 27, 32 and 37. Lower (negative) BD-rate implies better compression rate. Table 1 shows that the presented MRL provides 0.46% bitrate savings for an all-intra and 0.2% for a random-access configuration on average. Furthermore, it provides 1.45% bitrate reduction for screen content test sequences, which are representing an increasingly important video application. Because of a fairly good trade-off between coding efficiency and complexity, the proposed MRL coding mode with MPM restriction was adopted into the current VVC draft standard.

The Versatile Video Coding (VVC) development has adopted the possibility to bypass the transform when the transform block size is equal to 4×4 from its predecessor High Efficiency Video Coding (HEVC). This so-called Transform Skip Mode (TSM) results in increased encoding time when extending it to transform block sizes up to 32×32 while the compression efficiency improvement is for screen content only. This paper presents the so-called Unified MTS scheme that makes TSM for luma transform block sizes up to 32×32 possible without the disadvantage of excessive encoding time increase by incorporating the TSM with the existing Multiple Transform Set (MTS) technique. The Unified MTS scheme achieves compression efficiency improvements, in terms of BD-rate, of about -5.0% in the All-Intra configuration and -5.4% in the Random-Access configuration, respectively, for the screen content sequences of the used test set. Compared to the straightforward extension of TSM to transform block sizes up to 32×32, the encoding time is about 25% less in the All-Intra configuration and about 21% less in the Random-Access configuration, respectively, whereas the compression efficiency improvements are only 0.04% less in the All-Intra configuration, and 0.09% less in the Random-Access configuration, respectively. Relative to the anchor using TSM for 4×4 transform blocks only, the encoding time is the same for natural content and 3% higher for screen content.

The H.265/MPEG-H High Efficiency Video Coding compliant encoding process faces the challenge of high computational complexity. Particularly, in the case of inter-picture prediction, most of the computational resources are allocated for the motion estimation (ME) process. In turn, ME and motion compensation enable improving coding efficiency by addressing the blocks of video frames as corresponding displacements from one or more reference blocks. These displacements do not necessarily have to be limited to integer sample positions, but may have an accuracy of half sample or quarter sample positions, which are identified during fractional sample refinement. In this paper, a context-based scheme for fractional sample refinement is proposed. The scheme takes the advantage of already obtained information in prior ME steps and provides significant flexibility in terms of parameterization. In this way, it adaptively achieves a desired tradeoff between computational complexity and coding efficiency. According to the experimental results obtained for an example algorithm utilizing the proposed framework, a significant decrease in the number of search points can be achieved. For instance, considering only 6 instead of 16 fractional sample positions results in a tradeoff of only 0.4% Bjøntegaard Delta-rate loss for high-definition video sequences compared with the conventional interpolation-and-search method.

Single instruction multiple data (SIMD) instructions have been commonly used to accelerate video codecs. The recently introduced High Efficiency Video Coding (HEVC) codec like its predecessors is based on the hybrid video codec principle and, therefore, is also well suited to be accelerated with SIMD. In this paper we present the SIMD optimization for the entire HEVC decoder for all major SIMD instruction set architectures. Evaluation has been performed on 14 mobile and PC platforms covering most major architectures released in recent years. With SIMD, up to 5× speedup can be achieved over the entire HEVC decoder, resulting in up to 133 and 37.8 frames/s on average on a single core for Main profile 1080p and Main10 profile 2160p sequences, respectively.

The state-of-the-art hybrid video coding standard H.264/AVC defines two entropy-coding schemes with different complexity-performance trade-offs. Supporting these two schemes within a single standard raises several problems ranging from higher efforts for product development to increased silicon costs for hardware implementations. To overcome these issues, this work proposes a unified and complexity-scalable entropy-coding framework that is based on PIPE/V2V. The proposed framework uses a single set of tools for all configurations and achieves the same complexity-performance trade-offs as the existing entropy-coding schemes through scalability.

In hybrid video coding, an entropy coding scheme transmits the quantized transform coefficients, resulting from block-based transformation and quantization of the difference between the prediction signal and the original signal, and additional side information. The state-of-the-art hybrid video coding standard H.264/AVC defines two different entropy coding schemes with different complexity-performance tradeoff. As a result, the support for two different entropy coding schemes has to be maintained and introduces several problems. To overcome these issues, a unified solution is proposed, which is based on the PIPE/V2V coding concept. It achieves the same complexity-performance trade-offs as the existing entropy coding schemes by scalability. The advantage of the proposed scheme over the existing concept is the usage of the same set of tools for all configurations. Simulation results and complexity analysis on hardware show the efficiency of the proposed scheme.

Advances in video compression technology have been driven by ever-increasing processing power available in software and hardware. The emerging High Efficiency Video Coding (HEVC) standard aims to provide a doubling in coding efficiency with respect to the H.264/AVC high profile, delivering the same video quality at half the bit rate. In this paper, complexity-related aspects that were considered in the standardization process are described. Furthermore, profiling of reference software and optimized software gives an indication of where HEVC may be more complex than its predecessors and where it may be simpler. Overall, the complexity of HEVC decoders does not appear to be significantly different from that of H.264/AVC decoders; this makes HEVC decoding in software very practical on current hardware. HEVC encoders are expected to be several times more complex than H.264/AVC encoders and will be a subject of research in years to come.

The joint development of the upcoming High Efficiency Video Coding (HEVC) standard by ITU-T Video Coding Experts Group and ISO/IEC Moving Picture Experts Group marks a new step in video compression capability. In technical terms, HEVC is a hybrid video-coding approach using quadtree-based block partitioning together with motion-compensated prediction. Even though a high degree of adaptability is achieved by quadtree-based block partitioning, this approach has certain intrinsic drawbacks, which may result in redundant sets of motion parameters being transmitted. Previous work has shown that those redundancies can effectively be removed by merging the leafs of a particular quadtree structure. Following this concept, a block merging algorithm for HEVC is now proposed. This algorithm generates a single motion parameter set for a whole region of contiguous motion-compensated blocks. In this paper, we describe the various components of the proposed block merging algorithm and, using experimental evidence, demonstrate their benefits in terms of coding efficiency.

Quadtree-based block partitioning together with motion-compensated prediction has proven to be an efficient approach in video compression. However, when dealing with spatially neighboring blocks in uniformly displaced regions, quadtree-based partitioning may lead to redundant sets of transmitted motion parameters. This paper proposes and describes a simple but efficient block merging algorithm that aims at removing those redundancies by using only a single parameter set for a whole motion-compensated region of contiguous blocks. Simulation results show that our proposed merging technique works more efficiently than the conceptually similar direct mode as, e.g., specified in H.264/AVC. Due its efficiency and simplicity, our proposed merging approach has been adopted into the first test model of the high efficiency video coding (HEVC) standardization project, as currently pursued by ITU-T VCEG and ISO/IEC MPEG.

Typical interpolation methods in video coding perform filtering of reference picture samples using FIR filters for motion-compensated prediction. This process can be viewed as a signal decomposition using basis functions which are restricted by the interpolating constraint. Using the concept of generalized interpolation provides a greater degree of freedom for selecting basis functions. We implemented generalized interpolation using a combination of IIR and FIR filters. The complexity of the proposed scheme is comparable to that of an 8-tap FIR filter. Bit rate savings up to 20% compared to the H.264/AVC 6-tap filter are shown.

This paper describes a novel video coding scheme that can be considered as a generalization of the block-based hybrid video coding approach of H.264/AVC. While the individual building blocks of our approach are kept simple similarly as in H.264/AVC, the flexibility of the block partitioning for prediction and transform coding has been substantially increased. This is achieved by the use of nested and pre-configurable quadtree structures, such that the block partitioning for temporal and spatial prediction as well as the space-frequency resolution of the corresponding prediction residual can be adapted to the given video signal in a highly flexible way. In addition, techniques for an improved motion representation as well as a novel entropy coding concept are included. The presented video codec was submitted to a Call for Proposals of ITU-T VCEG and ISO/IEC MPEG and was ranked among the five best performing proposals, both in terms of subjective and objective quality.

Abstract-A video coding architecture is described that is based on nested and pre-configurable quadtree structures for flexible and signal-adaptive picture partitioning. The primary goal of this partitioning concept is to provide a high degree of adaptability for both temporal and spatial prediction as well as for the purpose of space-frequency representation of prediction residuals. At the same time, a leaf merging mechanism is included in order to prevent excessive partitioning of a picture into prediction blocks and to reduce the amount of bits for signaling the prediction signal. For fractional-sample motion-compensated prediction, a fixed-point implementation of the maximal-order minimum-support algorithm is presented that uses a combination of infinite impulse response and FIR filtering. Entropy coding utilizes the concept of probability interval partitioning entropy codes that offers new ways for parallelization and enhanced throughput. The presented video coding scheme was submitted to a joint call for proposals of ITU-T Visual Coding Experts Group and ISO/IEC Moving Picture Experts Group and was ranked among the five best performing proposals, both in terms of subjective and objective quality.

Decoder-side motion vector derivation (DMVD) using template matching has been shown to improve coding efficiency of H.264/AVC based video coding. Instead of explicitly coding motion vectors into the bitstream, the decoder performs motion estimation in order to derive the motion vector used for motion compensated prediction. In previous works, DMVD was performed using a full template matching search in a limited search range. In this paper, a candidate based fast search algorithm replaces the full search. While the complexity reduction especially for the decoder is quite significant, the coding efficiency remains comparable. While for the full search algorithm BD-bitrate savings of 7.4% averaged over CIF and HD sequences according to the VCEG common conditions for IPPP high profile are observed, the proposed fast search achieves bitrate reductions of up to 7.5% on average. By further omitting sub-pel refinement, average savings observed for CIF and HD are still up to 7%.