This paper presents a CNN to reduce the encoding time of a VVC-based intra-picture encoder. For encoding a 32 × 32 block, the CNN estimates two partitioning parameters that restrict the allowed coding block width and height. To estimate them such that the encoder skips testing inefficient partitioning modes, we train the CNN as follows: First, we generate training data by encoding sequences without the CNN. While encoding, we test all combinations of the two parameters for each 32 × 32 block and store the resulting Lagrangian rate-distortion-time (RDT) cost. We use the recorded cost to derive the loss function when training the CNN. Consequently, the CNN is trained such that it minimizes the Lagrangian RDT cost. Our CNN reduces the encoding time by 50% with a bit rate increase of 0.9%, which outperforms existing CNN-based approaches. Our generic training approach could also be applied for other encoder parameters.

This paper presents two new methods for fast VVC intra-picture encoding. Both are based on an approach that uses a CNN for blockadaptive parameter estimation. The parameters restrict the multitype-tree (MTT) partitionings tested by the encoder. The methods aim for an improvement of the approach by further constraints with additional parameters. Adding parameters increases the time required for training data generation exponentially. This raises the question which parameters to add and how. To explore further partitioning restrictions, the first method adds parameters controlling the block sizes the MTT can start from. Although this leads to four parameters, we can exploit that some of their combinations are invalid. To investigate whether testing fewer prediction and transform modes is feasible, the second method adds a single parameter that restricts their number jointly. The paper evaluates hypothetical and actual encoding time reductions for VTM-10.2. The first method outperforms our other and other existing method: The encoding time decreases by 50% with a bit rate increase of 0.7%.

This paper presents a new method for fast VVC intra-picture encoding using a CNN. The CNN operates on the original samples of$\mathbf{32}\times \mathbf{32}$blocks. Given a current block, it derives for each of the block's multi-type trees (MTTs), which are nested in quad-tree (QT) nodes, a parameter pair. The parameter pairs constrain the minimum width and height of the sub-blocks in their MTTs. This enables the CNN to control the number of tested MTT splits with fine granularity. To skip modes while maintaining the rate-distortion (RD) performance, we train the CNN considering the Lagrangian rate-distortion-time (RDT) cost caused by the derived parameters. First, we generate training data by encoding; when reaching a quad-tree node in a$\mathbf{32}\times \mathbf{32}$block, we encode the associated MTT with varying parameter pair values and record the resulting the RD and time cost. Then, when the CNN outputs parameters in training, we estimate the related RDT cost of the$\mathbf{32}\times \mathbf{32}$block using the recorded data. For this, we model the dependency between RDT cost and the parameters by emulating the encoder's RD optimization process. This way, we train the CNN while considering the RDT cost with an accuracy that is sufficient to outperform existing approaches. The approach achieves an encoding time reduction of 50% with a bit rate increase of only 0.7% for VTM-10.2.

In the current version of the evolving Versatile Video Coding (VVC) standard, the Matrix-Based Intra Prediction (MIP) tool is included which is based on data-driven training techniques. This paper describes how an efficient fixed point implementation of the MIP prediction modes was designed. First, a method is presented that improves the integer quantization of the floating-point matrices resulting from the training by geometrically transforming the input and output. Second, a soft-clipping function can be used during the training stage to restrict the range of floating-point matrix entries. This enables the quantization of the trained coefficients with a fixed bit depth and precision. The clipping itself has no impact on the compression efficiency. The obtained predictors are incorporated into the Versatile Video Coding Test Model 7. All-Intra bit-rate savings of 0.6 % across different resolutions in terms of the Bjøntegaard-Delta bit rate (BD-rate) are reported.

In this paper, we describe a video coding design that enables a higher coding efficiency than the HEVC standard. The proposed video codec follows the design of block-based hybrid video coding, but includes a number of advanced coding tools. A part of the incorporated advanced concepts was developed by the Joint Video Exploration Team, while others are newly proposed. The key aspects of these newly proposed tools are the following. A video frame is subdivided into rectangles of variable size using a binary partitioning with variable split ratios. Three new approaches for generating spatial intra prediction signals are supported: A line-wise application of conventional intra prediction modes, coupled with a mode-dependent processing order, a region-based template matching prediction method and intra prediction modes based on neural networks. For motion-compensated prediction, a multi-hypothesis mode with more than two motion hypotheses can be used. In transform coding, mode dependent combinations of primary and secondary transforms are applied. Moreover, scalar quantization is replaced by trellis-coded quantization and the entropy coding of the quantized transform coefficients is improved. The intra and inter prediction signals can be filtered using an edge-preserving diffusion filter or a non-linear DCT-based thresholding operation. The video codec includes an adaptive in-loop filter for which one of three classifiers can be chosen on a picture basis. We also incorporated an optional encoder control, which adjusts the quantization parameters based on a perceptually motivated distortion measure. In a random access scenario, our proposed video codec achieves luma BD-rate savings between 32.5% for HDR HLG UHD and 39.6% for SDR UHD over the HEVC (HM software) anchor for different categories of test sequences.

We train a neural network to perform intra picture prediction for block based video coding. Our network has multiple prediction modes which co-adapt during training to minimize a loss function. By applying the l1-norm and a sigmoid-function to the prediction residual in the DCT domain, our loss function reflects properties of the residual quantization and coding stages present in the typical hybrid video coding architecture. We simplify the resulting predictors by pruning them in the frequency domain, thus greatly reducing the number of multiplications otherwise needed for the dense matrix-vector multiplications. Also, by quantizing the network weights and using fixed point arithmetic, we allow for a hardware friendly implementation. We demonstrate significant coding gains over state of the art intra prediction.

In a template matching (TM) intra method, the neighboring samples of the current block are regarded as a template. The decoder searches for the best template match in the current reconstructed picture using an error minimizing metric like sum of squared differences (SSD). The prediction signal is generated by copying the samples of the adjacent block to the selected template match. The increased decoder complexity from the search algorithm makes it less attractive for modern applications like video calling. In a previous publication [1], we presented a region-based template matching (RTM) approach for intra coding. Compared to the conventional TM methods which searches for the template match in a complete search window, RTM searches in a region of the search window. Thus, RTM offers a better trade-off between coding efficiency and decoder complexity. Nevertheless, the memory requirements and number of computations to be carried at the decoder are still high, making RTM difficult for hardware realization. This paper aims to address these issues.

This paper presents a novel method for a data-driven training of affine-linear predictors which perform intra prediction in state-of-the-art video coding. The main aspect of our training design is the use of subband decomposition of both the input and the output of the prediction. Due to this architecture, the same set of predictors can be shared across different block shapes leading to a very limited memory requirement. Also, the computational complexity of the resulting predictors can be limited such that it does not exceed the complexity of the conventional angular intra prediction. In the training itself, a loss function modelling the bit-rate of the DCT-transformed residuals is used. The obtained predictors are incorporated into the Versatile Video Coding Test Model 3 in addition to the conventional intra prediction modes. All-Intra bit-rate savings ranging from 0.8% to 1.4% across different resolutions have been measured in terms of the Bjøntegaard-Delta bit rate (BD-rate).

This paper presents a data-driven training of affine- linear predictors which perform intra-picture prediction for video coding. The trained predictors use a single line of reconstructed boundary samples as input like the conventional intra prediction modes. For large blocks, the presented predictors initially transform the input samples via Discrete Cosine Transform. This allows to omit high frequency coefficients and consequently reduce the input dimension. The output is the result of a single matrix-vector multiplication and offset addition. Here, the predictors only construct certain coefficients in the frequency domain. The final prediction signal is then obtained by inverse transform. The coefficients of the prediction modes need to be stored in advance, requiring 0.273 MB of memory. The training employs a recursive block partitioning, where the loss function targets to approximate the bit-rate of the DCT-transformed block residuals. The obtained predictors are incorporated into the Versatile Video Coding Test Model 4. The authors report All- Intra bit-rate savings ranging from 0.7% to 2.0% across different resolutions in terms of the Bjøntegaard-Delta bit rate (BD-rate).

Besides model-based development, model-based quality assurance and the tighter integration of static and dynamic quality assurance activities are becoming increasingly relevant in the development of software-intensive systems. Thus, this paper reports on an empirical study aimed at investigating the promises regarding quality improvements and cost savings. The evaluation comprises data from 13 industry case studies conducted during a three-year large-scale research project in the transportation domain (automotive, avionics, rail system). During the evaluation, we identified major goals and strategies associated with (integrated) model-based analysis and testing and evaluated the improvements achieved. The aggregated results indicate an average cost reduction of between 29% and 34% for verification and validation and of between 22% and 32% for defect removal. Compared with these cost savings, improvements regarding test coverage (~8%), number of remaining defects (~13%), and time to market (~8%) appear less noticeable.

The paper describes a scalable video coding extension of the upcoming HEVC video coding standard for spatial and quality scalable coding. Besides coding tools known from scalable profiles of prior video coding standards, it includes new coding tools that further improve the enhancement layer coding efficiency. The effectiveness of the proposed scalable HEVC extension is demonstrated by comparing the coding efficiency to simulcast and single-layer coding for several test sequences and coding conditions.

In this paper we present a new local and multiscale disparity estimation algorithm. Results show that the proposed method can preserve arbitrarily shaped depth discontinuities. Some results obtained from this method are shown as depth maps and synthetic views produced using the estimated disparity maps.

The joint development of the upcoming High Efficiency Video Coding (HEVC) standard by ITU-T Video Coding Experts Group and ISO/IEC Moving Picture Experts Group marks a new step in video compression capability. In technical terms, HEVC is a hybrid video-coding approach using quadtree-based block partitioning together with motion-compensated prediction. Even though a high degree of adaptability is achieved by quadtree-based block partitioning, this approach has certain intrinsic drawbacks, which may result in redundant sets of motion parameters being transmitted. Previous work has shown that those redundancies can effectively be removed by merging the leafs of a particular quadtree structure. Following this concept, a block merging algorithm for HEVC is now proposed. This algorithm generates a single motion parameter set for a whole region of contiguous motion-compensated blocks. In this paper, we describe the various components of the proposed block merging algorithm and, using experimental evidence, demonstrate their benefits in terms of coding efficiency.

Recently, ITU-T VCEG and ISO/IEC MPEG have started a new joint standardization activity on video coding, called High Efficiency Video Coding (HEVC). In this paper, the new transform coding techniques in the HEVC Test Model are de- scribed, including the residual quadtree (RQT) approach and coded block pattern signaling. Experimental results showing the advantage of using larger block size transforms, especially for high resolution video material, are presented. Furthermore, the trade-off between coding gain and encoder complexity in terms of the maximum RQT depth is discussed.

Quadtree-based block partitioning together with motion-compensated prediction has proven to be an efficient approach in video compression. However, when dealing with spatially neighboring blocks in uniformly displaced regions, quadtree-based partitioning may lead to redundant sets of transmitted motion parameters. This paper proposes and describes a simple but efficient block merging algorithm that aims at removing those redundancies by using only a single parameter set for a whole motion-compensated region of contiguous blocks. Simulation results show that our proposed merging technique works more efficiently than the conceptually similar direct mode as, e.g., specified in H.264/AVC. Due its efficiency and simplicity, our proposed merging approach has been adopted into the first test model of the high efficiency video coding (HEVC) standardization project, as currently pursued by ITU-T VCEG and ISO/IEC MPEG.

This paper describes a novel video coding scheme that can be considered as a generalization of the block-based hybrid video coding approach of H.264/AVC. While the individual building blocks of our approach are kept simple similarly as in H.264/AVC, the flexibility of the block partitioning for prediction and transform coding has been substantially increased. This is achieved by the use of nested and pre-configurable quadtree structures, such that the block partitioning for temporal and spatial prediction as well as the space-frequency resolution of the corresponding prediction residual can be adapted to the given video signal in a highly flexible way. In addition, techniques for an improved motion representation as well as a novel entropy coding concept are included. The presented video codec was submitted to a Call for Proposals of ITU-T VCEG and ISO/IEC MPEG and was ranked among the five best performing proposals, both in terms of subjective and objective quality.

Abstract-A video coding architecture is described that is based on nested and pre-configurable quadtree structures for flexible and signal-adaptive picture partitioning. The primary goal of this partitioning concept is to provide a high degree of adaptability for both temporal and spatial prediction as well as for the purpose of space-frequency representation of prediction residuals. At the same time, a leaf merging mechanism is included in order to prevent excessive partitioning of a picture into prediction blocks and to reduce the amount of bits for signaling the prediction signal. For fractional-sample motion-compensated prediction, a fixed-point implementation of the maximal-order minimum-support algorithm is presented that uses a combination of infinite impulse response and FIR filtering. Entropy coding utilizes the concept of probability interval partitioning entropy codes that offers new ways for parallelization and enhanced throughput. The presented video coding scheme was submitted to a joint call for proposals of ITU-T Visual Coding Experts Group and ISO/IEC Moving Picture Experts Group and was ranked among the five best performing proposals, both in terms of subjective and objective quality.