Recent video codecs, e.g. AV1, VVC, apply a Non-rectangular (NR) partitioning to combine prediction signals using a smooth blending around the boundary, followed by a rectangular transform (TX) on the whole block. TX on each NR residual separately is not yet supported. A recent NR TX technique [1] demonstrated promising gains in an experimental setup outside the reference software. This method employs the regular inverse-DCT at the decoder to reconstruct a rectangular signal while discarding the signal outside the region of interest. This design is appealing due to the minimal changes required at the decoder. The method uses a partitioned 2D DCT as a dictionary to find a sparse representation of the NR signal, with scaled representations serving as TX coefficients. These coefficients typically exhibit properties distinct from those of DCT TX coefficients. Therefore, the established entropy coding schemes in video codecs, which are primarily designed for DCT coefficients, are not well-suited for optimally encoding these TX coefficients.

This work introduces an additional boundary for extended wedge mode in the reference software of AVM. Wedge mode, introduced in AV1 and modified later, employs a non-rectangular partitioning mode to combine two prediction signals. However, the available wedge masks might not be sufficient to handle diverse video content. In this work, an adaptive boundary selection scheme to construct the wedge masks is proposed. A set of two boundaries are used for this purpose. Additionally, three signaling schemes are presented, with different levels of complexity, and one low-complexity design which limits the adaptivity to smaller block sizes while maintaining similar performance. Experimental results using AVM common testing conditions showed promising gains. In random access configuration, class A5 demonstrates 0.89% PSNR-Y BD rate gain, while classes A1, B1, and A3 show approximately 0.08% PSNR-Y BD rate gain.

The wedge mode in Alliance of Open Media Video Model (AVM) divides a block into two non-rectangular regions to combine two prediction signals. Rather than employing a sharp transition between these signals, the wedge mode utilizes a gradual transition around the boundary. Currently, there is only one such wedge boundary integrated in the AVM reference software, which is insufficient for handling diverse video content. Recently, an additional relatively smooth boundary has been explored using several signaling schemes. While this adaptive wedge boundary demonstrated promising improvements for short sequences, the signaling overhead associated with it restricted its overall potential. The current work uses this smooth boundary and proposes three simple switching schemes, where the boundary is selected based on specific parameters. While this approach reduces flexibility, it mitigates the requirement for signalling, which has proven to be more beneficial. This provides a PSNR-Y BD-rate gain of up to 0.05% in Random Access (RA) and 0.10% gain in Low Delay (LD) configuration. The improvements are particularly significant for high-resolution classes; in class A1, one scheme achieves 0.38% gain in RA, and another 0.24% gain in LD. A High Level Syntax scheme is presented, which combines the performance of 0.10% in RA and 0.12% in LD by signalling 2 bits per sequence.

Reference Picture Padding is utilized in a variety of video coding standards. It allows motion vectors to point partly outside the reference picture in inter prediction. This approach provides advantages in compression efficiency. The method employed in current standards is repetative padding, which copies the pixels at the picture border outward. The major advantage of this approach is its low complexity. However, it is not optimal in terms of compression efficiency. Recently, motion-compensated padding has been introduced, which utilizes already coded content in the padding process. Another padding method is template matching-based padding, with the main downside being its high computational complexity. In this work, we propose a complexity reduction of the method by applying different measures, including an improved virtual target block increase and an early termination of the search, among other improvements. As a result, we achieve a decoder runtime increase of 1% and a BD-rate of -0.33% in a low delay subpicture scenario, outperforming motion-compensated padding with a decoder runtime increase of 9% and a BD-rate of -0.19%. In the random-access configuration we can show a consistent improvement by combining the methods over the stand-alone methods.

Reference picture padding is the task of padding the outside of the reference picture for inter prediction. This task is done to accommodate for motion vectors that extend partly outside the picture, thereby increasing compression efficiency. The established approach involves simply duplicating the boundary pixels outward. While this solution boasts low complexity, it often results in suboptimal compression performance in numerous cases. Template matching-based reference picture padding presents itself as a promising method to further increase compression efficiency and reduce artifacts. However, its primary drawback lies in its high computational complexity. One potential solution entails restricting the number of considered candidates per search step to a very small set to maintain a feasibly low increase in decoder runtime. This, however, compromises the compression efficiency. In this study, we propose a novel approach that maintains the efficiency gains, while significantly reducing the increase in decoder runtime. This method primarily focuses the computational complexity on pixels more frequently utilized in inter prediction. Additionaly, we introduce an early stopping criterion that terminates the search if at least one similar candidate is found. Through these modifications we achieve a reduction in decoder runtime increase from 1252% to 159%, while maintaining −0.37% compared to −0.41% Bj⊘ntegaard delta rate in a Versatile Video Coding subpicture coding scenario.

The VVC codec is applied to the task of multispectral image (MSI) compression using adaptive and scalable coding structures. In a “plain” VVC approach, concepts from picture-to-picture temporal prediction are employed for decorrelation along the MSI’s spectral dimension. The popular principle component analysis (PCA) for spectral decorrelation is further evaluated in combination with VVC intra-coding for spatial decorrelation. This approach is referred to as PCA-VVC. A novel adaptive MSI compression algorithm, named HPCLS, is introduced, that uses PCA and inter-prediction for spectral and VVC intra-coding for spatial decorrelation. Further, a novel adaptive scalable approach is proposed, that provides a separately decodable spectrally scaled preview of the MSI in the compressed file. Information contained in the preview is exploited in order to reduce the overall file size. All schemes are evaluated on images from the ARAD HS data set containing outdoor scenes with a high variety in brightness and color. We found that “Plain” VVC is outperformed by both PCA-VVC and HPCLS. HPCLS shows advantageous rate-distortion (RD) behavior compared to PCA-VVC for reconstruction quality above 51 dB PSNR. The performance of the scalable approach is compared to the combination of an independent RGB preview and one of HPCLS or PCA-VVC denoted as simulcast. The scalable approach shows significant benefit especially at higher preview qualities. A more detailed version of this article can be found on arXiv1.

Reference picture padding is needed in areas close to the picture boundary. It lifts the restriction of motion vectors not to point over the boundary when using inter prediction. Especially, when independently decodable subpictures are used e.g. in viewport-adaptive streaming, many boundaries occur, where padding is necessary. In such cases, the prediction quality of reference (sub)picture padding has an increased impact on the coding performance. Template matching has shown to work well for texture prediction in video coding. The paper shows that it also improves coding performance when applied in the context of reference picture padding. Experimental results on a set of test sequences demonstrate consistent coding gains and an average Bjøntegaard delta rate reduction of −0.38% and −0.48% for two sets of sequences with frequent utilization of reference picture padding. A drawback of template matching is its computational complexity. A preliminary investigation shows that the complexity can be reduced by a factor of 330 while maintaining about 60% of the Bjøntegaard delta rate savings.

Boundary extension refers to the extension of a picture boundary to enable inter prediction from regions outside the picture. In current video coding schemes, only non-adaptive approaches are used with a constant prediction which continues the boundary samples in the extension region. This causes artifacts in the region of the boundary which may be strongly visible. Especially when 360° video is coded using independently decodable subpictures, extension-related artifacts can occur at all subpicture boundaries and are not limited to the picture boundary area. Thereby, the handling of subpicture boundaries becomes more important. In this paper, an adaptive boundary extension method is investigated with explicit signaling that uses angular prediction for the extension task. It is shown that angular prediction modes are promising candidates for an extension by isolating the impact of the prediction improvement from the signaling cost. The scheme is implemented in the VVC test model, with a simple signaling method that leads to coding gain for over 40% of the subpictures. Preliminary results indicate Bj0ntegaard delta rate savings of about 0.1% when only selected subpictures are considered. This can be considered significant given that only a small area of the prediction signal is affected by the method. A major advantage of the explicit signaling approach is seen in the fact that the encoder can influence the predictions in the boundary region, such that subpicture transitions are more consistent.

This paper presents a transform coding technique for non-rectangular 2-D signals by extending the signal into a rectangular block in order to enable conventional block-based transform coding. The technique could be suitable for coding residuals of prediction blocks using geometric partitioning which has been adopted into the draft Versatile Video Coding standard. The extension of the non-rectangular signal is found using a sparse solution set generated by applying Orthogonal Matching Pursuits using partitioned transform bases. The method developed in this paper is based on Discrete Cosine Transform. Results achieved in an experimental setup outside of the video coding loop are presented for signals of triangular and trapezoidal shape in comparison to the shape-adaptive DCT. Encouraging gains are observed specifically for larger block sizes and in dependency of the quantization parameter and the partitioning shape.