The past decade has witnessed the huge success of deep learning in well-known artificial intelligence applications such as face recognition, autonomous driving, and large language model like ChatGPT. Recently, the application of deep learning has been extended to a much wider range, with Neural Network-Based Video Coding (NNVC) being one of them. NNVC can be performed at two different levels: embedding neural network-based (NN-based) coding tools into a classical video compression framework or building the entire compression framework upon neural networks. This article elaborates our studies in response to the recent exploration efforts in JVET (Joint Video Experts Team of ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC29) in the name of NNVC, falling in the former category. Specifically, in this article, we propose two advanced NN-based video coding technologies, i.e., NN-based intra prediction and NN-based in-loop filtering, which have been investigated for several meeting cycles in JVET and then adopted into the reference software, i.e., NNVC. In addition, we further propose a Small Ad-hoc Deep-Learning Library (SADL), which provides integer-based inference capabilities for neural networks to ensure interoperability across different systems. SADL has been adopted as the inference platform of all neural networks in NNVC. Extensive experiments on top of the NNVC have been conducted to evaluate the effectiveness of the proposed techniques. Compared with VTM-11.0_nnvc, the proposed two NN-based coding tools jointly achieve {11.94%, 21.86%, 22.59%}, {9.18%, 19.76%, 20.92%}, and {10.63%, 21.56%, 23.02%} BD-rate reductions on average for {Y, Cb, Cr} under random-access, low-delay, and all-intra configurations, respectively.

The state-of-the-art neural network-based (NN-based) in-loop filters for video coding are built on convolutional neural networks. The Joint Video Experts Team (JVET) activities investigate NN-based in-loop filters for two operation points, the high operation point (HOP) which provides highest possible gains at a high complexity and the low operation point (LOP) which is constrained on a low complexity. This paper focuses on the LOP network. We apply a DCT and reshaping to the inputs and an inverse DCT and inverse reshaping to the outputs of LOP. The spatial resolution inside the network is reduced by a factor of four while the final output still has the same number of pixels. The complexity in MAC/pixel (multiplyaccumulate operations per pixel) is therefore also reduced by a factor of four. This freed-up complexity is instead spent on increasing the number of backbone blocks and channels so the LOP complexity is matched. Our network has a complexity of $16.9 \mathrm{kMAC} /$ pixel and 0.2 M parameters (LOP: 17 kMAC/pixel, 0.05 M parameters). The BD-rate impact compared to the NNVC-7.1 anchor is reported to be −0.48% for RA and −0.17% for AI with the float model, and −0.44% for RA and −0.18% for AI with the integer model.

This paper presents a novel video encoding algorithm called Block Importance Mapping. In this method, blocks are assigned a quantization parameter, QP, offset based on how likely the block samples are to be used as references in encoding of nearby pictures. The reusability estimation is based on a motion search in pictures immediately before and after the current picture in output order. Blocks that are likely to be used as reference blocks are coded with a lower QP value, i.e., higher quality, whereas blocks that are deemed unlikely to be referenced are coded with a higher QP value. This method has been implemented in the VVC reference software VTM and tested on various configurations and contents. It is reported to provide BD-rate savings of 1.93% for PSNR and 3.69% for MS-SSIM in Random Access and 2.26% for PSNR and 4.29% for MS-SSIM in Low Delay coding. For HDR content, wPSNR BD-rate savings of 1.18% in Random Access configuration is reported.

This paper describes a bilateral filter that is being proposed as a coding tool for the Versatile Video Codec (VVC). The filter acts as a loop filter in parallel with the sample-adaptive offset (SAO) filter. Both the proposed filter and SAO act on the same input samples, each filter produces an offset, and these offsets are then added to the input sample to produce an output sample that, after clipping, goes to the next stage. The method has been implemented and tested according to the common test conditions in VVC test model version 5.0. For the all-intra configuration, we report a BD rate figure of -0.4% with an encoder run time increase of 6% and a decoder run time increase of 4%. For the random access configuration, the BD rate figure is -0.5% with an encoder run time increase of 2% and a decoder run time increase of 2%.

This paper describes a pre-processing method for HDR video on linear RGB data before converting to a Y'CbCr 4:2:0 representation. The method targets the luminance artifacts that can arise in saturated colors after compression when PQ Y'CbCr 4:2:0 NCL is used. After processing, the resulting Y'CbCr 4:2:0 representation is also more compressible, leading to objective BD rate results of -2.4%. Subjective improvements in compressed material are also clearly visible.

This paper proposes the use of a bilateral filter as a coding tool for video compression. The filter is applied after transform and reconstruction, and the filtered result is used both for output as well as for spatial and temporal prediction. The implementation is based on a look-up table (LUT), making it fast enough to give a reasonable trade-off between complexity and compression efficiency. By varying the center filter coefficient and avoiding storing zero LUT entries, it is possible to reduce the size of the LUT to 2202 bytes. It is also demonstrated that the filter can be implemented without divisions, which is important for full custom ASIC implementations. The method has been implemented and tested according to the common test conditions in JEM version 5.0.1. For still images, or intra frames, we report a 0.4% bitrate reduction with a complexity increase of 6% in the encoder and 5% in the decoder. For video, we report a 0.5% bitrate reduction with a complexity increase of 3% in the encoder and 0% in the decoder.

In this paper we present a solution to a luminance artifact problem that occurs when conventional non-constant luminance Y'CbCr and 4:2:0 subsampling is combined with the type of highly non-linear transfer functions typically used for High Dynamic Range (HDR) video. These luminance artifacts can be avoided by selecting a luma code value that minimizes the luminance error. Subjectively, the quality improvement is clearly visible even for uncompressed video. Improvements in tPSNR-Y of up to 20 dB have been observed, compared to conventional subsampling. Crucially, no change in the decoder is needed.

This paper describes high-quality compression of high dynamic range (HDR) video using existing tools such as the HEVC Main 10 profile, the SMPTE ST 2084 (PQ) transfer function, and the BT.2020 non-constant luminance Y'CbCr color representation. First, we present novel mathematical bounds that reduce complexity of luminance-preserving subsampling (luma adjustment). A nested look-up table allows for further speedup. Second, an adaptive QP scheme is presented that obtains a better bit allocation balance between dark and bright areas of the picture. Third, a method to control the bit allocation balance between chroma and luma by adjusting the chroma QP offset is presented. The result is a considerable increase in perceptual quality compared to the anchors used in the 2015 MPEG High Dynamic Range/Wide Color Gamut Call for Evidence. All techniques are encoder-side-only, making them compatible with a regular decoder capable of supporting HEVC Main10/PQ/BT.2020, which is already available in some TV sets on the market.

Abstract In this paper we investigate low‐bitrate compression of scalar textures such as alpha maps, down to one or two bits per pixel. We present two new techniques for 4 × 4 blocks, based on the idea from ETC to use index tables. We demonstrate that although the visual quality of the alpha maps is greatly reduced at these low bit rates, the quality of the final rendered images appears to be sufficient for a wide range of applications, thus allowing bandwidth savings of up to 75%. The 2 bpp version improves PSNR with over 2 dB compared to BTC at the same bit rate. The 1 bpp version is, to the best of our knowledge, the first public 1 bpp texture compression algorithm, which makes comparison hard. However, compared to just DXT5‐compressing a subsampled texture, our 1 bpp technique improves PSNR with over 2 dB. Finally, we show that some aspects of the presented algorithms are also useful for the more common bit rate of four bits per pixel, achieving PSNR scores around 1 dB better than DXT5, over a set of test images.

During the past few years, mobile phones and other handheld devices have gone from only handling dull text-based menu systems to, on an increasing number of models, being able to render high-quality three-dimensional graphics at high frame rates. This paper is a survey of the special considerations that must be taken when designing graphics processing units (GPUs) on such devices. Starting off by introducing desktop GPUs as a reference, the paper discusses how mobile GPUs are designed, often with power consumption rather than performance as the primary goal. Lowering the bus traffic between the GPU and the memory is an efficient way of reducing power consumption, and therefore some high-level algorithms for bandwidth reduction are presented. In addition, an overview of the different APIs that are used in the handheld market to handle both two-dimensional and three-dimensional graphics is provided. Finally, we present our outlook for the future and discuss directions of future research on handheld GPUs.

The mobile phone is one of the most widespread devices with rendering capabilities. Those capabilities have been very limited because the resources on such devices are extremely scarce; small amounts of memory, little bandwidth, little chip area dedicated for special purposes, and limited power consumption. The small display resolutions present a further challenge; the angle subtended by a pixel is relatively large, and therefore reasonably high quality rendering is needed to generate high fidelity images.To increase the mobile rendering capabilities, we propose a new hardware architecture for rasterizing textured triangles. Our architecture focuses on saving memory bandwidth, since an external memory access typically is one of the most energy-consuming operations, and because mobile phones need to use as little power as possible. Therefore, our system includes three new key innovations: I) an inexpensive multisampling scheme that gives relatively high quality at the same cost of previous inexpensive schemes, II) a texture minification system, including texture compression, which gives quality relatively close to trilinear mipmapping at the cost of 1.33 32-bit memory accesses on average, III) a scanline-based culling scheme that avoids a significant amount of z-buffer reads, and that only requires one context. Software simulations show that these three innovations together significantly reduce the memory bandwidth, and thus also the power consumption.