In this paper, we present a novel in-loop filter for video coding which is based on a convolutional neural network (CNN). For that, the adaptive loop filter (ALF) of Versatile Video Coding (VVC) is generalized to define the model architecture for a CNN-based in-loop filter which requires significantly lower computational complexity compared to other existing CNN-based in-loop filters. Experimental results show that, under the JVET common test conditions, BD-rate savings of 1.84% and 1.83% can be achieved compared to VVC for the all-intra and random-access configurations, respectively. At the same time, the computational complexity is at only about 455 multiplications per luma sample.

In this paper, a data-driven generalization of the adaptive loop filter (ALF) of Versatile Video Coding (VVC) is presented. It is shown how the conventional ALF process of classification and FIR filtering can be generalized to define a natural model architecture for convolutional neural network (CNN) based in-loop filters. Experimental results show that over VVC, average bit-rate savings of 3.85%/4.75% and 4.39%/4.33% can be achieved for the all intra and random access configurations in the low- and high-QP settings.

Recently, convolutional neural network (CNN)-based in-loop filters have been introduced for video coding and they show huge coding gains. However, one of the main issues of this approach is the high computational complexity of these filters. In this paper, we present various settings for CNN-based in-loop filters targeting on the reduction of their decoder complexity and describe the corresponding gain-complexity trade-offs. For this, we introduce an effective complexity measure and show that it is possible to notably reduce this value for some CNN-based in-loop filters while keeping the compression gain over Versatile Video Coding (VVC).

In this paper, we describe a video coding design that enables a higher coding efficiency than the HEVC standard. The proposed video codec follows the design of block-based hybrid video coding, but includes a number of advanced coding tools. A part of the incorporated advanced concepts was developed by the Joint Video Exploration Team, while others are newly proposed. The key aspects of these newly proposed tools are the following. A video frame is subdivided into rectangles of variable size using a binary partitioning with variable split ratios. Three new approaches for generating spatial intra prediction signals are supported: A line-wise application of conventional intra prediction modes, coupled with a mode-dependent processing order, a region-based template matching prediction method and intra prediction modes based on neural networks. For motion-compensated prediction, a multi-hypothesis mode with more than two motion hypotheses can be used. In transform coding, mode dependent combinations of primary and secondary transforms are applied. Moreover, scalar quantization is replaced by trellis-coded quantization and the entropy coding of the quantized transform coefficients is improved. The intra and inter prediction signals can be filtered using an edge-preserving diffusion filter or a non-linear DCT-based thresholding operation. The video codec includes an adaptive in-loop filter for which one of three classifiers can be chosen on a picture basis. We also incorporated an optional encoder control, which adjusts the quantization parameters based on a perceptually motivated distortion measure. In a random access scenario, our proposed video codec achieves luma BD-rate savings between 32.5% for HDR HLG UHD and 39.6% for SDR UHD over the HEVC (HM software) anchor for different categories of test sequences.

In-loop filtering is an important task in video coding, as it refines both the reconstructed signal for display and the pictures used for inter-prediction. At the current stage of the Versatile Video Coding (VVC) standardization, there are three in-loop filtering procedures consisting of deblocking filter (DBF), sample adaptive offset (SAO) and adaptive loop filter (ALF). Among them, SAO is the simplest in-loop filtering process and highly effective in removing coding artifacts. It simply modifies decoded samples by conditionally adding an offset value to each sample after the application of the DBF. For this, a classification is applied for each sample location, which gives a partition of the set of all sample locations. After that, an offset value is added to all samples associated with each class. Therefore, the performance of SAO essentially relies on how its classification behaves. In this paper, we introduce a novel classification method for SAO. Based on this, we derive an additional SAO filtering process which we call post sample adaptive offset (PSAO). Experimental results show the effectiveness of our proposed PSAO filtering process. On average, 0.42%, 0.31% and 0.33% additional coding gains can be achieved on top of VTM-5.0 for all intra (AI), random access (RA) and low delay with B pictures (LB) configurations, respectively.

In video coding, adaptive loop filter (ALF) has attracted attention due to its increasing coding performances. Recently ALF has been further developed for its extension, which introduces geometry transformation-based adaptive loop filter (GALF) outperforming the existing ALF techniques. The main idea of ALF is to apply a classification to obtain multiple classes, which gives a partition of a set of all pixel locations. After that, a Wiener filter is applied for each class. Therefore, the performance of ALF essentially relies on how its classification behaves. In this paper, we introduce a novel classification method, Multiple feature-based Classifications ALF (MCALF) extending a classification in GALF and show that it increases coding efficiency while only marginally raising encoding complexity. The key idea is to apply more than one classifier at the encoder to group all reconstructed samples and then to select a classifier with the best RD-performance to carry out the classification process. Simulation results show that around 2% bit rate reduction can be achieved on top of GALF for some selected test sequences.

Applications such as magnetic resonance imaging acquire imaging data by point samples of their Fourier transform. This raises the question of balancing the efficiency of the sampling strategies with the approximation accuracy of an associated reconstruction procedure. In this paper, we introduce a novel sampling-reconstruction scheme based on a random anisotropic sampling pattern and a compressed sensing--type reconstruction strategy with a variant of dualizable shearlet frames as sparsifying representation system. For this scheme, we prove asymptotic almost optimality in an approximation theoretic sense for cartoon-like functions as a model class for the imaging data. Finally, we present numerical experiments showing the superiority of our scheme over other approaches.

Wavelets and their associated transforms are highly efficient when approximating and analyzing one-dimensional signals. However, multivariate signals such as images or videos typically exhibit curvilinear singularities, which wavelets are provably deficient in sparsely approximating and also in analyzing in the sense of, for instance, detecting their direction. Shearlets are a directional representation system extending the wavelet framework, which overcomes those deficiencies. Similar to wavelets, shearlets allow a faithful implementation and fast associated transforms. In this article, we will introduce a comprehensive carefully documented software package coined ShearLab 3D (www.ShearLab.org) and discuss its algorithmic details. This package provides MATLAB code for a novel faithful algorithmic realization of the 2D and 3D shearlet transform (and their inverses) associated with compactly supported universal shearlet systems incorporating the option of using CUDA. We will present extensive numerical experiments in 2D and 3D concerning denoising, inpainting, and feature extraction, comparing the performance of ShearLab 3D with similar transform-based algorithms such as curvelets, contourlets, or surfacelets. In the spirit of reproducible research, all scripts are accessible on www.ShearLab.org.

This paper proposes an image interpolation algorithm exploiting sparse representation for natural images. It involves three steps: (a) obtaining an initial estimate of the high resolution image using linear methods like FIR filtering, (b) promoting sparsity in a selected dictionary through thresholding and (c) extracting high frequency information from the approximation and adding it to the initial estimate. For the sparse modeling, a shearlet dictionary is chosen to yield a multi-scale directional representation. The proposed algorithm is compared to several state-of-the-art methods to assess its objective as well as subjective performance. Compared to the cubic spline interpolation method, an average PSNR gain of around 0.7 dB is observed over a dataset of 200 images.

Over the past few years, various representation systems which sparsely approximate functions governed by anisotropic features, such as edges in images, have been proposed. Alongside the theoretical development of these systems, algorithmic realizations of the associated transforms are provided. However, one of the most common shortcomings of these frameworks is the lack of providing a unified treatment of the continuum and digital world, i.e., allowing a digital theory to be a natural digitization of the continuum theory. In this paper, we introduce a new shearlet transform associated with a nonseparable shearlet generator, which improves the directional selectivity of previous shearlet transforms. Our approach is based on a discrete framework, which allows a faithful digitization of the continuum domain directional transform based on compactly supported shearlets introduced as means to sparsely encode anisotropic singularities of multivariate data. We show numerical experiments demonstrating the potential of our new shearlet transform in 2D and 3D image processing applications.

It is now widely acknowledged that analyzing the intrinsic geometrical features of the underlying image is essential in many applications including image processing. In order to achieve this, several directional image representation schemes have been proposed. In this paper, we develop the discrete shearlet transform (DST) which provides efficient multiscale directional representation and show that the implementation of the transform is built in the discrete framework based on a multiresolution analysis (MRA). We assess the performance of the DST in image denoising and approximation applications. In image approximations, our approximation scheme using the DST outperforms the discrete wavelet transform (DWT) while the computational cost of our scheme is comparable to the DWT. Also, in image denoising, the DST compares favorably with other existing transforms in the literature.