An unfocused plenoptic light field (LF) camera places an array of microlenses in front of an image sensor in order to separately capture different directional rays arriving at an image pixel. Using a conventional Bayer pattern, data captured at each pixel is a single color component (R, G or B). The sensed data then undergoes demosaicking (interpolation of RGB components per pixel) and conversion to an array of sub-aperture images (SAIs). In this paper, we propose a new LF image coding scheme based on graph lifting transform (GLT), where the acquired sensor data are coded in the original captured form without pre-processing. Specifically, we directly map raw sensed color data to the SAIs, resulting in sparsely distributed color pixels on 2D grids, and perform demosaicking at the receiver after decoding. To exploit spatial correlation among the sparse pixels, we propose a novel intra-prediction scheme, where the prediction kernel is determined according to the local gradient estimated from already coded neighboring pixel blocks. We then connect the pixels by forming a graph, modeling the prediction residuals statistically as a Gaussian Markov Random Field (GMRF). The optimal edge weights are computed via a graph learning method using a set of training SAIs. The residual data is encoded via low-complexity GLT. Experiments show that at high PSNRs-important for archiving and instant storage scenarios-our method outperformed significantly a conventional light field image coding scheme with demosaicking followed by High Efficiency Video Coding (HEVC).

In an increasingly connected world, consumer video experiences have diversified away from traditional broadcast video into new applications with increased use of non-camera-captured content such as computer screen desktop recordings or animations created by computer rendering, collectively referred to as screen content. There has also been increased use of graphics and character content that is rendered and mixed or overlaid together with camera-generated content. The emerging Versatile Video Coding (VVC) standard, in its first version, addresses this market change by the specification of low-level coding tools suitable for screen content. This is in contrast to its predecessor, the High Efficiency Video Coding (HEVC) standard, where highly efficient screen content support is only available in extension profiles of its version 4. This paper describes the screen content support and the five main low-level screen content coding tools in VVC: transform skip residual coding (TSRC), block-based differential pulse-code modulation (BDPCM), intra block copy (IBC), adaptive color transform (ACT), and the palette mode. The specification of these coding tools in the first version of VVC enables the VVC reference software implementation (VTM) to achieve average bit-rate savings of about 41% to 61% relative to the HEVC test model (HM) reference software implementation using the Main 10 profile for 4:2:0 screen content test sequences. Compared to the HM using the Screen-Extended Main 10 profile and the same 4:2:0 test sequences, the VTM provides about 19% to 25% bit-rate savings. The same comparison with 4:4:4 test sequences revealed bit-rate savings of about 13% to 27% for$Y'C_{B}C_{R}$and of about 6% to 14% for$R'G'B'$screen content. Relative to the HM without the HEVC version 4 screen content coding extensions, the bit-rate savings for 4:4:4 test sequences are about 33% to 64% for$Y'C_{B}C_{R}$and 43% to 66% for$R'G'B'$screen content.

In many state-of-the-art compression systems, signal transformation is an integral part of the encoding and decoding process, where transforms provide compact representations for the signals of interest. This paper introduces a class of transforms called graph-based transforms (GBTs) for video compression, and proposes two different techniques to design GBTs. In the first technique, we formulate an optimization problem to learn graphs from data and provide solutions for optimal separable and nonseparable GBT designs, called GL-GBTs. The optimality of the proposed GL-GBTs is also theoretically analyzed based on Gaussian-Markov random field (GMRF) models for intra and inter predicted block signals. The second technique develops edge-adaptive GBTs (EA-GBTs) in order to flexibly adapt transforms to block signals with image edges (discontinuities). The advantages of EA-GBTs are both theoretically and empirically demonstrated. Our experimental results show that the proposed transforms can significantly outperform the traditional Karhunen-Loeve transform (KLT).

Recent video codecs with multiple separable transforms can achieve significant coding gains using asymmetric trigonometric transforms (DCTs and DSTs), because they can exploit diverse statistics of residual block signals. However, they add excessive computational and memory complexity on large transforms (32-point and larger), since their practical software and hardware implementations are not as effi-cient as of the DCT-2. This article introduces a novel technique to design low-complexity approximations of trigonometric transforms. The proposed method uses DCT-2 computations, and applies orthogonal "adjustments" to approximate the most important basis vectors of the desired transform. Experimental results on the Versatile Video Coding (VVC) reference software show that the proposed approach significantly reduces the computational complexity, while providing practically identical coding efficiency.

A plenoptic light field (LF) camera places an array of microlenses in front of an image sensor, in order to separately capture different directional rays arriving at an image pixel. Using a Bayer pattern, data captured at each pixel is a single color component (R, G or B). The sensed data then undergoes demosaicking (interpolation of RGB components per pixel) and conversion to a series of subaperture images. In this paper, we propose a novel LF image coding scheme based on graph lifting transform, where the acquired sensor data are coded in their original form without pre-processing. Specifically, demosaicking is not performed, and instead we first map raw sensed color data directly to subaperture image 2D grids, then encode the color pixels, which are sparse in spatial distribution, via a graph lifting transform. Our method avoids redundancies stemming from demosaicking, and operates in the original RGB domain without color conversion and sub-sampling. The graph lifting transform efficiently encodes irregularly spaced pixels in each subaperture image, resulting in compact representations. Experiments show that at high PSNRs - important for archiving and instant storage scenarios - our method outperforms demosaicking followed by intra-only High Efficiency Video Coding (HEVC) significantly.

Hyperspectral imaging captures the spectral responses of different wavelengths per pixel for an entire image. Because the number of spectral bands is large, efficient compression of hyperspectral images is important. Leveraging on recent advances in graph signal processing (GSP), in this paper we propose to encode a hyperspectral image in groups of ω spectral bands using graph wavelets, exploiting correlations along both the spatial and the spectral dimensions. Specifically, along the spatial dimension, we estimate the inter-pixel correlations for all adjacent pixel pairs from the last image in the previous coded band group. Along the spectral dimension, we first divide an image into different spatial regions with similar spectral responses, and encode the spectral signature (correlations along the spectrum) for each region as side information (SI). The spatial / spectral correlations are used to compute edge weights to construct a graph for signal-adaptive graph wavelet based compression. Experimental results suggest that our proposal can outperform existing schemes noticeably at comparable complexity.

In this paper, we propose a graph-based lifting transform for intra-predicted video sequences. The transform can approximate the performance of a Graph Fourier Transform (GFT) for a given graph, but does not require computing eigenvectors. A predict-update bipartition is designed based on a Gaussian Markov Random Field (GMRF) model with the goal to minimize the energy in the prediction set. Additionally, a novel re-connection method is applied for multi-level graphs, leading to significant gain for the proposed bipartition method and for the conventional MaxCut based bipartition. Experiments on intra-predicted video sequences show that the proposed method, even considering the extra overhead for edge information, outperforms the Discrete Cosine Transform (DCT) and approximates the performance of the higher complexity GFT.

In this paper, we propose a new edge model for edge adaptive graph-based transforms (EA-GBTs) in video compression. In particular, we consider step and ramp edge models to design graphs used for defining transforms, and compare their performance on coding intra and inter predicted residual blocks. In order to reduce the signaling overhead of block-adaptive coding, a new edge coding method is introduced for the ramp model. Our experimental results show that the proposed methods outperform classical DCT-based encoding and that ramp edge models provide better performance than step edge models for intra predicted residuals.

This paper introduces a novel class of transforms, called graph-based separable transforms (GBSTs), based on two line graphs with optimized weights. For the optimal GBST construction, we formulate a graph learning problem to design two separate line graphs using row-wise and column-wise residual block statistics, respectively. We also analyze the optimality of resulting separable transforms for both intra and inter predicted residual block models. Moreover, we show that separable DCT and ADST (DST-7) are special cases of the GBSTs. Our experimental results demonstrate that the proposed optimized transforms outperform 2-D DCT/ADST and separable KLT.

In video coding, motion compensation is an essential tool to obtain residual block signals whose transform coefficients are encoded. This paper proposes novel graph-based transforms (GBTs) for coding inter-predicted residual block signals. Our contribution is twofold: (i) We develop edge adaptive GBTs (EA-GBTs) derived from graphs estimated from residual blocks, and (ii) we design template adaptive GBTs (TA-GBTs) by introducing simplified graph templates generating different set of GBTs with low transform signaling overhead. Our experimental results show that proposed methods significantly outperform traditional DCT and KLT in terms of rate-distortion performance.

We present a novel edge adaptive depth map coding based on lifting on graphs. The transform is localized, of low complexity, and guarantees perfect reconstruction as long as a proper predict-update split is defined. During the transform process, data in the prediction set are predicted by data in the update set; the prediction errors are then stored for encoding. In order to reduce the energy of the prediction residue, we propose to use optimized sampling on graphs to select the update set. Experiments show that the optimized sampling approach achieves better results than the conventional maximum cut based splitting in terms of transform efficiency and reconstruction quality. In addition, performance using the lifting transform is comparable to the state-of-the-art graph based depth map encoder using graph Fourier transform (GFT), which requires high complexity for signal projection.

Multi-view video transmission poses great challenges because of its data size and dimension. Therefore, how to design efficient 3D scene representations and coding (of luminance and geometry) has become a critical research topic. Recently, the graph-based representation (GBR) is introduced, which provides a lossless compression of multi-view geometry by connecting informative pixels among views. This representation has been shown as a promising alternative to the classical depth-based representation, where the view synthesis accuracy is hard to control. In this work, we study the luminance compression under GBR, which is not well considered in existing literature. With a proper structural reformulation, we show that the graph-based transform can be applied on the GBR paradigm, hence better extracting the correlation among pixels along graph connections. Moreover, we extend the popular SPIHT coding scheme to further improve coding efficiency. The experimental results show that our method leads to better RD coding performance as compared the classical luminance coding algorithms.