In the past decade, the development of transform coding techniques has achieved significant progress and several advanced transform tools have been adopted in the new generation Versatile Video Coding (VVC) standard. In this paper, a brief history of transform coding development during VVC standardization is presented, and the transform coding tools in the VVC standard are described in detail together with their initial design, incremental improvements and implementation aspects. To improve coding efficiency, four new transform coding techniques are introduced in VVC, which are namely Multiple Transform Selection (MTS), Low-Frequency Non-separable Secondary Transform (LFNST) and Sub-Block Transform (SBT), as well as a large (64-point) type-2 DCT. The experimental results on VVC reference software (VTM-9.0) show that average 4.5% and 3.6% overall coding gain can be achieved by the VVC transform coding tools for All Intra and Random Access configurations, respectively.

In many video coding systems, separable transforms (such as two-dimensional DCT-2) have been used to code block residual signals obtained after prediction. This paper proposes a parametric approach to build graph-based separable transforms (GBSTs) for video coding. Specifically, a GBST is derived from a pair of line graphs, whose weights are determined based on two non-negative parameters. As certain choices of those parameters correspond to the discrete sine and cosine transform types used in recent video coding standards (including DCT-2, DST-7 and DCT-8), this paper further optimizes these graph parameters to better capture residual block statistics and improve video coding efficiency. The proposed GBSTs are tested on the Versatile Video Coding (VVC) reference software, and the experimental results show that about 0.4% average coding gain is achieved over the existing set of separable transforms constructed based on DCT-2, DST-7 and DCT-8 in VVC.

This article describes the main video coding technologies included in a joint proposal submitted by Qualcomm and Technicolor, in response to a Call for Proposals (CfP) issued by ITU-T SG16 WP3 Q.6 (VCEG) and ISO/IEC JTC1/SC29/WG11 (MPEG) in Oct. 2017. The proposal contains the majority of the tools that have been adopted into the Joint Exploration Model (JEM), developed in the exploratory phase that preceded the CfP. A flexible multi-tree type (MTT) block-partitioning scheme is proposed to extend the quadtree and binary tree (QTBT) based partitioning in JEM by including triple tree (TT) and asymmetric binary tree (ABT) partitions. In addition, several JEM tools in intra and inter prediction, transforms and arithmetic coding are modified, and new tools such as sign prediction and motion compensated padding are proposed. Objective standard dynamic range (SDR) gains of 43.1% and 15.5% in terms of average luma BD-rate improvement have been achieved for the CfP constraint set 1 (random-access configuration) relative to HEVC/H.265 (HM) and JEM anchors, respectively. For the CfP constraint set 2 (low-delay configuration), the average luma BD-rate improvements are 33.7% relative to the HM anchor and 12.7% relative to the JEM anchor. The proposed codec scored highly in both subjective evaluations and objective metrics and was among the best-performing CfP proposals.

In many state-of-the-art compression systems, signal transformation is an integral part of the encoding and decoding process, where transforms provide compact representations for the signals of interest. This paper introduces a class of transforms called graph-based transforms (GBTs) for video compression, and proposes two different techniques to design GBTs. In the first technique, we formulate an optimization problem to learn graphs from data and provide solutions for optimal separable and nonseparable GBT designs, called GL-GBTs. The optimality of the proposed GL-GBTs is also theoretically analyzed based on Gaussian-Markov random field (GMRF) models for intra and inter predicted block signals. The second technique develops edge-adaptive GBTs (EA-GBTs) in order to flexibly adapt transforms to block signals with image edges (discontinuities). The advantages of EA-GBTs are both theoretically and empirically demonstrated. Our experimental results show that the proposed transforms can significantly outperform the traditional Karhunen-Loeve transform (KLT).

Recent video codecs with multiple separable transforms can achieve significant coding gains using asymmetric trigonometric transforms (DCTs and DSTs), because they can exploit diverse statistics of residual block signals. However, they add excessive computational and memory complexity on large transforms (32-point and larger), since their practical software and hardware implementations are not as effi-cient as of the DCT-2. This article introduces a novel technique to design low-complexity approximations of trigonometric transforms. The proposed method uses DCT-2 computations, and applies orthogonal "adjustments" to approximate the most important basis vectors of the desired transform. Experimental results on the Versatile Video Coding (VVC) reference software show that the proposed approach significantly reduces the computational complexity, while providing practically identical coding efficiency.

In existing video coding standards such as H.264/AVC and HEVC, the intra prediction is typically derived using fixed, symmetric prediction filters along the prediction direction, e.g., in planar mode, top-right and bottom-left samples are predicted using symmetric prediction filters. However, in case of asymmetric availability of neighboring reference samples, the performance of intra prediction filters designed in HEVC may not be optimal. To further refine the intra prediction and achieve higher accuracy of prediction samples, this paper proposes low-complexity refinements over HEVC intra prediction, which are applied on frequently used planar, DC, horizontal and vertical modes. The proposed method only requires simple addition and bit-shift operations on top of HEVC's intra prediction implementation. Experimental results show that, an average of 0.7% coding gain is achieved for intra coding with no increase in run-time complexity.

This paper introduces a novel framework for combining multiple weighted graphs into a single optimized weighted graph. In our framework, we first develop a statistical formulation for the graph combining problem with a maximum likelihood criterion, and derive its optimality conditions. We then use these conditions to formulate the deterministic graph combining problem and propose a solution. Our experimental results show that the proposed solution provides better modeling compared to the commonly used averaging method. The introduced framework has various applications in signal processing and machine learning.

In this paper, we propose a new edge model for edge adaptive graph-based transforms (EA-GBTs) in video compression. In particular, we consider step and ramp edge models to design graphs used for defining transforms, and compare their performance on coding intra and inter predicted residual blocks. In order to reduce the signaling overhead of block-adaptive coding, a new edge coding method is introduced for the ramp model. Our experimental results show that the proposed methods outperform classical DCT-based encoding and that ramp edge models provide better performance than step edge models for intra predicted residuals.

This paper introduces a novel class of transforms, called graph-based separable transforms (GBSTs), based on two line graphs with optimized weights. For the optimal GBST construction, we formulate a graph learning problem to design two separate line graphs using row-wise and column-wise residual block statistics, respectively. We also analyze the optimality of resulting separable transforms for both intra and inter predicted residual block models. Moreover, we show that separable DCT and ADST (DST-7) are special cases of the GBSTs. Our experimental results demonstrate that the proposed optimized transforms outperform 2-D DCT/ADST and separable KLT.

This paper introduces row-column transforms (RCTs) which are 2D non-separable transforms defined with the aid of a set of 1-D linear transforms and a basis ordering permutation. We propose a novel method for the design of row-column transforms that approximate desired complex transforms (such as KLTs, SOTs, etc.) so that most of the performance of the approximated transforms is retained at significantly reduced complexity. Given a non-separable block transform of interest, our method designs an RCT by (i) optimizing a set of 1-D transforms applied to rows and columns of the signal block and (ii) finding the best transform coefficient ordering permutation. Our experimental results show that optimized RCTs closely approach the compression performance of the desired non-separable transforms while retaining the computational complexity of separable transforms.

For the last few decades, the application of signal-adaptive transform coding to video compression has been stymied by the large computational complexity of matrix-based solutions. In this paper, we propose a novel parametric approach to greatly reduce the complexity without degrading the compression performance. In our approach, instead of following the conventional technique of identifying full transform matrices that yield best compression efficiency, we look for the best transform parameters defining a new class of transforms, called HyGTs, which have low complexity implementations that are easy to parallelize. The proposed HyGTs are implemented as an extension of High Efficiency Video Coding (HEVC), and our comprehensive experimental results demonstrate that proposed HyGTs improve average coding gain by 6% bit rate reduction, while using 6.8 times less memory than KLT matrices.

In traditional image and video coding schemes, separable transforms are typically employed due to their low-complexity implementations. However, the compression efficiency of separable transforms is limited for most natural image/video blocks which generally have arbitrarily directed edge and texture patterns. It is well known that non-separable transforms can achieve better compression efficiency for directional texture patterns, yet they are computationally complex, especially for larger block sizes. In order to achieve higher transform coding gains with relatively low-complexity implementations, in this paper, we propose non-separable secondary transforms (NSSTs). The proposed approach applies a secondary non-separable transform on a sub-block of low frequency coefficients generated using a primary separable transform, such as discrete cosine transform (DCT). Since the proposed NSST is a non-separable transform applied on low frequency coefficients in a much smaller block size, which typically captures most of the signal energy, better coding gains can be achieved with at a relatively low-computational cost. Experimental results show that, compared to the latest HEVC reference software (HM16.6), the proposed method achieves up to a significant 12% coding gain for Intra coding.

Spectrum sensing is an essential functionality of cognitive radio wireless networks (CRWNs) that enables detecting unused frequency sub-bands for dynamic spectrum access. This paper proposes a compressed spectrum sensing framework by (i) constructing a sparsity basis in wavelet domain that helps compressed sensing at sub-Nyquist rates and (ii) applying a wavelet-based singularity detector on the reconstructed signal to identify available frequency sub-bands with low complexity. In particular, for the compressed sensing, an optimized Haar wavelet basis is employed to sparsely represent piecewise constant (PWC) signals which closely approximates the frequency spectrum of a sensed signal. Our simulation results show that our proposed framework outperforms existing compressed spectrum sensing methods by providing higher accuracy at lower sampling rates.

In video coding, motion compensation is an essential tool to obtain residual block signals whose transform coefficients are encoded. This paper proposes novel graph-based transforms (GBTs) for coding inter-predicted residual block signals. Our contribution is twofold: (i) We develop edge adaptive GBTs (EA-GBTs) derived from graphs estimated from residual blocks, and (ii) we design template adaptive GBTs (TA-GBTs) by introducing simplified graph templates generating different set of GBTs with low transform signaling overhead. Our experimental results show that proposed methods significantly outperform traditional DCT and KLT in terms of rate-distortion performance.

The Karhunen-Loeve transform (KLT) is known to be optimal for decorrelating stationary Gaussian processes, and it provides effective transform coding of images. Although the KLT allows efficient representations for such signals, the transform itself is completely data-driven and computationally complex. This paper proposes a new class of transforms called graph template transforms (GTTs) that approximate the KLT by exploiting a priori information known about signals represented by a graph-template. In order to construct a GTT (i) a design matrix leading to a class of transforms is defined, then (ii) a constrained optimization framework is employed to learn graphs based on given graph templates structuring a priori known information. Our experimental results show that some instances of the proposed GTTs can closely achieve the rate-distortion performance of KLT with significantly less complexity.

This paper introduces a novel spectral anomaly detection method by developing a graph-based filtering framework. In particular, we consider the problem of unsupervised data anomaly detection over wireless sensor networks (WSNs) where sensor measurements are represented as signals on a graph. In our framework, graphs are chosen to capture useful proximity information about measured data. The associated graph-based filters are then employed to project the graph signals on normal and anomaly subspaces, and resulting projections are used in detection of data anomalies. The proposed approach has two main advantages over the standard spectral technique, principal component analysis (PCA). Firstly, graph-based filtering allows us to incorporate structural information known a priori (e.g., distance between sensors) in addition to data. Secondly, it provides localized transformations leading to effective distributed anomaly detection. Our experimental results show that our proposed solution outperforms PCA-based and distributed clustering-based anomaly detection methods in terms of receiver operating characteristics (ROCs).

This paper presents novel QoS extensions to distributed control plane architectures for multimedia delivery over large-scale, multi-operator Software Defined Networks (SDNs). We foresee that large-scale SDNs shall be managed by a distributed control plane consisting of multiple controllers, where each controller performs optimal QoS routing within its domain and shares summarized (aggregated) QoS routing information with other domain controllers to enable inter-domain QoS routing with reduced problem dimensionality. To this effect, this paper proposes (i) topology aggregation and link summarization methods to efficiently acquire network topology and state information, (ii) a general optimization framework for flow-based end-to-end QoS provision over multi-domain networks, and (iii) two distributed control plane designs by addressing the messaging between controllers for scalable and secure inter-domain QoS routing. We apply these extensions to streaming of layered videos and compare the performance of different control planes in terms of received video quality, communication cost and memory overhead. Our experimental results show that the proposed distributed solution closely approaches the global optimum (with full network state information) and nicely scales to large networks.

This demo abstract describes an initial design of a new adaptive video streaming protocol for device-to-device WiFi-based mobile platforms and its software implementation. For the demonstration, two mobile servers and two mobile users will be deployed verifying that our device-to-device adaptive video streaming implementation works with desirable user experience.

OpenFlow is a programmable network protocol and associated hardware designed to effectively manage and direct traffic by decoupling control and forwarding layers of routing. This paper presents an analytical framework for optimization of forwarding decisions at the control layer to enable dynamic Quality of Service (QoS) over OpenFlow networks and discusses application of this framework to QoS-enabled streaming of scalable encoded videos with two QoS levels. We pose and solve optimization of dynamic QoS routing as a constrained shortest path problem, where we treat the base layer of scalable encoded video as a level-1 QoS flow, while the enhancement layers can be treated as level-2 QoS or best-effort flows. We provide experimental results which show that the proposed dynamic QoS framework achieves significant improvement in overall quality of streaming of scalable encoded videos under various coding configurations and network congestion scenarios.

This paper proposes a new Quality of Service (QoS) optimized routing architecture for video streaming over large-scale multi-domain OpenFlow networks managed by a distributed control plane, where each controller performs optimal routing within its domain and shares summarized intra-domain routing data with other controllers to reduce problem dimensionality for calculating inter-domain routing. We apply the proposed architecture to streaming of scalable (layered) videos, where the base layer routes are dynamically optimized to fulfill a required QoS level, while enhancement layers follow traditional shortest path. We show that the proposed solution approaches the expensive non-scalable globally optimal solution (single controller for the whole network) in terms of received video quality under various congestion scenarios.

OpenFlow is a clean-slate Future Internet architecture that decouples control and forwarding layers of routing, which has recently started being deployed throughout the world for research purposes. This paper presents an optimization framework for the OpenFlow controller in order to provide QoS support for scalable video streaming over an OpenFlow network. We pose and solve two optimization problems, where we route the base layer of SVC encoded video as a lossless-QoS flow, while the enhancement layers can be routed either as a lossy-QoS flow or as a best effort flow, respectively. The proposed approach differs from current QoS architectures since we provide dynamic rerouting capability possibly using non-shortest paths for lossless and lossy QoS flows. We show that dynamic rerouting of QoS flows achieves significant improvement on the video's overall PSNR under network congestion.