The inordinate power consumption of contemporary display panels accelerates battery depletion in mobile devices. Existing display-power reduction approaches are suboptimal because they incur additional computations and latency on the mobile device or because they assume a server-client model in which a single entity controls the server and the client. To overcome these limitations, we extract standardized metadata at a server and transmit it to a client along with the video bitstream. This metadata includes quality indicators, associated power controls, and contrast-enhancement information. The metadata guarantees that specified quality levels are achieved while avoiding flicker and minimizing power consumption. Furthermore, the metadata has minimal computational, storage & power overheads and, because it is standardized, the metadata benefits servers and clients in open ecosystems wherein all entities function independently. Our research has been implemented on mobile devices and provides, on average, 32.5 percent power reduction at high quality and up to 84 percent power reduction at acceptable quality. The ISO/IEC Moving Pictures Expert Group has recently standardized our proposed metadata in the Green Metadata Standard.

Super resolution (SR) for real-life video sequences is a challenging problem due to complex nature of the motion fields. In this paper, a novel blind SR method is proposed to improve the spatial resolution of video sequences, while the overall point spread function of the imaging system, motion fields, and noise statistics are unknown. To estimate the blur(s), first, a nonuniform interpolation SR method is utilized to upsample the frames, and then, the blur(s) is(are) estimated through a multi-scale process. The blur estimation process is initially performed on a few emphasized edges and gradually on more edges as the iterations continue. Also for faster convergence, the blur is estimated in the filter domain rather than the pixel domain. The high-resolution frames are estimated using a cost function that has the fidelity and regularization terms of type Huber-Markov random field to preserve edges and fine details. The fidelity term is adaptively weighted at each iteration using a masking operation to suppress artifacts due to inaccurate motions. Very promising results are obtained for real-life videos containing detailed structures, complex motions, fast-moving objects, deformable regions, or severe brightness changes. The proposed method outperforms the state of the art in all performed experiments through both subjective and objective evaluations. The results are available online at http://lyle.smu.edu/~rajand/Video_SR/.

Screen content video coding is becoming increasingly important in various applications, such as desktop sharing, video conferencing, and remote education. In general, compared to natural camera-captured content, screen content has different characteristics, such as sharp edges. In this paper, we propose a novel intra prediction scheme for screen content video. In the proposed scheme, bilinear interpolation in angular intra prediction in HEVC is selectively replaced by nearest-neighbor (NN) interpolation to preserve the sharp edges in screen content video. We present two different variants of NN interpolation. In the first implicit pixel-based method, both the encoder, and the decoder determine whether to perform NN interpolation based on the prediction pixels. The second method comprises of the encoder performing a Rate-Distortion search at a block-level, and explicitly signaling a flag to the decoder to indicate when to use the NN interpolation. Both the proposed variants provide significant gains over HEVC, and simulation results show that average gains of 3.3% BD-bitrate are achieved for screen content video. The HEVC proposal of this method was accepted in the core experiments, and would be a technology under consideration in the ongoing Screen Content Coding extension of HEVC scheduled to begin in March 2014.

Screen content video coding is becoming increasingly important in various applications, such as desktop sharing, video conferencing, and remote education. In January 2014, the ITU-T and ISO/IEC MPEG jointly issued call for proposals for screen content coding as an extension of HEVC. In general, compared to natural camera-captured video content, screen content has different characteristics, e.g., sharper edges, and fewer unique colors on a block-by-block basis. Palette coding for screen content has been recently proposed in the ongoing HEVC Range Extensions standardization. Palette coding utilizes the fact that there are few unique colors in screen content video blocks, and tries to send palettes of these unique colors. However, the size of these palettes can expand, especially in highresolution video. Therefore, to reduce the overall bits for palette transmission, we present various palette prediction techniques in this paper. We first analyze the characteristics of palettes in screen content, and then propose various palette prediction techniques. Our experiments on HEVC test sequences in state-of-the-art HM range extensions software for HEVC show significant improvements over palette coding without any prediction, at almost negligible additional complexity.

In this paper, we provide an overview of the DCT/DST transform scheme for intra coding in the HEVC standard. A unique feature of this scheme is the use of DST-VII transforms in addition to DCT-II. We further derive factorizations for fast joint computation of DCT-II and DST-VII transforms of several sizes. Simulation results for the DCT/DST scheme in the HM reference software for HEVC are also provided together with a discussion on computational complexity.

In this paper, we present a secondary transform scheme for inter-layer prediction residue in scalable video coding (SVC). Efficient prediction of the co-located blocks from the base layer (BL) can significantly improve the enhancement layer (EL) coding in SVC, especially when the temporal information from previous EL frames is less correlated than the co-located BL information. Previously, Guo et al. showed that because of the peculiar frequency characteristics of EL residuals, the conventional DCT Type-2 transform is suboptimal and is often outperformed by either the DCT Type-3, or DST Type-3 when these transforms are applied to the EL residuals. However, their proposed technique requires upto 8 additional transform cores, two of which are of size 32×32. Here, in this work, we propose a low-complexity secondary transform scheme, where the proposed transform is applied only to the lower frequency coefficients after DCT. Our proposed transform scheme requires at most only 2 additional cores. We also present a fast implementation for a class of 4×4 secondary transforms, which actually is a cascade of two DCT Type-2 using only 6 multiplications. Simulation results show that our proposed scheme provides significant BD-Rate improvement over the conventional DCT-based coding scheme in the ongoing scalable extensions of HEVC standardization.

The objective in developing compact descriptors for visual image search is building an image retrieval system that works efficiently and effectively under bandwidth and memory constraints. Selecting local descriptors to be processed, and sending them to the server for matching is an integral part of such a system. One such image search and retrieval system is the Compact Descriptors for Visual Search (CDVS) standardization test model being developed by MPEG which has an efficient local descriptor selection criteria. However, all the existing selection parameters in CDVS are based on low-level features. In this paper, we propose two “mid-level” local descriptor selection criteria: Visual Meaning Score (VMS), and Visual Vocabulary Score (VVS) which can be seamlessly integrated into the existing CDVS framework. A mid-level criteria explicitly allows selection of local descriptors closer to a given set of images. Both VMS and VVS are based on visual words (patches) of images, and provide significant gains over the current CDVS standard in terms of matching accuracy, and have very low implementation cost.

Image matching and search is gaining significant commercial importance nowadays due to various applications it enables such as augmented reality, image-queries for internet search, etc. Many researchers have effectively used color information in an image to improve its matching accuracy. These techniques, however, cannot be directly used for large scale mobile visual search applications that pose strict constraints on the size of the extracted features, computational resources and the system accuracy. To overcome this limitation, we propose a new and effective technique to incorporate color information that can use the SIFT extraction technique. We conduct our experiments on a large dataset containing around 33, 000 images that is currently being investigated in the MPEG-Compact Descriptors for Visual Search Standard and show substantial improvement compared to baseline.

In this paper, we present a secondary transform scheme for inter-layer prediction residue in scalable video coding (SVC). Efficient prediction of the co-located blocks from the base layer (BL) can significantly improve the enhancement layer (EL) coding in SVC, especially when the temporal information from previous EL frames is less correlated than the co-located BL information. However, Guo et al. showed that because of the peculiar frequency characteristics of EL residuals, the conventional DCT Type-2 transform is suboptimal and is often outperformed by either the DCT Type-3, or DST Type-3 when these transforms are applied to the EL residuals. However, their proposed technique requires upto 8 additional transform cores, two of which are of size 32×32. Here, in this work, we propose a secondary transform scheme, where the proposed transform is applied only to the lower 8x8 frequency coefficients after DCT, for block sizes 8×8 to 32×32. Our proposed transform scheme requires at most only 2 additional cores. We also propose a low-complexity 8x8 Rotational Transform as a special case of secondary transforms in this paper. Simulation results show that the proposed transform scheme provides significant BD-Rate improvement over the conventional DCT-based coding scheme for video sequences in the ongoing scalable extensions of HEVC standardization.

In this paper, we present a DCT/DST based transform scheme that applies either the conventional DCT or type-7 DST for all the video-coding intra-prediction modes: vertical, horizontal, and oblique. Our approach is applicable to any block-based intra prediction scheme in a codec that employs transforms along the horizontal and vertical direction separably. Previously, Han, Saxena, and Rose showed that for the intra-predicted residuals of horizontal and vertical modes, the DST is the optimal transform with performance close to the KLT. Here, we prove that this is indeed the case for the other oblique modes. The optimal choice of using DCT or DST is based on intra-prediction modes and requires no additional signaling information or rate-distortion search. The DCT/DST scheme presented in this paper was adopted in the HEVC standardization in March 2011. Further simplifications, especially to reduce implementation complexity, which remove the mode-dependency between DCT and DST, and simply always use DST for the 4 × 4 intra luma blocks, were adopted in the HEVC standard in July 2012. Simulation results conducted for the DCT/DST algorithm are shown in the reference software for the ongoing HEVC standardization. Our results show that the DCT/DST scheme provides significant BD-rate improvement over the conventional DCT based scheme for intra prediction in video sequences.

In this paper, we present a transform scheme where a secondary transform is applied after the conventional DCT for intra as well as inter prediction residues. Our approach is applicable to any block-based video codec that employs transforms along the horizontal and vertical direction separably. The secondary transform is applied to the lower K ( K=4 or 8) frequency coefficients of the output of conventional DCT at block with dimensions 8 and larger. The proposed transform scheme has low complexity as it is applied only to the top-left portion of the DCT output, especially in the context of large blocks such as 32 × 32 where an alternate non-DCT 32 × 32 transform would have a prohibitive implementation hardware cost. The proposed technique is single-pass, and the choice of whether to use the secondary transform is solely based on the prediction direction for intra residue, and on transform unit location in the prediction unit for the inter residue. The scheme requires no additional signaling information or R-D search. Our simulation results show that the proposed transform scheme provides significant BD-rate improvement over the conventional DCT-based coding scheme. Finally, we also show how to implement the proposed secondary transforms with low latency in hardware.

To improve coding efficiency and visual quality in video coding, adaptive loop filtering (ALF) methods using directional features to classify pixels for filter adaptation have been proposed in the literature. However, the associated complexity has been identified as an obstacle for realistic hardware implementation and deployment. This paper presents an ALF design, which efficiently classifies video blocks by computing the direction and strength of local gradients only at subset of pixels. As compared to the state-of-the-art ALF approach utilizing directional features for classification, our design achieves appealing complexity-efficiency trade-off, with more than 85% reductions in computation, 75% reduction in terms of the number of pixels to be accessed, and reduces the worst-case number of filters from 15 to 8; while being able to preserve more than 85% of ALF gams in different coding structures.

In this paper, we present a secondary transform that is applied in a codec after the conventional DCT for all the video-coding intra prediction modes. Our approach is applicable to any block-based intra prediction scheme that employs transforms along the horizontal and vertical direction separably. The secondary transform is applied to the lower 8×8 frequency coefficients of the output of conventional DCT at block sizes 8×8 and higher. The proposed transform scheme has low complexity as it is applied only to the top-left portion of DCT output, especially in the context of large blocks such as 32×32 where an alternate transform of size 32×32 other than DCT would be expensive to be implemented in hardware. The proposed technique works in single-pass, and the choice of when to use the secondary transform is solely based on the intra prediction mode and requires no additional signaling information or R-D search. Our simulation results show that the proposed transform scheme provides significant BD-Rate improvement over the conventional DCT-based coding scheme for intra prediction of video sequences in the ongoing HEVC standardization.

In this paper, we present a novel video fingerprinting algorithm which leverages the concept of perceptual similarity between different video sequences. Inspired by the popular structural similarity (SSIM) index, we quantify the perceptual similarity between different video sequences by proposing a perceptual distance metric (PDM) which is utilized in the matching stage of our proposed video fingerprinting algorithm. PDM requires very simple features, viz., block means and therefore has extremely low complexity in both the feature extraction part, as well as during the matching stage. We also show how to use an order statistic in the proposed distance measure to improve the system performance for localized block-based artifacts such as the logo artifact. Simulation results for the proposed fingerprinting algorithm show significant gains over other video fingerprinting techniques on different video datasets for numerous heavy video artifacts.

In this paper, a unified computational complexity model is proposed to predict the H.264/AVC encoding and decoding computing cycles on popular ARM featured mobile platform. We have developed an analytical complexity model considering the video spatial resolution (i.e., frame size), temporal resolution (i.e., frame rate), and amplitude resolution (i.e., signal amplitude which is usually controlled by compression quantization parameter (QP)). Our proposed model has been validated for H.264/AVC encoding, where x264 is chosen as the typical mobile H.264 encoder. The same analytical model is also extended and verified for the H.264/AVC decoding using FFmpeg. Extensive simulations have been carried out to experiment different scenarios using various video sources at different frame sizes (e.g., HD to QCIF), frame rates (e.g., 60 to 3.75 fps) and bit rates (either using constant QP or rate control). Results demonstrate the high accuracy of our proposed model, with the average relative prediction error less than 7%.

In this paper, we present a transform scheme that utilizes a secondary transform after the conventional DCT for intra as well as inter prediction residues. Our approach is applicable to any block-based video codec that employs transforms along the horizontal and vertical direction separably. The secondary transform is applied to the lower K (K=4 or 8) frequency coefficients of the output of conventional DCT at block with dimensions 8 and higher. The proposed transform scheme has low complexity as it is applied only to the top-left portion of DCT output, especially in the context of large blocks such as 32×32 where an alternate transform of size 32×32 other than DCT would be expensive to be implemented in hardware. The proposed technique works in single-pass, and the choice of whether to use the secondary transform is solely based on the prediction direction for intra residue, and transform unit location in the prediction unit for the inter residue. The scheme requires no additional signaling information or R-D search. Our simulation results show that the proposed transform scheme provides significant BD-Rate improvement over the conventional DCT-based coding scheme for video sequences in the ongoing HEVC standardization.

Mobile video streaming has become one of the most popular applications in the trend of smartphone booming and the prevalence of 3G/4G networks, i.e., HSPA, HSPA+, and LTE. However, the prohibitively high power consumption by 3G/4G radios in smartphones reduces battery life significantly and thus severely hurts user experience. To tackle this challenge, we designed GreenTube, a system that optimizes power consumption for mobile video streaming by judiciously scheduling downloading activities to minimize unnecessary active periods of 3G/4G radio. GreenTube achieves this by dynamically managing the downloading cache based on user viewing history and network condition. We implemented GreenTube on Android-based smartphones. Experimental results show that GreenTube achieves large power reductions of more than 70% (on the 3G/4G radio) and 40% (for the whole system). We believe GreenTube is a desirable upgrade to the Android system, especially in the light of increasing LTE popularity.

This paper proposes an analytical power consumption model for H.264/AVC video decoding using hardware (HW) accelerator on popular mobile platforms. Our proposed model is expressed as the product of the power functions of video spatial resolution (i.e., frame size) and temporal resolution (i.e., frame rate). We have demonstrated that the same analytical model is applicable to different platforms. Model parameters are fixed for a specific platform. This indicates that HW accelerated video decoding is independent of the video content. Simulation results show the high accuracy for video decoding power prediction using proposed model, with the maximum relative prediction error less than 10%. Together with the video bit rate and perceptual quality models published in separated works, we propose to solve the power-rate optimized mobile video streaming problem, so as to maximum the video quality given the limited access network bandwidth and battery life for mobile devices.

To improve video coding efficiency, the Rotational Transform (ROT) was proposed for adaptive switching between different transforms cores. The Karhunen Loeve Transform (KLT) is known to be optimal for given residual but requires much side information to be signaled to the decoder. The Discrete Cosine transform (DCT) is known to be close to optimal but for strongly directional components, it is sub-optimal. The main idea of ROT is that small modification of DCT coefficients can improve energy compaction. The ROT is implemented as a secondary transform applied after the primary DCT. The ROT matrix is sparse and thus enjoys relatively small computational complexity and memory usage increment. The encoder tries every rotational transform from the dictionary. Only one number, the ROT index, needs to be signaled to the decoder. Because the ROT is an orthogonal transform, encoder search is greatly simplified: distortion can be estimated in frequency domain and no inverse transformation is needed. This makes the ROT an efficient way to improve image/video compression. The ROT Coding gain for Intra slice is 2-3% in the HM 1.0 software implementation.

In this paper, we present a mode-dependent transform scheme that applies either the conventional DCT or type-7 DST for all the video-coding intra-prediction modes: vertical, horizontal or oblique. Our approach is applicable to any block-based intra prediction scheme in a codec, that employs transforms along the horizontal and vertical direction separably. Previously Han, Saxena & Rose showed that for the intra-predicted residuals of horizontal and vertical modes, the DST is the optimal transform with performance close to KLT. Here we prove that this is indeed the case for the other oblique modes. The choice of using DCT/DST is based on intra-prediction modes, and requires no additional signaling information or Rate-Distortion search. Simulations are conducted for the DCT/DST algorithm in TMuC 0.9, the reference software for the ongoing HEVC standardization. Our results show that the DCT/DST scheme provides significant BD-Rate improvement over the DCT for intra prediction in video sequences.

Although the discrete wavelet transform (DWT) is a powerful tool for signal and image processing, it has three serious disadvantages: shift sensitivity, poor directionality, and lack of phase information. To overcome these disadvantages, we introduce multidimensional, mapping-based, complex wavelet transforms that consist of a mapping onto a complex function space followed by a DWT of the complex mapping. Unlike other popular transforms that also mitigate DWT shortcomings, the decoupled implementation of our transforms has two important advantages. First, the controllable redundancy of the mapping stage offers a balance between degree of shift sensitivity and transform redundancy. This allows us to create a directional, nonredundant, complex wavelet transform with potential benefits for image coding systems. To the best of our knowledge, no other complex wavelet transform is simultaneously directional and nonredundant. The second advantage of our approach is the flexibility to use any DWT in the transform implementation. As an example, we exploit this flexibility to create the complex double-density DWT: a shift-insensitive, directional, complex wavelet transform with a low redundancy of (3M - 1)/(2M - 1) in M dimensions. No other transform achieves all these properties at a lower redundancy, to the best of our knowledge. By exploiting the advantages of our multidimensional, mapping-based complex wavelet transforms in seismic signal-processing applications, we have demonstrated state-of-the-art results.

The directionality and phase information provided by nonredundant complex wavelet transforms (NCWTs) provide significant potential benefits for image/video processing and compression applications. However, because existing NCWTs are created by downsampling filtered wavelet coefficients, the finest scale of these transforms has a resolution 4/spl times/ lower than the real input signal. In this paper, we propose a linear-phase, semi-orthogonal, directional NCWT design using a novel triband filter bank. At the finest scale, the resulting transform has a resolution 3/spl times/ lower than the real input signal. We provide a design example to demonstrate three important properties for image/video processing applications: directionality, magnitude coherency, and phase coherency.

Shift sensitivity, poor directional selectivity and lack of phase information are three major disadvantages of the discrete wavelet transform, In earlier research, we demonstrated that projection-based complex wavelet transforms have excellent directional selectivity and explicit phase information, In this paper, we discuss the theory of projection-based complex wavelet transforms and we prove that these transforms can have reduced shift sensitivity with low transform redundancy. We also provide experimental results to demonstrate the reduced shift sensitivity of our transforms.

Shift variance and poor directional selectivity, two major disadvantages of the discrete wavelet transform, have previously been circumvented either by using highly redundant, non-separable wavelet transforms or by using restrictive designs to obtain a pair of wavelet trees with a transform-domain redundancy of 4.0 in 2D. We demonstrate that excellent shift-invariance properties and directional selectivity may be obtained with a transform-domain redundancy of only 2.67 in 2D. We achieve this by projecting the wavelet coefficients from Selesnick's (Selesnick et al., 2000) almost-shift-invariant, double-density wavelet transform so as to separate approximately the positive and negative frequencies, thereby increasing directionality. Subsequent decimation and a novel inverse projection maintain the low redundancy while ensuring perfect reconstruction. Although our transform generates complex-valued coefficients allowing processing capabilities that are impossible with real-valued coefficients, it may be implemented with a fast algorithm that uses only real arithmetic. To demonstrate the efficacy of our new transform, we show that it achieves state-of-the-art performance in a seismic image-processing application.

Shift sensitivity and poor directionality, two major disadvantages of the discrete wavelet transform, have previously been circumvented either by using highly redundant, non-separable wavelet transforms or by using restrictive designs to obtain a pair of wavelet trees with a transform-domain redundancy of 4.0 in 2D. We demonstrate that excellent shift-invariance properties and directional selectivity may be obtained with a transform-domain redundancy of only 2.67 in 2D. We achieve this by projecting the wavelet coefficients from Selesnick's (see Wavelet Applications VII, Proceedings of SPIE, 2000) shift-insensitive, double-density wavelet transform so as to separate approximately the positive and negative frequencies, thereby increasing directionality. Subsequent decimation and a novel inverse projection maintain the low redundancy while ensuring perfect reconstruction. Although our transform generates complex-valued coefficients that provide valuable phase information, it may be implemented with a fast algorithm that uses only real arithmetic. To demonstrate the efficacy of our new transform, we show that it achieves state-of-the-art performance in a seismic image-processing application.

Poor directional selectivity, a major disadvantage of the 2D separable discrete wavelet transform (DWT), has heretofore been circumvented either by using highly redundant, nonseparable wavelet transforms or by using restrictive designs to obtain a pair of wavelet trees. In this paper, we demonstrate that superior directional selectivity may be obtained with no redundancy in any separable wavelet transform. We achieve this by projecting the wavelet transform coefficients onto the Softy space of signals and decimating before processing. A novel reconstruction step guarantees perfect reconstruction within this critically-sampled framework.

This paper describes the first steps toward a multiwavelet system that may retain the advantages of a traditional multiwavelet system while alleviating some of its disadvantages. We attempt to achieve this through the introduction of a novel property-the disjoint support of the multiscaling functions. We derive the conditions on the matrix filter coefficients that guarantee the disjoint support of multiscaling functions. The preliminary results demonstrate that multiwavelet systems with this property may be arbitrarily complex. We then establish the existence of multiwavelet systems with two scaling functions and approximation order=2.