This paper describes a video coding technology proposal submitted by MediaTek in April 2018 in response to a joint call for proposals (CfP) issued by ITU-T VCEG and ISO/IEC MPEG. The proposal uses the conventional block-based hybrid coding approach with a breakthrough design of block structure, substantial improvements of coding tools, and new inventions of loop filtering techniques. First, the block structure is designed using a highly flexible partitioning scheme, where each coding tree unit in a picture is split into multiple coding units (CUs) by a recursive quaternary tree followed by a recursive binary-ternary tree. The prediction unit (PU) and transform unit (TU) concepts are unified with the CU concept, i.e., PU and TU are basically the same as CU. Moreover, inter prediction, intra prediction, transform, quantization, in-loop filtering, and entropy coding are all improved. Finally, new loop filtering techniques are invented, where the proposed convolutional neural network loop filtering is of particular interest. The proposed video codec achieves 43.81% average bit rate savings based on objective measures compared to the High Efficiency Video Coding (HEVC) anchors and is among the best-performing CfP responses both objectively and subjectively. It has been partially adopted into the working draft of the Versatile Video Coding (VVC) standard.

Image compression has always been an important topic in the last decades due to the explosive increase of images. The popular image compression formats are based on different transforms which convert images from the spatial domain into compact frequency domain to remove the spatial correlation. In this paper, we focus on the exploration of data-driven transform, Karhunen-Loéve transform (KLT), the kernels of which are derived from specific images via Principal Component Analysis (PCA), and design a high efficient KLT based image compression algorithm with variable transform sizes. To explore the optimal compression performance, the multiple transform sizes and categories are utilized and determined adaptively according to their rate-distortion (RD) costs. Moreover, comprehensive analyses on the transform coefficients are provided and a band-adaptive quantization scheme is proposed based on the coefficient RD performance. Extensive experiments are performed on several class-specific images as well as general images, and the proposed method achieves significant coding gain over the popular image compression standards including JPEG, JPEG 2000, and the state-of-the-art dictionary learning based methods.

Statistical dependencies among different color components can be exploited to improve the compression efficiency of video coding. In the Range Extensions (RExt) of High Efficiency Video Coding (HEVC) standard, a cross-component prediction tool is adopted. In each transform unit, it explicitly signals a scaling parameter for residual prediction for the second and third color components respectively from the first color component. This paper presents a guided cross-component residual prediction method for the inter prediction coding. Instead of explicit signaling, the residual prediction parameter is derived by analyzing the statistical correlations among the corresponding color components in the motion compensation prediction signal. Compares with the cross-component prediction tool in the HEVC RExt, the proposed method can have 0.6% and 0.9% averaged bit rate savings in coding RGB video in random access and low delay B configurations, respectively. Compares with cross-component prediction tool off, the bit rate savings are about 14% in both configurations.

• A large-scale JND-based coded video quality dataset is presented. • The VideoSet contains 220 5-s sequences in four resolutions coded by H.264/AVC. • The subjective test procedure, JND data cleaning and properties are described. • The significance and implications of the VideoSet are discussed. • This work points out a clear path to data-driven perceptual coding. A new methodology to measure coded image/video quality using the just-noticeable-difference (JND) idea was proposed in Lin et al. (2015). Several small JND-based image/video quality datasets were released by the Media Communications Lab at the University of Southern California in Jin et al. (2016) and Wang et al. (2016) [3]. In this work, we present an effort to build a large-scale JND-based coded video quality dataset. The dataset consists of 220 5-s sequences in four resolutions (i.e., 1920 × 1080 , 1280 × 720 , 960 × 540 and 640 × 360 ). For each of the 880 video clips, we encode it using the H.264/AVC codec with QP = 1 , … , 51 and measure the first three JND points with 30 + subjects. The dataset is called the “VideoSet”, which is an acronym for “Video Subject Evaluation Test (SET)”. This work describes the subjective test procedure, detection and removal of outlying measured data, and the properties of collected JND data. Finally, the significance and implications of the VideoSet to future video coding research and standardization efforts are pointed out. All source/coded video clips as well as measured JND data included in the VideoSet are available to the public in the IEEE DataPort (Wang et al., 2016 [4]).

The emerging standards for screen content coding (SCC) and 3D video coding known as High Efficiency Video Coding (HEVC)-SCC and 3D-HEVC, respectively, have been developed as extensions of HEVC. Compared with camera-captured video, screen content video and depth maps usually contain significantly fewer colors or sample intensities and much sharper edges in a frame. In this paper, a new coding method named segmental prediction is proposed to code screen content and depth maps more efficiently. First, samples in a prediction block are classified into several segments according to the sample intensities. Second, a particular predictor is calculated for each segment. Then an offset for a segment is derived at the encoder side and signaled to the decoder side. Finally, a single sample value is assigned to each segment in the reconstructed block. The experimental results show that the proposed approach can achieve a 2.2% average Bjontegaard Distortion-rate (BD-rate) saving under the HEVC-SCC common test conditions (CTCs) for sequences of the category YUV, text and graphics with motion, 1080p, with the all intra configuration. It can also achieve a 0.6% average BD-rate saving for synthesized views under the 3D-HEVC CTCs with the random access configuration.

Palette mode is the new coding tool that has been adopted in the Screen Content Coding Extensions of High Efficiency Video Coding (HEVC SCC). Palette mode can represent colour clusters for screen content efficiently and can be summarized into two parts: palette coding tools and colour index map coding tools. This paper proposes two techniques to improve colour index map coding: transition copy and prediction across coding unit boundary. The former is for exploring the correlation between an index and all of the previously coded indices; the latter for utilizing the correlation between the content along the coding unit boundary. Experimental results reportedly show that, compared with HEVC SCC, the proposed techniques can achieve 3.2%, 2.2%, and 1.7% BD-rate savings compared with HEVC SCC for “YUV, text & graphics with motion, 1080p & 720p sequences” under all intra, random access, and low-delay B common test conditions, respectively.

In screen content video, the spatial correlation among pixels shows different characteristics as compared to natural content video. Intra-picture motion compensation (or intra block copy in HEVC) plays a key role in reducing the bit rate of representing high resolution video with contents such as text and graphics. The efficiency of intra block copy is highly related to the accuracy of block vector prediction. In this paper, several block vector prediction methods are proposed to improve the performance of intra block copy technology in HEVC screen content coding extension. Simulation results show that an average bit rate reduction of 8.0% can be achieved for typical 1080p text and graphics sequences in all intra configuration, when compared to the standard's test model SCM-1.0. As a result, some of the proposed methods have been adopted into the standard working draft and reference software.

Palette mode, as a new coding tool, is adopted in the Screen Content Coding Extensions of High Efficiency Video Coding (HEVC SCC) that is being defined by the Joint Collaborative Team on Video Coding (JCT-VC). In the palette mode, pixels in a coding unit (CU) are represented by selected representative colours according to the characteristics of screen contents in which pixel values usually concentrate on few colour values. This paper introduces the palette mode in HEVC SCC, and our contributions to the palette mode are also presented. Experimental results show that disabling the palette mode suffers 20.3%, 13.0%, and 7.5% BD-rate increases for “YUV, text & graphics with motion, 1080p & 720p sequences” under all intra (AI), random access (RA), and low-delay B (LB) common test conditions, respectively.

As the state-of-the-art video coding standard, High Efficiency Video Coding (HEVC) has been finalized in January 2013. To support the coding of multiple views and associated depth data, the development of the HEVC extension for 3D video coding is also about to be finalized. New coding unit (CU) level coding tools are added to the HEVC design to improve the compression capabilities for both video views and depth data. In this paper, we introduce a new coding mode termed as “Single Depth Intra Coding Mode” to efficiently code the smooth area within a depth map. The concept of single depth mode is to simply reconstruct a coding block with a single depth value based on block merging scheme under the HEVC quad-tree based block partitioning. The proposed single depth mode has been adopted into the working draft and 3D-HEVC test model 12 (HTM-12.0). The experimental results evaluated under the common test conditions (CTC) for 3D-HEVC show that the proposed technique achieves up to 5% bit-rate saving for depth coding and on average 0.3% bit-rate saving regarding the synthesis results over the texture and depth bit-rate with decoding time reduced by 4%.

The 3D extension of High Efficiency Video Coding (3D-HEVC) is a new international video coding standard that has been developed by the Joint Collaborative Team on 3D Video Coding Extensions (JCT-3V). It aims at improving the coding efficiency of 3D and multi-view videos by introducing new coding tools to utilize the correlations between views and between texture and depth components. In this paper, we propose an inter-view motion prediction (inter-view merge candidate) and an inter-component motion prediction (texture merge candidate) to explore the inter-view and the inter-component redundancies. The proposed schemes were adopted in 3D-HEVC and the experimental results demonstrate that the proposed inter-view merge candidate and texture merge candidate achieve significant BD-rate reductions of 20% for dependent texture views and 9% for the synthesized texture views under the common test conditions used for 3D-HEVC standardization.

In the screen content coding extension for HEVC, intra block copy (IBC) was adopted to explore the redundancy between reduplicated patterns, which often occurs in a picture with screen contents. Besides reduplication, symmetry is also often observed in screen-content and even in natural pictures. In this paper, symmetric intra block copy (SIBC) is therefore proposed to further improve IBC. By flipping the reference block of IBC horizontally or vertically, SIBC may provide better predictions than IBC for symmetric patterns. Experimental results show that SIBC can achieve 1.2% and 1.9% coding gains in all intra configurations on sequences in the category of YUV, text & graphics with motion, 1080p for lossy and lossless coding respectively.

High Efficiency Video Coding (HEVC) adopts hybrid transform coding scheme to improve the transform performance. However, it does not consider the influence of the prediction unit boundary. This paper proposes an adaptive boundary dependent transform optimization scheme for HEVC. Based on the transform unit boundary type identification by checking whether it lies at a prediction unit boundary or not, some additional transform kernels, including DCT-IV, are incorporated for transform. Moreover, the additional adopted transform kernels are generated by re-using the existing kernels in HEVC. Therefore, the butterfly structure for transform implementation can be preserved to facilitate parallel computing, and computational complexity increasing can be avoided. Experimental results show that the coding performance improvements can be up to 0.96% and 0.64% for low delay P and random access testing configurations respectively.

The target of 3D video coding is to compress Multiview Video plus Depth (MVD) format data, which consist of a texture image and its corresponding depth map. In the MVD format, the depth map plays an important role for successful services in 3D video applications, because it enables the user to experience 3D by generating arbitrary intermediate views. The depth map has a strong correlation with its associated texture data, so it can be utilized to improve texture coding efficiency. This paper introduces a novel and efficient depth-based texture coding scheme. It includes depth-based motion vector prediction, block-based view synthesis prediction, and adaptive luminance compensation, which were adopted in an AVC-compatible 3D video coding standard. Simulation results demonstrate that the proposed scheme reduces the total coding bitrates of texture and depth by 19.06% for the coded PSNR and 17.01% for the synthesized PSNR in a P-I-P view prediction structure, respectively.

To achieve high coding performance for screen content, the intra block copy (IntraBC) performs block matching within a limited area of the reconstructed samples inside the current picture. We further extend its notion from the unit of square coding unit (CU) to non-square prediction unit (PU) partitions. This design is then justified by theoretical and empirical analyses which reveal the same fact that blocks coded by IntraBC mode tend to enable more at smaller partition levels. Besides, the syntax design of the proposed method is fully aligned with that of inter partition modes. Therefore the architecture-wise change in video codec design can be minimized. The experimental results justify the effectiveness of the proposed mode for coding of screen content video. In particular, up to 19.5% rate reduction (with an average of 12.7%) relative to the HM-12.0+RExt-4.1 anchor can be achieved on top of the usage of CU-based IntraBC prediction.

This paper presents a novel inter-view motion prediction technique used in 3D-HEVC which provides efficient compression for motion vectors. The 3D extension of HEVC (High Efficiency Video Coding) namely 3D-HEVC is under development in JCT-3V for coding multi-view video and depth data. Inter-view motion prediction takes benefit of the inter-view correlation between views by inferring the motion information of a view from the already coded motion information in another view as well as the local disparity between two views. Experimental results show that inter-view motion prediction in 3D-HEVC provides an average bit-rate savings of 15% for the dependent views.

This paper introduces a novel inter-layer adaptive filtering method for scalable extension of High Efficiency Video Coding (HEVC) standard, which is being developed by the Joint Collaborative Team on Video Coding (JCT-VC). The scalable extension of HEVC is currently utilizing an interlayer texture prediction method, in which the prediction of texture at enhancement layer is derived from the collocated reconstructions of base layer. The proposed method applies an adaptive filter to the reconstructions of base layer and generates the predictor of enhancement layer. It aims at enhancing the coding efficiency of enhancement layer by further reducing the redundant texture information in interlayer texture prediction. Two techniques are presented in this paper. The first one applies the adaptive filter to the reconstructions of base layer, which is followed by the fixed upsampling in spatial scalability. In the second one, an adaptive upsampling filter is further applied to filtered base layer reconstructions to generate the pixels at interpolated positions. Compared with the Scalable Mode under Consideration (SMuC) version 0.1.1 of HEVC scalable extension, average 0.8% and 3.2% bit-rate reductions are achieved for spatial scalability and SNR scalability respectively.

This paper introduces a novel inter-layer intra mode prediction method for scalable extension of High Efficiency Video Coding (HEVC) standard, which is being developed by the Joint Collaborative Team on Video Coding (JCT-VC). In HEVC and its scalable extension, thirty-five intra prediction modes are adopted to reduce the spatial redundancy of luma texture within one frame, which may introduce noticeable overhead of delivering the intra mode information in the scalable bit-stream. In this paper, the correlation of intra modes between different layers is exploited to improve the efficiency of intra mode coding in enhancement layer. The intra modes in enhancement layer are mainly predicted with the ones of collocated blocks at base layer. Two techniques are presented in this paper with some differences in terms of the derivation of intra mode predictor from base layer and the selection of three most probable modes at enhancement layer. Compared to Test Model version 1.0 of HEVC scalable extension, 0.4% and 0.1% bit-rate reductions can be achieved with technique 2 in All Intra 2x spatial scalability and All Intra 1.5x spatial scalability respectively, while technique 1 can achieve 0.2% and 0.1% reductions. There is no obvious running time increase for both encoder and decoder.

In the context of scalable video coding, we present in this paper a general prediction mechanism to combine temporal and inter-layer reference signals for compressing the enhancement-layer (EL). By utilizing the proposed combinations, the EL temporal prediction residue can be further predicted from a referenced base-layer (BL) temporal residue; or, the inter-layer prediction residue between EL and BL can itself be temporal predicted, leading to improved coding efficiency. Furthermore, we investigate the complexity associated with the proposed generalized combined prediction method, and present techniques to significantly reduce its worst-case complexity while still achieving substantial coding gains. Simulation results on HEVC Scalable Extension reported average of 1.6%, 2.7%, 4.2% EL+BL BD-rate gains for random access, low-delay B and low-delay P configurations respectively, with only about 2% decoding runtime increase. The worst-case complexity of the proposed combined prediction, in terms of computation and memory bandwidth, is only about 120% and 75% of the worst-case complexity of HEVC temporal prediction.

In scalable video coding, reconstructed base-layer samples can be used as predictor for compressing enhancement-layer samples. This inter-layer predictor typically exhibits specific type of artifacts, arising from coarse quantization and upsampling processes. To suppress these artifacts, we propose an inter-layer filtering scheme which adapts to local orientations. On a sample-by-sample basis, it first determines the orientation class by comparing simple gradient measurements. According to the determined orientation, a corresponding directional filter constructed with predetermined coefficients, is selected and then applied. Our filtering scheme achieves low encoding latency and also avoids complicated normalization processes typically associated with other low-latency filtering methods. Based on complexity analysis and simulation results, we demonstrate that, compared with other inter-later filtering approaches, our method provides a viable complexity-efficiency trade-off for practical scalable video applications.

This paper proposes a progressive motion vector resolution (PMVR) method for High Efficiency Video Coding (HEVC). In the proposed scheme, high motion vector (MV) resolutions, e.g. 1/4 or 1/8 pixel resolution, are employed for MVs near to the motion vector predictor (MVP) and low MV resolutions are employed for MVs far from the MVP. The range of each MV resolution is indicated by a threshold parameter. And a new motion vector difference (MVD) derivation method is designed to encode MVD efficiently. Experimental results show that PMVR with 1/8 pixel motion search can achieve a BD-rate gain up to 16% with almost the same coding time with HM8.0, and for PMVR without 1/8 pixel motion search, up to 6.1% BD-rate gain can be achieved with 9% encoding time saving on average.

For showing 2D video contents on 3DTVs, 2D-to-3D conversion is required to convert 2D contents to 3D ones. In general, the conversion process consists of depth map generation, which estimates the 3D geometry of the scene, and rendering, which produces output stereo images. We propose a depth map refinement algorithm which uses adaptive decimation and guided interpolation to refine the depth map. The proposed approach eliminates unnecessary textures and keeps object boundaries on the depth map. In the refined depth map, the transition of depth values within an object is smooth, the object boundaries on the depth map are aligned to those on the input image, and the foreground can be clearly separated from the background. These characteristics on the depth map help to achieve better 3D visual perception. Compared to the state-of-the-art approaches, the blurriness and object boundary alignment are easier to be adjusted by the proposed two-stage operations, and the computational complexity of the proposed algorithm is much lower.

This paper presents a mechanism of using rectangular prediction blocks for Intra prediction in the emerging High Efficiency Video Coding (HEVC) standard. In the previous and current video coding standards, Intra predictions have been processed on square blocks, e.g. 16×16, 8×8 and 4×4 pixel blocks H.264/MPEG4 AVC, and 2N×2N or N×N square prediction units (PU) in the current HEVC. In this paper, it is proposed to include 2N×N and N×2N prediction block sizes (or PU types) in a 2N×2N coding unit (CU) for the Intra prediction, as in the Inter prediction. Experimental results show that, together with other tools (e.g. NSQT) significant coding gain, i.e. average 1.8% BD-rate reduction with the use of 2-point prediction and transforms, or 1.4% BD-rate reduction without the use of 2-point prediction and transforms (for “All Intra” configuration) can be achieved, compared with HM5.0 anchor.

New High Efficiency Video Coding (HEVC) standard is designed to provide substantial coding efficiency improvement compared to H.264/AVC. Latest subjective testing shows 50% improvement has been achieved. Many new technologies contribute to the overall improvement. Intra prediction with 35 modes is one of the key improvements. Associated with that, there is a new intra mode coding method to efficiently signal the selected modes. This paper presents this new intra mode coding method that has been adopted by HEVC. In this method, the 35 intra modes are divided into two categories. One category includes 3 most probable modes (MPMs) and another category includes 32 remaining modes. In doing so, shorter codeword is used for coding MPMs and fixed length coding is used to code the remaining modes. Experimental results show the 3MPMs based method improve the coding efficiency compared to the prior art method used in H.264/AVC.

In High Efficiency Video Coding (HEVC), advanced motion vector prediction (AMVP) is adopted to predict current motion vector by utilizing a competition-based scheme from a given candidate set, which include both the spatial and temporal motion vectors. In order to enhance the practicability of the AMVP, a simplified AMVP is proposed. Firstly, by analyzing the importance of the spatial and temporal candidates, we reduce the number of the candidates involved in the competition set and simplify the redundancy checking process, which will decrease the complexity of the decoder as well as improve the robustness of the decoder. Secondly, we simplify the zero motion adding process which will occur only when the number of existing candidates is less than the predefined number. Experimental results show that the proposed scheme provides no loss in random access and low delay conditions. These two simplifications have been proposed and adopted into the HEVC standard.

This paper provides a technical overview of a newly added in-loop filtering technique, sample adaptive offset (SAO), in High Efficiency Video Coding (HEVC). The key idea of SAO is to reduce sample distortion by first classifying reconstructed samples into different categories, obtaining an offset for each category, and then adding the offset to each sample of the category. The offset of each category is properly calculated at the encoder and explicitly signaled to the decoder for reducing sample distortion effectively, while the classification of each sample is performed at both the encoder and the decoder for saving side information significantly. To achieve low latency of only one coding tree unit (CTU), a CTU-based syntax design is specified to adapt SAO parameters for each CTU. A CTU-based optimization algorithm can be used to derive SAO parameters of each CTU, and the SAO parameters of the CTU are inter leaved into the slice data. It is reported that SAO achieves on average 3.5% BD-rate reduction and up to 23.5% BD-rate reduction with less than 1% encoding time increase and about 2.5% decoding time increase under common test conditions of HEVC reference software version 8.0.

Recently, an efficient coding tool named adaptive interpolation filtering (AIF) has been proposed to hybrid video coding scheme. By introducing Wiener filter into the fractional-pixel interpolation procedure, AIF can reduce the inter-prediction error and improve coding efficiency significantly. However, the training-based Wiener filter mechanism brings AIF an inherent multi-pass encoding structure, which imposes big burdens on the encoder in terms of huge computational complexity and memory access. In this paper, we propose a single-pass-based localized adaptive interpolation filtering (SPL-AIF) algorithm for video coding, which can reduce the complexity of AIF dramatically without sacrifice of its outstanding coding performance. The proposed SPL-AIF algorithm is based on the observation that there is a high correlation among optimal interpolation filters of consecutive frames, and different regions in a frame often possess different statistical characteristics. Accordingly, the proposed algorithm can be designed including two major parts. First, a competitive filter set which includes the optimal interpolation filters of several previous frames as well as the fixed H.264/AVC interpolation filters is built up for the coding of the current frame. Then a rate-distortion optimization criterion is used to select the best one at macroblock (MB) level. In order to reduce overhead, a predictive coding method is used to compress the filter signaling flag for each MB. Experimental results show that, by using the proposed algorithm, the encoding complexity can be reduced significantly while the average coding gain in Bjöntegaard distortion bit-rate reduction can be improved about 1% compared with the multi-pass AIF. The proposed method has been adopted into the Video Coding Expert Group Key Technology Area software.

A new video coding tool, sample adaptive offset (SAO), is introduced in this paper. SAO has been adopted into the Working Draft of the new video coding standard, High-Efficiency Video Coding (HEVC). The SAO is located after deblocking in the video coding loop. The concept of SAO is to classify reconstructed pixels into different categories and then reduce the distortion by simply adding an offset for each category of pixels. The pixel intensity and edge properties are used for pixel classification. To further improve the coding efficiency, a picture can be divided into regions for localization of offset parameters. Simulation results show that SAO can achieve on average 2% bit rate reduction and up to 6% bit rate reduction. The run time increases for encoders and decoders are only 2%.

High Efficiency Video Coding (HEVC) is a new international video coding standard that has been developed by the Joint Collaborative Team on Video Coding (JCT-VC). In this paper, an overview of the motion vector coding techniques for HEVC is presented. Our three proposed coding tools for the motion vector predictor (MVP) in the Inter, Skip and Merge modes of HEVC are also presented, which includes a new location of the temporal MVP, a priority-based derivation method of spatial MVPs, and a derivation method of temporal MVPs. A combination of these three tools can achieve on average 1.3%, 1.8%, 1.2% and 2.2% bit rate reductions for high efficiency random access, low complexity random access, high efficiency low delay, and low complexity low delay, respectively.

Hand tracking is a fundamental task in a gesture recognition system. Most previous works tracked the hand position on color images and relied heavily on skin color information. However, color information is very vulnerable to lighting variations and skin color varies across difference human races. Furthermore, one can not effectively discriminate faces or other skin-color-like objects from hands when using skin color detection. In this paper, we propose a hand tracking algorithm that uses depth images only, and also a hand click detection method to initialize the hand tracking automatically. We show that depth images suffice and are advantageous to real-time hand tracking. A region growing technique is applied to segment the hand region on depth images. Then a mean-shift based algorithm accurately locates the hand center in the segmented hand region. The experimental results show that the proposed tracking algorithm runs at 300+ FPS, and the average error of the tracked 3D hand positions is less than 1 centimeter. The proposed method enables a plethora of potential applications to natural Human-Computer Interaction (HCI), and is adequate for embedded systems of consumer electronics because of its low complexity and low bandwidth requirement.

In this paper, a one-pass encoding algorithm is proposed for adaptive loop filter (ALF) in high-efficiency video coding (HEVC). ALF can improve both subjective and objective video quality, but it also requires a lot of encoding passes (i.e. picture buffer accesses) that will significantly increase external memory access, encoding latency, and power consumption. Therefore, we propose a method to estimate filtering distortion without performing real filter operation. The number of encoding passes can be effectively reduced from 16 to 1. Combined with an initial guess of filter-on/off blocks by using time-delayed filters, the proposed one-pass algorithm only induces average 0.17% BD-rate increase.

An adaptive interpolation filtering (AIF) algorithm has been proposed to improve the conventional hybrid video coding scheme recently. Although such an algorithm does improve coding efficiency significantly, its encoding complexity, in terms of computational complexity and memory access, increases dramatically due to its inherent two-pass encoding structure. In this paper, we propose a novel single-pass based algorithm which can reduce the complexity of AIF significantly. In our method, a competitive filter set which includes optimal filters trained from several previous frames and the fixed H.264 filters is considered for the current coding frame. A rate-distortion optimization (RDO) criterion is then used to select the best one at macro-block (MB) level. In order to reduce overhead, a predictive coding method is used to compress the filter type for each MB. Experimental results show that, by using the proposed algorithm, the encoding complexity can be significantly reduced without sacrifice of coding gain.

Intra-coding plays an important role in video coding schemes and it has become a hot research topic of the next generation video coding standard. In this paper, we propose two techniques to improve the efficiency of the prediction and the transform, respectively, for intra-coding. First, we introduce an overlapped block intra-prediction (OBIP) method, which makes use of the intra directional prediction modes of neighboring blocks as well as the mode of the current block. A position dependent weighted sum of several possible predictors will be treated as the final prediction. Second, we propose a multiple-model Karhunen-Loève transform (MMKLT) technique to further improve the mode dependent directional transform (MDDT) method. More than one KLT will be trained offline for each intra-prediction mode based on residual variances. Experimental results indicate that the proposed two methods together can achieve an average bit-rate reduction about 11% compared with H.264 or 5% compared with MDDT in all-intra coding.

Adaptive interpolation filtering (AIF) algorithms have been proposed to enhance the hybrid video coding scheme recently. Although these algorithms can improve coding efficiency significantly, their encoders suffer huge increase in complexity in terms of latency and memory access due to its inherent multi-pass encoding procedure. In this paper, we present a novel single-pass solution for these algorithms, which allows optimal selection among different interpolation filters. In this solution, time-delayed interpolation filters are used to achieve single-pass encoding, and localized ratedistortion (RD) selection is used to compensate the possible coding loss from time-delayed filters. Experimental results show that the proposed method is efficient for All AIF techniques in current ITU-T/SG16 reference software. By using the proposed method, single-pass encoding with multiple AIF filters can be achieved while maintaining similar coding efficiency as multi-pass AIF.

A texture representation should corroborate various functions of a texture. In this paper, we present a novel approach that incorporates texture features for retrieval in an examplar-based texture compaction and synthesis algorithm. The original texture is compacted and compressed in the encoder to obtain a thumbnail texture, which the decoder then synthesizes to obtain a perceptually high quality texture. We propose using a probabilistic framework based on the generalized EM algorithm to analyze the solutions of the approach. Our experiment results show that a high quality synthesized texture can be generated in the decoder from a compressed thumbnail texture. The number of bits in the compressed thumbnail is 400 times lower than that in the original texture and 50 times lower than that needed to compress the original texture using JPEG2000. We also show that, in terms of retrieval and synthesization, our compressed and compacted textures perform better than compressed cropped textures and compressed compacted textures derived by the patchwork algorithm.

A technique for film grain noise extraction, modeling and synthesis is studied and applied to high-definition video coding in this paper. Film grain noise enhances the natural appearance of pictures in high-definition video and should be preserved in coded video. However, the coding of video contents with film grain noise is expensive. In previous art, it was proposed to enhance the coding performance by extracting film grain noise from the input video at the encoder as a preprocessing step, and by resynthesizing and adding it back to the decoded video at the decoder as a postprocessing step. In a novel implementation of this approach, we first remove film grain noise from image/video with a variational denoising approach without distorting its original content. Then, we present a parametric model (consisting of a small set of parameters) to generate film grain noise that is close to the actual one in terms of a couple of observed statistical properties, such as the power spectral density and the crosschannel spectral correlation. Under this framework, the coding gain of denoised video is higher while the visual quality of the final reconstructed video is well preserved. Experimental results are provided to demonstrate the superior performance of the proposed approach.

A new technique for film grain noise extraction, modeling and synthesis is proposed and applied to the coding of high definition video in this work. The film grain noise is viewed as a part of artistic presentation by people in the movie industry. On one hand, since the film grain noise can boost the natural appearance of pictures in high definition video, it should be preserved in high-fidelity video processing systems. On the other hand, video coding with film grain noise is expensive. It is desirable to extract film grain noise from the input video as a pre-processing step at the encoder and re-synthesize the film grain noise and add it back to the decoded video as a post-processing step at the decoder. Under this framework, the coding gain of the denoised video is higher while the quality of the final reconstructed video can still be well preserved. Following this idea, we present a method to remove film grain noise from image/video without distorting its original content. Besides, we describe a parametric model containing a small set of parameters to represent the extracted film grain noise. The proposed model generates the film grain noise that is close to the real one in terms of power spectral density and cross-channel spectral correlation. Experimental results are shown to demonstrate the efficiency of the proposed scheme.

An automatic method is presented to enhance the visibility of dark areas of an image while preserving its natural look. This method consists of three major parts: classification, global adjustment and local adjustment. Firstly, an image is classified into one of several types for different global/local processing or no processing. The method adaptively maps a luminance value into a value based on a piecewise linear mapping curve in order to increase contrast in dark areas. The global and local mapping curves are based on histogram equalization with modifications. The local enhancement is non-overlapped block-based and is applied only when necessary. To avoid blocking artifacts generated by block-based local adjustment, the luminance value of each pixel is adjusted according to an interpolated mapping curve derived from the block mapping curves of nearby blocks. The combination of global and local enhancements achieves good overall enhancement with local adaptability without excessive complexity

Predictive motion estimation techniques have become widely used in video coding systems due to their reasonable coding performance and low computational requirement compared to the brute force full search method. However, most existing predictive motion estimation algorithms focus mainly on local motions and consider motion predictions from only neighboring image blocks (either spatially or temporally). In this paper, a reliable global motion predictor or GMP is introduced to improve predictive motion estimation for sequences with fast camera motions. A simple global motion model is applied for global motion estimation. The derived global motion vector for each macroblock is then used as the GMP during the predictive motion estimation for all possible block modes or types, together with up to 8 local motion predictors for P frame motion estimation and up to 12 local motion predictors for B frame motion estimation. The comparison between the coding performance with and without GMP shows very significant differences; roughly 1-2 dB’s improvement is achieved by GMP for sequences with very fast camera motions. The computational overhead for GMP is very small, less than 2% of the total computation cycles in our H.264 encoder.

Multimedia data security is very important for multimedia commerce on the Internet such as video-on-demand and real-time video multicast. Traditional cryptographic algorithms/systems for data security are often not fast enough to process the vast amount of data generated by multimedia applications to meet real-time constraints. This paper presents a joint encryption and compression framework in which video data are scrambled efficiently in the frequency domain by employing selective bit scrambling, block shuffling and block rotation of the transform coefficients and motion vectors. The new approach is very simple to implement, yet provides considerable levels of security and different levels of transparency, and has a very limited adverse impact on compression efficiency and no adverse impact on error resiliency. Furthermore, it allows transcodability/scalability, and other content processing functionalities without having to access the cryptographic key and perform decryption and re-encryption.

It has been widely observed that improved motion field representation (e.g. fractional-pel and OBMC) is beneficial to video coding. In this paper, we investigate efficient motion field representation in the wavelet domain. We aim to demonstrate that motion estimation in the wavelet domain can: 1) effectively solve the occlusion problem; 2) transform the aperture from a problem into a property that can be exploited in video coding. Our study leads to a novel anisotropic motion field representation in the wavelet domain. Our new wavelet-based motion field offers more flexibility than traditional representations in the spatial domain and opens the door to an improved understanding of relationship between motion and intensity uncertainty model.

This paper presents a novel framework of motion compensated prediction (MCP) techniques in the wavelet domain for high performance video coding. Our analysis reveals fundamental limitations with previous ad-hoc wavelet-based video coders from the motion accuracy point of view. We demonstrate that the phase associated with any wavelet transform carries critical information of the motion accuracy and we propose to restore the motion accuracy by considering the wavelet coefficients of the previous frame with all different phases. Our all-phase MCP approach can be viewed as predicting the wavelet coefficients from an over-complete expansion of the previous frame. Experimental results have shown that restoration of motion accuracy in the wavelet domain can dramatically improve the efficiency of MCP. The video coder (MCP-WT) built upon the MCP of wavelet coefficients has achieved 2 to 3 dB gain over existing the MPEG-2 coder at the bit rate of 1 to 9 Mbps. Moreover, MCP techniques in the wavelet domain offer a promising new ground for developing efficient scalable video coders.

This paper studies the problem of lossless compression of computer generated compound images that contain not only photographic images but also text and graphic images. We present a simple backward adaptive classification scheme to separate the image source into three classes: smooth regions, text regions and image regions. Different probability models are assigned within each class to maximize the compression performance. We also extend our scheme to exploit the interplane dependency for coding color images. The segmentation results of the reference color plane are used as the contexts for the classification and coding of the current color plane. Our new lossless coder significantly outperforms current state-of-the-art coders such as CALIC and JPEG-LS for compound images with modest computational complexity.

This paper presents a novel block-based segmentation and adaptive coding (BSAC) algorithm for visually lossless compression of scanned documents that contain not only photographic images but also text and graphic images. For such a compound image source, we structure the image into nonoverlapping blocks and classify each block into four different classes based on the empirical statistics within the block. Different coding strategies are applied to different classes in order to achieve the very best compression performance. Our new block-based image coder is able to provide visually lossless compression of scanned documents at the bit rate of around 1/spl sim/1.5 bpp with modest computational complexity and very low memory requirement.

One common visual optimization strategy for image compression is to exploit the visual masking effect where artifacts are locally masked by the image acting as a background signal. In this paper, we present a point-wise extended visual masking approach that nonlinearly maps the wavelet coefficients to a perceptually uniform domain prior to quantization by taking advantages of both self-contrast masking and neighborhood masking effects, thus achieving very good visual quality. It is essentially a coefficient-wise adaptive quantization without any overhead. It allows bitstream scalability, as opposed to many previous works. The proposed scheme has been adopted into the working draft of JPEG-2000 Part II.

We review the various tools in JPEG 2000 that allow the users to take advantages of the various properties of the human visual system such as spatial frequency sensitivity and the visual masking effect. We show that the visual tool sets in JPEG 2000 are much richer than what was available in JPEG, where only locally invariant frequency weighting can be exploited.

This paper presents a fast and efficient way for color re-indexing that tends to maximize the compression performance of a palette-based compression system. The proposed scheme relates the index difference of neighboring pixels to the potential cost of bits. It optimizes the assignment of index values to colors in a one-step look-ahead greedy fashion. Experimental results suggest that the proposed re-indexing scheme can reduce the bit rate by up to 43%, when compared to a previously proposed intensity-based color-indexing scheme. Furthermore, we show that with the proposed color re-indexing scheme, the palette-based JPEG-LS and palette-based JPEG-2000 can often outperform the graphical interchange format (GIF) significantly.

In typical block-based video coding, the rate-control scheme allocates a target number of bits to each frame of a video sequence and selects the block quantization parameters to meet the frame targets. In this work, we present a new technique for assigning such targets. This method has been adopted in the test model TMN10 of H.263+, but it is applicable to any video coder and is particularly useful for those that use B frames. Our approach selects the frame targets using formulas that result from combining an analytical rate-distortion optimization and a heuristic technique that compensates for the distortion dependency among frames. The method does not require pre-analyses, and encodes each frame only once; hence, it is geared toward low-complexity real-time video coding. We compare this new frame-layer bit allocation in TMN10 to that in MPEG-2's TM5 for a variety of bit rates and video sequences.

Multimedia data security is very important for multimedia commerce on the Internet such as video-on-demand and real-time video multicast. Traditional cryptographic algorithms for data security are often not fast enough to process the vast amount of data generated by the multimedia applications to meet the real-time constraints. This paper presents a joint encryption and compression framework in which video data are scrambled efficiently in the frequency domain by employing selective bit scrambling, block shuffling and block rotation of the transform coefficients and motion vectors. The new approach is very simple to implement, yet provides considerable level of security, has minimum adverse impact on the compression efficiency, and allows transparency, transcodability, and other content processing functionalities without accessing the cryptographic key.

An important motivation for the development of the emerging H.263+ and MPEG-4 coding standards is to enhance the quality of highly compressed video for two-way, real-time communications. In these applications, the delay produced by bits accumulated in the encoder buffer must be very small, typically below 100 ms, and the rate control strategy is responsible for encoding the video with high quality and maintaining a low buffer delay. In this work, we present a simple rate control technique that achieves these two objectives by smartly selecting the values of the quantization parameters in typical discrete cosine transform video coders. To do this, we derive models for bit rate and distortion in this type of coders, in terms of the quantization parameters. Using Lagrange optimization, we minimize distortion subject to the target bit constraint, and obtain formulas that indicate how to choose the quantization parameters. We implement our technique in H.263 and MPEG-4 coders, and compare its performance to TMN7 and VM7 rate control when the encoder buffer is small, for a variety of video sequences and bit rates. This new method has been adopted as a rate control tool in the test model TMN8 of H.263+ and (with some modifications) in the verification model VM8 of MPEG-4.

It is well known that the fixed rate coder achieves optimality when all coefficients are coded with the same rate-distortion (R-D) slope. In this paper, we show that the performance of the embedded coder can be optimized in a rate-distortion sense by coding the coefficients with decreasing R-D slope. We denote such a coding strategy as rate-distortion optimized embedding (RDE). RDE allocates the available coding bits first to the coefficient with the steepest R-D slope, i.e., the largest distortion decrease per coding bit. The resultant coding bitstream can be truncated at any point and still maintain an optimal R-D performance. To avoid the overhead of coding order transmission, we use the expected R-D slope, which can be calculated from the coded bits and is available in both the encoder and the decoder. With the probability estimation table of the QM-coder, the calculation of the R-D slope can be just a lookup table operation. Experimental results show that the rate-distortion optimization significantly improves the coding efficiency in a wide bit rate range.

An adaptive wavelet transform algorithm for scalable image coding is proposed. A quadtree segmentation scheme based on an entropy criterion is proposed to segment the image into several regions to allow for adaptive filtering using different types of prototype wavelet filters. A joint bit allocation and coding method is presented to efficiently code the segmented image in an embedding fashion. Experimental results show that the proposed scheme generally provides higher peak signal-to-noise ratios (PSNRs) at higher bit rates, and significantly better visual quality at lower bit rates, compared to fixed-filter wavelet transform based coding schemes.

Arithmetic coding is a powerful lossless data compression technique that has attracted much attention. It provides more flexibility and better efficiency than Huffman coding does. However, the multiplications needed in its encoding and decoding algorithms are very undesirable. Rissanen and Mohiuddin (1989) have proposed a simple scheme to avoid the multiplications. The present authors found that the performance of their proposed scheme might degrade significantly in some cases. They propose a multiplication-free multialphabet arithmetic code which can be shown to have minor performance degradation in all cases. In the proposed scheme, each multiplication is replaced by a single shift-and-add. The authors prove, by both theoretical analysis and simulation results, that the degradation of the proposed multiplication-free scheme is always several times (2-7 times in the present experiments) smaller than that of the Rissanen-Mohiuddin's scheme.

The error detection and correction capability of a linear block code is usually analyzed by its minimum distance and weight distribution function. A different analysis tool, the syndrome distribution of m-bit error patterns, is introduced in the paper. Additional detailed information about the capability of a code can be obtained from the syndrome distribution, e.g., undetected percentages, miscorrected percentages, as well as the weight distribution function. A fast algorithm is provided to compute the syndrome distribution vectors of m-bit error patterns for all m. The cyclic codes defined in asynchronous transfer mode (ATM) for broadband ISDN are used as examples to show the strength of this new analysis tool. A rough, but simple, estimation of the undetected percentages is also introduced in the paper.>

Arithmetic coding is a powerful lossless data compression technique that has attracted much attention in recent years. In this paper, we give a quantitative analysis of the performance degradation caused by the finite word-length registers used to compute the arithmetic codes. Such analysis is important for determining the appropriate word length in practical applications. Two algorithms of computing arithmetic codes are discussed. Our analysis results show that the degradation of one algorithm is reduced by half for every increased bit of word length while the degradation of the other algorithm is reduced to a quarter. Computer simulation results are also given to verify the theoretical analysis.>

Transport of MPEG2 video and associated audio and data over ATM (asynchronous transfer mode) networks has attracted much interest. A defined MPEG2 transport packet of 188 bytes can fit into 4 ATM cells with 32 bits left for forward error correction (FEC) codes and cell sequence numbers. The author discusses the design of FECs to correct the error patterns generated by different scrambler and line codes used by several ATM physical layers. By choosing generator polynomials carefully, a single-error-correcting Reed-Solomon code can correct not only the single-byte error patterns but also the double-byte error patterns processed by a scrambler. A method of distributing 16 bits of FEC into 4 ATM cells (thus, 4 bits of FEC each cell) to accommodate a cell sequence number in each cell is also given.>

Multi-point ISDN videoconferencing with video bridging in network-based servers represents a viable new network service. The paper presents a detailed technical analysis of a continuous presence video bridge using the H.261 video coding standard. The authors first compare the pros and cons of coded domain versus pel domain video bridges. The architecture and the required operations of a coded domain bridge using H.261 are then investigated. They derive the bounds of the bridge delay and the required buffer size for the implementation of the bridge. The delay and the buffer occupancy of the video bridge depend on the order, complexity, and the bit-distribution of the input video sources. To investigate a typical case, the authors simulate the delay and the buffer occupancy of a video bridge. They also provide a heuristic method to estimate the delay in a typical case. Several techniques are discussed to minimize the bridge delay and the buffer size. Finally, they simulate intra slice coding and show that the delay and the buffer size can be reduced significantly using this technique.>

Multi-point videoconferencing provides the full benefits of teleconference but also incurs more involved technical issues. This paper does a detailed analysis of a continuous presence video bridge using the H.261 video coding standard. We first describe the architecture and the required operations of a coded domain bridge using H.261. We then derive the bounds of the bridge delay and the required buffer size for the implementation of the bridge. The delay and the buffer occupancy of the video bridge depend on the order, complexity, and the bit- distribution of the input video sources. To investigate a typical case, we simulate the delay and the buffer occupancy of a video bridge. We also provide a heuristic method to estimate the delay in a typical case. Several techniques were discussed to minimize the bridge delay and the buffer size. Finally, we simulate an intra slice coding and show that the delay and the buffer size can be reduced significantly using this technique.

Arithmetic coding is a powerful lossless data compression technique that has attracted much attention in recent years. It provides more flexibility and better efficiency than the celebrated Huffman coding does. However, the multiplications needed in its encoding and decoding algorithms are very undesirable. Rissanen and Mohiuddin have proposed a simple scheme to avoid the multiplications. We found that the performance of their proposed scheme might degrade significantly in some cases. In this paper, we propose a multiplication-free multialphabet arithmetic code which can be shown to have only minor performance degradation in all cases. In our proposed scheme, each multiplication is replaced by a single shift-and-add. We prove, by both theoretical analysis and simulation results, that the degradation of the proposed multiplication-free scheme is always several times smaller than that of the Rissanen-Mohiuddin's scheme.

An HDTV coding scheme that is capable of achieving high quality below 20 Mb/s is presented. The system employs motion estimation/compensation followed by four-band subband splitting. The band signals are then subject to quantization and coded by an innovative high-order entropy coder. It is shown that this high-order entropy coder generates a bit rate nearly 30% lower than that of run-length/Huffman coding for motion-compensated frame differences. About 15% improvement is observed for the intrafield coded data. To reduce the complexity of the high-order entropy coder for high-speed implementation, two techniques, codebook reduction and nonuniform quantization of conditioning samples, are developed. A method for selecting conditioning samples and their order, based on mutual information between samples, is proposed and shown to outperform several other systematic and intuitive approaches. Simulations show that the new system, at about 20 Mb/s, achieves picture quality comparable to that of a 120-Mb/s six-band intrafield coding system previously investigated.>

The recently developed incremental-tree extension technique is used to design the conditional tree for high-order conditional entropy coding. In order to reduce its complexity, the authors introduce two techniques: code table reduction and nonlinear quantization of conditioning pixels. These two techniques substantially reduce the number of code tables, the number of conditioning pixel address-bits, and the number of states with only a minor penalty in performance. The authors also propose a suboptimal method to determine the pixel sequence of the conditioning states. Using these techniques, the high-order conditional entropy coding of images becomes practical and has shown significant improvement over the conventional runlength/variable-length coding scheme.>

High-order conditional entropy coding has not been practical due to its high complexity and lack of hardware to extract the conditioning state efficiently. The authors adopt the recently developed incremental-tree-extension technique to design the conditional tree for high-order conditional entropy coding. In order to make the high-speed conditional entropy coder feasible, they introduce several key innovations in the areas of complexity reduction and hardware architecture. For complexity reduction, they develop two techniques: code table reduction and nonlinear quantization of conditioning pixels. For hardware architecture, they propose a pattern-matching technique for fast conditioning state extraction and a multistage pipelined structure to handle the case of a large number of conditioning pixels. Using the complexity reduction techniques and the hardware structures. the authors demonstrate that it is possible to implement practical high-order conditional entropy codecs using current low-cost very large-scale integration (VLSI) technology.>

Run-length coding (RLC) and variable-length coding (VLC) are widely used techniques for lossless data compression. A high-speed entropy coding system using these two techniques is considered for digital high definition television (HDTV) applications. Traditionally, VLC decoding is implemented through a tree-searching algorithm as the input bits are received serially. For HDTV applications, it is very difficult to implement a real-time VLC decoder of this kind due to the very high data rate required. A parallel structured VLC decoder which decodes each codeword in one clock cycle regardless of its length is introduced. The required clock rate of the decoder is thus lower, and parallel processing architectures become easy to adopt in the entropy coding system. The parallel entropy coder and decoder are designed for implementation in two experimental prototype chips which are designed to encode and decode more than 52 million samples/s. Some related system issues, such as the synchronization of variable-length codewords and error concealment, are also discussed.>