Screen content has become one of the prominent mediums in the increasingly connected world. With the prevalence of remote collaboration and communication such as virtual conferences and online education, recent years have witnessed a dramatic increase in the data volume of the screen content. Screen content compression serves as the fundamental technology in fostering the storage, transmission, and exhibition of screen content. In this article, we target the super-resolution of the compressed screen content images, intending to tackle the real-world challenge problems. A dataset is proposed for the super-resolution of the screen contents contaminated with different compression distortion levels. Subsequently, we introduce the principle of the multi-hypothesis into the super resolution and propose a new paradigm for the restoration of the compressed screen content images. The luminance and sharpness similarity metric is adopted in the network learning to better adapt to the screen content characteristic and ensure perceptual fidelity. Experimental results verify the superiority and effectiveness of the proposed method.

Intra block copy (IBC) has been verified as an effective tool for screen content coding. By redefining the reference area for IBC, we show that the coding efficiency can be much improved without increase the on-chip memory requirement in VVC.

For the latest video coding standard Versatile Video Coding (VVC), the encoding complexity is much higher than previous video coding standards to achieve a better coding efficiency, especially for intra coding. The complexity becomes a major barrier of its deployment and use. Even with many fast encoding algorithms, it is still practically important to control the encoding complexity to a given level. Inspired by rate control algorithms, we propose a scheme to precisely control the intra encoding complexity of VVC. In the proposed scheme, a Time-PlanarCost (viz. Time-Cost, or T-C) model is utilized for CTU encoding time estimation. By combining a set of predefined parameters and the T-C model, CTU-level complexity can be roughly controlled. Then to achieve a precise picture-level complexity control, a framework is constructed including uneven complexity pre-allocation, preset selection and feedback. Experimental results show that, for the challenging intra coding scenario, the complexity error quickly converges to under 3.21%, while keeping a reasonable time saving and rate-distortion (RD) performance. This proves the efficiency of the proposed methods.

Versatile Video Coding (VVC) is capable of achieving approximately 25% bitrate reduction compared with High Efficiency Video Coding (HEVC) at the same objective quality under all intra configuration. Meanwhile, VVC sacrifices the encoding complexity by 26 times the encoding time of HEVC, which makes it impractical to use VVC without optimization. In view of the fact that most of complexity is due to the novel block partitioning structure in VVC, this paper focuses on predicting the partitioning structure with convolutional neural networks. Specifically, we first formulate the partitioning prediction problem into two alternatives: bottom-up-based where the split type of subblock boundaries is first predicted and then used to infer the partitioning structure of each coding unit, top-down-based where the probability distribution in the ensemble partitioning space is first derived and then used to decide the partitioning structure of each coding unit. Then, we address both formulations using convolutional neural networks. When evaluating on top of VTM7.0, proposed schemes perform favorably against state-of-the-art works. In particular, the bottom-up-based method can bring on average 52.3% encoding complexity reduction with 0.46% BD-rate increase while the top-down-based method could provide on average 39.5% encoding complexity reduction with 0.28% BD-rate increase. In addition, the proposed schemes can offer scalability in terms of coding efficiency and complexity trade-off.

Recently years have witnessed an increasing interest in the neural network-based video coding, including end2end and hybrid schemes. To foster the research in this emerging field and provide a benchmark, we propose the Grand Challenge on Neural Network-based Video Coding (GC-NNVC) in the ISCAS 2022. In this paper, we review the grand challenge results and share interesting observations resulted from the challenge. This challenge includes two tracks, the hybrid-based and end2endbased. Different neural network-based coding schemes are evaluated according to their coding efficiency and innovations in methodologies in each track. To facilitate a solid comparison with conventional video coding techniques, the decoded sequences are evaluated in YUV 4:2:0 color format and PSNR is adopted as the distortion metric. Compared with the reference software of HEVC, submissions in hybrid track typically exhibit promising BD-rate reduction while submissions from the end2end track perform worse.

Versatile Video Coding (VVC) achieves promising compression performances compared with High Efficiency Video Coding (HEVC) while sacrificing the encoding speed. This paper focuses on predicting the partitioning structures with convolutional neural networks to speedup the VVC encoder. Specifically, we formulate the partitioning prediction problem into two alternatives: implicit partitioning prediction based on the split type of subblock boundaries and explicit partitioning prediction from the ensemble partitioning space. Then, we address both formulations using convolutional neural networks.

In this paper, we concentrate on the super-resolution (SR) of compressed screen content video, in an effort to address the real-world challenges by considering the underlying characteristics of screen content. Firstly, we propose a new dataset for the SR of screen content video with different distortion levels. Meanwhile, we design an efficient SR structure that could capture the characteristics of compressed screen content video and manipulate the inner-connections in consecutive compressed low-resolution frames, facilitating the high-quality recovery of the high-resolution counter-part. Moreover, we design a new loss function for network training to better remedy the compression distortion and perceptual distortion. Experimental results demonstrate the effectiveness and superiority of the proposed method.

The quad-tree plus binary-tree (QTBT) partition structures was adopted into the versatile video coding (VVC) standard. In the QTBT partition structure, a dual tree structure can be applied on intra slices where the partitioning structures for luminance and chrominance components are separate. The dual tree structure is efficient for camera captured videos. However, it may be not the case for screen content videos. Therefore, adaptive dual tree structure is proposed in this paper wherein the coding structure of each coding tree unit is switched between separate and joint coding structure to adapt the textures adaptively. The proposed scheme is implemented into the reference software of the VVC standard-VTM. Simulation results report that up to 4.5% BD bitrate saving can achieve on the typical screen content videos when compared to VTM5.

The emerging visual quality assessment metric VMAF that fuses several elementary metrics by SVM regression has shown a higher correlation with human perception. In this paper, we introduce VMAF into the traditional video coding task as the distortion metric, which needs to be optimized to explore the potential of perceptual quality improvement. Specifically, we propose a multi-granularity VMAF based rate-distortion optimization framework. A frame level visual quality adaption is first conducted by taking the quantization characteristics into account at the coarse-grained adjustment step. Within each frame, a CTU level Lagrangian multiplier and corresponding quantization parameter adaption are carried out based on the content of each CTU at the fine-grained adjustment step. The proposed method has been incorporated into the latest video coding standard – VVC. Experimental results show compared with the conventional rate-distortion optimization for SSE, the proposed method achieves an average 3.30% BD-rate reduction in VMAF.

In an increasingly connected world, consumer video experiences have diversified away from traditional broadcast video into new applications with increased use of non-camera-captured content such as computer screen desktop recordings or animations created by computer rendering, collectively referred to as screen content. There has also been increased use of graphics and character content that is rendered and mixed or overlaid together with camera-generated content. The emerging Versatile Video Coding (VVC) standard, in its first version, addresses this market change by the specification of low-level coding tools suitable for screen content. This is in contrast to its predecessor, the High Efficiency Video Coding (HEVC) standard, where highly efficient screen content support is only available in extension profiles of its version 4. This paper describes the screen content support and the five main low-level screen content coding tools in VVC: transform skip residual coding (TSRC), block-based differential pulse-code modulation (BDPCM), intra block copy (IBC), adaptive color transform (ACT), and the palette mode. The specification of these coding tools in the first version of VVC enables the VVC reference software implementation (VTM) to achieve average bit-rate savings of about 41% to 61% relative to the HEVC test model (HM) reference software implementation using the Main 10 profile for 4:2:0 screen content test sequences. Compared to the HM using the Screen-Extended Main 10 profile and the same 4:2:0 test sequences, the VTM provides about 19% to 25% bit-rate savings. The same comparison with 4:4:4 test sequences revealed bit-rate savings of about 13% to 27% for$Y'C_{B}C_{R}$and of about 6% to 14% for$R'G'B'$screen content. Relative to the HM without the HEVC version 4 screen content coding extensions, the bit-rate savings for 4:4:4 test sequences are about 33% to 64% for$Y'C_{B}C_{R}$and 43% to 66% for$R'G'B'$screen content.

Due to the widespread deployment of fingerprint/face/speaker recognition systems, the risk in these systems, especially the adversarial attack, has drawn increasing attention in recent years. Previous researches mainly studied the adversarial attack to the vision-based systems, such as fingerprint and face recognition. While the attack for speech-based systems has not been well studied yet, although it has been widely used in our daily life. In this article, we attempt to fool the state-of-the-art speaker recognition model and present speaker recognition attacker , a lightweight multi-layer convolutional neural network to fool the well-trained state-of-the-art speaker recognition model by adding imperceptible perturbations onto the raw speech waveform. We find that the speaker recognition system is vulnerable to the adversarial attack, and achieve a high success rate on both the non-targeted attack and targeted attack. Besides, we present an effective method by leveraging a pretrained phoneme recognition model to optimize the speaker recognition attacker to obtain a tradeoff between the attack success rate and the perceptual quality. Experimental results on the TIMIT and LibriSpeech datasets demonstrate the effectiveness and efficiency of our proposed model. And the experiments for frequency analysis indicate that high-frequency attack is more effective than low-frequency attack, which is different from the conclusion drawn in previous image-based works. Additionally, the ablation study gives more insights into our model.

Affinity, which represents whether two pixels belong to a same instance, is an equivalent representation to the instance segmentation labels. Conventional works do not make an explicit exploration on the affinity. In this article, we present two instance segmentation schemes based on pixel affinity information and show the effectiveness of affinity in both aspects. For proposal-free method, we predict pixel affinity for each image and then propose a simple yet effective graph merge algorithm to cluster pixels into instances. It shows that the affinity is powerful as an instance-relevant information to guide the clustering procedure in proposal-free instance segmentation. For proposal-based methods, we extend conventional framework with affinity head and introduce affinity as attached supervision in training phase. Without any additional inference cost, we can improve the performance of existing proposal-based instance segmentation methods, which shows that the affinity can also be applied as an auxiliary loss and training with such extra loss is beneficial to the training progress. Experimental results show that our schemes achieve comparable performance to other state-of-the-art instance segmentation methods. With Cityscapes training data, the proposed proposal-free method achieves 28.8 AP and the proposal-based method gets 27.2 AP both on test sets.

Recently, neural style transfer has drawn many attentions and significant progresses have been made, especially for image style transfer. However, flexible and consistent style transfer for videos remains a challenging problem. Existing training strategies, either using a significant amount of video data with optical flows or introducing single-frame regularizers, have limited performance on real videos. In this paper, we propose a novel interpretation of temporal consistency, based on which we analyze the drawbacks of existing training strategies; and then derive a new compound regularization. Experimental results show that the proposed regularization can better balance the spatial and temporal performance, which supports our modeling. Combining with the new cost formula, we design a zero-shot video style transfer framework. Moreover, for better feature migration, we introduce a new module to dynamically adjust inter-channel distributions. Quantitative and qualitative results demonstrate the superiority of our method over other state-of-the-art style transfer methods. Our project is publicly available at: https://daooshee.github.io/CompoundVST/.

In this paper, we focus on computationally modeling of the local texture correlations, in an effort to better explore the coding modes with higher priorities in the rate-distortion optimized intra coding. In particular, strong correlations and continuities of local texture with neighboring blocks have been revealed in our analysis, and empirical justifications provide us inspirations on the joint optimization of rate-distortion-complexity when angular modes become finer to adapt the local textures. We examine the philosophy with extensive experiments conducted for refining the intra full-RD list. The results show that better coding performance with on average 0.72% and 3.00% BD-Rate savings for the natural scene and screen content sequences can be achieved in AVS3 test model HPM-5.0 under all intra configuration, with negligible encoding and decoding time variations.

Palette mode is a powerful tool for screen content coding in the upcoming versatile video coding (VVC) standard. In the palette mode, escape samples are employed to handle an outlier case. In this paper, a quantization parameter (QP) dependent fixed-length binarization is proposed for escape samples coding to 1) simplify the design and 2) improve coding efficiency. The length is calculated according to the QP for the current block and correspondingly, the dequantization process can be also implemented by only left shifting. The proposed method is evaluated with VVC reference software VTM-6.0 on typical sequences containing "text and graphics with motion". Experimental results report that the proposed scheme can achieve up to 2.0% BD-rate savings compared to VTM-6.0 with a much simpler design.

Due to the widespread deployment of fingerprint/face/speaker recognition systems, attacking deep learning based biometric systems has drawn more and more attention. Previous research mainly studied the attack to the vision-based system, such as fingerprint and face recognition. While the attack for speaker recognition has not been investigated yet, although it has been widely used in our daily life. In this paper, we attempt to fool the state-of-the-art speaker recognition model and present speaker recognition attacker, a lightweight model to fool the deep speaker recognition model by adding imperceptible perturbations onto the raw speech waveform. We find that the speaker recognition system is also vulnerable to the attack, and we achieve a high success rate on the non-targeted attack. Besides, we also present an effective method to optimize the speaker recognition attacker to obtain a trade-off between the attack success rate with the perceptual quality. Experiments on the TIMIT dataset show that we can achieve a sentence error rate of 99.2% with an average SNR 57.2dB and PESQ 4.2 with speed rather faster than real-time.

Attacking deep learning based biometric systems has drawn more and more attention with the wide deployment of fingerprint/face/speaker recognition systems, given the fact that the neural networks are vulnerable to the adversarial examples, which have been intentionally perturbed to remain almost imperceptible for human. In this paper, we demonstrated the existence of the universal adversarial perturbations (UAPs) for the speaker recognition systems. We proposed a generative network to learn the mapping from the low-dimensional normal distribution to the UAPs subspace, then synthesize the UAPs to perturbe any input signals to spoof the well-trained speaker recognition model with high probability. Experimental results on TIMIT and LibriSpeech datasets demonstrate the effectiveness of our model.

In recent years, neural style transfer has attracted more and more attention, especially for image style transfer. However, temporally consistent style transfer for videos is still a challenging problem. Existing methods, either relying on a significant amount of video data with optical flows or using singleframe regularizers, fail to handle strong motions or complex variations, therefore have limited performance on real videos. In this paper, we address the problem by jointly considering the intrinsic properties of stylization and temporal consistency. We first identify the cause of the conflict between style transfer and temporal consistency, and propose to reconcile this contradiction by relaxing the objective function, so as to make the stylization loss term more robust to motions. Through relaxation, style transfer is more robust to inter-frame variation without degrading the subjective effect. Then, we provide a novel formulation and understanding of temporal consistency. Based on the formulation, we analyze the drawbacks of existing training strategies and derive a new regularization. We show by experiments that the proposed regularization can better balance the spatial and temporal performance. Based on relaxation and regularization, we design a zero-shot video style transfer framework. Moreover, for better feature migration, we introduce a new module to dynamically adjust inter-channel distributions. Quantitative and qualitative results demonstrate the superiority of our method over state-of-the-art style transfer methods.

In the emerging next generation video coding standard Versatile Video Coding (VVC) developed by the Joint Video Exploration Team (JVET), sub-block-based inter-prediction plays a key role in promising coding tools such as Affine Motion Compensation (AMC) and sub-block-based Temporal Motion Vector Prediction (sbTMVP). With sub-block-based inter-prediction, a coding block is divided into sub-blocks, and the motion information of each sub-block is derived individually. Although sub-block-based inter-prediction can provide a higher quality prediction benefiting from a finer motion granularity, it still suffers two problems: uneven prediction quality and boundary discontinuity. In this paper, we present a method of interweaved prediction to further improve sub-block-based inter-prediction. With interweaved prediction, a coding block with AMC or sbTMVP mode is divided into sub-blocks with two different dividing patterns, so that a corner position of a sub-block in one dividing pattern coincides with the central position of a sub-block in the other dividing pattern. Then two auxiliary predictions are generated by AMC or sbTMVP with the two dividing patterns, independently. The final prediction is calculated as a weighted-sum of the two auxiliary predictions. Theoretical analysis and statistical data prove that interweaved prediction can significantly mitigate the two problems in sub-block-based inter-prediction. Simulation results show that the proposed methods can achieve 0.64% BD-rate saving on average with the random access configurations. On sequences with rich affine motions, the average BD-rate saving can be up to 2.54%.

Bi-directional optical flow (BDOF) is an efficient coding tool that has been recently adopted into Versatile Video Coding (VVC) standard. With BDOF, bi-predictive prediction samples of one coding block are enhanced via higher-precision motion vectors (MVs) derived from its two reference blocks. In this way, the energy of prediction error could be reduced, resulting in better coding performance. In VVC, the derived motion information is only used to enhance prediction samples. In this paper, it is proposed to use the derived motion information to also refine decoded MVs. The refined MVs may be used as spatial motion vector prediction (MVP) for the following coding units (CUs), as the temporal MVP for the subsequent pictures, and in the deblocking filtering process. Furthermore, the refined MVs can be used to perform motion compensation (MC) again to further improve the quality of the prediction samples. Simulation results show that the proposed methods can achieve -1.18% BD-rate saving in average under the random access configuration on top of the existing BDOF design in VVC.

Due to the fast inference and excellent learning capability, deep learning has become an effective means for image denoising and attracted considerable attention recently. However, for the images with rich textures and structures, the performance of deep learning approaches is still unsatisfactory. To address this issue, we develop a new convolutional neural network (CNN) for subband image denoising and name it SDCNN. In the proposed approach, we first decompose images into transform domain and denoise the coefficients of various subbands. By incorporating frequency information with spatial context, SDCNN is more effective in recovering image details. In particular, the introduced procedure of subband transform also plays the role of downsampling and enlarges the receptive field without increasing depth or sacrificing efficiency of network. Experimental results show that the SDCNN achieves promising results in terms of both objective and subjective performance.

Affine Motion Model (AMM) based inter prediction, which can represent complex motions such as zooming, rotation or shearing, has been adopted into the Versatile Video Coding (VVC) standard. AMM is defined by Control Point Motion Vectors (CPMVs) in VVC. On the other hand, Adaptive Motion Vector Resolution (AMVR) has also been adopted into VVC standard due to a favorable trade-off between the Motion Vector (MV) precision and the bit consumption on MV Differences (MVDs). However, AMVR is only applied to the Translational Motion Model (TMM), and AMM cannot benefit from it. In this paper, it is proposed to extend AMVR to AMM. Specifically, 1-pixel, 1/4-pixel and 1/16-pixel MV precisions are allowed and can be selected adaptively by each affine-inter mode coded Coding Unit (CU). Simulation results reportedly show that the proposed method can achieve 0.32% BD-rate saving on average under the Random Access configuration.

The Joint Video Exploration Team (JVET) has been developing an emerging standard Versatile Video Coding (VVC), which includes screen contents as one of its requirements. Intra block copy (IBC) and palette coding are the two powerful coding tools for screen content coding. In this paper, a compound palette mode is proposed to exploit the advantages of both IBC and palette coding, which allows samples to be reconstructed by either IBC predictions or palette entries. The proposed method is evaluated with VVC reference software VTM4 on typical sequences containing "text and graphics with motion". Experimental results report significant coding gain that the proposed scheme can achieve 7.80% and 1.03% BD-rate savings under AI conditions on average when compared with VTM4 and the existing palette scheme.

Efficient compression of omnidirectional video is important for emerging virtual reality applications. To compress this kind of video, each frame is projected to a 2D plane [e.g., equirectangular projection (ERP) map] first, adapting to the input format of existing video coding systems. At the display side, an inverse projection is applied to the reconstructed video to restore signals in spherical domain. Such a projection, however, makes presentation and encoding in different domains. Thus, an encoder agnostic to the projection performs inefficiently. In this paper, we analyze how a projection influences the distortion measurements in different domains. Based on the analysis, we propose a scheme to optimize the encoding process based on signals' distortion in spherical domain. With the proposed optimization, an average 4.31% (up to 9.67%) luma BD-rate reduction is achieved for ERP in random access configuration. The corresponding bit saving is averagely 10.84% (up to 34.44%) when considering the viewing field being π/2 . The proposed method also benefits other projections and viewport settings, with a marginal complexity increase.

Establishing an interactive screen sharing system that supports ultra high resolution (such as 4k) is challenging, with latency and frame rate playing important roles in user experience. The screen frame needs to be compressed efficiently without consuming extensive computational resources. We present a hardware-accelerated system for real-time screen sharing, which decreases encoding workload by exploiting content redundancies between successive screen frames. We propose a multiple codec approach that utilizes several encoders with H.264 Advanced Video Coding (H.264/AVC) of different input sizes, creating savings in encoding time by selecting the appropriate one for updated screen content. An optimized metadata processing method is proposed as well. Small but distant updates within a frame can be split into independent frames for more efficient compression, which is also beneficial for interactive latency. In the evaluation, the proposed system takes less encoding time than general single codec implementation in common screen sharing scenarios. Measurement for latency shows that the end-to-end latency for 4K resolution screen sharing is only about 17-25 ms, which makes the proposed system suitable for various applications in local wired and wireless connections.

Cascaded Regression (CR) based methods have been proposed to solve facial landmarks detection problem, which learn a series of descent directions by multiple cascaded regressors separately trained in coarse and fine stages. They outperform the traditional gradient descent based methods in both accuracy and running speed. However, cascaded regression is not robust enough because each regressor's training data comes from the output of previous regressor. Moreover, training multiple regressors requires lots of computing resources, especially for deep learning based methods. In this paper, we develop a Self-Iterative Regression (SIR) framework to improve the model efficiency. Only one self-iterative regressor is trained to learn the descent directions for samples from coarse stages to fine stages, and parameters are iteratively updated by the same regressor. Specifically, we proposed Landmarks-Attention Network (LAN) as our regressor, which concurrently learns features around each landmark and obtains the holistic location increment. By doing so, not only the rest of regressors are removed to simplify the training process, but the number of model parameters is significantly decreased. The experiments demonstrate that with only 3.72M model parameters, our proposed method achieves the state-of-the-art performance.

The recent development of CNN-based image dehazing has revealed the effectiveness of end-to-end modeling. However, extending the idea to end-to-end video dehazing has not been explored yet. In this paper, we propose an End-to-End Video Dehazing Network (EVD-Net), to exploit the temporal consistency between consecutive video frames. A thorough study has been conducted over a number of structure options, to identify the best temporal fusion strategy. Furthermore, we build an End-to-End United Video Dehazing and Detection Network (EVDD-Net), which concatenates and jointly trains EVD-Net with a video object detection model. The resulting augmented end-to-end pipeline has demonstrated much more stable and accurate detection results in hazy video.

Screen content coding plays an important role in many applications. To meet the growing demands of screen content coding, the emerging AV1 video codec incorporates several coding tools, which are specially designed for screen content utilizing its distinctive characteristics. Among these tools, the intra block copy utilizes the characteristic that repeating patterns frequently occur in screen content. This paper presents the technology of intra block copy in AV1. In particular, to efficiently search the predictor in the reconstructed regions of the current picture, AV1 uses the hash matching method at the encoder side. For the generation of hash table, a bottom-to-up manner is adopted to reduce the redundant computation and then decrease the encoding time. In addition, several constraints are involved to facilitate hardware design. Experimental results demonstrate that the intra block copy in AV1 can bring 27.1% bitrate saving for screen content. When compared with the non hash-based intra block copy, the hash-based method achieves 12.2% bitrate saving.

Deep convolutional neural networks have demonstrated their powerfulness in a variety of applications. However, the storage and computational requirements have largely restricted their further extensions on mobile devices. Recently, pruning of unimportant parameters has been used for both network compression and acceleration. Considering that there are spatial redundancy within most filters in a CNN, we propose a frequency-domain dynamic pruning scheme to exploit the spatial correlations. The frequency-domain coefficients are pruned dynamically in each iteration and different frequency bands are pruned discriminatively, given their different importance on accuracy. Experimental results demonstrate that the proposed scheme can outperform previous spatial-domain counterparts by a large margin. Specifically, it can achieve a compression ratio of 8.4x and a theoretical inference speed-up of 9.2x for ResNet-110, while the accuracy is even better than the reference model on CIFAR-110.

Transform, as one of the most important modules of mainstream video coding systems, seems very stable over the past several decades. However, recent developments indicate that bringing more options for transform can lead to coding efficiency benefits. In this paper, we go further to investigate how the coding efficiency can be improved over the state-of-the-art method by adapting a transform for each block. We present a variable block-sized signal-dependent transforms (SDTs) design based on the High Efficiency Video Coding (HEVC) framework. For a coding block ranged from $4\times4$ to $32\times32$ , we collect a quantity of similar blocks from the reconstructed area and use them to derive the Karhunen-Loève transform. We avoid sending overhead bits to denote the transform by performing the same procedure at the decoder. In this way, the transform for every block is tailored according to its statistics, to be signal-dependent. To make the large block-sized SDTs feasible, we present a fast algorithm for transform derivation. Experimental results show the effectiveness of the SDTs for different block sizes, which leads to up to 23.3% bit-saving. On average, we achieve BD-rate saving of 2.2%, 2.4%, 3.3%, and 7.1% under AI-Main10, RA-Main10, RA-Main10, and LP-Main10 configurations, respectively, compared with the test model HM-12 of HEVC. The proposed scheme has also been adopted into the joint exploration test model for the exploration of potential future video coding standard.

Traditional intra prediction usually utilizes the nearest reference line to generate the predicted block when considering strong spatial correlation. However, this kind of single-line-based method does not always work well due to at least two issues. One is the incoherence caused by the signal noise or the texture of other objects, where this texture deviates from the inherent texture of the current block. The other reason is that the nearest reference line usually has worse reconstruction quality in block-based video coding. Due to these two issues, this paper proposes an efficient multiple-line-based intra-prediction scheme to improve coding efficiency. Besides the nearest reference line, further reference lines are also utilized. The further reference lines with a relatively higher quality can provide potentially better prediction. At the same time, the residue compensation is introduced to calibrate the prediction of boundary regions in a block when we utilize further reference lines. To speed up the encoding process, this paper designs several fast algorithms. The experimental results show that compared with HM-16.9, the proposed fast search method achieves a 2.0% bit saving on average and up to 3.7% by increasing the encoding time by 112%.

Screen content coding plays an important role in many applications. Conventional reference picture management (RPM) strategies developed for natural content may not work well for screen content. This is because many regions in screen content remain static for a long time, causing a lot of repetitive contents to stay in the decoded picture buffer. The repetitive contents are not conducive to inter prediction, but still occupy valuable memory. This paper proposes a diversity-based RPM scheme for screen content coding. The concept of diversity is introduced for the reference picture set (RPS) to help formulate the RPM problem. By maximizing the diversity of RPS, more potentially better predictions are provided. Better compression performance can then be achieved. Meanwhile, the proposed scheme is nonnormative and compatible with existing video coding standards, such as High Efficiency Video Coding. The experimental results show that, for low delay screen content coding, the bit saving of the proposed scheme is 4.9% on average and up to 13.7%, without increasing encoding time.

Unlike camera-captured video, screen content (SC) often contains a lot of repeating patterns, which makes some blocks used as references much more important than others. However, conventional rate-distortion optimization (RDO) schemes in video coding do not consider the dependence among image blocks, which often leads to a locally optimal parameter selection, especially for SC. In this paper, we present a weighted RDO scheme for SC coding (SCC), in which the repeating characteristics are taken into account when deciding RD tradeoff for each block. For one block, the number being referenced by the current picture and following pictures is estimated and based on the number, we set a proper weight in the RDO process to reflect its importance from a global point of view. To estimate the number being referenced, we propose a hash-based method to approximate the results to avoid the complexity of direct search. Experimental results show that compared with the High Efficiency Video Coding SCC reference software, 10.1%, 14.5%, and 2.2% on average and up to 25.7%, 39.8%, and 4.6% bit saving can be achieved by considering weights provided by our scheme for hierarchical-B, IBBB, and all intra coding structures, respectively. Thanks to our hash-based design, the complexity increase brought by the proposed scheme is marginal.

In the latest High Efficiency Video Coding (HEVC) development, i.e., HEVC screen content coding extensions (HEVC-SCC), a hash-based inter-motion search/block matching scheme is adopted in the reference test model, which brings significant coding gains to code screen content. However, the hash table generation itself may take up to half the encoding time and is thus too complex for practical usage. In this paper, we propose a hierarchical hash design and the corresponding block matching scheme to significantly reduce the complexity of hash-based block matching. The hierarchical structure in the proposed scheme allows large block calculation to use the results of small blocks. Thus, we avoid redundant computation among blocks with different sizes, which greatly reduces complexity without compromising coding efficiency. The experimental results show that compared with the hash-based block matching scheme in the HEVC-SCC test model (SCM)-6.0, the proposed scheme reduces about 77% of hash processing time, which leads to 12% and 16% encoding time savings in random access (RA) and low-delay B coding structures. The proposed scheme has been adopted into the latest SCM. A parallel implementation of the proposed hash table generation on graphics processing unit (GPU) is also presented to show the high parallelism of the proposed scheme, which achieves more than 30 frames/s for 1080p sequences and 60 frames/s for 720p sequences. With the fast hash-based block matching integrated into x265 and the hash table generated on GPU, the encoder can achieve 11.8% and 14.0% coding gains on average for RA and low-delay P coding structures, respectively, for real-time encoding.

This paper proposes a deep learning method for intra prediction. Different from traditional methods utilizing some fixed rules, we propose using a fully connected network to learn an end-to-end mapping from neighboring reconstructed pixels to the current block. In the proposed method, the network is fed by multiple reference lines. Compared with traditional single line-based methods, more contextual information of the current block is utilized. For this reason, the proposed network has the potential to generate better prediction. In addition, the proposed network has good generalization ability on different bitrate settings. The model trained from a specified bitrate setting also works well on other bitrate settings. Experimental results demonstrate the effectiveness of the proposed method. When compared with high efficiency video coding reference software HM-16.9, our network can achieve an average of 3.4% bitrate saving. In particular, the average result of 4K sequences is 4.5% bitrate saving, where the maximum one is 7.4%.

Redundancy is necessary for a storage system to achieve reliability. Frequent errors in large-scale storage systems, for example, cloud, make it desirable to reduce the cost of recovery. Among all types of data in cloud storage, videos generally occupy significant amounts of space due to high volumes and the rapid development of video sharing and video-on-demand services. Unlike general data, videos can tolerate a certain level of quality degradation. This paper investigates multilayer video representations, such as scalable videos and simulcast streaming, and proposes an unequal error protection scheme based on local reconstruction codes (LRC) for video storage. By providing less protection for less important layers or video copies, a better tradeoff between storage and repair cost is achieved. Both theoretical and simulation results show that such a tradeoff can be achieved over the LRC with equal error protection, though the recovered video quality might be slightly lower in rare cases.

Traditional intra prediction schemes usually only use the nearest adjacent reference line to generate the prediction. Although the nearest reference line generally has the strongest statistical correlation with current block, the farther non-adjacent reference lines can still provide potential better prediction in some cases. Thus, in this paper, not only the nearest reference line but also the farther reference lines are utilized to help intra prediction. When using the farther reference lines, an additional residue compensation procedure is introduced to further refine the prediction. In particular, this paper designs three solutions to meet different complexity requirements. They are multiple line-based intra prediction (MLIP), fast search for multiple line-based intra prediction (FS-MLIP), and dual line-based intra prediction (DLIP). Experimental results verify the effectiveness of the proposed methods. When compared with HM-16.9, the proposed MLIP achieves 2.4% bit saving on average with the encoding time increasing about 362%. The FS-MLIP achieves 2.0% bit saving on average with the encoding time increasing about 114%. The DLIP achieves 0.9% bit saving on average with the encoding time increasing about only 15%.

This paper proposes an image dehazing model built with a convolutional neural network (CNN), called All-in-One Dehazing Network (AOD-Net). It is designed based on a re-formulated atmospheric scattering model. Instead of estimating the transmission matrix and the atmospheric light separately as most previous models did, AOD-Net directly generates the clean image through a light-weight CNN. Such a novel end-to-end design makes it easy to embed AOD-Net into other deep models, e.g., Faster R-CNN, for improving high-level tasks on hazy images. Experimental results on both synthesized and natural hazy image datasets demonstrate our superior performance than the state-of-the-art in terms of PSNR, SSIM and the subjective visual quality. Furthermore, when concatenating AOD-Net with Faster R-CNN, we witness a large improvement of the object detection performance on hazy images.

Traditional intra prediction methods exploit some fixed rules to generate prediction, which might not be adaptive enough to handle complicated contents. In this paper, we investigate applying deep neural network to improve the state-of-the-art intra prediction. Considering the characteristics of block-based video coding framework, we propose a fully connected network for intra prediction where all layers except non-linear ones are fully connected. In the proposed network, the inputs are multiple reference lines of the current block and the output is the prediction for the block. When compared with the traditional intra prediction method, the richer context of current block is exploited. For this reason, the proposed network is capable of providing more accurate prediction. Experimental results demonstrate the effectiveness of proposed network. When integrated into the HEVC reference software, the proposed method can achieve up to 3.3% bitrate saving and an average of 1.6% bitrate saving for 4K sequences.

Emerging virtual reality (VR) applications bring much challenge to video coding for 360-degree videos. To compress this kind of video, each picture should be projected to a 2D plane (e.g. equirctangular projection map) first, adapting to the input of existing video coding systems. At the display side, an inverse projection is performed before viewport rendering. However, such a project introduces much different levels of distortions depending on the location, which makes the rate-distortion optimization process in video coding much inefficient. In this paper, we consider the distortion in spherical domain and analyse its influence to the rate-distortion optimization process. Then we derive the optimal rate-distortion relationship in spherical domain and present its optimal solution based on HEVC/H.265. Experimental results show that the proposed method can bring up to 11.5% bit-saving compared with the current HEVC/H.265 anchor for 360-degree video coding.

Different from conventional video, screen content video often contains large movements and abrupt changes between adjacent frames. These distinct characteristics bring great challenges to the implementation of rate control in screen content coding (SCC). This paper proposes a novel rate control scheme for SCC considering the delay constraint. First, a pre-analyzer is designed to collect the information of the proceeding frames, which are about to be encoded. Then, after the information collection, more rational bit allocation strategy is adopted to keep the encoder from both buffer overflow and underflow. Furthermore, a delay constraint condition is derived for buffer and pre-analyzer to avoid additional delay. Experimental results demonstrate that the proposed scheme achieves more accurate rate control accuracy and 3.10 dB gain on average when compared with the existing rate control scheme in HM-16.8+SCM-7.0.

A light field consists of multiple views of a scene, which can be arranged and encoded like a pseudo sequence. Since the correlations between views are not equal and indeed content dependent, a well-constructed coding order and adaptive prediction structure will improve performance. In this paper, we propose an adaptive prediction structure for light field compression. While the coding order is inherited from the 2-D hierarchical coding order, the prediction structure is determined by the differences between scale-invariant feature transform (SIFT) descriptors of the views. Experimental results show that the proposed method leads to on average 5.71% BD-rate reduction compared with fixed prediction structure.

Recently, there is a resurgence of interest in uncoded transmission for wireless visual communication. While conventional coded systems suffer from cliff effect as the channel condition varies dynamically, uncoded linear-transformed transmission (ULT) provides elegant quality degradation for wide channel SNR range. ULT skips non-linear operations, such as quantization and entropy coding. Instead, it utilizes linear decorrelation transform and linear scaling power allocation to achieve optimized transmission. This paper presents a theoretical analysis for power-distortion optimization of ULT. In addition to the observation in our previous work that a decorrelation transform can bring significant performance gain, this paper reveals that exploiting the energy diversity in transformed signal is the key to achieve the full potential of decorrelation transform. In particular, we investigated the efficiency of ULT with exact or inexact signal statistics, highlighting the impact of signal energy modeling accuracy. Based on that, we further proposed two practical energy modeling schemes for ULT of visual signals. Experimental results show that the proposed schemes improve the quality of reconstructed images by 3~5 dB, while reducing the signal modeling overhead from hundreds or thousands of meta data to only a few meta data. The perceptual quality of reconstruction is significantly improved.

We consider efficient image transmission via time-varying channels. To improve the performance, we propose a new distributed compressive sensing (CS) scheme that can leverage similar images in the cloud. It is featured by channel SNR and bandwidth scalability, high efficiency, and low encoding complexity. For each image, a compressed thumbnail is first transmitted after forward error correction (FEC) and modulation to retrieve similar images and generate a side information (SI) in the cloud. The residual image after subtracting the decompressed thumbnail is then coded and transmitted by CS through a very dense constellation without FEC. The linearly and ratelessly generated CS measurements make it capable of achieving both graceful quality degradation (GD) with the channel SNR and bandwidth scalability in a universal scheme. A mode decision and transform-domain power allocation are introduced for better bandwidth usage and protection against channel errors. At the decoder, a two-step CS decoding is performed to recover the residual signal, where both the local and nonlocal correlations within the image and that with the SI are exploited. Simulations on landmark images and an AWGN channel show that the received image quality gracefully increases with the channel SNR and bandwidth. Furthermore, it outperforms existing schemes both subjectively and objectively by up to 11 dB gains compared with the state-of-the-art transmission scheme with GD, i.e. SoftCast.

Discrete Cosine Transform (DCT) has been the commonly used transform for a few decades in image/video coding. However, DCT does not work well on the blocks having anisotropic correlations. In this paper, based on the adaptive dictionary, we propose a new online transform scheme using Orthogonal Matching Pursuit (OMP) for High Efficiency Video Coding (HEVC). For a coding block, we construct its dictionary by exploiting non-local correlations from the reconstructed regions. The OMP algorithm is implemented to obtain the sparse transform coefficients. Experimental results show that the BD-rate savings of the proposed scheme for the sequences with strong edges can be up to 19.9%.

This paper presents diagonal motion partitions (DMP) for inter prediction in HEVC. In addition to the square and rectangular partitions, we propose to add diagonal shaped partitions to match different motion parts with oblique boundaries. Considering the overlap of pixels along the partition boundaries, the calculation of sum of absolute differences (SAD) and the motion compensation for the pixels on the boundaries are weighted. Besides, the residues of a diagonal prediction unit (PU) are augmented to form a rectangular one, so as to perform Hadamard transform on the residues. We also revise the advanced motion vector prediction (AMVP) and merge candidates based on the diagonal motion partitions. Experimental results show that on average 0.8%-1.0% BD-rate reduction can be achieved by DMP.

The Range Extensions (RExt) of the High Efficiency Video Coding (HEVC) standard have recently been approved by both ITU-T and ISO/IEC. This set of extensions targets video coding applications in areas including content acquisition, postproduction, contribution, distribution, archiving, medical imaging, still imaging, and screen content. In addition to the functionality of HEVC Version 1, RExt provide support for monochrome, 4:2:2, and 4:4:4 chroma sampling formats as well as increased sample bit depths beyond 10 bits per sample. This extended functionality includes new coding tools with a view to provide additional coding efficiency, greater flexibility, and throughput at high bit depths/rates. Improved lossless, near-lossless, and very high bit-rate coding is also a part of the RExt scope. This paper presents the technical aspects of HEVC RExt, including a discussion of RExt profiles, tools, applications, and provides experimental results for a performance comparison with previous relevant coding technology. When compared with the High 4:4:4 Predictive Profile of H.264/Advanced Video Coding (AVC), the corresponding HEVC 4:4:4 RExt profile provides up to ~25$ %, ~32%, and ~36% average bit-rate reduction at the same PSNR quality level for intra, random access, and low delay configurations, respectively.

A screen content coding (SCC) extension to High Efficiency Video Coding (HEVC) is currently under development by the Joint Collaborative Team on Video Coding, which is a joint effort from the ITU-T Video Coding Experts Group and the ISO/IEC Moving Picture Experts Group. The main goal of the HEVC-SCC standardization effort is to enable significantly improved compression performance for videos containing a substantial amount of still or moving rendered graphics, text, and animation rather than, or in addition to, camera-captured content. This paper provides an overview of the technical features and characteristics of the current HEVC-SCC test model and related coding tools, including intra-block copy, palette mode, adaptive color transform, and adaptive motion vector resolution. The performance of the SCC extension is compared against existing standards in terms of bitrate savings at equal distortion.

Template matching (TM) was proposed in the literature a decade ago to efficiently remove non-local redundancies within an image without transmitting any overhead of displacement vectors. However, the large computational complexity introduced at both the encoder and the decoder, especially for a large search range, limits its widespread use. This paper proposes a hash-based line-by-line template matching (hLTM) for lossless screen image coding, where the non-local redundancy commonly exists in text and graphics parts. By hash-based search, it can largely reduce the search complexity of template matching without an accuracy degradation. Besides, the line-by-line template matching increases prediction accuracy by using a fine granularity. Experimental results show that the hLTM can significantly reduce both the encoding and decoding complexities by 68 and 23 times, respectively, compared with the traditional TM with a search radius of 128. Moreover, when compared with High Efficiency Video Coding screen content coding test model SCM-1.0, it can largely improve coding efficiency by up to 12.68% bits saving on screen contents with rich texts/graphics.

An uncoded transmission scheme called SoftCast has recently shown great potential for wireless video transmission. Unlike conventional approaches, SoftCast processes input images only by a series of transformations and modulates the coefficients directly to a dense constellation for transmission. The transmission is uncoded and lossy in nature, with its noise level commensurate with the channel condition. This paper presents a theoretical analysis for an uncoded visual communication, focusing on developing a quantitative measurements for the efficiency of decorrelation transform in a generalized uncoded transmission framework. Our analysis reveals that the energy distribution among signal elements is critical for the efficiency of uncoded transmission. A decorrelation transform can potentially bring a significant performance gain by boosting the energy diversity in signal representation. Numerical results on Markov random process and real image and video signals are reported to evaluate the performance gain of using different transforms in uncoded transmission. The analysis presented in this paper is verified by simulated SoftCast transmissions. This provide guidelines for designing efficient uncoded video transmission schemes.

Fractional-pel motion compensation is very good at improving video coding efficiency, especially for camera-captured content. But for screen content, which is obtained from a computer desktop, motion vectors with integer-precision may be enough to represent the motion in different pictures. Using fractional-pel motion compensation for such content is a waste of bits. Thus, adaptive motion compensation precision is helpful for improving coding efficiency, especially for screen content coding. Usually, to select suitable motion compensation precision, multi-pass encoding is introduced, which significantly increases the encoding time. This paper presents a fast encoding algorithm for adaptive motion compensation precision used in screen content coding by hash-based block matching. With the proposed method, multi-pass encoding is avoided and most of the benefits brought by adaptive motion compensation precision are preserved. The experimental results show that with the proposed method, up to 7.7% bit saving is obtained without a significant impact on encoding time.

Motion estimation plays an important role in video coding standards, such as H.264/AVC and HEVC. In this paper, we propose a multi-stage hash based motion estimation algorithm for HEVC, which enables hash based motion estimation for natural videos. In the proposed method, the prediction blocks significantly different from the current prediction unit will be eliminated in the motion estimation process. Locality sensitive hashing functions are used to measure the difference between the input block and predicted blocks. The proposed algorithm is implemented into the HM 12.0 software, and the simulation results show that the complexity of motion estimation is significantly reduced with negligible coding performance loss.

We present a compound image compression scheme based on the dictionary-based Lempel-Ziv-Markov chain algorithm (LZMA), under the framework of High Efficiency Video Coding (HEVC). Through matching strings from the sliding window dictionary, LZMA exploits the characteristics of the repeated patterns over the text and graphics regions of compound images, and represents them compactly. To obtain high compression efficiency even for noisy text and graphics contents, we have modified LZMA to support both lossless and lossy compression. We develop and treat it as a new intramode of HEVC. Experimental results show that the proposed scheme achieves significant coding gains for compound image compression. Thanks to the introduction of the lossy LZMA, the compression performance for noisy compound images is improved for more than 5dB in terms of PSNR in comparison with the lossless LZMA scheme.

Redundancy is necessary for a storage system to recover from errors. The frequent errors in large-scale systems, e.g. cloud, make it desired to reduce the recovery cost. Among all kinds of data stored in the cloud, video takes a large portion due to its large data volume. The other characteristic of video is that a certain distortion can be tolerated. This paper investigates using scalable video representation and unequal error protection scheme to reduce the storage and recovery costs in the cloud. By introducing more protection for the base layer and less on the enhancement layers, it can achieve a better tradeoff between storage and reconstruction costs although the reliability for the enhancement layer sacrifices a little. Simulation results based on local reconstruction codes (LRC) show that comparing with the existing (12, 2, 2) LRC code in Windows Azure Storage, the reconstruction cost can be reduced from 6x to 3x at the same storage cost at the expense of possible video quality loss.

Screen content usually has much different motion characteristics compared with conventional videos, which makes exiting rate control schemes unsuitable. This paper analyses the motion characteristics of screen content and proposes an efficient rate control method for screen content coding, by improving the bit allocation and model parameter adaptation strategies. The proposed rate control algorithm is able to both control bitrate accurately and improve the quality of the entire video sequence. The experimental results demonstrate that the proposed algorithm achieves smaller bitrate errors and better coding performance. Compared with the existing rate control scheme in High Efficiency Video Coding (HEVC) reference software, the proposed algorithm improves the coding efficiency by 5.6% on average while obtaining smaller bitrate errors.

Emerging screen content coding brings great challenges to rate control due to significantly different characteristics of screen content from conventional video, e.g. large motion, frequent scene changes. This paper proposes an efficient rate control scheme for screen content coding by considering the characteristics of screen content. We first classify pictures of a video sequence into different groups by comparing the current picture with its neighbours. Then for each group, we apply different strategies to bit allocation and parameter updating process based on the characteristics of each group. The proposed algorithm can control the bitrate accurately without introducing any additional encoding delay. Experiment results show that the proposed scheme can significantly improve PSNR with a more accurate bitrate compared with the existing rate control scheme.

Screen content refers to computer generated content like text, graphics, and animations. In such video, many regions may remain static for a long period after a sudden change. Traditional hierarchical Quantization Parameter (QP) setting may not be able to handle these regions efficiently because the encoder probably needs to refine the quality of these static regions multiple times. It will cost more bits while the quality of the static regions may reach the expected degree which the flat QP setting is able to achieve. This paper proposes using different QP settings for different regions in a picture. Region classification algorithms are developed to determine whether a flat or hierarchical QP setting is used. Experimental results demonstrate that the proposed scheme can achieve an average bitrate reduction of 3.1%, and up to 8.1% bitrate reduction for IBBB coding. The proposed method improves coding efficiency without increasing encoding complexity.

This paper proposes a distributed compressive sensing (CS) scheme for robust image transmission over unknown or time-varying channels with highly correlated images at the decoder. A compressed thumbnail is first transmitted after digital forward error correction (FEC) and modulation to retrieve highly correlated images and generate a side information (SI) at the decoder. The current residual image after subtracting the decompressed thumbnail is then coded and transmitted by CS through a very dense constellation without FEC. The linear representation of the residual signal by CS measurements and rateless sampling makes it able to achieve graceful degradation and bandwidth scalability without channel feedback. Moreover, a transform-domain power allocation is employed before random sampling to protect against channel errors. At the decoder, both the nonlocal correlations within the original image and the correlation with the SI are exploited in CS decoding via a low-rank regulation on similar patches. After CS decoding, a block-wise minimum-mean-square-error (MMSE) reconstruction using the SI is further performed in the spatial domain to enhance the reconstruction quality. Simulations on landmark images and an unknown Gaussian channel show that an up to 10 dB gain is achieved at low channel SNRs compared with the state-of-the-art uncoded image transmission scheme, i.e. SoftCast, when highly correlated images are available at the decoder.

Screen content videos often have mixed content consisting of various types such as natural content, text and graphics in the same picture. To achieve high coding efficiency for this content, new coding tools are developed in the High Efficiency Video Coding (HEVC) standard Screen Content Coding (SCC) extension. Among them, intra block copy (IntraBC) allows a nonlocal intra prediction from the coded region of the same picture. However, the rate-distortion optimization (RDO) scheme of the screen content coding still follows that of the HEVC reference software and each block is optimized locally. The screen content characteristics are not fully utilized in the current RDO scheme. This paper presents a weighted RDO scheme for the intra coding of screen content videos, which measures the importance of each block within the picture first and then larger distortion weights are applied to the blocks with larger importance in the RDO. In this way, a better rate-distortion trade-off can be achieved for the picture instead of the blocks themselves. The experimental results show that up to 5.5% coding efficiency gain can be achieved compared with the reference software.

With multimedia flourishing on the Web, it is easy to find similar images for a query, especially landmark images. Traditional image coding, such as JPEG, cannot exploit correlations with external images. Existing vision-based approaches are able to exploit such correlations by reconstructing from local descriptors but cannot ensure the pixel-level fidelity of the reconstruction. In this paper, a cloud-based distributed image coding (Cloud-DIC) scheme is proposed to exploit external correlations for mobile photo uploading. For each input image, a thumbnail is transmitted to retrieve correlated images and reconstruct it in the cloud by geometrical and illumination registrations. Such a reconstruction serves as the side information (SI) in the Cloud-DIC. The image is then compressed by a transform-domain syndrome coding to correct the disparity between the original image and the SI. Once a bitplane is received in the cloud, an iterative refinement process is performed between the final reconstruction and the SI. Moreover, a joint encoder/decoder mode decision at block, frequency, and bitplane levels is proposed to adapt to different correlations. Experimental results on a landmark image database show that the Cloud-DIC can largely enhance the coding efficiency both subjectively and objectively, with up to 5-dB gains and 70% bits saving over JPEG with arithmetic coding, and perform comparably at low bitrates with the intra coding of the High Efficiency Video Coding standard with a much lower encoder complexity.

Although the High Efficiency Video Coding (HEVC) standard significantly improves the coding efficiency of video compression, it is unacceptable even in offline applications to spend several hours compressing 10 s of high-definition video. In this paper, we propose using a multicore central processing unit (CPU) and an off-the-shelf graphics processing unit (GPU) with 3072 streaming processors (SPs) for HEVC fast encoding, so that the speed optimization does not result in loss of coding efficiency. There are two key technical contributions in this paper. First, we propose an algorithm that is both parallel and fast for the GPU, which can utilize 3072 SPs in parallel to estimate the motion vector (MV) of every prediction unit (PU) in every combination of the coding unit (CU) and PU partitions. Furthermore, the proposed GPU algorithm can avoid coding efficiency loss caused by the lack of a MV predictor (MVP). Second, we propose a fast algorithm for the CPU, which can fully utilize the results from the GPU to significantly reduce the number of possible CU and PU partitions without any coding efficiency loss. Our experimental results show that compared with the reference software, we can encode high-resolution video that consumes 1.9% of the CPU time and 1.0% of the GPU time, with only a 1.4% rate increase.

By considering the increasing importance of screen contents, the high efficiency video coding (HEVC) standard includes screen content coding as one of its requirements. In this paper, we demonstrate that enabling frame level block searching in HEVC can significantly improve coding efficiency on screen contents. We propose a hash-based block matching scheme for the intra block copy mode and the motion estimation process, which enables frame level block searching in HEVC without changing the HEVC syntaxes. In the proposed scheme, the blocks sharing the same hash values with the current block are selected as prediction candidates. Then the hash-based block selection is employed to select the best candidates. To achieve the best coding efficiency, the rate distortion optimization is further employed to improve the proposed scheme by balancing the coding cost of motion vectors and prediction difference. Compared with HEVC, the proposed scheme achieves 21% and 37% bitrate saving with all intra and low delay configurations with encoding time reduction. Up to 59% bitrate saving can be achieved on sequences with large motions.

Screen contents with complex structure contain random combination of texts, graphics and camera-captured images, which makes them difficult to be compressed efficiently by traditional video codecs. In this paper, we propose a 2-D dictionary based scheme to exploit the repeated patterns on screen content. In the proposed scheme, the current block is predicted from the reconstructed region using a hash-based block searching scheme. A hierarchical two-level hash based searching scheme is designed to find the best matching block for each block. The first-level hash function is used to search the blocks similar to the current block in the constructed 2-D dictionary. The second-level hash function is used to update the 2-D dictionary, which filters out the identical blocks from the blocks found using the first-level hash function. The proposed scheme is incorporated into HEVC framework as an additional mode. Experimental results show that the proposed scheme achieves significantly coding performance improvements on screen contents compared with HEVC.

In this paper, we propose an adaptive weighted distortion optimization algorithm for improving the coding performance of video represented in RGB color space. The YCbCr is the color space, which is usually adopted in video coding. And with 4:2:0 planar format, it generally uses the ratio 6:1:1 metrics objective peak signal-to-noise ratio (PSNR) for quality assessment. In this paper, we still use 6:1:1 combined PSNR as a universal criterion to optimize video coded in RGB domain and measure the final quality of decoded video as well. The new distortion metric in RGB coding is derived from the optimal target, and the modification is only applied on the encoder side. The proposed algorithm achieves about 26% bit-saving on average and maximum 44% bit-saving in HEVC range extensions reference software HM12.0 RExt4.1. Additionally, the visual quality of video coded by the proposed algorithm is also improved.

This paper proposes a cloud-based distributed image coding scheme (Cloud-DIC) to exploit the strong correlations with external partial-duplicate images in the cloud. It features both high coding efficiency and low encoder complexity, which makes it suitable for photo sharing on mobile devices. To get the side information in the cloud, a thumbnail of the current image is transmitted to retrieve highly correlated images and reconstruct through geometrical registration and adaptive patched-based stitching. The current image is then compressed by a transform-domain syndrome coding, bitplane by bitplane. Once a bitplane is received, the decoded high-quality image is further used to refine the side information in the cloud, which will benefit the coding of following bitplanes and the reconstruction. Experimental results on a landmark image database show that it can largely enhance the coding efficiency both subjectively and objectively with up to 5 dB gains and 58% bits saving over JPEG.

The line-based intra block copy (IntraBC) technique is newly proposed in the High Efficient Video Coding (HEVC) Range Extensions to deal with repeated patterns within a picture for screen content coding. One challenge for line-based IntraBC is the large overhead by transmitting displacement vectors (DV) for each line. In this paper, a DV prediction is proposed to reduce such an overhead. Moreover, a flipping copy based prediction is proposed to better exploit the correlations within screen content. Experimental results show that our scheme can provide a BD-rate reduction of 11.0% compare with the HEVC Range Extension anchor with the encoding time increased by 23% and no decoding time increase.

This paper introduces 1-D dictionary mode designed for screen content coding. Two 1-D dictionary modes are designed to improve the coding efficiency for screen content. The first one is called normal dictionary mode, in which a virtual dictionary should be maintained and all the prediction comes from the virtual dictionary. The other one is called reconstruction based dictionary mode, where no virtual dictionary is to be maintained and all the previously reconstructed pixels in the same picture can be used for prediction. Hash based search is designed to find matching for both dictionary modes efficiently. 1-D dictionary mode with variable block sizes are also supported in the proposed scheme. The experimental results show the proposed algorithm achieves about 10% ~ 18.4% bit saving for different coding structures. The bit saving is up to 60% for the proposed method.

This paper introduces a unified framework of hash-based matching method for screen content coding. Screen content has some different characteristics from camera-captured content, such as large motion and repeating patterns. Hash-based matching is proposed to better explore the correlation in screen content, thus, improving the coding efficiency. The proposed method can handle both intra picture and inter picture block matching with variable block sizes in a unified framework. The proposed framework is also easy to be extended to handle other motion models to further improve the coding efficiency of screen content. We also develop fast encoding algorithms to make full use of the hash results. The experimental results show the proposed algorithm achieves about 12% bit saving while saving more than 25% encoding time. The bit saving is up to 57% and the encoding time saving is up to 60% for the proposed method.

High Efficiency Video Coding (HEVC) uses a very flexible tree structure to organize coding units, which leads to a superior coding efficiency compared with previous video coding standards. However, such a flexible coding unit tree structure also places a great challenge for encoders. In order to fully exploit the coding efficiency brought by this structure, huge amount of computational complexity is needed for an encoder to decide the optimal coding unit tree for each image block. One way to achieve this is to use parallel computing enabled by many-core processors. In this paper, we analyze the challenge to use many-core processors to make coding unit tree decision. Through in-depth understanding of the dependency among different coding units, we propose a parallel framework to decide coding unit trees. Experimental results show that, on the Tile64 platform, our proposed method achieves averagely more than 11 and 16 times speedup for 1920x1080 and 2560x1600 video sequences, respectively, without any coding efficiency degradation.

High Efficiency Video Coding (HEVC) provides superior coding efficiency than previous video coding standards at the cost of increasing encoding complexity. The complexity increase of motion estimation (ME) procedure is rather significant, especially when considering the complicated partitioning structure of HEVC. To fully exploit the coding efficiency brought by HEVC requires a huge amount of computations. In this paper, we analyze the ME structure in HEVC and propose a parallel framework to decouple ME for different partitions on many-core processors. Based on local parallel method (LPM), we first use the directed acyclic graph (DAG)-based order to parallelize coding tree units (CTUs) and adopt improved LPM (ILPM) within each CTU (DAGILPM), which exploits the CTU-level and prediction unit (PU)-level parallelism. Then, we find that there exist completely independent PUs (CIPUs) and partially independent PUs (PIPUs). When the degree of parallelism (DP) is smaller than the maximum DP of DAGILPM, we process the CIPUs and PIPUs, which further increases the DP. The data dependencies and coding efficiency stay the same as LPM. Experiments show that on a 64-core system, compared with serial execution, our proposed scheme achieves more than 30 and 40 times speedup for 1920 × 1080 and 2560 × 1600 video sequences, respectively.

In this paper, we propose a novel coding and transmission scheme, called LineCast, for broadcasting satellite images to a large number of receivers. The proposed LineCast matches perfectly with the line scanning cameras that are widely adopted in orbit satellites to capture high-resolution images. On the sender side, each captured line is immediately compressed by a transform-domain scalar modulo quantization. Without syndrome coding, the transmission power is directly allocated to quantized coefficients by scaling the coefficients according to their distributions. Finally, the scaled coefficients are transmitted over a dense constellation. This line-based distributed scheme features low delay, low memory cost, and low complexity. On the receiver side, our proposed line-based prediction is used to generate side information from previously decoded lines, which fully utilizes the correlation among lines. The quantized coefficients are decoded by the linear least square estimator from the received data. The image line is then reconstructed by the scalar modulo dequantization using the generated side information. Since there is neither syndrome coding nor channel coding, the proposed LineCast can make a large number of receivers reach the qualities matching their channel conditions. Our theoretical analysis shows that the proposed LineCast can achieve Shannon's optimum performance by using a high-dimensional modulo-lattice quantization. Experiments on satellite images demonstrate that it achieves up to 1.9-dB gain over the state-of-the-art 2D broadcasting scheme and a gain of more than 5 dB over JPEG 2000 with forward error correction.

Screen content like cartoons, captures of typical computer screens or video with text overlay or news ticker is an important category of video, which needs new techniques beyond the existing video coding techniques. In this paper, we analyze the characteristics of screen content and coding efficiency of HEVC on screen content. We propose a new coding scheme, which adopts a non-transform representation, separating screen content into color component and structure component. Based on the proposed representation, two coding modes are designed for screen content to exploit the directional correlation and non-translational changes in screen video sequences. The proposed scheme is then seamlessly incorporated into the HEVC structure and implemented into HEVC range extension reference software HM9.0. Experimental results show that the proposed scheme achieves up to 52.6% bitrate saving compared with HM9.0. On average, 35.1%, 29.2% and 23.6% bitrate saving are achieved with intra, random-access and low-delay configurations, respectively. The visual quality of the decoded video sequences is also significantly improved by reducing ringing artifacts around sharp edges and reserving the shape of text without blur.

This paper presents a Lagrange multiplier determination method and Quantization Parameter (QP) refinement algorithm used in the Rate-Distortion Optimization (RDO) process to improve the coding efficiency for Audio Video Coding Standard (AVS) (IEEE 1857). This paper investigates the Lagrange multiplier setting problem for different kinds of pictures in the encoding process of AVS. When Lagrange multiplier is determined, to minimize the RD (Rate-Distortion) cost, the optimal QP can be selected by multiple-QP optimization. However, this kind of optimization increases the encoding time significantly. This paper analyzes the relationship between the QP value and Lagrange multiplier in this paper. And then this relationship is applied in the encoding process of AVS to refine the predetermined QP value. The experimental results show that proposed algorithm can lead to significant bit saving compared with the default anchor of AVS. The proposed method improves the coding efficiency without increasing the processing time.

This paper presents an adaptive weighted distortion optimization algorithm used in the Rate-Distortion Optimization (RDO) process of the High Efficiency Video Coding (HEVC). RDO is an important tool to improve the coding efficiency. Usually the distortion weights of different color components are equal or predetermined. In this paper, an adaptive weighted distortion optimization algorithm is introduced to improve the coding efficiency. The distortion weight is estimated according to the previous coded pictures belonging to the same temporal level, such that encoding complexity is almost unchanged. With the proposed adaptive weighted distortion optimization method, on average about 3.3% and up to 10.6% bit-saving are obtained based on the latest HEVC reference software, HM-8.0 and the corresponding common test conditions. The proposed algorithm can also be applied to other coding schemes such as H.264/MPEG-4 AVC.

This paper presents a Quantization Parameter (QP) refinement algorithm used in Rate-Distortion Optimization (RDO) process to improve the coding efficiency for High Efficiency Video Coding (HEVC). To minimize the RD cost, QP is one of the parameters that can be optimized. Usually, multiple-QP optimization can be applied to choose the best QP value. However, this kind of optimization increases the encoding complexity significantly. We analyze the relationship between the QP value and Lagrange multiplier in this paper firstly. And then we apply the relationship between QP and Lagrange multiplier to the encoding process of HEVC. The proposed QP refinement method has already been adopted into HM (the HEVC reference software). The experimental results show that proposed algorithm can achieve about 1.4%~1.9% bit saving for luma component on average compared with the default anchor of HM-8.0. The bit saving on chroma components are larger than that of luma. The proposed method improves the coding efficiency without increasing the processing time.

In wireless scenarios where the channel condition may vary drastically, visual communication systems using source and channel coding generally suffer from threshold effect. An uncoded transmission scheme called SoftCast [1]-[3], however, was recently shown to provide both graceful quality transition and competitive performance. In SoftCast, image signal is directly modulated to a dense constellation using proper power for transmission, solely after employing a transform for energy compaction, leaving out conventional quantization, entropy coding and channel coding. The received signal is lossy in nature, with its noise level commensurate with the channel condition. This paper presents a theoretical analysis for uncoded visual communication, focusing on the role of transform and the quantitative measurement of transform gain in a generalized uncoded transmission framework with optimal power allocation. Our analysis reveal that the energy distribution among signal elements plays an important role in the power-distortion performance. Further analysis show that the energy compaction capability of decorrelation transform can bring significant gain by boosting the energy diversity in signal representation. Numerical analysis results are reported for Markov random signals and natural images, respectively. The performance of typical transforms, e.g. KLT, DCT and DWT, and the effect of different transform sizes or levels are evaluated. These analysis results are verified by simulations.

By employing 35 intra prediction modes, HEVC standard performs better to remove spatial redundancy between the current block and its neighbors. Although 3D video coding has adopted HEVC intra prediction and Depth Modeling Modes (DMM) technology to improve the performance, the Explicit Wedgelet Partition Mode within DMM brings unaffordable complexity to compress depth map. Based on the way of obtaining the picture texture from the mode with Sum of Absolute Transform Difference in rough mode decision, we propose a fast scheme to determine Wedgelet Partition in intra prediction, which leads to significant computational saving with marginal BD-rate increase after decoder-side view synthesis.

Screen content refers to image/video generated by electronic devices. Existing state-of-art image/video coding standards don't exploit the anisotropic features. This paper proposes an adaptive LZMA-based coding scheme for screen content to achieve better coding efficiency. Adaptive scanning and directional matching are incorporated into LZMA in our scheme. Both of them exploit the directional correlations in screen content to increase the matching probability of LZMA, and thus achieve better coding performance. Experimental results show that the proposed scheme significantly outperforms existing LZMA-based coding schemes on screen content coding.

The mean-square-error (MSE) distortion criterion used in the state-of-the-art video coding standards, e.g. H.264/AVC and the High Efficiency Video Coding (HEVC) under standardization, is widely criticized for poor measurement of perceived visual quality. Existing research on perceptual video coding mainly employs low-level features of images/video, which cannot take into account the big picture people see. This paper proposes a scene-aware perceptual video coding scheme (SAPC), which accommodates human visual perception of the scene by reconstructing the scene from video and perform scene-based bits allocation. To be specific, more bits are allocated to the foreground object and its boundaries considering that people tend to pay more attention to the foreground and object boundaries are prone to blur at low bitrates for object occlusion. The structure from motion (SFM) technology is employed for scene reconstruction. Experiments taking HEVC as the benchmark show that our algorithm can give better visual quality than the original HEVC encoder at the same bitrate.

High Efficiency Video Coding (HEVC), not only provides a much better coding efficiency than previous video coding standards, but also shows significantly superior performance than other image coding schemes when applied to image coding. However, the improvement is at the cost of significant increase of encoding complexity. In this paper, we focus on retaining the high coding efficiency provided by HEVC while largely reducing its encoding complexity for image coding. By applying various techniques including optimized coding structure parameters, coding unit early termination, fast intra prediction and transform skip mode decision, we significantly reduce the complexity of HEVC intra coding while keeping most of its coding efficiency. Experimental results show that our light-weight HEVC encoder can save about 82% coding time compared with original HEVC encoder. With a slight loss to the HEVC reference software, the proposed scheme still gains about 19% in BD-BR compared with H.264/AVC.

Compared with H.264/AVC, the emerging video coding standard, High Efficiency Video Coding (HEVC), introduces a lots of intra prediction modes in intra coding, which brings a real challenge in practice. Although the Rough Mode Decision (RMD) scheme in HEVC provides a fast mode decision, it is a fixed strategy for different blocks with the same size and does not consider their different characteristics. In this paper, we analyze the relation between a block's texture characteristics and its best coding mode. Based on the analysis, we further develop an adaptive strategy for fast mode decision in intra coding for HEVC. Experimental results show significant computational saving in All-Intra High Efficiency and All-Intra Low Complexity test conditions compared to the default HM-4.0 with a marginal BD-rate increase.

Slice-parallel video coding with a fixed number of slices and uniformly allocated coding regions would cause extra waiting time among slice threads, especially when the encoding machine is also busy running other applications. In this paper, we propose to adaptively decide the number of slices before encoding each frame by computing the expected value of the coding time of the encoder under two kinds of machine states: busy and free based on a Markov system. We also manage to balance computational complexity among slices by utilizing the estimated encoding time of each allocation unit in the frame to be encoded. Experimental results show that the proposed scheme can further improve the coding speed of an H.264 based fixed-slice parallel encoder by 30% on average, while the rate-distortion performance change is marginal.

In this paper, we present several strategies for transcoding from H.264/AVC bit streams to High Efficiency Video Coding (HEVC) bit streams. Because HEVC and AVC share the similar coding architecture, we try to exploit the information in AVC bit streams as much as possible. For inter picture, we utilize the power spectrum based rate-distortion optimization (PS-RDO) model as well as the input residual, modes and motion vectors to estimate the best coding unit (CU) split quad tree, the best prediction unit (PU) mode and the best motion vector of each PU partition For intra picture, we propose to reduce the CU and PU partition candidates. The proposed strategies can significantly reduce the transcoding complexity in terms of reduced processing for RDO evaluations, motion estimation, motion compensation as well as fractional pixel interpolation operations. Experiment results show that the proposed transcoding methods can achieve a good tradeoff between coding efficiency and transcoding complexity.

This paper presents the results of compression comparison tests between the current state of the emerging High Efficiency Video Coding (HEVC) draft standard and the current dominant standard H.264/MPEG-4 AVC (High Profile) as an anchor reference. The conditions used for the comparison tests were designed to reflect relevant application scenarios and to enable a fair comparison to the maximum extent feasible, i.e. using comparable quantization settings, reference frame buffering, etc. The testing was generally configured in favour of using a relatively strong H.264/MPEG-4 AVC anchor reference. Several of the encoder optimizations currently found in the HEVC software are tested and shown to be helpful to improve the H.264/MPEG-4 AVC anchor performance. When compared to the improved anchor encoder configurations, the HEVC draft design currently provides a bit rate savings for equal PSNR of about 39% for random access applications, 44% for low-delay use, and 25% for all-intra use.

Compared to the CAVLC in H.264/AVC, CAVLC in HEVC has been improved much. One mechanism that contributes to the improvement is adaptive code word swapping based on previous coded symbol in the early design of HEVC. However, such a instantly swapping strategy often makes a wrong estimation of the real statistics and may lead to inefficient coding that may even worse than without adaptation. In this paper, we analyze potential loss caused by instantly swapping and propose a counter-based adaptation scheme to further improve the entropy coding efficiency. We discuss the bit-depth of those counters that achieves a trade-off between the adaptation capability and a good estimation of the statistics. The proposed scheme introduces a sum counter for each code table and a counter for each code. The sum counter determines when to shift all the counters to prevent overflow and make sure symbols far away influence less. The experimental results show that the counter based algorithm brings about 0.7% bits saving compared with the instantly swapping adaptation algorithm. Compared with the CAVLC without adaptation, the performance gain of the counter adaptation algorithm is about 4.0% bits saving. A variant of the proposed counter-based adaptation has been adopted by HEVC design.

This paper presents a joint rate-distortion optimization coding scheme for H.264/AVC intra coding, which uses cluster computing technologies to jointly optimize coding parameters among different image blocks. We identify two kinds of dependencies, inter sub-block dependency and inter macroblock dependency, ignored in the ordinary rate-distortion optimization and a joint RD cost is designed in the optimization process to consider those dependencies. We show that with our proposed scheme, a significant better coding performance can be achieved with the standard-compliant bitstreams. Experiment results show up to 5% bits saving compared to the H.264/AVC reference software.

Simultaneously capturing of color and depth videos, e.g. with Kinect, favors many applications and has become very popular. Efficient representation and compression of such data is important yet challenging. In this paper, we have designed an object-based coding system to compress Kinect-like depth and color videos. Segmentation is first conducted to obtain different object planes, where a mask image is utilized to identify them. We compress depth and color images respectively using the proposed object-based coding codec, which is designed based on High Efficiency Video Coding (HEVC). The mask image is losslessly compressed by adding a new context-based mode to HEVC. To assure the alignment of object boundaries on the depth image and those on the color image, a pre-processing is conducted over the depth image. The separate coding of the different object planes for the depth image can avoid the inefficiency coding of edges blocks at object boundaries and thus bring obvious coding gain. Moreover, the attractive functionality of “content-based” coding which permits the transmission of the interested object planes rather than an entire image provides a practical way to decrease the bitrate.

Motion compensation with multiple reference pictures has been widely used during the development of the emerging High Efficiency Video Coding (HEVC) standard, which greatly helps to improve the coding efficiency. Usually, a heuristic strategy is exploited to use the nearest reconstructed pictures as references. However, such a strategy may not be efficient on all occasions, especially when different content characteristics and coding settings are considered. In this paper, we investigate how to manage reference pictures so as to achieve better rate-distortion performance under the memory constraint of the decoded picture buffer at the decoder. We formulate the reference picture management as an optimization problem and approximate its optimal solution. Moreover, we explore how to adjust quality for each picture according to the reference structure to further improve coding efficiency. For some coding cases, where a complicated encoder optimization is unaffordable, we also develop fast algorithms to get the most benefit from reference picture selection. Among them, one strategy has been adopted by the HEVC software and common test conditions to generate the anchor. Experimental results show that the proposed full search algorithm and fast search algorithms achieve significant bitrate reduction.

Computers are developing along with a new trend from the dual-core and quad-core processors to ones with tens or even hundreds of cores. Multimedia, as one of the most important applications in computers, has an urgent need to design parallel coding algorithms for compression. Taking intraframe/image coding as a start point, this paper proposes a pure line-by-line coding scheme (LBLC) to meet the need. In LBLC, an input image is processed line by line sequentially, and each line is divided into small fixed-length segments. The compression of all segments from prediction to entropy coding is completely independent and concurrent at many cores. Results on a general-purpose computer show that our scheme can get a 13.9 times speedup with 15 cores at the encoder and a 10.3 times speedup at the decoder. Ideally, such near-linear speeding relation with the number of cores can be kept for more than 100 cores. In addition to the high parallelism, the proposed scheme can perform comparatively or even better than the H.264 high profile above middle bit rates. At near-lossless coding, it outperforms H.264 more than 10 dB. At lossless coding, up to 14% bit-rate reduction is observed compared with H.264 lossless coding at the high 4:4:4 profile.

Visually summarizing web pages is an attractive approach that provides users an effective and friendly interface to identify desired contents at a first glance for search and re-finding tasks. Using dominant images in web pages is generally reliable for this purpose. However, dominant images are often unavailable in many web pages. To solve this problem, we first propose a new approach to summarize those web pages without any dominant images by retrieving relevant external images from the Internet. However, relevant external images are sometimes unreliable. To take the advantages of these two kinds of images, we further propose a clustering based algorithm to select the best summarization among all of internal and external images. This algorithm leverages relevance and dominance of images as the prior information. Experimental results show that our approach achieves 0.098 and 0.082 NDCG1gain on a human labeled data set, compared with relevant external image and dominant image, respectively. Our user study also indicates that the images selected by our algorithm are useful as the summarization of web pages.

This paper proposes a pure line-by-line image coding scheme (LBC) for many-core computers. The proposed scheme can simultaneously use tens and even hundreds of cores to compress one HD image. In LBC, an input image is coded line by line sequentially and each line is divided into many short segments at equal length that are coded independently. Experimental results show that the proposed scheme can speed the encoding up to 14 times through 16 cores, whereas parallel H.264 intra-frame coding can only speed up less than 5 times. In terms of coding performance, efficient intra predictions are proposed to make full use of the strong correlations among neighboring lines. Experiments results also show that the proposed scheme can have a comparable and even better performance than H.264 high profile at middle bit rates. At high bit rates, the proposed scheme can outperform H.264 high profile more than 10dB.

This paper presents a parallel arithmetic coding scheme in which supports a large degree of parallelism with a marginal cost in terms of the coding efficiency. The parallelism is brought by coding the bits using multiple arithmetic coders. We identify two types of losses in coding efficiency by breaking the dependency among the data: the loss by breaking the probability prediction process and the loss by breaking the information for context decision at the starting of each slice. We further analyze these losses quantitatively and find that the loss by breaking probability adaptation process takes the most of losses. A coding method is proposed in this paper to compensate such a loss by sending additional data. Experimental results show that our proposed method can compensate the loss well, especially for large scale parallel, whereas the overhead is moderate.

Compound images are combinations of text, graphics and natural images. They possess characteristics different from those of natural images, such as a strong anisotropy, sparse color histograms and repeated patterns. Former research on compressing them has mainly focused on developing certain strategies based on some of these characteristics but has failed so far to fully exploit them simultaneously. In this paper, we investigate the combination of four up-to-date strategies to construct a comprehensive scheme for compound image compression. We have implemented these strategies as four types of modes with variable block sizes. Experimental results show that the proposed scheme achieves significant coding gains for compound image compression at all bitrates.

We investigate how to improve video coding efficiency via optimized reference frame selection using large-scale computation resources, e.g., a cloud. We first formulate the optimization problem for reference frame selection in video coding, which can be simplified to a manageable level. Given the maximum number of reference frames for encoding one frame, we give the upper bound of the coding efficiency on the High Efficiency Video Coding (HEVC) platform, which, although ideal, may require a huge amount of reference frame buffering at the decoder. Then we give a solution and the corresponding performance when the reference frame buffer size at the decoder is constrained. Experimental results show that when the number of reference frames is four, the proposed encoding scheme can achieve up to 16.9% bit-saving compared to HEVC, the state-of-the-art video coding system. The proposed encoding scheme is standard-compliant and can also be applied to H.264/AVC to improve coding efficiency.

Scalable video coding provides an efficient way to serve video contents at different quality levels. Based on the development of emerging High Efficiency Video Coding (HEVC), we propose two coarse granular scalable (CGS) video coding schemes here. In scheme A, we present a multi-loop solution in which the fully reconstructed base pictures are utilized in the enhancement layer prediction. By inserting the reconstructed base picture (BP) into the list of reference pictures of the collocated enhancement layer frame, we enable the coarse granular quality scalability of HEVC with very limited changes. On the other hand, scheme B supports single loop decoding. It contains three inter-layer predictions similar to the scalable extension of H.264/AVC. Compared to scheme A, it decreases the decoding complexity by avoiding the motion compensation, deblocking filtering (DF) and adaptive loop filtering (ALF) in the base layer. The effectiveness of our proposed two coding schemes is evaluated by comparing with single-layer coding and simulcast.

One problem in the current High Efficiency Video Coding design is that the entropy decoding, or the parsing process of one frame may depend on syntax elements on other frames. Thus, a small error may lead to immediate failure of decoding of the current frame and all the following inter frames. This paper analyzes the cause of this vulnerability of HEVC bitstreams. Based on the analysis, we further present two solutions to make a tradeoff between the coding efficiency and the error robustness by cutting down the error propagation caused by temporal motion vector prediction. One solution is to redesign some code tables so that the dependency on other frames' syntax elements can be broken. The other solution is to insert recovery points for the coded bitstream, so that the decoding process can recover from errors. The pros and cons of these two solutions are discussed. Experimental results show that the proposed methods can solve the parsing problem in HEVC with a marginal influence on the coding performance and provide significant improvements for the decoded video when there are errors.

Compound images/videos are a mixture of text, graphics and natural images/video. Despite extensive research on compound image coding, rare research on compound video coding has been reported in the literature. This paper proposes three approaches to exploit inter-frame correlations in compound video. One is the motion compensation aided base color (MCA-BC) approach, which handles a special type of motion on text/graphics parts: foreground/background color change. The other two are motion-compensated residue base color and index map (MC-BCIM) and the motion-compensated residue scalar quantization (MC-RSQ) approach, both of which exploit spatial correlations among motion compensated residues. Experiments based on the High-Efficiency Video Coding (HEVC) reference software HM0.9 show that the proposed scheme largely improves the coding efficiency for more than 3dB on compound video with text/graphics and still keep a comparable performance on natural video content.

In lifting-based directional wavelet transforms, different subsampling patterns may show significant difference for directional signals in image coding. This paper investigates the influence of subsampling in directional wavelet transform. We show that the best subsampling depends on the direction and the directionality strength of the signal. To improve the coding performance, we further propose a subsampling-adaptive directional wavelet transform, which can use different subsampling patterns adaptively and according to the local characteristics of the image. To handle the boundary transition when subsampling changes, a phase completion process is applied to ensure that wavelet transform with various subsampling can be performed without introducing boundary effects and performance loss. Experimental results show that the proposed transform can achieve significant coding gain in image coding compared to other existing directional wavelet transforms.

For natural images, there are usually repeating similar contents but hard to be well predicted locally. Prediction using template matching is an effective technology to exploit such a non-local correlation. In this paper, we propose an alternative scheme to further exploit the non-local correlation. In the proposed scheme, template matching is also used to search for probable similar references to the current block to be coded. We then use these references to train an adaptive transform, which most likely reflects the statistical characteristic of the current block. The proposed scheme can further exploit correlation between the current block and more possible references. Compared to the scheme that only integrates prediction by template matching, the proposed scheme shows improvement about 0.45dB PSNR increase or 9.3% bit saving on average, which leads to 1dB's gain or 19.5% bit saving on average compared to the state-of-the-art scheme without using template matching.

This paper proposes a line-based approach for image coding. Unlike conventional block-based image coding, the proposed method takes one line as a basic unit in coding and reconstruction. Such a structure allows us to perform better predictions between neighboring units and affords us more flexibility when coding each unit. We also present an efficient prediction method using adaptive prediction filters for this structure. Based on the neighboring information, we apply the best prediction filter on each line to get high-quality predictions. We show that the proposed line-based coding can perform comparatively or even better than the state-of-the-art block-based coding. When combing line-based and block-based coding together, the state-of-the-art image coding can be much improved.

Transform-based image coding has been the mainstream for many years, as witnessed in from the early effort in JPEG to the recent advances in HD Photo. Traditionally, a 2-D transform used in image coding is always implemented separately along the vertical and horizontal directions, respectively. However, it is usually true that many image blocks contain oriented structures (e.g., edges) and/or textures that do not follow either the vertical or horizontal direction. The traditional 2-D transform thus may not be the most appropriate one to these image blocks. This well-known fact has recently triggered several attempts towards the development of directional transforms so as to better preserve the directional information in an image block. Some of these directional transforms have been applied in image coding, demonstrating a significant coding gain. This paper presents an overview of these directional transforms as well as a discussion of some existing problems and their potential solutions.

Visual summarization is an attractive new scheme to summarize web pages, which can help achieve a more friendly user experience in search and re-finding tasks by allowing users quickly get the idea of what the web page is about and helping users recall the visited web page. In this paper, we perform a careful study on the recently proposed visual summarization approaches, including the thumbnail of the web page snapshot, the internal image in the web page which is representative of the content in the page, and the visual snippet which is a synthesized image based on the internal image, the title, and the logo found in the web page. Moreover, since the internal image based summarization approach hardly works when the representative internal images are unavailable, we propose a new strategy, which retrieves the representative image from the external to summarize the web page. The experimental results suggest that the various summarization approaches have respective advantages on different types of web pages. While internal images and thumbnails can provide a reliable summarization on web pages with dominant images and web pages with simple structure respectively, the external images are regarded as a useful information to complement the internal images and are demonstrated very useful in helping users understanding new web pages. The visual snippet performs well on the re-finding tasks since it incorporates the title and logo which are advantageous on identifying the visited web pages.

Line-based coding has shown its potential in improving the coding efficiency of intra-frame/image coding due to its flexibility in prediction. In our previous work we proposed an efficient line-based image coding method (LIC) by adaptive line-by-line prediction (ALP) and adaptive residue coding. In this paper we further improve this line-based coding scheme by exploiting long-distance correlations in prediction. Line-by-line template matching (LTM) is introduced to perform the long-distance prediction. Experiments in the KTA software show that the LTM scheme can effectively improve the performance of LIC at low bitrates and also brings some improvement at high bitrates. Up to 1db gain is achieved compared to previous LIC and 1.5db compared to KTA on images with regular patterns or strong edges. Up to 20% rate reduction over KTA is also achieved on some standard video sequences with different resolutions.

While directional adaption is introduced into traditional transforms, different orders of two 1-D transforms will result in different results of one 2-D transform. Based upon an anisotropic image model, this paper analyzes the effect of transform orders in terms of theoretical coding gain. Our results reveal that the transform orders have little effect on the coding gain with full decomposition, good directional modes and good interpolation. However, in practical compression schemes, since high-pass bands are not decomposed fully because of the consideration on complexity, different transform orders have different coding performances, which can be solved by an adaptive transform order. Motivated by our analyzed results, a directional filtering transform (dFT, in order to distinguish from the common usage on DFT) is proposed in this paper to better exploit correlations among samples in H.264 intraframe coding. It provides an evenly distributed set of prediction modes with an adaptive transform order. Both interblock and intrablock correlations are exploited in this scheme. Experimental results in H.264 intraframe coding demonstrate its superiority both objectively and subjectively.

In this paper, we present the design of directional lapped transforms for image coding. A lapped transform, which can be implemented by a prefilter followed by a discrete cosine transform (DCT), can be factorized into elementary operators. The corresponding directional lapped transform is generated by applying each elementary operator along a given direction. The proposed directional lapped transforms are not only nonredundant and perfectly reconstructed, but they can also provide a basis along an arbitrary direction. These properties, along with the advantages of lapped transforms, make the proposed transforms appealing for image coding. A block-based directional transform scheme is also presented and integrated into HD Phtoto, one of the state-of-the-art image coding systems, to verify the effectiveness of the proposed transforms.

Summary form only given. This paper studies how to better exploit intra correlation to compress images. In general, edge and texture areas of images exhibit strong anisotropic property. The correlation among samples is determined by not only their distance but also the link orientation. Traditional transforms are not efficient on handling this anisotropic correlation. Therefore, in this paper we propose a directional filtering transform (dFT, in order to distinguish from the common usage on DFT) to exploit local anisotropic correlation among samples. Similar to directional prediction in H.264 intra-frame coding, but it adopts the hierarchal structure to decrease the distance between samples to be predicted and that are used for prediction. From another viewpoint, the dFT prediction resembles the directional wavelet transform without update, which can take both intra-block and inter-block correlations into account.

Intra-predictive transforms are a kind of block-based transforms that can exploit both the intra- and inter-block correlations. This paper analyzes the coding gains of intra-predictive transforms for the Gaussian process. The tight upper bound of the coding gain is derived, which is shown to be better than both the discrete cosine transform and the Karhunen Loe¿ve transform. The optimal intra-predictive transform that achieves the upper bound is also presented. Actual coding results on images verify the effectiveness of the optimal intra-predictive transform.

This paper proposes the directional filtering transform (dFT, in order to distinguish from the common usage on DFT) to better exploit intra-frame correlation in H.264 intra-frame coding. It consists of a directional filtering and an optional DCT transform. In the proposed directional filtering, there are two different approaches. One is the uni-directional filtering (UDF) that is similar to H.264 directional intra prediction. In this approach, only samples from neighboring blocks can be used in prediction. Another is bidirectional filtering (BDF) that exploits the correlations among samples from not only neighboring blocks but also the current block. The prediction structure in this approach is hierarchical multi-layer. In this paper, we present mathematical analyses on UDF and BDF and show the advantage to combine them together. The proposed dFT is integrated into H.264 intra-frame coding too. The preliminary experimental results in H.264 demonstrate its superiority.

This paper presents a new transform framework, intra-predictive transforms, for block-based image coding. The proposed framework unifies cross-block prediction and transforms within a block. Thus, it can exploit both the intra- and inter- block correlations. The intra-prediction in H.264 can be viewed as a special case of intra-predictive transforms. The new framework provides more flexibility to design new transform for coding. As an example, a new intra-predictive transform based on frequency domain prediction is also presented to show the advantages of the proposed framework.

This paper presents a scheme to design directional lapped transforms. Lapped transforms can be factorized into lifting steps. By introducing directional operator into each lifting step, the directional lapped transform is constructed. The directional lapped transform proposed not only preserves the advantages of lapped transforms, it also can represent directional signals more efficiently. An image coding scheme using the directional lapped transform is also described. Compared to the state-of-the-art image coding using lapped transform, HD photo, the proposed scheme shows more than 20 dB's gain for artificial images with strong directional correlations. And for natural images, up to 1.5 dB's gain can also be observed.

Recent nonlocal means filter (NLM) [1] is lack of mean conservation property, that is, image can have different mean before and after being processed. It is not suppose to happen when dealing with zero-mean noise. In this paper, a compensation procedure is proposed as post-process of NLM. The constraint of mean conservation is imposed on NLM through adding to the restored image of NLM with a compensation image. The compensation image is obtained via processing the method noise of NLM. Experiments show that the compensation procedure is able to significantly improve the performance of NLM.

The recent nonlocal means filter (NLM) exhibits exciting performance for image or video denoising. However, the estimation of NLM is biased and tends to keep the value of noise. In the process of NLM, the target pixel (the pixel being processed) is involved in patch distance measuring, which, in turn, controls the restoration of the target pixel. Such design makes the estimation of the target pixel to be biased to the noise. The biased estimation may degrade the reconstruction especially when the noise energy is large. In this paper, the cause and influence of such biased estimation phenomenon is analyzed in detail. We will show that the biased estimation can be extremely harmful or helpful, depending on the degree of noise. Taking advantage of this, a new method is proposed to adaptively control the contribution of target pixel in patch distance measuring. This method achieves much better denoising performance compared with NLM.

In this paper, we propose a new adaptive lifting scheme that not only locally adapts the filtering directions to the orientations of image features, but also adapts the lifting filters to the statistic properties of image signal. The proposed approach refines previous adaptive directional lifting-based wavelet transform (ADL) by combining directional lifting and adaptive lifting filters to form a unified framework. The image signal is first segmented into regions of textures and edges with close directional features. The lifting filters are then effectively designed. The prediction step is designed to minimize the prediction error of the image signal, and the update step is designed to minimize the reconstruction error. Significant improvements on objective and subjective quality over conventional 2-D wavelet transform and previous ADL transform are achieved.

The adaptive directional lifting-based wavelet transform (ADL) locally adapts the filtering directions to the local properties of the image. In this letter, instead of using the conventional interpolation filter for the directional prediction with fractional-pel accuracy, a new two-dimensional nonseparable adaptive interpolation filter is proposed. The adaptive filter is calculated for every fractional-pel direction so as to minimize the energy of the prediction error. The tradeoff between reducing the prediction error and the overhead to code the interpolation filter is discussed. This enables coding gains of up to 0.98 dB, compared to ADL coder, and up to 2.4 dB, compared to the JPEG 2000 for typical test images.

In existing pyramid-based spatially scalable coding schemes, such as H.264/MPEG-4 SVC (scalable video coding), video frame at a certain high-resolution layer is mainly predicted either from the same frame at the next lower resolution layer, or from the temporal neighboring frames within the same resolution layer. But these schemes fail to exploit both kinds of correlation simultaneously and therefore cannot remove the redundancies among resolution layers efficiently. This paper extends the idea of spatiotemporal subband transform and proposes a general in-scale motion compensation technique for pyramid-based spatially scalable video coding. Video frame at each high-resolution layer is partitioned into two parts in frequency. Prediction for the lowpass part is derived from the next lower resolution layer, whereas prediction for the highpass part is obtained from neighboring frames within the same resolution layer, to further utilize temporal correlation. In this way, both kinds of correlation are exploited simultaneously and the cross-resolution-layer redundancy can be highly removed. Furthermore, this paper also proposes a macroblock-based adaptive in-scale technique for hybrid spatial and SNR scalability. Experimental results show that the proposed techniques can significantly improve the spatial scalability performance of H.264/MPEG-4 SVC, especially when the bit-rate ratio of lower resolution bit stream to higher resolution bit stream is considerable.

In most existing in-band wavelet video coding schemes, over-complete wavelet transform is used for the motion-compensated temporal filtering (MCTF) of each spatial subband. It can overcome the shift-variance of critical sampling wavelet transform and improve the coding efficiency of the in-band scheme. However, a dilemma exists in the current implementations of in-band MCTF (IBMCTF), which is whether or not to exploit the spatial highpass subbands in motion compensation of the spatial lowpass subband. The absence of the spatial highpass subbands will result in significant quality loss in the reconstructed full-resolution video, whereas the presence of the spatial highpass subbands may bring serious mismatch error in the decoded low-resolution video since the corresponding highpass subbands may be unavailable at the decoder. In this paper, we first analyze the mismatch error propagation in decoding the low-resolution video. Based on our analysis, we then propose a frame-based cross-resolution leaky prediction scheme for IBMCTF. It can make a good tradeoff between alleviating the low-resolution mismatch and improving the full-resolution coding efficiency. Experimental results show that the proposed scheme can dramatically reduce the mismatch error by 0.3-2.5 dB for low resolution, while the performance loss is marginal for high resolution.

Traditional 2D DCT implemented by separable 1D transform in horizontal and vertical directions does not take image orientation features in a local window into account. To improve it, we propose to introduce directional primary operations to the lifting-based DCT and thereby derive a new directional DCT-like transform. A JPEG-wise image coding scheme is also proposed to evaluate the performance of the proposed directional DCT-like transform. In this scheme, each 8times8 block can be transformed along different direction in order to better adapt to local orientation properties of image. The experimental results show that the performance of the proposed directional DCT can dramatically outperform the conventional DCT up to 2 dB even without modifying entropy coding.

We propose an image coding scheme using 2-D anisotropic dual-tree discrete wavelet transform (DDWT). First, we extend 2-D DDWT to anisotropic decomposition, and obtain more directional subbands. Second, an iterative projection-based noise shaping algorithm is employed to further sparsify anisotropic DDWT coefficients. At last, the resulting coefficients are rearranged to preserve zero-tree relationship so that they can be efficiently coded with SPIHT. Experimental results show that our proposed scheme outperforms JPEG2000 and SPIHT at low bit rates despite the redundancy of DDWT.

This paper extends the so called generalized in-scale coding technique developed in our previous work to a macroblock-based adaptive in-scale motion compensation framework for H.264/MPEG-4 SVC (scalable video coding). In this framework, the in-scale technique is employed to generate predictions for video frames at high resolution layers, which can utilize both temporal correlation and cross-resolution correlation simultaneously. The lowpass content of a high resolution frame is predicted from its lower resolution layer and the highpass content is predicted from neighboring frames at the same resolution layer. For quality and resolution combined scalability, the prediction from lower resolution layer is not always better than temporal prediction. The in-scale technique is thereby employed adaptively at macroblock level according to the local characteristics of video signal. Techniques for motion estimation and mode decision are also investigated. Experimental results demonstrate that the scheme proposed in this paper can outperform H.264/MPEG-4 SVC significantly. The coding performance is quite promising especially for high-fidelity video coding

In existing video coding schemes with spatial scalability based on pyramid frame representation, such as the ongoing H.264/MPEG-4 SVC (scalable video coding) standard, video frame at a high resolution is mainly predicted either from the lower-resolution image of the same frame or from the temporal neighboring frames at the same resolution. Most of these prediction techniques fail to exploit the two correlations simultaneously and efficiently. This paper extends the in-scale prediction technique developed for wavelet video coding to a generalized in-scale motion compensation framework for H.264/MPEG-4 SVC. In this framework, for a video frame at a high resolution layer, the lowpass content is predicted from the information already coded in lower resolution layer, but the highpass content is predicted by exploiting the neighboring frames at current resolution. In this way, both the cross-resolution correlation and temporal correlation are exploited simultaneously, which leads to much higher efficiency in prediction. Preliminary experimental results demonstrate that the proposed framework improves the spatial scalability performance of current H.264/MPEG-4 SVC. The improvement is significant especially for high-fidelity video coding. In addition, another advantage over wavelet-based in-scale scheme is achieved that the proposed framework can support arbitrary down-sampling and up-sampling filters.

This paper provides an overview of the Barbell lifting coding scheme that has been adopted as common software by the MPEG ad hoc group on further exploration of wavelet video coding. The core techniques used in this scheme, such as Barbell lifting, layered motion coding, 3D entropy coding and base layer embedding, are discussed. The paper also analyzes and compares the proposed scheme with the oncoming scalable video coding (SVC) standard because the hierarchical temporal prediction technique used in SVC has a close relationship with motion compensated temporal lifting (MCTF) in wavelet coding. The commonalities and differences between these two schemes are exhibited for readers to better understand modern scalable video coding technologies. Several challenges that still exist in scalable video coding, e.g., performance of spatial scalable coding and accurate MC lifting, are also discussed. Two new techniques are presented in this paper although they are not yet integrated into the common software. Finally, experimental results demonstrate the performance of the Barbell-lifting coding scheme and compare it with SVC and another well-known 3D wavelet coding scheme, MC embedded zero block coding (MC-EZBC).

The motion compensated temporal filtering (MCTF) technique, which is extensively used in 3-D wavelet video coding schemes nowadays, leads to signal coupling among various spatial subbands because motion alignment is introduced in the temporal filtering. Using all spatial subbands as a reference enables MCTF to fully take advantage of temporal correlation across frames but inevitably brings drifting problem in supporting spatial scalability. This paper first analyzes the signal coupling phenomenon and then proposes a quantitative model to describe signal propagation across spatial subbands during the MCTF process. The signal propagation is modeled for a single MC step based on the shifting effect of wavelet synthesis filters and then it is extended to multilevel MCTF. This model is called subband coupling aware signal propagation (SCASP) model in this paper. Based on the model, we further propose a subband coupling aware rate allocation scheme as one possible solution to the above dilemma in supporting spatial scalability. To find the optimal rate allocation among all subbands for a specified reconstruction resolution, the SCASP model is used to approximate the reconstruction process and derive the synthesis gain of each subband with regard to that reconstruction. Experimental results have fully demonstrated the advantages of our proposed rate allocation scheme in improving both objective and subjective qualities of reconstructed low-resolution video, especially at middle bit rates and high bit rates.

Traditional 2-D discrete cosine transform (DCT) implemented by separable 1-D transform in horizontal and vertical directions does not take image orientation features in a local window into account. To improve it, we propose to introduce directional primary operations to the lifting-based DCT and thereby derive a new directional DCT-like transform, whose transform matrix is dependent on directional angle and interpolation used there. Furthermore, the proposed transform is compared with the straightforward one of first rotated and then transformed. A JPEG-wise image coding scheme is also proposed to evaluate the performance of the proposed directional DCT-like transform. The first 1-D transform is performed according to image orientation features, and the second 1-D transform still in the horizontal or vertical direction. At the same time, an approach is proposed to optimally select transform direction of each block because selected directions of neighboring blocks will influence each other. The experimental results show that the performance of the proposed directional DCT-like transform can dramatically outperform the conventional DCT up to 2 dB even without modifying entropy coding.

In subband image and video compression, the issue of dyadic spatial scalability with a downsizing ratio of 2:1 has already been widely investigated. In some application scenarios, however, other downsizing ratios can be desirable. This paper proposes a wavelet coding scheme to support non-dyadic spatial scalability. By extending lifting-based transform structure, we show how wavelet transform can directly support non-dyadic spatial scalability. Specially, based on the extended lifting framework, a wavelet transform is designed to decompose an image with size 3W×3H to a low-pass subband with size 2W×2H and three high-pass subbands to support 3:2-ratio spatial scalability. The characteristic of proposed wavelet low-pass filter is investigated and experimental results show that the low resolution image produced by our scheme has good quality while the coding performance just has marginal loss compared to Daubechies 9/7 filter.

The progressive fine granularity scalable (PFGS) coding is a promising technique for streaming video applications. The macroblock-based PFGS coding proposes several INTER modes for the enhancement macroblock coding and provides a simple method to determine a coding mode for each macroblock. Because it does not consider the drifting error possibly occurs in decoder, the mode decision method is not effective. In this paper, we present a mode decision scheme to make a trade-off between the low drifting error and high coding efficiency, by adding a middle bit-rate loop in encoder to estimate the possible drifting error in decoder and using an R-D model to estimate the improvement of coding efficiency by selecting high quality reference. Then a nearly optimal coding mode for each enhancement macroblock can be easily determined. The experimental results show that the proposed scheme can effectively reduce the drifting error at low bit rates, while maintaining improved coding efficiency at high bit rates

This paper proposes a subband adaptive motion compensated temporal filtering (MCTF) technique for scalable video coding and introduces a revised synthesis gain model for the quantization in this adaptive MCTF scheme. In scalable video coding, hierarchical MCTF is extensively adopted to exploit the temporal correlation across video frames. In this hierarchical MCTF structure, the strength of temporal correlation varies with the level of temporal transform and varies with the various spatial frequency components in a frame. The reconstruction noises also have diverse strength at various subbands. According to the correlation and noise characteristics of various subbands, we can adjust the strength of motion compensated prediction step in MCTF to maximally take the advantage of temporal correlation but restrict the propagation of reconstruction noise. The quantization step of each subband is also adjusted according to synthesis gain determined by the MCTF structure. In this way an adaptive MCTF scheme is formed and the proposed technique improves the coding performance of scalable video coding

This paper presents a novel high-efficient hybrid open-close loop based fine granularity scalable (HOCFGS) coding framework supporting different decoding complexity applications. The open-loop motion compensation for inter-pictures is introduced to efficiently exploit the temporal correlation among adjacent pictures for both base layer and FGS enhancement layer within a wide bit-rate range. An efficient rate-distortion optimized macro-block mode decision rule is used to reduce drifting error and achieve comparable coding performance at the lowest bit-rate point (base layer) with non-scalable coding. An approach like MPEG-4 FGS coding with close-loop motion compensation only at base layer is used for some inter-pictures to stop the drifting error propagation. Furthermore, to achieve better coding performance for these inter-pictures, block based progressive fine granularity scalable (BLPFGS) is introduced, in which leaky prediction is used to generate high-quality reference for FGS enhancement layer. In BLPFGS coding, efficient bit-plane coding and de-blocking techniques are investigated to improve the coding performance for FGS enhancement layer, especially at low bit-rate points. The coding performance for the proposed method is verified by integrating it into MPEG SVC reference software.

To handle the mismatch problems of spatial-domain motion aligned temporal filtering (MATF) in providing spatial scalability, this paper presents a novel in-scale motion aligned temporal filtering (ISMATF) for highly scalable video coding. For each lifting step of ISMATF, the predict or update is carried out at various resolutions in a scale-by-scale way and the operation at higher resolution is designed to embed the operations at lower resolutions to prevent redundancy in frame representation. The proposed ISMATF has several main features, which as a whole differentiates it from other schemes (e.g. spatial-domain MATF, in-band MATF, straightforward multi-resolution MATF) and makes it effective in video coding with spatial scalabilities. These features include 1) Multi-resolution lifting with perfect reconstruction capability; 2) non-redundancy in frame representation, which favors highly scalable video coding; 3) temporal filtering performed in the image-domain of corresponding resolution, which is different from in-band schemes and preserves high efficiency of motion compensation; 4) no mismatch between the encoder and decoder, no matter which resolution is decoded. The proposed ISMATF scheme solves the mismatch problems of SDMATF at low resolution while maintains a good performance at high resolution. It can outperform redundant multi-resolution MATF schemes by up to 1.0 dB at high resolution.

A lifting implementation of the discrete wavelet transform applied along motion trajectories has recently gained a lot of attention in the video community as strong candidates in incoming scalable video coders. We generalize the coding scheme for classical lifting-based motion compensation temporal filtering and permit the codec to choose adaptively between the original reference frames and new fading-compensated reference frames to predict residuals while maintaining the invertibility of the inter-frame transform. Experimental results show that the proposed algorithm not only significantly improves subjective visual quality of the temporal low-pass frames, but also has 0.15-0.3 dB gain in PSNR performance compared with the normal (5, 3) lifting schemes.

This paper presents an improved update operator for H.264 scalable extension, which is a new joint standardization effort of ISO/IEC and ITU-T. Energy distributed update (EDU) is utilized to reduce decoder complexity. The relationship between EDU and the optimum update in B. Girod et al., (2005) is revealed. Moreover, for the constrained motion-compensated lifting structure employed in H.264 scalable extension, we derive a set of new update coefficients to improve coding performance based on distortion analysis. Experimental results show that the proposed update operator can improve coding efficiency with less complexity and meanwhile, frame-wise quality fluctuation is also reduced

Subband video coding is an elegant scheme to fulfill high performance scalable video coding. In this paper, a new update scheme, energy distributed update steps (EDU), is proposed for the temporal transform in lifting based motion compensated video coding. The idea is to update where predict is made by distributing high-pass signals to the low-pass frame. The scheme avoids complex and inaccurate inversion of the motion information that used in the traditional update steps, thus it reduces computations in temporal transform. Experimental results show that the coding performances can be improved up to 0.77 dB.

This paper proposes a framework of scalable motion estimation and coding with the structure of multilayers for 3D wavelet video coding. The motion representation consists of multiple layers. The encoder uses motion of all layers to perform analysis, while the decoder may receive only part of motion for synthesis. Different from other schemes, each layer of motion is a point optimized at a certain range of bit-rate. We observe that the distortion introduced by motion mismatch is highly independent with the rate for texture in a wide range. Therefore, to make the best trade-off between motion and texture under the constraint of a given bit rate, a motion layer decision algorithm is used to find the appropriate number of motion layers to be included into the bit-stream. The proposed framework also supports the spatial and temporal scalabilities of motion. Experimental results show significant improvement at low bit-rates and nearly no loss at high bit-rates with layered motion coding and optimal motion decision. The performance is approaching to the convex hull of those with multiple sets of nonscalable motion.

Three-dimensional (3D) wavelet-based scalable video coding provides a viable alternative to standard MC-DCT coding. However, many current 3D wavelet coders experience severe boundary effects across group of pictures (GOP) boundaries. This paper proposes a memory-efficient transform technique via lifting that effectively computes wavelet transforms of a video sequence continuously on the fly, thus eliminating the boundary effects due to limited length of individual GOPs. Coding results show that the proposed scheme completely eliminates the boundary effects and gives superb video playback quality.

This paper introduces a lossy to lossless coding technique for compression of multitarget fluorescence in situ hybridization (M-FISH) images using 3-D embedded subband coding with optimal truncation (3-D ESCOT) (Xu et al.). With a lifting-based integer wavelet decomposition, 3-D ESCOT achieves about twice as much compression as Lempel-Ziv (WinZip) coding-the current method for archiving M-FISH images. The lossy coding performance of 3-D ESCOT is significantly better than that of 2-D based JPEG-2000.

The rapid development of wired and wireless networks tremendouslyfacilitates communications between people. However, most of thecurrent wireless networks still work in low bandwidths, and mobiledevices still suffer from weak computational power, short batterylifetime and limited display capability. We developed a very lowbit-rate bi-level video coding technique, which can be used invideo communications almost anywhere, anytime on any device. Thespirit of this method is that rather than giving highest priorityto the basic colors of an image as in conventional DCT-basedcompression methods, we give preference to the outline features ofscenes when we have limited bandwidths. These features can berepresented by bi-level image sequences that are converted fromgray-scale image sequences. By analyzing the temporal correlationbetween successive frames and flexibilities in the scenepresentation using bi-level images, we achieve very high ratioswith our bi-level video compression scheme. Experiments show thatin low bandwidths, our method provides clearer shape, smoothermotion, shorter initial latency and much cheaper computational costthan do DCT-based methods. Our method is especially suitable forsmall mobile devices such as handheld PCs, palm-size PCs and mobilephones that possess small display screens and light computationalpower, and work in low bandwidth wireless networks. We have builtPC and Pocket PC versions of bi-level video phone systems, whichtypically provide QCIF-size video with a frame rate of 5-15 fps fora 9.6 Kbps bandwidth.

As the Internet and wirless networks are developed rapidly, the demand of communicating anywhere, anytime on any device emerges. However, most of the current wireless networks still work in low bandwidths, and mobile devices still suffer from weak computational power, short battery lifetime and limited display capability. We developed portrait video phone systems that can run on Pcs and Pocket Pcs at very low bit rates through the Internet. The core technology that portrait video phones employ is the so-called portrait video (or bi-level video) codec. Portrait video codec first converts a full-color video into a black/white image sequence and then compresses it into a black/white portrait-like video. Portrait video processes clearer shape, smoother motion, shorter initial latency, and cheaper computational cost than MPEG2, MPEG4 and H.263 for low bandwidths. Typically the portrait video phone provides QCIF-size video with a frame rate of 5-15 fps for a 9.6 Kbps video bandwidth. The portrait video is so small that it can even be transmitted through an HTTP proxy as text. Experiments show that the portrait video phones work well on ordinary GSM wireless telecommunication networks.

Concentric mosaics offer a quick solution to construct a virtual copy of a real environment, and navigate in the virtual environment. However, the huge amount of data associated with concentric mosaics is a heavy burden for its application. A 3D wavelet transform-based compressor has been proposed in previous work to compress the concentric mosaics. In this paper, we greatly improve the performance of the 3D wavelet coder with a data rearrangement mechanism called “smart rebinning”. The proposed scheme first aligns the concentric mosaic image shots along the horizontal direction and then rebins the shots into multi-perspective panoramas. Smart rebinning greatly improves the cross shot correlation and enables the coder to better explore the redundancy among shots. Experimental results show that the performance of the 3D wavelet coder improves an average of 4.3dB with the use of smart rebinning. The proposed coder outperforms MPEG-2 coding of concentric mosaics by an average of 3.7dB.