Recent advances in extreme image compression have revealed that mapping pixel data into highly compact latent representations can significantly improve coding efficiency. However, most existing methods compress images into 2-D latent spaces via convolutional neural networks (CNNs) or Swin Transformers, which tend to retain substantial spatial redundancy, thereby limiting overall compression performance. In this paper, we propose a novel Mixed RWKV-Transformer (MRT) architecture that encodes images into more compact 1-D latent representations by synergistically integrating the complementary strengths of linear-attention-based RWKV and self-attention-based Transformer models. Specifically, MRT partitions each image into fixed-size windows, utilizing RWKV modules to capture global dependencies across windows and Transformer blocks to model local redundancies within each window. The hierarchical attention mechanism enables more efficient and compact representation learning in the 1-D domain. To further enhance compression efficiency, we introduce a dedicated RWKV Compression Model (RCM) tailored to the structure characteristics of the intermediate 1-D latent features in MRT. Extensive experiments on standard image compression benchmarks validate the effectiveness of our approach. The proposed MRT framework consistently achieves superior reconstruction quality at bitrates below 0.02 bits per pixel (bpp). Quantitative results based on the DISTS metric show that MRT significantly outperforms the state-of-the-art 2-D architecture GLC, achieving bitrate savings of 43.75%, 30.59% on the Kodak and CLIC2020 test datasets, respectively.

With the rapid advancement of 3D visualization, 3D Gaussian Splatting (3DGS) has emerged as a leading technique for real-time, high-fidelity rendering. While prior research has emphasized algorithmic performance and visual fidelity, the perceptual quality of 3DGS-rendered content, especially under varying reconstruction conditions, remains largely underexplored. In practice, factors such as viewpoint sparsity, limited training iterations, point downsampling, noise, and color distortions can significantly degrade visual quality, yet their perceptual impact has not been systematically studied. To bridge this gap, we present 3DGS-QA, the first subjective quality assessment dataset for 3DGS. It comprises 225 degraded reconstructions across 15 object types, enabling a controlled investigation of common distortion factors. Based on this dataset, we introduce a no-reference quality prediction model that directly operates on native 3D Gaussian primitives, without requiring rendered images or ground-truth references. Our model extracts spatial and photometric cues from the Gaussian representation to estimate perceived quality in a structure-aware manner. We further benchmark existing quality assessment methods, spanning both traditional and learning-based approaches. Experimental results show that our method consistently achieves superior performance, highlighting its robustness and effectiveness for 3DGS content evaluation. The dataset and code are made publicly available to facilitate future research in 3DGS quality assessment.

Recent advances in video generation techniques have given rise to an emerging paradigm of generative video coding for Ultra-Low Bitrate (ULB) scenarios by leveraging powerful generative priors. However, most existing methods are limited by domain specificity (e.g., facial or human videos) or excessive dependence on high-level text guidance, which tend to inadequately capture fine-grained motion details, leading to unrealistic or incoherent reconstructions. To address these challenges, we propose Trajectory-Guided Generative Video Coding (dubbed T-GVC), a novel framework that bridges low-level motion tracking with high-level semantic understanding. T-GVC features a semantic-aware sparse motion sampling pipeline that extracts pixel-wise motion as sparse trajectory points based on their semantic importance, significantly reducing the bitrate while preserving critical temporal semantic information. In addition, by integrating trajectory-aligned loss constraints into diffusion processes, we introduce a training-free guidance mechanism in latent space to ensure physically plausible motion patterns without sacrificing the inherent capabilities of generative models. Experimental results demonstrate that T-GVC outperforms both traditional and neural video codecs under ULB conditions. Furthermore, additional experiments confirm that our framework achieves more precise motion control than existing text-guided methods, paving the way for a novel direction of generative video coding guided by geometric motion modeling.

This paper proposes a novel neural B-frame video compression framework with hybrid implicit motion modeling. In our approach, implicit motion modeling replaces the rate-consuming yet less effective flow-based explicit motion modeling to improve overall RD performance. Specifically, an interpolated frame is first generated from the forward and backward reference frames to enrich the temporal priors. A Hybrid Temporal Prior Extractor (HTPE) is then introduced to exploit these priors, where a hybrid feature extractor combining Content-Aware Depthwise Separable Convolution (CADSC) and Linear Attention Duality (LAD) adaptively captures local and global temporal features, respectively. Finally, the enriched temporal prior features are leveraged in the main encoder/decoder to enable implicit motion modeling, and are further integrated into the entropy model to improve the accuracy of entropy estimation for the discrete latent representation.

In point cloud compression, the quality of a reconstructed point cloud relies on both the global structure and the local context, with existing methods usually processing global and local information sequentially and lacking communication between these two types of information. In this paper, we propose a point-voxel interlaced network (PVINet), which captures global structural features and local contextual features in parallel and performs interactions at each scale to enhance feature perception efficiency. Specifically, PVINet contains a voxel-based encoder ($\mathcal {E}_{v}$) for extracting global structural features and a point-based encoder ($\mathcal {E}_{p}$) that models local contexts centered at each voxel. Particularly, a novel conditional sparse convolution is introduced, which applies point embeddings to dynamically customize kernels for voxel feature extraction, facilitating feature interactions from$\mathcal {E}_{p}$to$\mathcal {E}_{v}$. During decoding, a voxel-based decoder employs conditional sparse convolutions to incorporate point embeddings as guidance to reconstruct the point cloud. Experiments on benchmark datasets show that PVINet delivers competitive performance compared to state-of-the-art methods.

Distance estimation and theoretical derivation in 3D space form the foundation basis for improving localization performance in wireless sensor networks (WSNs). Localization is a pivotal challenge in wireless sensor network (WSN) applications. To address this issue, we propose a probability-based distance estimation (PDE) model and a distance correction strategy based on expected hops (DCSEH). First, the PDE model is constructed from the multi-hop probability distribution of nodes, from which the upper bound and average distance for anchor nodes to detect target nodes under different hop counts are derived. Second, the DCSEH strategy effectively mitigates transmission-path detours in wireless node communication. Finally, the constructed loss function is embedded into a multi-objective genetic algorithm to predict the position of each unknown node in three-dimensional space. Extensive experiments demonstrate that the proposed method achieves state-of-the-art 3D localization performance on both random and multimodal datasets.

Omnidirectional videos (ODVs) present distinct challenges for accurate audio-visual saliency prediction due to their immersive nature, which combines spatial audio with panoramic visuals to enhance the user experience. While auditory cues are crucial for guiding visual attention across the panoramic scene, the interaction between audio and visual stimuli in ODVs remains underexplored. Existing models primarily focus on spatiotemporal visual cues and treat audio signals separately from their spatial and temporal contexts, often leading to misalignments between audio and visual content and undermining temporal consistency across frames. To bridge these gaps, we propose a novel audio-induced saliency prediction model for ODVs that holistically integrates audio and visual inputs through a multi-modal encoder, an audio-visual interaction module, and an audio-visual transformer. Unlike conventional methods that isolate audio cue locations and attributes, our model employs a query-based framework, where learnable audio queries capture comprehensive audio-visual dependencies, thus enhancing saliency prediction by dynamically aligning with audio cues. Besides, we introduce a novel consistency loss to enforce temporal coherence in saliency regions across frames. Extensive experiments demonstrate that our model outperforms state-of-the-art methods in predicting audio-visual salient regions in ODVs, establishing its robustness and superior performance.

Recently, neural image compression (NIC) has made remarkable progress. Two key parts of NIC are the encoder-decoder and the entropy model. For the encoder-decoder, a larger effective receptive field (ERF) means a stronger transformation ability. Existing methods usually enlarge the ERF at the expense of complexity, which is intolerable. To address this issue, we propose a multi-scale depthwise separable dilated convolution (MSDSDC) to build the encoder-decoder. Specifically, we first construct a depthwise separable dilated convolution (DSDC) by using the depthwise separable strategy in dilated convolution to reduce its complexity. Subsequently, multi-scale features extracted by three DSDCs with varying dilation rates are fused to expand the ERF of the encoder-decoder, consequently enhancing its transformation capability. Besides, we design a multi-distribution mixture entropy model (MDMEM) to further enhance the flexibility of latent representation probability modeling. The experimental results demonstrate that our proposed method achieves the best balance between rate-distortion performance and complexity.

Deep unfolding compressive sensing (CS) has experienced remarkable advancements. However, there still exist two challenges: (1) Many algorithms either use uniform block-based sampling, which ignore the fact that the content of different blocks is different, or allocate the sampling rate referring to complete signal before CS sampling, which is not always feasible in real-world scenarios. (2) Traditional CNN is difficult to capture broader contextual priors during iterative recovery. In this paper, we propose a novel network ASMFNet to solve the above two issues. Specifically, to address the first issue, we introduce a dual-branch network featuring a basic sampling branch to acquire reference image and an adaptive sampling branch by median filtering for allocating remaining sampling rate adaptively. For the second problem, we use Swin Transformer and feature fusion block to increase the feature interactions. Experimental results demonstrate that our proposed method outperforms existing methods.

Guided image super-resolution (GISR) aims to reconstruct a high-resolution (HR) target image from its low-resolution (LR) counterpart with the guidance of a HR image from another modality. Existing learning-based methods typically employ symmetric two-stream networks to extract features from both the guidance and target images, and then fuse these features at either an early or late stage through manually designed modules to facilitate joint inference. Despite significant performance, these methods still face several issues: i) the symmetric architectures treat images from different modalities equally, which may overlook the inherent differences between them; ii) lower-level features contain detailed information while higher-level features capture semantic structures. However, determining which layers should be fused and which fusion operations should be selected remain unresolved; iii) most methods achieve performance gains at the cost of increased computational complexity, so balancing the trade-off between computational complexity and model performance remains a critical issue. To address these issues, we propose a Dual-level Cross-modality Neural Architecture Search (DCNAS) framework to automatically design efficient GISR models. Specifically, we propose a dual-level search space that enables the NAS algorithm to identify effective architectures and optimal fusion strategies. Moreover, we propose a supernet training strategy that employs a pairwise ranking loss trained performance predictor to guide the supernet training process. To the best of our knowledge, this is the first attempt to introduce the NAS algorithm into GISR tasks. Extensive experiments demonstrate that the discovered model family, DCNAS-Tiny and DCNAS, achieve significant improvements on several GISR tasks, including guided depth map super-resolution, guided saliency map super-resolution, guided thermal image super-resolution, and pan-sharpening. Furthermore, we analyze the architectures searched by our method and provide some new insights for future research.

As the consumption of multimedia content continues to rise, audio and video have become central to everyday entertainment and social interactions. This growing reliance amplifies the demand for effective and objective audio-visual quality assessment (AVQA) to understand the interaction between audio and visual elements, ultimately enhancing user satisfaction. However, existing state-of-the-art AVQA methods often rely on simplistic machine learning models or fully connected networks for audio-visual signal fusion, which limits their ability to exploit the complementary nature of these modalities. In response to this gap, we propose a novel no-reference AVQA method that utilizes joint cross-attention fusion of audio-visual perception. Our approach begins with a dual-stream feature extraction process that simultaneously captures long-range spatiotemporal visual features and audio features. The fusion model then dynamically adjusts the contributions of features from both modalities, effectively integrating them to provide a more comprehensive perception for quality score prediction. Experimental results on the LIVE-SJTU and UnB-AVC datasets demonstrate that our model outperforms state-of-the-art methods, achieving superior performance in audio-visual quality assessment.

Recently, neural network-based in-loop filters have been rapidly developed, effectively improving the reconstruction quality and compression efficiency in video coding. Existing deep in-loop filters typically employed networks with fixed structures to process all image blocks. However, under various bitrate conditions, compressed image blocks with different textures exhibit varying degradations, which poses a challenge for high-quality and low-complexity filtering. Additionally, different complexity requirements for coding tools in various scenarios limit the versatility of fixed models. To address these problems, a content-aware dynamic in-loop filter (dubbed DILF) with adjustable complexity is proposed in this paper. Specifically, DILF comprises a policy network and a filtering network. For each reconstructed image block, the policy network dynamically generates a filtering network topology based on pixel information and the quantization parameter (QP), guiding the filtering network to skip redundant layers and conduct content-aware image enhancement, thereby improving the filtering performance. In addition, by introducing a user-defined balancing factor into the policy network, the content-aware filtering network topology can be further adjusted according to user’s requirements, facilitating adjustable complexity with a single model. We integrate DILF into Versatile Video Coding (VVC) to replace the built-in deblocking filter. Extensive experiments demonstrate the efficiency of DILF in processing image blocks with varying degrees of degradation and its flexibility in controlling complexity. When the balancing factor is set to 2e-5, DILF achieves bitrate savings of 8.07%, 17.97%, and 20.93% on average for YUV components over VVC reference software VTM-11.0 under all-intra configuration. Compared to static networks with fixed structures, DILF demonstrates superior performance and lower computational complexity.

Omnidirectional images, offering immersive 360° views, have gained significant attention, but assessing their perceptual quality, especially for stereoscopic content, remains a complex challenge. A major limitation lies in the fact that head-mounted devices restrict the viewer’s experience to a single viewport at a time, necessitating a comprehensive understanding of how multiple viewport images interact and aggregate during the viewing process. Moreover, the depth dimension inherent in stereoscopic content further complicates the 360° visual experience, a factor often oversimplified by existing methods, limiting their ability to accurately differentiate perceptual quality across viewports. To address these challenges, we propose a novel no-reference quality assessment model for stereoscopic omnidirectional images. Our approach integrates binocular vision principles within a viewport hypergraph convolutional network framework. First, guided by the unique viewing patterns of stereoscopic omnidirectional images, our model selects panoramic viewports that align with human visual preferences. Next, we devise an image feature extraction network that simulates the binocular fusion and rivalry mechanisms within the human visual system, leveraging a twin encoder-decoder network and tensor decomposition to capture key features. Finally, to assess overall image quality, we introduce a hypergraph structure module that captures complex positional and content-based interactions among sampled viewports through the Graph Influence Network. Extensive experiments on the NBU-SOID, SOLID, and LIVE 3D VR databases demonstrate the superior accuracy and robustness of our model compared to state-of-the-art methods.

Video deblurring is a fundamental problem in low-level vision, and many methods have employed designs based on CNNs and transformers. Traditional CNNs often require deeper architectures to achieve a larger receptive field, which may not be optimal for spatially non-uniform blurs and intense motion blurs. While transformers offer a large receptive field, their quadratic complexity due to attention designs typically imposes a significant computational burden. In addressing these issues, we present an Attentive Large Kernel Network with Mixture of Experts (ALK-MoE). In ALK-MoE, an attentive large kernel backbone network is proposed. On one hand, it inherently extends the network’s receptive field through its large kernel design. On the other hand, it addresses the quadratic complexity of attention by employing a sophisticated attention design, thus maintaining its ability to capture long-range dependencies. Furthermore, to achieve more precise and robust alignment of inter-frame features using optical flow for better utilization of clear frames, a mixture of experts model is proposed. It involves integrating optical flow updates between different experts in a residual manner. Our ablation experiments and experiments on multiple datasets indicate that ALK-MoE achieves comparable or superior performance compared to Transformer-based methods, with lower complexity.

Recently, a large number of image compressive sensing (CS) methods with deep unfolding networks (DUNs) have been proposed. However, existing methods either use fixed-scale blocks for sampling that leads to limited insights into the image content or employ a plain convolutional neural network (CNN) in each iteration that weakens the perception of broader contextual prior. In this paper, we propose a novel DUN (dubbed SVASNet) for image compressive sensing, which achieves scale-variable adaptive sampling and hybrid-attention Transformer reconstruction with a single model. Specifically, for scale-variable sampling, a sampling matrix-based calculator is first employed to evaluate the reconstruction distortion, which only requires measurements without access to the ground truth image. Then, a Block Scale Aggregation (BSA) strategy is presented to compute the reconstruction distortion under block divisions at different scales and select the optimal division scale for sampling. To realize hybrid-attention reconstruction, a dual Cross Attention (CA) submodule in the gradient descent step and a Spatial Attention (SA) submodule in the proximal mapping step are developed. The CA submodule introduces inter-phase inertial forces in the gradient descent, which improves the memory effect between adjacent iterations. The SA submodule integrates local and global prior representations of CNN and Transformer, and explores local and global affinities between dense feature representations. Extensive experimental results show that the proposed SVASNet achieves significant improvements over the state-of-the-art methods.

Training an effective policy on complex goal-reaching tasks with sparse rewards is an open challenge. It is more difficult for the task of reaching remote goals (RRG), as the unavailability of the original rewards and large Wasserstein distance between the distributions of desired goals and initial states make existing methods for common goal-reaching tasks inefficient or even completely ineffective. In this article, we propose progressively learning to reach remote goals by continuously updating boundary goals (PLUB), which solves RRG tasks by reducing the Wasserstein distance between the distributions of boundary goals and desired goals. Specifically, the concept of boundary goal is introduced, which is the set of the closest achieved goals for each desired goal. In addition, to reduce the computational complexity caused by the Wasserstein distance, the closest moving distance is introduced, which is its upper bound, and also the expectation of the distance between the desired goal and the closest boundary goal. By selecting the appropriate intermediate goal from all boundary goals and continuously updating boundary goals, both the closest moving distance and the Wasserstein distance can be reduced. As a result, RRG tasks degenerate into common goal-reaching tasks that can be efficiently solved by a combination of hindsight relabeling and the learning from demonstrations (LfD) method. Extensive experiments on several robotic manipulation tasks demonstrate that PLUB can bring substantial improvements over the existing methods.

Mesh denoising (MD) is a critical task in geometry processing, as meshes from scanning or AIGC techniques are susceptible to noise contamination. The challenge of MD lies in the diverse nature of mesh facets in terms of geometric characteristics and noise distributions. Despite recent advancements in deep learning-based MD methods, existing MD networks typically neglect the consideration of geometric characteristics and noise distributions. In this paper, we propose Hyper-MD, a hyper-network-based approach that addresses this limitation by dynamically customizing denoising parameters for each facet based on its noise intensity and geometric characteristics. Specifically, Hyper-MD is composed of a hyper-network and an MD network. For each noisy facet, the hyper-network takes two angles as input to customize parameters for the MD network. These two angles are specially defined to reveal the noise intensity and geometric characteristics of the current facet, respectively. The MD network receives a facet patch as input, and outputs the denoised normal using the customized parameters. Experimental results on synthetic and real-scanned meshes demonstrate that Hyper-MD outperforms state-of-the-art mesh denoising methods.

Deep network-based image Compressive Sensing (CS) has attracted much attention in recent years. However, there still exist the following two issues: 1) Existing methods typically use fixed-scale sampling, which leads to limited insights into the image content. 2) Most pre-trained models can only handle fixed sampling rates and fixed block scales, which restricts the scalability of the model. In this paper, we propose a novel scale-aware scalable CS network (dubbed S2-CSNet), which achieves scale-aware adaptive sampling, fine granular scalability and high-quality reconstruction with one single model. Specifically, to enhance the scalability of the model, a structural sampling matrix with a predefined order is first designed, which is a universal sampling matrix that can sample multi-scale image blocks with arbitrary sampling rates. Then, based on the universal sampling matrix, a distortion-guided scale-aware scheme is presented to achieve scale-variable adaptive sampling, which predicts the reconstruction distortion at different sampling scales from the measurements and select the optimal division scale for sampling. Furthermore, a multi-scale hierarchical sub-network under a well-defined compact framework is put forward to reconstruct the image. In the multi-scale feature domain of the sub-network, a dual spatial attention is developed to explore the local and global affinities between dense feature representations for deep fusion. Extensive experiments manifest that the proposed S2-CSNet outperforms existing state-of-the-art CS methods.

The emergence of virtual reality technology has made stereoscopic omnidirectional images (SOI) easily accessible and prompted the need to evaluate their perceptual quality. At present, most stereoscopic omnidirectional image quality assessment (SOIQA) methods rely on one of the projection formats, i.e., Equirectangular Projection (ERP) or CubeMap Projection (CMP). However, while ERP provides global information and the less distorted CMP complements it by providing local structural guidance, research on leveraging both ERP and CMP in SOIQA remains limited, hindering a comprehensive understanding of both global and local visual cues. Motivated by this gap, our study introduces a novel dual-stream perception-driven network for blind quality assessment of stereoscopic omnidirectional images. By integrating both ERP and CMP, our method effectively captures both global and local information, marking the first attempt to bridge this gap in SOIQA, particularly through deep learning methodologies. We employ an inter-intra feature fusion module, which considers both the inter-complementarity between ERP and CMP and the intra-relationships within CMP images. This module dynamically and complementarily adjusts the contributions of features from both projections and effectively integrates them to achieve a more comprehensive perception. Besides, deformable convolution is employed to extract the local region of interest, simulating the orientation selectivity of the primary visual cortex. Finally, with the features of left and right views of SOI, a stereo cross attention module that simulates the binocular fusion mechanism is proposed to predict the quality score. Extensive experiments are conducted to evaluate our model and the state-of-the-art competitors, demonstrating that our model has achieved the best performance on the databases of LIVE 3D VR, SOLID, and NBU.

Mesh denoising is a fundamental task in geometry processing, and recent studies have demonstrated the remarkable superiority of deep learning-based methods in this field. However, existing works commonly rely on neural networks without explicit designs for noise and geometry which are actually fundamental factors in mesh denoising. In this paper, by jointly considering noise intensity and geometric characteristics, a novel Filtering Coefficient Learner (FCL for short) for mesh denoising is developed, which delicately generates coefficients to filter face normals. Specifically, FCL produces filtering coefficients consisting of a noise-aware component and a geometry-aware component. The first component is inversely proportional to the noise intensity of each face, resulting in smaller coefficients for faces with stronger noise. For the effective assessment of the noise intensity, a noise intensity estimation module is designed, which predicts the angle between paired noisy-clean normals based on a mean filtering angle. The second component is derived based on two types of geometric features, namely the category feature and face-wise features. The category feature provides a global description of the input patch, while the face-wise features complement the perception of local textures. Extensive experiments have validated the superior performance of FCL over SOTA works in both noise removal and feature preservation.

Given different instructions, large vision-language models (LVLMs) exhibit different degrees of object hallucinations, posing a significant challenge to the evaluation of object hallucinations. Overcoming this challenge, existing object hallucination evaluation methods average the results obtained from a set of instructions. However, these methods fail to provide consistent evaluation across instruction sets that generate image descriptions of significantly different lengths. In this paper, we present the first systematic investigation of the effect of instructions on object hallucinations in LVLMs, with a specific focus on the role played by image description lengths. A valuable finding is that instructions indirectly affect hallucinations through the length of image descriptions. The longer the image description, the higher the object hallucination degree. Accordingly, we fit an informative length-hallucination curve, upon which a fine-grained evaluation framework named LeHaCE is introduced for evaluating object hallucinations at any given image description length. LeHaCE evaluates the object hallucination degree at a uniform image description length to mitigate the effect of description lengths, promoting stability and fairness. Moreover, LeHaCE incorporates the curve slope as an innovative hallucination evaluation metric, reflecting the extent to which the object hallucination degree is affected by the image description length, achieving a more comprehensive evaluation. Experimental results demonstrate that LeHaCE provides a more stable, fair, and comprehensive evaluation of object hallucinations in LVLMs compared to existing methods.

Sensor location awareness is a critical issue in internet of things applications. For more accurate location estimation, the two issues should be considered extensively: 1) how to sufficiently utilize the connection information between multiple nodes and 2) how to select a suitable solution from multiple solutions obtained by the Euclidean distance loss. In this paper, a DV-Hop localization based on the distance estimation using multi-node (DEMN) and the hop loss in WSNs is proposed to address the two issues. In DEMN, when multiple anchor nodes can detect an unknown node, the distance expectation between the unknown node and an anchor node is calculated using the cross domain information and is considered as the expected distance between them, which narrows the search space. When minimizing the traditional Euclidean distance loss, multiple solutions may exist. To select a suitable solution, the hop loss is proposed, which minimizes the difference between the real and its predicted hops. Finally, the Euclidean distance loss calculated by the DEMN and the hop loss are embedded into the multi-objective optimization algorithm. The experimental results show that the proposed method gains 86.11% location accuracy in the randomly distributed network, which is 6.05% better than the DEM-DV-Hop, while DEMN and the hop loss can contribute 2.46% and 3.41%, respectively.

Event cameras, offering high temporal resolutions and high dynamic ranges, have brought a new perspective to address common challenges in monocular depth estimation (e.g., motion blur and low light). However, existing CNN-based methods insufficiently exploit global spatial information from asynchronous events, while RNN-based methods show a limited capacity for effective temporal cues utilization for event-based monocular depth estimation. To this end, we propose a event-based monocular depth estimator with recurrent transformers, namely EReFormer. Technically, we first design a transformer-based encoder-decoder that utilizes multi-scale features to model global spatial information from events. Then, we propose a Gate Recurrent Vision Transformer (GRViT), introducing a recursive mechanism into transformers, to leverage rich temporal cues from events. Finally, we present a Cross Attention-guided Skip Connection (CASC), performing cross attention to fuse multi-scale features, to improve global spatial modeling capabilities. The experimental results show that our EReFormer outperforms state-of-the-art methods by a margin on both synthetic and real-world datasets. Our open-source code is available at https://github.com/liuxu0303/EReFormer.

Intra prediction is a vital tool in video coding that eliminates the spatial redundancy within a frame to enhance compression efficiency. Conventional intra prediction methods employ multiple directional prediction modes to describe textures in local areas. Recently, research on neural network-based intra prediction has achieved great success. The block-context pairs are divided into multiple clusters according to a predefined relationship, and a corresponding network is trained and applied for each cluster. However, the networks in these methods adopt fixed parameters to predict diverse image blocks, making it hard to cope with various textures in natural images. Inspired by recent works on parameter prediction, in this paper, we propose a meta-network-based intra prediction method, called MetaIP, that dynamically customizes the network parameters for each block sample in a given cluster. MetaIP consists of a meta-subnetwork and a prediction subnetwork. For an image block, the meta-subnetwork takes its neighboring reference pixels and some auxiliary information (e.g., quantization parameter) as inputs to generate customized parameters first. Then, the prediction subnetwork uses the customized parameters to infer the predicted block. MetaIP can generate multiple sets of network parameters corresponding to multiple modes for an image block. The optimal mode is determined by the rate-distortion optimization. MetaIP is integrated into VVC to assist or replace the directional prediction modes to evaluate its performance. The experimental results demonstrate that MetaIP with four prediction modes achieves an average of 3.84% and 1.96% bitrate saving for the luma component over VTM-17.0 when assisting or replacing VVC intra modes, respectively.

The sphere-to-plane projection of 360-degree video introduces substantial stretched redundant data, which is discarded when reprojected to the 3D sphere for display. Consequently, encoding and transmitting such redundant data is unnecessary. Highly redundant blocks can be referred to as all-zero blocks (AZBs). Detecting these AZBs in advance can reduce computational and transmission resource consumption. However, this cannot be achieved by existing AZB detection techniques due to the unawareness of the stretching redundancy. In this paper, we first derive a latitude-adaptive redundancy detection (LARD) approach to adaptively detect coefficients carrying redundancy in transformed blocks by modeling the dependency between valid frequency range and the stretching degree based on spectrum analysis. Utilizing LARD, a latitude-redundancy-aware AZB detection scheme tailored for fast 360-degree video coding (LRAS) is proposed to accelerate the encoding process. LRAS consists of three sequential stages: latitude-adaptive AZB (L-AZB) detection, latitude-adaptive genuine-AZB (LG-AZB) detection and latitude-adaptive pseudo-AZB (LP-AZB) detection. Specifically, L-AZB refers to the AZB introduced by projection. LARD is used to detect L-AZB directly. LG-AZB refers to the AZB after hard-decision quantization and zeroing redundant coefficients. A novel latitude-adaptive sum of absolute difference estimation model is built to derive the threshold for LG-AZB detection. LP-AZB refers to the AZB in terms of rate-distortion optimization considering redundancy. A latitude-adaptive rate-distortion model is established for LP-AZB detection. Experimental results show that LRAS can achieve an average total encoding time reduction of 25.85% and 20.38% under low-delay and random access configurations compared to the original HEVC encoder, with only 0.16% and 0.13% BDBR increases and 0.01dB BDPSNR loss, respectively. The transform and quantization time savings are 60.13% and 59.94% on average.

Inspired by certain optimization solvers, the deep unfolding network (DUN) has attracted much attention in recent years for image compressed sensing (CS). However, there still exist the following two issues: 1) In existing DUNs, most hyperparameters are usually content independent, which greatly limits their adaptability for different input contents. 2) In each iteration, a plain convolutional neural network is usually adopted, which weakens the perception of wider context prior and therefore depresses the expressive ability. In this article, inspired by the traditional Proximal Gradient Descent (PGD) algorithm, a novel DUN for image compressed sensing (dubbed DUN-CSNet) is proposed to solve the above two issues. Specifically, for the first issue, a novel content adaptive gradient descent network is proposed, in which a well-designed step size generation sub-network is developed to dynamically allocate the corresponding step sizes for different textures of input image by generating a content-aware step size map, realizing a content-adaptive gradient updating. For the second issue, considering the fact that many similar patches exist in an image but have undergone a deformation, a novel deformation-invariant non-local proximal mapping network is developed, which can adaptively build the long-range dependencies between the nonlocal patches by deformation-invariant non-local modeling, leading to a wider perception on context priors. Extensive experiments manifest that the proposed DUN-CSNet outperforms existing state-of-the-art CS methods by large margins.

Deep learning-based image compressive sensing (CS) methods have achieved great success in the past few years. However, most of them are content-independent, with a spatially uniform sampling rate allocation for the entire image. Such practises may potentially degrade the performance of image CS with block-based sampling, since the content of different blocks in an image is different. In this article, we propose a novel rate-adaptive image CS neural network (dubbed RACSNet) to achieve adaptive sampling rate allocation based on the content characteristics of the image with a single model. Specifically, a measurement domain-based reconstruction distortion is first used to guide the sampling rate allocation for different blocks in an image without access to the ground truth image. Then, a step-wise training strategy is designed to train a reusable sampling matrix, which is capable of sampling image blocks to generate the compressed measurements under arbitrary sampling rates. Subsequently, a pyramid-shaped initial reconstruction sub-network and a hierarchical deep reconstruction sub-network that fuse the measurement information of different scales are put forward to reconstruct image blocks from the compressed measurements. Finally, a reconstruction distortion map and an improved loss function are developed to eliminate the blocking artifacts and further enhance the CS reconstruction. Experimental results on both objective metrics and subjective visual qualities show that the proposed RACSNet achieves significant improvements over the state-of-the-art methods.

Guided filter is a fundamental tool in computer vision and computer graphics, which aims to transfer structure information from the guide image to the target image. Most existing methods construct filter kernels from the guidance itself without considering the mutual dependency between the guidance and the target. However, since there typically exist significantly different edges in two images, simply transferring all structural information from the guide to the target would result in various artifacts. To cope with this problem, we propose an effective framework named deep attentional guided image filtering, the filtering process of which can fully integrate the complementary information contained in both images. Specifically, we propose an attentional kernel learning module to generate dual sets of filter kernels from the guidance and the target and then adaptively combine them by modeling the pixelwise dependency between the two images. Meanwhile, we propose a multiscale guided image filtering module to progressively generate the filtering result with the constructed kernels in a coarse-to-fine manner. Correspondingly, a multiscale fusion strategy is introduced to reuse the intermediate results in the coarse-to-fine process. Extensive experiments show that the proposed framework compares favorably with the state-of-the-art methods in a wide range of guided image filtering applications, such as guided super-resolution (SR), cross-modality restoration, and semantic segmentation. Moreover, our scheme achieved the first place in the real depth map SR challenge held in ACM ICMR 2021. The codes can be found at https://github.com/zhwzhong/DAGF.

Existing image compressed sensing (CS) coding frameworks usually solve an inverse problem based on measurement coding and optimization-based image reconstruction, which still exist the following two challenges: (1) the widely used random sampling matrix, such as the Gaussian Random Matrix (GRM), usually leads to low measurement coding efficiency, and (2) the optimization-based reconstruction methods generally maintain a much higher computational complexity. In this article, we propose a new convolutional neural network based image CS coding framework using local structural sampling (dubbed CSCNet) that includes three functional modules: local structural sampling, measurement coding, and Laplacian pyramid reconstruction. In the proposed framework, instead of GRM, a new local structural sampling matrix is first developed, which is able to enhance the correlation between the measurements through a local perceptual sampling strategy. Besides, the designed local structural sampling matrix can be jointly optimized with the other functional modules during the training process. After sampling, the measurements with high correlations are produced, which are then coded into final bitstreams by the third-party image codec. Last, a Laplacian pyramid reconstruction network is proposed to efficiently recover the target image from the measurement domain to the image domain. Extensive experimental results demonstrate that the proposed scheme outperforms the existing state-of-the-art CS coding methods while maintaining fast computational speed.

Images captured under low-light conditions suffer from several combined degradation factors, including low brightness, low contrast, noise, and color bias. Many learning-based techniques attempt to learn the low-to-clear mapping between low-light and normal-light images. However, they often fall short when applied to low-light images taken in wide-contrast scenes because uneven illumination brings illumination-varying noise and the enhanced images are easily over-saturated in highlight areas. In this article, we present a novel two-stage method to tackle the problem of uneven illumination distribution in low-light images. Under the assumption that noise varies with illumination, we design an illumination-aware transformer network for the first stage of image restoration. In this stage, we introduce the Illumination-aware Attention Block featured with Illumination-aware Multi-head Self-attention, which incorporates different scales of illumination features to guide the attention module, thereby enhancing the denoising and reconstruction capabilities of the restoration network. In the second stage, we innovatively introduce a cubic auto-knee curve transfer with a global parameter predictor to alleviate the over-exposure caused by uneven illumination. We also adopt a white balance correction module to address color bias issues at this stage. Extensive experiments on various benchmarks demonstrate the advantages of our method over state-of-the-art methods qualitatively and quantitatively.

Inspired by certain optimization solvers, the deep unfolding network (DUN) usually inherits a multi-phase structure for image compressive sensing (CS). However, in existing DUNs, the message transmission within and between phases still faces two issues: 1) the roughness of transmitted information, e.g., the low-dimensional representations. 2) the inefficiency of transmitted policy, e.g., simply concatenating deep features. In this paper, by unfolding the Proximal Gradient Descent (PGD) algorithm, a novel gradient guided DUN (G2 -DUN) for image CS is proposed, in which a gradient map is delicately introduced within each phase for providing richer informational guidance at both intra-phase and inter-phase levels. Specifically, corresponding to the gradient descent (GD) of PGD, a gradient guided GD module is designed, in which the gradient map can adaptively guide step size allocation for different textures of input image, realizing a content-aware gradient updating. On the other hand, corresponding to the proximal mapping (PM) of PGD, a gradient guided PM module is developed, in which the gradient map can dynamically guide the exploring of deep textural priors in multi-scale space, achieving the dynamic perception of the proposed deep model. By introducing the gradient map, the proposed message transmission system not only facilitates the informational communication between different functional modules within each phase, but also strengthens the inferential cooperation among cascaded phases. Extensive experiments manifest that the proposed G2 -DUN outperforms existing state-of-the-art CS methods.

Abstract Depth map super resolution has been paid much attention in 3D applications due to the limitation of depth sensors. Few textures in objects with clear contours along them is the most important characteristic of depth map. An efficient image representation should be directional, multiscale and anisotropic. From this we propose a novel multiscale and multidirection depth map super resolution framework with semantic inference to improve the quality of depth maps. In this framework, a multiscale and multidirection depth map contour fusion scheme captures and assembles intrinsic geometrical structures through a multiview non‐subsampled contourlet transform manner. This scheme not only isolates the discontinuities of contours but retains the smoothness along the contours. The semantic inference is also utilized to segment and label the depth map into objects/backgrounds‐level which are coplanar. Furthermore, a semantic‐aware label refinement strategy is introduced to correct the rarely inaccurate labels of the label map for upscaling the target pixel with pixels in the same object or background. Experimental results on benchmark depth map dataset demonstrate that the proposed multiscale and multidirection depth map super resolution framework with semantic inference has a significant improvement than the state‐of‐the‐art algorithms both visually and quantitatively.

The acquisition of point clouds is usually accompanied by noise due to imperfect laser scanning or image-based reconstruction techniques. Deep learning-based methods have achieved impressive performance in point cloud denoising. However, the features captured by a denoising network from noisy point clouds are usually contaminated by noise during training. The feature noise will lead to the oscillation of back-propagated gradients, which interferes with parameter optimization and reduces the denoising performance. In this paper, we propose to explicitly clean up feature noise for point cloud denoising from two aspects: feature noise cleaning and network training. From the first aspect, we propose the feature clean network (FCNet for short) to explicitly clean up the feature noise. From the second aspect, we train FCNet by a teacher-student learning model to learn the noise-free features under the guidance of feature domain losses. Specifically, FCNet is designed with emphasis on two modules: non-local self-similarity (NSS) and weighted average pooling (WAP). NSS module smooths features through a non-local filter based on the inherent non-local self-similarity of point clouds. WAP module applies original weights calculated by the statistical outlier removal algorithm to suppress the feature noise induced by outliers. In the teacher-student learning model, we introduce a clean input using the noisy point and its clean neighbors. The teacher network accepts the clean input to capture noise-free features. The student network is trained to imitate the teacher network to learn noise-free features by minimizing the feature loss. The experiments on synthetic and real scanned point clouds show that FCNet outperforms state-of-the-art point cloud denoising methods.

Point clouds obtained by 3D scanning or reconstruction are usually accompanied by noise. Filtering-based point cloud denoising methods are simple and effective, but they are limited by the manually defined coefficients. Deep learning has shown excellent ability in automatically learning parameters. In this paper, a filtering network named PointFilterNet (PFN for short) is proposed to denoise point clouds by the combination of filtering and deep learning. Instead of directly outputting denoised points using networks, PFN generates the filtering denoised points through learned coefficients. Specifically, PFN outputs three coefficient vectors. These coefficient vectors are then used to filter the coordinates of points in the neighborhood of the noisy point. PFN consists of two parts: an outlier recognizer and a denoiser, both of which generate different but indispensable filtering coefficients. The outlier recognizer reduces the interference of outliers by assigning small coefficients to them. The denoiser is designed to progressively denoise point clouds which accords to the perception process of the human visual system. The experiments on synthetic and real scanned point clouds demonstrate that PFN outperforms state-of-the-art point cloud denoising works. Compared with DMR, the Chamfer distance of PFN is reduced by 22.07% on PCN-dataset.

Intra prediction is a crucial part of video compression, which utilizes local information in images to eliminate spatial redundancy. As the state-of-the-art video coding standard, Versatile Video Coding (H.266/VVC) employs multiple directional prediction modes in intra prediction to find the texture trend of local areas. Then the prediction is made based on reference samples in the selected direction. Recently, neural network-based intra prediction has achieved great success. Deep network models are trained and applied to assist the HEVC and VVC intra modes. In this paper, we propose a novel tree-structured data clustering-driven neural network (dubbed TreeNet) for intra prediction, which builds the networks and clusters the training data in a tree-structured manner. Specifically, in each network split and training process of TreeNet, every parent network on a leaf node is split into two child networks by adding or subtracting Gaussian random noise. Then data clustering-driven training is applied to train the two derived child networks using the clustered training data of their parent. On the one hand, the networks at the same level in TreeNet are trained with non-overlapping clustered datasets, and thus they can learn different prediction abilities. On the other hand, the networks at different levels are trained with hierarchically clustered datasets, and thus they will have different generalization abilities. TreeNet is integrated into VVC to assist or replace intra prediction modes to test its performance. In addition, a fast termination strategy is proposed to accelerate the search of TreeNet. The experimental results demonstrate that when TreeNet is used to assist the VVC Intra modes, TreeNet with depth = 3 can bring an average of 3.78% bitrate saving (up to 8.12%) over VTM-17.0. If TreeNet with the same depth replaces all VVC intra modes, an average of 1.59% bitrate saving can be reached.

JPEG, which was developed 30 years ago, is the most widely used image coding format, especially favored by the resource-deficient devices, due to its simplicity and efficiency. With the evolution of the Internet and the popularity of mobile devices, a huge amount of user-generated JPEG images are uploaded to social media sites like Facebook and Flickr or stored in personal computers or notebooks, which leads to an increase in storage cost. However, the performance of JPEG is far from the-state-of-the art coding methods. Therefore, the lossless recompression of JPEG images is urgent to be studied, which will further reduce the storage cost while maintaining the image fidelity. In this paper, a hybrid coding framework for the lossless recompression of JPEG images (LLJPEG) using transform domain intra prediction is proposed, including block partition and intraprediction, transform and quantization, and entropy coding. Specifically, in LLJPEG, intra prediction is first used to obtain a predicted block. Then the predicted block is transformed by DCT and then quantized to obtain the predicted coefficients. After that, the predicted coefficients are subtracted from the original coefficients to get the DCT coefficient residuals. Finally, the DCT residuals are entropy coded. In LLJPEG, some new coding tools are proposed for intra prediction and the entropy coding is redesigned. The experiments show that LLJPEG can reduce the storage space by 29.43% and 26.40% on the Kodak and DIV2K datasets respectively without any loss for JPEG images, while maintaining low decoding complexity.

Deep network-based image Compressed Sensing (CS) has attracted much attention in recent years. However, the existing deep network-based CS schemes either reconstruct the target image in a block-by-block manner that leads to serious block artifacts or train the deep network as a black box that brings about limited insights of image prior knowledge. In this paper, a novel image CS framework using non-local neural network (NL-CSNet) is proposed, which utilizes the non-local self-similarity priors with deep network to improve the reconstruction quality. In the proposed NL-CSNet, two non-local subnetworks are constructed for utilizing the non-local self-similarity priors in the measurement domain and the multi-scale feature domain respectively. Specifically, in the subnetwork of measurement domain, the long-distance dependencies between the measurements of different image blocks are established for better initial reconstruction. Analogically, in the subnetwork of multi-scale feature domain, the affinities between the dense feature representations are explored in the multi-scale space for deep reconstruction. Furthermore, a novel loss function is developed to enhance the coupling between the non-local representations, which also enables an end-to-end training of NL-CSNet. Extensive experiments manifest that NL-CSNet outperforms existing state-of-the-art CS methods, while maintaining fast computational speed.

3D meshes are widely employed to represent geometry structure of 3D shapes. Due to limitation of scanning sensor precision and other issues, meshes are inevitably affected by noise, which hampers the subsequent applications. Convolultional neural networks (CNNs) achieve great success in image processing tasks, including 2D image denoising, and have been proven to own the capacity of modeling complex features at different scales, which is also particularly useful for mesh denoising. However, due to the nature of irregular structure, CNNs-based denosing strategies cannot be trivially applied for meshes. To circumvent this limitation, in the paper, we propose the local surface descriptor (LSD), which is able to transform the local deformable surface around a face into 2D grid representation and thus facilitates the deployment of CNNs to generate denoised face normals. To verify the superiority of LSD, we directly feed LSD into the classical Resnet without any complicated network design. The extensive experimental results show that, compared to the state-of-the-arts, our method achieves encouraging performance with respect to both objective and subjective evaluations.

Source camera identification (SCI) technology has attracted increasing attentions over the past few years. However, the existing methods suppress image content with denoising filters that are largely agnostic to the specific sensor pattern noise (SPN) signal of interest. Such practices may potentially degrade the performance of SPN-based SCI due to un-reliable SPNs, especially when forensic images are transmitted through social networking platforms. In this paper, we address the problem of SPN-based device identification and propose a multi-scale feature fusion network (MSFFN) to boost the sensor-based source camera identification attribution. Specifically, several image patches of different scales are selected and input into the MSFFN to extract the SPN. The MSFFN is a multi-scale encoder-decoder structure, which is used to suppress image content and improve source attribution. Subsequently, the content-independent SPN features of different scales are fused. At last, the fused features are used for image source identification. Experimental results compared with the state-of-the-art demonstrate that the proposed scheme achieves significant improvements, especially in the accuracy of social networking image source identification.

By integrating certain optimization solvers with deep neural network, deep unfolding network (DUN) has attracted much attention in recent years for image compressed sensing (CS). However, there still exist several issues in existing DUNs: 1) For each iteration, a simple stacked convolutional network is usually adopted, which apparently limits the expressiveness of these models. 2) Once the training is completed, most hyperparameters of existing DUNs are fixed for any input content, which significantly weakens their adaptability. In this paper, by unfolding the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA), a novel fast hierarchical DUN, dubbed FHDUN, is proposed for image compressed sensing, in which a well-designed hierarchical unfolding architecture is developed to cooperatively explore richer contextual prior information in multi-scale spaces. To further enhance the adaptability, series of hyperparametric generation networks are developed in our framework to dynamically produce the corresponding optimal hyperparameters according to the input content. Furthermore, due to the accelerated policy in FISTA, the newly embedded acceleration module makes the proposed FHDUN save more than 50% of the iterative loops against recent DUNs. Extensive CS experiments manifest that the proposed FHDUN outperforms existing state-of-the-art CS methods, while maintaining fewer iterations.

Low-light enhancement aims to recover a high contrast normal light image from a low-light image with bad exposure and low contrast. Inspired by curve adjustment in photo editing software and Chebyshev approximation, this paper presents a novel model for brightening low-light images. The proposed model, ChebyLighter, learns to estimate pixel-wise adjustment curves for a low-light image recurrently to reconstruct an enhanced output. In ChebyLighter, Chebyshev image series are first generated. Then pixel-wise coefficient matrices are estimated with Triple Coefficient Estimation (TCE) modules and the final enhanced image is recurrently reconstructed by Chebyshev Attention Weighted Summation (CAWS). The TCE module is specifically designed based on dual attention mechanism with three necessary inputs. Our method can achieve ideal performance because adjustment curves can be obtained with numerical approximation by our model. With extensive quantitative and qualitative experiments on diverse test images, we demonstrate that the proposed method performs favorably against state-of-the-art low-light image enhancement algorithms.

Inter prediction is a crucial part of hybrid video coding frameworks, utilized to exploit the temporal redundancy in video sequences and improve the coding performance. During inter prediction, a predicted block is typically derived from reference pictures using motion estimation and motion compensation. To improve the coding performance of inter prediction, a neural network based enhancement to inter prediction (NNIP) is proposed in this paper. NNIP is composed of three networks, namely residue estimation network, combination network, and deep refinement network. Specifically, first, a residue estimation network is designed to estimate the residue between current block and its predicted block using their available spatial neighbors. Second, the feature maps of the estimated residue and the predicted block are extracted and concatenated in a combination network. Finally, the concatenated feature maps are fed into a deep refinement network to generate a refined residue, which is added back to the predicted block to derive a more accurate predicted block. NNIP is integrated in HEVC to evaluate its efficiency. The experimental results demonstrate that NNIP can achieve 4.6%, 3.0%, and 2.7% BD-rate reduction on average under LDP, LDB, and RA configurations compared to HEVC.

Depth map records distance between the viewpoint and objects in the scene, which plays a critical role in many real-world applications. However, depth map captured by consumer-grade RGB-D cameras suffers from low spatial resolution. Guided depth map super-resolution (DSR) is a popular approach to address this problem, which attempts to restore a high-resolution (HR) depth map from the input low-resolution (LR) depth and its coupled HR RGB image that serves as the guidance. The most challenging issue for guided DSR is how to correctly select consistent structures and propagate them, and properly handle inconsistent ones. In this paper, we propose a novel attention-based hierarchical multi-modal fusion (AHMF) network for guided DSR. Specifically, to effectively extract and combine relevant information from LR depth and HR guidance, we propose a multi-modal attention based fusion (MMAF) strategy for hierarchical convolutional layers, including a feature enhancement block to select valuable features and a feature recalibration block to unify the similarity metrics of modalities with different appearance characteristics. Furthermore, we propose a bi-directional hierarchical feature collaboration (BHFC) module to fully leverage low-level spatial information and high-level structure information among multi-scale features. Experimental results show that our approach outperforms state-of-the-art methods in terms of reconstruction accuracy, running speed and memory efficiency.

Video watermarking technology has attracted increasing attention in the past few years, and a great deal of traditional and deep learning-based methods have been proposed. However, these existing methods usually suffer from the following two challenges: First, most algorithms cannot resist camcorder recording attack, which limits their practical application. Second, watermark embedding may cause substantial degradation of video quality. Through analyzing the unique distortions presented in the camcorder recording process, including geometric distortion, temporal sampling distortion, sensor distortion and processing distortion, this paper proposes a novel spatio-temporal context based adaptive camcorder recording watermarking scheme STACR. In STACR, considering the geometric distortion and video visual quality, we embed the watermark by constructing a spatio-temporal histogram and incorporate a content features based adaptive locating algorithm to select embedding blocks and embedding strengths. As for the temporal sampling attack, we put forward a watermark correlation-based synchronization algorithm and combine it with cross-validation. Moreover, to resist the sensor distortion, we design a local matching-based algorithm to improve the extraction accuracy. In addition, grouped and repeated embedding strategies are combined to cope with the processing distortion. Experimental results compared with the state-of-the-art show that the proposed scheme achieves high video quality and is robust to geometric attacks, compression, scaling, transcoding, recoding, frame rate changes and especially for camcorder recording.

Watermarking technology has attracted increasing attentions in the past few years, and a great deal traditional and deep learning-based methods have been proposed. However, these methods usually suffer from the following three challenges: First, the current algorithms are designed separately for images or videos, and there is no universal solution. Second, most algorithms cannot resist screen recording, which limits its application. Third, some algorithms can only embed fixed-length watermarks and cannot handle the embedding capacity flexibly. In this paper, a novel watermarking scheme is proposed based on spatial and temporal histograms, in which two types of histogram watermarking are designed: One is constructed in the spatial domain, using the low-frequency characteristics of the image to change the shape of the histogram and embed the watermark. The other is established in the time domain, which uses the similarity of adjacent frames and combines texture features to modify the shape of the temporal histogram to embed the watermark. Experimental results compared with the state-of-the-art demonstrate that the proposed scheme achieves superior performance.

Recently, a few image compressed sensing (CS) methods based on deep learning have been developed, which achieve remarkable reconstruction quality with low computational complexity. However, these existing deep learning-based image CS methods focus on exploring intraframe correlation while ignoring interframe cues, resulting in inefficiency when directly applied to video CS. In this paper, we propose a novel video CS framework based on a convolutional neural network (dubbed VCSNet) to explore both intraframe and interframe correlations. Specifically, VCSNet divides the video sequence into multiple groups of pictures (GOPs), of which the first frame is a keyframe that is sampled at a higher sampling ratio than the other nonkeyframes. In a GOP, the block-based framewise sampling by a convolution layer is proposed, which leads to the sampling matrix being automatically optimized. In the reconstruction process, the framewise initial reconstruction by using a linear convolutional neural network is first presented, which effectively utilizes the intraframe correlation. Then, the deep reconstruction with multilevel feature compensation is proposed, which compensates the nonkeyframes with the keyframe in a multilevel feature compensation manner. Such multilevel feature compensation allows the network to better explore both intraframe and interframe correlations. Extensive experiments on six benchmark videos show that VCSNet provides better performance over state-of-the-art video CS methods and deep learning-based image CS methods in both objective and subjective reconstruction quality.

Mesh denoising is a critical technology in geometry processing that aims to recover high-fidelity 3D mesh models of objects from noise-corrupted versions. In this work, we propose a learning-based mesh normal denoising scheme, calledNormalNet, which employs deep networks to find the correlation between the volumetric representation and denoised face normal. Overall,NormalNetfollows the iterative framework of filtering-based mesh denoising. During each iteration, firstly, a local partition normalization strategy is applied to split the local structure around each face into dense voxels, in which both the structure and face normal information can be preserved during this transformation. Benefiting from the thorough information preservation, we can use simple residual networks, which employ the volumetric representation as the input and produce the learned denoised face normal, to achieve satisfactory results. Finally, the vertex positions are updated according to the denoised normals. Besides introducing normalization into mesh denoising, our main contributions include a classification-based training faces selection strategy for balancing the training set and a mismatched-faces rejection strategy for removing the mismatched faces between noisy mesh and ground truth. Compared to state-of-the-art works,NormalNetcan effectively remove noise while preserving the original features and avoiding pseudo-features.

Distinguishing between geometric features and noise is of paramount importance for mesh denoising. In this paper, a graph-based feature-preserving mesh normal filtering scheme is proposed, which includes two stages: graph-based feature detection and feature-aware guided normal filtering. In the first stage, faces in the input noisy mesh are represented by patches, which are then modelled as weighted graphs. In this way, feature detection can be cast as a graph-cut problem. Subsequently, an iterative normalized cut algorithm is applied on each patch to separate the patch into smooth regions according to the detected features. In the second stage, a feature-aware guidance normal is constructed for each face, and guided normal filtering is applied to achieve robust feature-preserving mesh denoising. The results of experiments on synthetic and real scanned models indicate that the proposed scheme outperforms state-of-the-art mesh denoising works in terms of both objective and subjective evaluations.

The widespread application of audio communication technologies has speeded up audio data flowing across the Internet, which made it a popular carrier for covert communication. In this paper, we present a cross-modal steganography method for hiding image content into audio carriers while preserving the perceptual fidelity of the cover audio. In our framework, two multi-stage networks are designed: the first network encodes the decreasing multilevel residual errors inside different audio subsequences with the corresponding stage sub-networks, while the second network decodes the residual errors from the modified carrier with the corresponding stage sub-networks to produce the final revealed results. The multi-stage design of proposed framework not only make the controlling of payload capacity more flexible, but also make hiding easier because of the gradual sparse characteristic of residual errors. Qualitative experiments suggest that modifications to the carrier are unnoticeable by human listeners and that the decoded images are highly intelligible.

The airborne laser scanning (ALS) point cloud has drawn increasing attention thanks to its capability to quickly acquire large-scale and high-precision ground information. Due to the complexity of observed scenes and the irregularity of point distribution, the semantic labeling of ALS point clouds is extremely challenging. In this paper, we introduce an efficient discretization based framework according to the geometric character of ALS point clouds, and propose an original intraclass weighted cross entropy loss function to solve the problem of data imbalance. We evaluate our framework on the ISPRS (International Society for Photogrammetry and Remote Sensing) 3D Semantic Labeling dataset. The experimental results show that the proposed method has achieved a new state-of-the-art performance in terms of overall accuracy (85.3%) and average F1 score (74.1%).

Most existing blind image quality assessment (BIQA) methods belong to supervised methods, which always need a large number of image samples and expensive subjective scores for training a quality prediction model. In this paper, we focus our attention on the unsupervised BIQA methods and put forward a novel unsupervised approach. The main idea of our method is to quantify the image quality degradation through measuring the structure, naturalness, and the perception quality variations of the distorted image from the pristine natural images. In specific, the structure variation is captured by the deviations of the image phase congruency and gradients distributions. The naturalness variation is characterized through the distributions variations of the locally mean subtracted and contrast normalized (MSCN) coefficients and the products of pairs of the adjacent MSCN coefficients. Compared with existing unsupervised methods, we initiatively introduce the perception quality measurement into the construction of unsupervised BIQA method, which is conducted by characterizing the prediction discrepancy between the image and its brain prediction based on the free-energy principle in the newly revealed brain theory. After feature extraction, we learn a pristine multivariate Gaussian (MVG) model with the extracted features from a set of pristine natural images. The quality of a new image is finally defined as the distance between its MVG model and the learned pristine MVG model. The extensive experiments conducted on LIVE, TID2013, CSIQ, Toyama, CID2013, and the Waterloo Exploration databases demonstrate that the proposed method achieves comparative prediction performance with the state-of-the-art BIQA methods.

In both H.264/AVC and HEVC, the angular prediction is adopted for intra coding, which only exploits the spatial correlation between the current block and its neighboring single line reference. This angular prediction can handle the main directional patterns well, however, lacks the ability to deal with other directions. In this paper, a multi-scale convolutional neural network based intra prediction is proposed to address this problem. Specifically, a predicted block is first generated by the angular prediction, then fed into the proposed network with its neighboring reconstructed L-shape to generate a more accurate predicted block. On one hand, the L-shape of multiple lines provides more reliable reconstructed pixels and more contextual information to get better prediction; on the other hand, the multi-scale feature extraction takes the advantage of the feature maps in different scales to further enhance the prediction. With this multi-scale structure, the L-shape can be used to refine both left-above and right-bottom pixels in the predicted block during the convolution operation. Experimental results demonstrate that compared with HEVC reference software HM 16.9, the proposed intra prediction can achieve an average of 3.4% (up to 5.6%) bitrate saving with all intra configuration.

In the study of compressed sensing (CS), the two main challenges are the design of sampling matrix and the development of reconstruction method. On the one hand, the usually used random sampling matrices (e.g. GRM) are signal independent, which ignore the characteristics of the signal. On the other hand, the state-of-the-art image CS methods (e.g. GSR and MH) achieve quite good performance, however with much higher computational complexity. To deal with the two challenges, we propose an image CS framework using convolutional neural network (dubbed CSNet) that includes a sampling network and a reconstruction network, which are optimized jointly. The sampling network adaptively learns the sampling matrix from the training images, which makes the CS measurements retain more image structural information for better reconstruction. Specifically, three types of sampling matrices are learned, i.e. floating-point matrix, {0,1}-binary matrix, and {-1,+1}-bipolar matrix. The last two matrices are specially designed for easy storage and hardware implementation. The reconstruction network, which contains a linear initial reconstruction network and a non-linear deep reconstruction network, learns an end-to-end mapping between the CS measurements and the reconstructed images. Experimental results demonstrate that CSNet offers state-of-the-art reconstruction quality, while achieving fast running speed. In addition, CSNet with {0,1}-binary matrix, and {-1,+1}-bipolar matrix gets comparable performance with the existing deep learning based CS methods, and outperforms the traditional CS methods. What's more, the experimental results further suggest that the learned sampling matrices can improve the traditional image CS reconstruction methods significantly.

Image paragraph generation aims to produce a complete description of a given image. This task is more challenging than image captioning, which only generates one sentence to describe the entire image. Traditional paragraph generation methods usually produce paragraph descriptions based on individual regions that are detected by a Region Proposal Network (RPN). However, relationships among visual objects are either ignored or utilized in an implicit manner in previous work. In this paper, we attempt to explore more visual information through a novel paragraph generation network that explicitly incorporates visual relationship semantics when producing descriptions. First, a novel Relation Pair Generative Adversarial Network (RP-GAN) is designed to locate regions that may cover subjective or objective elements. Then, their relationships are inferred through an attention-based network. Finally, the visual features and relationship semantics of valid relation pairs are taken as inputs by a Long Short-Term Memory (LSTM) network for generating sentences. The experimental results show that by explicitly utilizing the predicted relationship information, our proposed method obtains more accurate and informative paragraph descriptions than previous methods.

An ideal quality assessment model should simulate the properties of the visual brain to be consistent with human evaluation. The visual brain appears to have both evolved to seek an efficient, decorrelated representation of image information and to “match” the statistics of the natural image. On one hand, the theoretical studies suggest that sparse representation resembles the strategy in the primary visual cortex of brain for representing natural images. On the other hand, the natural scene statistics have driven the evolution of human visual system and have also inspired the understanding and simulating of visual perception. Inspired by these observations, in this paper, we propose a novel reduced-reference stereoscopic image quality assessment metric using sparse representation and natural scene statistics to simulate the visual perception of the brain. Specifically, the distribution statistics of the classified visual primitives extracted by sparse representation are used to measure the visual information, which is closely related to the hierarchical progressive process of human visual perception. Particularly, the mutual information of classified primitives between two view images is derived as a binocular cue to simulate the binocular fusion process. The maximum mechanism that is applied to select the visual information is a pooling mechanism with which complex cells use the maximal stimuli from a group of simple cells during the transfer process in the primary visual cortex. The natural scene statistics of locally normalized luminance coefficients are used to evaluate the natural losses due to the presence of distortions. The differences of the visual information and the natural scene statistics between the original and distorted images are used to compute the quality score by a prediction function which is trained using support vector regression. Experimental results show that the proposed metric outperforms the state-of-the-art stereoscopic image quality assessment metrics on LIVE 3D IQA database and NBU-MDSID Phase-II database, and delivers competitive performance on Waterloo IVC 3D database.

Recently, deep learning based image Compressed Sensing (CS) methods have been proposed and demonstrated superior reconstruction quality with low computational complexity. However, the existing deep learning based image CS methods need to train different models for different sampling ratios, which increases the complexity of the encoder and decoder. In this paper, we propose a scalable convolutional neural network (dubbed SCSNet) to achieve scalable sampling and scalable reconstruction with only one model. Specifically, SCSNet provides both coarse and fine granular scalability. For coarse granular scalability, SCSNet is designed as a single sampling matrix plus a hierarchical reconstruction network that contains a base layer plus multiple enhancement layers. The base layer provides the basic reconstruction quality, while the enhancement layers reference the lower reconstruction layers and gradually improve the reconstruction quality. For fine granular scalability, SCSNet achieves sampling and reconstruction at any sampling ratio by using a greedy method to select the measurement bases. Compared with the existing deep learning based image CS methods, SCSNet achieves scalable sampling and quality scalable reconstruction at any sampling ratio with only one model. Experimental results demonstrate that SCSNet has the state-of-the-art performance while maintaining a comparable running speed with the existing deep learning based image CS methods.

To tackle three main problems in depth map super resolution (SR) process, which are texture copy artifacts, blurred edge artifacts and jagged edge artifacts, we propose a depth map super resolution with denoising method based on multiscale directional fusion via nonsubsampled contourlet transform (NCST). We first transform low resolution depth maps of multiple views via NSCT. Then NSCT coefficients are denoised by a BayesShrink threshold in nonsubsampled directional filter banks (NSDFB) domain and fused by the max coefficient absolute value (mCAV) rule respectively within each scale and direction. Finally the fused coefficients are synthesized and upscaled to a high resolution depth map utilizing a modified edge-guided joint bilateral filter. Experimental results demonstrate that our method significantly outperforms the state-of-the-art super resolution algorithms quantitatively and visually while mitigating the corrupted noise.

G-Cast is a wireless visual communication scheme that conveys visual information via image gradient. It is inspired by the characteristics of human vision systems and can provide improved perceptual quality. G-Cast is power efficient but bandwidth demanding, because gradient data have double the size of the original image. This paper presents a scheme named CG-Cast for scalable image transmission in bandwidth-limited wireless scenarios. It employs a compressive-gradient-based image representation to describe perceptually sensitive image details and reduce the bandwidth requirement at the same time, combining gradient-based visual representation with compressive sensing techniques. The compressive gradient data are transmitted in a pseudo-analog way so that it achieves elegant quality transition in a wide channel signal-to-noise ratio (CSNR) range. CG-Cast also sends a small set of low-frequency data in digital a way to provide the global and local luminance of the image. We developed an effective optimization algorithm for the decoder to reconstruct the original image from the received noisy compressive gradient and the low-frequency part of the image. Experimental results demonstrate that the proposed scheme improves the quality of received images remarkably under different CSNR and channel bandwidth conditions.

JPEG images captured in poor lighting conditions suffer from both low luminance contrast and coarse quantization artifacts due to lossy compression. Performing dequantization and contrast enhancement in separate back-to-back steps would amplify the residual compression artifacts, resulting in low visual quality. Leveraging on recent development in graph signal processing (GSP), we propose to jointly dequantize and contrast-enhance such images in a single graph-signal restoration framework. Specifically, we separate each observed pixel patch into illumination and reflectance via Retinex theory, where we define generalized smoothness prior and signed graph smoothness prior according to their respective unique signal characteristics. Given only a transform-coded image patch, we compute robust edge weights for each graph via low-pass filtering in the dual graph domain. We compute the illumination and reflectance components for each patch alternately, adopting accelerated proximal gradient (APG) algorithms in the transform domain, with backtracking line search for further speedup. Experimental results show that our generated images outperform the state-of-the-art schemes noticeably in the subjective quality evaluation.

In this paper, we propose a novel depth restoration algorithm from RGB-D data through combining characteristics of local and non-local manifolds, which provide low-dimensional parameterizations of the local and non-local geometry of depth maps. Specifically, on the one hand, a local manifold model is defined to favor local neighboring relationship of pixels in depth, according to which, manifold regularization is introduced to promote smoothing along the manifold structure. On the other hand, the non-local characteristics of the patch-based manifold can be used to build highly data-adaptive orthogonal bases to extract elongated image patterns, accounting for self-similar structures in the manifold. We further define a manifold thresholding operator in 3D adaptive orthogonal spectral bases-eigenvectors of the discrete Laplacian of local and non-local manifolds-to retain only low graph frequencies for depth maps restoration. Finally, we propose a unified alternating direction method of multipliers optimization framework, which elegantly casts the adaptive manifold regularization and thresholding jointly to regularize the inverse problem of depth maps recovery. Experimental results demonstrate that our method achieves superior performance compared with the state-of-the-art works with respect to both objective and subjective quality evaluations.

Depth information is being widely used in many real-world applications. However, due to the limitation of depth sensing technology, the captured depth map in practice usually has much lower resolution than that of color image counterpart. In this paper, we propose to combine the internal smoothness prior and external gradient consistency constraint in graph domain for depth super-resolution. On one hand, a new graph Laplacian regularizer is proposed to preserve the inherent piecewise smooth characteristic of depth, which has desirable filtering properties. A specific weight matrix of the respect graph is defined to make full use of information of both depth and the corresponding guidance image. On the other hand, inspired by an observation that the gradient of depth is small except at edge separating regions, we introduce a graph gradient consistency constraint to enforce that the graph gradient of depth is close to the thresholded gradient of guidance. We reinterpret the gradient thresholding model as variational optimization with sparsity constraint. In this way, we remedy the problem of structure discrepancy between depth and guidance. Finally, the internal and external regularizations are casted into a unified optimization framework, which can be efficiently addressed by ADMM. Experimental results demonstrate that our method outperforms the state-of-the-art with respect to both objective and subjective quality evaluations.

Camera images in reality are easily affected by various distortions, such as blur, noise, blockiness, and the like, which damage the quality of images. The complexity of distortions in camera images raises significant challenge for precisely predicting their perceptual quality. In this paper, we present an image quality assessment (IQA) approach that aims to solve this challenging problem to some extent. In the proposed method, we first extract the low-level and high-level statistical features, which can capture the quality degradations effectively. On the one hand, the first kind of statistical features are extracted from the locally mean subtracted and contrast normalized coefficients, which denote the low-level features in the early human vision. On the other hand, the recently proposed brain theory and neuroscience, especially the free-energy principle, reveal that the human brain tries to explain its encountered visual scenes through an inner creative model, with which the brain can produce the projection for the image. Then, the quality of perceptions can be reflected by the divergence between the image and its brain projection. Based on this, we extract the second type of features from the brain perception mechanism, which represent the high-level features. The low-level and high-level statistical features can play a complementary role in quality prediction. After feature extraction, we design a neural network to integrate all the features and convert them to the final quality score. Extensive tests performed on two real camera image datasets prove the validity of our method and its advantageous predicting ability over the competitive IQA models.

Lossy image compression is generally formulated as a joint rate-distortion optimization problem to learn encoder, quantizer, and decoder. Due to the non-differentiable quantizer and discrete entropy estimation, it is very challenging to develop a convolutional network (CNN)-based image compression system. In this paper, motivated by that the local information content is spatially variant in an image, we suggest that: (i) the bit rate of the different parts of the image is adapted to local content, and (ii) the content-aware bit rate is allocated under the guidance of a content-weighted importance map. The sum of the importance map can thus serve as a continuous alternative of discrete entropy estimation to control compression rate. The binarizer is adopted to quantize the output of encoder and a proxy function is introduced for approximating binary operation in backward propagation to make it differentiable. The encoder, decoder, binarizer and importance map can be jointly optimized in an end-to-end manner. And a convolutional entropy encoder is further presented for lossless compression of importance map and binary codes. In low bit rate image compression, experiments show that our system significantly outperforms JPEG and JPEG 2000 by structural similarity (SSIM) index, and can produce the much better visual result with sharp edges, rich textures, and fewer artifacts.

The compressed sensing (CS) has been successfully applied to image compression in the past few years as most image signals are sparse in a certain domain. Several CS reconstruction models have been proposed and obtained superior performance. However, these methods suffer from blocking artifacts or ringing effects at low sampling ratios in most cases. To address this problem, we propose a deep convolutional Laplacian Pyramid Compressed Sensing Network (LapC-SNet) for CS, which consists of a sampling sub-network and a reconstruction sub-network. In the sampling sub-network, we utilize a convolutional layer to mimic the sampling operator. In contrast to the fixed sampling matrices used in traditional CS methods, the filters used in our convolutional layer are jointly optimized with the reconstruction sub-network. In the reconstruction sub-network, two branches are designed to reconstruct multi -scale residual images and muti -scale target images progressively using a Laplacian pyramid architecture. The proposed LapCSNet not only integrates multi-scale information to achieve better performance but also reduces computational cost dramatically. Experimental results on benchmark datasets demonstrate that the proposed method is capable of reconstructing more details and sharper edges against the state-of-the-arts methods.

Depth information is being widely used in many real-world tasks, such as 3DTV, 3D scene reconstruction, multi-view rendering, etc. However, the captured depth maps in practice usually suffer from quality degradations, including low-resolution and noise corruption, which limit their further applications. Noise-aware super-resolution of depth maps is a challenging task and has received increasingly more attention in recent years. In this paper, we propose a novel method based on the plug-and-play scheme, which casts two powerful graph-based tools-the graph Laplacian regularizer and 3D graph Fourier transform-into a unified ADMM optimization framework. It can be performed in an iterative manner with easily treatable convex optimization sub-problems. Experiments results demonstrate that our method achieves superior performance compared with the state-of-the-art works with respect to both objective and subjective quality evaluations.

Gaussian random matrix (GRM) has been widely used to generate linear measurements in compressed sensing (CS) of natural images. However, there actually exist two disadvantages with GRM in practice. One is that GRM has large memory requirement and high computational complexity, which restrict the applications of CS. Another is that the CS measurements randomly obtained by GRM cannot provide sufficient reconstruction performances. In this paper, a Deep neural network based Sparse Measurement Matrix (DSMM) is learned by the proposed convolutional network to reduce the sampling computational complexity and improve the CS reconstruction performance. Two sub-networks are included in the proposed network, which are the sampling sub-network and the reconstruction sub-network. In the sampling sub-network, the sparsity and the normalization are both considered by the limitation of the storage and the computational complexity. In order to improve the CS reconstruction performance, a reconstruction sub-network are introduced to help enhance the sampling sub-network. So by the offline iterative training of the proposed end-to-end network, the DSMM is generated for accurate measurement and excellent reconstruction. Experimental results demonstrate that the proposed DSMM outperforms GRM greatly on representative CS reconstruction methods.

As a successful deep model applied in image compressed sensing, the Compressed Sensing Network (CSNet) has demonstrated superior performance to the previous handcrafted models in both running speed and reconstruction quality. However, CSNet trains different models for different sampling rates that hinders it from practical usage since too many models need to store. In this paper, we propose multi-scale deep network for image compressed sensing. We still use a sampling network to learn the sampling operator and implement the compressed sampling process. Given the compressed measurements, the reconstruction network directly maps them to the desired reconstructed images. There are three main differences in comparison with CSNet. Firstly, this paper proposes to use an unified deep reconstruction network for all sampling rates that decreases large amount of storage requirements. Secondly, we redesign a better deep reconstruction network using the popular residual learning technology. Finally, we investigate an image local smooth prior based loss function to enhance image structural information. Extensive experimental results show that the proposed multi-scale deep network based image compressed sensing method outperforms many other state-of-the-art methods.

High Efficiency Video Coding (HEVC) introduces a number of new coding tools such as 35 intra prediction modes, improved coding unit structure and SAO filter and so on to improve the compression capacity of encoding. Compared with H.264/AVC, the compression performance increased by nearly 50% at the cost of the algorithm complexity increasing significantly. One of the most time-consuming part is the CU partition process which taking over 50% of the encoding time. In this paper, in order to reduce the computational pressure on the encoder side, we design an algorithm based on contour detection and SVM classifier. First, we adopt an improved Canny algorithm which is special designed for CU partition task. Then, based on SVM classifier, we introduce a new CU partition decision strategy. Experimental results show that the proposed algorithm achieves about 50% time saving on average with negligible loss of coding efficiency compared with HEVC reference software HM16.0.

With the rapid advancement of 3D applications, depth map super resolution has been a serious problem to be solved. Many researches has proposed depth map super resolution methods focusing on spatial domain which can't produce clear edges in the super resolution results. However, different from color images, depth maps have clear edges along them with internal smoothness. In this paper, we propose a novel multiresolution contourlet transform fusion based depth map super resolution method to enhance the quality of depth maps with preserving more contour information. We first transform the depth maps of multiple views via contourlet transform (CT) into multiresolution coefficients. Then we fuse the coefficients of same resolution to get the fused coefficients. Finally the target depth map is upscaled utilizing the prevalent upscaling framework JBU or WMF. Depth map fusion in transform domain is first proposed to improve the quality of the target depth map. A CT based depth map fusion model can not only produce sharper edges, but alleviate the noise in the depth map. Experimental results demonstrate that our proposed method outperforms many state-of-the-art algorithms in both objectively and subjectively.

HEVC is the latest video coding standard, in which inter prediction plays an important role to reduce the temporal redundancy. The accuracy of inter prediction is limited since only temporal information is used in conventional algorithms. In this paper, we propose a neural network based inter prediction algorithm for HEVC by using the spatial-temporal information. In the proposed algorithm, we first design a neural network architecture consisting of a fully connected network (FCN) and a convolutional neural network (CNN). Then the spatial neighboring pixels and the temporal neighboring pixels are inputted into FCN. The output of FCN and the prediction of current block are inputted into CNN, which will results the more accurate prediction of current block. Experimental results demonstrate that the proposed method can achieve average 1.7% (up to 8.6%) BD-rate reduction in low delay P test condition compared to HM 16.9.

In this paper, we propose a robust contrast enhancement algorithm based on cartoon and texture layer decomposition. Specifically, the cartoon layer is expected to be generally smoothing but with sharp edges at the foreground and background boundaries, for which we propose a quadratic form of graph total variation (GTV) as the prior to promote signal smoothness along graph structure. For the texture layer, a re-weighted GTV is tailored to remove noises while preserving true image details. Finally, an optimization objective function is formulated, which casts image decomposition, contrast enhancement and noise reduction into a unified framework. We propose an efficient algorithm to solve it. Experimental results show that our generated images outperform state-of-the-art schemes noticeably in subjective quality evaluation.

Compressed Sensing (CS) has been successfully applied to image compression in the past few years. However, there are still several challenges that restrict its applications in practice including large memory requirement and unsatisfactory reconstruction performance. To address these challenges, in this paper, we propose a classification guided deep convolutional network for image compressed sensing (CCSNet), which includes a sampling sub-network and a reconstruction sub-network. In the sampling sub-network, multiple convolutional layers are used to sample the original image, which significantly reduces the parameters of the sampling matrix while causes performance degradation moderately compared against existing convolution based sampling methods. In the reconstruction sub-network, a novel two-branch architecture is proposed to improve the adaptability of the model to various textures in natural images. The first branch, named the classification branch, is to classify the sampled measurements of the original image to one of the predefined textural classes. The second branch, named the reconstruction branch, consists of multiple sub-branches, which are responsible for reconstructing the original images belonging to the corresponding textural classes. By jointly utilizing two sub-networks, the entire network can be trained in the form of end-to-end metric with a joint loss function. Experimental results demonstrate that the proposed method provides a significant quality improvement in terms of PSNR compared against state-of-the-art methods.

Paragraph generation of images is a new concept, aiming to produce multiple sentences to describe a given image. In this paper, we propose a paragraph generation network with introducing visual relationship detection. We first detect regions which may contain important visual objects and then predict their relationships. Paragraphs are produced based on object regions which have valid relationship with others. Compared with previous works which generate sentences based on region features, we explicitly explore and utilize visual relationships in order to improve final captions. The experimental results show that such strategy could improve paragraph generating performance from two aspects: more details about object relations are detected and more accurate sentences are obtained. Furthermore, our model is more robust to region detection fluctuation.

Traditional image compressed sensing (CS) coding frameworks solve an inverse problem that is based on the measurement coding tools (prediction, quantization, entropy coding, etc.) and the optimization based image reconstruction method. These CS coding frameworks face the challenges of improving the coding efficiency at the encoder, while also suffering from high computational complexity at the decoder. In this paper, we move forward a step and propose a novel deep network based CS coding framework of natural images, which consists of three sub-networks: sampling sub-network, offset sub-network and reconstruction sub-network that responsible for sampling, quantization and reconstruction, respectively. By cooperatively utilizing these sub-networks, it can be trained in the form of an end-to-end metric with a proposed rate-distortion optimization loss function. The proposed framework not only improves the coding performance, but also reduces the computational cost of the image reconstruction dramatically. Experimental results on benchmark datasets demonstrate that the proposed method is capable of achieving superior rate-distortion performance against state-of-the-art methods.

Depth information is being widely used in many real-world applications. However, due to the limitation of depth sensing technology, the captured depth map in practice usually has much lower resolution than that of color image counterpart. In this paper, we propose to joint exploit the internal smoothness prior and external gradient consistency constraint in graph domain for depth super-resolution. On one hand, a new graph Laplacian regularizer is proposed to the preserve the inherent piecewise smooth characteristic of depth, which has desirable filtering properties. On the other hand, inspired by an observation that the gradient of depth is zero except at edge separating regions, we introduce a graph gradient consistency constraint to enforce that the graph gradient of depth is close to the thresholded gradient of guidance. Finally, the internal and external regularizations are casted into a unified optimization framework, which can be efficiently addressed by ADMM. Experiments results demonstrate that our method outperforms the state-of-the-art with respect to both objective and subjective quality evaluations.

Deep learning, e.g., convolutional neural networks (CNNs), has achieved great success in image processing and computer vision especially in high-level vision applications, such as recognition and understanding. However, it is rarely used to solve low-level vision problems such as image compression studied in this paper. Here, we move forward a step and propose a novel compression framework based on CNNs. To achieve high-quality image compression at low bit rates, two CNNs are seamlessly integrated into an end-to-end compression framework. The first CNN, named compact convolutional neural network (ComCNN), learns an optimal compact representation from an input image, which preserves the structural information and is then encoded using an image codec (e.g., JPEG, JPEG2000, or BPG). The second CNN, named reconstruction convolutional neural network (RecCNN), is used to reconstruct the decoded image with high quality in the decoding end. To make two CNNs effectively collaborate, we develop a unified end-to-end learning algorithm to simultaneously learn ComCNN and RecCNN, which facilitates the accurate reconstruction of the decoded image using RecCNN. Such a design also makes the proposed compression framework compatible with existing image coding standards. Experimental results validate that the proposed compression framework greatly outperforms several compression frameworks that use existing image coding standards with the state-of-the-art deblocking or denoising post-processing methods.

Moving object segmentation and classification from compressed video plays an important role in intelligent video surveillance. Compared with H.264/AVC, High Efficiency Video Coding (HEVC) introduces a host of new coding features that can be further exploited for moving object segmentation and classification. In this paper, we present a real-time approach to segment and classify moving objects using unique features directly extracted from the HEVC compressed domain for video surveillance. In the proposed method, first, motion vector (MV) interpolation for intra-coded prediction unit (PU) and MV outlier removal are employed for preprocessing. Second, blocks with nonzero MVs are clustered into the connected foreground regions using the four-connectivity component labeling algorithm. Third, object region tracking based on temporal consistency is applied to the connected foreground regions to remove the noise regions. The boundary of moving object region is further refined by the coding unit size and PU size. Finally, a person-vehicle classification model using bag of spatial-temporal HEVC syntax words is trained to classify the moving objects, either persons or vehicles. The experimental results demonstrate that the proposed method provides solid performance and can classify moving persons and vehicles accurately.

Millions of user-generated images are uploaded to social media sites like Facebook daily, which translate to a large storage cost. However, there exists an asymmetry in upload and download data: only a fraction of the uploaded images are subsequently retrieved for viewing. In this paper, we propose a cloud storage system that reduces the storage cost of all uploaded JPEG photos, at the expense of a controlled increase in computation mainly during download of requested image subset. Specifically, the system first selectively re-encodes code blocks of uploaded JPEG images using coarser quantization parameters for smaller storage sizes. Then during download, the system exploits known signal priors-sparsity prior and graph-signal smoothness prior-for reverse mapping to recover original fine quantization bin indices, with either deterministic guarantee (lossless mode) or statistical guarantee (near-lossless mode). For fast reverse mapping, we use small dictionaries and sparse graphs that are tailored for specific clusters of similar blocks, which are classified via tree-structured vector quantizer. During image upload, cluster indices identifying the appropriate dictionaries and graphs for the re-quantized blocks are encoded as side information using a differential distributed source coding scheme to facilitate reverse mapping during image download. Experimental results show that our system can reap significant storage savings (up to 12.05%) at roughly the same image PSNR (within 0.18 dB).

The free-energy principle in recent studies of brain theory and neuroscience models the perception and understanding of the outside scene as an active inference process, in which the brain tries to account for the visual scene with an internal generative model. Specifically, with the internal generative model, the brain yields corresponding predictions for its encountered visual scenes. Then, the discrepancy between the visual input and its brain prediction should be closely related to the quality of perceptions. On the other hand, sparse representation has been evidenced to resemble the strategy of the primary visual cortex in the brain for representing natural images. With the strong neurobiological support for sparse representation, in this paper, we approximate the internal generative model with sparse representation and propose an image quality metric accordingly, which is named FSI (free-energy principle and sparse representation-based index for image quality assessment). In FSI, the reference and distorted images are, respectively, predicted by the sparse representation at first. Then, the difference between the entropies of the prediction discrepancies is defined to measure the image quality. Experimental results on four large-scale image databases confirm the effectiveness of the FSI and its superiority over representative image quality assessment methods. The FSI belongs to reduced-reference methods, and it only needs a single number from the reference image for quality estimation.

Recent years have witnessed the growing popularity of hashing for large-scale multimedia retrieval. Extensive hashing methods have been designed for data stored in a single machine, that is, centralized hashing . In many real-world applications, however, the large-scale data are often distributed across different locations, servers, or sites. Although hashing for distributed data can be implemented by assembling all distributed data together as a whole dataset in theory, it usually leads to prohibitive computation, communication, and storage costs in practice. Up to now, only a few methods were tailored for distributed hashing, which are all unsupervised approaches. In this paper, we propose an efficient and effective method called supervised distributed hashing (SupDisH), which learns discriminative hash functions by leveraging the semantic label information in a distributed manner. Specifically, we cast the distributed hashing problem into the framework of classification, where the learned binary codes are expected to be distinct enough for semantic retrieval. By introducing auxiliary variables, the distributed model is then separated into a set of decentralized subproblems with consistency constraints, which can be solved in parallel on each vertex of the distributed network. As such, we can obtain high-quality distinctive unbiased binary codes and consistent hash functions with low computational complexity, which facilitate tackling large-scale multimedia retrieval tasks involving distributed datasets. Experimental evaluations on three large-scale datasets show that SupDisH is competitive to centralized hashing methods and outperforms the state-of-the-art unsupervised distributed method significantly.

In the latest video coding standard, namely, high-efficiency video coding (HEVC), intra coding efficiency is significantly improved by a quadtree partition structure and more intra prediction modes. For intra coding, 35 intra modes are employed including 33 angular intra prediction (AIP) modes that are effective for blocks with strong directions, and a planar mode and a dc mode that are used to predict smooth regions. However, intra prediction in HEVC still cannot handle complicated blocks well. To deal with this problem, this paper proposes a hybrid intra prediction method to improve the intra prediction efficiency. The proposed hybrid intra prediction method consists of three parts: adaptive template matching prediction (ATMP) by exploring nonlocal correlation, combined local and nonlocal prediction by exploring both local correlation for AIP and non-local correlation for ATMP, and combined neighboring modes prediction that can generate smooth prediction by exploring more local correlations. Experimental results suggest that the proposed hybrid intra prediction method achieves 2.8% BD-rate reduction on average for luma compared to HEVC reference software HM-14.0 under all intra main configurations. The gain can be up to 6.9%. When integrated into joint exploration model -1.0, the proposed method still can achieve 1.2% BD-rate reduction for luma.

Summary form only given. Traditional intra prediction methods for HEVC rely on using the nearest reference lines for predicting a block, which ignore much richer context between the current block and its neighboring blocks and therefore cause inaccurate prediction especially when weak spatial correlation exists between the current block and the reference lines. To overcome this problem, in this paper, an intra-prediction convolutional neural network (IPCNN) is proposed for intra prediction, which exploits the rich context of the current block and therefore is capable of improving the accuracy of predicting the current block. Meanwhile, the reconstruction of the three nearest blocks can also be refined. To the best of our knowledge, this is the first paper that directly applies CNNs to intra prediction for HEVC. Experimental results validate the effectiveness of applying CNNs to intra prediction and the proposed method can achieve 0.70% bitrate reduction compared to HEVC reference software HM-14.0.

Summary form only given. Traditional image coding standards (such as JPEG and JPEG2000) make the decoded image suffer from many blocking artifacts or noises since the use of big quantization steps. To overcome this problem, we proposed an end-to-end compression framework based on two CNNs, as shown in Figure 1, which produce a compact representation for encoding using a third party coding standard and reconstruct the decoded image, respectively. To make two CNNs effectively collaborate, we develop a unified end-to-end learning framework to simultaneously learn CrCNN and ReCNN such that the compact representation obtained by CrCNN preserves the structural information of the image, which facilitates to accurately reconstruct the decoded image using ReCNN and also makes the proposed compression framework compatible with existing image coding standards.

The human visual perception is a layered progressive process that brain assimilates visual information gradually, from primary information, structural information to detailed information. Recently, the visual primitives (atoms in the dictionary) extracted by sparse representation have been shown to be highly related to the layered progressive process of human visual perception. In this paper, the visual primitives are first classified into three categories: DCprimary, sketch and texture in terms of their inherent properties regarding tothe perceptual information. Then, we propose a novel reduced reference (RR) image quality assessment (IQA) metric using perceptual information represented by entropy of classified primitives (EoCP). Specifically, EoCP is a measurement of the distribution statistics of the visual primitives, which can represent the visual information. The differences of EoCPs between the reference image and its distorted version are calculated as features to characterize perceptual loss. The extracted features (only three scalars) are used to compute the quality score by a prediction function which is trained using support vector regression(SVR). Experimental results on LIVE, CSIQ and TID2013 image databases demonstrate that the proposed metric achieves high consistency with the human perception and show competitive performance with state-of-the-art IQA metrics.

Traditional works have shown that patches in a natural image tend to redundantly recur many times inside the image, both within the same scale, as well as across different scales. Make full use of these multi-scale information can improve the image restoration performance. However, the current proposed deep learning based restoration methods do not take the multi-scale information into account. In this paper, we propose a dilated convolution based inception module to learn multi-scale information and design a deep network for single image super-resolution. Different dilated convolution learns different scale feature, then the inception module concatenates all these features to fuse multi-scale information. In order to increase the reception field of our network to catch more contextual information, we cascade multiple inception modules to constitute a deep network to conduct single image super-resolution. With the novel dilated convolution based inception module, the proposed end-to-end single image super-resolution network can take advantage of multi-scale information to improve image super-resolution performance. Experimental results show that our proposed method outperforms many state-of-the-art single image super-resolution methods.

The compressed sensing (CS) theory has been successfully applied to image compression in the past few years as most image signals are sparse in a certain domain. Several CS reconstruction models have been recently proposed and obtained superior performance. However, there still exist two important challenges within the CS theory. The first one is how to design a sampling mechanism to achieve an optimal sampling efficiency, and the second one is how to perform the reconstruction to get the highest quality to achieve an optimal signal recovery. In this paper, we try to deal with these two problems with a deep network. First of all, we train a sampling matrix via the network training instead of using a traditional manually designed one, which is much appropriate for our deep network based reconstruct process. Then, we propose a deep network to recover the image, which imitates traditional compressed sensing reconstruction processes. Experimental results demonstrate that our deep networks based CS reconstruction method offers a very significant quality improvement compared against state-of-the-art ones.

Stereoscopic vision is a complex system which receives and integrates perceptual information from both monocular and binocular cues. In this paper, a novel reduced-reference stereoscopic image quality assessment scheme is proposed, based on the visual perceptual information measured by entropy of classified primitives (EoCP) and mutual information of classified primitives (MIoCP), named as DCprimary, sketch and texture primitives respectively, which is in accordance with the hierarchical progressive process of human visual perception. Specifically, EoCP of each-view image are calculated as monocular cue, and MIoCP between two-view images is derived as binocular cue. The Maximum (MAX) mechanism is applied to determine the perceptual information. The perceptual information differences between the original and distorted images are used to predict the stereoscopic image quality by support vector regression (SVR). Experimental results on LIVE phase II asymmetric database validate the proposed metric achieves significantly higher consistency with subjective ratings and outperforms state-of-the-art stereoscopic image quality assessment methods.

Motion estimation (ME) plays an important role in most video coding standards since it can improve the coding performance significantly. Block matching ME (BMME) based on translation motion model is employed in both H.264/Advanced Video Coding and High Efficiency Video Coding (HEVC), which assumes that all the pixels in a block undergo the same motion. However, BMME cannot represent the complex motion accurately, such as rotation, zoom, and deformation motion. To resolve the problem, a merge mode for deformable block motion information derivation is proposed, which considers bilinear interpolation model, six-parameter affine model, and four-parameter affine model simultaneously. The sum of absolute transformed differences criterion is used to select the best model for a block. In the proposed merge mode, the motion information of the current block is derived from the motion information of its neighbors. It does not need to perform ME and transmit motion parameters. The proposed merge mode is integrated into HEVC test model (HM) 14.0. The simulation results show that compared with HM14.0, the proposed merge mode can bring 1.3%, 1.7%, and 2.7% Bjøntegaard delta (BD) bitrate savings on average for test sequences of HEVC, and 11.0%, 15.2%, and 17.4% BD bitrate savings for deformation sequences in random access main, low-delay B main, and low-delay P main test conditions, respectively, while the increased encoding and decoding complexities are moderate.

The latest video coding standard High Efficiency Video Coding (HEVC) achieves about a 50% bit-rate reduction compared with H.264/AVC under the same perceptual video quality. For intra coding, a coding unit (CU) is recursively divided into a quadtree-based structure from the largest CU 64 × 64 to the smallest CU 8 × 8. Also, up to 35 intra-prediction modes are allowed. These two techniques improve the intra-coding performance significantly. However, the encoding complexity increases several times compared with H.264/AVC intra coding. In this paper, fast intra-mode decision and CU size decision are proposed to reduce the complexity of HEVC intra coding while maintaining the rate-distortion (RD) performance. For fast intra-mode decision, a gradient-based method is proposed to reduce the candidate modes for rough mode decision and RD optimization. For fast CU size decision, the homogenous CUs are early terminated first. Then two linear support vector machines that employ the depth difference and HAD cost ratio (and RD cost ratio) as features are proposed to perform the decisions of early CU split and early CU termination for the rest of the CUs. Experimental results show that the proposed fast intra-coding algorithm achieves about a 54% encoding time reduction on average with only a 0.7% BD-rate increase for the HEVC reference software HM 14.0 under all-intra configuration.

Given the prevalence of joint photographic experts group (JPEG) compressed images, optimizing image reconstruction from the compressed format remains an important problem. Instead of simply reconstructing a pixel block from the centers of indexed discrete cosine transform (DCT) coefficient quantization bins (hard decoding), soft decoding reconstructs a block by selecting appropriate coefficient values within the indexed bins with the help of signal priors. The challenge thus lies in how to define suitable priors and apply them effectively. In this paper, we combine three image priors-Laplacian prior for DCT coefficients, sparsity prior, and graph-signal smoothness prior for image patches-to construct an efficient JPEG soft decoding algorithm. Specifically, we first use the Laplacian prior to compute a minimum mean square error initial solution for each code block. Next, we show that while the sparsity prior can reduce block artifacts, limiting the size of the overcomplete dictionary (to lower computation) would lead to poor recovery of high DCT frequencies. To alleviate this problem, we design a new graph-signal smoothness prior (desired signal has mainly low graph frequencies) based on the left eigenvectors of the random walk graph Laplacian matrix (LERaG). Compared with the previous graph-signal smoothness priors, LERaG has desirable image filtering properties with low computation overhead. We demonstrate how LERaG can facilitate recovery of high DCT frequencies of a piecewise smooth signal via an interpretation of low graph frequency components as relaxed solutions to normalized cut in spectral clustering. Finally, we construct a soft decoding algorithm using the three signal priors with appropriate prior weights. Experimental results show that our proposal outperforms the state-of-the-art soft decoding algorithms in both objective and subjective evaluations noticeably.

In this paper, we propose a novel sparsity-based image error concealment (EC) algorithm through adaptive dual dictionary learning and regularization. We define two feature spaces: the observed space and the latent space, corresponding to the available regions and the missing regions of image under test, respectively. We learn adaptive and complete dictionaries individually for each space, where the training data are collected via an adaptive template matching mechanism. Based on the piecewise stationarity of natural images, a local correlation model is learned to bridge the sparse representations of the aforementioned dual spaces, allowing us to transfer the knowledge of the available regions to the missing regions for EC purpose. Eventually, the EC task is formulated as a unified optimization problem, where the sparsity of both spaces and the learned correlation model are incorporated. Experimental results show that the proposed method outperforms the state-of-the-art techniques in terms of both objective and perceptual metrics.

Transform is used to compact the energy of the blocks into a small number of coefficients and is widely used in recent image/video coding standards. In the latest video coding standard high efficiency video coding (HEVC), a combination of discrete cosine transform (DCT) and discrete sine transform (DST) is adopted to transform the residuals from intra prediction. Since the DCT and DST are the fixed transforms that are derived from the Gauss-Markov model, some of residual blocks may not be compacted well by the DCT/DST. In this paper, we propose a signal dependent transform based on singular value decomposition (SVD) for HEVC intracoding. The proposed transform (SDT-SVD) is derived by performing SVD on the synthetic block and applied to the residual block considering the structural similarity between them. Furthermore, we extend SDT-SVD to template matching prediction (TMP) to further improve the intracoding performance. Experimental results show that the proposed transform on angular intra prediction (AIP) outperforms the latest HEVC reference software with a bit rate reduction of 1.0% on average and it can be up to 2.1%. When the proposed transform is extended to TMP-based intracoding, the overall bit rate reduction is 2.7% on average and can be up to 5.8%.

Social media sites like Facebook are obligated to store all photos uploaded by an ever growing user base-which translates to an increasingly expensive storage cost-but only a fraction of uploaded images are revisited thereafter. In this paper, we propose a cloud storage system that trades off computation of a small fraction of requested images with storage of all photos. The key idea is to re-encode uploaded JPEG photos with coarser quantization parameters (QP) for permanent storage, then exploit a signal sparsity prior during inverse mapping to recover fine quantization bin indices via a maximum a posteriori (MAP) formulation. Because by design the system guarantees recovery of an original compressed image (either with exactly the same input fine quantization bin indices or has visual quality indistinguishable by human eyes), from the user's viewpoint it is a normal cloud storage, while from the operator's viewpoint there is pure compression gain and hence lower storage cost. Experimental results show that our storage system can reap significant storage savings (up to 20%) at roughly the same image PSNR (within 0.13dB).

The Group-based Sparse Representation (GSR) is able to sparsely represent natural images in the domain of group, which enforces the intrinsic local sparsity and nonlocal self-similarity of images simultaneously in a unified framework. And the GSR-driven L0 minimization method for image restoration has been proposed. This paper expands the application of GSR from image restoration to low lighting image enhancement. The GSR is not used to represent the natural images anymore, but representing the transmission map of the haze image and recovering it. Because the transmission map is very important for the low lighting image enhancement, the dark channel prior based enhancement method with the enhanced transmission map can get a better enhanced results. Different from other methods, we evaluate the quality of the enhanced images not only by qualitative analysis but also by quantitative results. Extensive experiments on low lighting image show that the GSR-based method gets a better enhancement result than many current state-of-the-art ones.

Recently, the concept of Entropy of Primitive (EoP) has been proposed to measure the image visual information. Some successful EoP based application also be developed. In this paper, we further explore the concept of EoP and propose an improved version: the L1 norm based EoP. Our EoP takes full account of the properties of a dictionary's layered structure and the characteristic of a basis pursuit method. Experimental results show that the L1 norm based EoP is superior to the L0 norm based one in measuring the image visual information. The curve of L1 norm based EoP holds a more consistent monotonicity with SSIM, its values is not trapped in the local convergence and the convergence value is less than that of the L0 norm based one. With the convergence characteristics of EoP, we further explore its application in stereoscopic image quality assessment (SIQA). With EoP as monocular cue and mutual information of primitive (MIP) as binocular cue, the relative entropy between the original stereoscopic image and the distorted one is used to compute the quality score by a prediction function which is trained using support vector regression (SVR). Extensive experimental results show that our new EoP based SIQA outperforms many state-of-the-art on the LIVE phase II databases.

High efficiency video coding (HEVC) is the latest video coding standard, in which coding performance improvement of intra prediction comes from flexible block partitioning and advanced directional prediction. Blocks with strong directionality and regular patterns can be handled well by this finer granular directional prediction. However, irregular patterns and noises often exist in natural videos, and reference samples are not fully used in directional intra prediction. In this paper, we propose a mode dependent intra smoothing filter algorithm to improve intra prediction, which employs a Markov model to improve coding efficiency of intra prediction. Experimental results demonstrate that the proposed method can achieve average 1.1% (up to 1.8%) BD-rate reduction in all intra high efficiency test condition compared to HM 16.0.

The latest high efficiency video coding (HEVC) standard achieves about 50% bit-rate reduction at equivalent visual quality compared to H.264/AVC. Sample adaptive offset (SAO) is one of the newly adopted tools right after deblocking filter, which can improve both coding efficiency and visual quality. However, for real-time encoding scenarios, the complexity of SAO is usually too high to be enabled. In this paper, a GPU-based optimization algorithm is proposed to reduce the complexity of SAO. Experiments are conducted based on the state-of-the-art open source HEVC encoder, i.e. X265. Results show that the proposed algorithm can reduce about 70% processing time of SAO on average without sacrifice of coding efficiency.

Since the purpose of objective image quality assessment is to be consistent with subjective image quality assessment as highly as possible, the understanding of the mechanisms of human visual system will certainly benefit the study of objective image quality assessment. Recent developments in brain theory and neuroscience, particularly the free-energy principle, account for the perception and understanding of visual scenes. As the free-energy principle conjectures, the brain tries to generate the corresponding prediction for its encountered scene by an internal generative model. On the other hand, sparse representation is evidenced to resemble the neural response properties of simple cells in the primary visual cortex. Conjunctively, in this paper, we suppose the prediction manner of the internal generative model in free-energy principle follows sparse representation and propose an image quality metric accordingly. Experiments on LIVE, TID2008 and CSIQ image databases demonstrate the effectiveness of the proposed image quality metric. Noteworthily, our metric needs little information (only a single scalar) of the reference image and is training-free.

Compressed domain moving object segmentation and classification plays an important role in many real-time applications, such as video indexing and intelligent video surveillance. Compared with the previous international video coding standards, such as H.264/AVC, HEVC introduces a host of new coding features. Therefore, moving object segmentation and classification directly from HEVC compressed videos represents a new challenge. In this paper, we develop a method for segmenting and classifying moving objects, specifically, persons and vehicles, in the HEVC compression domain. We first train a classifier to determine if an image patch belongs to the foreground objects or background using HEVC syntax features. This will generate a bounding box which locates the object in the video frame. We then train a second classification model to classify the moving objects, either persons or vehicles, using bags of spatial-temporal HEVC syntax words. Our extensive experimental results demonstrate that the approach provides the remarkable performance and can classify moving person and vehicles accurately and robustly.

As a 3D extension of the Audio Video coding Standard (AVS), 3D-AVS is used to improve the coding efficiency of multi-view video. However, the improved performance has also resulted in increased coding complexity. Taking into account the real-time request, how to reduce the complexity of the encoder is becoming very critical. In this paper, a fast encoder decision algorithm including two strategies is proposed. The first one is an early skip/direct mode decision algorithm, and the second one is an adaptive CU minimum splitting algorithm. Both of the two algorithms are applied to dependent texture views to reduce unnecessary visits by exploiting inter-view correlations between the dependent view and the base view. Experimental results show that the proposed algorithm can save 34.2% encoding time with 0.07% BD-rate loss on the overall texture views compared to RFD-4.0 (3D-AVS test model) under the common test condition (CTC). Both of the two strategies have been adopted into the 3D-AVS reference software and disabled in the configuration file under CTC.

Recent years have witnessed the success of the proposed line-based distributed coding and transmission system for broadcasting satellite images (LineCast). The LineCast achieves high compression efficiency and provides smooth multicast capability with soft broadcast and coset coding, in which the precision of side information is crucial. However, the generated side information is rough by averaging several candidates based on template matching. In this paper, we propose a low-rank approximation based LineCast scheme for video broadcasting (LRALineCast) to generate more precise side information by exploiting the cross-frame correlation. The side information generation problem is thus reformulated as a low-rank matrix completion problem, where the approximately low-rank matrix is rearranged by the similar lines searched by template matching. LRALineCast achieves good broadcast performance and avoids the annoying cliff effect by utilizing soft broadcast transmission. The experimental results show that LRALineCast outperforms the state-of-the-art 2D broadcasting scheme and LineCast with up to 2dB coding gain.

In the large body of research literature on image restoration, very few papers were concerned with compression-induced degradations, although in practice, the most common cause of image degradation is compression. This paper presents a novel approach to restoring JPEG-compressed images. The main innovation is in the approach of exploiting residual redundancies of JPEG code streams and sparsity properties of latent images. The restoration is a sparse coding process carried out jointly in the DCT and pixel domains. The prowess of the proposed approach is directly restoring DCT coefficients of the latent image to prevent the spreading of quantization errors into the pixel domain, and at the same time, using online machine-learned local spatial features to regulate the solution of the underlying inverse problem. Experimental results are encouraging and show the promise of the new approach in significantly improving the quality of DCT-coded images.

In this paper, a new compressive sampling-based image coding scheme is developed to achieve competitive coding efficiency at lower encoder computational complexity, while supporting error resilience. This technique is particularly suitable for visual communication with resource-deficient devices. At the encoder, compact image representation is produced, which is a polyphase down-sampled version of the input image; but the conventional low-pass filter prior to down-sampling is replaced by a local random binary convolution kernel. The pixels of the resulting down-sampled pre-filtered image are local random measurements and placed in the original spatial configuration. The advantages of the local random measurements are two folds: 1) preserve high-frequency image features that are otherwise discarded by low-pass filtering and 2) remain a conventional image and can therefore be coded by any standardized codec to remove the statistical redundancy of larger scales. Moreover, measurements generated by different kernels can be considered as the multiple descriptions of the original image and therefore the proposed scheme has the advantage of multiple description coding. At the decoder, a unified sparsity-based soft-decoding technique is developed to recover the original image from received measurements in a framework of compressive sensing. Experimental results demonstrate that the proposed scheme is competitive compared with existing methods, with a unique strength of recovering fine details and sharp edges at low bit-rates.

Arguably the most common cause of image degradation is compression. This papers presents a novel approach to restoring JPEG-compressed images. The main innovation is in the approach of exploiting residual redundancies of JPEG code streams and sparsity properties of latent images. The restoration is a sparse coding process carried out jointy in the DCT and. pixel domains. The prowess of the proposed approach is directly restoring DCT coefficients of the latent image to prevent the spreading of quantization errors into the pixel domain, and at the same time using on-line machine-learnt local spatial features to regulate the solution of the underlying inverse problem. Experimental results are encouraging and show the promise of the new approach in significantly improving the quality of DCT-coded images.

Gaussian random matrix (GRM) has been widely used to generate linear measurements in compressive sensing (CS) of natural images. However, in practice, there actually exist two problems with GRM. One is that GRM is non-sparse and complicated, leading to high computational complexity and high difficulty in hardware implementation. The other is that regardless of the characteristics of signal the measurements generated by GRM are also random, which results in low efficiency of compression coding. In this paper, we design a novel local structural measurement matrix (LSMM) for block-based CS coding of natural images by utilizing the local smooth property of images. The proposed LSMM has two main advantages. First, LSMM is a highly sparse matrix, which can be easily implemented in hardware, and its reconstruction performance is even superior to GRM at low CS sampling sub rate. Second, the adjacent measurement elements generated by LSMM have high correlation, which can be exploited to greatly improve the coding efficiency. Furthermore, this paper presents a new framework with LSMM for block-based CS coding of natural images, including measurement generating, measurement coding and CS reconstruction. Experimental results show that the proposed framework with LSMM for block-based CS coding of natural images greatly enhances the existing CS coding performance when compared with other state-of-the-art image CS coding schemes.

In this paper, the hierarchical frame based video compressed sensing (CS) framework is proposed, which outperforms the traditional framework through the better exploitation of frames correlation with reference frames, the unequal sample subrates setting among frames in different layers and the reduction of the error propagation. By considering the spatial and temporal correlations of the video sequence, a spatial-temporal sparse representation based recovery is proposed for this framework. The similar blocks in both the current frame and these recovered reference frames are composed as a spatial-temporal group, which is defined as the unit of the sparse representation. By exploiting the low dimensional subspace description of each group, the video CS recovery is converted as a low-rank matrix approximation problem, which can be solved by exploiting the hard thresholding and the gradient descent. Experimental results show that the proposed method achieves better performance against both the state-of-art still-image CS recovery algorithms and the existing residual domain based video CS reconstruction approaches.

Given the prevalence of JPEG compressed images on the Internet, image reconstruction from the compressed format remains an important and practical problem. Instead of simply reconstructing a pixel block from the centers of assigned DCT coefficient quantization bins (hard decoding), we propose to jointly reconstruct a neighborhood group of pixel patches using two image priors while satisfying the quantization bin constraints. First, we assume that a pixel patch can be approximated as a sparse linear combination of atoms from an offline-learned over-complete dictionary. Second, we assume that a patch, when interpreted as a graph-signal, is smooth with respect to an appropriately defined graph that captures the estimated structure of the target image. Finally, neighboring patches in the optimization have sufficient overlaps and are forced to be consistent, so that blocking artifacts typical in JPEG decoded images are avoided. To find the optimal group of patches, we formulate a constrained optimization problem and propose a fast alternating algorithm to find locally optimal solutions. Experimental results show that our proposed algorithm outperforms state-of-the-art soft decoding algorithms by up to 1.47dB in PSNR.

Recent studies have shown that one of the main reasons inducing visual discomfort is accommodation-vergence conflict. To evaluate visual discomfort induced by this conflict, this paper proposes a stereo visual comfort assessment (SVCA) metric based on the measurement of accommodation and vergence for stereoscopic image. Here, accommodation corresponds to the monocular focusing process which is modeled by two-view images' joint entropy; vergence corresponds to binocular fusion which is modeled by two-view images' mutual information. The joint entropy and mutual information are calculated by the visual primitives extracted from two-view images. In this paper, accommodation-vergence conflict is expressed as the ratio of the mutual information over joint entropy. To evaluate the proposed metric, a subjective experiment is conducted to construct a ground truth database. The experimental results show that the proposed SVCA metric achieves a highly competitive performance with some state-of-the-art SVCA models.

Different from general algorithms of haze removal and low lighting image enhancement, which only use the information of image to process, this paper adds a reference image to get more information for the algorithm and focuses on enhancing region of interest of an image based on the reference one. With the reference image, the haze one can be divided into Region of Interest (RoI) and Region of no Interest (non-RoI). Furthermore, the reference image can provide more useful information for computing the transmission map and atmospheric light. For the non-RoI region, a more robust transmission map and minimizing reconstruction error cost function based method to estimate atmospheric light has been proposed. Because the atmospheric light is a global variable, the optimized one is also suitable for the RoI region. With the global optimized atmospheric light, an optimized transmission map can be got for the RoI region. The RoI region can be enhanced via the optimal transmission map and atmosphere light. Theoretical analysis gives eloquent proof proving that the proposed method is definitely better than the traditional dark-channel-prior-based methods due to our better transmission map and atmosphere light. Extensive experiments also show the expected results.

In this paper, we propose a macro-block (MB) level rate control algorithm for low delay H.264/AVC video communication based on the ρ domain rate model. In the proposed algorithm, an exponential model is used to characterize the relation between ρ and the quantization step (Qstep) at the MB level, with which the quantization parameter (QP) for a MB can be obtained. Furthermore, a switched QP calculation scheme is introduced to obtain the QP for each MB to avoid large deviation of the actual frame size from the target bit budget. Compared with the original ρ domain rate control, the proposed method can achieve better video quality and improved bit-rate accuracy. Meanwhile, the computational complexity is also significantly reduced.

A well-known problem in video conferencing is gaze mismatch. Instead of relying exclusively on online captured data for rendering, a recent work first trains offline dictionaries using a large image database of movie and TV stars to learn "beautiful" features. During real-time conferencing, one can then simultaneously correct gaze and beautify the subject's facial components in single images by seeking sparse linear combination of pre-trained dictionary atoms for face reconstruction. Extending on this work, we focus on joint gaze correction / face beautification for video. First, we define a large search space invariant to scale, shift and rotation for facial feature beautification based on SIFT. We then address two practical issues unique to video: i) how beautified results can be temporally consistent across group of pictures (GOP), and ii) how blinking eyes can be beautified even though the training database contains only open-eye facial images. Experimental results show that our method achieves the desired temporal consistency, and the blinking process is smooth and natural.

During the process of image acquisition, images are often subject to out-of-focus or defocus blur because of the improper adjustment of the camera's focal length, this image blur will degrade the image quality. However, in the literature, image quality assessment (IQA) methods dedicated to evaluating the quality of images with out-of-focus blur remain few. Therefore, in this paper, we focus our attention on the quality assessment of images that suffer from out-of-focus blur and propose an objective quality assessment method accordingly. Concretely, we construct a dedicated out-of-focus blurred image dataset, which is composed of 150 images subjected to different degrees of out-of-focus blur and the mean opinion scores (MOSs). Then, we propose a specific objective quality metric for the blurred images, which combines image sharpness assessment and saliency-guided pooling strategy. Experimental results demonstrate the proposed metric highly correlates with human judgements of image quality.

As a 3D video extension of Audio Video Coding Standard (AVS), 3D-AVS is being developed to improve the coding efficiency of multi-view video. Since multi-view video is composed of projections of the same scenery from different viewpoints at the same time instant, it contains a large amount of inter-view redundancies. To exploit the inter-view correlation, this paper presents a method to derive the motion parameters for a coding unit (CU) in the dependent view from the already coded inter-view picture. The algorithm is based on quad-tree partitioning, each CU could be recursively split into four sub-CUs of the same size and whether each sub-CU is further split is determined by comparing the derived motion parameters. Experimental results show that the proposed method provides 8.7% BD-rate saving for both video 1 and video 2 in low delay configuration and the BD-rate saving is up to 14.5% and 13.7% on video 1 and video 2 for Balloons. This method has been proposed and adopted into the 3D-AVS standard.

In the latest HEVC video coding standard, angular intra prediction (AIP) applies 35 modes including 33 angular modes which can handle blocks with direction information well, and other 2 modes (DC and planar) which are used to predict smooth blocks. However, for blocks with complex texture, these modes may not give good predictions. Some complex blocks can be predicted well by template matching prediction (TMP) which was proposed to predict blocks having similar patterns in the coded regions of the same frame without the cost of large overheads. In this paper, a novel hybrid AIP/TMP is proposed to improve the prediction efficiency. Experimental results show that the proposed method can achieve a coding gain of about 2.5% for high resolution sequences on average compared to the AIP in HEVC. The gain can be up to 4.2%.

Inter prediction plays an important role in most video encoding systems since it could significantly improve coding performance. The more accurate the prediction of the current block, the smaller the residual, and the higher coding efficiency could be achieved accordingly. In this paper, a localized weighted prediction method is proposed to improve inter prediction accuracy. The linear regression improvement model is employed to modify the prediction pixel values. The weighting parameters are estimated in both encoder and decoder, no additional bits are required to be transmitted. The proposed method shows better coding performance than previous methods, including the explicit weighted prediction method in High Efficiency Video Coding (HEVC). Experimental results show that the BD bit rate saving of the proposed method is up to 7.4% compared to HM12.0, while the decoding complexity is almost the same.

In many multimedia applications, such as online speech, video chat and online conference, multiple source videos are synthesized in a single scene for explicit presentation and the synthesized video is compressed for transmission. The source video with important contents deserves more compression resources for quality preservation under the bandwidth constraint. To address this problem, a region-of-interest (ROI) based coding scheme for synthesized video is proposed in this paper aiming at achieve better and consistent quality for ROI source videos with the bitrate meeting the constraint bandwidth. In the proposed coding scheme, ROI based rate-distortion (R-D) models are established, in which different R-D models are built for different source video. Then an objective function is defined with respect to the video quality and the consistency of video quality. By minimizing the objective function, the optimal quantization parameters for the ROI and non-ROI source videos are obtained. The experimental results show that the proposed coding scheme achieves better and consistent quality for ROI source videos.

The advent and popularity of Kinect provides a new choice and opportunity for hand gesture recognition (HGR) research. In this study, the authors propose a discriminating features extraction for HGR, in which features from red, green and blue (RGB) images and depth images are both explored. More specifically, histogram of oriented gradient feature, local binary pattern feature, structure feature and three‐dimensional voxel feature are first extracted from RGB images and depth images, then these features are further reduced with a novel deflation orthogonal discriminant analysis, which enhances the discriminative ability of the features with supervised subspace projection. The extensive experimental results show that the proposed method improves the HGR performance significantly.

Wireless video broadcast poses a challenge to the conventional visual communication in providing simultaneously each receiver the best video quality under its channel condition. Soft video broadcast, as a newly emerged wireless video broadcast scheme, is able to accommodate multiple receivers of different channel SNRs. However, the current soft video broadcast frameworks such as SoftCast require the bandwidth of the wireless channel to match the number of video coefficients per second. When the channel bandwidth is larger, the existing frameworks become not very efficient in bandwidth expansion. More importantly, it is possible that the users in broadcast applications have different bandwidths. However, none of the existing soft video broadcast frameworks considers bandwidth heterogeneity. In this paper, we propose a soft video broadcast framework, called LayerCast, which can simultaneously accommodate heterogeneous users with diverse SNRs and diverse bandwidths. The bandwidth expansion problem is solved by applying layered coset coding. More importantly, we derive a globally optimal power allocation between layers and, within each layer, between each DCT chunk. In simulations, the proposed framework outperforms SoftCast of up to 4 dB in video PSNR, and outperforms H.264-based framework up to 8 dB in broadcast.

In this paper, we propose a novel reduced reference stereoscopic image quality assessment (RR-SIQA) metric by using binocular perceptual information (BPI). BPI is represented by the distribution statistics of visual primitives in left and right views' images, which are extracted by sparse coding and representation . Specifically, entropy of the left view's image and entropy of the right view's image are used to represent monocular cue. Their mutual information is used to represent binocular cue. Constructively, we represent BPI as three numerical indicators . The difference of the original and distorted images' BPIs is taken as perceptual loss vector. The perceptual loss vector is used to compute the quality score for a stereoscopic image by a prediction function which is trained using support vector regression (SVR). Experimental results show that the proposed metric achieves significantly higher prediction accuracy than the state-of-the-art reduced reference SIQA methods and better than several state-of-the-art full reference SIQA methods on the LIVE phase II asymmetric databases.

In this paper, an effective multiple description image coding technique is developed to achieve competitive coding efficiency at low encoder complexity, while being standard compliant. The new technique is particularly suitable for visual communication over packet-switched networks and with resource-deficient wireless devices. To keep the encoder simple and standard compliant, multiple descriptions are produced by quincunx spatial multiplexing. Each side description is a polyphase down sampled version of the input image, but the conventional low-pass filter prior to downsampling is replaced by a local random binary convolution kernel. The pixels of each resulting side description are local random measurements and placed in the original spatial configuration. The advantages of local random measurements are two folds: 1) preservation of high-frequency image features that are otherwise discarded by low-pass filtering, 2) each side description remains a conventional image and can therefore be coded by any standardized codec to remove statistical redundancy of larger scales. The decoder performs joint upsampling of received description(s) and recovers the image from local random measurements in a framework of compressive sensing. Experimental results demonstrate that the proposed multiple description image codec is competitive in rate-distortion performance compared with existing methods, with a unique strength of recovering fine details and sharp edges at low bit rates.

Image interpolation is one of the most elementary imaging research topics. A number of image interpolation methods have been developed and tested on uncompressed images in the literature. However, a lot of videos have already been stored or have to be transmitted in compressed format due to the storage limitation or the bandwidth limitation. The existed image interpolation methods may not be efficient when directly applied to compressed images or videos. Inspired by the success of the intra prediction in HEVC and the edge-directed image interpolation methods, a directional intra frame interpolation for HEVC compressed video is proposed. The main idea is to use the directional prediction information in compressed low-resolution video bitstreams to estimate the associated high-resolution video. For intra frames, the prediction direction information is taken into account as context in the directional interpolation. When a pixel is decompressed with a small prediction residual, the interpolation is performed along its block direction. The interpolation weight for each block direction is off-line trained by the Wiener filter based on the representative video sequences. For each pixel with a large prediction residual, a piecewise autoregressive model is used as a regularization term into the interpolation function. Extensive experiments demonstrate that the proposed method achieves better performance than the traditional methods such as Bicubic, KR, LAZA, NEDI and SAI.

This paper presents an arithmetic coding scheme for DCT coefficients in video compression, in which the number of non-zero coefficients, significant map and level information for a DCT block are used as coding elements. To exploit the statistical correlations, an hierarchical dependency context model (HDCM) is proposed, where the number of non-zero coefficients and scanned position are used to capture the magnitude varying tendency of DCT coefficients. Then a new binary arithmetic coding using HDCM (HDCMBAC) is proposed to code the coding elements. Experimental results demonstrate that HDCMBAC can achieve the similar coding performance as CABAC at low and high QPs. Meanwhile the context modeling and arithmetic decoding in HDCMBAC can be carried out in parallel, since the context dependency only exists among different parts of basic coding elements in HDCM.

Gaze mismatch is a common problem in video conferencing, where the viewpoint captured by a camera (usually located above or below a display monitor) is not aligned with the gaze direction of the human subject, who typically looks at his counterpart in the center of the screen. This means that the two parties cannot converse eye-to-eye, hampering the quality of visual communication. One conventional approach to the gaze mismatch problem is to synthesize a gaze-corrected face image as viewed from center of the screen via depth-image-based rendering (DIBR), assuming texture and depth maps are available at the camera-captured viewpoint(s). Due to self-occlusion, however, there will be missing pixels in the DIBR-synthesized view image that require satisfactory filling. In this paper, we propose to jointly solve the hole-filling problem and the face beautification problem (subtle modifications of facial features to enhance attractiveness of the rendered face) via a unified dual sparse coding framework. Specifically, we first train two dictionaries separately: one for face images of the intended conference subject, one for images of “beautiful” human faces. During synthesis, we simultaneously seek two code vectors - one is sparse in the first dictionary and explains the available DIBR-synthesized pixels, the other is sparse in the second dictionary and matches well with the first vector up to a restricted linear transform. This ensures a good match with the intended target face, while increasing proximity to “beautiful” facial features to improve attractiveness. Experimental results show naturally rendered human faces with noticeably improved attractiveness.

High Efficiency Video Coding (HEVC) is the emerging video coding standard, which provides equivalent subjective quality with about 50% bit rate reduction compared to H.264/AVC High Profile. However, the improvement of coding efficiency is obtained at the expense of increased computational complexity. In this paper, a fast algorithm for HEVC intra coding is proposed. Firstly, a depth range prediction method by exploiting the correlation between the neighboring Coding Tree Units (CTUs) is proposed. Secondly, the rate distortion costs and Sum of Absolute Difference with Hadamard transform (HSAD) of recently encoded CUs are used to decide whether current CU will be further divided or not. Finally, only the intra prediction modes (IPMs) with lower precision are employed in the rough mode decision (RMD) and IPMs for rate distortion optimization (RDO) are reduced based on the correlation between neighboring CUs. Experimental results show that our proposed algorithm can achieve average 54% (up to 57%) encoding time saving while causing negligible RD performance loss (1.0% BD-rate increase on average) in All Intra High Efficiency test condition compared with HM 10.0.

High Efficiency Video Coding (HEVC) has drastically improved the coding efficiency, and can provide more than 23% bit rate reduction compared to its predecessor H264/AVC for all intra (AI) configuration. However, the improvement on coding efficiency is obtained at the expense of much more intensive computation complexity. In this paper, a fast intra optimization scheme is proposed. Firstly, a shrinkage scheme for the number of mode prediction is proposed based on gradient variance. Then a novel estimation model for rate distortion cost is provided based on Hadamard transform and quantization to better determine the mode prediction for RQT. Experimental results demonstrate that, compared with the original HEVC reference encoder implementation in AI configuration, the proposed algorithms can achieve more than 26% coding complexity reduction on average with ignorable coding performance degradation.

This paper presents a novel strategy for high-fidelity image restoration by characterizing both local smoothness and nonlocal self-similarity of natural images in a unified statistical manner. The main contributions are three-fold. First, from the perspective of image statistics, a joint statistical modeling (JSM) in an adaptive hybrid space-transform domain is established, which offers a powerful mechanism of combining local smoothness and nonlocal self-similarity simultaneously to ensure a more reliable and robust estimation. Second, a new form of minimization functional for solving the image inverse problem is formulated using JSM under a regularization-based framework. Finally, in order to make JSM tractable and robust, a new Split Bregman-based algorithm is developed to efficiently solve the above severely underdetermined inverse problem associated with theoretical proof of convergence. Extensive experiments on image inpainting, image deblurring, and mixed Gaussian plus salt-and-pepper noise removal applications verify the effectiveness of the proposed algorithm.

In this paper, we propose a novel image interpolation algorithm via graph-based Bayesian label propagation. The basic idea is to first create a graph with known and unknown pixels as vertices and with edge weights encoding the similarity between vertices, then the problem of interpolation converts to how to effectively propagate the label information from known points to unknown ones. This process can be posed as a Bayesian inference, in which we try to combine the principles of local adaptation and global consistency to obtain accurate and robust estimation. Specially, our algorithm first constructs a set of local interpolation models, which predict the intensity labels of all image samples, and a loss term will be minimized to keep the predicted labels of the available low-resolution (LR) samples sufficiently close to the original ones. Then, all of the losses evaluated in local neighborhoods are accumulated together to measure the global consistency on all samples. Moreover, a graph-Laplacian-based manifold regularization term is incorporated to penalize the global smoothness of intensity labels, such smoothing can alleviate the insufficient training of the local models and make them more robust. Finally, we construct a unified objective function to combine together the global loss of the locally linear regression, square error of prediction bias on the available LR samples, and the manifold regularization term. It can be solved with a closed-form solution as a convex optimization problem. Experimental results demonstrate that the proposed method achieves competitive performance with the state-of-the-art image interpolation algorithms.

Recovering images from corrupted observations is necessary for many real-world applications. In this paper, we propose a unified framework to perform progressive image recovery based on hybrid graph Laplacian regularized regression. We first construct a multiscale representation of the target image by Laplacian pyramid, then progressively recover the degraded image in the scale space from coarse to fine so that the sharp edges and texture can be eventually recovered. On one hand, within each scale, a graph Laplacian regularization model represented by implicit kernel is learned, which simultaneously minimizes the least square error on the measured samples and preserves the geometrical structure of the image data space. In this procedure, the intrinsic manifold structure is explicitly considered using both measured and unmeasured samples, and the nonlocal self-similarity property is utilized as a fruitful resource for abstracting a priori knowledge of the images. On the other hand, between two successive scales, the proposed model is extended to a projected high-dimensional feature space through explicit kernel mapping to describe the interscale correlation, in which the local structure regularity is learned and propagated from coarser to finer scales. In this way, the proposed algorithm gradually recovers more and more image details and edges, which could not been recovered in previous scale. We test our algorithm on one typical image recovery task: impulse noise removal. Experimental results on benchmark test images demonstrate that the proposed method achieves better performance than state-of-the-art algorithms.

Traditional patch-based sparse representation modeling of natural images usually suffer from two problems. First, it has to solve a large-scale optimization problem with high computational complexity in dictionary learning. Second, each patch is considered independently in dictionary learning and sparse coding, which ignores the relationship among patches, resulting in inaccurate sparse coding coefficients. In this paper, instead of using patch as the basic unit of sparse representation, we exploit the concept of group as the basic unit of sparse representation, which is composed of nonlocal patches with similar structures, and establish a novel sparse representation modeling of natural images, called group-based sparse representation (GSR). The proposed GSR is able to sparsely represent natural images in the domain of group, which enforces the intrinsic local sparsity and nonlocal self-similarity of images simultaneously in a unified framework. In addition, an effective self-adaptive dictionary learning method for each group with low complexity is designed, rather than dictionary learning from natural images. To make GSR tractable and robust, a split Bregman-based technique is developed to solve the proposed GSR-driven ℓ0 minimization problem for image restoration efficiently. Extensive experiments on image inpainting, image deblurring and image compressive sensing recovery manifest that the proposed GSR modeling outperforms many current state-of-the-art schemes in both peak signal-to-noise ratio and visual perception.

In this paper, we propose an efficient image super-resolution algorithm based on hierarchical and collaborative sparse representation (HCSR). Motivated by the observation that natural images typically exhibit multi-modal statistics, we propose a hierarchical sparse coding model which includes two layers: the first layer encodes individual patches, and the second layer jointly encodes the set of patches that belong to the same homogeneous subset of image space. We further present a simple alternative to achieve such target by identifying optimal sparse representation that is adaptive to specific statistics of images. Specially, we cluster images from the offline training set into regions of similar geometric structure, and model each region (cluster) by learning adaptive bases describing the patches within that cluster using principal component analysis (PCA). This cluster-specific dictionary is then exploited to optimally estimate the underlying HR pixel values using the idea of collaborative sparse coding, in which the similarity between patches in the same cluster is further considered. It conceptually and computationally remedies the limitation of many existing algorithms based on standard sparse coding, in which patches are independently encoded. Experimental results demonstrate the proposed method appears to be competitive with state-of-the-art algorithms.

In this paper, we propose a unified framework to perform progressive image restoration based on hybrid graph Laplacian regularized regression. We first construct a multi-scale representation of the target image by Laplacian pyramid, then progressively recover the degraded image in the scale space from coarse to fine so that the sharp edges and texture can be eventually recovered. On one hand, within each scale, a graph Laplacian regularization model represented by implicit kernel is learned which simultaneously minimizes the least square error on the measured samples and preserves the geometrical structure of the image data space by exploring non-local self-similarity. In this procedure, the intrinsic manifold structure is considered by using both measured and unmeasured samples. On the other hand, between two scales, the proposed model is extended to the parametric manner through explicit kernel mapping to model the inter-scale correlation, in which the local structure regularity is learned and propagated from coarser to finer scales. Experimental results on benchmark test images demonstrate that the proposed method achieves better performance than state-of-the-art image restoration algorithms.

Compressive Sensing (CS) theory shows that a signal can be decoded from many fewer measurements than suggested by the Nyquist sampling theory, when the signal is sparse in some domain. Most of conventional CS recovery approaches, however, exploited a set of fixed bases (e.g. DCT, wavelet, contour let and gradient domain) for the entirety of a signal, which are irrespective of the nonstationarity of natural signals and cannot achieve high enough degree of sparsity, thus resulting in poor rate-distortion performance. In this paper, we propose a new framework for image compressive sensing recovery via structural group sparse representation (SGSR) modeling, which enforces image sparsity and self-similarity simultaneously under a unified framework in an adaptive group domain, thus greatly confining the CS solution space. In addition, an efficient iterative shrinkage/thresholding algorithm based technique is developed to solve the above optimization problem. Experimental results demonstrate that the novel CS recovery strategy achieves significant performance improvements over the current state-of-the-art schemes and exhibits nice convergence.

In this paper, a novel quadratic ρ-domain frame layer rate control algorithm is proposed for Low Delay (LD) configuration in High Efficiency Video Coding (HEVC). Firstly, we propose a quadratic ρ-domain rate quantization (R-Q) model to establish the relationship between bit rate and quantization parameter. Subsequently, inspired by the specialized QP determination and reference picture set (RPS) mechanism, we design a RPS based hierarchical partition structure for rate control. Based on the hierarchical partition, an efficient hierarchical bit allocation scheme is provided. Finally, the quantization parameter (QP) is calculated via the proposed R-Q model after the bit allocation to meet the target bits. Experimental results demonstrate the proposed algorithm can significantly improve the R-D performance compared to the state-of-the-art rate control scheme for HEVC. Moreover, our proposed algorithm also provides smaller mismatch between the target bit rate and the actual bit rate.

In this paper, we propose to estimate the end-to-end distortion of virtual view for error-resilient depth map coding by analyzing the view-rendering process. The end-to-end distortion is estimated according to the expectation of depth distortion. The expectation of depth distortion is estimated based on the error-propagated distortion from the reference frame, which is calculated recursively for each pixel by considering the source characteristics, channel condition and error concealment method. The error-propagated distortion in terms of each frame is updated after the frame is encoded, which is used for the subsequent frames. Experimental results demonstrate that the proposed algorithm achieves substantial and consistent gains in comparison with the random intra updating method and the conventional rate-distortion method for error-resilient depth map coding.

In the pursuit of natural image prior, responses to the specific filter bank and the character of sparse representation are of the most important clues. Based on these clues, many effective and successful algorithms are proposed and widely used in low vision tasks. Up to now, the corresponding researches with these clues are developed in relatively independent ways. In this paper, taking K-SVD as an example of sparse representation and Fields of experts (FoE) as an example of responses to the specific filter bank, we demonstrate the inherent relationship between them. The filters of FoE stand for the components that fire rarely on natural images, while the redundant dictionary of K-SVD depicts the primary components to some extent. They are two complementary pursuits of natural images space. We further bridge the gap between these two methods by proposing a method to get adaptive filters for FoE from the redundant dictionary of K-SVD. Instead of pursuing the state-of-the-art performance, our research gives a suggestive and unique view point from the essence of natural image space pursuit.

We propose a sparsity-based soft decoding approach to restore compressed images directly in the transform domain of compression (DCT domain specifically examined in this paper). Restoring transform coefficients rather than pixel values prevents the propagation of quantization errors in the image domain. As natural images are statistically non-stationary with spatially varying sparse representations, we develop an adaptive block-wise sparsity-based restoration method that learns and exploits local statistics. Specially, for each DCT block, we collect sample blocks via non-local patch grouping to learn a compact dictionary based on principal component analysis. The resulting block-specific dictionary is used to estimate the corresponding DCT coefficients by a technique of collaborative sparse coding, in which the similarity between sample DCT patches used in dictionary construction is further considered. Experimental results are encouraging and demonstrate that the proposed soft decoding approach performs competitively on restoring compressed images against existing methods.

In this paper, we propose a full reference Stereoscopic Video Quality Assessment (SVQA) algorithm based on the Stereo Just-Noticeable Difference (SJND) model. Firstly, SJND mimic the human binocular visual system characteristics from four factors, including: sensitivity of luminance contrast, spatial masking, temporal masking and binocular masking. Secondly, based on the SJND model, the full reference SVQA is developed, by capturing spatio-temporal distortions and binocular perceptions. Finally, experimental results have demonstrated that the proposed SVQA outperforms other four current evaluation methods and has a good consistency with the observers' subjective perception.

In this paper, a fast multi reference frame motion estimation algorithm is proposed to reduce the computational complexity for high efficiency video coding (HEVC). Firstly, according to the reference frame distribution, we define the motion complexity for a frame. Based on the motion complexity, an initial reference frame set (RFS) is constructed to reduce the number of the reference frame for motion estimation. Then for each prediction unit (PU), the average distortion per pixel together with the motion vector difference (MVD) in the first reference frame is employed to shrink the RFS in order to early terminate the motion estimation process. In addition, to enhance the robustness of the proposed scheme, an expansion method for RFS is proposed to guarantee the video quality. Experimental results demonstrate that the proposed scheme can significantly save the encoding time. For Low Delay testing configuration, over 30% of time saving can be achieved on average with ignorable performance loss.

In real-time screen remoting system, frame rate is one of essential factors that affect user experience. Therefore, how to compress diversity of screen contents fast and efficiently is a key issue. Existing video codecs such as H.264 are always used in such a system for screen compression. However, arbitrary-sized regions with large motion always exist in typical screen content videos, which lead to a lower encoding speed and higher bit-rate, thereby decrease the frame rate. This paper proposes an efficient motion detection algorithm, which is fast and efficient for large motion regions. In specific, a region-based motion detection is used to find motion vectors instead of traditional block based motion estimation. The motion vectors are then utilized by H.264 encoder for normal motion compensated prediction. Experimental results show that the proposed algorithm can reduce both encoding time and bit-rate significantly.

A novel coding strategy for block-based compressive sensing named spatially directional predictive coding (SDPC) is proposed, which efficiently utilizes the intrinsic spatial correlation of natural images. At the encoder, for each block of compressive sensing (CS) measurements, the optimal prediction is selected from a set of prediction candidates that are generated by four designed directional predictive modes. Then, the resulting residual is processed by scalar quantization (SQ). At the decoder, the same prediction is added onto the de-quantized residuals to produce the quantized CS measurements, which is exploited for CS reconstruction. Experimental results substantiate significant improvements achieved by SDPC-plus-SQ in rate distortion performance as compared with SQ alone and DPCM-plus-SQ.

Recent years we have witnessed a striking rise of the compound video, which is generated by computers. The High Efficiency Video Coding standard (HEVC) adopts the intra transform skip mode to improve the coding efficiency of compound video. In this paper, we improve the intra transform skip mode from two aspects. Firstly, we study the statistical character of the intra prediction residual and propose a separate coefficient scan order for the intra transform skip mode. Secondly, we analyze the probability of the intra transform skip mode for different coding unit (CU) sizes and prediction unit (PU) sizes, and then propose one limitation to reduce the applied range of intra transform skip mode. The experimental results show that the first method can achieve 1.6%, 1.1%, and 0.7% luma coding gains on average in terms of BD-rate for class F under HE10-AI, HE10-RA, and HE10-LB conditions respectively; the second method can bring 9% encoder run time reduction with almost the same BD-rate changes. The proposed methods have been partially adopted into HEVC standard.

Binocular suppression theory states that the stereo vision quality is not much influenced by asymmetric degradation of the individual views. Based on these findings, mixed resolution (MR) multi-view framework jointly utilizes the lower and full resolution images to reduce the data amount, while maintaining good stereo vision quality. To enhance the resolution of the lower resolution image, a novel superresolution scheme for the MR multi-view framework is presented in the paper. It is based on the assumption that the image is modeled as a 2D piecewise auto-regressive process. In the scheme, each pixel to be interpolated is estimated as the linear weighted summation of the pixels, which are consisted of the spatial neighboring ones from lower resolution image in the current view and the ones from the full resolution image in the neighboring views. To get the corresponding pixels in the neighboring views that match the scene to be reconstructed, a window based scene-matching approach is used. Through exploiting the spatial correlation and the inter-view correlation, the proposed scheme achieves a significant gain in PSNR and visual quality for the test sequences.

The specialized QP determination and reference picture set (RPS) mechanism in High Efficiency Video Coding (HEVC) significantly improves the coding efficiency. In this paper, we propose a frame layer rate control algorithm for HEVC based on this specialized mechanism. Firstly, we design a RPS based hierarchical partition structure. Then an efficient hierarchical bit allocation scheme is proposed. Moreover, a novel header bits ratio prediction method is also provided to further improve the accuracy of bit allocation. Finally, an improved mean of absolute difference (MAD) prediction scheme is provided for quadratic R-Q model to calculate an accurate quantization parameter. Experimental results demonstrate the proposed algorithm has much better R-D performance. Compared with the state-of-the-art rate control scheme for HEVC, the BD-Rate gain can be up to 30% and BD-PSNR gain can be 0.6dB on average in Low_Delay configuration. Moreover, our proposed algorithm also provides smaller mismatch between the target bit rate and the actual bit rate.

Recently, total variation (TV) based minimization algorithms have achieved great success in compressive sensing (CS) recovery for natural images due to its virtue of preserving edges. However, the use of TV is not able to recover the fine details and textures, and often suffers from undesirable staircase artifact. To reduce these effects, this paper presents an improved TV based image CS recovery algorithm by introducing a new nonlocal regularization constraint into CS optimization problem. The nonlocal regularization is built on the well known nonlocal means (NLM) filtering and takes advantage of self-similarity in images, which helps to suppress the staircase effect and restore the fine details. Furthermore, an efficient augmented Lagrangian based algorithm is developed to solve the above combined TV and nonlocal regularization constrained problem. Experimental results demonstrate that the proposed algorithm achieves significant performance improvements over the state-of-the-art TV based algorithm in both PSNR and visual perception.

In this paper, a multi-scale face hallucination method is proposed to produce high-resolution (HR) face images from low-resolution (LR) ones according to the specific face characteristics and priors based on frequency bands analysis. In the first scale, the middle-resolution (MR) images are generated based on a patch-based learning method in DCT domain. In this scale, the DC coefficients and AC coefficients are estimated separately. In the second scale, a DCT upsampling for low frequency band restoration and a high frequency band restoration are combined to generate the final high-resolution face images. Extensive experiments show that the proposed algorithm achieves significant improvement.

One of the most striking properties of natural image statistics is their scale invariance. Intuitively, a natural image always contains the same contents of different scales and dually the same contents of same scale exist throughout scales of the image. Different from the previous scale invariance related work decomposing an image to its local band-pass filter components, this paper seeks a general model of the natural image paths distribution to describe the scale invariance in the visual world and then a novel strategy for high-fidelity image restoration is presented by characterizing nonlocal self-similarity of natural images throughout scales in a unified statistical manner, which offers a powerful mechanism of combining natural images scale invariance and nonlocal self-similarity simultaneously to ensure a more reliable and robust estimation. Extensive experiments on image restoration from partial random samples manifest that the proposed algorithm achieves significant performance improvements over the current state-of-the-art schemes.

Video broadcasting is a popular application of wireless network, whose main challenge is to accommodate different users with different channel conditions. Recently, a novel `D-Cast' approach based on distributed source coding (DSC) is proposed. It can avoid error propagation and still achieve high compression efficiency in inter frame coding by utilizing coset coding and soft broadcast. However, D-CAST is not very efficient because of rough side information. In this work, we present a novel soft mobile video broadcast approach based on side information refinement algorithm (SIR-CAST) to improve the quality of the side information. Moreover, SIR-Cast optimizes the estimate of the quantifying step (Qstep) which is corresponding to the refined side information. Thus, SIR-CAST outperforms D-CAST about 1dB-2dB in video PSNR while maintaining the similar graceful degradation feature as D-CAST.

With the rapid development of digital video technology, frame rate up conversion is widely used. In this paper, a novel motion vector refinement method for frame rate up conversion on depth based 3D video is proposed. Our method involves two major stages in frame rate up conversion which are motion estimation and motion vector filtering. In the motion estimation process, the depth constraint to block matching algorithm is introduced in the bi-directional motion estimation method to obtain the motion vectors. In the motion vector filtering process, a depth-guided filter is designed to enhance the consistence of motions in the same depth plane. The refined motion vectors are used for frame interpolation. Experimental results show that the proposed method achieves 0.45 dB gain in terms of PSNR on average and improves the visual quality of the frame rate up-converted video.

In the High Efficiency Video Coding (HEVC) based 3D video coding, 3D-HEVC, the disparity vector (DV) derivation is critical for inter-view motion prediction, inter-view residual prediction, disparity-compensated prediction (DCP) or any other tools exploiting inter-view correlation. In HTM-5.0.1, the DV is derived from some spatial and temporal neighbors to locate a corresponding block in another view. This paper advocates modifications of the DV derivation to reduce the complexity and to achieve slightly better coding efficiency. Firstly, we propose to remove the additional temporal block, unify the DV searching order for all views, and to impose restrictions on the temporal blocks for memory access bandwidth and complexity reduction in DV derivation. We also propose an improved DV searching order for slightly better coding performance. These proposed methods were adopted into the 3D-HEVC standard in the 3rd JCT-3V meeting in Jan. 2013.

In recent years, video broadcast has become a popular application, but the traditional hierarchical design requires the source to pick a bitrate and video resolution for encoding before transmission, it cannot be efficient to accommodate users with different channel quality. The proposed DCAST scheme can solve this problem. DCAST uses the ME and MC technology to generate a predicted frame which helps current frame to do coset coding. However, in the ME process, DCAST uses fixed block size. In this paper, we use the variable block size motion estimation to replace the fixed block size motion estimation, it can effectively reduce the block effect and improve the quality of the reconstructed frame generated by MC and predicted frame. The DCAST with variable block size motion estimation is 0.5dB better than DCAST with fixed block size motion estimation.

A growing societal awareness about privacy and security push the development of signal processing techniques in the encrypted domain. Data compression in encrypted domain attracts much attention recently years due to its avoiding the leakage of data source during compression. This paper proposes an improved block-by-block compression scheme of encrypted image with flexible compression ratio. The original image is encrypted by permuting the blocks of the image and then permuting the pixels in the blocks. In the compression, pixels chosen randomly used as reference information, and remaining pixels are compressed by coset code. At the decoder side, side information (SI) which is generated by combining correlation among blocks and image restoration from partial random samples (IRPRS) is utilized to assist the decompression. Moreover, an adaptive system parameters selection method is also given in this paper. The experimental results show that the proposed method can achieve a better reconstructed result compared with the earlier method.

One important problem in distributed video coding is to estimate the variance of the correlation noise between the video signal and its decoder side information. This variance is hard to estimate due to the lack of the motion vectors at the encoder side. In this paper, we first propose a linear model to estimate this variance by referring the zero motion prediction at the encoder based on a Markov field assumption. Furthermore, not only the prediction noise from the video signal itself but also the additional noise due to wireless transmission is considered in this paper. We applied our correlation estimation method in our recent distributed wireless visual communication framework called DCAST. The experimental results show that the proposed method improves the video PSNR by 0.5-1.5dB while avoiding motion estimation at encoder.

This paper proposes a novel framework called DCast for distributed video coding and transmission over wireless networks, which is different from existing distributed schemes in three aspects. First, coset quantized DCT coefficients and motion data are directly delivered to the channel coding layer without syndrome or entropy coding. Second, transmission power is directly allocated to coset data and motion data according to their distributions and magnitudes without forward error correction. Third, these data are transformed by Hadamard and then directly mapped using a dense constellation (64K-QAM) for transmission without Gray coding. One of the most important properties in this framework is that the coding and transmission rate is fixed and distortion is minimized by allocating the transmission power. Thus, we further propose a power distortion optimization algorithm to replace the traditional rate distortion optimization. This framework avoids the annoying cliff effect caused by the mismatch between transmission rate and channel condition. In multicast, each user can get approximately the best quality matching its channel condition. Our experiment results show that the proposed DCast outperforms the typical solution using H.264 over 802.11 up to 8 dB in video PSNR in video broadcast. Even in video unicast, the proposed DCast is still comparable to the typical solution.

Video broadcasting is a popular application of wireless network. However, the existing layered approaches can hardly accommodate users with diverse channel conditions as analog communication can do. The newly emerged `soft cast' approach, utilizing soft broadcast, provides smooth multicast performance but is not very efficient in inter frame compression. In this work, we propose a motion-aligned wireless video multicast scheme DCAST. Instead of using conventional close loop prediction (CLP), DCAST is based on distributed source coding (DSC) theory. This helps DCAST to avoid error propagation but still achieve high compression efficiency in inter frame coding. DCAST outperforms soft cast 5dB in video PSNR while maintaining the similar graceful degradation feature as soft cast.

In this paper, we propose a novel multi-scale spatial error concealment algorithm to combine the modeling strengthes of the parametric and nonparametric Bayesian regression. We progressively recover missing blocks in the scale space from coarse to fine so that the sharp edges and texture in the finest scale can be eventually recovered. On one hand, in each scale, the nonparametric part of the methodology is used to exploit the intra-scale correlation, which relies on the data itself to dictate the structure of the model. In this procedure, the non-local self-similarity property is utilized as a fruitful resource for abstracting a priori knowledge of images. On the other hand, the parametric part is used to explicitly model the inter-scale correlation, in which the local structure regularity is thoroughly explored to recover the sharp edges and major texture features of images. It is not respected if only the nonparametric modeling is considering. We achieve the best of both worlds within a multi-scale framework. Experimental results on benchmark test images demonstrate that the proposed method achieves very competitive performance with the state-of-the-art error concealment algorithms.

Compressed Sensing (CS) has drawn quite an amount of attention as a joint sampling and compression approach. Its theory shows that a signal can be decoded from many fewer measurements than suggested by the Nyquist sampling theory, when the signal is sparse in some domain. So one of the most significant challenges in CS is to seek a domain where a signal can exhibit a high degree of sparsity and hence be recovered faithfully. Most of conventional CS recovery approaches, however, exploited a set of fixed bases (e.g. DCT, wavelet and gradient domain) for the entirety of a signal, which are irrespective of the nonstationarity of natural signals and cannot achieve high enough degree of sparsity, thus resulting in poor rate-distortion performance. In this paper, we propose a new framework for compressed sensing recovery via collaborative sparsity (RCoS), which enforces local two-dimensional sparsity and nonlocal three-dimensional sparsity simultaneously in an adaptive hybrid space-transform domain, thus substantially utilizing intrinsic sparsities of natural images and greatly confining the CS solution space. In addition, an efficient augmented Lagrangian based technique is developed to solve the above optimization problem. Experimental results on a wide range of natural images are presented to demonstrate the efficacy of the new CS recovery strategy.

Although ordinary least squares (OLS) regression achieves great success in clean image interpolation, its effectiveness is questionable in the scenario of web images which are usually compressed beforehand. The inherent flaw of OLS is that it is asymmetric, the perturbation is only confined on the right side of the linear system. It is not reasonable for web images. Considering the drawback of OLS, in this paper, we propose an efficient web image interpolation algorithm based on total least squares (TLS) regression. In the proposed method, small perturbations are allowed in both side of the system, which are optimized by TLS in a patch-based manner. Furthermore, we develop a weighted version of TLS to consider contribution diversity of different samples and patches in model estimation, which can efficiently remove the influence of outliers in regression. Experimental results on benchmark test images demonstrate the efficiency of our method.

Motion estimation (ME) plays an important role in most video encoding systems since it could significantly affect coding performance. However, both the next generation video coding standard High Efficiency Video Coding (HEVC) and the current video coding standard H.264/MPEG-4 AVC employ block matching motion estimation (BMME) which is based on translation motion model. This makes it difficult to represent the complex motion accurately such as rotation, zoom, and etc. In this paper, we propose an adjacent-block-based prediction model to improve the prediction performance of a deformable block. Based on this model, the motion information of each 4×4 block, i.e. the minimum partition (MP) in a coding unit (CU) of HEVC, is derived from the motion information of the nearest neighbors to the four corners of current prediction unit (PU). We integrate our method into HEVC as an additional choice of its merge mode. Simulation results show that our proposed method has better performance compared to HM4.0, the BD bit rate saving is up to 15.4%, while the encoding and decoding complexities are almost the same.

Most existing image denoising algorithms can only deal with a single type of noise, which violates the fact that the noisy observed images in practice are often suffered from more than one type of noise during the process of acquisition and transmission. In this paper, we propose a new variational algorithm for mixed Gaussian-impulse noise removal by exploiting image local consistency and nonlocal consistency simultaneously. Specifically, the local consistency is measured by a hyper-Lap lace prior, enforcing the local smoothness of images, while the nonlocal consistency is measured by three-dimensional sparsity of similar blocks, enforcing the nonlocal self-similarity of natural images. Moreover, a Split-Bregman based technique is developed to solve the above optimization problem efficiently. Extensive experiments for mixed Gaussian plus impulse noise show that significant performance improvements over the current state-of-the-art schemes have been achieved, which substantiates the effectiveness of the proposed algorithm.

Stereoscopic image quality assessment has been widely studied in last decades; however, the research on 3D quality of experience (QoE) is proposed recently. As a part of human stereo perception, 3D QoE plays an important role to stereoscopic image quality assessment. In this paper, an objective metric is proposed based on the hypothesis that binocular vision system is sensitive to the structure of low-level features and its discrepancy between the two view images of a stereoscopic image pair. Specifically, the correlation between the left and right views of a stereoscopic image pair could reflect the QoE. To represent the structure of low-level features, in each view of the stereoscopic image pair, the phase congruency (PC) and the saliency map are employed as the primary and secondary features to compose a feature map. To compute the correlation between the two views, a local matching function is suggested to weight the discrepancy between the two feature maps and generate a local quality. Then these local quality values are combined to derive a single quality score. The proposed metric is evaluated on one public subjective assessment database. The experimental results indicate that our metric exhibits good performance.

Learning-based image super-resolution aims to reconstruct high-frequency (HF) details from the prior model trained by a set of high- and low-resolution image patches. In this paper, HF to be estimated is considered as a combination of two components: main high-frequency (MHF) and residual high-frequency (RHF), and we propose a novel image super-resolution method via dual-dictionary learning and sparse representation, which consists of the main dictionary learning and the residual dictionary learning, to recover MHF and RHF respectively. Extensive experimental results on test images validate that by employing the proposed two-layer progressive scheme, more image details can be recovered and much better results can be achieved than the state-of-the-art algorithms in terms of both PSNR and visual perception.

Wireless video broadcasting is a popular application of mobile network. However, the traditional approaches have limited supports to the accommodation of users with diverse channel conditions. The newly emerged Softcast approach provides smooth multicast performance but is not very efficient in inter frame compression. In this work, we propose a new video multicast approach: WaveCast. Different from softcast, WaveCast utilizes motion compensated temporal filter (MCTF) to exploit inter frame redundancy, and utilizes conventional framework to transmit motion information such that the MVs can be reconstructed losslessly. Meanwhile, WaveCast transmits the transform coefficients in lossy mode and performs gracefully in multicast. In experiments, Wave-Cast outperforms softcast 2dB in video PSNR at low channel SNR, and outperforms H.264 based framework up to 8dB in broadcast.

In this paper, we propose a novel image interpolation algorithm, which is formulated via combining both the local autoregressive (AR) model and the nonlocal adaptive 3-D sparse model as regularized constraints under the regularization framework. Estimating the high-resolution image by the local AR regularization is different from these conventional AR models, which weighted calculates the interpolation coefficients without considering the rough structural similarity between the low-resolution (LR) and high-resolution (HR) images. Then the nonlocal adaptive 3-D sparse model is formulated to regularize the interpolated HR image, which provides a way to modify these pixels with the problem of numerical stability caused by AR model. In addition, a new Split-Bregman based iterative algorithm is developed to solve the above optimization problem iteratively. Experiment results demonstrate that the proposed algorithm achieves significant performance improvements over the traditional algorithms in terms of both objective quality and visual perception.

In this paper, a practical video coding scheme is developed to realize state-of-the-art video coding efficiency with lower encoder complexity at low bit-rate, while supporting standard compliance and error resilience. Such an architecture is particularly attractive for wireless visual communications. At the encoder, multiple descriptions of a video sequence are generated in the spatio-temporal domain by temporal multiplexing and spatial adaptive downsampling. The resulting side descriptions are interleaved with each other in temporal domain, and still with conventional square sample grids in spatial domain. As such, each side description can be compressed without any change to existing video coding standards. At the decoder, each side description is first decompressed, and then reconstructed to original resolution with the help of the other side description. In this procedure, the decoder recover the original video sequence in a constrained least squares regression process, using 2D or 3D piecewise autoregressive model according to different prediction modes. In this way, the spatial and temporal correlation is sufficiently explored to achieve superior quality. Experiment results demonstrate the proposed video coding scheme outperforms H.264 in rate-distortion performance at low bit-rates and achieves superior visual quality at medium bit-rates as well.

Video transcoding is an efficient way to reduce the bitrate or convert the format of the original video stream to meet the requirements of different applications and various channel capacity. In this paper, we propose a fast multiview video transcoder (MVT) for bitrate reduction. Different from the H.264 transcoder, the inter-view prediction information in the input video stream is utilized to reduce the complexity of transcoding. Besides, we also utilize the mode and selected reference frame information in original stream to accelerate RD optimization calculations. Experimental results show that the proposed transcoder can achieve significant computation reduction while maintaining close RD performance compared to the fully decode and re-encode transcoder (FDET).

In this paper, we creatively present a viewpoint-free hand gesture recognition system based on Kinect sensor. Through depth image, we build Point Clouds of user. Then, we estimate the current optimal viewpoint, i.e., the front, and project Point Clouds to that direction. Through that process we in great extent overcome the viewpoint-dependency issue. To match hand types, we propose an improved shape context to describe each hand gesture and use the Hungarian algorithm to calculate match degree. Our method is quite straightforward, however the experimental results prove that by this means gestures can be recognized independent of viewpoints with great accuracy. Besides, it is fast and robust, thus can be applied under various realistic scenarios in realtime.

Video plus the corresponding depth maps is now the most popular data format for 3D video coding, as it is convenient to synthesize an arbitrary intermediate virtual view without the need to display the depth map for end users. How to compress the depth map efficiently is one of the key problems in 3DVC. In this paper we propose a new distortion model for depth map coding by considering the view synthesis distortion caused by depth distortion and the disparity rounding problem. Simulation results show that the proposed scheme can achieve 34% bit-rate saving for depth map coding, or 1.28 dB gain in terms of PSNR on average for synthesized virtual views.

In High Efficiency Video Coding (HEVC), advanced motion vector prediction (AMVP) is adopted to predict current motion vector by utilizing a competition-based scheme from a given candidate set, which include both the spatial and temporal motion vectors. In order to enhance the practicability of the AMVP, a simplified AMVP is proposed. Firstly, by analyzing the importance of the spatial and temporal candidates, we reduce the number of the candidates involved in the competition set and simplify the redundancy checking process, which will decrease the complexity of the decoder as well as improve the robustness of the decoder. Secondly, we simplify the zero motion adding process which will occur only when the number of existing candidates is less than the predefined number. Experimental results show that the proposed scheme provides no loss in random access and low delay conditions. These two simplifications have been proposed and adopted into the HEVC standard.

Frame rate up-conversion (FRUC) improves the viewing experience of a video because the motion in a FRUC-constructed high frame-rate video looks more smooth and continuous. This paper proposes a multiple hypotheses Bayesian FRUC scheme for estimating the intermediate frame with maximum a posteriori probability, in which both temporal motion model and spatial image model are incorporated into the optimization criterion. The image model describes the spatial structure of neighboring pixels while the motion model describes the temporal correlation of pixels along motion trajectories. Instead of employing a single uniquely optimal motion, multiple “optimal” motion trajectories are utilized to form a group of motion hypotheses. To obtain accurate estimation for the pixels in missing intermediate frames, the motion-compensated interpolations generated by all these motion hypotheses are adaptively fused according to the reliability of each hypothesis. We revealed by numerical analysis that this reliability (i.e., the variance of interpolation errors along the hypothesized motion trajectory) can be measured by the variation of reference pixels along the motion trajectory. To obtain the multiple motion fields, a set of block-matching sizes is used and the motion fields are estimated by progressively reducing the size of matching block. Experimental results show that the proposed method can significantly improve both the objective and the subjective quality of the constructed high frame rate video.

Video transcoding is proposed for the bitrate adaption, spatial and/or temporal resolutions adaption, and video format conversion. In video streaming application, it converts videos at server to the compatible versions demanded by networks or clients' devices, so that the videos can be delivered over networks and displayed in the clients' devices successfully. This paper provides a universal rate control scheme for various video transcoding purposes. First, a new rate-distortion (R-D) model is established theoretically for better representing the real R-D feature of transcoding. Second, a window-level rate control algorithm is proposed for providing smooth visual quality with compliant buffer constraint by utilizing the two-pass R-D model and a new proposed sliding window buffer control strategy. Finally, a universal rate control scheme for transcoding is developed based on the established R-D model and the proposed window-level rate control algorithm. The extensive experimental results demonstrate that as compared to other state-of-the-art rate control algorithms for transcoding, the proposed scheme can achieve more bit control accuracy with the average mismatch below 0.2%, and much more consistent visual quality with 0.1 dB-0.3 dB peak-to-signal noise ratio improvement in average, while with low computational complexity.

In this paper, auto regressive (AR) model is applied to error concealment for block-based packet video coding. In the proposed error concealment scheme, the motion vector for each corrupted block is first derived by any kind of recovery algorithms. Then each pixel within the corrupted block is replenished as the weighted summation of pixels within a square centered at the pixel indicated by the derived motion vector in a regression manner. Two block-dependent AR coefficient derivation algorithms under spatial and temporal continuity constraints are proposed respectively. The first one derives the AR coefficients via minimizing the summation of the weighted square errors within all the available neighboring blocks under the spatial continuity constraint. The confidence weight of each pixel sample within the available neighboring blocks is inversely proportional to the distance between the sample and the corrupted block. The second one derives the AR coefficients by minimizing the summation of the weighted square errors within an extended block in the previous frame along the motion trajectory under the temporal continuity constraint. The confidence weight of each extended sample is inversely proportional to the distance toward the corresponding motion aligned block whereas the confidence weight of each sample within the motion aligned block is set to be one. The regression results generated by the two algorithms are then merged to form the ultimate restorations. Various experimental results demonstrate that the proposed error concealment strategy is able to improve both the objective and subjective quality of the replenished blocks compared to other methods.

In this paper, we present a pose based approach for locating and recognizing human actions in videos. In our method, human poses are detected and represented based on deformable part model. To our knowledge, this is the first work on exploring the effectiveness of deformable part models in combining human detection and pose estimation into action recognition. Comparing with previous methods, ours have three main advantages. First, our method does not rely on any assumption on video preprocessing quality, such as satisfactory foreground segmentation or reliable tracking; Second, we propose a novel compact representation for human pose which works together with human detection and can well represent the spatial and temporal structures inside an action; Third, with human detection taken into consideration in our framework, our method has the ability to locate and recognize multiple actions in the same scene. Experiments on benchmark datasets and recorded cluttered videos verified the efficacy of our method.

In this paper, we propose a novel image super-resolution algorithm, referred to as interpolation based on transductive regression with local and global consistency (TRLGC). Our algorithm first constructs a set of local interpolation models which can predict the intensity labels of all image samples, and a loss term will be minimized to keep the predicted labels of available low-resolution (LR) samples sufficiently close to the original ones. Then, all of the losses evaluated in local neighborhoods are accumulated together to measure the global consistency on all samples. Furthermore, a graph-Laplacian based manifold regularization term is incorporated to penalize the global smoothness of intensity labels, such smoothing can alleviate the insufficient training of the local models and make them more robust. Finally, we construct a unified objective function to combine together the accumulated loss of the locally linear regression, square error of prediction bias on the available LR samples and the manifold regularization term, which could be solved with a closed-form solution as a convex optimization problem. In this way, a transductive regression algorithm with local and global consistency is developed. Experimental results on benchmark test images demonstrate that the proposed image super-resolution method achieves very competitive performance with the state-of-the-art algorithms.

Summary form only given. In low bit rate video coding, the frame rate of input sequence can be reduced to the half or even smaller portion by skipping or deleting frames before compression, and then the temporal resolution is restored via up-sampling at the decoder side. Numerous algorithms have been developed to address the problem of temporal resolution improvement. Actually, the quality of up-sampled frames depends on not only the performance of up-sampling method but also the information maintained in the down-sampled video sequence. To improve the quality of up-sampled frames and smooth the quality between the up-sampled and decompressed frames, this paper proposes an up-sampling dependent frame rate reduction, which is shown in Fig. 1. The proposed low bit rate video coding scheme is composed of up-sampling dependent frame rate reduction, compression, decompression and up-sampling components. The proposed frame rate reduction method is hinged to the temporal up-sampling. It is noted that there is a feedback between frame rate reduction and up-sampling in the proposed up-sampling dependent frame rate reduction, of which the goal is to obtain a down-sampled sequence maintaining more information about the frames to be up-sampled at the decoder side.

Stereoscopic video is an important manner for 3-D video applications, and robust stereoscopic video transmission has posed a technical challenge for stereoscopic video coding. In this paper, an auto-regressive (AR) model based error concealment scheme is proposed for stereoscopic video coding to address the challenging problem. The proposed error concealment scheme includes a temporal AR model for independent view, and a temporal-interview AR model for inter-view predicted view. First, appropriate motions and disparities for lost blocks are derived. Then, the proposed AR model coefficients are computed according to the spatial neighboring pixels and their temporal-correlated and interview-correlated pixels indicated by the selected prediction directions. Finally, applying the AR model, each pixel of the lost block is interpolated as a weighted summation of pixels in the reference frame along the selected prediction directions. Simulation results show that the performance of the proposed scheme is superior to conventional temporal error concealment methods for stereoscopic video coding.

Saliency detection is essential to visual attention modelling and various computer vision tasks. Representation and measurement are two important issues for saliency models. Good representation and reasonable measurement are both critical issues in modelling visual saliency mechanism. For every input image, we obtain a self-adaptive dictionary that describes the image content effectively and image prior that forces sparsity in every location in the image using the K-SVD algorithm. For saliency measurement, background firing rate (BFR) is defined for each sparse features and it is followed by feature activation rate (FAR) computation to measure the bottom-up visual saliency.

Inspired by the success of structural similarity index (SSIM), some image quality assessment (IQA) methods have been developed recently. To achieve better performance, this paper proposes a new visual attention (VA) model that combines saliency based VA and visual importance based VA, under the assumptions that humans often pay more attention to the regions with important content in the beginning of evaluating a given image and then the regions with poor quality. Then the proposed VA model is incorporated into SSIM. The experiments on LIVE database and TID2008 database demonstrate its improvements over the latest state-of-the-art IQA methods and the information content weighted SSIM measure (IW-SSIM).

For video frame rate up-conversion, new frames need to be interpolated along motion trajectories and inserted between the original adjacent frames. This paper proposes a Bayesian frame interpolation strategy, which considers both the spatial (i.e. intra-frame) correlation model and the temporal (i.e. inter-frame) correlation model after motion compensation. Different from the conventional schemes that perform interpolation using only one estimated motion vector field (MVF), the proposed strategy estimates multiple MVFs. To cope with different scales of frame contents, we generate the multiple MVFs through variable block size motion estimation. A criterion is adopted to estimate reliability of the motion compensated prediction frames from different motion fields. Experimental results demonstrate that the proposed strategy improves the interpolation performance remarkably.

In this paper, we present an efficient side information extrapolation scheme with temporal and spatial consistency for low delay Wyner-Ziv video coding. Our method is based on the regularized local linear regression (RLLR) model, in which each pixel in SI is approximated as a linear weighted combination of samples within a local temporal neighborhood. The optimal model parameters are estimated by projecting the transformation function onto the temporal training samples to exploit motion-related dependency. During this procedure, moving weights are incorporated into the objective function to express the relative importance of training samples in estimating parameters of the model. Furthermore, spatial correlation is explored by imposing an additional local smoothness penalty, which does good to estimate the occluded regions and complex motion regions. The learned function is smooth and locally linear, and can be obtained with a closed-form solution by solving a convex optimization problem. Experimental results demonstrate that the RLLR method achieves very competitive SI extrapolation performance compared with the state-of-the-art methods.

Multiview video coding (MVC) improves the coding efficiency by motion estimation (ME) and disparity estimation (DE). ME and DE at encoder side involve in heavy computation, which needs to be further reduced for practical applications. This paper presents fast disparity estimation by utilizing depth information to reduce DE's computational complexity. First, the coordinate offset of the encoding block is calculated by 3D warping in the reference view frame utilizing depth map. Then the search region for DE is adjusted according to the coordinate offset. Experimental results verify that the proposed algorithm can save roughly 50% coding time with negligible loss of coding efficiency.

Towards a compact video feature representation, we propose a novel feature selection methodology for action recognition based on the saliency maps of videos. Since saliency maps measure the perceptual importance of the pixels and regions in videos, selecting features using saliency maps enables us to find a feature representation that covers the informative parts of a video. Because saliency detection is a bottom-up procedure, some appearance changes or motions that are irrelevant to actions may also be detected as salient regions. To further improve the purity of the feature representation, we prune these irrelevant salient regions using the saliency values distribution and the spatial-temporal distribution of the salient regions. Extensive experiments are conducted to demonstrate that the proposed feature selection method largely improves the performance of bag-of-video-words model on action recognition based on three different attention models including a static attention model, a motion attention model and their combination.

Mobile video broadcasting is a popular application of wireless network. However, the traditional layer based approaches have limited supports to the accommodation of users with diverse channel conditions. The newly emerged Softcast approach provides smooth multicast performance but is not very efficient in inter frame compression. In this work, we propose a new video multicast approach: D-cast. Instead of using conventional close loop prediction (CLP), D-cast is based on distributed source coding (DSC) theory. This helps D-cast to avoid error propagation but still achieve high compression efficiency in inter frame coding. D-cast outperforms softcast 3--5dB in video PSNR while maintaining the similar graceful degradation feature as softcast. In addition, D-cast is efficient in not only narrow band channel but also wide band channel.

Context-adaptive binary arithmetic coding (CABAC) is one of the most time-consuming modules in H.264/AVC decoder. A potential way to accelerate CABAC is by parallelization. However, the context modeling process for level information in CABAC is highly serial in nature and can not be parallelized in the coefficient level. In order to improve the throughput of CABAC, in this paper we present a parallel context model for level information. The key feature of the model is to use the total number of the significant coefficients and the scanned position of the current significant coefficient in the quantized transform coefficient block as the context information. Since the context information is independent of the previously decoded significant coefficients, parallel decoding in coefficient level is achieved. In experiments, the proposed context model achieves the similar compression efficiency as the CABAC.

In this paper, a mode-dependent intra frame interpolation method is proposed for H.264/AVC compressed video. The intra prediction mode information is taken into account in the interpolation filter design. For each intra prediction mode, an optimal Wiener filter is trained based on the representative video sequences. Therefore the trained filter is adaptive to the intra prediction mode. Furthermore, the quantization parameter is also explored as context information for filter selection. Extensive experiments demonstrate that the proposed method achieves better performance than the traditional methods such as Bicubic, Bilinear, LAZA and NEDI, while keeping low computational complexity.

Just noticeable difference (JND) model can reflect the least perceptible distortion from images, including 2D images and stereoscopic images. As we know, for the perception of human visual system (HVS), stereoscopic images have quite different characteristics from 2D images, since stereoscopic images contain not only planar information, but also depth information. This paper proposes a joint JND (JJND) model based on depth perception for stereoscopic images. Firstly, disparity estimation is performed in order to decompose the image into the occlusion region and the non-overlapped region. Then, different JND thresholds are applied on different regions, according to the depth information of the region, which can be derived from the disparity of the region. Experimental results verified our model's validity for stereoscopic images.

In this paper, a novel algorithm for high-quality image restoration is proposed. The contributions of this work are two-fold. First, a new form of minimization function for solving image inverse problems is formulated via combining local total variation model and nonlocal adaptive 3-D sparse representation model as regularizers under the regularization- based framework. Second, a new Split-Bregman based iterative algorithm is developed to solve the above optimization problem efficiently associated with proved theoretical convergence property. Experimental results on image restoration from partial random samples have shown that the proposed algorithm achieves significant performance improvements over the current state-of-the-art schemes and exhibits nice convergence property.

As the next generation standard of video coding, the High Efficiency Video Coding (HEVC) is intended to provide significantly better coding efficiency than all existing video coding standards. To improve the coding efficiency of intra frame coding, up to 34 intra prediction modes are defined in HEVC. The best mode among these pre-defined intra prediction modes is selected by rate-distortion optimization (RDO) for each block. If all directions are tested in the RDO process, it will be very time-consuming. To alleviate the encoder computation load, this paper proposes a new method to reduce the candidates in RDO process. In addition, the direction information of the neighboring blocks is made full use of to speed up intra mode decision. Experimental results show that the proposed scheme provides 20% and 28% time savings in intra high efficiency and low complexity cases on average compared to the default encoding scheme in HM 1.0 with almost the same coding efficiency. This algorithm has been proposed to HEVC standard and partially adopted into the HEVC test model.

Inspired by Witsenhausen and Wyner's 1980 (now expired) patent on “interframe coder for video signals,” this paper presents a Witsenhausen-Wyner video codec, where the motion-compensated previously decoded video frame is used at the decoder as side information for joint decoding. Specifically, we replace predictive Inter coding in H.264/AVC by the syndrome-based coding scheme of Witsenhausen and Wyner, while keeping the Intra and Skip modes of H.264/AVC unchanged. We employ forward motion estimation at the encoder and send the motion vectors to help generate side information at the decoder, since our focus is not on low-complexity encoding. We also examine the tradeoff between the motion vector resolution and coding efficiency. Within the Witsenhausen-Wyner coding mode, we optimize the decision between syndrome coding and entropy coding among different discrete cosine transform (DCT) bands and among different bit-planes within each DCT coefficient. Extensive simulations of video transmission over wireless networks show that Witsenhausen-Wyner video coding is more robust against channel errors than H.264/AVC. The price paid for enhanced error-resilience with Witsenhausen-Wyner coding is a small loss in compression efficiency.

The linear regression model is a very attractive tool to design effective image interpolation schemes. Some regression-based image interpolation algorithms have been proposed in the literature, in which the objective functions are optimized by ordinary least squares (OLS). However, it is shown that interpolation with OLS may have some undesirable properties from a robustness point of view: even small amounts of outliers can dramatically affect the estimates. To address these issues, in this paper we propose a novel image interpolation algorithm based on regularized local linear regression (RLLR). Starting with the linear regression model where we replace the OLS error norm with the moving least squares (MLS) error norm leads to a robust estimator of local image structure. To keep the solution stable and avoid overfitting, we incorporate the l(2)-norm as the estimator complexity penalty. Moreover, motivated by recent progress on manifold-based semi-supervised learning, we explicitly consider the intrinsic manifold structure by making use of both measured and unmeasured data points. Specifically, our framework incorporates the geometric structure of the marginal probability distribution induced by unmeasured samples as an additional local smoothness preserving constraint. The optimal model parameters can be obtained with a closed-form solution by solving a convex optimization problem. Experimental results on benchmark test images demonstrate that the proposed method achieves very competitive performance with the state-of-the-art interpolation algorithms, especially in image edge structure preservation.

In rate control, smooth picture quality and smooth buffer occupancy are both important but contrary to each other at a given bit rate. How to get a good tradeoff between them was not devoted much attention previously. To deal with this problem, a theoretical window model is proposed in this paper, in which several adjacent frames grouped as a window are considered together. The smoothness of both picture quality and buffer occupancy can be gracefully achieved by regulating the size of the window. To illustrate the usage of window model, a window-level rate control algorithm cooperated with the traditional ρ-domain rate-distortion model is further introduced. In experiments, we first show how the proposed window model achieves the tradeoff between picture quality smoothness and buffer smoothness, and then demonstrate the significant PSNR improvement, accuracy of bit control and consistency of visual quality of the proposed window-level rate control algorithm.

Traditional methods for image downsampling commit to remove the aliasing artifacts. However, the influences on the quality of the image interpolated from the downsampled one are usually neglected. To tackle this problem, in this paper, we propose an interpolation-dependent image downsampling (IDID), where interpolation is hinged to downsampling. Given an interpolation method, the goal of IDID is to obtain a downsampled image that minimizes the sum of square errors between the input image and the one interpolated from the corresponding downsampled image. Utilizing a least squares algorithm, the solution of IDID is derived as the inverse operator of upsampling. We also devise a content-dependent IDID for the interpolation methods with varying interpolation coefficients. Numerous experimental results demonstrate the viability and efficiency of the proposed IDID.

Summary form only given. In this paper, an error resilient JU-DFMC is proposed for video transmission over error-prone channels. In the proposed error resilient JU-DFMC, a new error resilient prediction structure of DFMC is firstly presented. The LQF can adaptively select reference frame according to different packet loss rate. Then the MB information is divided into two partition header information (A) and texture coefficients (B). Based on the partition, an end-to-end distortion model is applied for macroblock (MB) level mode decision. Finally a frame level rate distortion cost scheme is proposed to determine how many times the header information will be transmitted in a high quality frame (HQF). The HQF (LTR) is given more protection. The experimental results show that the proposed method can achieve better performance than the previous DFMC schemes. In the future, how to determine LQF header transmission times will be further exploited.

In this paper, auto regressive (AR) model is applied to error concealment for block-based packet video encoding. Each pixel within the corrupted block is restored as the weighted summation of corresponding pixels within the previous frame in a linear regression manner. Two novel algorithms using weighted least squares method are proposed to derive the AR coefficients. First, we present a coefficient derivation algorithm under the spatial continuity constraint, in which the summation of the weighted square errors within the available neighboring blocks is minimized. The confident weight of each sample is inversely proportional to the distance between the sample and the corrupted block. Second, we provide a coefficient derivation algorithm under the temporal continuity constraint, where the summation of the weighted square errors around the target pixel within the previous frame is minimized. The confident weight of each sample is proportional to the similarity of geometric proximity as well as the intensity gray level. The regression results generated by the two algorithms are then merged to form the ultimate restorations. Various experimental results demonstrate that the proposed error concealment strategy is able to increase the peak signal-to-noise ratio (PSNR) compared to other methods.

Intra prediction is one effective method to remove the spatial redundancies in intra frame coding. Better intra prediction will result in the residual with less energy, which will decrease the number of bits needed to reconstruct the signal at decoder. To improve the accuracy of intra prediction, a context-adaptive pixel based prediction (CAPBP) algorithm is proposed in this paper. For each pixel within the target block to be encoded, the prediction is calculated as the linear weighted summation of the reconstructed pixels within the left column and the above row. Based on the assumption that pixels having the same coordinates within one block own the same prediction weights, we calculate the corresponding weights for each pixel within the target block by the least square method. The same processing is also performed at decoder; hence the prediction weights do not need to be sent to the decoder. Experimental results verify that the proposed algorithm is able to improve the efficiency of intra frame coding up to 0.5dB.

The Random forest classifier comes to be the working horse for visual recognition community. It predicts the class label of an input data by aggregating the votes of multiple tree classifiers. However, the classification performances of these tree classifiers are different. The random forest classifier ignores the difference by simply assigning them equal weights in voting for the final classification decision. Also, the random forest classifier only casts votes from individual tree classifiers without considering their compositions which would be more accurate. In this paper, we propose to tackle the two points by discovering weighted decision rules from the tree classifiers' output sets on training data. By treating the outputs of the tree classifiers on each data as a digital itemset, we want to find discriminative patterns (either containing the output of a single tree classifier or a set of tree classifiers) from the itemsets of training data. We employ an efficient data mining algorithm, the Emerging Pattern (EP) Mining, to search such discriminative patterns and weight them according to their discriminative powers. A set of decision rules are built from these discovered patterns and the final outputs of the Random Forest are made using these decision rules. We call the proposed classifier Emerging Pattern (EP) Random Forest. Experimental results on action categorization problems confirm that the proposed method really improve the performance of the traditional Random Forest classifier.

Distributed video coding (DVC) was proposed to meet the low complexity encoding requirement, and it was verified to work more efficiently than H.264/AVC intra coding on video sequences with low motion. This makes DVC suitable for surveillance application. This paper presents a background aided surveillance-oriented distributed video coding system. A high quality background frame is encoded for each group of pictures (GOP), which can provides high quality SI for the background parts of the Wyner-Ziv (WZ) frames. Consequently, bit rate for the WZ frames can be reduced. Experimental results demonstrate that the proposed system can decrease the bit rate by up to 67.4% when compared with traditional DVC codec.

In this paper, we present an efficient image interpolation scheme by using regularized local linear regression (RLLR). On one hand, we introduce a robust estimator of local image structure based on moving least squares, which can efficiently handle the statistical outliers compared with ordinary least squares based methods. On the other hand, motivated by recent progress on manifold based semi-supervise learning, the intrinsic manifold structure is explicitly considered by making use of both measured and unmeasured data points. In particular, the geometric structure of the marginal probability distribution induced by unmeasured samples is incorporated as an additional locality preserving constraint. The optimal model parameters can be obtained with a closed-form solution by solving a convex optimization problem. Experimental results demonstrate that our method outperform the existing methods in both objective and subjective visual quality over a wide range of test images.

Humans are the ultimate receivers of the visual information contained in an image, so the reasonable method of image quality assessment (IQA) should follow the properties of the human visual system (HVS). In recent years, IQA methods based on HVS-models are slowly replacing classical schemes, such as mean squared error (MSE) and Peak Signal-to-Noise Ratio (PSNR). IQA-structural similarity (SSIM) regarded as one of the most popular HVS-based methods of full reference IQA has apparent improvements in performance compared with traditional metrics in nature, however, it performs not very well when the images' structure is destroyed seriously or masked by noise. In this paper, a new efficient fovea based structure similarity image quality assessment (FSSIM) is proposed. It enlarges the distortions in the concerned positions adaptively and changes the importances of the three components in SSIM. FSSIM predicts the quality of an image through three steps. First, it computes the luminance, contrast and structure comparison terms; second, it computes the saliency map by extracting the fovea information from the reference image with the features of HVS; third, it pools the above three terms according to the processed saliency map. Finally, a commonly experimental database LIVE IQA is used for evaluating the performance of the FSSIM. Experimental results indicate that the consistency and relevance between FSSIM and mean opinion score (MOS) are both better than SSIM and PSNR clearly.

Recently multiview video coding (MVC) standard has been finalized as an extension of H.264/AVC by Joint Video Team (JVT). In the project Joint Multiview Video Model (JMVM) for the standardization, illumination compensation (IC) is adopted as a useful tool. In this paper, a novel illumination compensation algorithm based on template is proposed. The basic idea of the algorithm is that the illumination of the current block has a strong correlation with its adjacent template. Based on this idea, firstly a template based illumination compensation method is presented, and then a template models selection strategy is devised to improve the illumination compensation performance. The experimental results show that the proposed algorithm can improve the coding efficiency significantly.

It is a challenging problem to robustly track moving objects from image sequences because of occlusions. Previous methods did not exploit depth information sufficiently. Based on multiple camera scenes, we propose a 3D silhouette tracking framework to resolve occlusions and recover the appearances in 3D space, which enhances tracking effectiveness. In the framework, 2D object silhouettes are initially gained by Snake. Then a Voxel Space Carving procedure is introduced to simultaneously generate the occlusion model and visual hull of objects. Next, we adopt Particle Filter to select the valuable parts of occlusion model and combine them with the initial object silhouettes to generate the updated visual hull. Finally, updated visual hull of the objects are re-projected to each view to obtain their final contours. The experiments under the public LAB and SCULPTURE datasets validate the feasibility and effectiveness of our framework.

This paper investigates how to exploit the artifacts constraint to enhance the quality of side information (SI) in distributed video coding (DVC). The idea originates from the observation that there are usually some regions shown as artifacts in the interpolated SI, which seriously degrade the overall performance of the DVC system, especially when the GOP size is large. Encouraged by the previous work about the backward channel in the literature, we propose a simple and effective DVC scheme based on artifact detection and removal. On one hand, texture analysis via dynamic range and local entropy is utilized to determine artifact regions in spatial domain. On the other hand, motion activity evaluation is used to determine artifact regions in temporal domain, and temporal local entropy difference is further employed to eliminate the effect of other factors like brightness variance. After the artifact blocks are detected, we re-code them in intra mode and replace the co-located ones at the decoder side so that the quality of SI is enhanced. Experimental results demonstrate that the proposed SI quality enhancement technique is effective.

H.264/AVC adopts 9 intra 4×4 prediction modes to improve the intra frame compression efficiency. This makes the intra frame coding and transcoding very complex. In this paper, a fast intra 4×4 mode decision algorithm is proposed to reduce the intra transcoding complexity. Firstly, the surrounding modes in the high bitrate and the modes of the neighboring blocks in the current bitrate are used to form the candidate modes. Secondly, the variance and mean of the intra prediction residual are used to further reduce the size of the candidate mode set. Thirdly, the intra mode context model is built from the high bitrate modes and updated by the low bitrate modes to order the candidate modes. An early skip scheme is used to terminate the mode decision process. Experimental results show that about 50% intra 4×4 modes can be saved with a penalty on coding efficiency less than 0.1dB. The transcoding performance can even gain up to 0.6 dB on some sequences.

Images encoded at low bit rate may suffer from blocking artifacts, which dramatically degrade the visual quality. In this paper, we propose a postprocessing image restoration scheme for JPEG compressed images based on the maximum a posteriori criterion. A degradation model, represent by additive Gaussian noise model, is proposed to simulate JPEG compression process, while the original image is modeled as a high order Markov random field (MRF) based on the fields of experts framework. Meanwhile, an enhancement algorithm is proposed to restore the high frequency (HF) components. Experiment results have demonstrated that the proposed scheme can reproduce higher-quality images in terms of both objective and subjective quality.

An interpolation oriented adaptive down-sampling algorithm is proposed for low bit-rate image coding in this paper. Given an image, the proposed algorithm is able to obtain a low resolution image, from which a high quality image with the same resolution as the input image can be interpolated. Different from the traditional down-sampling algorithms, which are independent from the interpolation process, the proposed down-sampling algorithm hinges the down-sampling to the interpolation process. Consequently, the proposed down-sampling algorithm is able to maintain the original information of the input image to the largest extent. The down-sampled image is then fed into JPEG. A total variation (TV) based post processing is then applied to the decompressed low resolution image. Ultimately, the processed image is interpolated to maintain the original resolution of the input image. Experimental results verify that utilizing the downsampled image by the proposed algorithm, an interpolated image with much higher quality can be achieved. Besides, the proposed algorithm is able to achieve superior performance than JPEG for low bit rate image coding.

Sign language recognition systems suffer from the problem of signer dependence. In this letter, we propose a novel method that adapts the original model set to a specific signer with his/her small amount of training data. First, affinity propagation is used to extract the exemplars of signer independent hidden Markov models; then the adaptive training vocabulary can be automatically formed. Based on the collected sign gestures of the new vocabulary, the combination of maximum a posteriori and iterative vector field smoothing is utilized to generate signer-adapted models. Experimental results on six signers demonstrate that the proposed method can reduce the amount of the adaptation data and still can achieve high recognition performance.

A motion-compensated deinterlacing scheme based on hierarchical motion analysis is presented. According to deinterlacing steps, our contribution can be divided into four parts: motion estimation, motion state analysis, motion consistency analysis, and finer-grained interpolation. In motion estimation, we introduce a Gaussian noise model for choosing the best motion vector for each block, and make a tradeoff between utilizing previous de-interlaced frames and avoiding error propagation. A directional interpolation method is also introduced in this part for backward fields. In motion state analysis, we define two motion states for each pixel, thus achieve a compromise between traditional block-based strategies and the extreme pixel-based case. In motion consistency analysis, we propose to measure both the motion vector consistency and the motion state consistency in order to determine whether the previous two parts should be performed again with a different block size. In finer-grained interpolation, we utilize a combination of recursive median filters to generate the final results. Experimental results show that all of the proposed techniques are effective, either objectively or subjectively. As a result, we can achieve much higher image quality, with an average gain of about 1.83 dB in terms of peak signal-to-noise ratio. Moreover, the increased computation complexity is marginal.

In dual frame motion compensation (DFMC), one short-term reference frame and one long-term reference frame (LTR) are utilized for motion compensation. The performance of DFMC is heavily influenced by the jump updating parameter and bit allocation for the reference frames. In this paper, first the rate-distortion performance analysis of motion compensated prediction in DFMC is presented. Based on this analysis, an adaptive jump updating DFMC (JU-DFMC) with optimal LTR selection and bit allocation is proposed. Subsequently, an error resilient JU-DFMC is further presented based on the error propagation analysis of the proposed adaptive JU-DFMC. The experimental results show that the proposed adaptive JU-DFMC achieves better performance over the existing JU-DFMC schemes and the normal DFMC scheme, in which the temporally most recently decoded two frames are used as the references. The performance of the adaptive JU-DFMC is significantly improved for video transmission over noisy channels when the specified error resilience functionality is introduced.

In the above titled paper (ibid., vol. 19, no. 9, pp. 1289-1301, Sep. 09), the column under 3-DRS in Table II is incorrect due to the author's table editing error. The correct table is presented here.

In this paper, a motion-aligned auto-regressive (MAAR) model is proposed for frame rate up conversion, where each pixel is interpolated as the average of the results generated by one forward MAAR (Fw-MAAR) model and one backward MAAR (Bw-MAAR) model. In the Fw-MAAR model, each pixel in the to-be-interpolated frame is generated as a linear weighted summation of the pixels within a motion-aligned square neighborhood in the previous frame. To derive more accurate interpolation weights, the aligned actual pixels in the following frame are also estimated as a linear weighted summation of the newly interpolated pixels in the to-be-interpolated frame by the same weights. Consequently, the backward-aligned actual pixels in the following frame can be estimated as a weighted summation of the corresponding pixels within an enlarged square neighborhood in the previous frame. The Bw-MAAR is performed likewise except that it is operated in the reverse direction. A damping Newton algorithm is then proposed to compute the adaptive interpolation weights for the Fw-MAAR and Bw-MAAR models. Extensive experiments demonstrate that the proposed MAAR model is able to achieve superior performance than the traditional frame interpolation methods such as MCI, OBMC, and AOBMC, and it is even better than STAR model for the most test sequences with moderate or large motions.

The goal of steganography is to hide information into media without disclosing the fact of existing communication. Currently, stganography such as least significant bit (LSB), quantization index modulation (QIM) and spread spectrum (SS), has become increasingly widespread. Steganalysis as a counterpart of stganography is to detect the presence of it. In this paper, we present a new universal steganalysis method based on statistical models of the imagepsilas discrete cosine transform (DCT) coefficients. In fact, the block-based DCT by proper reorganization of its coefficients can have similar characteristics to wavelet transforms. The presented universal steganalysis method utilizes these characteristics to build statistical models of the image and its prediction-error image. Features extracted from the re-organization DCT blocks of host images and theirs prediction-error images and features extracted from steg images and theirs prediction-error images are used to train the SVM classifier. In the testing, features from those potential images are inputted the trained-well classifier to determine where the potential images are stego images or not. The experiments have shown that the proposed method outperforms in general prior-arts of steganalysis methods based on wavelet transform domain.

In 3D video applications, the virtual view is generally rendered by the compressed texture and depth. The texture and depth compression with different bit-rate overheads can lead to different virtual view rendering qualities. In this paper, we analyze the compression-induced rendering distortion for the virtual view. Based on the 3D warping principle, we first address how the texture and depth compression affects the virtual view quality, and then derive an upper bound for the compression-induced rendering distortion. The derived distortion bound depends on the compression-induced depth error and texture intensity error. Simulation results demonstrate that the theoretical upper bound is an approximate indication of the rendering quality and can be used to guide sequence-level texture/depth rate allocation for 3D video compression.

No-reference measurement of blurring artifacts in images is a challenging problem in image quality assessment field. One of the difficulties is that the inherently blurry regions in some natural images may disturb the evaluation of blurring artifacts. In this paper, we study the image gradients along local image structures and propose a new perceptual blur metric to deal with the above problem. The gradient profile sharpness of image edge is efficiently calculated along horizontal or vertical direction. Then the sharpness distribution histogram rectified by just noticeable distortion (JND) threshold is used to evaluate the blurring artifacts and assess the image quality. Experimental results show that the proposed method can achieve good image quality prediction performance.

The coding efficiency of distributed video coding system is significantly determined by the side information quality and correlation model. Motivated by theoretical analysis of the maximum likelihood treatment for linear regression model, we propose a novel joint online learning model for side information generation and correlation model estimation in this paper. In our proposed scheme, each pixel in the side information is approximated as the linear weighted combination of samples within a local spatio-temporal neighboring space. Weights are trained in a self-feedback fashion, during which the correlation model parameters can also be achieved. The efficiency of the proposed joint learning model is confirmed experimentally.

Recently, multi-view distributed video coding (MDVC) receives more and more attention, as its low-complexity encoder and high-complexity decoder coding paradigm suits for many applications such as sensor networks, in which several view sequences are required to be coded by a few power-constraint encoders. Modeling the correlation noises between original frame and side information frame is a hot research issue in distributed video coding (DVC), since it is a vital factor affecting the coding efficiency. This paper firstly proposes a novel method to model the correlation noises in MDVC. And an algorithm to online estimate the model at decoder using the knowledge of adjacent views is also presented. Experiment results show that the proposed correlation model can significantly improve coding efficiency.

In H.264/AVC, the bits used for the Intra_4times4 prediction mode information usually occupy a high percentage in intra coding. Towards this issue, we present an efficient coding method for the Intra_4times4 prediction mode information. Firstly, a 3-order Markov random field is introduced to model the correlation among neighboring 4times4 blocks at picture level. Secondly, based on the conditional probabilities learned in this model, we build up a context adaptive coding scheme to code the Intra_4times4 prediction mode information. Although the probabilities and the coding scheme are initialized off-line, they can be revised by automatic adjustments. Thus the proposed algorithm is robust to a variety of video sequences. Experimental results demonstrate that the proposed method can obtain a gain up to 0.3 dB in all I-frames coding without involving any serious computational burden.

This paper proposes a multi-hypothesis based Wyner-Ziv (WZ) decoder for the multi-view distributed video coding (MDVC). Two hypotheses, the intra-view SI and the interview SI, are fed together into the WZ decoder in the proposed scheme. A multi-hypothesis based correlation model (MHBCM) is presented to fully exploit the redundancy between these two SI frames and the original frame. The MHBCM is also applied on the optimal minimum mean-square error reconstruction of the quantized samples. The simulation results show that the proposed algorithms are able to significantly improve the coding efficiency of the MDVC system.

In this paper, we propose a novel two-pass reconstruction algorithm for the Wyner-Ziv (WZ) frames in distributed video coding (DVC), in which the traditional reconstructed WZ frame is utilized to perform motion estimation to obtain a more accurate motion field. During the motion estimation, the block, as well as its neighboring pixels are concerned. An overlapped block motion compensation is subsequently performed with the help of the motion field, consequently, an enhanced prediction for the WZ frame can be obtained, based on which an improved reconstruction can be achieved. Simulation results show that both the objective and subjective quality of WZ frames can be improved significantly.

This paper proposes an image partition based approach to enhance side information quality in low delay distributed video coding (DVC). The proposed method employs a checkerboard pattern to group blocks of the Wyner-Ziv frame into two sets, where one set is DPCM encoded and the other set is DVC encoded. These two sets are encoded independently and decoded successively. At decoder, DPCM set will be first reconstructed. Then the temporal concealment tool, such as boundary matching algorithm, is performed to conceal blocks in the DVC set. An improved side information is subsequently obtained for DVC set, based on which a higher compression can be achieved. Simulation results indicate that a more promising performance can be achieved when compared with existing motion extrapolated approach.

Motivated by theoretical analysis of the curve fitting problem based on equivalent kernel, in this paper we propose a local adaptive learning and fusion model for side information interpolation in distributed video coding. In the proposed model, each pixel in the interpolated frame is approximated as the linear combination of samples within a local spatio-temporal window using kernel parameters as weight. The size of training window can be adaptive to the motion characteristic of video, from samples in which the kernel parameters can be locally learned. In order to further improve the quality of interpolated frames, we introduce a belief-projection based fusion strategy with adaptive weights for multiple interpolated results which are with the same time index. Experimental results demonstrate that the proposed learning and fusion model is effective in performance for side information interpolation in distributed video coding.

In this paper, a high efficient temporal error concealment scheme based on auto-regressive (AR) model is proposed for video coding. The proposed AR based error concealment scheme includes a forward AR model for P slice, and a bi-direction AR model for B slice. First, we utilize the block matching algorithm (BMA) to select the best motions for lost blocks from the motions of available neighboring blocks. Then, the proposed AR model coefficients are computed according to the spatial neighboring pixels and their temporal-correlated pixels indicated by the selected best motions. Finally, applying the AR model, each pixel of the lost block is interpolated as a weighted summation of pixels in the reference frame along the selected best motions. Simulation results show that the performance of the proposed scheme is superior to conventional temporal error concealment methods.

Traditionally, rate control consists of bit allocation and QP decision on R-QP model. In bit allocation, target bits is further confined if buffer overflows. Meanwhile, rate control should also give as smooth as possible picture quality. However, there is no explicit relationship between picture quality and encoding parameters, so the coding result on picture quality is usually unpredictable and uncontrollable. On the condition of smooth picture quality, the smooth buffer occupancy is preferable certainly. In our work, we first proposed a "window model" formulating the size of window and variations of picture quality and buffer occupancy. Thus, given the constraint on picture quality and buffer occupancy, the compliant coding result about them can be expected employing window model. Second, a window-level rate distortion (RD) model inspired by the traditional rho-domain model is introduced. Lastly, the evaluation of our proposal is presented with elaborate experiments.

This paper proposed a novel Region of Interest (ROI) based rate control algorithm for video communication systems, in which the subjective quality of ROI can be adjusted according to users' requirements. In the proposed scheme, in order to analyze the relationship between subjective quality and encoding parameters, a structural similarity index map - quantization parameter (SSIM-QP) model is established. Through this relation, the possible visual quality range of ROI is defined according to the range of ROI QP, which is predicted by rate control algorithm. Then, with interest levels being identified within the visual quality range, resources allocated to ROI is determined. Finally, considering both the quality of ROI and the entire frame, resource allocation is slightly adjusted. Experimental results demonstrate that the subjective visual quality of ROI in our proposed scheme can be adjustable improved compared to the existing rate control algorithms.

An auto-regressive (AR) based side information (SI) generation is proposed in this paper for block based chessboard pattern Wyner-Ziv (WZ) coding, where each WZ frame is split into two sets at encoder and then encoded separately. At the decoder, one set of the WZ frame will be firstly reconstructed, and then proposed AR model is used to generate the SI of the other set, where each pixel is generated as a linear weighted summation of pixels within two square windows in the previous and following reconstructed WZ/key frames along the motion trajectory. To obtain high quality SI for the second set, reconstructed pixels in the four neighboring blocks of the first set are employed to derive accurate AR coefficients. Several experimental results demonstrate that the proposed AR model is able to improve the quality of the SI for the second set of the WZ frame, which leads to the improvement of the rate-distortion performance of the WZ coding.

This letter proposes a distributed video coding (DVC) scheme based on the human visual system (HVS). As we know, HVS can seldom sense any changes below the just-noticeable-difference (JND) distortion threshold due to its underlying sensitivity and masking property. Therefore, the un-noticeable signal differences between the original frame and side information need not to be corrected in DVC. Experimental results demonstrate that the proposed algorithm can save the bits-rates significantly without degrading the subjective quality of the reconstructed frames.

This paper proposes a novel approach to early determination of zero-quantized 8 × 8 discrete cosine transform (DCT) coefficients for fast video encoding. First, with the dynamic range analysis of DCT coefficients at different frequency positions, several sufficient conditions are derived to early determine whether a prediction error block (8 × 8) is an all-zero or a partial-zero block, i.e., the DCT coefficients within the block are all or partially zero-quantized. Being different from traditional methods that utilize the sum of absolute difference (SAD) of the entire prediction error block, the sufficient conditions are derived based on the SAD of each row of the prediction error block. For partial-zero blocks, fast DCT/IDCT algorithms are further developed by pruning conventional 8-point butterfly-based DCT/IDCT algorithms. Experimental results exhibit that the proposed early determination algorithm greatly reduces computational complexity in terms of DCT/IDCT, quantization, and inverse quantization, as compared with existing algorithms.

Fine-granular scalable (FGS) technologies in H.264/AVC-based scalable video coding (SVC) provide a flexible foundation to accommodate different network capacities. To support efficient quality extraction, it is important to obtain the rate-distortion (R-D) or Distortion-Rate (D-R) function of each individual picture or a group of pictures (GOP). In this paper, firstly, the R-D function of SVC FGS pictures is analyzed with generalized Gaussian model and the D-R curve is proved to be a concave function overall. Considering the current sub-bitplane technology, the D-R function is revisited and inferred to be linear under MSE criterion within an FGS level, which also explains why the observed D-R curve with PSNR criterion is a piece-wise convex function. Secondly, the drift issue of SVC is analyzed, and a simple and effective distortion model is proposed to estimate the reconstruction distortion with drift error. Thirdly, with the above analysis and models, a virtual GOP concept is introduced, and a new priority setting algorithm is designed to achieve the optimal R-D performance in a virtual GOP. The D-R slope of each FGS packet and the D-R function of each virtual GOP are also obtained during the process. Finally, the D-R slopes of FGS levels are used in quality layer assignment to achieve equivalent coding efficiency to the SVC test model but with significantly reduced complexity. The D-R functions of virtual GOPs are utilized to design a practical method for smooth quality reconstruction. Compared to the prior methods, the smoothed video quality is improved not only objectively but also subjectively.

This paper proposes a spatio-temporal auto regressive (STAR) model for frame rate upconversion. In the STAR model, each pixel in the interpolated frame is approximated as the weighted combination of a sample space including the pixels within its two temporal neighborhoods from the previous and following original frames as well as the available interpolated pixels within its spatial neighborhood in the current to-be-interpolated frame. To derive accurate STAR weights, an iterative self-feedback weight training algorithm is proposed. In each iteration, first the pixels of each training window in the interpolated frames are approximated by the sample space from the previous and following original frames and the to-be-interpolated frame. And then the actual pixels of each training window in the original frame are approximated by the sample space from the previous and following interpolated frames and the current original frame with the same weights. The weights of each training window are calculated by jointly minimizing the distortion between the interpolated frames in the current and previous iterations as well as the distortion between the original frame and its interpolated one. Extensive simulation results demonstrate that the proposed STAR model is able to yield the interpolated frames with high performance in terms of both subjective and objective qualities.

H.264/AVC scalable video coding (H.264/AVC SVC), as the scalable extension of H.264/AVC, offers the flexible adaptivity in terms of spatial, temporal and SNR scalabilities for the generated bitstream. However, such compressed video still suffers from the bad playback quality when packet loss occurs over unreliable networks. In this paper, we present an error concealment algorithm to tackle the whole-picture loss problem in H.264/AVC SVC when hierarchical B-picture coding is used to support temporal scalability. In the proposed algorithm, by taking advantage of the temporal relationship among the adjacent video pictures, the motion information of the lost picture is derived simply and efficiently based on the principle of temporal direct mode. Utilizing the derived motion information, the lost picture is concealed by performing motion compensation on the correctly received temporally previous and future video pictures. The experimental results demonstrate that as a post-processing tool, the proposed error concealment algorithm is able to significantly improve both the objective and subjective qualities of the decoded video pictures in the presence of packet losses when compared to the error concealment algorithm used in H.264/AVC SVC reference software. The proposed method can also be applied to H.264/AVC with hierarchical B-picture coding for error concealment.

In dual frame video coding, one short-term reference frame (STR) and one long-term reference frame (LTR) are available for motion compensation. The STR is the previous frame of current frame. The LTR remains static for a few frames, and then jump forward. In this paper, for different GOP length and bits allocation of the LTR, the coding performance of dual frame motion compensation is analyzed. The rate-distortion modeling of multi-hypothesis motion compensated prediction is employed to analyze the performance of dual frame motion compensation.

In this paper, we propose a novel three-tiered network model for image hallucination based on the learnt knowledge composed of image patches relating low and high resolution. A common problem of previous hallucination methods is that irregularities are usually introduced into the constructed high-resolution images. We remove the irregularities in three steps. First, the hallucination with primal sketch priors is performed to construct a coarse high-frequency component. Second, enhancement is implemented to enforce local compatibility between the patches in the constructed component. Third, a Markov network is utilized to refine the enhanced high-frequency component. Experiments demonstrate that our model can hallucinate higher-quality images than existing methods.

Sign Language Recognition (SLR) systems are mostly based on Hidden Markov Model (HMM) and have achieved excellent results. However, the assumption of frame independence in HMM makes it inconsistent with the characteristic of strong temporal correlation in sign language signals. Polynomial Segment Model (PSM) explicitly represents the temporal evolution of sign language features as a Gaussian process with time-varying parameters. In this paper PSM is first introduced to SLR framework to solve the temporal correlation problem. Considering the correlation among the coefficients of polynomial trajectorypsilas different orders, Mahalanobis distance is used as the classification criterion to evaluate the likelihood of test data. Experimental results show that our method outperform the conventional HMM methods by 6.81% in recognition accuracy.

In this paper, an improved spatial scalable wavelet domain distributed video coding (DVC) scheme is proposed. In this scheme, we adaptively apply differential pulse code modulation (DPCM) Wyner-Ziv (WZ) coding and discrete cosine transform (DCT) based intra block coding to reduced-resolution layer. Due to the low energy nature of high-pass subbands, WZ coding can be directly used. At decoder, in full-resolution layer WZ frame decoding, an efficient side information (SI) generation technique for low-pass subband is proposed, in which motion compensated interpolation (MCI) is performed on full-resolution signal. On the other hand, a refined technique is also employed to generate the SI for high-pass subbands by taking advantage of inter-band correlation. Experimental results show that the proposed scheme provides an efficient wavelet domain DVC coding paradigm.

This paper proposes a Wyner-Ziv (WZ) switching scheme for multiple bit-rate (MBR) video streaming over networks. Identical video content is encoded into a set of normal streams, which are generated by conventional hybrid video coding with multiple bit rates, so that streaming can dynamically switch among these normal streams according to available bandwidth. At encoder side, the WZ codec generates a switching stream by compressing the reconstructed frames of a certain normal stream that will be switched to, no matter which normal stream it switches from. At decoder side, for switching to the same frame, the same WZ switching stream is used to reconstruct the switching-to frame by taking the switching-from frame as the side information. The number of required WZ bits depends on the inherent mutual correlation between two frames switching to and from. Since the WZ switching streams are generated independently of the normal switching-from streams, given normal streams that can switch from any one to another, the proposed scheme reduces the number of switching streams from to . Furthermore, switching streams do not deteriorate the coding efficiency of normal streams when no switching occurs. However a big problem here, similar to requesting bits in distributed video coding, is how many WZ bits should be transmitted when a switching happens because the streaming scenario does not tolerate too much extra delay caused by the requests back and forth. Therefore, a Laplacian model, which is proved in the simplified case, is proposed to characterize the correlation between switching-to and switching-from frames. It can be used to accurately estimate the number of WZ bits at the server side.

Utilizing video correlations among views would definitely improve multiview video compression in terms of coding efficiency, which usually requests an expensive system to collect videos from different cameras and jointly compress them. Thanks to recent developments on distributed video coding, this paper proposes a new multiview video coding scheme based on Wyner-Ziv (WZ) coding technique, in which the complicated temporal and interview correlation exploration process is shifted from the encoder side to the decoder side so that broadband raw data traffic and high intensive computation for jointly encoding can be avoided. At the encoder side, a wavelet-based WZ scheme is proposed to compress video of every camera. Furthermore, in order to better utilize correlation in wavelet domain, all coefficients are organized as that done in SPIHT bit plane by bit plane. At the decoder side, a more flexible prediction technique that can jointly utilize temporal and view correlations is proposed to generate side information. Finally, experimental results show the proposed scheme significantly outperforms the conventional intra-frame coding for better random access and is even close to the inter-frame coding for better efficiency. Furthermore, compressed data is much robust when it is transmitted over an error-prone channel.

This paper proposes a fine granularity scalable (FGS) coding using cycle-based leaky prediction, in which the multiple leaky factors are used to yield enhancement layer prediction to make a good compromise between coding efficiency and drift error. In this proposed method, first, the error propagation for leaky prediction with two leaky factors is theoretically analyzed in case only the base-layer bitstream and part of the enhancement-layer bitstream are available at the decoder. Based on this analysis, in this paper, we investigate how to effectively introduce enhancement-layer information into the prediction loop for enhancement-layer coding by the proper leaky factors to constrain drift error while keeping high coding efficiency. Furthermore, a coefficient scaling approach in the transform domain is proposed to address the decoding complexity issue for multiple reconstructions of partial enhancement layers at different quality levels. Finally, an encoder optimization approach is presented to further control drift error for multiple FGS layers coding. The experimental results show that compared to AR-FGS in JSVM, the proposed method can significantly improve the coding performance over a wide range of bitrates.

In this paper, we describe the acquisition and contents of a large-scale Chinese face database: the CAS-PEAL face database. The goals of creating the CAS-PEAL face database include the following: 1) providing the worldwide researchers of face recognition with different sources of variations, particularly pose, expression, accessories, and lighting (PEAL), and exhaustive ground-truth information in one uniform database; 2) advancing the state-of-the-art face recognition technologies aiming at practical applications by using off-the-shelf imaging equipment and by designing normal face variations in the database; and 3) providing a large-scale face database of Mongolian. Currently, the CAS-PEAL face database contains 99 594 images of 1040 individuals (595 males and 445 females). A total of nine cameras are mounted horizontally on an arc arm to simultaneously capture images across different poses. Each subject is asked to look straight ahead, up, and down to obtain 27 images in three shots. Five facial expressions, six accessories, and 15 lighting changes are also included in the database. A selected subset of the database (CAS-PEAL-R1, containing 30 863 images of the 1040 subjects) is available to other researchers now. We discuss the evaluation protocol based on the CAS-PEAL-R1 database and present the performance of four algorithms as a baseline to do the following: 1) elementarily assess the difficulty of the database for face recognition algorithms; 2) preference evaluation results for researchers using the database; and 3) identify the strengths and weaknesses of the commonly used algorithms.

This paper proposes an enhanced RVLC coder, context-adaptive reversible variable length coder (CRVLC), for DCT coefficients by using the techniques of data sub-partitioning and context modeling. The data sub-partitioning means that the data part of DCT coefficients is split into several small sub-partitions. As each sub-partition can be reversibly decoded by RVLC, more data as well as higher error resilience can be obtained. The context modeling exploits the correlation of DCT coefficients for further compression. This modeling defines the contexts by hierarchical-dependent information. The information is also available in the backward decoding, so that it supports the reversible decoding. And with it the data outputted by CRVLC can be naturally placed into multiple sub-partitions.

JVT SVC has adopted AR-FGS (fine granularity scalability with adaptive reference) to efficiently encode the progressive sequences for low-delay applications. Generally, low-delay applications demand low complexity. However, to some extent, the significant gain of AR-FGS is achieved at the expense of much higher coding complexity. In this paper, we propose an efficient FGS coding scheme for low-delay applications that can more smartly compress the interlaced video sources by fully utilizing the special temporal correlation of interlaced sequences. Compared to AR-FGS in coding interlaced sequences, the new approach is able to significantly reduce both the encoding and decoding complexity while maintaining comparable compression performance. Moreover, benefited from the introduced open-loop MC, the proposed scheme can also support additional temporal scalability.

The multi-view video coding improves the coding efficiency by utilizing motion-compensated prediction (MCP) and disparity-compensated prediction (DCP). However, the complexity of the inter frame prediction is very high, especially when the rate-distortion optimization is used. This paper presents a fast inter frame prediction algorithm to reduce the complexity. Firstly the prediction type is decided according to reference frames. Then some unuseful search regions in view direction are removed. Finally a fast inter mode decision strategy is proposed based on the relationship between MCP and DCP. Experimental results verify that the proposed algorithm can greatly increase the speed of prediction with negligible loss of coding efficiency.

A novel error concealment method is proposed for two-view based stereoscopic video coding to address the challenging problem of adaptively combining inter-view correlation and temporal correlation. First, the disparity vectors of the lost macroblocks' neighboring macroblocks are used to recover the lost or erroneously received motion or disparity vectors. Then we propose a novel error concealment method based on overlapped block motion and disparity compensation, whose weights are determined by the side match criterion and viewpoints. Simulation results show that the subjective and objective performances of the proposed technique are both superior to those of conventional temporal error concealment methods for stereoscopic video coding.

This paper presents some techniques for efficient motion estimation (ME) implementation on fixed-point digital signal processor (DSP) for high resolution video coding. First, challenges in large search-range ME are discussed. Then, based on the statistics analysis of the best matched block, correlations in variable block-size ME are removed on algorithm level while keeping almost the same performance as the original fast search strategy. Based on this modification and data reuse technique, this paper proposes an "index search" method for DSP implementation, which is a good solution to balance both high coding efficiency and high coding speed on the DSP platform. In "index search", ME is divided into three steps and highly parallel pipelines are designed for each step. The fully use of pipelines brings about 85% time reduction during ME. "Index search" can also be easily extended to achieve different search range and search patterns.

The in-loop deblocking filter contains highly adaptive processing on both sample level and block edge level, which inevitably appears in the loop kernel of the algorithm. Therefore it is a challenge for parallel processing on a digital signal processor (DSP) platform. In this paper, pipelined DSP solutions to the in-loop deblocking filter in AVS1-P2 are presented. First, the whole filter process is divided into six sub-processes, so that the global filter structure can be improved to achieve regular processing flow. Then software pipelines are designed for these sub-processes, with elaborately allocating functional units and carefully choosing enhanced assembly instructions based on the DSP platform. The simulated results show that this efficient implementation can easily support real-time filter processing for high resolution videos.

A spatio-temporal autoregressive model is proposed in this paper to address the problem of frame rate up conversion. Every pixel in a skipped frame is generated as a linear combination of pixel values from forward and backward reference frames. At the beginning of the presented scheme, the coarse model parameters are computed according to the given initial pixel values for skipped frames. Then the coarse parameters are refined by an iteration process, during which we also interpolate the original low rate frames by the two closest generated skipped frames to derive more accurate parameters. Experimental results verify that the proposed algorithm significantly improves both the subjective and objective quality of the interpolated frames.

The direct mode used in the bi-predictive pictures (B-pictures) can efficiently improve the coding performance of B pictures, because it has small overhead and obtains a predictive picture from two reference pictures. The traditional temporal direct mode (TDM) derives the motion vector of the current block by scaling the motion vector of the co-located block in the backward reference picture. However, when the current block and its co-located block in backward reference picture belong to different objects with different motion directions, the prediction efficiency of TDM is drastically reduced. In this paper, we propose an improved direct mode prediction method. In the method, a virtual reference picture is generated using the pixel projection technique. Then the virtual reference picture is used to predict the direct mode blocks in B pictures. The proposed method can enhance the prediction performance of the direct mode blocks and achieve a higher coding efficiency.

In this paper, we propose a simple and effective rate-distortion model of fine-granular SNR scalable (FGS) layer in JVT scalable video coding (SVC). First, we introduce generalized Gaussian distributions (GGD) to model the distributions of the 16 (4*4) integer transform coefficients of the SVC FGS residue picture. Then we analyze the quantization scheme of SVC FGS coding, under which we analyze the distortion-rate function of generalized Gaussian model and conclude that the derivative of the distortion-rate function usually decreases to the traditional number 6.02 dB/bit as the coding rate increases. For some special cases where the shape of GGD is smooth, the derivative of the distortion-rate function increases to 6.02 dB/bit as the rate increases in the actual coding rate range. Guided by the observations, an effective and flexible rate-distortion model is proposed to approximate the actual rate-distortion function of SVC FGS layer. The average estimation error is only 0.07 dB in our extensive experiments. And our analysis and models bring us much insight into the SVC FGS coding and its rate-distortion function.

In inter-picture coding, block-based frequency transform is usually carried out on the predicted errors for each interblock to remove the spatial correlation among them. However, it can not always do well since the predicted errors in some inter-blocks have marginal or diagonal correlation. A good solution is to omit transform operations for the predicted errors of those inter-blocks with low correlation before quantization operation. The same phenomenon also can be observed in Fine Grain Scalability (FGS) layer coding. In this paper, an adaptive prediction error coding method in spatial and frequency domain with lower complexity is considered for FGS layer coding. Transform operation is only needed when there are non-zero reconstructed coefficients in spatially co-located block in base layer. The experimental results show that compared with FGS coding in JSVM, higher coding efficiency can be achieved with lower computational complexity at decoder since inverse transform is no longer needed for those predicted errors coded in spatial domain at encoder.

Reduced resolution update (RRU) video coding provides a better compression performance than H.264/AVC does. It is implemented by encoding the predicted residuals after down-sampling at the encoder and at the decoder, is up-sampled to the full-resolution size while motion compensation prediction signal are still yielded in the full-resolution spatial domain. In this paper, the authors first extend the frame-level RRU to the macroblock-level to obtain better coding performance than the frame-level RRU. Furthermore, a novel macroblock-level reduced resolution coding scheme was proposed which adaptively selects the part of DCT coefficients in the zigzag scan order to obtain better coding efficiency in terms of rate distortion performance. Experimental results exhibit that the proposed methods can achieve obvious improvements in both compression efficiency and decoding complexity compared with the frame-level RRU and H.264/AVC.

Multi-view video coding is becoming a very active research topic, as multi-view video system provides the interactive feature which makes viewers experience the free viewpoint navigation within the range covered by the shooting cameras compared with the traditional single view video. Multi-view video coding aims at compressing the redundancy between views besides temporal correlations within each view. Rapid view random access is a basic requirement for multi-view video communication. Though inter-view prediction enhances the coding efficiency, it limits the rapid view random access capability. In this paper, we propose three approaches to provide low-delay view random access capability while keeping the high rate-distortion performance. The proposed techniques, including SP/SI frame coding, interleaved view coding and secondary representation coding, vastly reduce the decoding delays while view random access occurs and greatly improve the ability of switching between views.

The coding performance can be further improved when the hierarchical B-picture coding is introduced into H.264/AVC. However, the existing rate control schemes can not work efficiently in such new coding framework. This paper proposes a novel rate control algorithm when hierarchical B-picture coding is used in H.264/AVC. Firstly, a set of scaling-factors applied in designing cascaded quantizer for the B frames at different temporal levels is introduced. Based on the designed scaling-factors, an efficient bit-allocation strategy for hierarchical B-picture coding is presented. The experiments show that the proposed rate control algorithm can further improve PSNR up to 0.7dB compared to the existing hierarchical B-picture coding in H.264/AVC, while the mismatch of target bit rate and real bit rate does not exceed 2%.

Discrete cosine transform (DCT) has been widely used in image/video coding systems, where zigzag scan is usually employed for DCT coefficient organization. However, due to local diversity of prediction errors, the fixed scan pattern, such as the traditional zigzag scan, is not efficient all the time for organizing the DCT transformed prediction errors. In this paper, the statistical distribution of prediction residuals is studied first, and two other scan patterns are introduced to complete the set of possible patterns. Then, a new adaptive scheme that adaptively chooses for each macroblock the best scan pattern to arrange the quantized DCT coefficients into arrays is presented. The best scan pattern is selected for each macroblock according to its own residual characteristics and hence leads to improved coding efficiency. Experimental results show that this macroblock-level adaptive scan scheme (MLASS) can always outperform the traditional zigzag one and the peak signal to noise ratio (PSNR) gain can be up to 0.65dB. Moreover, the extra two scan patterns do not increase memory requirement much. Additionally, a fast scan pattern decision algorithm is given for complexity reduction. This fast version of MLASS can keep almost the same coding efficiency while with much lower complexity.

This paper presents a new context modeling technique for arithmetic coding of DCT coefficients in video compression. A key feature of the new technique is the inclusion of all previously coded coefficient magnitudes in a DCT block in context modeling. This enables adaptive arithmetic coding to exploit the redundancy of the high-order Markov process in the DCT domain with a few conditioning states. In addition, a context weighting technique is used to further improve the coding efficiency. The complexity of the new arithmetic coding scheme is slightly lower than that of context-based adaptive binary arithmetic coding (CABAC) of H.264. Moreover, the scheme is made compatible to the AVS baseline profile. It achieves on average 13% improvement in compression ratio over context-based two dimension variable length coding (C2DVLC) designed for the DCT domain, and a similar coding efficiency as the CABAC technique in H.264.

For low-delay applications in Scalable Video Coding, there are two alternative coding strategies based on AR-FGS for compressing multiple FGS layers. However, their coding efficiencies both suffer largely from the inherent drifting errors. In this paper, a more efficient multiple FGS layers coding structure is presented, which can provide higher coding performance within wide bitrate range and stronger error resilience. This is achieved by incorporating the partial-reconstructed enhancement layer references, instead of the complete-reconstructed ones, into the motion-compensated prediction loop of the FGS layers. Thereupon the prediction drift can be effectively decreased, especially for the middle bitrate points. Further, through selecting different-quality enhancement layer references generated with cycle-based reconstruction mechanism, more flexible video quality can be supported to be more suitable for varied practical application requirements.

This paper presents a novel B-picture coding based on block-wise adaptive motion accuracy (BAMA) with low-complexity motion compensation (MC). BAMA is able to adaptively select the motion accuracy for each inter-block in a B-picture depending on whether it is of bidirectional prediction, in which compared to the unidirectionally predicted block, lower motion accuracy is applied to the bidirectionally predicted block to reduce MC complexity. To further reduce MC complexity for the bidirectionally predicted luminance block with subpixel motion vectors in both directions, the forward and backward MC interpolations are merged according to the linear operation property of MC interpolation. In addition, a simplified MC interpolation method is also introduced for chrominance components for the bidirectionally predicted block. When integrating the proposed methods into H.264/AVC reference software, the experimental results demonstrate that they can significantly reduce MC complexity for a B-picture while yielding the comparable rate distortion performance in comparison with H.264/AVC although they no longer generate H.264/AVC compliant bitstream.

For a typical video distribution system, the video contents are first compressed and then stored in the local storage or transmitted to the end users through networks. While the compressed videos are transmitted through error-prone networks, error robustness becomes an important issue. In the past years, a number of rate-distortion (R-D) optimized coding mode selection schemes have been proposed for error-resilient video coding, including a recursive optimal per-pixel estimate (ROPE) method. However, the ROPE-related approaches assume integer-pixel motion-compensated prediction rather than subpixel prediction, whose extension to H.264 is not straightforward. Alternatively, an error-robust R-D optimization (ER-RDO) method has been included in H.264 test model, in which the estimate of pixel distortion is derived by simulating decoding process multiple times in the encoder. Obviously, the computing complexity is very high. To address this problem, we propose a new end-to-end distortion model for R-D optimized coding mode selection, in which the overall distortion is taken as the sum of several separable distortion items. Thus, it can suppress the approximation errors caused by pixel averaging operations such as subpixel prediction. Based on the proposed end-to-end distortion model, a new Lagrange multiplier is derived for R-D optimized coding mode selection in packet-loss environment by taking into account of the network conditions. The rate control and complexity issues are also discussed in this paper

The major challenges that sign language recognition (SLR) now faces are developing methods that solve large-vocabulary continuous sign problems. In this paper, transition-movement models (TMMs) are proposed to handle transition parts between two adjacent signs in large-vocabulary continuous SLR. For tackling mass transition movements arisen from a large vocabulary size, a temporal clustering algorithm improved from k-means by using dynamic time warping as its distance measure is proposed to dynamically cluster them; then, an iterative segmentation algorithm for automatically segmenting transition parts from continuous sentences and training these TMMs through a bootstrap process is presented. The clustered TMMs due to their excellent generalization are very suitable for large-vocabulary continuous SLR. Lastly, TMMs together with sign models are viewed as candidates of the Viterbi search algorithm for recognizing continuous sign language. Experiments demonstrate that continuous SLR based on TMMs has good performance over a large vocabulary of 5113 Chinese signs and obtains an average accuracy of 91.9%

In this paper, we propose a novel bit-stream switching scheme for the multiple bit-rate (MBR) video streaming, in which a Wyner-Ziv coded frame is used to overcome the mismatch between the MBR streams when the switching occurs. With the proposed technique, the MBR streams can be independently encoded without losing any coding efficiency. Similar to distributed video coding, the proposed Wyner-Ziv switching scheme also faces the challenge of rate allocation at the server side. To solve this problem, we propose a new correlation model based on the analysis on the reconstructed frames from the streams with different bit rates. Accordingly, the number of transmitted bits can be on-line calculated based on the correlation model without any feedback from the decoder. With the proposed technique, the actually transmitted Wyner-Ziv bits are only few more than the truly requested bits. However, the delay due to the bit requesting process can be avoided.

In this paper, a low-complexity adaptive block-size transform (ABT) scheme is proposed for the H.264/AVC. The integer 8x8/4x4 discrete cosine transforms with an extended relation are applied to the proposed ABT scheme for reducing the implementation complexities. In the proposed scheme, the 8x8/4x4 transforms can be merged together and share the same scale matrix. As a result, the transform units and storage resources can be effectively saved in both encoder and decoder. The 8x8 transform extended from the 4x4 transform of H.264/AVC can be easily implemented with several simple arithmetic operators, and its intermediate results can be strictly limited within 16-bit. The experimental results show that the proposed ABT scheme based on extended transforms achieves a similar coding performance to the original ABT scheme for H.264/AVC.

JVT scalable video coding (SVC) provides high coding efficiency for progressive video sequences with combined scalability. However, interlaced SVC is only the straightforward extension of H.264/AVC interlaced coding with the similar FGS coding technique for progressive coding. Based on the particular temporal correlation of interlaced video sequences, this paper presents a novel and efficient FGS coding scheme for interlaced SVC, which is able to achieve higher compression efficiency and further temporal scalability by introducing additional temporal decomposition stage when coding key pictures. The advantages of our proposed interlaced FGS coding scheme are verified by integrating it into JVT-SVC reference software.

In this paper, we propose a novel hybrid de-interlacing algorithm, which effectively combines two motion-compensated (MC) de-interlacing techniques: MC median filtering (MCMF) and adaptive recursive (AR) with one spatial approach: line averaging (LA). Despite of its drawbacks, AR is one of the best methods nowadays. MCMF helps reduce flickers and LA is very robust to erroneous motion vectors. The interpolation switches among these methods based on the proposed measurement of texture smoothness and motion vector (MV) reliability. MCMF is adopted when MV is reliable and texture is rich. LA is used when MV is unreliable and texture is smooth. AR is applied to the remaining regions. Experimental results show that the proposed algorithm is superior to the compared algorithms in terms of peak signal-to-noise ratio (PSNR) and that the de-interlaced videos have very high subjective quality

Because of its ability to preserve signal edges while filtering out impulsive noises, median filtering is widely used in signal processing applications, e.g. deinterlacing. An edge-based median filtering (EMF) algorithm is proposed for adaptive deinterlacing. A criterion of motion vector reliability (MVR) is also introduced for better interpolation. For each motion compensated block, two motion vectors between opposite-parity fields and one between same-parity fields are taken into account. Experiments show that the proposed MVR and EMF are both very efficient. Outputs of the proposed EMF are much more similar to original progressive videos than those of objectively best EMF methods nowadays, without obvious visual distortions. Finally, the proposed EMF and MVR criterion are shown to be suitable for texture-based adaptive deinterlacing

A motion aligned spatial scalable video coding scheme (MA-SSC) is proposed in this paper. Different from the traditional spatial scalable coding schemes derived from MPEG-2, in the proposed scheme only one set of intra or inter prediction modes are optimally selected by jointly considering the base and enhancement layers. Thus, it saves one set of macroblock (MB) mode and motion vectors. Moreover, the combined motion estimation can reduce the residual coding bits of the base layer. The MA-SSC and traditional spatial scalable coding schemes are both implemented based on H.264 reference software to evaluate their performance. Simulation results show that the enhancement layer coding efficiency of MA-SSC is up to 0.6dB better than that of the traditional scheme, while the base layer coding efficiency of MA-SSC decreases less than 0.3db compared with the single-layer coding

In this paper, a low-complexity 8times8/4times4 adaptive block-size transform (ABT) scheme is tentatively proposed for the Chinese audio and video coding standard (AVS). In the proposed ABT scheme, an integer 8times8 transform is derived from the integer 4times4 transform used in AVS according to a transform extension principle. The 8times8 transform not only has high energy compacted property but also can be merely implemented within several additions and shifts, and all intermediate results are limited with 16-bit. The 8times8 transform and 4times4 transform can be merged together and share the same scale matrix so that the hardware units and storage resources are efficiently saved for both encoder and decoder. The experimental results on numerous sequences show that the proposed ABT scheme can achieve significant performance improvement for AVS

H.264/AVC is the latest video coding standard developed by MPEG and ITU. AVS is the latest video coding standard developed by China. The two new video coding standards will co-exist in the market in the future. So it is necessary to transcode the AVS format to the H.264/AVC format or vice versa. Because the AVS video coding standard is based on the 8times8 block and also uses the intra prediction technology, it is possible to use the AVS intra mode information to predict the H.264/AVC intra mode. In this paper, a fast intra mode decision algorithm was proposed to reduce the computation complexity for AVS to H.264/AVC intra transcoding. The evaluation results show that the proposed algorithm can save at least 50% computation with about 0.1 dB loss in PSNR

In this paper, we propose a novel algorithm to design an optimal non-uniform scalar quantizer for distributed video coding, which aims at achieving a coding rate close to joint conditional entropy of the quantized video frames given the side information. Wyner-Ziv theory on source coding is employed as the basic coding principle and the asymmetric scenario is considered. In this algorithm, a probability distribution model, which considers the influence of the joint distribution of input source and side information to the coding performance, is established and used as the optimality condition firstly. Then, a modified Lloyd Max algorithm is used to design the scalar quantizer to give an optimal quantization for input source before coding. Experimental results show that compared to uniform scalar quantization, proposed algorithm can improve coding performance largely, especially at low bit rate

This paper presents a novel high-efficient hybrid open-close loop based fine granularity scalable (HOCFGS) coding framework supporting different decoding complexity applications. The open-loop motion compensation for inter-pictures is introduced to efficiently exploit the temporal correlation among adjacent pictures for both base layer and FGS enhancement layer within a wide bit-rate range. An efficient rate-distortion optimized macro-block mode decision rule is used to reduce drifting error and achieve comparable coding performance at the lowest bit-rate point (base layer) with non-scalable coding. An approach like MPEG-4 FGS coding with close-loop motion compensation only at base layer is used for some inter-pictures to stop the drifting error propagation. Furthermore, to achieve better coding performance for these inter-pictures, block based progressive fine granularity scalable (BLPFGS) is introduced, in which leaky prediction is used to generate high-quality reference for FGS enhancement layer. In BLPFGS coding, efficient bit-plane coding and de-blocking techniques are investigated to improve the coding performance for FGS enhancement layer, especially at low bit-rate points. The coding performance for the proposed method is verified by integrating it into MPEG SVC reference software.

Data partitioning (DP) is an efficient error-resilient video coding tool. Its contribution to performance improvement in the error-prone environment arises from the superior error concealment mechanisms that are available with the help of protected data partitions. Since error-concealment in terms of DP is closely related to coding mode, it is desirable to have an optimized coding mode selection scheme. However, the existing coding mode selection techniques usually assume that the same error-concealment mechanism is used for a block when it is lost, and the associated distortion also remains the same. Obviously, this assumption is not true when DP involves. In this paper, a generalized end-to-end distortion model is proposed for the rate-distortion optimized coding mode selection, which fully utilizes the superior error-concealment mechanism in terms of DP. The proposed distortion model is also advantageous in the suppression of approximation errors caused by pixel average operations such as sub-pixel interpolation and deblocking filter. Therefore, it can lead to a low-complexity solution for real-time applications such as live streaming

In this paper, the generalized Gaussian distribution is employed first to model the DCT coefficients of image data from MPEG-4 fine-granularity scalability (FGS) frame. Then, according to the quantization theory, the distortion-rate function of the generalized Gaussian model is analyzed and it is concluded that the derivative of the distortion-rate function first decreases, and then increases up to the boundary of 6.02 as the bit rate increases. For actual FGS coding, the derivative of actual distortion-rate function usually decreases as the rate increases, and then begins to increase slowly at a comparatively high bit rate. Finally, based on above observations, a rate-distortion (R-D) model is proposed to approximate the actual distortion-rate function. Experiments show that the proposed R-D model is accurate and flexible.

In video coding, motion vectors always account for a large number of bits and affect coding efficiency largely. In this paper, we propose an efficient motion vector prediction algorithm for multiview video coding (MVC), which can predict motion vectors from adjacent views and achieve good prediction accuracy. We first investigate the correlations among different views and describe the disparity between adjacent views as global motion. The affine model is used to compute the global parameters between frames of adjacent views. At least one view is coded independently without interview prediction. After that, motion vectors of the frame to be coded can be derived from the motion vectors of the co-located coded frame in adjacent view using the global motion information. A rate-distortion optimization scheme is used to choose between the proposed method and traditional motion compensated prediction method. Experimental results show that, compared to simulcast coding, the proposed algorithm can achieve good performance and improve the coding efficiency up to 0.8 dB in PSNR.

Feature extraction for object representation plays an important role in automatic object detection system. As the spatial histograms consist of marginal distribution of image over local patches, object texture and shape are simultaneously preserved by the spatial histogram representation. In this paper, we propose methods of learning informative features for spatial histogram-based object detection. We employ Fisher criterion to measure the discriminability of each spatial histogram feature and calculate features correlation using mutual information. In order to construct compact feature sets for efficient classification, we propose informative selection algorithm to select uncorrelated and discriminative spatial histogram features. The proposed approaches are tested on two different kinds of objects: car and video text. The experimental results show that the proposed approaches are efficient in object detection.

Visual tracking could be formulated as a state estimation problem of target representation based on observations in image sequences. Approaching visual tracking problem in the Bayesian filter framework, how to sample the state evolution model to generate hypothesis of high confidence level is a critical factor. In this paper, we introduce an interacting multiple model estimation (IMME) framework for adaptive visual tracking. The essence of the IMME framework is that the state is estimated by integrating several different models in parallel and by interacting among those models' estimates probabilistically. Based on the IMME framework, we propose a new variation of particle filter named interacting multiple model particle filter (IMMPF), in which the hypotheses can be sampled from several different state evolution models adaptively. Experiments show that, when compared with the standard particle filter, the IMMPF generates better hypotheses resulting in better tracking results, especially when the target behaves along several motion modes randomly.

To leave the maximum flexibility in encoder to optimize the trade-off between coding performance and complexity, in video coding standards such as H.264 AVC, H.263 and MPEG-4 etc, any number of B pictures and any arrangement of P pictures within a group of pictures (GOP) of arbitrary length are permitted. In addition, multiple reference picture prediction is also permitted in some video coding systems such as H.264/AVC to achieve efficient coding by allowing the encoder to select reference pictures among a large number of coded pictures. Both of the above cases without fixing the temporal distance between forward and backward reference pictures require the division operation for deriving the motion vectors of direct mode, which can efficiently exploit the temporal correlation among pictures and does not require any bits for coding the motion vectors. However, the division is an expensive and undesired operation in video decoder hardware design. Although the H.264/AVC video standard has provided a scaling technique to tackle this problem, unfortunately, its performance is also deteriorated. This paper presents a new scaling technique to both remove the division operation for deriving direct mode motion vectors and efficiently improve the accuracy of derived direct mode motion vectors compared with the scaling technique in H.264/AVC.

In the conventional video codec, B pictures usually employ five prediction modes, including bi-prediction, direct prediction, forward prediction, backward prediction and intra modes. Among them, the direct and bi-prediction modes, whose prediction values are achieved from both forward and backward reference pictures, are more efficient in exploiting the temporal correlation between the reference pictures and the current B picture. In addition, the direct mode does not require any bits for coding the motion vectors. Therefore, the blocks coded with direct or bi-predictive mode are usually much more than the other modes. To further take advantage of the bi-prediction, We propose a new bi-predictive coding technique, which can achieve a good tradeoff between the bit-rate saving for motion vector coding and the prediction accuracy. Moreover, we also propose spatial motion vector prediction and motion vector scaling techniques to improve the accuracy of the derived direct mode motion vectors. All these techniques have been adopted in the AVS video coding standard.

It is well observed that usually the run-length of successive zero coefficients becomes longer and the magnitude of non-zero coefficients gets smaller while the DCT subband frequency increases. Therefore, the probability distributions of level/run combinations should be different at different DCT positions. Based on this observation, an efficient context based 2D-VLC entropy coder, especially designed for 8/spl times/8 DCT, is proposed to exploit this context characteristic. The key element of the proposed coder is to design multiple 2D-VLC tables and each can be used context-adaptively to match different level/run combination probability distributions better. Another key element is the usage of exponential-Golomb codes, which keep the multiple tables with low memory requirement. In terms of coding efficiency, the experimental results show that the proposed method can gain up to 0.23 dB when compared to the one-table-for-one-block coding method.

Intra-prediction is a widely used technique in intra coding. In H.264, nine directional prediction modes are used, and in AVS (audio-video coding standard of China), six modes are used. The paper proposes a multiple mode, intra-prediction scheme to achieve a better balance between prediction precision and side information cost. Three kinds of additional modes are introduced to increase prediction precision from different aspects. A significant performance gain can be achieved using this multiple mode intra-prediction, with little complexity increase.

The major difficulty for large vocabulary sign recognition lies in the huge search space due to a variety of recognized classes. How to reduce the recognition time without loss of accuracy is a challenging issue. In this paper, a fuzzy decision tree with heterogeneous classifiers is proposed for large vocabulary sign language recognition. As each sign feature has the different discrimination to gestures, the corresponding classifiers are presented for the hierarchical decision to sign language attributes. A one- or two- handed classifier and a hand-shaped classifier with little computational cost are first used to progressively eliminate many impossible candidates, and then, a self-organizing feature maps/hidden Markov model (SOFM/HMM) classifier in which SOFM being as an implicit different signers' feature extractor for continuous HMM, is proposed as a special component of a fuzzy decision tree to get the final results at the last nonleaf nodes that only include a few candidates. Experimental results on a large vocabulary of 5113-signs show that the proposed method dramatically reduces the recognition time by 11 times and also improves the recognition rate about 0.95% over single SOFM/HMM.

The major difficulty for large vocabulary sign language or gesture recognition lies in the huge search space due to a variety of recognized classes. How to reduce the recognition time without loss of accuracy is a challenge issue. In this paper, a hierarchical decision tree is first presented for large vocabulary sign language recognition based on the divide-and-conquer principle. As each sign feature has the different importance to gestures, the corresponding classifiers are proposed for the hierarchical decision to gesture attributes. One- or two- handed classifier with little computational cost is first used to eliminate many impossible candidates. The subsequent hand shape classifier is performed on the possible candidate space. SOFM/HMM classifier is employed to get the final results at the last non-leaf nodes that only include few candidates. Experimental results on a large vocabulary of 5113-signs show that the proposed method drastically reduces the recognition time by 11 times and also improves the recognition rate about 0.95% over single SOFM/HMM.

In this paper, a novel face representation, named Face-Specific Subspaces (FSS), is proposed first. Then the corresponding face identification strategy is provided. This method is motivated, but essentially different, from Eigenface. In the proposed method, each face holds one private face subspace, while in Eigenface all faces share one common face subspace. To enable the proposed approach to tackle single example problem, a technique to derive multi-samples from single example is further developed. Extensive experiments on Yale face database Bern face database, and our 350 subjects database show that our method makes impressive performance improvement when compared with conventional Eigenface and template matching, which intensively indicates the robustness of our approach. against appearance variances due to expression, illumination and pose.

Motion based video indexing is an important and active research area in content-based video retrieval. It explores the dynamic characteristics of video content and provides techniques for video representation and retrieval. In this paper, a new motion-based approach is proposed, in which the image generated from motion activity is used to index video. The proposed approach firstly computes the accumulation measurement of motion activity on the grids of video frames along the time axis. Then, the computed measurement is quantized into several gray levels and a gray image is generated from all measurements on the grids. The generated image called motion activity map (MAM) reserves the motion information on different spatial locations of video. The intensity of MAM is corresponding to the magnitude of motion activity. All MAMs are organized into a hierarchical tree according to the video structure. Therefore, user can browse video by the MAMs tree. The proposed approach provides a hierarchical, coarse to fine view of video and thus makes interactive video retrieval more simply and intuitively.

A novel approach to localize human iris centers in facial images is proposed. From the estimated nose center of the detected face, the proposed method searches for the possible iris edge by gradually growing a rectangle, and then localizes the iris center in an expected region by searching for the pixel, which has the minimum average intensity. Experimental results show that this method has good performance for facial images under variations in pose and lighting direction. The speed of the method is notably fast. It has been successfully applied to a face recognition system for providing the eye locations for aligning faces.

Recent success in discrete cosine transform (DCT) image coding is mainly attributed to recognition of the importance of data organization and representation. Currently, there are several competitive DCT-based coders such as DCT-based embedded image coding (EZDCT) (see Xiong et al., Z., 1996) and significance tree quantization (STQ) (see Davis, G. and Chawla, S., 1997). In the wavelet context, morphological representation of wavelet data has achieved the best compression performance. The representatives are morphological representation of wavelet data (MRWD) (see Servetto, S. et al., 1999) and significance-linked connected component analysis (see Chai, B.-B. et al., 1999). We show that, by proper reorganization of its coefficients, the block-based DCT can have similar characteristics, such as energy compaction, cross-subband similarity, decay of magnitude across subband, etc., to the wavelet transform. These characteristics can widen DCT applications relevant to image compression, image retrieval, pattern recognition, etc. We then present an image coder utilizing these characteristics by morphological representation of DCT coefficients (MRDCT). The experiments show that MRDCT is among the state-of-the-art DCT-based image coders reported in the literature. For example, for the Lena image at 0.25 bpp, MRDCT outperforms JPEG, STQ and EZDCT in peak signal-to-noise ratio by 1.0, 1.0, and 0.3 dB, respectively.

A low-complexity and low-memory entropy coder (LLEC) is proposed for image compression. The two key elements in the LLEC are zerotree coding and Golomb-Rice (1966, 1991) codes. Zerotree coding exploits the zerotree structure of transformed coefficients for higher compression efficiency. G-R codes are used to code the remaining coefficients in a variable-length codes/variable-length integer manner resulting in JPEG similar computational complexity. The proposed LLEC does not use any Huffman table, significant/insignificant list, or arithmetic coding, and therefore its memory requirement is minimized with respect to any known image entropy coder. In terms of compression efficiency, the experimental results show that discrete cosine transform (DCT)- and discrete wavelet transform (DWT)-based LLEC outperforms baseline JPEG and embedded zerotree wavelet coding (EZW) at the given bit rates, respectively. For example, LLEC outperforms baseline JPEG by an average of 2.2 dB on the Barbara image and is superior to EZW by an average of 0.2 dB on the Lena image. When compared with set partition in hierarchical trees, LLEC is inferior by 0.3 dB, on average, for both Lena and Barbara. In addition, LLEC has other desirable features, such as parallel processing support, region of interest coding, and as a universal entropy coder for DCT and DWT.

Summary form only given. A lossless/near lossless coding scheme, DACLIC, is presented. The proposed scheme attempts to remove redundancy in a given image in the spatial domain. The redundancy removal is achieved by block direction prediction and context-based error modeling. The block direction operation in DACLIC first partitions an image into blocks. Pixels within each incoming block are analyzed resulting in a best directional prediction for that block. The best direction is chosen from a given set that results in the minimum prediction error. Removal of redundancy by the block direction technique is not possible for removing all possible redundancy in a given image. Another decorrelation part of the DACLIC scheme is the context-based error modeling which exploits context-dependent DPCM error structures. The DACLIC scheme is primarily used as a lossless image compression technique. However, the scheme can be easily extended to near-lossless compression applications by introducing a small quantization loss. This small quantization loss is restricted to an absolute error not exceeding a prescribed value n for all pixels in a given image. Application of block direction and context modeling reduces a given image into residuals. This residual typically has a lower entropy than the given image. A quadtree Rice coder (QRC) is proposed as an entropy coder of DACLIC. An arithmetic coder is also given as an option. The QRC operates on residual blocks with low computing complexity that compares favorably with the residual coding method used by LOCO-I as the proposed QRC is two-dimensional in nature. For near-lossless compression with a larger value of n, banding artifacts are visible in the decoded image. In the DACLIC system, a postprocessing technique is proposed to remove the banding artifacts.

Summary form only given. FACOLA (face coder based on location and attention) is proposed for potential applications such as compression of face pictures used in IC and ID cards. The face locator locates a face in an image using template matching and eigenface techniques. The attention detector detects high attention and low attention in the located face according to their different frequency characteristics. For high attention, usually high quality or lossless coding is required. The DCT is not suitable for such a case because it will not lead to an efficient compaction of the image energy and variable length coding (VLC) cannot be done efficiently. So a DPCM coder with three directional predictors is presented instead of the DCT. The prediction difference is quantized and the entropy coded DPCM coder supports lossless and lossy compression. The DCT coder is adopted for low attention and non-face area compression using different quantization factors.