Recent advancements in 3D scanning technology have spurred extensive research on indoor scene point clouds. However, point clouds captured by 3D sensors are often sparse, noisy, and irregular, posing significant challenges for downstream tasks. While learning-based upsampling methods have achieved promising results, they often struggle with large-scale indoor scenes due to the high computational cost of K-Nearest Neighbor (KNN) search and insufficient reconstruction of fine-grained structures. To address these issues, we propose S 2 PU-Net, a sparse semantic-guided progressive point cloud upsampling network for indoor scenes. Specifically, we introduce a progressive upsampling mechanism based on sparse tensors, where a progressive Point Generation Block is designed to generate and prune intermediate point clouds iteratively. By employing multiple generative deconvolutions and adaptive pruning with different coefficients, our method progressively reconstructs the target point cloud while preserving fine-grained local structures. Furthermore, we integrate semantic information into each layer of the hourglass encoder through a Sparse Semantic Embedding (SSE) module, enhancing point feature extraction. To further strengthen feature representation, we propose a Multi-Scale SSE (MSSE) module, which refines the fused features across spatial and channel dimensions. Extensive experiments on multiple indoor scene datasets demonstrate that our method consistently outperforms state-of-the-art approaches across most evaluation metrics.

We present a novel spherical image compression approach based on Spherical Transformer Block(STB) that takes advantage of HEALPix's hierarchical structure, as illustrated in Figure 1. More specifically, we exploit the hierarchical properties of HEALPix to partition the sphere into several smaller windows and apply multi-head attention within each window, thereby alleviating the computational burden of the spherical Transformer. However, as there is no communication between neighboring windows, we incorporate spherical convolution to aggregate information from adjacent windows. In addition, we propose an inverted residual bottleneck module for feature embedding and a feedforward neural network. Experimental results show that our method outperforms the OSLO [1] approach in terms of performance, while also achieving faster codec times.

Emerging virtual reality (VR) applications bring significant challenges to spherical image compression. Spherical images are first converted into planar images using projections like the equirectangular projection (ERP) to facilitate compression. Methods based on deep neural networks (DNNs) have achieved optimal rate-distortion (R-D) performance in planar image compression. However, the non-uniform sampling of ERP makes the R-D optimization process inefficient when using DNN-based planar compression methods. To address this problem, we propose spherical DNNs for learning based spherical image compression using uniform sampling and ordered rooted tree based index of the Spherical Measure-Based Spherical Image Representation (SMSIR). Specifically, we first define basic spherical operations under the ordered rooted tree based index, including spherical convolution and window transformer, to exploit both local and non-local correlations on the sphere, respectively. We then construct a spherical convolution and a self-attention integrated transformer module named SMixFormer, which simultaneously considers both the enlargement of the receptive fields of local windows and the capture of local and non-local correlations. Furthermore, we introduce a spherical transformer context model with an ordering following the ordered rooted tree based index to enhance the accuracy of the entropy model. To optimize our model, we collect a high-resolution and high-quality spherical image dataset from the Internet. Experimental results demonstrate that our approach outperforms traditional image compression standards, including JPEG, JPEG2000, and BPG. Compared to the learning-based hyperprior planar image compression model, our method achieves a bitrate reduction of over 16%.

Deep unfolding networks have gained increasing attention in the field of compressed sensing (CS) owing to their theoretical interpretability and superior reconstruction performance. However, most existing deep unfolding methods often face the following issues: (1) they learn directly from single-channel images, leading to a simple feature representation that does not fully capture complex features; and (2) they treat various image components uniformly, ignoring the characteristics of different components. To address these issues, we propose a novel wavelet-domain deep unfolding framework named WTDUN, which operates directly on the multi-scale wavelet sub-bands. Our method utilizes the intrinsic sparsity and multi-scale structure of wavelet coefficients to achieve a tree-structured sampling and reconstruction, effectively capturing and highlighting the most important features within images. Specifically, the design of tree-structured reconstruction aims to capture the inter-dependencies among the multi-scale sub-bands, enabling the identification of both fine and coarse features, which can lead to a marked improvement in reconstruction quality. Furthermore, a wavelet domain adaptive sampling method is proposed to greatly improve the sampling capability, which is realized by assigning measurements to each wavelet sub-band based on its importance. Unlike pure deep learning methods that treat all components uniformly, our method introduces a targeted focus on important sub-bands, considering their energy and sparsity. This targeted strategy lets us capture key information more efficiently while discarding less important information, resulting in a more effective and detailed reconstruction. Extensive experimental results on various datasets validate the superior performance of our proposed method.

End-to-end learned image compression exploits the expressive power of nonlinear transform modules to de-correlate the spatial redundancies of image contents. Due to its long-range attention scheme, transformer-based transforms can explore more global features for better reconstruction. However, transformer modules bring in indispensable computational costs, and the coarse utilization of transformer in learned image compression cannot meet the coding efficiency. In this paper, we propose a novel graph-structured swin-transformer for learned image compression, shown in Figure 1 . We assume that the global receptive field of attention map should be sparse not dense, while the local neighboring correlations must be strong.

Point Cloud Geometry Compression (PCGC) methods have become increasingly popular for storing and transmitting point clouds due to their efficiency. However, most of the existing PCGC methods have some limitations. First, they do not explore the spatial correlation among different frequency components. Second, they ignore the inter-channel correlation in the latent feature domain. To address these limitations, we propose our Space-Channel Mixed Context (SCMC) model, which leverages both the spatial correlation and the inter-channel correlation to improve the accuracy of estimated probability and achieve significant bits saving. The SCMC model consists of a Frequency Decomposition Module (FDM) and a Grouped Channel Context Model (GCCM). The FDM decomposes the point cloud into multiple spatial frequency components and models the spatial dependency from low frequency to high frequency. The GCCM groups feature channels in an even way and embeds a spatial context model into each group to improve coding efficiency without compromising running time. Additionally, we introduce SparseCNN-based Feature Transform (SFT) modules to enhance the reconstruction quality, and we propose a framework called MC-PCGC for efficient point cloud compression. Our extensive experimental results demonstrate that our approach significantly improves compression performance.

Point cloud upsampling concerns producing a dense and uniform point set from a sparse and irregular one. Current upsampling methods primarily encounter two challenges: (i) insufficient uni-modal representations of sparse point clouds, and (ii) inaccurate estimation of geometric details in dense point clouds, resulting in suboptimal upsampling results. To tackle these challenges, we propose MVP-Net, a multi-view depth image guided cross-modal detail estimation distillation network for point cloud upsampling, in which the multi-view depth images of point clouds are fully explored to guide upsampling. Firstly, we propose a cross-modal feature extraction module, consisting of two branches designed to extract point features and depth image features separately. This setup aims to produce sufficient cross-modal representations of sparse point clouds. Subsequently, we design a Multi-View Depth Image to Point Feature Fusion (MVP) block to fuse the cross-modal features in a fine-grained and hierarchical manner. The MVP block is incorporated into the feature extraction module. Finally, we introduce a paradigm for multi-view depth image-guided detail estimation and distillation. The teacher network fully utilizes paired multi-view depth images of sparse point clouds and their dense counterparts to formulate multi-hierarchical representations of geometric details, thereby achieving high-fidelity reconstruction. Meanwhile, the student network takes only sparse point clouds and their multi-view depth images as input, and it learns to predict the multi-hierarchical detail representations distilled from the teacher network. Extensive qualitative and quantitative results on both synthetic and real-world datasets demonstrate that our method outperforms state-of-the-art point cloud upsampling methods.

Deep unfolding network (DUN) is a powerful technique for image compressive sensing that bridges the gap between optimization methods and deep networks. However, DUNs usually rely heavily on single-domain information, overlooking the inter-domain dependencies. Therefore, such DUNs often face the following challenges: 1) information loss due to the inefficient representation within a single domain, and 2) limited robustness due to the absence of inter-domain dependencies. To overcome these challenges, we propose a deep unfolding framework D^3U-Net that establishes a dual-domain collaborative optimization scheme. This framework introduces both visual representations from the image domain and multi-resolution analysis provided by the wavelet domain. Such dual-domain representations constrain the feasible region within the solution space more accurately. Specifically, we design a consistency-difference collaborative mechanism to capture inter-domain dependencies effectively. This mechanism not only enhances the fidelity of reconstruction but also enriches the depth and breadth of extracted features, improving the overall robustness and reconstruction quality. Moreover, we develop an inter-stage transmission pathway to minimize the information loss during transmission while broadcasting multi-scale features in a frequency-adaptive manner. Extensive experimental results on various benchmark datasets show the superior performance of our method.

The Transformer architecture has surpassed traditional CNN-based methods in the field of learned image compression, primarily due to its expansion of the receptive field. In learned image compression, the effective receptive field is crucial. Although the Transformer theoretically has an extensive receptive field, in image compression models, its effective receptive field is much smaller than the theoretical value, accompanied by higher computational costs. To address this challenge, this paper proposes an innovative Multi-scale Spatial Channel Fusion(MSCF) mechanism that not only brings the effective receptive field of CNNs on par with Transformers but also retains the low complexity and high efficiency of CNNs. Additionally, learned image compression tends to lose a significant amount of high-frequency components. To compensate for this deficiency, we introduce a High-Frequency Enhancement(HFE) module. We integrate the MSCF mechanism and the HFE module into the MLIC++ framework. Experimental results indicate that our proposed model, Multiscale Feature Extraction and High-Frequency Enhancement for Learned Image Compression (HMLIC) achieves a substantial performance improvement over the baseline model across the Kodak, CLIC Professional Validation and Tecnick test datasets, while incurring only a minimal increase in model complexity.

Guided depth map super-resolution (GDSR) is one of the mainstream methods in depth map super-resolution, as high-resolution color images can guide the reconstruction of the depth maps and are often easy to obtain. However, how to make full use of extracted guidance information of the color image to improve the depth map reconstruction remains a challenging problem. In this paper, we first design a multi-scale feedback module (MF) that extracts multi-scale features and alleviates the information loss in network propagation. We further propose a novel multi-scale feedback network (MSF-Net) for guided depth map super-resolution, which can better extract and refine the features by sequentially joining MF blocks. Specifically, our MF block uses parallel sampling layers and feedback links between multiple time steps to better learn information at different scales. Moreover, an inter-scale attention module (IA) is proposed to adaptively select and fuse important features at different scales. Meanwhile, depth features and corresponding color features are interacted using cross-domain attention conciliation module (CAC) after each MF block. We evaluate the performance of our proposed method on both synthetic and real captured datasets. Extensive experimental results validate that the proposed method achieves state-of-the-art performance in both objective and subjective quality.

In the Coded Aperture Snapshot Spectral Imaging (CASSI) systems, hyperspectral images (HSIs) reconstruction methods are employed to recover 3D signals from 2D compressive measurements. Among these methods, deep unfolding networks exhibit the benefits of interpretability and high efficiency, but they still have some notable shortcomings. Firstly, existing methods primarily exploit the spatial-spectral domain information of HSIs, neglecting exploration of the frequency domain, which is also beneficial to 3D HSIs. Secondly, current unfolding networks have limited utilization of information between different stages, failing to fully explore their relevance and thereby limiting the effectiveness of the overall framework. To address these issues, in this paper, we propose an integrated framework with dual-domain feature fusion and multi-level memory enhancement. Specifically, the former represents the first attempt to utilize frequency domain information in the feature space of HSIs overcoming the limitation of spatial-spectral domain features and thereby improving the data expression ability of the network by extracting dual-domain features. Simultaneously, our verification experiments also show that the proposed dual-domain feature representation can indeed extract complementary feature information in HSIs. Moreover, the latter aims to use the structural characteristics of the U-Net network to fully extract the correlation of information between different stages by designing a multi-level memory enhancement network. Extensive experimental results on various datasets validate the superiority of the proposed approach in both subjective and objective outcomes. Our proposed method achieves an average of 0.4dB improvement over the best counterpart method. And the code can be obtained from the link: https://github.com/yingyangke/DFFMM.

Adaptive transform coding is gaining more and more attention for better mining of image content over fixed transforms such as discrete cosine transform (DCT). As a special case, graph transform learning establishes a novel paradigm for the graph-based transforms. However, there still exists a challenge for graph transform learning-based image codecs design on natural image compression, and graph representation cannot describe regular image samples well over graph-structured data. Therefore, in this article, we propose a cross-channel graph-based transform (CCGBT) for natural color image compression. We observe that neighboring pixels having similar intensities should have similar values in the chroma channels, which means that the prominent structure of the luminance channel is related to the contours of the chrominance channels. A collaborative design of the learned graphs and their corresponding distinctive transforms lies in the assumption that a sufficiently small block can be considered smooth, meanwhile, guaranteeing the compression of the luma and chroma signals at the cost of a small overhead for coding the description of the designed luma graph. In addition, a color image compression framework based on the CCGBT is designed for comparing DCT on the classic JPEG codec. The proposed method benefits from its flexible transform block design on arbitrary sizes to exploit image content better than the fixed transform. The experimental results show that the unified graph-based transform outperforms conventional DCT, while close to discrete wavelet transform on JPEG2000 at high bit-rates.

Recently, point-based point cloud geometry compression has attracted great attention due to its superior performance at low bit rates. However, lacking an efficient way to represent the local geometric correlation well, most existing methods [1, 2, 3] can hardly extract fine local features accurately. Thus it’s difficult for them to obtain high-quality reconstruction of local geometry of point clouds.

Recently, neural image compression has received increasing attention as it has achieved superior rate-distortion performance than conventional image compression methods. However, most neural image compression models are fixed rate models, they need to train multiple models for different bit rates, which greatly increases the burden of computation and storage. While some variable-rate models scale the latent features only in the channels, which do not consider the differences in the content of each image. To address these issues, firstly we design a Content Gain Attention Module (CGAM) to learn the relationship between the content of each image and the bit rates. Then we propose a variable-rate neural image compression framework, which utilizes CGAM modules to achieve variable rate in one single model. The framework takes a source image and a Lagrange multiplier λ as inputs and outputs a compressed image with variable rates. Moreover, to achieve more precise rate control, an accurate m, R-λ model is further proposed. Besides, the Channel Gain Unit is utilized to assist variable rate, which scales latent features in a channel-wise manner. Extensive experimental results show that our method is effective and outperforms existing models.

In the coded aperture snapshot spectral compressive imaging (CASSI) system, hyperspectral image (HSI) reconstruction methods are employed to recover 3D signals from 2D compressive measurements. Among these methods, although the deep unfolding networks show the advantages of interpretability and high efficiency, they still have the limitations of insufficient feature utilization and low efficiency of information interaction between stages. To solve these problems, in this paper two well-designed techniques, dubbed as dual-domain feature learning and feature memory-enhanced module, are introduced into a deep unfolding network. The former is proposed to enhance the representation ability of deep networks, while the latter is designed to efficiently promote the cross-stage information interaction. Extensive experimental results validate the efficiency and effectiveness of the proposed method.

Point cloud upsampling aims to generate a dense and uniform point set from a sparse and irregular point set. The core challenge is to accurately restore the geometric structure and local details. To overcome the challenge, this paper presents a novel frequency-aware attention based point cloud upsampling approach, which combines graph filtering and channel attention based on the detection of high spatial-frequency components like edges and contours in the human visual system. To aggregate the features more efficiently, an intra-feature and inter-feature (I2-feature) aggregation block and a structure sensitive transformer block are introduced. On one hand, the I2-feature aggregation block serves to create a complete local representation of each point by aggregating intra and inter features. On the other hand, the structure sensitive transformer block aims to enhance the quality of the expanded point features by capturing the global geometric structures and the fine local details. Furthermore, to improve the quality of the coarse output, a multi-scale spatial refinement unit is applied, which leverages attentional feature fusion and multi-scale attention. Extensive qualitative and quantitative results on both synthetic and real-scanned datasets validate our proposed scheme outperforms state-of-the-art point cloud upsampling methods.

It’s difficult to effectively remove redundancy in Li-DAR point clouds due to their extremely sparse and nonuniform distribution. Taking advantage of both octree-based methods and voxel-based schemes, we propose to design an effective temporal-spatial context to compress the sequence octree-structured point cloud data into a more compact bitstream. In this paper, we first build a temporal-spatial multiscale context for the deep learning entropy model. It further utilize the correlation of sequential point cloud data from both the spatial domain and temporal domain. In terms of spatial context, we design a hierarchical dependency in an octree to encode the occupancy information of each non-leaf octree node into a bitstream. We propose to further group the nodes according to their octant which effectively expands the context receptive field. In terms of temporal context, the KNN algorithm is applied to explore the most relative context with the strongest dependency in the temporal domain. Finally, we design a voxel re-localization network to convert the discrete voxels into refined 3D points, which makes up for the coordinate loss in the process of generating an octree. The quantitative evaluation shows that our method outperforms state-of-the-art baselines with saving most bitrate on KITTI Odometry dataset, and achieving the best reconstreuction benefit by the designed refinement module.

Recently, a variable-rate image compression conditioned on a pixel-wise quality map achieved an outstanding rate-distortion trade-off compared to the approaches based on multiple models. However, it is hard to find an appropriate pixel-wise quality map for the target rate, and the hyperprior is only used to capture the probability distribution of latent representation, which causes its inefficient utilization. In this paper, we propose a variable-rate image compression network with shallow to deep hyperprior trained with a uniform quality map generated by the trade-off parameter$\lambda $of the rate-distortion optimization. The shallow to deep hyperprior structure enables the shallow hyperprior to improve the reconstruction image quality by compensating the latent representation as side information while estimating its distribution. Inspired by the fact that a uniform quality map can continuously scale the latent representations with the continuous change of$\lambda $, we build an individual$R$-$\lambda $model, which characterizes the relation between R and$\lambda $. With this$R$-$\lambda $model, we can perform precise and continuous rate control in the compression. To the best of our knowledge, this$R$-$\lambda $model is the first work to propose a continuous rate control for variable-rate learning-based image compression. Meanwhile, benefiting from the uniform values of the map, our method can handle various distortion metrics, such as MS-SSIM. Extensive experimental results show that the proposed model has achieved SOTA compression performance in variable-rate learning-based methods, achieving comparable compression performance compared with VVC. Furthermore, the rate control of our scheme shows very high accuracy.

In this paper, we present a deep correlated image set compression scheme based on Distributed Source Coding(DSC) and multi-scale image fusion. As there exists strong correlation among images in a similar image set, we propose to utilize such correlation to generate side information at decoder side for each image in the set. Specifically, a reference structure of the image set is generated by building a minimum spanning tree according to the similarity between two images at encoder. With the reference structure, the side information of each image to be decoded can be generated based on the decoded reference image. And our network learns the correlation between an image and its side information in the training phase. Based on the principle of DSC, the side information can provide additional information such as rich details at decoder side. To make full use of the side information, the initially decoded image and the additional side information are fused at different scales. A decompressed image enhancement network is introduced to reduce the compression artifacts of the decoded images. Extensive experimental results compared with other mainstream methods validate the superior performance of our scheme in both terms of subjective and objective quality.

Point clouds captured in real-world applications are often in-complete due to the limited sensor resolution, viewpoint, and occlusion. Therefore, recovering the completion point clouds from incomplete ones becomes an important work in many practical applications. However, most of previous work only focus on the point-to-point relationship between the reconstructed point cloud and groundtruth(GT), not fully exploring their local geometry relationship, which results in the inaccu-rate and nonuniform local details. To solve this problem, we propose a novel local geometry preserving point cloud com-pletion network(LGP-Net). By promoting the consistency of local geometry features between the reconstruction and GT, the proposed LGP-Net can preserve more accurate local de-tails. To further explore the local geometry correlation at dif-ferent scales, a multi-scale local geometry consistency is also proposed. Moreover, the consistency between the features at different scales are proposed to exploit the correlation of features under different resolutions. Quantitative and qualitative results on the benchmark dataset demonstrate that our LGP-Net achieves superior performance over several state-of-the-art methods significantly.

Motion estimation is a key component to remove the temporal redundancy in video compression. It is well-known that bi-directional motion estimation outperforms sequential motion estimation because of its capability to use both forward and backward reference frames. Previous approaches perform a coarse motion prediction operation to further remove motion spatial redundancy, which heavily rely on the regularity of the motion. However, most motions in natural video sequences are extremely complicated and objects usually move irregularly. To solve this problem, in this paper, we propose a fine motion prediction network by learning an importance map between the bi-directional references. Our designed network can generate more accurate prediction of the motion, yielding less residual. And it is also universal for videos with complex and irregular motions. Both objective and subjective quality results validate the effectiveness of our approach.

Unlike natural images, the topology similarity among meshes can hardly be handled with classical deep learning because of their irregular structures. Parameterization provides a way to represent meshes in the form of geometry and normal images, which reflects the correlation between neighboring sample locations. Generative Adversarial Networks (GANs) can efficiently generate images without explicitly computing probability densities of the underlying distribution. However, existing GANs such as Coupled Generative Adversarial Network (CoGAN) generally have two drawbacks: (1) Inability to process unnatural images. (2) Insufficient exploration of the inherent relation between normal and the corresponding geometry image. To address these issues, this paper proposes an efficient method named Prediction-Compensation Generative Adversarial Network (PCGAN) to learn a joint distribution of both geometry and normal images, which aims for generating meshes with two GANs. The consistency of two GANs for the geometry and the normal is guaranteed by utilizing a sequence of prediction-compensation pairs. The sequence can estimate the normal image from the geometry image and compensate the geometry from normal progressively. Particularly, the prediction has a closed-form expression, which provides high estimation accuracy and reduces training complexity. Extensive experimental results on facial mesh generation indicate that our PCGAN outperforms CoGAN and other architectures in retaining the geometry of the faces and in generating realistic face meshes with rich facial attributes such as facial expression and morphology. Moreover, quantitative evaluations demonstrate our superior performance compared with the methods mentioned above.

The edge information plays a key role in the restoration of a depth map. Most conventional methods assume that the color image and depth map are consistent in edge areas. However, complex texture regions in the color image do not match exactly with edges in the depth map. In this paper, firstly, we point out that in most cases the consistency between normal map and depth map is much higher than that between RGB-D pairs. Then we propose a dual normal-depth regularization term to guide the restoration of depth map, which constrains the edge consistency between normal map and depth map back and forth. Moreover, considering the bimodal characteristic of weight distribution that exists in depth discontinuous areas, a reweighted graph Laplacian regularizer is proposed to promote this bimodal characteristic. And this regularization is incorporated into a unified optimization framework to effectively protect the piece-wise smoothness(PWS) characteristics of depth map. By treating depth image as graph signal, the weight between two nodes is adapted according to its content. The proposed method is tested for both noise-free and noisy cases, and is compared against the state-of-the-art methods on both synthesis and real captured datasets. Extensive experimental results demonstrate the superior performance of our method compared with most state-of-the-art works in terms of both objective and subjective quality evaluations. Specifically, our method is more effective on edge areas and more robust to noises.

Under the extreme conditions such as excessive light, insufficient light or high-speed motion, the detection of vehicles by frame-based cameras still has challenges. Event cameras can capture the frame and event data asynchronously, which is of great help to address the object detection under the aforementioned extreme condition. We propose a fusion network with Attention Fusion module for vehicle object detection by jointly utilizing the features of both frame and event data. The frame and event data are separately fed into the symmetric framework based on Gaussian YOLOv3 to model the bounding box (bbox) coordinates of YOLOv3 as the Gaussian parameters and predict the localization uncertainty of bbox with a redesigned cross-entropy loss function of bbox. The feature maps of these Gaussian parameter and confidence map in each layer are deeply fused in the Attention Fusion module. Finally, the feature maps of the frame and event data are concatenated to the detection layer to improve the detection accuracy. The experimental results show that the method presented in this paper outperforms the state-of-the-art methods only using the traditional frame-based network and the joint network combining the event and frame information.

The edge information plays key role in the restoration of depth map. Most conventional methods assume that the RGB-D pairs are consistent in edge areas. In this paper, firstly, we point out that in most cases the consistency between normal map and depth map(N-D pairs) are much higher than that be-tween RGB-D pairs. Then we propose a dual regularization term to guide the restoration of depth map, which constrains the consistency between N-D pairs back and forth. Moreover, a reweighted graph Laplacian prior is incorporated into a unified optimization framework to effectively protect piece-wise smoothness(PWS) characteristics of depth map. By treating depth maps as graph signals, the weight between two nodes is adapted according to its content. Extensive experimental results demonstrate the superior performance of our method compared with other state-of-the-art works in terms of objective and subjective quality evaluations.

Removing undesirable reflections from a single image captured through a glass surface is of broad application to various image processing and computer vision tasks, but it is an ill-posed and challenging problem. Existing traditional single image reflection removal(SIRR) methods are often less efficient to remove reflection due to the limited description ability of handcrafted priors. State-of-the-art learning based methods often cause instability problems because they are designed as unexplainable black boxes. In this paper, we present an explainable approach for SIRR named model-guided unfolding network(MoG-SIRR), which is unfolded from our proposed reflection removal model with non-local autoregressive prior and dereflection prior. In order to complement the transmission layer and the reflection layer in a single image, we construct a deep learning framework with two streams by integrating reflection removal and non-local regularization into trainable modules. Extensive experiments on public benchmark datasets demonstrate that our method achieves superior performance for single image reflection removal.

The Spherical Measure Based Spherical Image Representation (SMSIR) has nearly uniformly distributed pixels in the spherical domain with effective index schemes. Based on SMSIR, the spherical wavelet transform can be efficiently designed, which can capture the spherical geometry feature in a compact manner and provides a powerful tool for spherical image compression. In this paper, we propose an efficient compression scheme for SMSIR images named Spherical Set Partitioning in Hierarchical Trees (S-SPIHT) using the spherical wavelet transform, which exploits the inherent similarities across the subbands in the spherical wavelet decomposition of a SMSIR image. The proposed S-SPIHT can progressively transform spherical wavelet coefficients into bit-stream, and generate an embedded compressed bit-stream that can be efficiently decoded at several spherical image quality levels. The most crucial part of our proposed S-SPIHT is the redesign of scanning the wavelet coefficients corresponding to different index schemes. We design three scanning methods, namely ordered root tree index scanning (ORTIS), dyadic index progressive scanning(DIPS) and dyadic index cross scanning(DICS)to efficiently reorganize the wavelet coefficients. These methods can effectively exploit the self-similarity between sub-bands and the fact that the high-frequency sub-bands mostly contain insignificant coefficients. Experimental results on widely-used datasets demonstrate that our proposed S-SPIHT outperforms the straightforward SPIHT for SMSIR images in terms of PSNR, S-PSNR and SSIM.

Image inpainting methods usually fail to reconstruct reasonable structure and fine-grained texture simultaneously. This paper handles this problem from a novel perspective of predicting low-frequency semantic structural contents and high-frequency detailed textures respectively, and proposes a multi-frequency probabilistic inference model(MPI model) to predict the multi-frequency information of missing regions by estimating the parametric distribution of multi-frequency features over the corresponding latent spaces. Firstly, in order to extract the information of different frequencies without any interference, wavelet transform is utilized to decompose the input image into low-frequency subband and high-frequency subbands. Furthermore, an MPI model is designed to estimate the underlying multi-frequency distribution of input images. With this model, closer approximation to the true posterior distribution can be constrained and maximum-likelihood assignment can be approximated. Finally, based on the proposed MPI model, a two-path network consisting of inference network(InferenceNet) and generation network(GenerationNet) is trained parallelly to enforce the consistency of global structure and local texture between the generated image and ground truth. We qualitatively and quantitatively compare our method with other state-of-the-art methods on Paris StreetView, CelebA, CelebAMask-HQ and Places2 datasets. The results show the superior performance of our method, especially in the aspects of realistic texture details and semantic structural consistency.

Due to the fact that sparse coding in redundant sparse dictionary learning model is NP-hard, interest has turned to the non-redundant sparsifying transform as its sparse coding is computationally cheap. However, natural images typically contain diverse textures that cannot be sparsified well by a non-redundant system. In this paper we propose a new approach for learning redundant sparsifying transform based on equi-angular frame, where the frame and its dual frame are corresponding to applying the forward and the backward transforms. The uniform mutual coherence in the sparsifying transform is enforced by the equi-angular constraint, which better sparsifies diverse textures. In addition, an efficient algorithm is proposed for learning the redundant transform. Experimental results for image representation illustrate the superiority of our proposed method over non-redundant sparsifying transforms. The image denoising results show that our proposed method achieves superior denoising performance, in terms of subjective and objective quality, compared to the K-SVD, the data-driven tight frame method, the learning based sparsifying transform and the overcomplete transform model with block cosparsity (OCTOBOS).

3D depth cameras have become more and more popular in recent years. However, depth maps captured by these cameras can hardly be used in 3D reconstruction directly because they often suffer from low resolution and blurring depth discontinuities. Super resolution of depth maps is necessary. In depth maps, the edge areas play more important role and demonstrate distinct geometry directions compared with natural images. However, most existing super-resolution methods ignore this fact, and they can not handle depth edges properly. Motivated by this, we propose a compound method that combines multi-direction dictionary sparse representation and autoregressive (AR) models, so that the depth edges are presented precisely at different levels. In the patch level, the depth edge patches with geometry directions are well represented by the pre-trained multi-directional dictionaries. Compared with a universal dictionary, multiple dictionaries trained from different directional patches can represent the directional depth patch much better. In the finer pixel level, we utilize an adaptive AR model to represent the local correlation patterns in small areas. Extensive experimental results on both synthetic and real datasets demonstrate that, the proposed model outperforms state-of-the-art depth map super-resolution methods in terms of both quantitative metrics and subjective visual quality.

Deep learning has achieved a preliminary success in image compression due to the ability to learn the nonlinear spaces with compact features that training samples belong to. Unfortunately, it is not straightforward for the network based image compression methods to code multiple highly related images. In this paper, we propose a co-prediction based image compression (CPIC) which uses the multi-stream autoencoders to collaboratively code the multiple highly correlated images by enforcing the co-reference constraint on the multi-stream features. Patch samples fed into the multi-stream autoencoder, are generated through corresponding patch matching under permutation, which helps the autoencoder to learn the relationship among corresponding patches from the correlated images. Each stream network consists of encoder, decoder, importance map network and binarizer. In order to guide the allocation of local bit rate of the binary features, the important map network is employed to guarantee the compactness of learned features. A proxy function is used to make the binary operation for the code layer of the autoencoder differentiable. Finally, the network optimization is formulated as a rate distortion optimization. Experimental results prove that the proposed compression method outperforms JPEG2000 up to 1.5 dB in terms of PSNR.

High-quality depth information has been increasingly used in many real-world multimedia applications in recent years. Due to the limitation of depth sensor and sensing technology, actually, the captured depth map usually has low resolution and black holes. In this paper, inspired by the geometric relationship between surface normal of a 3D scene and their distance from camera, we discover that surface normal map can provide more spatial geometric constraints for depth map reconstruction, as depth map is a special image with spatial information, which we called 2.5D image. To exploit this property, we propose a novel surface normal data guided depth recovery method, which uses surface normal data and observed depth value to estimate missing or interpolated depth values. Moreover, to preserve the inherent piecewise smooth characteristic of depth maps, graph Laplacian prior is applied to regularize the inverse problem of depth maps recovery and a graph Laplacian regularizer(GLR) is proposed. Finally, the spatial geometric constraint and graph Laplacian regularization are integrated into a unified optimization framework, which can be efficiently solved by conjugate gradient(CG). Extensive quantitative and qualitative evaluations compared with state-of-the-art schemes show the effectiveness and superiority of our method.

Unlike images, the topology similarity among meshes can hardly be handled with traditional signal processing tools because of their irregular structures. Geometry image parameterization provides a way to represent 3D meshes in the form of 2D geometry and normal images. However, most existing methods, including the CoGAN are not suitable for such unnatural images corresponding to meshes. To solve this problem, we propose a Prediction Generative Adversarial Network (PGAN) to learn a joint distribution of geometry and normal images for generating meshes. Particularly, we enforce a prediction constraint on the geometry GAN and normal GAN in our PGAN utilizing the inherent relationship between the geometry and normal. The experimental results on face mesh generation indicate that our PGAN outperforms in generating realistic face models with rich facial attributes such as facial expression and retaining the geometry of the faces.

Background based coding is an effective scheme to improve the coding efficiency of surveillance videos. However, it takes a long time to generate a high quality background picture (BG-picture). And the encoding of the high quality BG-picture will increase the bitrate abruptly. To solve these problems, a progressive background updating based coding scheme is proposed in this paper. In the proposed scheme, the BG-picture is updated block by block. To improve the overall coding efficiency, an importance map is designed to select the valid background blocks (B-blocks) progressively which will be encoded with high quality. It is worth noting that only the valid B-blocks is encoded instead of the entire BG-picture. Compared with the reference software of Versatile Video Coding (VVC), the proposed scheme achieves about 23.3 percent bit-rate saving on average.

The rapid growth of surveillance videos poses a huge challenge for video coding technology. To make the best use of the special characteristics of surveillance videos, the prediction and coding methods with high-quality background picture (BG-picture) has been proposed. However, a large number of frames are always required for training in traditional background modeling methods, and too many bits are spent to encode the BG-picture. Therefore, we propose a double background based coding scheme for surveillance videos, in which two background frames are generated from the reconstructed frames and original frames respectively. Then residual frame between the original background and the reconstructed background is encoded to reduce the bit cost of the BG-picture. The experiments on surveillance videos shows that compared with HM14.0, the proposed method can achieve about 17 percent bit rate saving on average. Up to 40 percent bit rate saving can be observed on surveillance videos.

Compressive light field sensing can recover the light field images with a large number of angular views from only a few acquisitions. Conventional reconstruction methods of light field images ignore the physical constraints of the light field images, moreover, the optimization of the projection and the dictionary is separated. In this paper, we present an efficient structural light field reconstruction model called collative optimization of structured projection and dictionary (COSPD) to optimize the projection and dictionary jointly. It takes advantage of the physical constraints of the light field to handle 4-dimension light field signals and the mutual coherence minimization. Thus, the presented COSPD model improves accuracy of reconstruction. Experimental results show that proposed schemes achieve objective and subjective performance improvements on light field images compared with conventional random projection methods.

Traditional synthesis/analysis sparse representation models signals in a one dimensional (1D) way, in which a multidimensional (MD) signal is converted into a 1D vector. 1D modeling cannot sufficiently handle MD signals of high dimensionality in limited computational resources and memory usage, as breaking the data structure and inherently ignores the diversity of MD signals (tensors). We utilize the multilinearity of tensors to establish the redundant basis of the space of multi linear maps with the sparsity constraint, and further propose MD synthesis/analysis sparse models to effectively and efficiently represent MD signals in their original form. The dimensional features of MD signals are captured by a series of dictionaries simultaneously and collaboratively. The corresponding dictionary learning algorithms and unified MD signal restoration formulations are proposed. The effectiveness of the proposed models and dictionary learning algorithms is demonstrated through experiments on MD signals denoising, image super-resolution and texture classification. Experiments show that the proposed MD models outperform state-of-the-art 1D models in terms of signal representation quality, computational overhead, and memory storage. Moreover, our proposed MD sparse models generalize the 1D sparse models and are flexible and adaptive to both homogeneous and inhomogeneous properties of MD signals.

The conventional sparse model relies on data representation in the form of vectors. It represents the vector-valued or vectorized one dimensional (1D) version of an signal as a highly sparse linear combination of basis atoms from a large dictionary. The 1D modeling, though simple, ignores the inherent structure and breaks the local correlation inside multidimensional (MD) signals. It also dramatically increases the demand of memory as well as computational resources especially when dealing with high dimensional signals. In this paper, we propose a new sparse model TenSR based on tensor for MD data representation along with the corresponding MD sparse coding and MD dictionary learning algorithms. The proposed TenSR model is able to well approximate the structure in each mode inherent in MD signals with a series of adaptive separable structure dictionaries via dictionary learning. The proposed MD sparse coding algorithm by proximal method further reduces the computational cost significantly. Experimental results with real world MD signals, i.e. 3D Multi-spectral images, show the proposed TenSR greatly reduces both the computational and memory costs with competitive performance in comparison with the state-of-the-art sparse representation methods. We believe our proposed TenSR model is a promising way to empower the sparse representation especially for large scale high order signals.

In this paper, we propose angle-normal images to reduce the number of normal component channels from three to two and present predicting the angle-normal images by the reconstructed geometry images. We implement the scheme on realistic meshes. Experimental results verify effectiveness of the proposed scheme. For geometry image codec, the proposed scheme outperforms up 1.78 dB PSNR gains, and average 0.57 dB PSNR gains. For normal image codec, the proposed scheme outperforms up 2.03 dB PSNR gains, and average 1.31 dB PSNR gains.

Summary form only given. Motion compensation is the key technique to reduce temporal redundancy in video coding. Interpolation filters are adopted to generate the inter frame prediction for motion compensation with fractional pixel accuracy. In existing video coding standards such as H.264/AVC and HEVC, a set of predefined interpolation filters is adopted in motion compensation. However, predefined interpolation filters cannot adapt to the video content, which may compromise the coding efficiency. In this paper, a content adaptive interpolation scheme is proposed for motion compensation. In the proposed scheme, a set of adaptive interpolation filters is derived for each frame as additional interpolation filters to minimize the inter prediction difference. Rate-distortion optimization is employed to choose between the predefined interpolation filters and the derived adaptive interpolation filters to achieve the best coding performance. The proposed scheme is implemented into the HM 12.1 software and. Experimental results show that the proposed scheme achieves 3.18 percent bit rate saving on average compared with HEVC.

Many fast integer-pixel motion estimation algorithms have been developed for the High Efficiency Video Coding Standard, however the speed of sub-pixel motion estimation still has room for improvement. A fast sub-pixel motion estimation algorithm is proposed in this paper to speed up the sub-pixel search process. First, the proposed scheme skips sub-pixel search process in smooth prediction units. Then a fast sub-pixel search algorithm based on texture direction analysis is proposed to further reduce the computational complexity of subpixel motion estimation. The simulation results show that compared with the Full Sub-pixel Search (FSPS), the encoding complexity of the whole motion estimation process can be reduced by an average of 40.9% with negligible coding performance loss.

As the projection of the real world, videos usually have many repeated patterns with similar structures cross regions, presenting strong non-local correlations. Moreover, different videos own different characteristics. Exploitation of the non-local correlations by off-line training of transforms has attracted considerable attention over the past years for compression. However, the samples used for training the transforms are usually collected from a predefined set of training videos to avoid the transmission of transform matrixes. There is no guarantee that the characteristics of those training videos and the corresponding transforms fit the current coding video very well. To address that, this paper proposes an internal-video mode dependent directional transform in HEVC for intra coding. In this scheme, for coding a video clip, based on different directions, a set of sample blocks is collected for each direction to train a Karhunen-Loeve transform at the video clip level. During encoding, the better transform between the proposed transform and original DCT/DST in HEVC is determined based on rate-distortion optimization. These transform matrixes are entropy coded to the bitstream. The proposed method can capture the statistical characteristics of the coded video clip and provide efficient transforms. Experimental results show that the proposed method achieves significant performance improvements in comparison with HEVC for intra coding, up to 13% BD-rate savings for the all intra configuration.

Recent years have witnessed the success of the proposed line-based distributed coding and transmission system for broadcasting satellite images (LineCast). The LineCast achieves high compression efficiency and provides smooth multicast capability with soft broadcast and coset coding, in which the precision of side information is crucial. However, the generated side information is rough by averaging several candidates based on template matching. In this paper, we propose a low-rank approximation based LineCast scheme for video broadcasting (LRALineCast) to generate more precise side information by exploiting the cross-frame correlation. The side information generation problem is thus reformulated as a low-rank matrix completion problem, where the approximately low-rank matrix is rearranged by the similar lines searched by template matching. LRALineCast achieves good broadcast performance and avoids the annoying cliff effect by utilizing soft broadcast transmission. The experimental results show that LRALineCast outperforms the state-of-the-art 2D broadcasting scheme and LineCast with up to 2dB coding gain.

Compressive light field photography has gained a significant research interest during the past years. One important problem in light field images is the effective representation of the structures. In this paper, based on collaborative projection and sparse dictionary, we propose a light field reconstruction optimization scheme for compressive light field photography. By utilizing the special structural feature of high correlations among light field views, an optimized structured dictionary is proposed, which is more memory and time efficient compared with general sparse dictionary. Furthermore, we propose the corresponding structured dictionary learning model and efficient algorithm to train the optimized dictionary with fixed projection. Simulation results on light field data set demonstrate the effectiveness of our optimizing scheme in terms of reconstruction quality compared with general sparse dictionary.

Motion estimation plays an important role in video coding standards, such as H.264/AVC and HEVC. In this paper, we propose a multi-stage hash based motion estimation algorithm for HEVC, which enables hash based motion estimation for natural videos. In the proposed method, the prediction blocks significantly different from the current prediction unit will be eliminated in the motion estimation process. Locality sensitive hashing functions are used to measure the difference between the input block and predicted blocks. The proposed algorithm is implemented into the HM 12.0 software, and the simulation results show that the complexity of motion estimation is significantly reduced with negligible coding performance loss.

Image super-resolution with sparsity prior provides promising performance. However, traditional sparse-based super resolution methods transform a two dimensional (2D) image into a one dimensional (1D) vector, which ignores the intrinsic 2D structure as well as spatial correlation inherent in images. In this paper, we propose the first image super-resolution method which reconstructs a high resolution image from its low resolution counterpart via a two dimensional sparse model. Correspondingly, we present a new dictionary learning algorithm to fully make use of the corresponding relationship of two pairs of 2D dictionaries of low and high resolution images, respectively. Experimental results demonstrate that our proposed image super-resolution with 2D sparse model outperforms state-of-the-art 1D sparse model based super resolution methods in terms of both reconstruction ability and memory usage.

Image super-resolution based on sparse model with patch clustering and nonlocal similarity provides promising performance. However, the traditional one dimensional (1D) sparse model enforces a 1D dictionary for every cluster of patches to capture complex structures and different features in images. The total dictionary will take expensive memory, which can be alleviated at cost of representation power. Recently, two dimensional (2D) sparse model has been proved to efficiently represent images and save memory usage. In this paper, we propose to integrate 2D sparse model with patch clustering and nonlocal similarity into a variational framework as 2D nonlocal sparse representation (2DNSR) for image SR to save memory cost and ensure SR performances. We also present a 2DNSR algorithm for image SR where each group of similar patches decompose on the respective 2D dictionaries. Experimental results on image SR demonstrate our proposed 2D nonlocal representation outperforms 2D sparse model and achieves competitive performance to state-of-the-art 1D nonlocal sparse models whereas with much less memory costs.

Two dimensional (2D) sparse representation provides promising performance in image denoising by cooperatively exploiting horizontal and vertical features inherent in images by two dictionaries. In this paper, we first propose integrating the 2D sparse model with clustering and nonlocal regularization into a unified variational framework, defined as 2D nonlocal sparse representation (2DNSR), for optimization. Within this framework, we then present a dictionary learning method for image denoising which jointly decomposes groups of similar noisy patches on subsets of 2D dictionaries. We finally present a 2DNSR-based algorithm for image denoising. Experimental results on image denoising show our proposed 2D nonlocal sparse representation outperforms the 2D sparse model and achieves competitive performance to state-of-the-art nonlocal sparse models whereas with much less memory costs.

By considering the increasing importance of screen contents, the high efficiency video coding (HEVC) standard includes screen content coding as one of its requirements. In this paper, we demonstrate that enabling frame level block searching in HEVC can significantly improve coding efficiency on screen contents. We propose a hash-based block matching scheme for the intra block copy mode and the motion estimation process, which enables frame level block searching in HEVC without changing the HEVC syntaxes. In the proposed scheme, the blocks sharing the same hash values with the current block are selected as prediction candidates. Then the hash-based block selection is employed to select the best candidates. To achieve the best coding efficiency, the rate distortion optimization is further employed to improve the proposed scheme by balancing the coding cost of motion vectors and prediction difference. Compared with HEVC, the proposed scheme achieves 21% and 37% bitrate saving with all intra and low delay configurations with encoding time reduction. Up to 59% bitrate saving can be achieved on sequences with large motions.

Screen contents with complex structure contain random combination of texts, graphics and camera-captured images, which makes them difficult to be compressed efficiently by traditional video codecs. In this paper, we propose a 2-D dictionary based scheme to exploit the repeated patterns on screen content. In the proposed scheme, the current block is predicted from the reconstructed region using a hash-based block searching scheme. A hierarchical two-level hash based searching scheme is designed to find the best matching block for each block. The first-level hash function is used to search the blocks similar to the current block in the constructed 2-D dictionary. The second-level hash function is used to update the 2-D dictionary, which filters out the identical blocks from the blocks found using the first-level hash function. The proposed scheme is incorporated into HEVC framework as an additional mode. Experimental results show that the proposed scheme achieves significantly coding performance improvements on screen contents compared with HEVC.

Dictionary learning for sparse representation has been an active topic in the field of image processing. Most existing dictionary learning schemes focus on the representation ability of the learned dictionary. However, according to the theory of compressive sensing, the mutual incoherence of the dictionary is of crucial role in the sparse coding. Thus incoherent dictionary is desirable to improve the performance of sparse representation based image restoration. In this paper, we propose a new incoherent dictionary learning model that minimizes the representation error and the mutual incoherence by incorporating the constraint of mutual incoherence into the dictionary update model. The optimal incoherent dictionary is achieved by seeking an optimization solution. An efficient algorithm is developed to solve the optimization problem iteratively. Experimental results on image denoising demonstrate that the proposed scheme achieves better recovery quality and converges faster than K-SVD while keeping lower computation complexity.

Conventional hierarchical image representation methods, e.g. Wavelet transform, use pre-determined filter banks which lack in adaption to the variant statistical characteristics of images. In this paper, we propose learning adaptive filter banks for hierarchical sparse image representation with a wavelet-like compact form using a deconvolutional network. The proposed scheme is verified by evaluating its sparsity in image representation. Experimental results demonstrate that the proposed scheme outperforms 9/7 and 5/3 wavelets transform in terms of both objective and subjective qualities under the same sparsity.

Screen content like cartoons, captures of typical computer screens or video with text overlay or news ticker is an important category of video, which needs new techniques beyond the existing video coding techniques. In this paper, we analyze the characteristics of screen content and coding efficiency of HEVC on screen content. We propose a new coding scheme, which adopts a non-transform representation, separating screen content into color component and structure component. Based on the proposed representation, two coding modes are designed for screen content to exploit the directional correlation and non-translational changes in screen video sequences. The proposed scheme is then seamlessly incorporated into the HEVC structure and implemented into HEVC range extension reference software HM9.0. Experimental results show that the proposed scheme achieves up to 52.6% bitrate saving compared with HM9.0. On average, 35.1%, 29.2% and 23.6% bitrate saving are achieved with intra, random-access and low-delay configurations, respectively. The visual quality of the decoded video sequences is also significantly improved by reducing ringing artifacts around sharp edges and reserving the shape of text without blur.

An analysis sparse model represents an image signal by multiplying it using an analysis dictionary, leading to a sparse outcome. It transforms an image (two dimensional signal) into a one-dimensional (1D) vector. However, this 1D model ignores the two dimensional property and breaks the local spatial correlation inside images. In this paper, we propose a two dimensional (2D) analysis sparse model. Our 2D model uses two analysis dictionaries to efficiently exploit the horizontal and vertical features simultaneously. The corresponding sparse coding and dictionary learning algorithm are also presented in this paper. The 2D sparse model is further evaluated for image denoising. Experimental results demonstrate our 2D analysis sparse model outperforms a state-of-the-art 1D analysis model in terms of both denoising ability and memory usage.

Sparse representation has been proved to be very efficient in machine learning and image processing. Traditional image sparse representation formulates an image into a one dimensional (1D) vector which is then represented by a sparse linear combination of the basis atoms from a dictionary. This 1D representation ignores the local spatial correlation inside one image. In this paper, we propose a two dimensional (2D) sparse model to much efficiently exploit the horizontal and vertical features which are represented by two dictionaries simultaneously. The corresponding sparse coding and dictionary learning algorithm are also presented in this paper. The 2D synthesis model is further evaluated in image denoising. Experimental results demonstrate our 2D synthesis sparse model outperforms the state-of-the-art 1D model in terms of both objective and subjective qualities.

Since Magnetic resonance(MR) Images commonly possess a blocky structure and have sparse representations under certain wavelet bases, total variation (TV) and wavelet domain ℓ1norm regularization are often enforced together (TV-wavelet method) to improve the recovery accuracy. However, this model ignores that a family of wavelet coefficients has a natural grouping of its components. In this paper, we propose a new TV-Group sparse model which combines TV and wavelet domain group sparse penalty. The corresponding algorithm based on composite splitting method is employed to approach this TV-Group sparse model. Experimental results show that our model can obviously improve both objective and subjective qualities of MR image recovery comparing with the TV-wavelet model.

Linear representation models are effective to represent the correlation in image interpolation. However, linear models usually lack constraints of the representation coefficient. In this paper, we propose a low rank matrix recovery based image interpolation to reinforce the sparsity of representation coefficient implicitly. Since both the local and nonlocal correlation is pervasive in natural images, we exploit such correlations by incorporating the local and nonlocal modeling, which fully utilizes the redundancy in images and improves the representation ability of our model. By minimizing the sum of the rank of data matrices which reflect the linear relationship among local patch pixels and nonlocal similar patch pixels, a precise low rank approximation of the missing pixels is obtained according to the low rank matrix recovery theory. A Split Bregman based minimization algorithm is developed to efficiently solve the low rank recovery problem. Extensive experimental results indicate the proposed method outperforms the traditional methods in both the objective and subjective visual quality.

Screen content refers to image/video generated by electronic devices. Existing state-of-art image/video coding standards don't exploit the anisotropic features. This paper proposes an adaptive LZMA-based coding scheme for screen content to achieve better coding efficiency. Adaptive scanning and directional matching are incorporated into LZMA in our scheme. Both of them exploit the directional correlations in screen content to increase the matching probability of LZMA, and thus achieve better coding performance. Experimental results show that the proposed scheme significantly outperforms existing LZMA-based coding schemes on screen content coding.

In order to show the realistic 3D mesh in geometry image-based 3D mesh compression, in addition to coding geometry image, normal-map image is usually required to code. But normal-map image are difficult to compress because it captures more details of the original mesh, and it has less spatial correlation between pixels than geometry image. This paper proposes a novel coding framework to solve this problem, we effectively predict the normal-map image based on the correlation between geometry image and normal-map image, and we also utilize the strong correlation among three components of normal-map image to improve the predicting accuracy. In this framework we only need to code geometry image and residual image which generated from normal-map image and its prediction. Experimental results show that comparing with the method which coding geometry image and normal-map image using JPEG2000 directly, our coding framework not only improves the coding efficiency of geometry images and normal-map images, but also enhances the realistic effect of 3D mesh significantly.

Computer generated compound images contain not only photographic images but also text and graphics images, which makes it difficult to be efficiently compressed using exiting image and video coding standards. This paper presents a new compound image coding scheme based on the intra coding framework of the upcoming high efficiency video coding (HEVC) standard. The proposed scheme first decomposes the compound image into color components and structure components. Then two-stage prediction scheme is employed to exploit the correlation among structure components, where the first prediction is generated by directional prediction and second prediction is generated by template matching. Experimental results show that the proposed scheme achieves up to 10.2 dB coding gain on compound images compared with HEVC. And on average it achieves 42.2 percent bitrate saving compared with HEVC.

H.264/AVC adopts intra coding technique to reduce spatial redundancies. In order to achieve better coding performance, there are nine candidate modes for 4×4 block unit to be selected by Rate-Distortion Optimization(RDO) process in the H.264/AVC encoder-side. However, too many intra candidate modes undoubtedly increase the transmitted bits that represent them. To solve this problem, a novel method for reducing the number of intra modes is proposed in this paper. For each 4×4 block, our algorithm can cluster the nine candidate modes into several groups and selects one mode from each group as a representative of all the modes in this group, so the cost of transmission can be reduced significantly because of the reduction of number of candidate modes. Experimental results show that the proposed method achieves 3.2% bit-rate reduction on average for various video sequences. The improvement of coding performance can be up to 7% bit-rate reduction at low bit rate, averagely.

Intra prediction plays an important role in reducing the spatial redundancy for intra frame encoding in H.264/AVC. In this paper, we propose a low-rank matrix completion based intra prediction to improve the prediction efficiency. According to the low-rank matrix completion theory, a low-rank matrix can be exactly recovered from quite limited samples with high probability under mild conditions. After moderate rearrangement and organization, image blocks can be represented as low-rank or approximately low-rank matrix. The intra prediction can then be formulated as a matrix completion problem, thus the unknown pixels can be inferred from limited samples with very high accuracy. Specifically, we novelly rearrange the encoded blocks similar to the current block to generate an observation matrix, from which the prediction can be obtained by solving a low-rank minimization problem. Experimental results demonstrate that the proposed scheme can achieve averagely 5.39% bit-rate saving for CIF sequences and 4.21% for QCIF sequences compared with standard H.264/AVC.

Existing image codec technologies are based on transform which make image signal can be compressed, while quantization has been used to control bit rates. Compressive sensing (CS), which is a novel signal processing and recovery method, can be applied to image decoding to replace inverse transform reconstruction. This paper proposes an error estimate method based on equalization quantization noise model for image codec. Due to the robust character of CS, it can upgrade the quality of reconstruction when error has been estimated accurately. With designed equalization matrix, a new norm constraint which can enhance the quality of CS recovery significantly has been shown. A CS-based JPEG decoding scheme based on quantization error estimate is also presented, and experimental evidence exhibits more gains over CS reconstruction without error estimation and original JPEG decoder.

Textures in many images or video scenes are difficult to code because of the large amount of visible detail. This paper proposes an image coding approach to solve this problem, in which we incorporate image decomposition and texture synthesis technology into the image coding framework. The key idea of our approach is to first decompose the original image into cartoon component u and texture component v with different basic characteristics, and then to synthesize the selected texture regions in texture component v. The cartoon component u and the non-synthetic regions in texture component v are compressed by JPEG. Experimental results show bit-rate savings of over 30% compared with JPEG at similar visual quality levels.

As the newest video coding standard, H.264/AVC uses the rate distortion optimization(RDO) method to select the optimal coding mode after the calculation of all the possible modes. The computational complexity is increased drastically, so it isn't fit for real time encoding and hardware design. In this paper, we propose a fast intra mode decision algorithm based on texture feature for H.264/AVC. First of all, we speculate the smooth degree of a macroblock(MB)'s texture according to the variance value of some pixels , then only choose the intra luma 16 × 16 block for the smooth texture or only choose the intra luma 4 × 4 block for the complex texture . Secondly, we choose part modes of intra 4 × 4 block or intra 16 × 16 block based on direction of texture. The experimental result show our algorithm can reduce the encoding time obviously with little change of PSNR and Bit-rate.