We present a novel spherical image compression approach based on Spherical Transformer Block(STB) that takes advantage of HEALPix's hierarchical structure, as illustrated in Figure 1. More specifically, we exploit the hierarchical properties of HEALPix to partition the sphere into several smaller windows and apply multi-head attention within each window, thereby alleviating the computational burden of the spherical Transformer. However, as there is no communication between neighboring windows, we incorporate spherical convolution to aggregate information from adjacent windows. In addition, we propose an inverted residual bottleneck module for feature embedding and a feedforward neural network. Experimental results show that our method outperforms the OSLO [1] approach in terms of performance, while also achieving faster codec times.

End-to-end learned image compression exploits the expressive power of nonlinear transform modules to de-correlate the spatial redundancies of image contents. Due to its long-range attention scheme, transformer-based transforms can explore more global features for better reconstruction. However, transformer modules bring in indispensable computational costs, and the coarse utilization of transformer in learned image compression cannot meet the coding efficiency. In this paper, we propose a novel graph-structured swin-transformer for learned image compression, shown in Figure 1 . We assume that the global receptive field of attention map should be sparse not dense, while the local neighboring correlations must be strong.

Recently, point-based point cloud geometry compression has attracted great attention due to its superior performance at low bit rates. However, lacking an efficient way to represent the local geometric correlation well, most existing methods [1, 2, 3] can hardly extract fine local features accurately. Thus it’s difficult for them to obtain high-quality reconstruction of local geometry of point clouds.

In this paper, we present a deep correlated image set compression scheme based on Distributed Source Coding(DSC) and multi-scale image fusion. As there exists strong correlation among images in a similar image set, we propose to utilize such correlation to generate side information at decoder side for each image in the set. Specifically, a reference structure of the image set is generated by building a minimum spanning tree according to the similarity between two images at encoder. With the reference structure, the side information of each image to be decoded can be generated based on the decoded reference image. And our network learns the correlation between an image and its side information in the training phase. Based on the principle of DSC, the side information can provide additional information such as rich details at decoder side. To make full use of the side information, the initially decoded image and the additional side information are fused at different scales. A decompressed image enhancement network is introduced to reduce the compression artifacts of the decoded images. Extensive experimental results compared with other mainstream methods validate the superior performance of our scheme in both terms of subjective and objective quality.

Motion estimation is a key component to remove the temporal redundancy in video compression. It is well-known that bi-directional motion estimation outperforms sequential motion estimation because of its capability to use both forward and backward reference frames. Previous approaches perform a coarse motion prediction operation to further remove motion spatial redundancy, which heavily rely on the regularity of the motion. However, most motions in natural video sequences are extremely complicated and objects usually move irregularly. To solve this problem, in this paper, we propose a fine motion prediction network by learning an importance map between the bi-directional references. Our designed network can generate more accurate prediction of the motion, yielding less residual. And it is also universal for videos with complex and irregular motions. Both objective and subjective quality results validate the effectiveness of our approach.

Removing undesirable reflections from a single image captured through a glass surface is of broad application to various image processing and computer vision tasks, but it is an ill-posed and challenging problem. Existing traditional single image reflection removal(SIRR) methods are often less efficient to remove reflection due to the limited description ability of handcrafted priors. State-of-the-art learning based methods often cause instability problems because they are designed as unexplainable black boxes. In this paper, we present an explainable approach for SIRR named model-guided unfolding network(MoG-SIRR), which is unfolded from our proposed reflection removal model with non-local autoregressive prior and dereflection prior. In order to complement the transmission layer and the reflection layer in a single image, we construct a deep learning framework with two streams by integrating reflection removal and non-local regularization into trainable modules. Extensive experiments on public benchmark datasets demonstrate that our method achieves superior performance for single image reflection removal.

The Spherical Measure Based Spherical Image Representation (SMSIR) has nearly uniformly distributed pixels in the spherical domain with effective index schemes. Based on SMSIR, the spherical wavelet transform can be efficiently designed, which can capture the spherical geometry feature in a compact manner and provides a powerful tool for spherical image compression. In this paper, we propose an efficient compression scheme for SMSIR images named Spherical Set Partitioning in Hierarchical Trees (S-SPIHT) using the spherical wavelet transform, which exploits the inherent similarities across the subbands in the spherical wavelet decomposition of a SMSIR image. The proposed S-SPIHT can progressively transform spherical wavelet coefficients into bit-stream, and generate an embedded compressed bit-stream that can be efficiently decoded at several spherical image quality levels. The most crucial part of our proposed S-SPIHT is the redesign of scanning the wavelet coefficients corresponding to different index schemes. We design three scanning methods, namely ordered root tree index scanning (ORTIS), dyadic index progressive scanning(DIPS) and dyadic index cross scanning(DICS)to efficiently reorganize the wavelet coefficients. These methods can effectively exploit the self-similarity between sub-bands and the fact that the high-frequency sub-bands mostly contain insignificant coefficients. Experimental results on widely-used datasets demonstrate that our proposed S-SPIHT outperforms the straightforward SPIHT for SMSIR images in terms of PSNR, S-PSNR and SSIM.

The recent years have witnessed the widespread of light field imaging in interactive and immersive visual applications. To record the directional information of the light rays, larger storage space is required by light field images compared with conventional 2D images. Hence, the efficient compression of light field image is highly desired for further applications. In this paper, we propose a novel light field image compression scheme using multi-branch spatial transformer networks based view synthesis. Firstly, a sparse subset of views are selected and are rearranged into a pseudo sequence to be encoded by a video codec at encoder. Then the other unselected views are synthesized based on the similarity between neighboring views with our proposed method at decoder. To better characterize the non-linear relationship between the sub-views, a multi-branch spatial transformer networks (MSTN) is designed to adaptively learn the affine transformations between the neighboring views, which are used to warp the input views to generate accurate approximation of the target views. Moreover, to better obtain the final view by the generated approximation views, the Wasserstein generative adversarial networks(WGAN) is applied with the improved training. Experimental results show the superior compression performance of our scheme compared with the state-of-the-art methods.

In recent years, the image inpainting technology based on deep learning has made remarkable progress, which can better complete the complex image inpainting task compared with traditional methods. However, most of the existing methods can not generate reasonable structure and fine texture details at the same time. To solve this problem, in this paper we propose a two-stage image inpainting method with structure awareness based on Generative Adversarial Networks, which divides the inpainting process into two sub tasks, namely, image structure generation and image content generation. In the former stage, the network generates the structural information of the missing area; while in the latter stage, the network uses this structural information as a prior, and combines the existing texture and color information to complete the image. Extensive experiments are conducted to evaluate the performance of our proposed method on Places2, CelebA and Paris Streetview datasets. The experimental results show the superior performance of the proposed method compared with other state-of-the-art methods qualitatively and quantitatively.

High-quality depth information has been increasingly used in many real-world multimedia applications in recent years. Due to the limitation of depth sensor and sensing technology, actually, the captured depth map usually has low resolution and black holes. In this paper, inspired by the geometric relationship between surface normal of a 3D scene and their distance from camera, we discover that surface normal map can provide more spatial geometric constraints for depth map reconstruction, as depth map is a special image with spatial information, which we called 2.5D image. To exploit this property, we propose a novel surface normal data guided depth recovery method, which uses surface normal data and observed depth value to estimate missing or interpolated depth values. Moreover, to preserve the inherent piecewise smooth characteristic of depth maps, graph Laplacian prior is applied to regularize the inverse problem of depth maps recovery and a graph Laplacian regularizer(GLR) is proposed. Finally, the spatial geometric constraint and graph Laplacian regularization are integrated into a unified optimization framework, which can be efficiently solved by conjugate gradient(CG). Extensive quantitative and qualitative evaluations compared with state-of-the-art schemes show the effectiveness and superiority of our method.

In this paper, we propose angle-normal images to reduce the number of normal component channels from three to two and present predicting the angle-normal images by the reconstructed geometry images. We implement the scheme on realistic meshes. Experimental results verify effectiveness of the proposed scheme. For geometry image codec, the proposed scheme outperforms up 1.78 dB PSNR gains, and average 0.57 dB PSNR gains. For normal image codec, the proposed scheme outperforms up 2.03 dB PSNR gains, and average 1.31 dB PSNR gains.