The continuous emission of spike stream offers more significant advantages over traditional fixed low sampling rate cameras. Although many reconstruction methods from spike streams have been proposed, the quality of recovered images remains suboptimal. Coarse-to-fine high-speed motion scene reconstruction reconstructs the spike sequence by dividing the dynamic and static regions. However, issues of limited texture richness and low contrasts are unsolved during the reconstruction of static spike. To address these issues, we propose a Coarse-to-Fine spatio-temporal Luminance-Aware Reconstruction (CFLAR) framework. Specially, we propose an adaptive luminance-aware reconstruction in spatio-temporal domain. To be specific, in the spatial domain, we perform region division and region merging of spike sequences based on luminance information, while non-uniform quantization of luminance information is mainly achieved through binary division and adaptive parameters division. In the temporal domain, we integrate alterable window length into the texture from playback to propose adaptive time shift window reconstruction, which enables to obtain reconstructed images with richer textures and higher contrasts. Experimental results demonstrate that our CFLAR method outperforms state-of-the-art approaches in terms of objective and subjective quality.

Single-view 3D reconstruction is a fundamental operation in computer vision. Although significant progress has been made by learning-based approaches, it remains a challenge that the reconstructed mesh is usually coarse since the geometric prior is ignored. In this paper, we propose a geometric prior based graph convolution neural network model (named G2CNN) for single-view 3D reconstruction with Loop subdivision. G2CNN is a data-driven deep neural network (DNN) with the geometry knowledge. To make the reconstructed results with abundant geometric details, we generate shapes with a coarse-to-fine strategy and utilize the Gaussian curvature loss as a geometric supervision. Furthermore, to produce the physically accurate 3D geometry, the mesh subdivision module is designed with Loop subdivision to exploit the vertex localizations and connectivity, which can refine and smooth the mesh surface. Experimental results on both synthesized data and real data demonstrate the effectiveness of our method in terms of both subjective and objective quality.

Depth map captured by 3D cameras usually suffers from low resolution and insufficient quality, which limits its applications in real world. Thus, it is an essential task to develop efficient and effective techniques to handle various depth degradations. In this paper, we propose a color guided depth map super-resolution method with nonlocal autoregressive modeling. Considering that textures in depth map demonstrate distinct geometry direction, we exploit the multi-directional dictionary which is effective in recovering subtle structures of depth patches. We further introduce two regularization terms into the sparse representation framework. Firstly, a patch based autoregressive model is introduced to represent the local patterns in a small area. Secondly, inspired by the structure consistence between depth map and color image, we propose a color guided nonlocal similarity to provide nonlocal constraint to the local structures, which is very helpful in preserving local structures and suppressing noise. Experimental results demonstrate the superior of our method compared with state-of-the-art methods.

Depth maps have been widely used in many real world applications, such as human-computer interaction and virtual reality. However, due to the limitation of current depth sensing technology, the captured depth maps usually suffer from low resolution and insufficient quality. In this paper, we propose a depth map super-resolution method via joint local gradient and nonlocal structural regularizations. Depth maps contain mainly smooth areas separated by textures which demonstrate distinct geometry direction characteristic. Motivated by this, we classify depth map patches according to their geometrical directions and learn a compact online dictionary in each class. We further introduce two regularization terms into the sparse representation framework. Firstly, a multi-directional total variation model is proposed to characterize the local patterns in the gradient domain. Secondly, a nonlocal autoregressive model is introduced to provide nonlocal constraint to the local structures, which can effectively restore image details and suppress noise. Quantitative and qualitative evaluations compared with state-of-the-art methods demonstrate that the proposed method achieves superior performance for various configurations of magnification factors and datasets.

As a powerful statistical signal modeling technique, sparse representation has been widely used in various image restoration (IR) applications. The sparsity-based methods have achieved leading performance in the past few decades. However, in recent years it has been surpassed by other methods, especially the recent deep learning based methods. In this paper, we address the question that whether sparse representation can be competitive again. The way we answer this question is to redesign it with a deep architecture. To be specific, we propose an end-to-end deep architecture that follows the process of the sparse representation based IR. In particular, we learn a sparse convolutional dictionary to replace the traditional dictionary, and a convolutional neural network (CNN) denoising prior to replace the image prior. Through end-to-end training, the parameters in convolutional dictionary and CNN denoiser can be jointly optimized. Experimental results on several representative IR tasks, including image denoising, deblurring and super-resolution, demonstrate that the proposed deep network can achieve superior performance against state-of-the-art model-based and learning-based methods.

Depth maps acquired by 3D cameras usually suffer from low resolution and insufficient quality, which makes it difficult to be directly used in visual depth perception and 3D reconstruction. To handle this problem, we propose a novel multi-direction dictionary and joint regularization model for high quality depth recovery. To enhance the sparsity, image is divided into classified patches according to the same geometrical direction and a compact dictionary is trained within each class. Then for a patch to be coded, the most relevant dictionary can be selected according to its geometrical direction. We further introduce two regularization terms into the reconstruction model. One is the anisotropic total variation (TV) defined by the local gradients of color/depth pair. The other is the nonlocal similarity to provide nonlocal constraint to the local structure. Experimental results demonstrate that our method outperforms other state-of-the-art methods in terms of both subjective quality and objective quality.

3D depth cameras have become more and more popular in recent years. However, depth maps captured by these cameras can hardly be used in 3D reconstruction directly because they often suffer from low resolution and blurring depth discontinuities. Super resolution of depth maps is necessary. In depth maps, the edge areas play more important role and demonstrate distinct geometry directions compared with natural images. However, most existing super-resolution methods ignore this fact, and they can not handle depth edges properly. Motivated by this, we propose a compound method that combines multi-direction dictionary sparse representation and autoregressive (AR) models, so that the depth edges are presented precisely at different levels. In the patch level, the depth edge patches with geometry directions are well represented by the pre-trained multi-directional dictionaries. Compared with a universal dictionary, multiple dictionaries trained from different directional patches can represent the directional depth patch much better. In the finer pixel level, we utilize an adaptive AR model to represent the local correlation patterns in small areas. Extensive experimental results on both synthetic and real datasets demonstrate that, the proposed model outperforms state-of-the-art depth map super-resolution methods in terms of both quantitative metrics and subjective visual quality.

Depth cameras have gained significant popularity due to their affordable cost in recent years. However, the resolution of depth map captured by these cameras is rather limited, and thus it hardly can be directly used in visual depth perception and 3D reconstruction. In order to handle this problem, we propose a novel multiclass dictionary learning method, in which depth image is divided into classified patches according to their geometrical directions and a sparse dictionary is trained within each class. Different from previous SR works, we build the correspondence between training samples and their corresponding register color image via sparse representation. We further use the adaptive autoregressive model as a reconstruction constraint to preserve smooth regions and sharp edges. Experimental results demonstrate that our method outperforms state-of-the-art methods in depth map super-resolution in terms of both subjective quality and objective quality.