Spike camera is a kind of neuromorphic sensor that uses a novel ``integrate-and-fire'' mechanism to generate a continuous spike stream to record the dynamic light intensity at extremely high temporal resolution. However, as a trade-off for high temporal resolution, its spatial resolution is limited, resulting in inferior reconstruction details. To address this issue, this paper develops a network (SpikeSR-Net) to super-resolve a high-resolution image sequence from the low-resolution binary spike streams. SpikeSR-Net is designed based on the observation model of spike camera and exploits both the merits of model-based and learning-based methods. To deal with the limited representation capacity of binary data, a pixel-adaptive spike encoder is proposed to convert spikes to latent representation to infer clues on intensity and motion. Then, a motion-aligned super resolver is employed to exploit long-term correlation, so that the dense sampling in temporal domain can be exploited to enhance the spatial resolution without introducing motion blur. Experimental results show that SpikeSR-Net is promising in super-resolving higher-quality images for spike camera.

Aging subjects with neurodegenerative conditions have multiple contributors and pathology progression patterns that result in heterogeneous disease biology and different disease phenotypes. Clinical data play a crucial role in disentangling such disease heterogeneity, but they are usually by noise, which can result in errors in clustering leading to spurious non-clinically relevant clusters. A limitation of conventional neuroimaging clustering methods is neglecting the potential bias caused by noise. To remove noise, we introduce adaptive regularization based on coefficient distribution modeling in transform domain. Different from traditional sparsity techniques that assume zero expectation of the coefficients, we use the data of interest to form the Laplace distributions so that they can depict the statistical characteristics more accurately. Furthermore, we use feature clusters to provide weak supervision for enhanced clustering of subjects. To this end, we employ nonnegative matrix tri-factorization to collaboratively cluster subjects and features. Experimental results on synthetic data and the real-life clinical dataset PRVENT-AD demonstrate superior effectiveness of the proposed approach.

Local and non-local attention-based methods have been well studied in various image restoration tasks while leading to promising performance. However, most of the existing methods solely focus on one type of attention mechanism (local or non-local). Furthermore, by exploiting the self-similarity of natural images, existing pixel-wise non-local attention operations tend to give rise to deviations in the process of characterizing long-range dependence due to image degeneration. To overcome these problems, in this paper we propose a novel collaborative attention network (COLA-Net) for image restoration, as the first attempt to combine local and non-local attention mechanisms to restore image content in the areas with complex textures and with highly repetitive details respectively. In addition, an effective and robust patch-wise non-local attention model is developed to capture long-range feature correspondences through 3D patches. Extensive experiments on synthetic image denoising, real image denoising and compression artifact reduction tasks demonstrate that our proposed COLA-Net is able to achieve state-of-the-art performance in both peak signal-to-noise ratio and visual perception, while maintaining an attractive computational complexity. The source code is available onhttps://github.com/MC-E/COLA-Net.

The recently invented retina-inspired spike camera has shown great potential for capturing dynamic scenes. Different from the conventional digital cameras that compact the photoelectric information within the exposure interval into a single snapshot, the spike camera produces a continuous spike stream to record the dynamic light intensity variation process. For spike cameras, image reconstruction remains an important and challenging issue. To this end, this paper develops a spike-to-image neural network (Spk2ImgNet) to reconstruct the dynamic scene from the continuous spike stream. In particular, to handle the challenges brought by both noise and high-speed motion, we propose a hierarchical architecture to exploit the temporal correlation of the spike stream progressively. Firstly, a spatially adaptive light inference subnet is proposed to exploit the local temporal correlation, producing basic light intensity estimates of different moments. Then, a pyramid deformable alignment is utilized to align the intermediate features such that the feature fusion module can exploit the long-term temporal correlation, while avoiding undesired motion blur. In addition, to train the network, we simulate the working mechanism of spike camera to generate a large-scale spike dataset composed of spike streams and corresponding ground truth images. Experimental results demonstrate that the proposed network evidently outperforms the state-of-the-art spike camera reconstruction methods.

Image restoration techniques generally use intrinsic correlations of image signals to reduce the uncertainty of the unknown signal and estimate the latent ground truth. Local and non-local correlation are the two major kinds of correlations utilized. They are different sources of correlations reflecting connections between different image data, but such a difference is not taken into consideration in most of the existing schemes. Typically, sparse representation based works use the same image data to exploit both local and non-local correlation in shared regularization. This paper aims to fully exploit local and non-local correlation of image contents separately so that near-optimal sparse representations are achieved and thus the uncertainty of signals is minimized. The proposed scheme adaptively selects different image data to exploit local and non-local correlations respectively. In particular, to exploit local correlation, the image data of interest are extracted from clustered rows of patch groups that consist of similar image contents. Experimental results on image denoising show that the proposed scheme not only outperforms state-of-the-art sparsity and low rank based methods, but also surpasses successful deep learning-based approaches in terms of PSNR, SSIM, and visual quality.

G-Cast is a wireless visual communication scheme that conveys visual information via image gradient. It is inspired by the characteristics of human vision systems and can provide improved perceptual quality. G-Cast is power efficient but bandwidth demanding, because gradient data have double the size of the original image. This paper presents a scheme named CG-Cast for scalable image transmission in bandwidth-limited wireless scenarios. It employs a compressive-gradient-based image representation to describe perceptually sensitive image details and reduce the bandwidth requirement at the same time, combining gradient-based visual representation with compressive sensing techniques. The compressive gradient data are transmitted in a pseudo-analog way so that it achieves elegant quality transition in a wide channel signal-to-noise ratio (CSNR) range. CG-Cast also sends a small set of low-frequency data in digital a way to provide the global and local luminance of the image. We developed an effective optimization algorithm for the decoder to reconstruct the original image from the received noisy compressive gradient and the low-frequency part of the image. Experimental results demonstrate that the proposed scheme improves the quality of received images remarkably under different CSNR and channel bandwidth conditions.

In image restoration tasks, image priors generally utilize correlation within image contents to predict the latent image signal. In this paper, we propose to jointly exploit both intra- and inter-patch correlation of the input image, so as to further reduce the uncertainty of the unknown signal, and thus improve the prediction of the latent image. The proposed scheme evolves from the low-rank regularization for non-local highly-correlated image contents. Since the underlying cost function to pursue minimal rank is hard to solve, we use non-convex smooth surrogates for the rank penalty. Two such surrogates are utilized in order to incorporate both intra- and inter-patch correlation. To tackle the optimization problem, we use iterative alternating direction technique to divide the problem into two subproblems, each of which is solved via an empirical Bayesian procedure built upon variational approximation. Experimental results on image denoising show that the proposed approach outperforms several state-of-the-art methods in terms of peak signal-to-noise ratio, structural similarity, and perceptual quality.

Summary form only given: Based on observations that the visual quality has strong correlation with image gradients, gradient based image SoftCast (G-Cast) [1] advocates to convey visual information by delivering image gradients. In G-Cast, both horizontal gradients and vertical gradients needs to be transmitted, even if the channel bandwidth is insufficient. This paper propose to send out the random projection measurements of the gradients instead of delivering gradients directly, so that data size can be reduced to an arbitrary ratio and channel bandwidth occupation can be lowered. We name this scheme as compressive gradient based SoftCast (CG-Cast). At CG-Cast sender, after generated by gradient transform, the gradients are down-sampled by random projection, sample rate of which is set according to the channel bandwidth condition. Then the produced measurements are sent out for raw OFDM transmission. A few lowfrequency components are also transmitted to tell the global luminance. At CG-Cast receiver, the received noisy measurements are used for compressive gradient based reconstruction procedure, which utilizes sparsity in gradient domain and non-local similarity in spatial domain [2]. The proposed method is compared with SoftCast [3] and compressive sensing (CS) in bandwidth limited and power constrained scenarios. To make fair comparison, these three schemes are tested under the same channel signal-to-noise ratio (CSNR) conditions to transmit equal amount of data for reconstruction, using equivalent power and bandwidth. CG-Cast outperforms SoftCast and CS in terms of SSIM and gradient signal-to-noise ratio (GSNR) at different bandwidth ratios. Comparing with SoftCast in different channel conditions, the average SSIM gain of all the tested images varies from 0.04 to 0.13, and the average GSNR gain ranges from 1.5dB to 2.9dB. CS is rather unstable in noisy conditions. SoftCast performs better than CS because of its power allocation.

In wireless visual communication systems, it is crucial to effectively utilize channel power and bandwidth in the pursue of optimal performance, and it is worthwhile to adapt the transmission scheme to human vision system (HVS) so as to achieve perceptually appealing results. Inspired by observations that visual quality of an image is closely related to the gradient data, this paper proposes to convey visual information by random projection measurements of image gradients in an analog framework. Since HVS is more sensitive to luminance variations of image contents, which are contained in the gradient data, the proposed scheme achieves better perceptual quality than conventional analog uncoded schemes like SoftCast. Besides, the gradient transform removes the low and medium frequency components of the image hence substantially reduces the power of the signal transmitted in the analog channel, thus evidently improves the power-distortion performance of the system. Furthermore, by applying random projection to the gradients, the number of transmitted data can be adjusted according to bandwidth conditions. Another contribution of this paper is to develop an effective optimization scheme for the compressive gradient based reconstruction problem. Experimental results validate the effectiveness of the proposed transmission and reconstruction scheme under different channel signal-to-noise ratio and bandwidth conditions.

Based on the observation that a matrix X consisted of non-local highly-correlated patches is of low rank, many image restoration methods use low-rank regularization to exploit correlation between image contents, so that the uncertainty of the unknown image signal can be reduced. To tackle the problem that the underlying cost function to pursue minimal rank is hard to solve, an effective way is to employ smooth non-convex surrogate log |XXT| for the rank penalty. Essentially, such technique only considers to utilize correlation within image patches. In this paper, we propose to jointly exploit both intra- and inter-patch correlation of the input image, so as to further reduce the uncertainty of the signal, and thus improve the prediction of the latent image. The corresponding two surrogates are integrated to incorporate both intra- and inter-patch correlation. To solve the optimization problem, we use iterative alternating direction technique to divide the problem into two subproblems, each of which is solved via an empirical Bayesian procedure built upon variational approximation. Experimental results show that the proposed approach outperforms several state-of-the-art methods in terms of PSNR and perceptual quality.

This paper combines an adaptive reconstruction based approach and a learning based technique into an effective scheme for single image super-resolution. Unlike conventional schemes that adopt pre-trained dictionaries to tell the relationship between high-resolution (HR) image and the low-resolution (LR) observation, the proposed method attempts to learn the joint distribution of highly-correlated patch couples from the input image itself instead of an external dataset, so that the learnt models are specially tailored for the current patches and thus can better fit the image data of interest. To be specific, we first apply spatially adaptive gradient sparsity regularization in the reconstruction of the HR image using the contour information, and then utilize the generated HR output to guide the joint distribution learning to infer the relationship between the highly-correlated HR and LR patches. In this way, we simultaneously exploit the inter-scale correlation as well as the local and non-local correlation of the image contents. Empirical results show that the performance of the proposed method is highly competitive with state-of-the-art schemes in terms of peak signal-to-noise ratio (PSNR) and perceptual quality.

Total variation (TV) regularization is widely used in image restoration to exploit the local smoothness of image content. Essentially, the TV model assumes a zero-mean Laplacian distribution for the gradient at all pixels. However, real-world images are nonstationary in general, and the zero-mean assumption of pixel gradient might be invalid, especially for regions with strong edges or rich textures. This paper introduces a nonlocal (NL) extension of TV regularization, which models the sparsity of the image gradient with pixelwise content-adaptive distributions, reflecting the nonstationary nature of image statistics. Taking advantage of the NL similarity of natural images, the proposed approach estimates the image gradient statistics at a particular pixel from a group of nonlocally searched patches, which are similar to the patch located at the current pixel. The gradient data in these NL similar patches are regarded as the samples of the gradient distribution to be learned. In this way, more accurate estimation of gradient is achieved. Experimental results demonstrate that the proposed method outperforms the conventional TV and several other anchors remarkably and produces better objective and subjective image qualities.

Prior knowledge plays an important role in image denoising tasks. This paper utilizes the data of the input image to adaptively model the prior distribution. The proposed scheme is based on the observation that, for a natural image, a matrix consisted of its vectorized non-local similar patches is of low rank. We use a non-convex smooth surrogate for the low-rank regularization, and view the optimization problem from the empirical Bayesian perspective. In such framework, a parameter-free distribution prior is derived from the grouped non-local similar image contents. Experimental results show that the proposed approach is highly competitive with several state-of-art denoising methods in PSNR and visual quality.

This paper proposes a new image denoising algorithm based on adaptive signal modeling and regularization. It improves the quality of images by regularizing each image patch using bandwise distribution modeling in transform domain. Instead of using a global model for all the patches in an image, it employs content-dependent adaptive models to address the non-stationarity of image signals and also the diversity among different transform bands. The distribution model is adaptively estimated for each patch individually. It varies from one patch location to another and also varies for different bands. In particular, we consider the estimated distribution to have non-zero expectation. To estimate the expectation and variance parameters for every band of a particular patch, we exploit the nonlocal correlation in image to collect a set of highly similar patches as the data samples to form the distribution. Irrelevant patches are excluded so that such adaptively learned model is more accurate than a global one. The image is ultimately restored via bandwise adaptive soft-thresholding, based on a Laplacian approximation of the distribution of similar-patch group transform coefficients. Experimental results demonstrate that the proposed scheme outperforms several state-of-the-art denoising methods in both the objective and the perceptual qualities.

This paper proposes a new image denoising approach using adaptive signal modeling and adaptive soft-thresholding. It improves the image quality by regularizing all the patches in image based on distribution modeling in transform domain. Instead of using a global model for all patches, it employs content adaptive models to address the non-stationarity of image signals. The distribution model of each patch is estimated individually and can vary for different transform bands and for different patch locations. In particular, we allow the distribution model for each individual patch to have non-zero expectation. To estimate the expectation and variance parameters for the transform bands of a particular patch, we exploit the non-local correlation of image and collect a set of similar patches as data samples to form the distribution. Irrelevant patches are excluded so that this non-local based modeling is more accurate than global modeling. Adaptive soft-thresholding is employed since we observed that the distribution of non-local samples can be approximated by Laplacian distribution. Experimental results show that the proposed scheme outperforms the state-of-the-art denoising methods such as BM3D and CSR in both the PSNR and the perceptual quality.

Conventional image and video communication systems are usually designed with the objective being to maximize the fidelity of reconstructed images measured by mean square errors (MSE). It is well known that the fidelity metric MSE may not reflect the visual quality perceived by human eyes. Recent advancements in image quality assessment tell us that the structural similarity (SSIM), especially the gradient similarity, reveals the perceptual fidelity of images more reliably. Inspired by this observation, this paper proposes a new image communication approach, which conveys the visual information in an image by transmitting the image gradients and recovers the image from the received gradient data at decoder side using statistical image prior knowledge. In particular, we designed a gradient-based image SoftCast scheme for wireless scenarios. Experimental results show that the proposed scheme can produce reconstruction images with much better perceptual quality. The advantage in perceptual quality is verified by the quality improvement measured by the metrics SSIM and gradient signal-to-noise ratio (GSNR).

Most existing image coding and communication systems aim to minimize the mean square error (MSE) of the pixels reconstructed at receivers. However, the quality metric MSE has long been criticized for not being consistent with the perception of human vision systems. This paper considers a gradient-based image SoftCast (G-Cast) scheme, based on the recent advancements in image quality assessment which indicate that gradient similarity is highly correlated with perceptual image quality. To reconstruct the image from the received noisy gradient data, we exploit the statistical characteristics of image gradients. Instead of using the very simple Laplacian distribution for image gradient as in the total variation (TV) model, we further exploit the non-local similarity of image patches. A non-local gradient sparsity regularization (NLGSR) method is developed and solved using augmented Lagrangian method. Experimental results show that the proposed scheme provides promising perceptual image quality, and the NLGSR reconstruction scheme outperforms the existing schemes remarkably.

Total-variation (TV) regularization is widely adopted in image restoration problems to exploit the feature that natural images are smooth with small gradient values at most regions. Basic TV method assumes identical zero-mean Laplacian distribution for the gradients at all pixels. However, for real-world images, the statistics of gradients may not be stationary, and the zero-mean assumption of gradients may not be valid either for a specific pixel. This paper presents a non-local extension of TV regularization for image restoration, called Non-Local Gradient Sparsity Regularization (NGSR). The NGSR model employs a separate gradient value distribution for each pixel. To figure out the distribution parameters, the NGSR method exploits a set of patches which are similar to the patch centered at current pixel and estimates the distribution parameter adaptively. Experimental results demonstrate that the proposed NGSR outperforms traditional TV remarkably for image restoration.

Inspired by the recent image quality assessment (IQA) studies which indicate that the image gradient data reflects the visual information more reliably than the image pixels, gradient based transmission scheme was recently proposed to pursue better perceptual quality for wireless visual communication. This paper develops an effective method to reconstruct high quality image from the received noisy gradient data. The proposed method utilizes both local correlation and non-local similarity within the image signal to regularize the reconstruction image. Principle component analysis (PCA) is employed to learn signal-adaptive two-dimensional (2D) transform basis, and 3D transform is performed on grouped similar patches to further decorrelate the coefficients. In this way, distortions can be effectively suppressed via adaptive collaborative shrinkage on the transform coefficients. Experimental results demonstrate that the proposed method improves the reconstruction performance remarkably compared with the existing schemes.