This paper proposes a new loss function to train a convolutional neural network (CNN) for multispectral and hyper-spectral (MS-HS) image fusion. The loss function is based on the relative dimensionless global error synthesis (ER-GAS), where we exchange the mean squared error (MSE) for its unbiased estimate using Stein’s risk unbiased estimate (SURE). The loss function has a good balance between the spectral and spatial information implied by the weighted MSE, therefore it does not need a parameter to balance the spectral and spatial terms as in MSE loss function, and it also converges faster than the MSE one. Additionally, the loss function enables unsupervised training and avoids overfit-ting, since it is derived by using SURE. Experimental results show that the proposed method yields good results and outperforms the competitive methods. Codes are available at https://github.com/hvn2/SURE-ERGAS

Multispectral remote sensing images are often have band-dependent image resolution due to cost and technical limitations. To address this, we developed a method that sharpens low-resolution (LR) images using high-resolution (HR) images. In this paper, we propose a novel unsupervised deep learning (DL) approach that involves unrolling an iterative algorithm into a deep neural network and training it using a loss function based on Stein’s risk unbiased estimate (SURE) to sharpen the LR bands (20 and 60 m) of Sentinel-2 (S2) to their highest resolution (10 m). This approach views traditional optimization model-based methods through a DL framework, improving interpretability and clarifying connections between the two approaches. Results from both simulated and real S2 datasets demonstrate that the proposed method outperforms competitive methods and produces high-quality images for the 20 m and 60 m bands. The codes are available at: https://github.com/hvn2/S2-Unrolling.

Recent advances in deep learning (DL) reveal that the structure of a convolutional neural network (CNN) is a good image prior (called deep image prior (DIP)), bridging the model-based and DL-based methods in image restoration. However, optimizing a DIP-based CNN is prone to over-fitting leading to a poorly reconstructed image. This paper derives a loss function based on Stein's unbiased risk estimate (SURE) for unsupervised training of a DIP-based CNN applied to the hyperspectral image (HSI) super-resolution. The SURE loss function is an unbiased estimate of the mean-square-error (MSE) between the clean low-resolution image and the low-resolution estimated image, which relies only on the observed low-resolution image. Experimental results on HSI show that the proposed method not only improves the performance, but also avoids overfitting. Codes are available at https://github.com/hvn2/SURE-MS-HS

Spectral and spatial correlation in hyperspectral images (HSIs) can be exploited in HSI processing because it directly induces a sparse and low-rank prior via linear transformations. Researchers have used the sparse and low-rank prior as an image prior for HSI restoration, such as denoising, deblurring, and super-resolution. This paper proposes a HSI denoising method that incorporates a sparse and low-rank prior with a deep image prior (DIP). The sparse and low-rank prior is obtained using the 2-dimensional discrete wavelet transform (2-D DWT), and singular value decomposition (SVD), while the DIP is provided by the structure of a convolutional neural network (CNN). The combination of a sparse and low-rank prior with a DIP views the CNN-based denoising method similar to a model-based method, inheriting the advantages of both model-based and CNN-based methods. Experimental results with simulated and real HSI datasets show that the proposed method outperforms the conventional sparse and low-rank based methods in both quantitative and qualitative performance. Codes are available at https://github.com/hvn2/DIP-SLR

Image fusion is utilized in remote sensing due to the limitation of the imaging sensor and the high cost of simultaneously acquiring high spatial and spectral resolution images. Optical remote sensing imaging systems usually provide images of high spatial resolution but low spectral resolution and vice versa. Therefore, fusing those images to obtain a fused image having both high spectral and spatial resolution is desirable in many applications. This paper proposes a fusion framework using an unsupervised convolutional neural network (CNN) and Stein’s unbiased risk estimate (SURE). We derive a new loss function for a CNN that incorporates back-projection mean-squared error with SURE to estimate the projected mean-square-error (MSE) between the fused image and the ground truth. The main motivation is that training a CNN with this SURE loss function is unsupervised and avoids overfitting. Experimental results for two fusion examples, multispectral and hyperspectral (MS-HS) image fusion, and multispectral and multispectral (MS-MS) image fusion, show that the proposed method yields high quality fused images and outperforms the competitive methods. Codes are be available at https://github.com/hvn2/Deep-SURE-Fusion.

This paper proposes a novel method for sharpening the 20 m bands of the multispectral images acquired by the Sentinel-2 (S2) constellation. We formulate the S2 sharpening as an inverse problem and solve it using an unsupervised convolutional neural network (CNN), called S2UCNN. The proposed method extends the deep image prior provided by a CNN structure with S2 domain knowledge. We incorporate a modulation transfer function-based degradation model as a network layer. We add the 10 m bands to both the network input and output to take advantage of the multitask learning. Experimental results with a real S2 dataset show that the proposed method outperforms the competitive methods on reduced-resolution data and gives very high quality sharpened image on full-resolution data.

Hyperspectral images (HSIs) are useful for many remote sensing applications. However, they are usually affected by noise that degrades the HSIs quality. Therefore, HSI denoising is important to improve the performance of subsequent HSI processing and analysis. In this article, we propose an HSI denoising method called Stein's unbiased risk estimate-convolutional neural network (SURE-CNN). The method is based on an unsupervised CNN and SURE. The main difference of SURE-CNN from existing supervised learning methods is that the SURE-based loss function can be computed only from noisy data. Since SURE is an unbiased estimate of the mean squared error (MSE) of an estimator, training a CNN using the SURE loss can yield similar results as using the MSE with ground truth in supervised learning. Also, a subspace version of SURE-CNN is proposed to reduce the running time. Extensive experimental results with both simulated and real data sets show that the SURE-CNN method outperforms the competitive methods in both objective and subjective assessments.

This paper addresses the hyperspectral image (HSI) denoising problem by using Stein's unbiased risk estimate (SURE) based convolutional neural network (CNN). Conventional deep learning denoising approaches often use supervised methods that minimize a mean-squared error (MSE) by training on noisy-clean image pairs. In contrast, our proposed CNN-based denoiser is unsupervised and only makes use of noisy images. The method uses SURE, which is an unbiased estimator of the MSE, that does not require any information about the clean image. Therefore minimization of the SURE loss function can accurately estimate the clean image only from noisy observation. Experimental results on both simulated and real hyperspectral datasets show that our proposed method outperforms competitive HSI denoising methods.

Sentinel-2 (S2) satellite constellations can provide multispectral images of 10 m, 20 m, and 60 m resolution for visible, near-infrared (NIR) and short-wave infrared (SWIR) in the electromagnetic spectrum. In this paper, we present a sharpening method based on a symmetric skipped connection convolutional neural network, called SSC-CNN, to sharpen 20 m bands using 10 m bands. The main advantage of SSC-CNN architecture is that it brings the features of the input branch to the output, thus improving convergence without using too many deep layers. The proposed method uses the reduced-scale combination of 10 m bands and 20 m bands, and the observed 20 m bands as the training pairs. The experimental results using two Sentinel-2 datasets show that our method outperforms competitive methods in quantitative metrics and visualization.