A high-resolution (HR) hyperspectral imaging at video rates can be achieved by fusing a multi-band low-resolution (LR) mosaiced image and a single-band HR panchromatic (PAN) image in a single shot within a short time. However, the current fusion methods always suffer from a spatial or spectral distortion. To alleviate this, we propose an equivariant Bayesian variational inference framework. Specifically, we decompose an HR hyperspectral image (HSI) into the principal component and sparsity residual, which are modeled as latent variables with Gaussian priors. Each component is estimated via a shared deep neural network (DNN) under a variational inference framework, leveraging the shared spatial structures to enhance parameter efficiency and reconstruction accuracy. Additionally, to tackle the challenge of unavailable ground truth in real-world scenarios, we integrate the equivariant imaging (EI) prior with the Bayesian framework. By enforcing the consistency between the transformed fusion result and the re-inference output, this strategy enables the network to learn beyond the range space. Furthermore, we propose to utilize the learnable degradation functions derived from the physical imaging model to enable the proposed framework, which ensures an enhanced performance by posing plausible constraints on parameters of the degradation functions. Specifically, we explicitly model the point spread function (PSF) and spectral response function (SRF) with learnable parameters and impose non-negativity and sum-to-one constraints. Extensive experiments conducted on both simulated and real-world datasets demonstrate the effectiveness of the proposed framework, paving the way for HSI computational imaging.

Existing mosaic-based snapshot hyperspectral imaging systems struggle to capture high resolution (HR) hyperspectral image (HSI), limiting its application. Fusing a low resolution (LR) mosaiced image with an HR panchromatic (PAN) image serves as a feasible solution to obtain the HR HSI. Therefore, we propose a dual-sensor based HSI imaging system, combining a $4\times 4$ spectral filter array (SFA) mosaiced image sensor with a co-aligned PAN image sensor to provide complementary spatial-spectral information. To reconstruct HR HSI, we propose an unsupervised equivariant imaging (EI)-based training framework with a learnable degradation function, overcoming the inaccessibility of ground truth and spectral response function (SRF). Specifically, we formulate the degradation process as a combination of $8\times 8$ mosaicing and $2\times 2$ average downsampling for the LR mosaiced image, while modeling the PAN image as a linear projection of the HR HSI using SRF. Since parameters of SRF are inaccessible, we propose to make them learnable to have an accurate estimation. By enforcing transformation equivariance between the input-output pair of the fusion network, the proposed framework ensures the reconstructed HSI preserves spatial-spectral consistency without relying on paired supervision. Furthermore, we instantiate the proposed HSI imaging system and collect a real-world dataset of 60 paired mosaiced / PAN images. The mosaiced image exhibits 16 spectral bands ranging from 722 to 896 nm and $1020\times 1104$ spatial pixels while the PAN image exhibits $2040\times 2208$ spatial pixels. Comprehensive experiments demonstrate that the proposed method exhibits high spatial consistency and spectral fidelity while maintaining computational efficiency.

The modulation transfer function tailored image filter (MTF-TIF) has long been regarded as the optimal filter for multispectral image pansharpening. It excels at simulating the camera's frequency response, thereby capturing finer image details and significantly improving pansharpening performance. However, we are skeptical about whether the pre-measured MTF is sufficient to describe the characteristics of actually acquired panchromatic image (PAN) and multispectral image (MSI). For example, any image resampling operations in geometric correction or image registration inevitably change the sharpness of acquired PAN and MSI, and the processed images no longer conform to the camera's MTF. Further, following the Wald protocol, in deep learning (DL) methods using MTF-TIF for downsampling images to construct training data does not satisfy the generalization consistency of training and testing. To prove our point, we propose a pair of symmetric frameworks based on DL in this paper, to find better image filters suitable for both traditional and DL pansharpening methods. We embed two learnable filters into the frameworks to simulate the optimal image filter, namely anisotropic Gaussian image filter and arbitrary image filter. Further, the proposed frameworks can capture subtle offsets between images and maintain the smoothness of the global deformation field. Extensive experiments on various satellite datasets demonstrate that the proposed frameworks can find better image filters than MTF-TIFs, which can achieve better pansharpening performance with stronger generalization ability.

Fusing hyperspectral images (HSIs) with multispectral images (MSIs) of higher spatial resolution has become an effective way to sharpen HSIs. Recently, deep convolutional neural networks (CNNs) have achieved promising fusion performance. However, these methods often suffer from the lack of training data and limited generalization ability. To address the above problems, we present a zero-shot learning (ZSL) method for HSI sharpening. Specifically, we first propose a novel method to quantitatively estimate the spectral and spatial responses of imaging sensors with high accuracy. In the training procedure, we spatially subsample the MSI and HSI based on the estimated spatial response and use the downsampled HSI and MSI to infer the original HSI. In this way, we can not only exploit the inherent information in the HSI and MSI, but the trained CNN can also be well generalized to the test data. In addition, we take the dimension reduction on the HSI, which reduces the model size and storage usage without sacrificing fusion accuracy. Furthermore, we design an imaging model-based loss function for CNN, which further boosts the fusion performance. The experimental results show the significantly high efficiency and accuracy of our approach.

Recently, fusing a low-resolution hyperspectral image (LR-HSI) with a high-resolution multispectral image (HR-MSI) of different satellites has become an effective way to improve the resolution of an HSI. However, due to different imaging satellites, different illumination, and adjacent imaging time, the LR-HSI and HR-MSI may not satisfy the observation models established by existing works, and the LR-HSI and HR-MSI are hard to be registered. To solve the above problems, we establish new observation models for LR-HSIs and HR-MSIs from different satellites, then a deep-learning-based framework is proposed to solve the key steps in multi-satellite HSI fusion, including image registration, blur kernel learning, and image fusion. Specifically, we first construct a convolutional neural network (CNN), called RegNet, to produce pixel-wise offsets between LR-HSI and HR-MSI, which are utilized to register the LR-HSI. Next, according to the new observation models, a tiny network, called BKLNet, is built to learn the spectral and spatial blur kernels, where the BKLNet and RegNet can be trained jointly. In the fusion part, we further train a FusNet by downsampling the registered data with the learned spatial blur kernel. Extensive experiments demonstrate the superiority of the proposed framework in HSI registration and fusion accuracy.

Image registration and the estimation of spatial and spectral blur kernels are essential steps before fusing panchromatic image (PAN) and multispectral image (MSI). Usually, these basic steps are performed separately, which will lead to error accumulation and ultimately affect the fusion performance. In this paper, we propose a novel deep learning (DL) based framework which can jointly register images and learn the blur kernels of original PAN and MSI. Specifically, we first construct a convolutional neural network (CNN) to learn the offsets between PAN and MSI, and the offsets are utilized to align the two images. Then, we analyze the relationship between the registered PAN and MSI, and design a tiny network for blur kernel learning. After solving the gradient derivation problems, we can combine the two networks and train them end-to-end. Experimental results on GF-2 satellite images demonstrate that the proposed method can significantly improve the fusion performance of some popular pansharpening methods.

Deep learning (DL) for pansharpening has recently attracted considerable attentions. To construct training data, DL based pansharpening approaches often downsample the original multispectral image (MSI) and panchromatic image (PAN) with fixed blur kernel, which can be different from the real point spread functions (PSF) of the satellites. And a mismatched blur kernel will cause the pansharpening performance to drop dramatically. In this paper, we propose a novel blur kernel learning method for pansharpening, which can learn the spatial and spectral blur kernels between PAN and MSI in an unsupervised way. Specifically, we analyze the relationship between PAN and MSI, and then construct a mini net for blur kernel learning. Once the spatial blur kernel is found, a convolutional neural network (CNN) for pansharpening is trained on the downsampled dataset using the learned spatial blur kernel. Experimental results on GF-2 images demonstrate the superiority of the proposed method.

Hyperspectral image (HSI) sharpening, which aims at fusing an observable low spatial resolution (LR) HSI (LR-HSI) with a high spatial resolution (HR) multispectral image (HR-MSI) of the same scene to acquire an HR-HSI, has recently attracted much attention. Most of the recent HSI sharpening approaches are based on image priors modeling, which are usually sensitive to the parameters selection and time-consuming. This paper presents a deep HSI sharpening method (named DHSIS) for the fusion of an LR-HSI with an HR-MSI, which directly learns the image priors via deep convolutional neural network-based residual learning. The DHSIS method incorporates the learned deep priors into the LR-HSI and HR-MSI fusion framework. Specifically, we first initialize the HR-HSI from the fusion framework via solving a Sylvester equation. Then, we map the initialized HR-HSI to the reference HR-HSI via deep residual learning to learn the image priors. Finally, the learned image priors are returned to the fusion framework to reconstruct the final HR-HSI. Experimental results demonstrate the superiority of the DHSIS approach over existing state-of-the-art HSI sharpening approaches in terms of reconstruction accuracy and running time.