Hyperspectral images (HSIs) and multispectral images (MSIs) possess complementary advantages, with HSI providing rich spectral information and MSI offering fine spatial details. Therefore, fusing HSI and MSI to obtain high-resolution hyperspectral images (HR-HSIs) with both high spatial detail and rich spectral information has drawn increased attention. However, convolutional neural network (CNN)-based methods are limited by their local receptive fields, while traditional Transformer architectures suffer from extremely high computational complexity when applied to HSI with numerous spectral bands. To address these issues, we propose a novel multiscale Spatial-Spectral Dilated Transformer (SSDT) network based on the Transformer framework. The proposed method adopts a dual-branch architecture, consisting of the Spatial Dilated MultiScale Transformer (Spa-DMST) and the Spectral Dilated Positional Embedded Transformer (Spe-DPET) modules. Specifically, Spa-MSDT introduces a multi-level dilated window mechanism to enable cross-scale spatial feature extraction, while Spe-PEDT employs position encoding-enhanced dilated attention to reduce computational complexity. To further optimize the feature fusion process, we design a Gating Fusion Reconstruction (GFR) module that integrates depthwise separable convolutions with a gating mechanism, effectively suppressing redundant information and enhancing detail reconstruction capabilities. Extensive experiments and visualized results on three simulated datasets and one real dataset demonstrate that the proposed method outperforms other state-of-the-art fusion methods. The code of this paper will be available at https://github.com/FxyPd/SSDT.

Hyperspectral anomaly detection (HAD) aims to identify anomalous objects from hyperspectral images (HSIs) whose spectral features significantly deviate from their surroundings. Existing HAD methods still reconstruct some anomalies during the background reconstruction process, which can seriously affect the detection accuracy. Consequently, inspired by Mamba’s ability to effectively model long sequences, we propose a multiscale Mamba reconstruction network for HAD (MMR-HAD) by enhancing the representation of the background and inhibit anomalies from being reconstructed. MMR-HAD first removes most of the anomalous pixels in HSIs using the random mask (RM) strategy, which reduces the interference of anomalous pixels on the background reconstruction and makes the background features more prominent. To further filter out the small amount of residual anomalous pixels, we propose the multiscale dilated attention background enhancement (MDABE) mechanism, which enhances the background representation. Finally, we apply the multiscale dynamic feature fusion (MDFF) strategy to reconstruct the background image, further extracting and strengthening the background information, thus obtaining a pure background image. MMR-HAD, which focuses on generating a pure background, has been experimentally validated on seven real hyperspectral datasets. The results demonstrate that it excels in background enhancement and anomaly suppression, significantly improving detection accuracy. Introducing Mamba offers a promising solution for HAD, with substantial potential for practical application.

Multispectral and hyperspectral image fusion (MHIF) seeks to combine high-resolution multispectral images (HR-MSIs) with low-resolution hyperspectral images (LR-HSIs) to create high-resolution hyperspectral images (HR-HSIs). Transformer-based architectures have recently become prominent in MHIF tasks due to their effective global self-attention mechanisms. However, the quadratic computational complexity of the global self-attention in Transformers presents significant challenges for practical applications. In this paper, we propose an enhanced spatial-frequency synergistic (ESFS) approach that leverages both spatial and frequency domain features to enhance fusion quality. Our ESFS framework introduces the condensed spatial augmentation module (CSAM), which condenses window features and employs cross-attention to balance extensive contextual understanding and detailed local feature extraction while reducing computational overhead. Additionally, we develop the selective frequency decomposition module (SFDM), which utilizes global filters composed of phase and amplitude information in the frequency domain to retain features, effectively capturing deep frequency domain characteristics and their interdependencies. Comprehensive experiments on three benchmark MHIF datasets demonstrate that our method achieves superior performance, establishing a new state-of-the-art (SOTA) in both quantitative metrics and visual quality assessments. The code is available at http://szu-hsilab.com/.

The joint classification of hyperspectral image (HSI) and light detection and ranging (LiDAR) data is gaining attention for its improved classification accuracy. However, effectively integrating the rich spectral information of HSI and the elevation features of LiDAR has remained a challenge in multimodal fusion. This article proposes a novel approach called progressive semantic enhancement network (PSENet) for hyperspectral and LiDAR classification based on a progressive joint spatial-spectral attention mechanism. PSENet mainly comprises two modules: the spatial grouping constraint (SAGC) module and the spectral weighting constraint (SEWC) module. The SAGC module extracts multiscale features in the spatial domain, while the SEWC module focuses on enhancing semantic features in spectral dimension. By gradually utilizing spatial and spectral constraint modules to progressively enhance feature extraction, PSENet integrates affluent information for a more refined classification of ground objects. Based on experimental results, it has been demonstrated that PSENet outperforms several most advanced methods on three datasets. The SAGC and SEWC modules proposed in PSENet enable the effective integration of the spatial, spectral, and elevation information from HSI and LiDAR, providing a promising way to perform classification more accurately. The source codes of this work will be publicly available at http://szu-hsilab.com/.

With its strong capability in modeling long-range dependencies, the Transformer achieves competitive performance in hyperspectral image (HSI) and multispectral image (MSI) fusion. However, existing Transformer-based methods face the trade-off between receptive field size and computational efficiency when dealing with spatially non-local features. Furthermore, the Transformer captures deep spectral relationships by modeling pairwise channel interactions. This global interaction may overlook features that contribute little to the overall context but are critical locally, thus affecting the accurate understanding of HSI content. To overcome these challenges, we propose a novel local-global collaborative network with Transformers (LGCT) specifically designed to achieve high-quality HSI reconstruction. The proposed LGCT includes two inverse feature streams to establish multiscale deep representations of the HSI and MSI features. The feature streams comprise collaborative Transformer blocks (CTBs) explicitly designed for the spectral and spatial domains. By combining global and local processing mechanisms, the proposed CTBs can efficiently emphasize potential crucial features that Transformer ignores when capturing deep spectral and spatial relationships, thus enabling efficient modeling of the spectral and spatial domains from details to the whole. Furthermore, to enhance the reusability of multiscale enhanced features from the spectral and spatial domains, a hierarchical and symmetric strategy is adopted to progressively fuse them to generate high-quality images. The results on both simulated and real datasets demonstrate the superior performance of the proposed method in terms of quantitative metrics and visual quality. The code will be released athttps://github.com/Hewq77/LGCT.

Hyperspectral cameras capture electromagnetic information within hundreds of narrow spectral bands, producing hyperspectral images (HSIs) with the capability to accurately characterize the attribute information of objects. However, mixed noise induced by instrument and atmospheric effects hinders the interpretations and applications of the HSIs. In this paper, we propose a novel subspace representation based mixed noise removal method for hyperspectral images via Robust Subspace Estimation and weighted Group Sparsity constraint (RoSEGS). An outlier detection method is proposed to effectively detect sparse noise and replace the sparse noise with new estimates. A subspace estimation strategy, which is robust to mixed noise, is proposed. The subspace is first estimated after sparse noise detection and then optimized iteratively. In addition to the introduction of a state-of-the-art denoiser based on the plug-and-play technique to exploit self-similarity characteristics of the eigen-images, we impose a weighted group sparse regularization on the eigen-images to better promote the group sparsity of the spatial differences between the eigen-images, which further improves the denoising performance. We performed extensive experiments on two simulated and two real HSIs to fully demonstrate the effectiveness of the proposed method in comparison with seven state-of-the-art competitors.

Hyperspectral images (HSIs) possess the capability to accurately characterize the attribute information of objects. However, they are usually obtained at a high spectral resolution with a compromise of its spatial resolution. In addition, they are easily contaminated by mixed noise induced by instrument and atmospheric effects. These disadvantages, to a certain degree, hinder the interpretations and applications of the HSIs. To overcome these limitations, in this paper, we propose a novel Mixed noise-oriented hyperspectral and multispectral image Fusion method, termed (MixFus). First, a sparse noise detection method is proposed by first leveraging a subset of specifically chosen hyperspectral bands to estimate noise in HSI and then employing Gaussian mixture models to detect sparse noise from the estimated noise. Then, a robust subspace estimation method is introduced by replacing the detected sparse noise with new estimates using median values within a sliding window for a better estimation of the subspace, which offers improved accuracy and robustness of subspace estimation. Finally, in addition to the introduction of a state-of-the-art image prior based on the plug-and-play technique to exploit self-similarity characteristics in the eigen-images, we also impose a weighted group sparse regularization on the eigen-images to better promote the group sparsity of the spatial differences between the eigen-images, which further improve the denoising performance. We evaluate the proposed method by performing extensive experiments on three reduced-resolution HSIs and a full-resolution HSI in comparison with seven state-of-the-art competitors. Experimental results demonstrate the superiority of the proposed method over the competitors in the fusion of hyperspectral and multispectral images against mixed noise.

The necessary prerequisite for effective data fusion is the strict registration of low-resolution hyperspectral images (LR-HSI) and high-resolution multispectral images (HR-MSI). However, registration requires a complex process that takes into account the effects of light, imaging angle, and geometric distortion of the image during acquisition. Therefore, to avoid complex registration, we focused on developing an unregistered HSI and MSI fusion method for pixel shifting, obtaining fused images with high resolution, high signal-to-noise ratio, and feature identifiability. We identified that the unregistered LR-HSI and HR-MSI in the case of pixel shift are very similar to the disparity maps in stereo vision. Inspired by this, we simulate the structure of stereo cameras to propose a stereo cross-attention network (SCANet) to achieve an accurate fusion of unregistered LR-HSI and HR-MSI. Considering the model complexity and computing efficiency, we design a simple and stackable stereo cross-fusion block (SCFBlock) based on a Transformer to simulate the process of light entering the left and right cameras by extracting the abstract features of the images. Moreover, the purpose of cross-convergence fusion self-attention (CCFSA) is to learn cross-complementary attention and collect contextual information in horizontal and vertical directions to fuse unregistered images using multi-directional cross-view information. We have conducted extensive experiments on Pavia University (PaviaU), Chikusei, and PYLake datasets. The results show that SCANet achieves superior or competitive performance in fusing unregistered LR-HSI and HR-MSI in comparison with the other competitors.

Fusing a Hyperspectral image (HSI) and a multispectral image (MSI) from different sensors is an economic and effective approach to get an image with both high spatial and spectral resolution, but localized changes between the multiplatform images can have negative impacts on the fusion. In this letter, we propose a novel sparsity constrained fusion method (SCFus) to fuse multiplatform HSIs and MSIs based on matrix factorization. Specifically, we imposed$\ell _{1}$norm on the residual term of the MSI to account for the localized changes between the hyperspectral and MSIs. Furthermore, we plugged a state-of-the-art denoiser, namely block-matching and 3-D filtering (BM3D), as the prior of the subspace coefficients by exploiting the plug-and-play framework. We refer to the proposed method as SCFus for hyperspectral and MSIs. Experimental results suggest that the proposed fusion method is more effective in fusing hyperspectral and MSIs than the competitors.

The high spectral resolution of hyperspectral images (HSIs) generally comes at the expense of low spatial resolution, which hinders the application of HSIs. Fusing an HSI and a multispectral image (MSI) from different sensors to get an image with the high spatial and spectral resolution is an economic and effective approach, but localized spatial and spectral changes between images acquired at different time instants can have negative impacts on the fusion results, which has rarely been considered in many fusion methods. In this article, we propose a novel group sparsity constrained fusion (GSFus) method to fuse hyperspectral and MSIs based on matrix factorization. Specifically, we imposed$\ell _{2,1}$norm on the residual term of the MSI to account for the localized interimage changes occurring during the acquisition of the hyperspectral and MSIs. Furthermore, by exploiting the plug-and-play framework, we plugged a state-of-the-art denoiser, namely block-matching and 3-D filtering (BM3D), as the prior of the subspace coefficients. We refer to the proposed fusion method as GSFus method. We performed fusion experiments on two kinds of datasets, i.e., with and without obvious localized changes between the HSIs and MSIs, and a full resolution dataset. Extensive experiments in comparison with seven state-of-the-art fusion methods suggest that the proposed fusion method is more effective on fusing hyperspectral and MSIs than the competitors.

Hyperspectral imaging measures the amount of electromagnetic energy across the instantaneous field of view at a very high resolution in hundreds or thousands of spectral channels. This enables objects to be detected and the identification of materials that have subtle differences between them. However, the increase in spectral resolution often means that there is a decrease in the number of photons received in each channel, which means that the noise linked to the image formation process is greater. This degradation limits the quality of the extracted information and its potential applications. Thus, denoising is a fundamental problem in hyperspectral image (HSI) processing. As images of natural scenes with highly correlated spectral channels, HSIs are characterized by a high level of self-similarity and can be well approximated by low-rank representations. These characteristics underlie the state-of-the-art methods used in HSI denoising. However, where there are rarely occurring pixel types, the denoising performance of these methods is not optimal, and the subsequent detection of these pixels may be compromised. To address these hurdles, in this article, we introduce RhyDe (Robust hyperspectral Denoising), a powerful HSI denoiser, which implements explicit low-rank representation, promotes self-similarity, and, by using a form of collaborative sparsity, preserves rare pixels. The denoising and detection effectiveness of the proposed robust HSI denoiser is illustrated using semireal and real data.

This article proposes a smart hyperspectral sensing strategy, implemented in the spectral domain, conceived for spaceborne sensor systems, where physical space, storage resources, and communication bandwidth are extremely scarce and expensive. Smart sensing means faster and hardware-friendly imaging. Instead of acquiring all band samples in the spectral domain, we randomly select a few band samples per spatial pixel location. A periodic structure of spectral band selector array (SBSA) is designed so that we can learn a subspace basis from subsamples, which is essential to the underlying hyperspectral image (HSI) recovery algorithm. This spectral subsampling sensing strategy yields a demosaicing problem. We propose a blind hyperspectral reconstruction technique termed hyperspectral demosaicing (Hy-demosaicing) exploiting spectral low-rankness and spatial correlation of HSIs. It is blind in the sense that the signal subspace is learned from measured spectral subsamples. The subspace basis is data-adaptive and provides a more compact representation than other non-adaptive representations. This adaptiveness leads to improved image recovery as illustrated in experiments with real data.

Due to the importance in many military and civilian applications, hyperspectral anomaly detection has attracted remarkable interest. Low-rank representation (LRR)-based anomaly detectors use the low-rank property to represent background pixels, and pixels that cannot be well represented are detected as anomalies. The ability of an LRR-based detector to separate background pixels and anomalous pixels depends on the dictionary representation ability, which usually can be enhanced by designing a proper prior for dictionary representation coefficients and constructing a better dictionary. However, it is not easy to handcraft effective and meaningful regularizers for dictionary coefficients. In this article, we propose a novel anomaly detection algorithm that uses a plug-and-play prior for representation coefficients and constructs a new dictionary based on clustering. Instead of cumbersomely handcrafting a regularizer for representation coefficients, we propose solving the anomaly detection problem using the plug-and-play framework, which enables us to plug state-of-the-art priors for representation coefficients. An effective convolutional neural network (CNN) denoiser is plugged into our framework to fully exploit the spatial correlation of representation coefficients. We also propose a modified background dictionary construction method, which carefully includes background pixels and excludes anomalous pixels from clustering results. We refer to the proposed anomaly detection method as plug-and-play denoising CNN regularized anomaly detection (DeCNN-AD) method. Extensive experiments were performed on five data sets in a comparison with eight state-of-the-art anomaly detection methods. The experimental results suggest that the proposed method is effective in anomaly detection and can produce better anomaly detection results than that of the comparison methods. The codes of this work will be available athttps://github.com/FxyPdfor the sake of reproducibility.

The ever-increasing spectral resolution of hyperspectral images (HSIs) is often obtained at the cost of a decrease in the signal-to-noise ratio of the measurements, thus calling for effective denoising techniques. HSIs from the real world lie in low-dimensional subspaces and are self-similar. The low dimensionality stems from the high correlation existing among the reflectance vectors, and self-similarity is common in real-world images. In this article, we exploit the above two properties. The low dimensionality is a global property that enables the denoising to be formulated just with respect to the subspace representation coefficients, thus greatly improving the denoising performance and reducing the computational complexity during processing. The self-similarity is exploited via a low-rank tensor factorization of nonlocal similar 3-D patches. The proposed factorization hinges on the optimal shrinkage/thresholding of the singular value decomposition (SVD) singular values of low-rank tensor unfoldings. As a result, the proposed method is user friendly and insensitive to its parameters. Its effectiveness is illustrated in a comparison with state-of-the-art competitors. A MATLAB demo of this work is available athttps://github.com/LinaZhuangfor the sake of reproducibility.