The continuous development of Earth observation (EO) technology has significantly increased the availability of multi-sensor remote sensing (RS) data. The fusion of hyperspectral image (HSI) and light detection and ranging (LiDAR) data has become a research hotspot. Current mainstream convolutional neural networks (CNNs) excel at extracting local features from images but have limitations in modeling global information, which may affect the performance of classification tasks. In contrast, modern graph convolutional networks (GCNs) excel at capturing global information, particularly demonstrating significant advantages when processing RS images with irregular topological structures. By integrating these two frameworks, features can be fused from multiple perspectives, enabling a more comprehensive capture of multimodal data attributes and improving classification performance. The paper proposes a spatial-spectral-structural feature fusion network (S3F2Net) for HSI and LiDAR data classification. S3F2Net utilizes multiple architectures to extract rich features of multimodal data from different perspectives. On one hand, local spatial and spectral features of multimodal data are extracted using CNN, enhancing interactions among heterogeneous data through shared-weight convolution to achieve detailed representations of land cover. On the other hand, the global topological structure is learned using GCN, which models the spatial relationships between land cover types through graph structure constructed from LiDAR data, thereby enhancing the model’s understanding of scene content. Furthermore, the dynamic node updating strategy within the GCN enhances the model’s ability to identify representative nodes for specific land cover types while facilitating information aggregation among remote nodes, thereby strengthening adaptability to complex topological structures. By employing a multi-level information fusion strategy to integrate data representations from both global and local perspectives, the accuracy and reliability of the results are ensured. Compared with state-of-the-art (SOTA) methods, the framework’s validity is verified on three real multimodal RS datasets. The source code will be available at https://github.com/slylnnu/S3F2Net.

Detail injection model-based methods are the mainstream pan-sharpening techniques for multispectral (MS) images. In recent years, the research on this type of method mainly focuses on optimizing the extraction and injection of panchromatic (PAN) image details, while paying less attention to the adaptive enhancement of MS image details. Due to the differences in spectral responses of different sources, it is difficult to effectively enhance or recover the multispectral details in the fused image. In this article, we analyze the limitations of the existing interpolation enhancement work from the perspective of model derivation, and propose an extended model of pan-sharpening detail injection based on “plug-and-play” (PnP) residual optimization. The model not only focuses on the flexibility in the choice of optimization routes and interpolation enhancement methods, but also emphasizes the universality of interpolation enhancement schemes across detail injection models. Our main contributions include the proposed residual interpolation optimization-based pan-sharpening extension model for detail injection oriented to additive and multiplicative rules, which successfully solves the problem of ineffective optimization in earlier related studies, especially for important multiresolution analysis (MRA) methods such as generalized Laplacian pyramid (GLP), and achieves universally effective optimization. In addition, through large-scale adaptive experiments, we selected 17 PnP methods including optimization model (OM) and deep learning (DL) methods for optimization tests, and comprehensively evaluated the applicability and effectiveness of the model. Comprehensive tests on 12 sets of images from five types of sensors on two public datasets show that our methods can achieve significant improvements in the main evaluation metrics, and the average ERGAS, Q2n, and HQNR metrics can reach 8.9%, 2.3%, and 6.3%, respectively. The source code of the proposed method can be downloaded fromhttps://github.com/JZ-Tao/EXP-PnP/.

Obtaining fine land surface change information from multitemporal hyperspectral images (HSIs) is a key goal pursued in remote sensing image processing. Recently, HSI change detection (HSI-CD) methods based on convolutional neural networks (CNNs) have achieved surprising detection results. One of the reasons is the support of large-scale labeled samples for network learning. However, the existence of mixed pixels greatly increases the difficulty of HSI interpretation, resulting in accurate pixel-level labeling work with a heavy burden and unable to meet the needs of time-sensitive applications. For this reason, achieving stable and high-precision CD with fewer samples is a difficult issue in this field. To address the above problems, a composite consistency-constrained self-ensemble framework (C3SelF) for HSI-CD is proposed, to alleviate the problems of low detection accuracy and instability caused by small samples. The framework mainly comprises two lightweight networks with the same structure aiming at accelerating the model inference process and thus improving the processing timeliness. The composite learning mode implements patch-wise classification loss, class-wise consistency loss on labeled samples, and patch-wise consistency loss on unlabeled samples under a multilevel noise perturbation strategy, which improves the classification results and reduces the labeling cost. Moreover, to exploit the multidimensional features contained in HSIs, a lightweight selective spatial-spectral feature joint network (S3Net) is designed to overcome over-fitting, and to deeply mine the discriminative information in unlabeled samples, a new sample screening strategy is designed to ensure the stability of the network during training unlabeled samples. Extensive experiments prove that the proposed C3SelF outperforms the state-of-the-art (SOTA) methods at a sampling rate of 0.1%, reaching 93.44% Kappa and 97.26% overall accuracy (OA) on the Farmland dataset. The source code of the proposed framework will be released athttps://github.com/zxylnnu/C3SelF.

The acquisition of multisource remote-sensing (RS) data has become more and more convenient due to the boom and innovation of RS imaging technology. The fusion and classification of hyperspectral images (HSIs) and Light Detection and Ranging (LiDAR) data has become a research hotspot because of their excellent complementarity and the vigorous development of deep learning (DL) provides effective methods. Most of the existing methods based on convolution neural networks (CNNs) have fixed convolution kernels, making it difficult to extract multiscale detailed features. In this letter, we propose a multiscale pyramid fusion framework based on spatial–spectral cross-modal attention (S2CA) for HSIs and LiDAR classification, which has strong multiscale information learning ability, especially in areas with complex information changes, thereby improving classification accuracy. Multiscale pyramid convolution is used to extract multiscale features, and an effective feature recalibration (EFR) module is used to enhance features and suppress useless information at each scale. To increase the interactivity of information between modes, we propose an S2CA module, which uses the features of different modes to enhance each other. Three real public datasets are used for the experiment. Compared with the existing advanced methods, the proposed method achieves the best results. The source code of the multiscale S2CA network (MS2CANet) will be public athttps://github.com/junhengzhu/MS2CANet.

Multisource remote sensing (RS) image fusion leverages data from various sensors to enhance the accuracy and comprehensiveness of Earth observation. Notably, the fusion of hyperspectral (HS) images and light detection and ranging (LiDAR) data has garnered significant attention due to their complementary features. However, current methods predominantly rely on simplistic techniques such as weight sharing, feature superposition, or feature products, which often fall short of achieving true feature fusion. These methods primarily focus on feature accumulation rather than integrative fusion. The transformer framework, with its self-attention mechanisms, offers potential for effective multimodal data fusion. However, simple linear transformations used in feature extraction may not adequately capture all relevant information. To address these challenges, we propose a novel multimodal contrastive fusion transformer (MCFT). Our approach employs convolutional neural networks (CNNs) for feature extraction from different modalities and leverages transformer networks for advanced fusion. We have modified the basic transformer architecture and propose a double position embedding mode to make it more suitable for RS image processing tasks. We introduce two novel modules: feature alignment module and feature matching module, designed to exploit both paired and unpaired samples. These modules facilitate more effective cross-modal learning by emphasizing the commonalities within the same features and the differences between features from distinct modalities. Experimental evaluations on several publicly available HS-LiDAR datasets demonstrate that proposed method consistently outperforms existing advanced methods. The source code for our approach is available at:https://github.com/SYFYN0317/MCFT.

In recent years, the significant success of deep learning (DL) in computer vision has contributed to its continuous development in the field of hyperspectral image (HSI) anomaly detection (AD). However, in practical applications, HSI-AD based on DL faces many challenges due to the inability to effectively acquire training samples and predict the types of anomaly targets. This makes it a challenging task, especially for AD in complex scenes. In this article, we propose an unsupervised DL framework for HSI-AD based on the multivariate probability distribution autoencoder (MPDA). First, to explore the distribution characteristics of high-dimensional data, we use the probability density histogram to statistically distribute the frequencies of each interval adaptively, dividing the HSI and obtaining an AD-guided image through the designed grid structure. Second, we propose an unsupervised multilayer autoencoder network based on energy-weighted skip connections. By coupling the detection-guided image in the network, we achieve reverse-guided module reconstruction, weakening the feature representation of anomalous in the reconstructed information and enhancing the separability of targets. Finally, we model the reconstructed error images using the multivariate skewed t-distribution based on data distribution characteristics to obtain the final AD map. Through the comparative experiments with other innovative AD algorithms on authentic HSI datasets captured in five different scenarios, the proposed algorithm demonstrates strong generalization and detection capabilities. The source code of the MPDA will be public athttps://github.com/muzhenhuam/MPDA/tree/master.

The hyper-sharpening technique fuses a high-resolution auxiliary image with a low spatial resolution hyperspectral (HS) image, aiming to enhance the spatial details of the HS image while preserving its spectral integrity. Although it extends from multispectral (MS) sharpening, the unique properties of HS data and the diversity of auxiliary images bring challenges to the application of MS sharpening methods. In this paper, a subspace dynamic combined sparse hyper-sharpening method for diverse auxiliary images is proposed. First, based on the dynamic total variation of MS sharpening, effective data reduction, and spectral coordination are realized by seeking reasonable subspace transformation paths and adopting band-matching processing for auxiliary images. Secondly, by associating the derivation of the relevant a priori with the residual representation, an idea of generalized “subspace + dynamic" regularization term is proposed. On this basis, a combination of regularization terms corresponding to dynamic gradient domain sparsity and dynamic nonlocal transform domain sparsity in subspace is explored. Finally, the alternating direction multiplier method and the improved closed-form solution are used for optimization. The general effectiveness of the proposed method is verified in three types of experiments (HS-PAN, HS-MS, HS-HS) on four public datasets. In the HS-HS class of experiments, where the inter-source correlation is low, the proposed method can overcome the spectral degradation assumption failure problem, with a PSNR improvement of 5.14% to 131.79% and an ERGAS improvement of 21.87% to 99.39% compared to the other nine methods. The code is at https://github.com/JZ-Tao/SDCS.

Hyperspectral image change detection (HSI-CD) based on deep learning (DL) has made significant progress. However, these methods rely significantly on the number of labeled data. Annotating HSI is a highly complex task that requires professional knowledge for guidance, resulting in a scarcity of high-quality labeled samples. The emergence of few-shot learning (FSL), which supports model learning from limited labeled samples, can address this issue. However, FSL-based methods still face some challenges: 1) existing methods mainly rely on single-source or homogenous cross-domain HSI data, which is difficult to adequately cope with the problem of scarcity of HSI labeled data; 2) most existing methods usually only focus on local features within patches and neglect interrelationships between patches, which is also important for model learning; and 3) transformers modeling long-range relationships rely on extensive labeled data, making it difficult to perform well in few-shot scenarios. Therefore we propose a cross-scene FSL framework based on interdomain collaboration (CSIDC-FSL) for HSI-CD. Specifically, the following is proposed: 1) FSL is performed on very high-resolution image (VHRI) and HSI, aiming to use the learnable information in VHRI with low annotation cost to help HSI-CD, reducing the dependence of the model on HSI annotation data while enabling multilevel feature hybrid perceptual CD; 2) a dual-information integrated mapping module (DI2M) is proposed, which designs a CNN and transformer integrated structure that can simultaneously focus on local features and class-wise long-range relationships to break the constraints of local perception of CNN while optimizing the performance of transformer under few-shot situations; and 3) the interdomain joint information allocation module (IDM) is designed to capture cross-scene domain-wise distribution features, and mitigate the impact of distribution differences in cross-scene data (VHRI and HSI) on knowledge learning and migration through the collaboratively consistent interdomain features. Under the condition of five samples per class, the CD results of CSIDC-FSL are better than those of recently advanced algorithms, with average improvements of 1.46%–1.5% for overall accuracy (OA) and average accuracy (AA), respectively. The code will be made available athttps://github.com/lsylnnu/CSIDC-FSL.

Using multitemporal hyperspectral images (HSIs) to obtain fine-grained land cover change information is an essential task in remote sensing (RS) image processing. Convolutional neural networks (CNNs), which have strong feature extraction and nonlinear regression capabilities, have recently aided in the advancement of this subject. However, the performance of these supervised methods is usually limited by small receptive field and less labeled samples. To this end, how to break through the aforementioned bottleneck and build a more suitable change detection (CD) framework for HSI is a crucial and challenging issue. To this end, a graph transformer-guided multitemporal information united framework for HSI-CD (GTransCD) is proposed, which mainly consists of the following three components: 1) applying transformer to the graph structure, a salient relationship strengthening graph transformer (GTrans) module is created, making it possible for the network to capture distant change information, and on this basis, local- and global-range information are aggregated simultaneously; 2) a gated change information fusion (GCF) unit is designed to inject the GTrans-guided change features into the original bitemporal concatenated features to further enhance the representation of change information in the network; and 3) a general HSI-CD framework that can organically blend change features guided by GTrans module with original features is proposed, with the intention of reducing the reliance on training samples by utilizing the semi-supervised learning mode of graph neural networks. numerous experiments demonstrate the proposed GTransCD surpasses the state-of-the-art methods and has a high level even at low sampling rates. The source code of the proposed framework will be released athttps://github.com/zxylnnu/GTransCD.

With the maturity and popularization of high-performance sensor technology, it is now possible to acquire huge amounts of very high-resolution (VHR) remote sensing images. The change detection (CD) for VHR images is currently receiving special attention for remote sensing earth observation applications, however, as a hot research field, it needs to be studied in depth to improve the detection accuracy of fine changes. To this end, a full-scale gated message passing framework (GaMPF) based on collaborative estimation for VHR remote sensing image change detection is proposed in this paper. On one hand, the key embedding representation is generated for each feature map by means of the collaborative estimation (CE) strategy; On the other hand, grounded in timing analysis, bitemporal features are sent selectively on dual paths according to the full-scale gated (FsG) mechanism. Specifically, this framework consists of the following four components: 1) Taking shared-weights Siamese network as an encoder to extract multi-scale features; 2) Generate a set of shared compact bases under the CE strategy and infer the key embedding representations on the basis of the shared bases for feature maps at the same level, considering the representations as the gated switches; 3) FsG mechanism is used as the mode of message passing between bitemporal images, which guides the information can be transmitted simultaneously on both within-and cross-temporal paths. 4) Creating a stepwise dense fusion module (DFM) as a decoder for predicting the change map. Experimental results show that the GaMPF proposed in this paper outperforms existing SOTA methods, and is particularly good at detecting edges and small objects. The source code will be released at https://github.com/zxylnnu/GaMPF.

Deep learning technologies have been successfully applied to hyperspectral (HS) image classification with remarkable performance. However, compared with traditional machine learning methods, neural networks usually need more data. In remote sensing (RS) research, obtaining a large number of labeled HS data is very difficult and expensive work. Simultaneously, the distribution of feature information is bound to be unbalanced, and tends to conform to the long tail. At present, the neighborhood information of unlabeled samples is usually ignored in HS image classification tasks based on semi-supervised learning. In this letter, we propose a new semi-supervised long-tail learning framework based on spatial neighborhood information (SLN-SNI), which can complete the HS image classification task under unbalanced small sample data. Specifically, a new semi-supervised learning strategy is proposed. On this basis, a new method to determine the label of unlabeled samples based on spatial neighborhood information (SNI) is proposed. The coarse classification results divided into three situations are judged again, and the accuracy of pseudo labels is improved. The performance of the proposed method is tested on three public HS image datasets. Compared with the current advanced methods have achieved a certain improvement.

Hyperspectral (HS) image change detection (CD) is an integral component of multitemporal remote-sensing (RS) Earth observation research. However, the existing HS image CD technology still has some problems, such as insufficient effective information extraction and weak correlation between shallow information and deep information, and so on. This letter proposes a new approach called attention-guided feature fusion network for multitemporal HS image CD (AgF2Net). This method is capable of efficiently retrieving and combining spatial–spectral (SS) features extracted from both shallow and deep layers of HS images, thereby enhancing the network’s capability to capture features from multitemporal HS images. The attention-guided enhanced joint feature extraction strategy is used to obtain a better change discriminative feature representation, and the multilevel features extracted from the backbone network are combined between spatial information and spectral information. The integrated channel feature fusion module (ICFFM) not only solves the problem of insufficient feature fusion at different levels, but also strengthens effective semantic information and forms features with more discriminative ability, while also realizing the advantages of multiple features and enhancing the network’s robustness and the accuracy of CD results. According to experimental results obtained from three publicly available datasets for detecting changes in HS images, the findings indicate that the proposed AgF2Net outperforms most advanced state-of-the-art (SOTA) methods. The source code of the AgF2Net will be public onhttps://github.com/NWH/AgF2Net.

Hyperspectral image change detection (HSI-CD) is a technique that detects changes in land cover occurring in a specific area within a closed time. At present, most existing methods for HSI-CD employ exceedingly intricate network architectures, leading to a high model complexity that hampers the achievement of a favorable trade-off between change detection accuracy and timeliness. Furthermore, existing methods often confine the feature extraction process to a single scale rather than multiple diverse scales. However, employing a multiscale approach for feature extraction allows for capturing finer-grained features encompassing more intricate details, as well as coarser-grained features that aggregate local information over a larger range. On the other hand, most existing methods overemphasize the complexity of the feature extraction process and underestimate the importance of the conversion process from bi-temporal features to valuable change features. To this end, a gated transmitting based multiscale siamese network (GTMSiam) is proposed, which mainly contains the following two portions: 1) dual branches with the siamese structure, which capture spatial features of the HSIs at multiple scales while preserving rich spectral information. Moreover, the siamese design effectively reduces the network parameters, thereby alleviating the computational complexity of the model. 2) gated change information transmitting module (GTM), which utilizes gated neural units to transform bi-temporal image features into land cover change information, while progressively transmitting change information at different scales. This enables the network to leverage diverse scale change information for comprehensive discrimination of land object changes. Experimental results on three publicly available datasets demonstrate the superior performance of the proposed GTMSiam. Simultaneously, the complexity analysis experiment proves that the GTMSiam can give consideration to both detection performance and timeliness. The source code of this letter will be released at https://github.com/zkylnnu/GTMSiam.

With the continuous improvement of satellite sensor performance, it is becoming easier to obtain different types of remote sensing (RS) data from multiple sensors, and the fusion of hyperspectral (HS) images and light detection and ranging (LiDAR) for land use/land cover classification has become a research hotspot. However, the current mainstream methods still have defects in feature extraction and feature fusion. In the feature extraction stage, previous methods usually use a single-scale patch as input and a fixed convolution kernel for feature extraction, which makes it difficult to extract features in line with different land cover types at the same time and to obtain high-quality features. Although multi-scale feature extraction can solve the one-sidedness problem of single-scale features, it also brings the challenge of high-dimensional multi-scale features. In the feature fusion stage, the current fusion methods are relatively simple. Therefore, we propose a dynamic scale hierarchical fusion network (DSHFNet) for fusion classification of HS images and LiDAR data. By calculating the similarity in the scale space and judging the information at different scales through the threshold value, the appropriate scale features are dynamically selected, the small-scale features are integrated into the large-scale features, and the dimensionality of the features is reduced. This method solves the unreliability problem of single-scale features and the high dimension problem of multi-scale features. In the feature fusion process, different attention modules are used for hierarchical fusion, spatial attention modules are used for shallow fusion and joint feature extraction, and modal attention modules are used for deep fusion of joint features and features from different sensors to achieve complete complementarity of features. Experimental evaluations on three real RS datasets demonstrate the superiority of the proposed method compared to existing methods. The source code can be downloaded at https://github.com/SYFYN0317/DSHFNet.

Due to the limitation of imaging equipment, it is difficult to acquire hyperspectral images with high spatial resolution directly. Existing approaches improve the resolution of HSIs by fusing multispectral image (MSI) and hyperspectral image (HSI). However, most of them are only feed-forward. They only learn low- to high-resolution feature mappings without considering the ill-posedness of super-resolution tasks, leading to a large solution space of mapping functions and making it difficult to learn a complete mapping function. Moreover, there is a large resolution difference between HSI and MSI, and some up-sampling operations are inevitably employed in the network. Nevertheless, traditional upsampling methods only represent pixel points in a discrete way, failing to adequately restore the continuous spatial and spectral information. To this end, this paper proposes a spatial-spectral implicit neural representation network for hyperspectral and multispectral image fusion (SS-INR). Inspired by the success of implicit neural representation(INR) in continuum reconstruction, we design spatial-INR and spectral-INR for spatial and spectral resolution reconstruction, respectively. SS-INR contains two processes: forward fusion (FF) and back-projection fusion(BPF). In the FF process, the input HSI is first spatially upsampled with Spatial-INR to overcome spatial resolution differences while performing initial fusion with MSI. In the BPF process, we explore the spatial and spectral degradation processes and use them as prior knowledge for error correction. Extensive experiments on five public hyperspectral datasets demonstrate the effectiveness of SS-INR, and SS-INR achieves competitive results compared with existing state-of-the-art fusion methods. The source code for SS-INR will be released at https://github.com/wxy11-27/SS-INR.

In recent years, hyperspectral image change detection (HSI-CD) based on deep learning has achieved high detection accuracy, but these methods obtain excellent detection results usually rely on having sufficient labeled samples to train the network. However, the production of HSI label is difficult, costly and inefficient. In practical tasks, often only a limited number of labeled samples can be obtained due to the limitation of timeliness. To address this problem, a cross heterogeneous domain few-shot learning framework based on bidirectional generation (BiG-FSLF) is proposed for HSI-CD, which aims to solve the few-shot problem of HSI-CD by few-shot learning (FSL), and to assist HSI-FSL perform better by obtaining learnable changed information (i.e., empirical knowledge) from another remote sensing data. Specifically, a multitask generation encoder (MLGenE) is designed to take on both the tasks of FSL and domain adaptation to achieve HSI-CD under the condition of cross heterogeneous domain few-shot. First, we take any pair of image data in a very high resolution image (VHRI) CD dataset as the source domain and HSI is used as the target domain, using sufficient labeled samples in source domain and a small number of labeled samples in target domain for FSL. Meanwhile, a bidirectional generation domain adaptation (BiGDA) method based on generative adversarial strategy is proposed to achieve adaptive alignment of the two heterogeneous domains (source and target domains) feature distributions, to mitigate the impact of the domain shift problem inherent to cross domain data on FSL. Abundant experiments with only five training samples on the publicly available popular HSI-CD datasets confirm that the proposed method can show great detection performance. The source code of the proposed framework will be released at https://github.com/lsylnnu/BiG-FSLF.

Hyperspectral image (HSI) change detection (CD) is a technique to accurately detect land cover changes by using HSIs with rich spatial-spectral information. In recent years, the HSI-CD methods based on convolutional neural networks (CNNs) have achieved great success because of their flexible and effective feature extraction ability. However, these methods often take the HSI patches as the input of the networks, which undoubtedly hinders the overall perception of the HSIs. Meanwhile, the valuable temporal information in HSIs is often underutilized. For this end, a triplet transformer framework (TriTF) based on parents-temporal attention and brother-spatial attention is proposed for HSI-CD. The proposed framework mainly contains the following three parts: 1) Transformer-based network backbone, which uses the self-attention to capture the correlation between arbitrarily two pixels in the same patch and extracts the global spatial correlation in the unit of encoded input patches; 2) parents-temporal attention (PTA) branch. Unlike the previous cross-temporal attention mechanisms of the “T1↔T2” mode which only consider the interaction between bi-temporal HSIs, this paper constructs a novel PTA of the “T1→T3←T2” mode which takes the difference-temporal image T3 as the core. The impact of bi-temporal HSIs on the land cover changes is more concerned in the PTA; 3) brother-spatial attention (BSA) branch. The most similar patch in the current training batch of each patch is defined as its brother patch. Furthermore, cross-spatial attention is applied to propagate the features of the brother patch to the current patch. Thus, the middle- and long-range dependencies can be utilized and the scope of feature propagation can be extended. In this paper, the experiments under low and high sampling rates are conducted and proved the outstanding change detection performance of the proposed TriTF when compared with abundant state-of-the-art (SOTA) CD algorithms. The source code of this paper will be released at https://github.com/zkylnnu/TriTF.

The characteristics of very high resolution (VHR) remote sensing images (RSIs) have higher spatial resolution inherently, and are easier to obtain globally compared with hyperspectral images (HSIs), making it possible to detect small-scale land cover changes in multiple applications. RSI change detection (RSI-CD) based on deep learning has been paid attention to and become a frontier research field in recent years, and is currently facing two challenging problems: The first is high dependence on registration between bi-temporal images caused by high spatial resolution; The other is high pseudo-change information response caused by low spectral resolution. In order to address the above-mentioned two problems, a novel RSI-CD framework called Geospatial-Awareness Network (GeSANet) based on the geospatial Position Matching Mechanism (PMM) with multi-level adjustment and the geo-spatial Content Reasoning Mechanism (CRM) with diverse pseudo-change information filtering is proposed. First of all, the PMM assigns independent two-dimensional offset coordinates to each position in the previous temporal image, afterwards, bilinear interpolation is employed to obtain the subpixel feature value after the offset, and the sparse results based on the difference are transmitted to the next level prediction to realize multi-level geospatial correction. The CRM extracts global features from the corrected sparse feature map in terms of dimensions, implementing effective discriminant feature extraction on basis of the original feature map in a stepwise refinement manner through the cross-dimension exchange mechanism, to filter out various pseudo-change information as well as maintain real change information. Comparison experiments with five recent SOTA methods are carried out on two popular datasets with diverse changes, the results show that the proposed method has good robustness and validity for multi-temporal RSI-CD. In particular, it has a strong comparative advantage in detecting small entity changes and edge details. The source code of the proposed framework can be downloaded from https://github.com/zxylnnu/GeSANet.

Deep learning methods have been extensively applied to hyperspectral (HS) image change detection task and achieved promising performance. However, the beneficial joint spatial-spectral-temporal information provided by the HS images has not been fully used. Since the bi-temporal HS images are highly symmetric, in this letter, we propose a novel Cross-Temporal Interaction Symmetric Attention Network (CSANet), which can effectively extract and integrate the joint spatial-spectral-temporal features of the HS images, at the same time to enhance feature discrimination ability of the changes. Specifically, we propose a novel Cross-Temporal Interaction Symmetric Attention (CSA) module to interact the bi-temporal HS information, where self attentions are combined to enhance the feature representation ability of each temporal image, and the cross-temporal attention is utilized to integrate the difference features oriented from each temporal feature embeding. On this basis, we design a siamese network structure equipped with the CSA to hierarchically extract the change information in a symmetric pattern. Experimental results on three public HS image change detection datasets show that the proposed CSANet change detection framework achieves a significant improvement when comparing with the state-of-the-art methods. The source code of the proposed framework will be released at https://github.com/srxlnnu.

Hyperspectral image (HSI) change detection (CD) aims at obtaining internal components’ change information of land cover and land use. In recent years, the development of convolutional neural networks (CNNs) has greatly promoted the research progress in this field. However, the fixed small-size convolution kernels used by CNNs have severely limited the receptive field of information. Another defect of most CNN-based models is their strong dependence on samples, and they are not competent for tasks with a small number of samples. Besides, the traditional CNN-based models can only perform convolution to learn the spatial–spectral features in the Euclidean space, which is not conducive to capturing the geometric changes in land covers in the HSIs. Differently, the graph attention network (GAT) has come into prominence due to its ability to capture the holistic topology structure of images flexibly, and the attention coefficients can be used to effectively model the long-range correlations between land covers. The semi-supervised nature of GAT is also well-suited to handle HSI-CD tasks with limited samples. Nevertheless, the pixel-level topology structure often generates expensive computational costs. To this end, a dual-branch framework based on temporal–spatial joint graph attention (TSJGAT) with complement strategy (CSDBF) is proposed for HSI-CD, which extracts superpixel- and pixel-level features from bitemporal HSIs in parallel and enables them to complement each other. The proposed CSDBF mainly consists of two branches: superpixel-level feature extraction branch (S-branch) and pixel-level feature extraction branch (P-branch). In the S-branch, we introduce the idea of GAT into HSI-CD for the first time and propose a novel TSJGAT module. Thus, the temporal–spatial features of HSIs are propagated and aggregated on the nonlinear graph structure, which makes the changed regions more discriminable. In the P-branch, pixel-level features are obtained by CNNs to correct uncertain factors caused by superpixel segmentation in the S-branch, which is complementary to the S-branch and lays a foundation for more accurate CD. Abundant experiments show that compared with other pioneer methods, the proposed CSDBF can improve the Kappa coefficient by more than 1.9% and 2.5% on average in general sampling rate situations and a low sampling rate situation, respectively, which shows better robustness and better detection accuracy than most existing state-of-the-art methods. The source code of this article can be downloaded fromhttps://github.com/zkylnnu/CSDBF.

Convolutional Neural Networks (CNNs) have been extensively applied to hyperspectral (HS) image classification tasks and achieved promising performance. However, for CNN based HS image classification methods, it is hard to depict the dependencies among HS image pixels in long-range distanced positions and bands. Moreover, the limited receptive field of the convolutional layers extremely hinders the development of the CNN structure. To tackle these problems, in this paper, the novel Bottleneck Spatial-Spectral Transformer (BS2T) is proposed to depict the long-range global dependencies of HS image pixels, which can be regarded as a feature extraction module for HS image classification networks. More specifically, inspired by Bottleneck Transformer in computer vision, for HS image feature extraction, the proposed BS2T is incorporated with a feature contraction module, a multi-head spatial-spectral self-attention (MHS2A) module and a feature expansion module. In this way, convolutional operations are replaced by the MHS2A to capture the long-range dependency of HS pixels regardless of their spatial position and distance. Meanwhile, in the MHS2A module, to highlight the spectral features of HS images, we introduce the spectral information and content spatial positional information to classical multi-head self-attentions to make the attentions more positional aware and spectral aware. On this basis, a dual-branch HS image classification framework based on 3D CNN and BS2T is defined for jointly extracting the local-global features of HS images. Experimental results on three public HS image classification datasets show that the proposed classification framework achieves a significant improvement when comparing with the state-of-the-art methods. The source code of the proposed framework can be downloaded from https://github.com/srxlnnu/BS2T.

Current remote sensing hardware technology is not yet able to acquire multiband remote sensing images with both high spatial and spectral resolution. As an important tool to compensate for the lack of spatial information acquisition of multispectral (MS) images, pan-sharpening has been an important and continuously active research area in remote sensing image processing. Although many methods have emerged, the problem of how to obtain high spatial resolution while effectively maintaining the spectral information of MS images has not been well solved. Many aspects still need further research. In this article, we first investigate the essential properties and rationality of two common framework types in the multiresolution analysis (MRA) sharpening method of pan-sharpening from the source perspective—the identical-resolution framework (IRF) derived from the generalized fusion application and the different-resolution framework (DRF) exclusive to the sharpening application, and show that the core difference between the two frameworks lies in the different ideas of utilizing the multiscale transformation, i.e., they tend to expand the scale space and model the spatially blurred degradation relationship between the sources, respectively. Both of them have their own advantages and disadvantages in handling detailed information, and neither of them can effectively deal with the “detail exclusivity” problem. Based on this, the idea of “entropy level matching” (ELM) of pan-sharpening is presented, and a comprehensive framework that can combine the advantages of the two types of frameworks is constructed, namely, the multiscale ELM framework. Furthermore, as an application of this framework, we propose a sharpening method shearlet transform-based entropy matching (STEM) built on the nonsubsampled shearlet as a multiscale transformation method. According to the difference in detail injection mode in it, it can be further divided into two sharpening methods based on additive mode and substitutive mode. The comparison experiments with 11 popular methods show that the proposed two sharpening methods can effectively improve the spatial resolution of MS images while keeping the spectral information well, and the comprehensive performance advantage is obvious. The source code of the proposed method can be downloaded fromhttps://github.com/JZ-Tao/STEM/.

The application of hyperspectral image (HSI) is more and more extensive, but the lower spatial resolution seriously affects its application effect. Using low-resolution hyperspectral image (LR-HSI) and high-resolution multispectral image (MSI) fusion technology to achieve super-resolution reconstruction of HSI has become a mainstream method. However, most of the existing fusion methods do not make full use of the large-scale range of remote sensing images, and neglect the preservation of spatial-spectral information in the fusion process. Considering that the spectral information in fused high-resolution hyperspectral image (HR-HSI) mainly depends on HSI, and the spatial information mainly depends on MSI, this paper proposes a full-scale linked Unet with spatial-spectral joint perceptual attention for hyperspectral and multispectral image fusion (FSL-Unet). The FSL-Unet consists of two modules, the first is spatial-spectral attention extraction module (SSAE), which is used to calculate the spectral attention of LR-HSI and the spatial attention of HR-MSI at different scales. The second is the full-scale link U-shaped fusion module (FLUF), which adopts a multi-level feature extraction strategy, using denser full-scale skip connections to explore feature information in a finer-grained range, enabling flexible combination of multi-scale and multi-path features. At the same time, we propose spatial-spectral joint peceptual attention (SSJPA) on the encoder side of FLUF. SSJPA can make full use of the attention maps computed by the SSAE, and then effectively embed spatial and spectral information into the fused image, enabling uninterrupted information transfer and aggregation. To demonstrate the effectiveness of FSL-Unet, we selected five public hyperspectral datasets for experiments. Compared with other eight state-of-the-art fusion methods, the experimental results show that the FSL-Unet achieves competitive results. The source code for FSL-Unet can be downloaded from https://github.com/wxy11-27/FSL-Unet.

Change detection techniques of hyperspectral images(HSI) has witnessed great improvements with the applications of deep convolutional networks (CNN). In this paper, we propose a novel HSI change detection framework based on 3D-Wavelet domain active convolutional neural network. First, the bi-temporal hyperspectral difference image is decomposed into directional subbands by the discrete 3D-Wavelet transform, which is capable of suppressing the noise information of the HSIs. Then, to enhance the change discrimination ability of the primary features, the directional subbands are concatenated from coarse to fine scales to form the initial 3D-Wavelet feature map. In the conventional implementation, active learning strategy is iteratively adopted to extract the deep features of the constructed feature map, in each active learning round, the most informative unlabeled samples will be selected to enlarge the training set, which greatly reduces the labor of annoteing data. Results on two realworld hyperspectral change detection datasets demonstrates the effectiveness of the proposed approach.

Image segmentation is a process of partitioning an image into non‐overlapping regions. Existing unsupervised image segmentation methods include level set, automatic thresholding and region‐based CV mode and so on. However, image segmentation as a key technology in the field of image processing has not been solved indeed, especially for images with complex texture. For this reason, the authors proposed a novel image segmentation algorithm based on NSST and the vector‐valued Chan–Vese (C–V) model. First, they obtained a multi‐scale representation by exploiting the non‐subsampled shearlet transform (NSST) to extract multi‐dimensional data in the image. Afterwards, they gave the vector‐valued C–V model, and applied it to all subbands of NSST, which are treated as a vector‐valued image. By comparing with other class methods, the experimental results show that the proposed method has better visual effects and lower error rates. But at the same time, it is a little time consuming. The proposed method is reasonable and effective, by taking full advantages of each subband's directional information during its diffusion process, compared with traditional C–V model.

The high spectral resolution of hyperspectral (HS) images provides the possibility of omnidirectional feature identification of objects. However, the high-dimensional features and the high redundancy information properties make data processing and the application of HS images extremely challenging. Thus, effectively expressing and correlating the intrinsic correlations of HS images by establishing a statistical model is of great significance. This article proposes a nonsubsampled shearlet transform hidden Markov forest (NSST-HMF) model. This new approach has three key characteristics: 1) the statistical properties of the NSST coefficients are studied in the spatial and spectral directions, respectively, and the “clustering” and “aggregation” properties are observed in both directions; 2) the HMF structure is proposed to depict the multidimensional collaborative correlation of the HS image NSST coefficients, and the proposed method considers the multidirectional transfer relationships among the Markov structure of HS images NSST coefficient for the first time, which significantly improves the prediction ability of the model; and 3) a novel HS-pansharpening approach based on the NSST-HMF model and amplitude modulation of large state probability in the high-frequency subband direction region is proposed. Experimental results show that our method can efficiently improve the spatial resolution of HS images while simultaneously preserving their spectral features. The HMF structure is first proposed in this article, which provides a way to depict the collaborative correlation of multichannel images.

Summary form only given. Shift-variance and computational complexity are bottleneck of existing wavelet-based motion estimation (ME). Moreover, to the best of our knowledge, few works have been reported on wavelet-domain ME of video objects (VOs). In this paper, we present an efficient wavelet-domain approach to ME of arbitrarily shaped VOs.