Deep learning-based methods have achieved significant success in remote sensing Earth observation data analysis. Numerous feature fusion techniques address multimodal remote sensing image classification by integrating global and local features. However, these techniques often struggle to extract structural and detail features from heterogeneous and redundant multimodal images, particularly in label-scarce scenarios. With the goal of introducing frequency domain learning to model key and sparse detail features, this paper introduces the spatial–spectral-frequency interaction network (S 2 Fin), which integrates pairwise fusion modules across the spatial, spectral, and frequency domains. Specifically, we propose a high-frequency sparse enhancement transformer to refine spectral signatures by adaptively enhancing discriminative high-frequency components. For spatial-frequency interaction, we present a depth-wise strategy: the adaptive frequency channel module fuses low-frequency structural information with enhanced details in shallow layers, while the high-frequency resonance mask amplifies modality-consistent regions in deep layers using phase similarity. In addition, a spatial–spectral attention fusion module bridges the gap between spectral and spatial branches at intermediate depths. Extensive experiments on four benchmark datasets demonstrate that S 2 Fin exhibits good robustness and generalization, and its performance significantly outperforms state-of-the-art methods in few-sample settings. The code is available at https://github.com/HaoLiu-XDU/SSFin .

Multispectral remote sensing image (MSI) semantic segmentation faces challenges of limited labeled data and significant scene variability. Although domain adaptation (DA) and domain generalization (DG) methods alleviate these issues to some extent, they still have limitations. DA requires target domain (TD) data, and DG has limited task adaptability. The recently emerged segment anything model (SAM) demonstrates exceptional zero-shot generalization capabilities, yet its visible-light training data and interactive prompt requirements prevent direct application to MSI segmentation tasks. To address these challenges, this article proposes a single-source domain defect-aware adaptation and style-modulated generalization network (SDSnet), which integrates two key innovations: defect-aware prompt learning that automatically focuses on high-difficulty regions through entropy-based defect detection, and style generalization learning that enhances cross-domain adaptability via codebook-based style modulation. Through knowledge distillation, SDSnet enables efficient inference using only the base network, without additional computational overhead. Extensive experiments on three TDs demonstrate SDSnet's superiority over state-of-the-art DA, DG, and SAM-based methods. Code will be available at https://github.com/zhaoboyu34526/SDSnet.

In recent years, unmanned aerial vehicle (UAV) technology has shown great potential for application in hyperspectral image (HSI) classification tasks due to its advantages of flexible scheduling and fast response. However, existing deep learning-based classification algorithms have not thoroughly studied the problems caused by the higher spatial resolution of UAV HSIs. Such as more severe intra-class variation and higher computational resource consumption. These issues limit classification performance. To address these challenges, this paper proposes a lightweight spatial-spectral deformable convolutional neural network (LS2DCNet) for UAV HSI classification. This network reduces computational resource usage and data processing time, meeting the fast-response application requirements while ensuring classification performance. First, spatial-spectral deformable convolution (S2DConv) is designed to construct a lightweight feature extraction network. This not only enhances the adaptive extraction of fine features but also reduces computational resource consumption, and improves response speed. Second, a dynamic labeling-based joint loss (DL-JLoss) is designed to dynamically learn the distribution relationship between data classes. This improves the network’s generalization performance. A more realistic experimental validation was conducted on three UAV HSI datasets using regionally divided training samples, and inference comparisons were performed on embedded devices. The results show that the proposed LS2DCNet exhibits a better overall performance in terms of classification accuracy and inference speed.The relevant code can be found at https://github.com/niuroushu/A-Lightweight-Spatial-Spectral-Deformable-CNN-for-UAV-Hyperspectral-Image-Classification.

Deep learning has garnered extensive attention in hyperspectral image (HSI) processing. However, its application in HSI semantic segmentation tasks has been relatively limited. Although segmentation methods can often interpret images up to two orders of magnitude faster than classification methods when interpreting images of the same scene, the segmentation task requires the training data to be fully labeled, i.e., each pixel has a corresponding label. Such data are scarce in HSI data. To address this problem, this article proposes a lightweight segmentation network based on a hierarchical attention segmentation network (LHAS), in which a generalized data augmentation (GDA) method is utilized to acquire relatively sufficient data for semantic segmentation. Specifically, the hierarchical attention module is designed to extract global and local information on HSI patches from different layers. A prototype auxiliary module (PAM) of cluster contrast has also been developed to enhance feature discrimination. Across two different datasets in various scenarios, the proposed LHAS demonstrates superior segmentation performance compared to existing methods, affirming its effectiveness. In addition, experiments conducted on embedded devices validate the efficacy of LHAS.

The availability of multisource remote sensing data provides the possibility for comprehensive observation. Convolutional neural networks (CNNs) naturally integrate multisource feature extractors and classifiers into an end-to-end multilayer design. However, CNN assumes data are independent and identically distributed. In practice, it is not always possible to access the labels or even data of the testing scenes. Therefore, the CNN-based methods have exposed its limitation on generalization ability. To solve the issue, a feature-distribution-independent network (FDINet) is designed for multisource remote sensing cross-domain classification without feature alignment and decoupling operations. On one hand, an elegantly designed baseline is used for extracting multisource cross-domain features. The baseline extracts the common line and texture features through shallow weight-sharing networks. More importantly, the modality prediction probability is used to measure the similarity between the source domains and the target domains, thereby improving cross-domain collaboration capabilities. On the other hand, the sharpness-aware feature discriminating (SAFD) strategy is developed for model optimization. Specifically, the generalization ability is improved by minimizing the sharpness of local optima. To avoid the decrease in feature discrimination caused by the gradient conflict between sharpness and overall loss, the discrimination constraints are designed to balance feature discrimination and generalization ability. Comprehensive experiments are conducted on two datasets, which demonstrate that the proposed FDINet outperforms other competitors in terms of quantitative and qualitative analyses.

Segment anything model (SAM), as a prompt-based image segmentation foundation model, demonstrates strong task versatility and domain generalization (DG) capabilities, providing a new direction for solving cross-scene segmentation tasks. However, SAM still has limitations in multispectral cross-domain segmentation tasks, mainly reflected in: 1) insufficient information utilization, which is reflected in the neglect of nonvisible spectral information and the shift information contained in source domain (SD) samples and target domain (TD) samples; and 2) lack of cross-domain strategies, which leads to insufficient cross-domain adaptation (DA) ability in downstream tasks. To address these challenges, we combine the respective advantages of masked autoencoder (MAE) and cross-domain strategies, propose an improved SAM DA network structure called domain information mining and state-guided adaptation network (DSAnet), aiming to enhance SAM's performance in multispectral cross-domain segmentation tasks from both data and task levels. At the data level, DSAnet incorporates a style masking learning component, which randomly masks image features and replaces them with domain-specific learnable tokens, integrated with the image reconstruction task, to mine the style information and domain invariance of the image itself. At the task level, DSAnet introduces domain state learning and style-guided segmentation: domain state learning, through a state sequence modeling approach, designs specific state representations for SD and TD to capture interdomain differences, thereby reducing task shift. Meanwhile, the learned domain state information can be directly applied to the inference stage. Style prompt segmentation guides the segmentation training process of SD images with TD style prompts, improving SAM's adaptability in cross-domain multispectral segmentation downstream tasks. Extensive experiments on three multitemporal multispectral image (MSI) datasets demonstrate the superiority of the proposed method compared to state-of-the-art cross-domain strategies and SAM variant methods.

Deep learning has received much attention in hyperspectral image (HSI) classification. However, most deep learning methods design relatively complex feature extraction and processing network modules for the characteristics of HSIs, which may not be necessary for relatively simple patch-based HSI classification tasks. The complex network structure and high feature channel dimension lead to large computational complexities, which limit the practical applicability of HSI. In this article, an elegant lightweight HSI classification-based information redistribution network (LIRnet) is proposed to separate and reaggregate the feature information to achieve feature information homogenization and extract discriminative feature information, respectively. The classification performance of LIRnet is better than that of existing methods on three different datasets in different scenarios, which proves its effectiveness. In addition, experiments on embedded devices verify the computational efficacy of LIRnet.

The semantic segmentation of wide-field remote sensing images plays a significant role in many fields. However, due to the complexity of the content of remote sensing images, the dataset often has an uneven distribution of land type between different classes and large gaps in the scales of different objects. This often creates great problems for fine segmentation. To solve the issues, an unbalanced class learning network with Scale-adaptive perception (UCSANet) is proposed, which can adaptively cope with Multi-scale objects and unbalanced classes. The design can be inserted in any convolution network easily and can enrich features without increasing too many parameters. The network groups feature and uses atrous convolutions with different dilated rates on different groups to extract Multi-scale features while separable convolutions reduce the amount of network parameters. Then, the fusion of features between different scales is achieved through the self-attention mechanism. Furthermore, a weight map is designed to adaptively combine the predictions of two segmentation heads with Cross-Entropy loss and Lovasz-Softmax loss respectively, which enable the network to focus on learning low-frequency classes without affecting high-frequency classes. Experimental results on GF-6 MSI datasets demonstrate that the proposed UCSANet performs significantly better than others and achieves multi-class segmentation more accurately.

With the development of remote sensing technology, more and more data sources are available for landcover classification tasks, such as hyperspectral images (HSIs), light detection and ranging (LiDAR) data, and synthetic aperture radar (SAR) data. Due to the unique information carried by different sources, the collaborative use of multiple remote sensing data has become a key research direction in landcover classification tasks. Most existing methods are only capable of dealing with two types of remote-sensing images, limiting the potential for the collaboration of more sources. Also, the traditional feature fusion method uses addition or concatenation manner to integrate information, which makes it difficult to make full use of complement characteristics between modalities. As a remedy, we propose a novel multimodal adaptive modulation network (MAMNet), for landcover classification tasks using multimodal remote sensing data. First, the cross-modal interacting module (CIM) is utilized for information absorption between modalities. The feature representation is enhanced, and the modality-specific information is preserved. Second, the modal attention layer (MAL) is designed for multimodal feature fusion. Softmax attention is utilized to eliminate redundant information among the three modal features. Finally, an adaptive multimodal margin loss (AMM loss) is proposed to balance the consistency and diversity of multimodal features. It encourages adjustable decision margins between sources, which enables the model to better utilize complementary information between modalities and partially avoids model-overfitting by defining a more difficult learning target. Experimental results on two benchmark remote sensing datasets show the effectiveness of the proposed method compared with several state-of-the-art approaches.

As an invasive plant in wetlands, Spartina alterniflora (S. alterniflora) causes immeasurable damage to wetland ecosystems. Observing S.alterniflora using multitemporal remote sensing data helps us better understand its further development and facilitates effective containment of its invasion trend. However, inconsistent representation across remote sensing data from different time periods poses a challenge. Fortunately, the utilization of unsupervised domain adaptation (UDA) techniques helps in addressing such issues and enables the exploration of rich temporal dimension information in multitemporal remote sensing data, revealing the spatio-temporal distribution characteristics of S.alterniflora. However, existing UDA methods mostly focus on directly aligning the global or intraclass distribution representations across domains, which overlooks the issue of significant differences between extreme domains and lacks exploration of interclass relationships. To address these limitations, an intermediate domain prototype class-level learning network (IDPNet) is proposed. IDPNet utilizes dynamically generated intermediate domain (ID) features to construct class prototypes while incorporating interclass information into the prototype construction, achieving the class-centered distribution alignment for adaptation. Moreover, intermediate domain feature generation module (IFM) is employed in IDPNet to blend the latent representations from various domains and generate ID features in real time. Additionally, the hierarchical feature fusion module (HFM) is designed to enable IDPNet to learn more discriminative and robust spatio-temporal distribution features, thereby reducing the loss of information from patches. Experimental results on two cross-year multispectral datasets demonstrate that the proposed IDPNet outperforms several state-of-the-art UDA methods.

Advances in multisource remote sensing have allowed for the development of more comprehensive observation. The adoption of deep convolutional neural networks (CNN) naturally includes spatial-spectral information, which has achieved promising performance in multisource data classification. However, challenges are still found with the extraction of spatial distribution and spectrum relationships, which eventually limit the classification performance. To solve the issue, a spatial-spectral perception network (S2PNet) is proposed to extract the advantages of different data sources and the cross information between data sources in a targeted manner. Specifically, the spatial perception network is developed to build the spatial distribution relationship from high-resolution images, while the spectral perception network extracts the spectrum relationship from spectral images. For perceiving cross information, a memory unit is utilized to store the features from different data sources in succession. In addition, the distance loss and reconstruction loss are introduced to keep the feature integrity, and the cross-entropy loss ensures that features can distinguish different classes. The comprehensive experiments are conducted on several datasets to validate the superiority of the proposed algorithm. The proposed S2PNet outperforms the considered classifiers with an average improvement of +0.77%, +5.62%, +1.58%, and +1.79% for overall accuracy values.

Convolutional neural networks (CNN) have attracted increasing attention in the field of multimodal cooperation. Recently, the adoption of CNN-based methods has achieved remarkable performance in multisource remote sensing data classification. However, it is still confronted with challenges in the aspect of complementarity extraction. In this paper, the adversarial complementary learning strategy is embedded into the CNN model called ACL-CNN, which is employed to extract the complementary information of the multisource data. The proposed ACL-CNN is able to filter out the common patterns and specific patterns from multisource data by conducting the adversarial max-min game. Especially, the modality-independent common patterns constitute the basic representation of the land-covers, while the specific patterns that are linearly independent of the common patterns that provide the supplementary representation. Therefore, the complementary information is mapped to a compact and discriminative representation. To eliminate the singularity noise, a learnable pattern sampling module (PSM) is designed to extract the mutual-exclusion relationship between specific patterns. Extensive experiments over three datasets demonstrate the superiority of the proposed ACL-CNN compared with several classification technologies.

Hyperspectral and multispectral images (HS/MS) fusion and classification as an important branch of data quality improvement and interpretation, has attracted increasing attention in recent years. However, the unavailable sensor prior still limits the performance of many traditional fusion methods, consequently deteriorating the classification results. Despite the unsupervised methods based on convolutional neural network (CNN) making a lot of attempts to mitigate the limitations, challenges with extracting the long-range dependencies hamper the performance. To address these impediments, a transformer-based baseline constructed by the cross-scale mixing attention (CSMFormer) is designed for HS/MS fusion and classification. Especially, the spatial-spectral mixer (SSMixer) is utilized to extract the long-range dependencies at large scale. Simultaneously, cross-scale feature calibration is achieved by combining information from the original scale. After that, nonlinear enhancement module (NLEM) is designed to encourage feature discrimination. Note that the spatial and spectral mixers can be replaced by any spatial-spectral feature extractors. Therefore, the proposed CSMFormer is flexible in data fusion, land-covers classification, segmentation, etc. Experiments about data fusion and land-covers classification on two HS/MS wetland remote sensing scenes demonstrate the superiority of the proposed CSMFormer baseline, improving the data quality and classification precision.

Joint classification using multisource remote sensing data for Earth observation is promising but challenging. Due to the gap of imaging mechanism and imbalanced information between multisource data, integrating the complementary merits for interpretation is still full of difficulties. In this article, a classification method based on asymmetric feature fusion, named asymmetric feature fusion network (AsyFFNet), is proposed. First, the weight-share residual blocks are utilized for feature extraction while keeping separate batch normalization (BN) layers. In the training phase, redundancy of the current channel is self-determined by the scaling factors in BN, which is replaced by another channel when the scaling factor is less than a threshold. To eliminate unnecessary channels and improve the generalization, a sparse constraint is imposed on partial scaling factors. Besides, a feature calibration module is designed to exploit the spatial dependence of multisource features, so that the discrimination capability is enhanced. Experimental results on the three datasets demonstrate that the proposed AsyFFNet significantly outperforms other competitive approaches.

Due to the limitations of single-source data, joint classification using multisource remote sensing data has received increasing attention. However, existing methods still have certain shortcomings when faced with feature extraction from single-source data and feature fusion between multisource data. In this article, a method based on multiscale interactive information extraction (MIFNet) for hyperspectral and synthetic aperture radar (SAR) image classification is proposed. First, a multiscale interactive information extraction (MIIE) block is designed to extract meaningful multiscale information. Compared with traditional multiscale models, it can not only obtain richer scale information but also reduce the model parameters and lower the network complexity. Furthermore, a global dependence fusion module (GDFM) is developed to fuse features from multisource data, which implements cross attention between multisource data from a global perspective and captures long-range dependence. Extensive experiments on the three datasets demonstrate the superiority of the proposed method and the necessity of each module for accuracy improvement.

Accurate remote sensing monitoring of wetland ground objects is of great significance for ecological protection. In this letter, a convolutional neural network based on feature intersecting learning (FIL-CNN) is designed for wetland classification using multisource remote sensing data. The multi-layer shift feature fusion (MSFF) and attention feature selection (AFS) modules are designed to extract the complementary merits. Specifically, the MSFF is applied to each feature extraction unit, and the asymmetric information fusion is achieved through the spatial position shift of grouped features. Thus, the diversified feature representation is achieved. In the prediction stage, the AFS is executed to explore the channel mutually exclusive relationship between multisource features, resulting in emphasizing the meaningful features and eliminating the unnecessary ones. The experimental results prove the effectiveness and generalization of the proposed FIL-CNN on the wetland datasets.

The monitoring of coastal wetlands is of great importance to the protection of marine and terrestrial ecosystems. However, due to the complex environment, severe vegetation mixture, and difficulty of access, it is impossible to accurately classify coastal wetlands and identify their species with traditional classifiers. Despite the integration of multisource remote sensing data for performance enhancement, there are still challenges with acquiring and exploiting the complementary merits from multisource data. In this article, the depthwise feature interaction network (DFINet) is proposed for wetland classification. A depthwise cross attention module is designed to extract self-correlation and cross correlation from multisource feature pairs. In this way, meaningful complementary information is emphasized for classification. DFINet is optimized by coordinating consistency loss, discrimination loss, and classification loss. Accordingly, DFINet reaches the standard solution-space under the regularity of loss functions, while the spatial consistency and feature discrimination are preserved. Comprehensive experimental results on two hyperspectral and multispectral wetland datasets demonstrate that the proposed DFINet outperforms other competitive methods in terms of overall accuracy.

Due to rich spectral and spatial information, the combination of hyperspectral and multispectral images (MSIs) has been widely used for Earth observation, such as wetland classification. However, mining of meaningful features and effective fusion of multisource remote sensing data are still urgent problems to be solved. In this article, graph-feature-enhanced selective assignment network (GSANet) is proposed. On the one hand, a graph feature extraction module (GFEM) is designed to extract topological structure information and combine with the rich spectral–spatial information. In particular, the features obtained by convolution are first mapped to the graph feature space, and the graph convolution operation is used to achieve propagation between nodes for preserving topological structure information. Moreover, to reduce the difference of graph features resulting from the mapping function and better explore the complementary properties of multisource data, a novel graph fusion strategy-graph dependence fusion is designed. A transition graph is generated to enhance the association and interaction between different graph features, so as to avoid the information loss caused by simple fusion operation. On the other hand, a selective feature assignment module (SFAM) is developed to adaptively assign weights to different discriminative features. SFAM assigns weights to different features to selectively emphasize informative features and suppress less useful ones. Extensive experiments are conducted on two multisource remote sensing datasets, and the improvement of at least 1.27% and 0.98% compared to other state-of-the-art work demonstrates the superiority of the proposed GSANet.

Wetland classification is of great significance for monitoring. Recently, collaborative analysis of multisource data has received special attention considering the limitations of single source data. In this paper, a wetland classification method based on feature exchange is proposed. Firstly, the weighting shared residual blocks are utilized for feature extraction. Then, the scaling factors in batch normalization (BN) self-determine the redundancy of current channel, which is replaced by another channel when the scaling factor is less than the threshold. To eliminate unnecessary channels and improve the generalization, sparsity constraint is employed on partial scaling factors. Experimental results on multisource wetland dataset demonstrate that the proposed method outperforms other competitive works.

Traditional synthetic-aperture radar (SAR) image change detection methods based on convolutional neural networks (CNNs) face the challenges of speckle noise and deformation sensitivity. To mitigate these issues, we proposed a multiscale capsule network (Ms-CapsNet) to extract the discriminative information between the changed and unchanged pixels. On the one hand, the multiscale capsule module is employed to exploit the spatial relationship of features. Therefore, equivariant properties can be achieved by aggregating the features from different positions. On the other hand, an adaptive fusion convolution (AFC) module is designed for the proposed Ms-CapsNet. The higher semantic features can be captured for the primary capsules. Feature extracted by the AFC module significantly improves the robustness to speckle noise. The effectiveness of the proposed Ms-CapsNet is verified on three real SAR data sets. The comparison experiments with four state-of-the-art methods demonstrate the efficiency of the proposed method. Our codes are available at https://github.com/summitgao/SAR_CD_MS_CapsNet.

High-quality sea ice monitoring is crucial to navigation safety and climate research in the polar regions. In this letter, a transferred multilevel fusion network (MLFN) is proposed for sea ice change detection from synthetic-aperture radar (SAR) images. Considering the fact that training data are limited in the task of sea ice change detection, a large data set was used to train the MLFN, and the deep knowledge can be transferred to sea ice analysis. In addition, cascade dense blocks are employed to optimize the convolutional layers. Multilayer feature fusion is introduced to exploit the complementary information among low-, mid-, and high-level feature representations. Therefore, more discriminative feature extraction can be achieved by the MLFN. Furthermore, the fine-tune strategy is utilized to optimize the network parameters. The experimental results on two real sea ice data sets demonstrated that the proposed method achieved better performance than other competitive methods.

Sea ice change detection from synthetic aperture radar (SAR) images can be regarded as a classification procedure, in which pixels are classified into changed and unchanged classes. However, existing methods usually suffer from the intrinsic speckle noise of multitemporal SAR images. To solve the problem, this letter presents a change detection method based on convolutional-wavelet neural networks (CWNNs). In CWNN, dual-tree complex wavelet transform is introduced into convolutional neural networks for changed and unchanged pixels' classification, and then, the effect of speckle noise is effectively reduced. In addition, a virtual sample generation scheme is employed to create samples for CWNN training, and the problem of limited samples is alleviated. Experimental results on two real SAR image data sets demonstrate the effectiveness and robustness of the proposed method.

Sea ice change detection from synthetic aperture radar (SAR) images is important for navigation safety and natural resource extraction. This paper proposed a sea ice change detection method from SAR images based on collaborative representation. First, neighborhood-based ratio is used to generate a difference image (DI). Then, some reliable samples are selected from the DI by hierarchical fuzzy C-means (FCM) clustering. Finally, based upon these samples, collaborative representation method is utilized to classify pixels from the original SAR images into unchanged and changed class. From there, the final change map can be obtained. Experimental results on two real sea ice datasets demonstrate the superiority of the proposed method over two closely related methods.