Change detection(CD) is important for Earth observation, emergency response and time-series understanding. Recently, data availability in various modalities has increased rapidly, and multimodal change detection (MCD) is gaining prominence. Given the scarcity of datasets and labels for MCD, unsupervised approaches are more practical for MCD. However, previous methods typically either merely reduce the gap between multimodal data through transformation or feed the original multimodal data directly into the discriminant network for difference extraction. The former faces challenges in extracting precise difference features. The latter contains the pronounced intrinsic distinction between the original multimodal data; direct extraction and comparison of features usually introduce significant noise, thereby compromising the quality of the resultant difference image. In this article, we proposed the MaCon framework to synergistically distill the common and discrepancy representations. The MaCon framework unifies mask reconstruction (MR) and contrastive learning (CL) self-supervised paradigms, where the MR serves the purpose of transformation while CL focuses on discrimination. Moreover, we presented an optimal sampling strategy in the CL architecture, enabling the CL subnetwork to extract more distinguishable discrepancy representations. Furthermore, we developed an effective silent attention mechanism that not only enhances contrast in output representations but stabilizes the training. Experimental results on both multimodal and monomodal datasets demonstrate that the MaCon framework effectively distills the intrinsic common representations between varied modalities and manifests state-of-the-art performance across both multimodal and monomodal CD. Such findings imply that the MaCon possesses the potential to serve as a unified framework in the CD and relevant fields. Source code will be publicly available once the article is accepted.

Cross-scene image classification aims to transfer prior knowledge of ground materials to annotate regions with different distributions and reduce hand-crafted cost in the field of remote sensing. However, existing approaches focus on single-source domain generalization to unseen target domains, and are easily confused by large real-world domain shifts due to the limited training information and insufficient diversity modeling capacity. To address this gap, we propose a novel multi-source collaborative domain generalization framework (MS-CDG) based on homogeneity and heterogeneity characteristics of multi-source remote sensing data, which considers data-aware adversarial augmentation and model-aware multi-level diversification simultaneously to enhance cross-scene generalization performance. The data-aware adversarial augmentation adopts an adversary neural network with semantic guide to generate MS samples by adaptively learning realistic channel and distribution changes across domains. In views of cross-domain and intra-domain modeling, the model-aware diversification transforms the shared spatial-channel features of MS data into the class-wise prototype and kernel mixture module, to address domain discrepancies and cluster different classes effectively. Finally, the joint classification of original and augmented MS samples is employed by introducing a distribution consistency alignment to increase model diversity and ensure better domain-invariant representation learning. Extensive experiments on three public MS remote sensing datasets demonstrate the superior performance of the proposed method when benchmarked with the state-of-the-art methods.

Supervised deep learning (DL)-based hyperspectral images change detection (HSIs-CD) has demonstrated excellent performance; however, current methods require many labeled training samples, and labeling the dataset is labor-intensive, limiting the application of high-precision supervised learning. Besides, there has been a lack of breakthroughs in unsupervised HSIs change detection (CD) methods due to the different feature distributions of bitemporal HSIs. Here, we propose a semi-supervised domain alignment transformer (DA-Former) for HSIs-CD to address the issues with limited samples. Specifically, a dual-branch transformer autoencoder (TAE) is designed, where the middle layer weights of the dual-branch transformer are shared, pulling features from different data into the same space. Moreover, the TAE is also trained cyclically to align the domains. Although the bitemporal HSIs features are cross domain, there is still confusion between the features of different objects. Thus, two fully connected (FC) layers are employed to classify the HSIs middle features extracted by the TAE into changed class or unchanged class with limited labeled data. Three HSIs-CD datasets are used to test this method, showing that TAE can align bitemporal HSIs domains and achieve the highest accuracy compared with benchmark approaches. The code of the proposed method will be published athttps://github.com/yanhengwang-heu/IEEE_TGRS_DA-Former.

Multi-feature SAR ship classification aims to build models that can process, correlate, and fuse information from both handcrafted and deep features. Although handcrafted features provide rich expert knowledge, current fusion methods inadequately explore the relatively significant role of handcrafted features in conjunction with deep features, the imbalances in feature contributions, and the cooperative ways in which features learn. In this paper, we propose a novel multi-feature collaborative fusion network with deep supervision (MFCFNet) to effectively fuse handcrafted features and deep features for SAR ship classification tasks. Specifically, our framework mainly includes two types of feature extraction branches, a knowledge supervision and collaboration module, and a feature fusion and contribution assignment module. The former module improves the quality of the feature maps learned by each branch through auxiliary feature supervision and introduces a synergy loss to facilitate the interaction of information between deep features and handcrafted features. The latter module utilizes an attention mechanism to adaptively balance the importance among various features and assign the corresponding feature contributions to the total loss function based on the generated feature weights. We conducted extensive experimental and ablation studies on two public datasets, OpenSARShip-1.0 and FUSAR-Ship, and the results show that MFCFNet is effective and outperforms single deep feature and multi-feature models based on previous internal FC layer and terminal FC layer fusion. Furthermore, our proposed MFCFNet exhibits better performance than the current state-of-the-art methods.

Accurate topography of intertidal mudflats of a fine resolution is fundamental data to further understand the coastal process and achieve targeted coastal management. However, the mapping of mudflat topography is hindered by poor accessibility of muddy environments and a short observation time window caused by periodic tides. Commonly as a revolutionary technique owing to its low cost, flexibility, and quality data, the unmanned aerial vehicle (UAV)-based SfM photogrammetry has been widely applied in coastal areas. The conventional UAV photogrammetric accuracy significantly depends on the number and distribution of ground control points (GCPs), limiting its mapping efficiency. With the increasingly available UAVs with onboard RTK, photogrammetry without GCPs is becoming a promising alternative. However, the ability of this advanced RTK-assisted UAV to capture centimeter-scale elevation changes in intertidal mudflats still remains unclear. For this reason, this paper aims to evaluate the potential of RTK-assisted UAVs in quantifying intertidal topographic changes. The results showed that the RTK-assisted UAV structure-from-motion (SfM) photogrammetry without GCPs could accurately capture fine-scale topographical features such as mudflat gradient and creeks with root-mean-squared errors (RMSE) of ± 3.3 cm, ± 2.8 cm, and ± 4.7 cm on X-, Y-, and vertical directions, respectively. Therefore, this study identified that RTK-assisted UAV photogrammetry could be used to quantify intertidal mudflat morphodynamics and calculate sediment deposition volume within the uncertainty in a more cost-effective way.

Semantic segmentation of remotely sensed images plays an important role in land resource management, yield estimation, and economic assessment. U-Net, a deep encoder–decoder architecture, has been used frequently for image segmentation with high accuracy. In this letter, we incorporate multiscale features generated by different layers of U-Net and design a multiscale skip connected and asymmetric-convolution-based U-Net (MACU-Net), for segmentation using fine-resolution remotely sensed images. Our design has the following advantages: (1) the multiscale skip connections combine and realign semantic features contained in both low-level and high-level feature maps; (2) the asymmetric convolution block strengthens the feature representation and feature extraction capability of a standard convolution layer. Experiments conducted on two remotely sensed data sets captured by different satellite sensors demonstrate that the proposed MACU-Net transcends the U-Net, U-Netpyramid pooling layers (PPL), U-Net 3+, among other benchmark approaches. Code is available athttps://github.com/lironui/MACU-Net.

Deep convolutional networks have been the most competitive method in remote sensing scene classification. Due to the diversity and complexity of scene content, remote sensing scene classification still remains a challenging task. Recently, the second-order pooling method has attracted more interest because it can learn higher-order information and enhance the nonlinear modeling ability of the networks. However, how to effectively learn second-order features and establish the discriminative feature representation of holistic images is still an open question. In this letter, we propose a first and second-order information fusion network (FSoI-Net) that can learn the first-order and second-order features at the same time, and construct the final feature representation by fusing the two types of features. Specifically, a self-attention-based second-order pooling (SaSoP) method based on covariance matrix is proposed to extract second-order features, and a fusion loss function is developed to jointly train the model and construct the final feature representation for the classification decision. The proposed network has been thoroughly evaluated on three real remote sensing scene datasets and achieved better performance than the counterparts.

The attention mechanism can refine the extracted feature maps and boost the classification performance of the deep network, which has become an essential technique in computer vision and natural language processing. However, the memory and computational costs of the dot-product attention mechanism increase quadratically with the spatiotemporal size of the input. Such growth hinders the usage of attention mechanisms considerably in application scenarios with large-scale inputs. In this letter, we propose a linear attention mechanism (LAM) to address this issue, which is approximately equivalent to dot-product attention with computational efficiency. Such a design makes the incorporation between attention mechanisms and deep networks much more flexible and versatile. Based on the proposed LAM, we refactor the skip connections in the raw U-Net and design a multistage attention ResU-Net (MAResU-Net) for semantic segmentation from fine-resolution remote sensing images. Experiments conducted on the Vaihingen data set demonstrated the effectiveness and efficiency of our MAResU-Net. Our code is available athttps://github.com/lironui/MAResU-Net.

Semantic segmentation of remote sensing images plays a crucial role in a wide variety of practical applications, including land cover mapping, environmental protection, and economic assessment. In the last decade, convolutional neural network (CNN) is the mainstream deep learning-based method of semantic segmentation. Compared with conventional methods, CNN-based methods learn semantic features automatically, thereby achieving strong representation capability. However, the local receptive field of the convolution operation limits CNN-based methods from capturing long-range dependencies. In contrast, Vision Transformer (ViT) demonstrates its great potential in modeling long-range dependencies and obtains superior results in semantic segmentation. Inspired by this, in this letter, we propose a class-guided Swin Transformer (CG-Swin) for semantic segmentation of remote sensing images. Specifically, we adopt a Transformer-based encoder–decoder structure, which introduces the Swin Transformer backbone as the encoder and designs a class-guided Transformer block to construct the decoder. The experimental results on ISPRS Vaihingen and Potsdam datasets demonstrate the significant breakthrough of the proposed method over ten benchmarks, outperforming both advanced CNN-based and recent Transformer-based approaches.

The fully convolutional network (FCN) with an encoder-decoder architecture has been the standard paradigm for semantic segmentation. The encoder-decoder architecture utilizes an encoder to capture multilevel feature maps, which are incorporated into the final prediction by a decoder. As the context is crucial for precise segmentation, tremendous effort has been made to extract such information in an intelligent fashion, including employing dilated/atrous convolutions or inserting attention modules. However, these endeavors are all based on the FCN architecture with ResNet or other backbones, which cannot fully exploit the context from the theoretical concept. By contrast, we introduce the Swin Transformer as the backbone to extract the context information and design a novel decoder of densely connected feature aggregation module (DCFAM) to restore the resolution and produce the segmentation map. The experimental results on two remotely sensed semantic segmentation datasets demonstrate the effectiveness of the proposed scheme.

Large-scale (large-area), fine spatial resolution satellite sensor images are valuable data sources for Earth observation while not yet fully exploited by research communities for practical applications. Often, such images exhibit highly complex geometrical structures and spatial patterns, and distinctive characteristics of multiple land-use categories may appear at the same region. Autonomous information extraction from these images is essential in the field of pattern recognition within remote sensing, but this task is extremely challenging due to the spectral and spatial complexity captured in satellite sensor imagery. In this research, a semi-supervised deep rule-based approach for satellite sensor image analysis (SeRBIA) is proposed, where large-scale satellite sensor images are analysed autonomously and classified into detailed land-use categories. Using an ensemble feature descriptor derived from pre-trained AlexNet and VGG-VD-16 models, SeRBIA is capable of learning continuously from both labelled and unlabelled images through self-adaptation without human involvement or intervention. Extensive numerical experiments were conducted on both benchmark datasets and real-world satellite sensor images to comprehensively test the validity and effectiveness of the proposed method. The novel information mining technique developed here can be applied to analyse large-scale satellite sensor images with high accuracy and interpretability, across a wide range of real-world applications.

Semantic segmentation of remote sensing images plays an important role in a wide range of applications, including land resource management, biosphere monitoring, and urban planning. Although the accuracy of semantic segmentation in remote sensing images has been increased significantly by deep convolutional neural networks, several limitations exist in standard models. First, for encoder–decoder architectures such as U-Net, the utilization of multiscale features causes the underuse of information, where low-level features and high-level features are concatenated directly without any refinement. Second, long-range dependencies of feature maps are insufficiently explored, resulting in suboptimal feature representations associated with each semantic class. Third, even though the dot-product attention mechanism has been introduced and utilized in semantic segmentation to model long-range dependencies, the large time and space demands of attention impede the actual usage of attention in application scenarios with large-scale input. This article proposed a multiattention network (MANet) to address these issues by extracting contextual dependencies through multiple efficient attention modules. A novel attention mechanism of kernel attention with linear complexity is proposed to alleviate the large computational demand in attention. Based on kernel attention and channel attention, we integrate local feature maps extracted by ResNet-50 with their corresponding global dependencies and reweight interdependent channel maps adaptively. Numerical experiments on two large-scale fine-resolution remote sensing datasets demonstrate the superior performance of the proposed MANet. Code is available athttps://github.com/lironui/Multi-Attention-Network.

This paper proposes an automatic method for detection and mapping of Espeletia plants from aerial images acquired by UAV drone. The proposed approach integrated a computer vision for automatic extraction of training zones and tested on three well-established machine learning algorithms to detect regions belonging to Espeletia plants. The main components of the method are: (i) data capture and preprocessing; (ii) automatic extraction of training zones; and (iii) classification procedure using machine learning algorithms. Experimental results show that the method can achieve accurate detection and mapping of Espeletia plants, with up to 98.3% accuracy.

In this letter, we propose a new approach for remote sensing scene classification by creating an ensemble of the recently introduced massively parallel deep (fuzzy) rule-based (DRB) classifiers trained with different levels of spatial information separately. Each DRB classifier consists of a massively parallel set of human-interpretable, transparent zero-order fuzzy IF...THEN... rules with a prototype-based nature. The DRB classifier can self-organize “from scratch” and self-evolve its structure. By employing the pretrained deep convolution neural network as the feature descriptor, the proposed DRB ensemble is able to exhibit human-level performance through a transparent and parallelizable training process. Numerical examples using benchmark data set demonstrate the superior accuracy of the proposed approach together with human-interpretable fuzzy rules autonomously generated by the DRB classifier.

Recent advances in computer vision and pattern recognition have demonstrated the superiority of deep neural networks using spatial feature representation, such as convolutional neural networks (CNNs), for image classification. However, any classifier, regardless of its model structure (deep or shallow), involves prediction uncertainty when classifying spatially and spectrally complicated very fine spatial resolution (VFSR) imagery. We propose here to characterize the uncertainty distribution of CNN classification and integrate it into a regional decision fusion to increase classification accuracy. Specifically, a variable precision rough set (VPRS) model is proposed to quantify the uncertainty within CNN classifications of VFSR imagery and partition this uncertainty into positive regions (correct classifications) and nonpositive regions (uncertain or incorrect classifications). Those “more correct” areas were trusted by the CNN, whereas the uncertain areas were rectified by a multilayer perceptron (MLP)-based Markov random field (MLP-MRF) classifier to provide crisp and accurate boundary delineation. The proposed MRF-CNN fusion decision strategy exploited the complementary characteristics of the two classifiers based on VPRS uncertainty description and classification integration. The effectiveness of the MRF-CNN method was tested in both urban and rural areas of southern England as well as semantic labeling data sets. The MRF-CNN consistently outperformed the benchmark MLP, support vector machine, MLP-MRF, CNN, and the baseline methods. This paper provides a regional decision fusion framework within which to gain the advantages of model-based CNN, while overcoming the problem of losing effective resolution and uncertain prediction at object boundaries, which is especially pertinent for complex VFSR image classification.

The formation of an anisotropic landscape is influenced by natural and/or human processes, which can then be inferred on the basis of geometric indices. In this study, two minimal bounding rectangles in consideration of the principles of mechanics (i.e. minimal width bounding (MWB) box and moment bounding (MB) box) were introduced. Based on these boxes, four novel shape indices, namely MBLW (the length-to-width ratio of MB box), PAMBA (area ratio between patch and MB box), PPMBP (perimeter ratio between patch and MB box) and ODI (orientation difference index between MB and MWB boxes), were introduced to capture multiple aspects of landscape features including patch elongation, patch compactness, patch roughness and patch symmetry. Landscape pattern was, thus, quantified by considering both patch directionality and patch shape simultaneously, which is especially suitable for anisotropic landscape analysis. The effectiveness of the new indices were tested with real landscape data consisting of three kinds of saline soil patches (i.e. the elongated shaped slightly saline soil class, the circular or half-moon shaped moderately saline soil, and the large and complex severely saline soil patches). The resulting classification was found to be more accurate and robust than that based on traditional shape complexity indices.