Multimodal cross-city semantic segmentation aims to adapt a network trained on multiple labeled source domains (MSDs) from one city to multiple unlabeled target domains (MTDs) in another city, where the multiple domains refer to different sensor modalities. However, remote sensing data from different sensors increases the extent of domain shift in the fused domain space, making feature alignment more challenging. Meanwhile, traditional fusion methods only consider complementarity within MSDs (or MTDs), which wastes cross-domain relevant information and neglects control over domain shift. To address the above issues, we propose a similarity-inspired fusion and invertible transformation learning network (SFITNet) for multimodal cross-city semantic segmentation. To alleviate the increasing alignment difficulty in multimodal fused domains, an invertible transformation learning strategy (ITLS) is proposed, which adopts a topological perspective on unsupervised domain adaptation. This strategy aims to simulate the potential distribution transformation function between the MSD and the MTD based on invertible neural networks (INNs) after feature fusion, thereby performing distribution alignment independently within the two feature spaces. A cross-domain similarity-inspired information interaction module (CDSiM) is also designed, which considers the correspondence between the MSD and the MTD in the fusion stage, effectively utilizes multimodal complementary information and promotes the subsequent alignment of fused domain shifts. The semantic segmentation tests are completed on the public C2Seg-AB dataset and a new multimodal cross-city Su-Wu dataset. Compared with some state-of-the-art techniques, the experimental results demonstrated the superiority of the proposed SFITNet.

With the growth of multisource sensor technology, multimodal learning has become pivotal in remote sensing (RS) image segmentation. Despite its potential, current methods face challenges in acquiring large-scale paired samples. When annotated optical images are available, but synthetic aperture radar (SAR) images lack annotations, learning discriminative features for SAR images from optical images becomes difficult. Unsupervised domain adaptation (UDA) offers a potential solution to this challenge, which we refer to as unpaired cross-modality UDA. In this article, we propose unlocking pseudolabel potential and alignment (ULPA) for unpaired cross-modality adaptation in RS image segmentation, a novel one-stage adaptation framework designed to enhance cross-modality knowledge transfer. Our approach employs a prototypical multidomain alignment (PMDA) strategy, which reduces the modality gap through contrastive learning between features and prototypes of identical classes across different modalities. In addition, we introduce the unreliable-sample-guided feature contrast (UFC) loss to address the underutilization of unreliable pixels during training. This strategy separates reliable and unreliable pixels based on prediction confidence, assigning unreliable pixels to a category-wise queue of negative samples, thus ensuring all candidate pixels contribute to the training process. Extensive experiments show that the integration of PMDA and UFC loss can lead to more effective cross-modality domain alignment and substantially boost the model's generalization capability.

Semi-supervised few-shot learning aims to alleviate the issue of insufficient labeled data with additional unlabeled samples. As for remote sensing images, complex contextual information leads to pseudo labeling with low confidence, which weakens the effect of semi-supervised few-shot classification. To solve these issues, a hypergraph matching network is proposed for semi-supervised few-shot scene classification of remote sensing images. Specifically, a hypergraph propagation module is designed to construct a hypergraph network, which can take advantage of adjacent samples with similar semantics and improve the representation ability of class prototypes. Then, a cross-layer prototype matching module is proposed to dynamically match features of different scales and angles, which aims to predict pseudo labels with high confidences. Experimental results on three public remote sensing datasets demonstrate that the proposed method can make effective utilization of additional unlabeled samples to enhance the classification performance of few-shot learning.

Unsupervised domain adaptation for remote sensing image semantic segmentation aims to train a deep model on the labeled source domain and apply it to the unlabeled target domain. However, resolution and scene inconsistencies of cross-domain remote sensing images lead to great distribution differences, which weakens the semantic segmentation effect. To solve the above issues, an unsupervised remote sensing image semantic segmentation method is proposed based on multi-scale contrastive domain adaptation. Firstly, the mean teacher model is introduced into the unsupervised domain adaptation paradigm to generate pseudo-labels for target domain data, thereby achieving the cross-domain segmentation capability. A dynamic class balance sampling method is proposed to mitigate the class imbalance problem in cross-domain data by increasing the sampling frequency of the categories with fewer samples. Then, a data augmentation method called cross-domain mixup is developed to reduce the gap between the source and target domains. Finally, a multi-scale cross-domain contrastive loss is developed, which introduces the contrastive learning to learn domain-consistent features across the source and target domains, resulting in a more coherent and discriminative feature representation. Experimental results show that the proposed method can yield superior performance for unsupervised remote sensing image semantic segmentation.

The fusion of optical, hyperspectral, and Synthetic Aperture Radar (SAR) images is essential for semantic segmentation in remote sensing, enabling more comprehensive land cover classification through multimodal data integration. However, disparities in spatial resolution and imaging characteristics across modalities impede effective feature alignment and fusion, degrading segmentation performance. To address this problem, we propose a novel Spectral-Geometric Iterative Fusion Network (SGIFN), specifically designed to handle multimodal semantic segmentation with inconsistent resolutions. The core innovation of SGIFN lies in its unified architecture that progressively aligns, integrates, and enhances multimodal features through three newly designed modules. The Spectral-Spatial Iterative Decoupling (SSID) module introduces a novel iterative mechanism to adaptively align and decouple optical and hyperspectral features. The Spectral-Geometric Synergistic Conditional Random Field (SGS-CRF) module captures both local and long-range spatial dependencies by synergizing geometric (SAR) and spectral information. The Class-Guided Multiscale Contrastive Aggregation (CG-MCA) module further strengthens feature representation across scales via multi-class, contrastive learning. We constructed a new multimodal remote sensing dataset comprising optical, hyperspectral, and SAR images with varying resolutions collected from the Wuhan and Suzhou regions. Experimental results show that SGIFN achieves an mIoU of 69.61% on the Suzhou dataset and 63.96% on the Wuhan dataset. These results demonstrate the effectiveness of SGIFN in handling multimodal data with inconsistent resolutions.

Intrusion detection systems are a crucial technique for securing the Internet of Things (IoT) from malicious attacks. Additionally, due to the continuous emergence of new vulnerabilities and unknown attack types, only a small number of attack samples in the IoT environments can be captured for analysis. In this work, we introduce an anchor graph clustering (AGC) method for intrusion detection to address the challenge of limited labeled samples in the IoT environments. AGC initially transforms the raw data into the embedding space to obtain more representative anchors. Then, AGC unifies anchor graph construction, anchor graph learning, and graph clustering into a unified framework, solving the resulting optimization problem through an iterative solution algorithm. Finally, AGC leverages the powerful analytical capabilities of graph learning to achieve fine-grained classification of low-quality labels. Experimental results on both real and synthetic datasets confirm that AGC can identify intrusions with high precision, while also being time-efficient in detection.

The feature fusion of optical and Synthetic Aperture Radar (SAR) images is widely used for semantic segmentation of multimodal remote sensing images. It leverages information from two different sensors to enhance the analytical capabilities of land cover. However, the imaging characteristics of optical and SAR data are vastly different, and noise interference makes the fusion of multimodal data information challenging. Furthermore, in practical remote sensing applications, there are typically only a limited number of labeled samples available, with most pixels needing to be labeled. Semi-supervised learning has the potential to improve model performance in scenarios with limited labeled data. However, in remote sensing applications, the quality of pseudo-labels is frequently compromised, particularly in challenging regions such as blurred edges and areas with class confusion. This degradation in label quality can have a detrimental effect on the model's overall performance. In this paper, we introduce the Difference-complementary Learning and Label Reassignment (DLLR) network for multimodal semi-supervised semantic segmentation of remote sensing images. Our proposed DLLR framework leverages asymmetric masking to create information discrepancies between the optical and SAR modalities, and employs a difference-guided complementary learning strategy to enable mutual learning. Subsequently, we introduce a multi-level label reassignment strategy, treating the label assignment problem as an optimal transport optimization task to allocate pixels to classes with higher precision for unlabeled pixels, thereby enhancing the quality of pseudo-label annotations. Finally, we introduce a multimodal consistency cross pseudo-supervision strategy to improve pseudo-label utilization. We evaluate our method on two multimodal remote sensing datasets, namely, the WHU-OPT-SAR and EErDS-OPT-SAR datasets. Experimental results demonstrate that our proposed DLLR model outperforms other relevant deep networks in terms of accuracy in multimodal semantic segmentation.

Quantifying the epistemic uncertainty for static information and dynamic fusion or reasoning processes is still unsolved for various epistemic uncertainty theories. This study focuses on the Dempster-Shafer evidence theory, which is of great ability in representing and fusing uncertain information with imprecision and ignorance on the basis of basic probability assignment (BPA) and Dempster combination rule (DCR). In order to effectively measure and infer the epistemic uncertainty for both static BPAs and dynamic fusion processes, a solution based on plausibility entropy is proposed in this study. At first, four new properties, called grouping, splitting, weighted additivity, and weighted subadditivity, are proved for the first time in this study to strengthen the theoretical foundation of plausibility entropy in measuring the uncertainty associated with a given BPA. Second, the upper bounds of uncertainty are derived for typical BPA-based multisource information fusion systems, including standard DCR, weighted DCR, discounted DCR fusion systems for evidence defined on the same frame of discernment (FOD), and the DCR fusion system for evidence defined on multiple distinct FODs. Several examples are given to illustrate these results.

Yufei Ma, Zihan Liang, Huangyu Dai, Ben Chen, Dehong Gao, Zhuoran Ran, Wang Zihan, Linbo Jin, Wen Jiang, Guannan Zhang, Xiaoyan Cai, Libin Yang. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

With the continuous development and advancement of neural networks, in the application of neural networks, users not only require neural networks to be able to complete a given task but also want to know when they can trust the network’s prediction results and when they need to be cautious about the prediction results. In response to the need for uncertainty estimation of neural networks, many researchers have invested in the study of uncertainty estimation. Existing uncertainty evaluation methods are difficult to apply to deep neural networks with large parameter scales, complex internal structures, and mappings between inputs and outputs that are hard to express. This paper proposes a key nodes based resampling uncertainty estimation method ((KN-RUE), which achieves uncertainty estimation of prediction results for arbitrarily given large-scale neural networks. In this method, the first step involves analyzing the differences in feature space between adversarial and clean samples, identifying the main nodes affected by adversarial samples, and determining the critical nodes within the network. Next, by resampling the parameters of key nodes, the model is extended while ensuring model performance as much as possible, thus completing the measurement of uncertainty in prediction results. Through experiments, the effectiveness of the extended model and the superiority of uncertainty estimation performance in KN-RUE have been verified.

Federated learning is a nascent distributed machine learning paradigm that enables multiple clients to collaborate in training a model for a specific task under the coordination of a central server, all while safeguarding the privacy of the user’s local data. Nevertheless, the constraint that distributed datasets must remain within local nodes introduces data heterogeneity in federated learning training. In this paper, we focus on how to mitigate the damage caused by the data heterogeneity of feature distribution skew in federated learning models during training. To achieve this goal, we propose a feature drift-corrected federated learning algorithm. We design a feature drift variable derived from the local models of clients and the global model of the server. This variable is incorporated into the client’s local loss function to rectify local model parameters. Additionally, we utilize the disparity between the global models before and after to regulate the local model. Validation experiments are conducted on multiple datasets exhibiting feature distribution skew. The implementation results demonstrate the efficacy of our approach in significantly enhancing the model performance of federated learning under feature distribution skew.

With recent advances in remote sensing technology, a wealth of multimodal data is available for applications. However, considering the domain differences between multimodal data and the alignment challenges in practical applications, it becomes important and challenging to integrate these data effectively. In this paper, we propose a multimodal prototype representation fusion network (MPRFN) for SAR and optical image fusion segmentation. Specifically, a more robust multimodal feature representation is provided by constructing multimodal category prototype representations that better capture the characteristics and distribution of each data. Meanwhile, a prototype-consistent semi-supervised learning method is proposed to improve the effectiveness of multimodal fusion semantic segmentation using a large number of unlabelled unimodal SAR images. Experiments on SAR and optical multimodal datasets show that the proposed method achieves state-of-the-art performance.

The utilization of deep learning methods in remote sensing image scene classification (RSISC) has gained significant attention, showcasing remarkable performance. However, these methods rely solely on the network for automatic weight assignment learning, which may introduce biases in attention calculations for remote sensing images. To address this issue, we propose a multi-view collaborative representation network (MCRNet) for RSISC. Specifically, we introduce a multiview collaborative representation framework (MCRF) to evaluate the impact of local features on key information within global features by different data augmentation. Furthermore, the introduction of a semantic summarization dictionary (SSD) aims to enhance the reconstruction of global semantic features through the optimization of a low-redundancy dictionary. Experiment results on two publicly available datasets confirm that the proposed model effectively improves the classification performance.

Recent studies on AI security have highlighted the vulnerability of Vision-Language Pre-training (VLP) models to subtle yet intentionally designed perturbations in images and texts. Investigating multimodal systems' robustness via adversarial attacks is crucial in this field. Most multimodal attacks are sample-specific, generating a unique perturbation for each sample to construct adversarial samples. To the best of our knowledge, it is the first work through multimodal decision boundaries to explore the creation of a universal, sample-agnostic perturbation that applies to any image. Initially, we explore strategies to move sample points beyond the decision boundaries of linear classifiers, refining the algorithm to ensure successful attacks under the top k accuracy metric. Based on this foundation, in visual-language tasks, we treat visual and textual modalities as reciprocal sample points and decision hyperplanes, guiding image embeddings to traverse text-constructed decision boundaries, and vice versa. This iterative process consistently refines a universal perturbation, ultimately identifying a singular direction within the input space which is exploitable to impair the retrieval performance of VLP models. The proposed algorithms support the creation of global perturbations or adversarial patches. Comprehensive experiments validate the effectiveness of our method, showcasing its data, task, and model transferability across various VLP models and datasets. Code: https://github.com/LibertazZ/MUAP

With Vision-Language Pre-training (VLP) models demonstrating powerful multimodal interaction capabilities, the application scenarios of neural networks are no longer confined to unimodal domains but have expanded to more complex multimodal V+L downstream tasks. The security vulnerabilities of unimodal models have been extensively examined, whereas those of VLP models remain challenging. We note that in CV models, the understanding of images comes from annotated information, while VLP models are designed to learn image representations directly from raw text. Motivated by this discrepancy, we developed the Feature Guidance Attack (FGA), a novel method that uses text representations to direct the perturbation of clean images, resulting in the generation of adversarial images. FGA is orthogonal to many advanced attack strategies in the unimodal domain, facilitating the direct application of rich research findings from the unimodal to the multimodal scenario. By appropriately introducing text attack into FGA, we construct Feature Guidance with Text Attack (FGA-T). Through the interaction of attacking two modalities, FGA-T achieves superior attack effects against VLP models. Moreover, incorporating data augmentation and momentum mechanisms significantly improves the black-box transferability of FGA-T. Our method demonstrates stable and effective attack capabilities across various datasets, downstream tasks, and both black-box and white-box settings, offering a unified baseline for exploring the robustness of VLP models.

The target recognition of synthetic aperture radar (SAR) data generally faces the issue of limited observational samples in practical applications. Recent few-shot SAR target recognition techniques based on meta-learning, which mainly focus on intricate meta-learning models without considering SAR imaging characteristics during model training, show promise. To address this issue, a novel few-shot transfer learning paradigm named causal intervention and parameter-free reasoning (CIPR) is proposed for SAR target recognition. In the proposed framework, causal intervention pretraining (CIP), which emphasizes causal features of SAR images, is developed to diminish spurious correlations caused by confounders. Moreover, variational inference approximates intricate alterations in SAR imaging angles and background clutter in a generative manner. To make predictions of the unlabelled query set without additional learnable parameters, a parameter-free label reasoning model based on optimal transport, which integrates label knowledge and effectively leverages the distribution characteristics of causal features, is introduced. Experiments on the moving and stationary target acquisition and recognition (MSTAR) dataset demonstrate that the proposed method achieves superior performance and has preferable robustness to large depression angle discrepancies.

Semantic segmentation is a significant task for remote sensing interpretation, which takes advantage of contextual semantic information to classify each pixel into a specific category. Most current methods apply convolutional neural networks (CNN) to learn feature representation from remote sensing images, which may ignore the global dependencies due to the limitation of convolutional kernels. Inspired by the global feature learning ability of Transformer, we propose a novel deep model called dual-path feature aware network (DPFANet), which combines the structure of CNN and Transformer for semantic segmentation of remote sensing images. DPFANet aims to learn effective modeling ability from local to global features of images. Simultaneously, an adaptive feature fusion network is developed to fuse features from dual-path networks. Moreover, an edge optimization block is applied to constrain the edge features, whose purpose is to obtain more representative features for segmentation. Experimental results on three public remote sensing datasets verify that our proposed network yields better segmentation performance compared to other related methods.

Hyperspectral cross-scene classification refers to using only labeled data from the source domain in training and testing directly on the target domain dataset. However, there are differences between the reflection spectra of objects with the same category, which makes the cross-scene classification performance drop significantly. The task of single-domain generalization (SDG) has received extensive attention to solve the above problem. To address the discrepancy between source and target domains, a spatial-spectral enhancement and causal constraint network (S2ECNet) in terms of both data enhancement and causal alignment is proposed in this paper. To make up for the lack of data diversity in the source domain, a generator is created inS2ECNet to simulate the spectral deviation and spatial deviation from the target domain. A causal contribution discriminator is also built inS2ECNet to solve the data bias problem caused by direct feature alignment, which constructs causal contribution vectors from a causal perspective and uses contrastive learning to constrain category labels, extracting “potential” causal invariances from spectral and spatial domains. The cross-scene classification test is completed on the Pavia dataset, the HyRANK dataset, and the Houston dataset, and compared with some advanced multimodal methods. The experimental results demonstrate the effectiveness of the proposed network.

How to fully utilize the rich but complex scattering characteristics in PolSAR data is still a challenge. In this paper, a hierarchical scattering-spatial interaction Transformer (HSSIT) for polarimetric SAR image classification is proposed to effectively combine scattering and spatial characteristics of PolSAR data. The proposed HSSIT adopts a multi-stage hierarchical structure to extract discriminative features. Specifically, spatial feature extraction branch (SFEB) is designed to improve the global information perception ability for spatial features, which combines the advantages of CNN and Transformer to extract local features and capture context dependencies between pixels. A scatter-aware branch (SAB) based on Transformer is proposed to model correlation between polarimetric scattering features. Furthermore, we further propose a cross attention based information exchange module, which aggregates the tokens from two branches to enhance the discrimination of features for land cover classification. Sufficient experiments are carried out on three widely used PolSAR datasets to certify the effectiveness and superiority of our proposed method.

The fusion of hyperspectral imagery (HSI) and light detection and ranging (LiDAR) data is widely used for land cover classification. However, due to different imaging mechanisms, HSI and LiDAR data always present significant image differences, and the dimensions and feature distributions of HSI and LiDAR are highly dissimilar. This makes it challenging to represent and correlate semantic information from multimodal data. Current methods for classifying pixel-by-pixel features, which rely on cascaded or attention-based fusion, cannot effectively use multimodal features. To achieve accurate classification results, extracting and fusing similar high-order semantic information and complementary discriminative information contained in multimodal data is vital. In this paper, we propose a cross-modal semantic enhancement network (CMSE) for multimodal semantic information mining and fusion. Our proposed CMSE framework extracts features from the image on multiple scales, capturing more representative local sparse features with different sizes of convolution kernels. To represent high-level semantic features related to land cover, we establish a Gaussian-weighted matrix and semantically transform the spatial and spectral features of distinct branches. Finally, we build a multi-level residual fusion module to incrementally fuse spectral features from HSI and elevation features from LiDAR. Additionally, we introduce a cross-modal semantically constrained loss to guide multimodal semantic feature alignment. We evaluate our approach on three multimodal remote sensing datasets, namely the Houston2013, Trento, and MUUFL datasets. The experimental results demonstrate that our proposed CMSE model achieves superior performance in terms of accuracy and robustness compared to other related deep networks.

Multitarget domain adaptation (MTDA) presents a formidable challenge in remote sensing image scene classification (RSICS), where the objective is to transfer knowledge from a labeled source domain to several unlabeled target domains. Compared to single-source-single-target domain adaptation (S3TDA), MTDA is inherently more complex due to domain shifts among multiple target domains. Directly merging the unique features of multitarget domains can result in corrupted information and poor classification performance. To address these challenges, we propose a hierarchical feature progressive alignment network (HFPAN) for RSICS in MTDA. First, our method introduces a fine-grained and contextual information extraction (FCIE) network to extract the global-local correlation in remote sensing (RS) images. Second, we construct a hierarchical feature embedding (HFE) framework that maintains hierarchical inter, intra constraints for the extracted features. Finally, we perform an alignment process for the constructed hierarchical features to minimize the differences in MTDA, progressing from coarse to fine granularity. To evaluate the efficacy of our proposed method, we conducted several cross-domain scene classification experiments on five public datasets. These experiments demonstrate the novelty of our approach and its ability to achieve improved classification performance.

In heterogeneous image change detection (HICD), preventing neural networks from distorting critical information is the main challenge of such methods based on deep translation. And most of these methods rely on a priori information to suppress the effects of changed pixels in the translation process, but the accuracy of the prior information will influence the results of translation. In this paper, we propose an end-to-end multi-domain constrained translation network (MDCTNet) for unsupervised HICD. The proposed MDCTNet utilizes an improved generative adversarial network (GAN) to generate target domain images realistically. Furthermore, to retain the content information of the source domain images, MDCTNet leverages contrastive learning to ensure the consistency of adjacent pixel relationships. Meanwhile, it employs high-frequency information consistency which preserves pivotal characteristics. We compare the proposed MDCTNet with state-of-the-art algorithms to verify the efficacy of the proposed technique. The experimental results on five real data sets demonstrate the effectiveness of the proposed method.

The pixel-level classification of multimodal remote sensing images plays a crucial role in the intelligent interpretation of remote sensing data. However, existing methods that mainly focus on feature interaction and fusion fail to address the challenges posed by confounders – brought by sensor imaging bias, limiting their performance. In this paper, we introduce causal inference into intelligent interpretation of remote sensing and propose a new Texture-Aware Causal Feature Extraction Network (TeACFNet) for pixel-level fusion classification. Specifically, we propose a two-stage causal feature extraction framework that helps networks learn more explicit class representations by capturing the causal relationships between multimodal heterogeneous data. In addition, to solve the problem of low-resolution land cover feature representation in remote sensing images, we propose the Refined Statistical Texture Extraction (ReSTE) module. This module integrates the semantics of statistical textures in shallow feature maps through feature refinement, quantization, and encoding. Extensive experiments on two publicly available datasets with different modalities, including Houston 2013 and Berlin datasets, demonstrate the remarkable effectiveness of our proposed method, which reaches a new state-of-the-art.

To improve the performance of deep neural networks, the Mixup method has been proposed to alleviate their memorization issues and sensitivity to adversarial samples. This provides networks with better generalization abilities. The learning principle of Mixup is essentially to train deep neural networks for regularization tasks with a convex combination of the original feature vectors and their labels. However, soft labels are generated directly using the mixing ratio without dealing with the uncertain information generated during the mixing process. Therefore, this paper proposes a new data augmentation method based on Mixup and Dempster-Shafer theory called DS-Mixup, which is a regularizer that can express and deal with the uncertainty caused by ambiguity. This method uses interval numbers to generate mass functions of mixed samples to model the distribution of set-valued random variables; then, ambiguous decision spaces are constructed, and soft labels with single-element subsets and multielement subsets are generated to further improve the delineation of decision boundaries during the training process. In addition, an evidence neural network with DS-Mixup is designed in this paper to accomplish recognition or classification tasks. Experimental results obtained on multimedia datasets, including attribute, image, text and signal data, show that the proposed method achieves more effective data augmentation effects and further improves the performance of deep neural networks.

Measuring the uncertainty of information is a crucial problem in many fields. Recent studies have found a new uncertainty measure for probabilities called “extropy” as a complementary dual function of classical Shannon entropy. In this article, the extropy measure of randomness is generalized to the case of information with epistemic uncertainty by means of a framework of Dempster-Shafer evidence theory. Specifically, a novel measure called plausibility extropy is proposed, which inherits the intriguing properties of original extropy. Moreover, the duality and complementarity between the proposed plausibility extropy and existing plausibility entropy are proved strictly, which constitutes an entropy-extropy combination for mass functions to measure the epistemic uncertainty. In addition, the maximum plausibility extropy is also studied in this article. Through comparing with existing extropy-like measures in Dempster-Shafer evidence theory, the rationality of proposed plausibility extropy is further demonstrated.

The utilization of multi-modal data from multi-sensors (e.g., hyperspectral and light detection ranging (LiDAR) data) to classify ground objects has been an important topic in remote sensing interpretation. However, complex background leads to the difficulty of extracting context relations, at the same time, redundancy and noise among multi-modal data bring great challenges to accurate classification. In this paper, we propose a novel spatial context and de-redundant fusion network (SCDNet) to fuse hyperspectral and LiDAR data for land cover classification. Specifically, a multi-scale attention fusion module is developed in the feature extraction stage, which adaptively fuses global and local information of different scales to obtain a more accurate spatial context. In the feature fusion stage, a fusion module based on gated mechanism is proposed, which can remove the redundant information of multi-mode data and obtain discriminative fusion features. We design a series of comparisons and ablation experiments on the Houston2013 dataset and Trento dataset, and the results demonstrate the effectiveness of the proposed method.

Both infrared and visible images have advantages for object detection, since infrared images can capture thermal characteristics of objects and visible images can provide high spatial resolution and clear texture details of objects. Combining infrared and visible images for object detection has many advantages, but how to fully utilize the inherent characteristics of these two data is still a challenging issue. To address this issue, a deblurring dictionary encoding fusion network (DDFN) is proposed for infrared and visible image object detection. Firstly, a dual-stream feature extraction backbone is structured, which aims to learn features based on the characteristics of different modalities. Then, pooling operations are applied to filter out key information and reduce the complexity of the network. Afterwards, a fuzzy compensation module is proposed, which aims to minimize the information loss of pooling process. Finally, a dictionary encoding fusion module is proposed to robustly excavate potential interactions between infrared and visible images, which can obtain fusion features with aggregating the local information of infrared features and the long-term dependent information of visible features. The proposed DDFN exhibits excellent performance on two benchmark bimodal datasets and shows superior capabilities in object detection of infrared-visible images.

Under the background of the rapid development of remote sensing (RS) technology, multimodal RS image classification has attracted great attention. Considerable research has been devoted to designing more adequate multimodal feature-level fusion networks. However, few have noted that in the process of feature fusion, if the multimodal heterogeneous features are quite different, direct fusion may introduce noise. This greatly affects the classification performance of the network. This letter proposes a shuffle feature enhancement-based fusion network (SFE-FN) for hyperspectral and light detection and ranging (LiDAR) classification, which effectively alleviates the aforementioned problems. Specifically, first, an SFE module is proposed to achieve self-enhancement and mutual enhancement of each modal feature to preliminary reduce the feature difference. Then, a cross-layer and cross-interaction module (CLCI) is designed to further enhance the consistency of features by updating parameters across layers. Finally, the proposed shuffle feature concatenation (SFC) module and the shuffle feature fusion (SFF) module are utilized to adequately merge fewer differentiated features. Experiments on Houston2013 and Trento datasets show that the proposed method is effective.

Few-shot learning (FSL) aims to train a model with limited samples for identifying novel category samples. As for remote sensing images, complex backgrounds may lead to large intra-class differences, and the number of labeled samples is quite smaller than that of large datasets, which both influence the classification performance. To solve these issues, a foreground-background contrastive learning (FBCL) is proposed for few-shot remote sensing image scene classification. Specifically, a foreground-background separation module is proposed to separate features between objects and background with supervised contrastive learning, which aims to improve the ability to distinguish foreground and background regions of remote sensing images. Moreover, a channel weight allocator is proposed to balance features of different dimensions, which can take full advantage of remote sensing image information. Experiments on three remote sensing datasets prove that the proposed few-shot method is able to produce superior classification results than other related approaches.

The existing deep networks have shown excellent performance in remote sensing scene classification (RSSC), which generally requires a large amount of class-balanced training samples. However, deep networks will result in underfitting with imbalanced training samples since they can easily bias toward the majority classes. To address these problems, a multigranularity decoupling network (MGDNet) is proposed for remote sensing image scene classification. To begin with, we design a multigranularity complementary feature representation (MGCFR) method to extract fine-grained features from remote sensing images, which utilizes region-level supervision to guide the attention of the decoupling network. Second, a class-imbalanced pseudolabel selection (CIPS) approach is proposed to evaluate the credibility of unlabeled samples. Finally, the diversity component feature (DCF) loss function is developed to force the local features to be more discriminative. Our model performs satisfactorily on three public datasets: UC Merced (UCM), NWPU-RESISC45, and Aerial Image Dataset (AID). Experimental results show that the proposed model yields superior performance compared with other state-of-the-art methods.

Remote sensing image semantic segmentation (RSISS) remains challenging due to the scarcity of labeled data. Semi-supervised learning can leverage pseudo-labels to enhance the model’s ability to learn from unlabeled data. However, accurately generating pseudo-labels for RSISS remains a significant challenge that severely affects the model’s performance, especially for the edges of different classes. In order to overcome these issues, we propose a semi-supervised semantic segmentation framework for remote sensing images based on edge-aware class activation enhancement (ECAE). Firstly, the baseline network is constructed based on the average teacher model, which separates the training of labeled and unlabeled data using student and teacher networks. Secondly, considering local continuity and global discreteness of object distribution in remote sensing images, the class activation mapping enhancement (CAME) network is designed to predict local areas more remarkably. Finally, the edge-aware network (EAN) is proposed to improve the performance of edge segmentation in remote sensing images. The combination of the CAME with the EAN further heightens the generation of high-confidence pseudo-labels. Experiments were performed on two publicly available remote sensing semantic segmentation datasets, Potsdam and ISPRS Vaihingen, which verify the superiorities of the proposed ECAE model.

Remote sensing image semantic segmentation is a crucial step in the intelligent interpretation of remote sensing. Most of the current approaches are based on the attention mechanism to enhance long-range representations. However, these works ignore the key problem of foreground-background imbalance, and their performances encounter a bottleneck. In this paper, we introduce mask classification into remote sensing image interpretation for the first time, and propose a novel mixed-mask Transformer (MMT) for remote sensing image semantic segmentation. Specifically, we propose a mixed-mask attention mechanism, a simple but effective module, which assists the network to learn more explicit intraclass and interclass correlations by capturing long-range interdependent representations. In addition, a progressive multi-scale learning strategy is proposed to solve the problem of large scale-varied targets in remote sensing images, which integrates semantic and visual representations of different scale targets by efficiently utilizing large scale feature maps in Transformer. Experimental results show that the proposed MMT exceeds the existing alternative approaches and achieves state-of-the-art performance on three semantic segmentation datasets.

Deep learning algorithms that can effectively extract features from different modalities have achieved significant performance in multimodal remote sensing (RS) data classification. However, we actually found that the feature representation of one modality is likely to affect other modalities through parameter back-propagation. Even if multimodal models are superior to their uni-modal counterparts, they are likely to be underutilized. To solve the above issue, a dual-branch dynamic modulation network is proposed for hyperspectral (HS) and light detection and ranging (LiDAR) data classification. Firstly, a novel dynamic multimodal gradient optimization (DMGO) strategy is proposed to control the gradient modulation of each feature extraction branch adaptively. Then, a multimodal bi-directional enhancement (MBE) module is developed to integrate features of different modalities, which aims to enhance the complementarity of HS and LiDAR data. Furthermore, a feature distribution consistency (FDC) loss function is designed to quantify similarities between integrated features and dominant features, which can improve the consistency of features across modalities. Experimental evaluations on Houston2013 and Trento datasets demonstrate that our proposed network exceeds several state-of-the-art multimodal classification methods in terms of fusion classification performance.

Remote sensing image object detection has been widely developed in many applications. Objects in remote sensing data have the characteristic of arbitrary directions, which leads to poor detection performance based on horizontal box detectors. To address this issue, a novel rotated object detection model based on binary smooth encoding and ellipse-like focus loss is proposed in this paper. Firstly, a multi-layer feature fusion network with attention mechanism is developed to extract features of multi-scale objects. Then, an anchor free detection module with binary smooth encoding is proposed, which aims to predict the rotated angles of objects. Moreover, an ellipse-like focus loss is proposed to obtain high-quality bounding boxes drawing near the object center. Experimental results on two public remote sensing datasets verify that the proposed method can yield superior detection performance than other related rotated object detection models.

Few-shot remote sensing image scene classification has gained more attention due to its ability to recognize novel categories with several annotated samples. However, it is a great challenge to extract category characteristics with insufficient labeled samples. To address this issue, an iterative distribution learning network (IDLN) is proposed for the scene classification of few-shot remote sensing images. Specifically, the proposed model is a cyclic iterative architecture, which is composed of three modules to enhance the classification performance. In each iteration, similarity distribution learning module is proposed to calculate feature relations among instances first, then label mapping module is developed as the few-shot classifier based on instructive knowledge, and, finally, the attention-based feature calibration module is proposed to modify features based on label relations, and the calibrated features are imported to the next iteration. Experimental results on two public remote sensing datasets demonstrate that the proposed network is able to achieve superior performance for few-shot remote sensing image scene classification.

Synthetic aperture radar (SAR) images can capture abundant spatial and polarimetric information of land cover objects, and thus polarimetric SAR (PolSAR) image classification has been developed for various applications. Combining the advantages of spatial and polarimetric information simultaneously is of great importance for PolSAR image classification. In this paper, a feature enhanced superpixel hypergraph neural network (FESHNN) is proposed for PolSAR image classification, which aims to take full advantage of spatial features and polarimetric features from PolSAR images. In the proposed model, superpixel hypergraph neural network is constructed for feature representation of superpixels, which aims to obtain spatial correlation and polarimetric correlation in a hypergraph. Then, a feature enhancement module is employed to refine the local features of pixels and the spatial features of superpixels, which aims to enhance the discrimination of feature representation. Experimental results on three PolSAR datasets demonstrate that the proposed method yields superior classification performance compared with other related approaches.

Deep learning has achieved excellent performance in remote-sensing image scene classification, since a large number of datasets with annotations can be applied for training. However, in actual applications, there is just a few annotated samples and a large number of unannotated samples in remote-sensing images, which leads to overfitting of the deep model and affects the performance of scene classification. In order to address these problems, a semi-supervised representation consistency Siamese network (SS-RCSN) is proposed for remote-sensing image scene classification. First, considering intraclass diversity and interclass similarity of remote-sensing images, Involution-generative adversarial network (GAN) is utilized to extract the discriminative features from remote-sensing images via unsupervised learning. Then, Siamese network with a representation consistency loss is proposed for semi-supervised classification, which aims to reduce the differences of labeled and unlabeled data. Experimental results on UC Merced dataset, RESICS-45 dataset, aerial image dataset (AID), and RS dataset demonstrate that our method yields superior classification performance compared with other semi-supervised learning (SSL) methods.

Semi-supervised few-shot learning is developed to train a classifier that can adapt to new tasks with limited labeled data and a fixed quantity of unlabeled data. Most semi-supervised few-shot learning methods select pseudo-labeled data of unlabeled set by task-specific confidence estimation. This work presents a task-unified confidence estimation approach for semi-supervised few-shot learning, named pseudo-loss confidence metric (PLCM). It measures the data credibility by the loss distribution of pseudo-labels, which is synthetical considered multi-tasks. Specifically, pseudo-labeled data of different tasks are mapped to a unified metric space by mean of the pseudo-loss model, making it possible to learn the prior pseudo-loss distribution. Then, confidence of pseudo-labeled data is estimated according to the distribution component confidence of its pseudo-loss. Thus highly reliable pseudo-labeled data are selected to strengthen the classifier. Moreover, to overcome the pseudo-loss distribution shift and improve the effectiveness of classifier, we advance the multi-step training strategy coordinated with the class balance measures of class-apart selection and class weight. Experimental results on four popular benchmark datasets demonstrate that the proposed approach can effectively select pseudo-labeled data and achieve the state-of-the-art performance.

Few-shot classification of hyperspectral image (HSI) has been increasingly abstracted attention due to its superiority of adopting to new HSI classification with only a few labeled data available. However, insufficient feature expression still bothers the improvement of performance. To address this issue, a deep self-attention and mutual-attention few-shot learning (SMA-FSL) method is proposed for HSI few-shot classification. Specifically, a deep 3D convolutional feature embedding network is utilized to extract the spectral-spatial feature at first. Then, self-attention and mutual-attention are used to ally the feature of different samples with same class and expand the class prototypes for more stable feature expression. Finally, the predicted results are obtained by calculating the distance between query set and aligned class prototypes. The experimental results on two well-know HSI datasets demonstrate that the proposed method achieves better performance compared with other related methods.

Remote sensing image scene classification, which requires large amounts of labeled data, plays a critical role in a range of fields.However, in the actual complex environment, the obtained remote sensing images are sometimes unlabeled due to data perturbation and the cost of manual labeling, which limits the training effect and generalization ability. To solve this issue, a semi-supervised siamese network with label fusion is proposed for remote sensing image scene classification. The siamese network is developed to extract features from remote sensing image, where loss function based on the low-entropy principle is constructed to select the unlabeled data as pseudo-label samples. The labeled and pseudo-label samples are mixed to further train the siamese network. The results on UC Merced dataset and WHU-RS19 show that our method is capable to achieve excellent performance compared with other semi-supervised learning methods.

With recent advances in remote sensing, abundant multimodal data are available for applications. However, considering the redundancy and the huge domain differences among multimodal data, how to effectively integrate these data is becoming important and challenging. In this paper, we proposed a triplet attention feature fusion network (TAFFN) for SAR and optical image fusion classification. Specifically, spatial attention module and spectral attention module based on self-attention mechanism are developed to extract spatial and spectral long-range information from the SAR image and optical image respectively, at the same time, cross-attention mechanism is proposed to capture the long-range interactive representation. Triplet attentions are concatenated to further integrate the complementary information of SAR and optical images. Experiments on a SAR and optical multimodal dataset demonstrate that the proposed method can achieve the state-of-the-arts performance.

Few-shot learning in image classification is developed to learn a model that aims to identify unseen classes with only few training samples for each class. Fewer training samples and new tasks of classification make many traditional classification models no longer applicable. In this paper, a novel few-shot learning method named multi-scale metric learning (MSML) is proposed to extract multi-scale features and learn the multi-scale relations between samples for the classification of few-shot learning. In the proposed method, a feature pyramid structure is introduced for multi-scale feature embedding, which aims to combine high-level strong semantic features with low-level but abundant visual features. Then a multi-scale relation generation network (MRGN) is developed for hierarchical metric learning, in which high-level features are corresponding to deeper metric learning while low-level features are corresponding to lighter metric learning. Moreover, a novel loss function named intra-class and inter-class relation loss (IIRL) is proposed to optimize the proposed deep network, which aims to strengthen the correlation between homogeneous groups of samples and weaken the correlation between heterogeneous groups of samples. Experimental results on mini ImageNet and tiered ImageNet demonstrate that the proposed method achieves superior performance in few-shot learning problem.

Z-number is an effective model to describe uncertainty in the real world. Under the condition that uncertainty reasoning is an important issue to process information, how to achieve Z-valuation uncertainty reasoning is a problem. As a Z-number involves both fuzzy and probabilistic uncertainty, main difficulty in the problem to be solved is accomplishing both two uncertainties' reasoning. In this paper, a novel Z-network model and its associated reasoning algorithm are proposed to overcome the difficulty. Structure of the proposed Z-network that contains three basic structures is directed acyclic graph, and this is similarly with Bayesian network (BN). Process of reasoning algorithm involves two parts: first, Bayesian reasoning is applied to establish an optimization model for probabilistic uncertainty reasoning in a Z-number; second, the arithmetic approach of discrete Z-number on if-then rule and maximum entropy approach are proposed for fuzzy uncertainty reasoning. Z-network is essentially an extended model on the basis of BN and properties of a Z-number for Z-valuation uncertainty reasoning. In application, a novel framework of dependence assessment in human reliability analysis is proposed based on Z-network, and a case study demonstrates its effectiveness.

The problem of different characters of heterogeneous synthetic aperture radar (SAR) images leads to poor performances for transfer learning of SAR image classification. To address this issue, a semisupervised model named as deep joint distribution adaptation networks (DJDANs) is proposed for transfer learning from a source SAR image to a different but similar target SAR image, which aims to match the joint probability distributions between the source domain and target domain. In the proposed DJDAN, a marginal distribution adaptation network is developed to map features across the domains into an augmented common feature subspace, which aims to match the marginal probability distributions and unify the dimensions. Then, a conditional distribution adaptation network is proposed to transfer knowledge across the domains, which aims to reduce the discrepancies of the conditional probability distributions and enhance the effectiveness of feature representation. Moreover, one-versus-rest classification is utilized in the proposed framework, which aims to improve the discrimination between the inside and outside class. Experimental results demonstrate the effectiveness of the proposed deep networks.

Cross-scene classification of hyperspectral image (HSI) has been increasingly researched due to its crucial utilization in practical applications. However, cross-scene data generally perform distribution discrepancy, which hampers the transfer learning performance. To address this issue, deep conditional distribution adaptation networks (DCDAN) are proposed for HSI cross-scene classification, which aim to reduce the distribution shift between a source domain and a target domain. The proposed deep network adopts a conditional constraint to match the class conditional distributions across domains, where a great number of training samples from the source domain and a small number of training samples from the target domain are utilized to train the deep model. Cross-scene classification results on two HSIs demonstrate that the proposed network is able to yield superior performance compared with some related methods.

Uncertainty quantification is very important in many applications. As a generalization of Dempster-Shafer theory, the theory of D numbers is a new theoretical framework for uncertainty reasoning. Measuring the uncertainty of knowledge or information represented by D numbers is an unsolved issue in that theory. In this paper, inspired by distance-based uncertainty measures for Dempster-Shafer theory, a total uncertainty measure for a D number is proposed based on its belief intervals. The proposed total uncertainty measure can simultaneously capture the discord, and nonspecificity, and nonexclusiveness involved in D numbers. And some basic properties of this total uncertainty measure, including range, monotonicity, generalized set consistency, are also presented. At last, an illustrative application about feature evaluation is given to verify the effectiveness of the proposed uncertainty measure.

The Dempster-Shafer evidence theory is widely used in many fields of information fusion because of its advantage in handling uncertain information. One of the key issues in this theory is how to make decision based on a basic probability assignment (BPA). Currently, a feasible scheme is transforming a BPA to a distribution of probabilities. However, little attention was paid to the correlation between BPA and probability distribution. In this paper, a novel method about the probability transformation based on a correlation coefficient of belief functions is proposed. The correlation coefficient is a new measurement, which can effectively measure the correlation between BPAs. The proposed method aims at maximizing the correlation coefficient between the given BPA and the transformed probability distribution. On the basis of this idea, the corresponding probability distribution can be obtained and could reflect the original information of the given BPA to the maximum extent. It is valid to consider that the proposed probability transformation method is reasonable and effective. Numerical examples are given to show the effectiveness of the proposed method.

OWA operator is an effective aggregation method in multi-criteria decision making problem. However, in some multi-criteria decision making cases, the criteria satisfactions have some uncertainty, for instance, which is a set of interval values at a series of different levels. For multi-criteria decision making problem, it is necessary to aggregate criteria satisfactions. But the linear ordering of criteria satisfactions is unknown at a specific level in these cases. Therefore, OWA operator cannot be applied to aggregate the satisfactions directly. In this paper, a new method, named as the Interval Value Exceedance Method (IVEM), is proposed. By using the proposed method, the domination relationship of criteria satisfactions for each level can be obtained. Then OWA operator can be used to aggregate these satisfactions based on the domination relationship, even if the linear ordering of satisfaction is unknown.

Axiomatic design (AD) provides a general theory for system and product development. In recent years, the principles of AD have been successfully applied to the decision-making field, and derived a fuzzy AD approach for fuzzy decision-making environment. In this work, the interest is paid on the theoretical developments and applications of AD in the uncertain environment expressed by Dempster–Shafer evidence theory. Based on the concept of belief structure satisfaction to uncertain target values, an evidential AD approach is proposed for decision making by combining the independence axiom and information axiom of AD with the framework of Dempster–Shafer theory. An illustrative example has demonstrated the effectiveness of the proposed approach. This work, on the one hand, has successfully generalized the principles of AD to the Dempster–Shafer uncertain environment; on the other hand, it has presented a successful application of the concept of belief structure satisfaction.

Atanassov's intuitionistic fuzzy set (IFS) is a generalization of a fuzzy set that can express and process uncertainty much better. There are various averaging operators defined for IFSs. In this paper, a new type of operator called an intuitionistic fuzzy entropy weighted power average ggregation operator is proposed. The entropy among IFSs is taken into consideration to determine the weights. What's more, the similarity is considered to measure the support degree between two elements of the IFS. Compared with other classical power average operators, the proposed operator is completely driven by data and fully takes into account the relationship among values. Finally, an illustrative example of multiple attribute group decision making is presented to show that the proposed operator is effective and practical.

Regret decision theory is a classic theory for decision problem. Recently, Yager proposed a generalized regret based decision-making method, which calculates the effective regret associated with an alternative by aggregating this alternative's all regrets across all the possible states of nature. The generalized regret based decision-making method that can be applied in many fields is understandable and effective. However, as Yager pointed, an issue limits the application of this method, that is, the generalized regret based decision-making method is lack of indifference to irrelevant alternatives. In this paper, we analyze the cause of this issue, that is, unreasonable alternatives may change other alternatives' regrets by changing the maximal payoff under the occurrence of a state of nature. Furthermore, a new method based on original model is proposed to reduce the impact of unreasonable alternatives according to a parameter called impact factor defined to measure an alternative's quality. Finally, several numerical examples are illustrated to show this new method's effectiveness.

Dempster-Shafer theory of evidence is widely applied to uncertainty modelling and knowledge reasoning because of its advantages in dealing with uncertain information. But some conditions or requirements, such as exclusiveness hypothesis and completeness constraint, limit the development and application of that theory to a large extend. To overcome the shortcomings and enhance its capability of representing the uncertainty, a novel model, called D numbers, has been proposed recently. However, many key issues, for example how to implement the combination of D numbers, remain unsolved. In the paper, we have explored the combination of D Numbers from a perspective of conflict redistribution, and propose two combination rules being suitable for different situations for the fusion of two D numbers. The proposed combination rules can reduce to the classical Dempster's rule in Dempster-Shafer theory under certain conditions. Numerical examples and discussion about the proposed rules are also given in the paper.

Dempster-Shafer (D-S) evidence theory is widely used for information fusion field. However, one of the main issues of D-S evidence theory is that, when large amount of focal elements in Basic Probability Assignment (BPA) are available, the fusion of BPA requires high computational cost and long computing time. This problem greatly limits its application. In this paper, a novel method for approximating a BPA based on correlation coefficient is present, which can reduce the computational cost of evidence combination effectively. At last, several numerical examples are given to illustrate the superiority of the proposed method.

How to fuse uncertain information in multiple attribute decision making (MADM) efficiently is still an open issue. The power average operation is an effective tool to aggregate interval data. However, existing methods to aggregate interval numbers based on power average operator are relatively complicated. In this paper, a simple and effective support function of interval data is proposed. Then, a novel interval number power average operation operator is presented. Finally, a practical MADM problem is used to show the efficiency of the developed method.