High-resolution Earth Observation technologies present unprecedented opportunities for geospatial analysis, yet traditional 2D aerial-view semantic segmentation remains limited by its inability to model spatial relationships and handle object occlusions. While 3D Aerial-view Segmentation (3DAS) has emerged to address these limitations, existing methods predominantly rely on 2D discriminative models pre-trained on natural scenes. These models struggle to accurately recognize aerial-view imagery, resulting in suboptimal performance due to significant domain discrepancies. This paper introduces ID-Splat, a novel object-centric framework that directly leverages multi-view object identities without discriminative information to enhance 3D semantic understanding. ID-Splat implements a two-stage process: first, Mask-object Tracking combines SAM and Point Tracking to establish robust and consistent object identities across multi-view aerial images; second, Object Integration & Propagation assigns these identities to 3D Gaussian Splatting (3DGS) points, enabling complete 3D segmentation through semantic propagation. Experimental results on the 3D-AS dataset demonstrate that ID-Splat significantly outperforms existing methods, particularly under sparse supervision conditions. ID-Splat also achieves state-of-the-art performance while reducing the need for extensive labeled data by effectively leveraging the inherent 3D structure.

Semi-supervised semantic segmentation (S4) has shown significant promise in reducing the burden of labor-intensive data annotation. However, existing methods mainly rely on pixel-level information, neglecting the strong region consistency inherent in remote sensing images (RSIs), which limits their effectiveness in handling the complex and diverse backgrounds of RSIs. To address this, we propose RegionMatch, a novel approach that leverages unlabeled data from a fresh object-level perspective, which is more tailored to the nature of semantic segmentation. We design the Pixel-Region Synergy Pseudo-Labeling strategy, which explicitly injects object-level contextual information into the S4 pipeline and promotes knowledge collaboration between pixel and region perspectives for generating high-quality pseudo-labels. In addition, we propose the Region Structure-Aware Correlation Consistency, which models object-level relationships by establishing inter-region correlations across images and pixel correlations within regions, providing more effective supervision signals for unlabeled data. Experimental results demonstrate that RegionMatch outperforms state-of-the-art methods on multiple authoritative remote sensing datasets, highlighting its superiority in the RSIs.

Unsupervised anomaly detection in hyperspectral images (HSI), aiming to detect unknown targets from backgrounds, is challenging for earth surface monitoring. However, current studies are hindered by steep computational costs due to the high-dimensional property of HSI and dense sampling-based training paradigm, constraining their rapid deployment. Our key observation is that, during training, not all samples within the same homogeneous area are indispensable, whereas ingenious sampling can provide a powerful substitute for reducing costs. Motivated by this, we propose an Asymmetrical Consensus State Space Model (ACMamba) to significantly reduce computational costs without compromising accuracy. Specifically, we design an asymmetrical anomaly detection paradigm that utilizes region-level instances as an efficient alternative to dense pixel-level samples. In this paradigm, a low-cost Mamba-based module is introduced to discover global contextual attributes of regions that are essential for HSI reconstruction. Additionally, we develop a consensus learning strategy from the optimization perspective to simultaneously facilitate background reconstruction and anomaly compression, further alleviating the negative impact of anomaly reconstruction. Theoretical analysis and extensive experiments across eight benchmarks verify the superiority of ACMamba, demonstrating a faster speed and stronger performance over the state-of-the-art. Code is released at https://github.com/PURE-melo/ACMamba.

As a useful remote sensing (RS) scene interpretation technique, multi-label RS scene classification (RSSC) always attracts researchers’ attention and plays an important role in the RS community. To assign multiple semantic labels to a single RS image according to its complex contents, the existing methods focus on learning the valuable visual features and mining the latent semantic relationships from the RS images. This is a feasible and helpful solution. However, they are often associated with high computational costs due to the widespread use of Transformers. To alleviate this problem, we propose a Mamba-based efficient network based on the newly emerged state space model called MLMamba. In addition to the basic feature extractor (convolutional neural network and language model) and classifier (multiple perceptrons), MLMamba consists of two key components: a pyramid Mamba and a feature-guided semantic modeling (FGSM) Mamba. Pyramid Mamba uses multi-scale scanning to establish global relationships within and across different scales, improving MLMamba’s ability to explore RS images. Under the guidance of the obtained visual features, FGSM Mamba establishes associations between different land covers. Combining these two components can deeply mine local features, multi-scale information, and long-range dependencies from RS images and build semantic relationships between different surface covers. These superiorities guarantee that MLMamba can fully understand the complex contents within RS images and accurately determine which categories exist. Furthermore, the simple and effective structure and linear computational complexity of the state space model ensure that pyramid Mamba and FGSM Mamba will not impose too much computational burden on MLMamba. Extensive experiments counted on three benchmark multi-label RSSC data sets validate the effectiveness of MLMamba. The positive results demonstrate that MLMamba achieves state-of-the-art performance, surpassing existing methods in accuracy, model size, and computational efficiency. Our source codes are available athttps://github.com/TangXu-Group/ multilabelRSSC/tree/main/MLMamba.

Unsupervised hyperspectral change detection (UHCD), detecting subtle changes between bi-temporal images without manual annotations, is an essential but challenging task in the earth observation community. The current modus operandi often performs it in a feature comparison manner, which is limited by variations in imaging conditions. We observe that fully supervised paradigms using limited annotations are capable of overcoming this challenge. Based on this, we introduce a novel Observational Learning Paradigm (OraL) for UHCD by mimicking fully supervised paradigms. OraL comprises two sequential stages: Observation, which designs a spatial-temporal observation strategy (STO) that records the learning consistency of pixels under different training steps and views, to obtain reliable pseudo-labels. Reproduction, which retrains the model with these pseudo-labels and introduces a distribution-aware spectral learning strategy (DSL) to adaptively increase their learning difficulty according to spectral distributions, enhancing the robustness and generalization of the model. Extensive experiments on several public hyperspectral image datasets demonstrate its state-of-the-art performance and pluggability for previous unsupervised methods. Code will be made available.

With remote sensing (RS) images’ resolution increasing, a single scene label cannot adequately represent RS scenes’ contents. Therefore, multilabel RS scene classification (MLRSSC) is gradually attracting the researchers’ attention. Many methods have been proposed recently, and most use deep features or semantic connections to complete MLRSSC. However, they ignore the combination of these two aspects. In addition, the high interclass similarity and low intraclass similarity of RS images limit the robustness of these methods. In this article, we propose a semantic-assisted feature integration network (SFIN) to overcome the above limitations. It contains a dual-scale feature extractor module (DFEM), a local semantic enhance module (LSEM), a cross-scale interactive attention module (CIAM), and a classifier module (CM). DFEM utilizes the convolutional neural networks (CNNs) to extract multiscale features from RS images. LSEM extracts semantic information and establishes their relationships at different scales. CIAM enhances the feature representation by interacting with the clues across different scales. CM completes the prediction of classification (CLA) results. Integrating them into an end-to-end framework, SFIN can discover the diverse and complex land covers hidden in RS images. Furthermore, to ensure the accuracy of explored semantics and enhance the SFIN’s feature extraction ability, we design a semantic supervision (SS) loss and a semantic-based contrastive learning (SB-CL) loss. They are in charge of the correctness and discrimination of the mined semantics. Along with the typical CLA loss, SFIN can be adequately trained. Extensive experiments have been conducted on four MLRSSC datasets, and the positive results demonstrate that SFIN outperforms many existing methods in MLRSSC tasks. Our source codes are available at:https://github.com/TangXu-Group/multilabelRSSC/tree/main/SFIN.

Recently auxiliary cross-scene information has been widely utilized to improve the hypersperctral image classification performance by knowledge transfer. However, recognition of different objects with the same semantic category is difficult when the object types in similar scenes are different or only limited similarity knowledge is provided. In this paper, a multi-modal feature interactive learning (MMFI) method is proposed based on both hyperspectral image modality and textual modality to distinguish similar objects, which enhances the transfer capability by utilizing the semantic prior from the textual modality. First, the adversarial domain mapping (ADM) module is designed to realize cross-domain knowledge transfer across different scenes in an adversarial learning manner. In particular, the noise is simulated as data distribution in different domains through domain mapping and aggregated with source and target domain data, which is then reconstructed and optimized to learn discriminative and conducive information for transfer. Then, the adaptive interactive learning (AIL) module acts on the latent features of the encoder to mine latent associations among the aggregated features and facilitate the expression of consistent features. In addition, few-shot learning with textual embedding enables more powerful semantic priors for few-shot prototypes, making up for insufficient recognition capability in the presence of hyperspectral image modality only. Experimental results on three datasets demonstrate the superiority of our method.

Change detection in remote sensing images aims at revealing interesting changes about the earth surface and has been one of the most important issues in earth observation. In recent years, lots of fully-supervised change detection methods have achieved good performance with the help of deep learning architectures, which rely on large amounts of pixel-level labels. However, obtaining high-quality pixel-level labels is laborious and expensive. To alleviate this problem, we propose a novel weakly-supervised change detection way via consistency instance segmentation called Box2Change, which requires only box-level labels and achieves competitive results to fully-supervised change detection method. Compared with pixel-level label, it is much more efficient to get box-level label, which locates the potential changed area by a rectangle box. There are two key components in the proposed method, the Changed Instance Segmentation (CIS) and the Self-Supervised Consistency Learning (SSCL) in affine space. The former generates multi-scale changed instances, which learns positional information from box-level labels and segments the instance boundaries within a given bounded region. The latter introduces affine transform and employs consistency constraints in a self-supervised manner to increases the robustness to pseudo-change situations caused by light or noise. In experiments, three popular public change detection datasets are tested and both visual and numerical assessment are discussed, where the proposed method exhibits competitive performance to fully-supervised methods and achieves the state-of-the-art results compared with the other weakly-supervised change detection methods.

Change detection (CD) in remote sensing aims to reveal meaningful surface changes and has been flourishing in recent years. Compared with fully-supervised methods based on pixel-level labels, image-level labels are easy to acquire, which reduces manual labor to a large extent. However, image-level labels lack spatial-and-shape information while containing the least semantic information, which poses a great challenge to the weakly-supervised CD task. Motivated by the prior that bi-temporal images have background semantic consistency, we propose Background-Driven and Foreground-Refined (BDFR-Net) to ameliorate the above problem. Specifically, there are two key components in the proposed method: the Background-Driven Reconstruction (BDR) with image-level supervision and the Foreground-Refined Learning (FRL) with affinity learning. The former generates changed regions of foreground and background separation, which activates the foreground from image-level supervision and constrains the foreground by maintaining spatial and semantic consistency in background regions. The latter introduces Complementary Fusion and Label Adaption (CFLA) strategies to further refine the foreground, which can mine complementary information from foreground sequences and suppress false activations. In addition, affinity learning is proposed to stabilize and supervise the above process. Complementary relationships between foreground and background are fully utilized. Tested on two popular CD datasets, the results demonstrate that our proposed BDFR-Net produces completely changed regions with clear boundaries and outperforms state-of-the-art weakly-supervised methods.

Remote sensing (RS) scene classification (RSSC) is a prominent research topic in the RS community. Multilevel feature fusion is an important way of addressing RS scene classification, and many methods have been proposed in recent years. Although they succeed, current methods can still be improved, particularly in distinguishing the contributions of different multilevel features and fully and effectively fusing them. To address the above issues and fully exploit the potential of multilevel features for RS scene classification tasks, we propose a new model named multiscale sparse cross-attention network (MSCN). It not only focuses on the effectiveness of feature learning but also emphasizes the rationality of feature fusion. In detail, MSCN first extracts multilevel features using a pre-trained ResNet50. Also, these features are divided into high- and low-level features according to the clues they involved. Then, a multiscale sparse cross-attention (MSC) module is developed to cross-fuse the high-level feature with various low-level features, thereby effectively mining helpful information from multilevel features. In the fusion process, MSC not only explores the multiscale messages in RS scenes but also mitigates the negative impact of irrelevant information by employing sparse operations. Third, a group convolutional block attention module (CBAM) enhancer (GCE) is presented to enhance the representation of classification features. GCE detects local salient information within classification features using grouped CBAM and further enhances crucial details by readjusting the CBAM attention weights. This way, the classification features’ discrimination can be improved. We conducted extensive experiments on three public RS scene classification datasets. The exceptional experimental results indicate that our proposed MSCN achieves superior classification accuracy, surpassing many existing methods. Our source codes are available at https://github.com/TangXu-Group/Remote-Sensing-Images-Classification/tree/main/MSCN.

Hyperspectral image (HSI) classification is crucial in the remote sensing (RS) community. In recent years, Transformers have been popular in this field due to their global information modeling capabilities. However, the quadratic complexity limits their performance under limited computational resources. Fortunately, a selective structured state space model named Mamba emerges. Like Transformer, it is good at modeling the long-distance relationships hidden in the pending data. Unlike Transformer, its complexity remains at a linear level. Therefore, a growing number of studies have been proposed to explore the usefulness of Mamba in HSI classification. Nevertheless, most of them only apply Mamba to HSIs directly but do not consider the inherent characteristics of HSIs properly. To exploit the potential of Mamba in HSI classification deeply, this paper presents a new spatial-spectral complementary Mamba with a spatial spiral scan named SpiralMamba. It mainly encloses three main components: a spatial Mamba encoder (SpaME), a spectral Mamba encoder (SpeME), and a spatial-spectral complementary fusion module (SSCFM). SpaME focuses on understanding the spatial context within HSIs. To this end, instead of the common scanning, a spatial spiral scan strategy is introduced to address the sequence transformation of non-causal HSIs. SpeME aims to comprehensively extract valuable spectral features from HSIs. To achieve this goal, besides developing a spectral bidirectional scan strategy, a multilayer convolution (MLC) is also incorporated to capture local variations within spectral tokens. SSCFM concentrates on building the complex connections between spatial and spectral features and fusing them. For this purpose, a relationship learning block (RLB) and a threshold enhancement mechanism (TEM) are developed. Positive experimental results counted on three public HSI datasets demonstrate the effectiveness of SpiralMamba. Our source codes are available at https://github.com/TangXu-Group/Hyperspectral-Images-Classification/tree/main/SpiralMamba.

Although modern Remote Sensing Object Detection (RSOD) methods have achieved advanced performance, they heavily rely on a large amount of annotated data. This paper explores semi-supervised RSOD to mitigate annotation costs, leveraging recent extensive research in generic Semi-Supervised Object Detection (SSOD) based on the self-training paradigm. Current SSOD methods encounter challenges in adapting to remote sensing images due to the complexity and variability of RSIs. Two key issues remain underexplored: the noise in pseudo-labels caused by model instability and the difficulty in distinguishing similar categories. This paper introduces the Temporal-Feedback Self-Training (TST) framework, a novel approach to tackle these challenges in semi-supervised RSOD. TST consists of two components: Temporal Consistency Based Pseudo-labels Certainty Estimation (TCE) and Temporal Self-Feedback Feature Refinement (TSF). TCE addresses pseudo-label noise during training by evaluating the stability of pseudo-label classification and localization over time series to assess the quality of pseudo-labels. On the other hand, TSF enhances pseudo-label quality by dynamically identifying the models confusing categories as feedback for feature refinement. Both components facilitate the progression of the self-training-based RSOD during training. We conducted extensive experiments on two challenging public datasets, DOTA and DIOR. The results demonstrate that the proposed TST and TCE components significantly improve the baseline models performance, surpassing the state-of-the-art generic SSOD method. This suggests that our approach is more effective than generic SSOD methods in addressing the challenges posed by remote sensing images.

The progress of brain cognition and learning mechanisms has provided new inspiration for the next generation of artificial intelligence (AI) and provided the biological basis for the establishment of new models and methods. Brain science can effectively improve the intelligence of existing models and systems. Compared with other reviews, this article provides a comprehensive review of brain-inspired deep learning algorithms for learning, perception, and cognition from microscopic, mesoscopic, macroscopic, and super-macroscopic perspectives. First, this article introduces the brain cognition mechanism. Then, it summarizes the existing studies on brain-inspired learning and modeling from the perspectives of neural structure, cognitive module, learning mechanism, and behavioral characteristics. Next, this article introduces the potential learning directions of brain-inspired learning from four aspects: perception, cognition, understanding, and decision-making. Finally, the top-ten open problems that brain-inspired learning, perception, and cognition currently face are summarized, and the next generation of AI technology has been prospected. This work intends to provide a quick overview of the research on brain-inspired AI algorithms and to motivate future research by illuminating the latest developments in brain science.

The cross-modal remote sensing image-text retrieval (CMRSITR) is a lively research topic in the remote sensing (RS) community. Benefiting from the large pretrained image-text models, many successful CMRSITR methods have been proposed in recent years. Although their performance is attractive, there are still some challenges. First, fine-tuning large pretrained models requires a significant amount of computational resources. Second, most large models are pretrained by natural images, which reduces their effectiveness in processing RS images. To tackle these challenges, we propose a new CMRSITR network named context and uncertainty-aware prompt (CUP). First, prompt tuning theory is introduced into CUP to eliminate the burden of optimization resources. By training the prompt tokens rather than all parameters, the large model's knowledge can be transferred to CMRSITR tasks with small trainable parameters. Second, considering the differences between natural-image-based prior clues and RS images, apart from adopting the free-prompt tokens, we develop a prompt generation module (PGM) to produce the RS-oriented prompt tokens. The specific prompt tokens are rich in object-level messages of RS images, which help CUP narrow the gaps between natural large models and RS images. Third, we further design an uncertainty estimation module (UEM) to whittle down the uncertainties caused by the model and data. This way, can not only the semantic misalignment and intraclass diversity imbalance problems be mitigated but also the RS clues can be deeply explored. Competitive experimental results counted on three public benchmark datasets demonstrate that our CUP can achieve competitive performance in the CMRSITR task compared with many existing methods. Our source codes are available at: https://github.com/TangXu-Group/Cross-modal-remote-sensing-image-and-text-retrieval-models/tree/main/CUP.

Weakly supervised object detection (WSOD) and semantic segmentation with image-level annotations have attracted extensive attention due to their high label efficiency. Multiple instance learning (MIL) offers a feasible solution for the two tasks by treating each image as a bag with a series of instances (object regions or pixels) and identifying foreground instances that contribute to bag classification. However, conventional MIL paradigms often suffer from issues, e.g., discriminative instance domination and missing instances. In this article, we observe that negative instances usually contain valuable deterministic information, which is the key to solving the two issues. Motivated by this, we propose a novel MIL paradigm based on negative deterministic information (NDI), termed NDI-MIL, which is based on two core designs with a progressive relation: NDI collection and negative contrastive learning (NCL). In NDI collection, we identify and distill NDI from negative instances online by a dynamic feature bank. The collected NDI is then utilized in a NCL mechanism to locate and punish those discriminative regions, by which the discriminative instance domination and missing instances issues are effectively addressed, leading to improved object- and pixel-level localization accuracy and completeness. In addition, we design an NDI-guided instance selection (NGIS) strategy to further enhance the systematic performance. Experimental results on several public benchmarks, including PASCAL VOC 2007, PASCAL VOC 2012, and MS COCO, show that our method achieves satisfactory performance. The code is available at: https://github.com/GC-WSL/NDI.

Remote sensing visual question answering (RSVQA) aims to answer the questions based on the content in remote sensing (RS) images. Due to the complexity of RS images, it is challenging to focus on regions relevant to the questions in the RS images. To this end, we propose a channel-selective multi-scale cross-attention (CSCa) model for RSVQA tasks. Specifically, we design a text-driven multi-scale feature extractor to extract question-related features in RS images. To obtain the cross-attention map in this extractor, we design a novel channel selection mechanism to capture more accurate question-related regions in RS images and develop a channel-wise contrastive learning task to align the semantics between image and text features. We set up experiments on RSVQA-LR and RSIVQA datasets. Experiment results show that our CSCa achieves excellent performance.

Vision Transformer (ViT) has achieved great success in the field of computer vision since it was proposed, and there have been many works applying ViT based models to remote sensing scene classification (RSSC) tasks. The proposal of Pyramid Vision Transformer (PVT) greatly reduces the calculation amount of the ViT while maintaining accuracy. But PVT did not utilize multi-scale information in remote sensing (RS) scenes, which is crucial for RSSC. This paper proposes a multi-scale sparse transformer (MST) based on PVT. MST enables the network to learn multi-scale representations of RS scenes through spatial reduction implementations at different scales. In addition, we employ sparse operations to adaptively guide the model’s attention towards semantically relevant regions during self-attention computation, thereby reducing interference from semantically irrelevant areas. Experiments conducted on the UCM and AID datasets demonstrate the outstanding performance of the proposed MST.

In remote sensing (RS), Few-Shot Novel View Synthesis (FS-NVS) focuses on creating images of unobserved viewpoints using limited training images. Recently, 3D Gaussian Splatting (3DGS) has drawn scholars’ attention by its increasing rendering speeds and providing an explicit neural representation for 3D scenes. However, 3DGS tends to overfit limited training data. To tackle this challenge, we propose a Pseudoview Regularized 3DGS (PR3DGS) FSNVS method for RS scenarios. Our PR3DGS method introduces a pseudo-views regularization module to discriminate synthetic RS images generated from training- or pseudo-viewpoints. Therefore, our PR3DGS method can effectively mitigate overfitting in seen views and enhance the model’s capability to generate more realistic RS images from novel viewpoints. Besides, the excellent experimental results on the LEVIR-NVS dataset demonstrate the effectiveness of our method in RS FSNVS.

In recent years, as a very important part of remote sensing interpretation, change detection (CD) has developed rapidly with deep learning and remote sensing interpretation. However, most of the existing CD methods focus on how to extract the spatial features of the bi-temporal image pairs, ignore the importance of the temporal features. To solve this problem, a new time-guided network (TG-Net) using temporal information to guide feature extraction is proposed in this paper. In TG-Net, we use a newly proposed time-guided feature fusion (TGF2) block that uses temporal information to guide spatial feature fusion to extract temporal and spatial information comprehensively. We conducted experiments on two publicly available remote sensing datasets, LEVIR-CD and WHU, and compared them with three common CD methods.

Benefiting from the thousands of continuous band information in hyperspectral images (HSIs), the task of HSI classification has become an indispensable part of the field of remote sensing. With the development of deep learning, deep learning techniques such as convolutional neural networks have been widely introduced into HSI classification research. However, most of these methods do not fully consider the potential relationship between the central pixel and surrounding neighborhoods. Therefore, we introduce a novel center mask self-attention network (CMSAN) to enable the model to effectively capture the association between the central pixel and its neighbors for better feature extraction. We conduct experiments on two publicly available HSI datasets. The positive results on both datasets fully demonstrate the effectiveness of our proposed method.

Remote sensing (RS) image change detection (CD) based on deep learning (DL), has received increasing attention recently. However, the general independent learning of bi-temporal images ignores the relationship between them, falling short in learning of the change information. In this paper, a Triple-Double (TD) framework with ability of conjoint cross-attention modeling and joint feature calibrating is proposed for CD. Specifically, the TD framework composed of Triple-branch encoder and Double-branch decoder is constructed to extract diverse features and acquire changed maps with the guidance of original edge cues. To enhance the perception of the connection between the bi-temporal features, the multi-scale difference guidance (MDG) module and conjoint cross-attention (CCA) module are designed for the dual-branch encoder, wherein the CCA introduces a novel and efficient rule for modeling the affinity in spatial and channel dimension simultaneously. Furthermore, a joint feature calibration (JFC) module is introduced to enhance the expression of feature diversity in the joint features within the single-branch encoder. Experimental results on three public datasets demonstrate the superiority of the proposed method compared to the state-of-the-art (SOTA) methods.

Change detection (CD) in remote sensing images aims at revealing earth surface changes between co-registered bitemporal images. A common way to reveal changed areas is to directly mix bitemporal features and generate CD results through supervised learning. However, a certain change usually corresponds to a real object in either of the two images, which exhibits coarse/fine shape in different scales. Therefore, a coarser-to-finer method called candidate-aware and change-guided network (CACG-Net) is proposed to effectively detect changes, where candidate objects are revealed and associated with interesting changes. Specifically, there are three key components. They are multistage change decoder (MCD), candidate-aware learning (CAL) and change guidance module (CGM). MCD reveals the most important changed objects in the coarse shape from the basic features extracted by the backbone (ResNet-18). To capture changes of interest, CAL is designed to select candidate objects in each temporal image, where a segmenter is utilized with variant change-losses. CGM intends to enrich the change details step-by-step through combining coarser change results and finer features, so that changed objects are gradually revealed in a coarser-to-finer way. Furthermore, deep supervision is employed throughout the layers of CACG-Net in the training procedure, which mitigates the learning difficulty in both deep and shallow layers. Test results on four popular datasets indicate that the proposed method outperforms several state-of-the-art CD algorithms in terms of accuracy and efficiency.

With the flourishing of remote sensing (RS) platform techniques, very high-resolution (VHR) images have become more and more popular in recent years, which benefit the task of semantic segmentation but bring new challenges as well. Small objects, such as cars and trees, only occupy a few pixels in VHR images and are usually hard to segment. Moreover, the overlap problem about similar ground objects, such as low vegetation and trees, always results in underperformance. In this article, a content-guided and class-oriented network (CGCO-Net) for VHR image semantic segmentation is proposed to tackle this problem. Specifically, an adaptive content-guided fusion (ACGF) module with deformable convolution is introduced to capture long-distance dependencies and spatial aggregation effectively. With the guidance of the high-level features, the semantic content knowledge is gradually aggregated into low-level features and the details of the original features could be preserved. In addition, a multiscale channel alignment module is introduced into the encoder–decoder structure to further extract the long-range context information and reduce the calculation consumption. In order to improve the ability of pixel-level classification, a class-oriented representation learning (CORL) way is designed with transformer blocks by class embedding and deep supervision, which gradually enhance the discrimination and benefit the final segmentation. Furthermore, a weighted loss function and a threshold optimization strategy are employed to alleviate the sample imbalance problem. Tested on three public datasets and compared with several state-of-the-art methods, the proposed CGCO-net achieves good performance in both qualitative and quantitative analysis.

Change detection (CD) is one of the important research topics in remote sensing (RS) image processing. Recently, convolutional neural networks (CNNs) have dominated the RSCD community. Many successful CNN-based models have been proposed, and they achieved cracking performance. Nevertheless, influenced by the limited receptive field, the CNN-based models are not good at capturing long-distance context dependencies within RS images, negatively impacting their performance. With the appearance of the visual transformer, the above problems have been mitigated. However, the high time costs of the transformer-based models limit their applicability. In addition, previous CD networks (whether CNN-based or transform-based) do not pay attention to the edges of changed areas, reducing the quality of change maps. To overcome the shortcomings discussed above, we propose a new CD method named edge-assisted adaptive transformer detector (EATDer). EATDer consists of a Siamese encoder and an edge-aware decoder. Each branch in the Siamese encoder encloses three self-adaption vision transformer (SAVT) blocks, which aim to capture the local and global information within RS images. Also, two branches are connected by full-range fusion modules (FRFMs), which focus on mining the temporal clues among bi-temporal RS images and pointing out the changed/unchanged messages. The edge-aware decoder first integrates the multiscale features obtained by the encoder using a restoring block. Then, it enhances the combined features by a refining block. Finally, based on the refined features, both the change and edge detection results can be produced. Along with a joint loss function, we can get high-quality change maps in which the changed areas are correct and have clear and smooth edges. The usefulness of our EATDer is validated by extensive experiments conducted on three popular RSCD datasets. Our source codes are available athttps://github.com/TangXu-Group/Remote-Sensing-Image-Change-Detection/tree/main/EATDer

Hyperspectral image (HSI) classification is a hot topic in remote sensing. A large number of studies have been proposed and achieved excellent performance. Most of them rely on accurate annotations. However, this requirement cannot always be met. Due to the complex contents within HSIs and the uncontrollable external interference factors, incorrect labels are inevitable. Thus, the study of noisy HSI classification is boomed. Some attempts have been made, and their central ideas are to filter the noisy samples from the training set. Although feasible, this would result in information loss, i.e., the contents covered by the removed samples are ignored. Besides, the characteristics of HSIs are not fully considered in many models. To overcome the above limitations, we develop a spatial pooling transformer network (SPTNet) and a noise-tolerant learning algorithm in this paper. SPTNet first uses a spectral feature extraction (SFE) module to capture the rich spectral information from HSI patches. Then, three spatial pooling transformers (SPTs) are constructed and stacked to explore the spatial knowledge and depress confusing clues caused by the HSI patch division. Finally, a standard transformer encoder is used to enhance the obtained spectral-spatial features for the downstream classification. To use SPTNet to handle noisy HSI classification, the noise-tolerant learning algorithm is designed. It encloses two parts, i.e., a data partition scheme and a label-independent similarity regularization. The data partition scheme divides the training data into clean and noisy sets. Then, the clean samples are used to train SPTNet with the classification loss function. At the same time, similarity regularization helps SPTNet to comprehensively understand HSIs by analyzing the resemblance between clean and noisy samples. Integrating two parts into a co-training framework, SPTNets can be trained under a noisy scenario. Four popular HSI datasets are selected to testify to our methods. The positive results demonstrate that the combination of SPTNet and the noise-tolerant learning algorithm is helpful to the noisy HSI classification. Our source codes are available at https://github.com/TangXu-Group/Hyperspectral-Images-Classification/tree/main/SPTNet-NTLA.

Remote sensing (RS) image-text retrieval (RSITR) aims to retrieve relevant texts (RS images) based on the content of a given RS image (text). Existing methods are used to employing the convolutional neural network (CNN) and recurrent neural network (RNN) as encoders to learn visual and textual features for retrieval. Although feasible, the global information hidden in different data does not receive the attention it deserves. To mitigate this problem, transformers have been introduced. Nevertheless, the complexity of RS images present challenges in directly introducing Transformer-based architectures to multimodal learning in RS scenes, particularly in visual feature extraction and cross-modal interaction. In addition, the textual captions are always simpler than the complex RS images, leading to a semantic description appearing in different images. This typical false-negative (FN) sample problem increases the difficulty of RSITR tasks. To address the above limitations, we propose a new RSITR model named prior-experience-based RS vision-language (PERSVL). First, the specific visual and text encoders are used to extract features from RS images and texts. Also, a high-level feature complement (HFC) module is developed based on the self-attention mechanism (SAM) for the visual encoder to explore the complex contents from RS images fully. Second, a dual-branch multimodal fusion encoder (DBMFE) is designed to complete the cross-modal learning. It comprises a dual-branch multimodal interaction (DBMI) module and a branch fusion module. DBMI is designed to fully explore the relationships between different modalities, enriching visual and textual features. The branch fusion module integrates the cross-modal features and utilizes a classification head to generate matching scores for retrieval. Finally, a learning from prior experiences (LPEs) module is designed to reduce the influence of FN samples by analyzing the historical data produced in the model training process. Experiments are conducted on three popular datasets, and the positive results show that our PERSVL model achieves superior performance compared with previous methods. By integrating the advantages of natural language and RS images, our PERSVL can be applied in various applications, such as environmental monitoring, disaster evaluation, and urban planning. Our source codes are available at:https://github.com/TangXu-Group/Cross-modal-remote-sensing-image-and-text-retrieval-models/tree/main/PERSVL.

Joint classification of hyperspectral image (HSI) and light detection and ranging (LiDAR) can simultaneously utilize rich spectral information and elevation information and has become a hot research topic in remote sensing (RS). Although many works have been proposed for this task, their performance cannot reach what we expected due to inadequate cross-modal feature learning and simple feature fusion. This article proposes a multiple information collaborative fusion network (MICF-Net) to overcome those limitations, which aims to leverage the essentially consistent spatial relationships and high-level semantic information in multimodal data to guide the extraction of multimodal fusion features. Specifically, MICF-Net first uses a simple two-branch convolutional neural network (CNN) for preliminary feature extraction. Then, a dual-branch cross-modal attention fusion transformer (CMAFT) is developed to mine global contextual content. By fusing the attention maps of two modalities and limiting their similarity, CMAFT can retain modality-specific information while achieving information interaction based on spatial relationships. Next, an adaptive mask modulation (AMM) module is designed to dynamically balance the learning rate of each modality to ensure the effectiveness of the features of all modalities. Finally, to mine the complementary information of HSI and LiDAR data, a semantic-guided feature fusion (SGFF) module is introduced. It achieves mutual guided learning by exchanging semantic information between two modalities. Positive experimental results counted on three popular HSI and LiDAR datasets demonstrate the effectiveness of the proposed MICF-Net. Our source codes are available athttps://github.com/TangXu-Group/Hyperspectral-Images-Classification/tree/main/MICF-Net.

Semi-supervised learning aims to exploit the potential of unlabeled data to enhance model performance, which makes it suitable for addressing the challenge of limited labeled data. As a popular technology, pseudo-label is widely applied in many semi-supervised remote sensing (RS) change detection methods. However, when facing limited labeled data, abundant low-quality pseudo-labels from a poorly-performing model hinder the effective enhancement of model performance. To address this issue, we propose a novel semi-supervised strategy, named ensemble cross pseudo supervision (ECPS). The utilization of ensemble learning to merge outputs from several change detection models enhances pseudo-label quality, leading to more accurate change information and a significant boost in model performance, even with limited labeled data. In this method, adopting crosswise supervision ensures that no additional inference costs caused by ensemble learning are consumed. This provides both high efficiency and effectiveness for identifying land-cover changes. On the other hand, a simple yet effective ensemble strategy is proposed, which allows to manually adjust the model’s tendency towards higher precision or recall for satisfying practical requirements. We conduct extensive experiments on four public RS change detection datasets, and the promising results demonstrate the superiority of the proposed method across various numbers of labeled samples. Our source codes are available at https://github.com/TangXu-Group/ECPS.

As an important topic in the remote sensing (RS) image processing community, change detection has attracted much attention from researchers, which aims to distinguish land-cover changes in a geographic position. This is a challenging task because the visual representations of land cover captured from RS images at different periods would vary widely and considerably, resulting in significant differences in feature representations. To alleviate this problem, many existing deep-based methods employ the parameter-shared strategy to map RS images into a common feature space for detecting the changes. Although they are feasible, the simple and single visual information learned by deep models is still not sophisticated enough for satisfactory results. To address this problem, we propose a forward dictionary learning (DL) model named forward DL detector (FDLdet) in this article. Besides the common visual features, our FDLdet takes into account the essential information, e.g., element composition and land-cover category, for change detection. FDLdet consists of a feature extractor, a coefficient generator, and a deep dictionary. Specifically, first, the feature extractor is used to extract shared deep features from RS images. Second, the coefficient generator transforms these deep features into word coefficients. Third, words within the deep dictionary are combined by word coefficients to generate the dictionary features with essential information. Finally, the dictionary features are used instead of deep features to detect land-cover changes. Extensive experiments are conducted on two public large-scale datasets, i.e., season-varying change detection (SVCD), Sun Yat-sen University change detection (SYSU-CD), and LEVIR change detection (LEVIR-CD). Experimental results demonstrate the effectiveness of the proposed FDLdet. Our source codes are available athttps://github.com/TangXu-Group/FDLdet.

Multilabel classification in remote sensing (RS) images aims to correctly predict multiple object labels in an RS image with the primary challenge of mining correlations among multiple labels. In this context, we argue that a scene can be treated as a high-level depiction of the interactions among multiple interconnected objects within the image. However, hierarchical relationships between the scene and local objects are often neglected in other state-of-the-art approaches. In this article, we consider multilabel classification as a global-to-local prediction process, whereas the scene of an image is first identified, followed by recognition of local objects in the image. To achieve this, we propose a novel hierarchical knowledge graph (HKG)-based framework for multilabel classification in RS images (ML-HKG). Specifically, we first construct a hierarchical KG to depict label correlations between scenes and objects and represent the hierarchical knowledge as interrelated scene- and object-level label embeddings. Subsequently, we generate a scene-aware enhanced feature map by recognizing scene categories in an image under the guidance of scene-level knowledge embeddings. Afterward, object-level embeddings are used to derive category-specific visual representations for final multilabel prediction. Extensive experiments on the UCM and AID datasets demonstrate the effectiveness of our framework.

With the rapid advances in deep learning techniques, remote sensing object detection has achieved remarkable achievements in recent years. However, tiny object detection remains unsatisfactory and suffers from two main drawbacks, including (1) the high sensitivity of IoU for location deviation in tiny objects and (2) the poor-quality feature representations of tiny objects. To address the aforementioned problems, we propose a Multi-stage Enhancement Network (MENet) that achieves the instance-level and feature-level enhancement of tiny objects from different stages of the detector. Since the IoU-based label assignment drastically deteriorates the positive samples for tiny objects, we first propose a Central Region-based (CR) label assignment to substitute it in the Region Proposal Network (RPN). The CR label assignment regards the anchors that fall into the central region of ground-truth boxes as positive samples, which provides more positive samples for tiny objects. Then, we design a Gated Context Aggregation (GCA) module that selectively aggregates valuable context information to enhance the feature representation of tiny objects. Additionally, we devise a positive RoI feature (pRoI) generator in the Region Convolutional Neural Network (R-CNN) to generate a rich diversity of high-quality positive RoI features for tiny objects. We conduct extensive experiments on AI-TOD and SODA-A datasets, and the results demonstrate the effectiveness of our proposed method.

In recent years, high-resolution (HR) remote sensing images (RSIs) segmentation has received growing attention. The huge number of pixels poses a challenge to the semantic segmentation algorithm, which is limited by the storage of GPUs, so the current methods for processing HR RSIs are categorized into two main categories, i.e., global methods and local methods. The former downsamples the original image and loses a lot of feature details. The latter crops the original image and fails to obtain global contextual information. Both types of methods lead to limited segmentation accuracy. In this article, we propose an end-to-end framework, called global injection network (GINet), which explores two levels of feature distribution and feature relationship to achieve tradeoff between global context and local details. In concrete terms, we propose the global-guided normalization (GGN) module, which injects global context information into local branch and modulates local features using global features to enhance the global perception of local branch. In addition, to constrain the spatial consistency of two branches, inspired by the knowledge distillation technique, we propose local affinity distillation (LAD) loss, which distills the relations in local features into global features to keep the similarity of the relationships corresponding to patches in the two branches. The comprehensive experimental results on three large-scale land-cover classification datasets, DeepGlobe ($2448 \times 2448$), Inria Aerial ($5000 \times 5000$), and GID-15 ($7200 \times 6800$), confirm the effectiveness and superiority of our method in HR semantic segmentation tasks.

In recent years, convolutional neural networks (CNNs)-based methods achieve cracking performance on hyperspectral image (HSI) classification tasks, due to its hierarchical structure and strong nonlinear fitting capacity. Most of them, however, are supervised approaches that need a large number of labeled data to train them. Conventional convolution kernels are fixed shape of rectangular with fixed sizes, which are good at capturing short-range relations between pixels within HSIs but ignore the long-range context within HSIs, limiting their performance. To overcome the limitations mentioned above, we present a dynamic multiscale graph convolutional network (GCN) classifier (DMSGer). DMSGer first constructs a relatively small graph at region-level based on a superpixel segmentation algorithm and metric-learning. A dynamic pixel-level feature update strategy is then applied to the region-level adjacency matrix, which can help DMSGer learn the pixel representation dynamically. Finally, to deeply understand the complex contents within HSIs, our model is expanded into a multiscale version. On the one hand, by introducing graph learning theory, DMSGer accomplishes HSI classification tasks in a semi-supervised manner, relieving the pressure of collecting abundant labeled samples. Superpixels are generally in irregular shapes and sizes which can group only similar pixels in a neighborhood. On the other hand, based on the proposed dynamic-GCN, the pixel-level and region-level information can be captured simultaneously in one graph convolution layer such that the classification results can be improved. Also, due to the proper multiscale expansion, more helpful information can be captured from HSIs. Extensive experiments were conducted on four public HSIs, and the promising results illustrate that our DMSGer is robust in classifying HSIs. Our source codes are available at https://github.com/TangXu-Group/DMSGer.

Remote sensing change detection (RSCD), with a view to probing surface changes between bi-temporal images, makes a spurt of progress with the continuous innovation of deep learning. However, the extraction of multi-scale features and the detection of small domain of variation as well as the detail information in RSCD task still has large development space. Besides, current existing methods mostly focus on learning regional information but pay less regard to boundary identification, which leads to inaccurate detection results. Therefore, a spatial-preserving and edge-orienting high-resolution network is proposed to address the problems. In the overall architecture, a dual-branch encoder consists of a pyramid feature extracted branch and an enhanced HR network branch is designed to extract muti-scale bi-temporal features and small change objectives, while two edge-orienting modules (EOM) are embedded in order to utilize edge prior knowledge for further improving the accuracy of change detection. Moreover, spatial-preserving module (SPM) based on the self-attention calculation in spatial dimension is applied in the pyramid part to alleviate the poor location information of the high-level features. The experimental results demonstrate that the proposed network outperforms the cited state-of-the-art methods on LEVIR change detection datasets (LEVIR-CD).

Change detection is developed to automatically identify semantic changes between remote sensing (RS) images captured at different points of time in a specific geographic location. Due to the limited availability of annotated data and the high complexity of the change detection problem, the performance of change detection models can not meet our expectations. To address this issue, many methods are proposed to solve this issue. However, ignoring the characteristics of the change detection task limits their performance. Therefore, we introduce a novel exchange data enhancement method (EDEM) strategy to generate image pairs to help the neural network to capture the temporal consistency in the data. We evaluate the proposed approach on two publicly available datasets and compare it with several state-of-the-art methods. The experimental results demonstrate that our proposed approach can effectively improve the performance of change detection models, achieving state-of-the-art performance on both datasets.

There is a small sample problem in hyperspectral image (HSI) classification task due to the difficulty of labeling samples. It is generally solved using a combination of few-shot learning and cross-domain method. In the paper, we propose a domain-specific and domain-common feature enhancement method for cross-domain few-shot HSI classification. It consists of a domain adaptation module and a feature enhancement module. The former is used to learn domain-specific features of both domains from the beginning of the network, and the latter is used to reduce domain differences by learning domain-common features through feature enhancement. The experimental results indicate that our proposed method performs better than the advanced classification methods.

The domain shift between the source and target domains limits the performance of traditional convolutional neural networks (CNNs) for feature extraction in remote sensing tasks. We propose an image translation network that uses generative adversarial networks (GANs) to transfer spectral distributions from training to test data, enhancing cross-domain semantic segmentation. Our approach fine-tunes the DeepLab-V3 framework on synthetic training data generated by the proposed network. Experimental results show improved performance in cross-domain semantic segmentation tasks for remote sensing images.

Few-shot remote sensing scene classification (FSRSSC) aims to make the model quickly adapt to new scenes with a small amount of annotation data. The large intra-class variance and high inter-class similarity in remote sensing (RS) scenes make this task more challenging. To this end, we propose a multi-scale interaction prototypical network, which pays attention to capturing multi-scale information of images during model learning, and then generates a prototype representation of mixed query information through a feature interaction module, thereby enhancing the rapid learning ability of the model, so as to reducing of the within-class and between-class variance ratio in RS scenes. The positive experimental results on UC-Merced and NWPU datasets demonstrate the effectiveness of our model in FSRSSC.

Synthetic aperture radar (SAR) image change detection is a hot but challenging task due to SAR images’ complex contents and inherent speckle noises. The expected change detection methods should reduce the influence of speckle noises, obtain the discriminative feature representations, and generate accurate change maps simultaneously. To these ends, we propose a new SAR image change detection method named feature fusion of information transfer network (FFITN). First, we develop a hybrid convolution block to depress the speckle noise impacts and explore the valuable information from SAR images. Thus, the feature extraction module (FEM) is constructed to obtain the multi-level features. Then, an information transfer module (ITM) is proposed to capture the salient regions from various aspects. Also, the salient knowledge is transferred among features at different levels to enhance their discrimination. Next, a self-attention-based feature fusion module (SAFFM) is introduced to fuse various features. Finally, a change map generation module (CMGM) with the clustering algorithm and specific loss functions is designed to produce the pseudo labels and change maps. Experimental results on three public SAR data sets demonstrate the model’s effectiveness. Our source codes are available at https://github.com/TangXu-Group/FFITN.

Hyperspectral image (HSI) classification is significant in remote sensing applications. However, most methods focus on the spectral and spatial correlation information in the neighborhood while ignoring the feature difference among global different pixels. In this article, we propose a spatial–spectral adaptive learning with pixelwise filtering (SSALPF) method to fully consider the discriminative information of pixels in different spatial locations, which mainly consists of a parallel spatial–spectral adaptive learning (SSAL) module and a pixelwise filtering (PF) module. Specifically, the former aims to obtain joint spatial–spectral discriminative features of each pixel point in a parallel manner and is used as a guide for adaptive selection of filter kernel. The latter uses the adaptive filter kernel to implement pixel-level filtering on HSI, in order to learn the discriminative features contained in different pixel points for classification. The adaptive filter kernel is generated by a linear combination of a predefined dictionary containing multiple filter bases. Experiments demonstrate that the proposed method is superior to other methods on popular hyperspectral datasets.

Recently, deep learning-based hyperspectral image (HSI) classification techniques are flourishing and exhibit good performance, where cross domain information is usually utilized to reduce the dependency on large labeled samples. However, the gap between source domain and target domain makes it difficult to carry out knowledge transfer directly. In this paper, an adversarial domain alignment with contrastive learning method is designed for the HSI classification task to achieve feature consistency that benefits transferring knowledge. In details, spectral alignment and semantic alignment are conducted in local and global levels respectively in an adversarial learning way, and the adversarial loss acts on both source and target domains. In order to learn specific features for objects with different spatial scales, a multi-scale selection module is constructed in semantic alignment to select channel features adaptively. Moreover, contrastive learning is employed to increase both robustness and sensitiveness, where augmented data from the same/different samples are forced to be similar/dissimilar with each other. The training process is conducted in a few-shot learning way then the few-shot classification loss, the adversarial loss and the contrastive loss is optimized together. Tested on one source dataset and four target datasets, the experimental results show that the proposed method outperforms the other comparisons.

As a content management technique, remote sensing (RS) scene classification (RSSC) always attracts researchers’ attention. In the past decades, many successful methods have been proposed. Nevertheless, their prerequisite is that there are large labeled data sets, which is a strict demand in practice. To resolve this contradiction, developing RSSC models with the help of few-shot learning (FSL) has become popular. Due to lacking prior knowledge, most of the existing few-shot RSSC models pay attention to the learning algorithm. However, they do not attach importance to the complex contents within RS scenes and the intricate inter-/intra-class relations between RS scenes. This would influence their performance negatively. In this paper, we propose a new few-shot RSSC model named multi-pretext-task prototypes guided dynamic contrastive learning network (MPCL-Net). MPCL-Net consists of a multi-pretext tasks generation sub-module, a deep feature learning sub-module, and a joint optimization sub-module. First, two RS-oriented pretext tasks are constructed under the self-supervised learning (SSL) framework in the multi-pretext tasks generation sub-module, which aim to explore multi-scale and rotation-invariant information from RS scenes. Second, a simple convolutional neural network (CNN) is developed in the deep feature learning sub-module to transform the RS scenes into visual features. Third, three loss functions are formulated and integrated in the joint optimization sub-module. Their goals are to fully capture the diverse land covers within RS scenes and compact/separate the intra-/inter-class samples with limited supervision. Finally, our MPCL-Net can be trained in a meta way. The positive results counted on the three public RS scene data sets confirm that our MPCL-Net is helpful to RSSC tasks under the few-shot scenario. Our source codes are available at https://github.com/TangXu-Group/Remote-Sensing-Images-Classification/tree/main/MPCL.

Cross-modal remote sensing image-text retrieval (CMRSITR) is a challenging topic in the remote sensing (RS) community. It has gained growing attention because it can be flexibly used in many practical applications. In the current deep era, with the help of deep convolutional neural networks (DCNNs), many successful CMRSITR methods have been proposed. Most of them first learn valuable features from RS images and texts respectively. Then, the obtained visual and textual features are mapped into a common space for the final retrieval. The above operations are feasible, however, two difficulties are still to be solved. One is that the semantics within the visual and textual features are misaligned due to the independent learning manner. The other one is that the deep links between RS images and texts cannot be fully explored by simple common space mapping. To overcome the above challenges, we propose a new model named interacting-enhancing feature transformer (IEFT) for CMRSITR, which regards the RS images and texts as a whole. First, a simple feature embedding module (FEM) is developed to map images and texts into the visual and textual feature spaces. Second, an information interacting-enhancing module (IIEM) is designed to simultaneously model the inner relationships between RS images and texts and enhance the visual features. IIEM consists of three feature interacting-enhancing (FIE) blocks, each of which contains an inter-modality relationship interacting (IMRI) sub-block and a visual feature enhancing (VFE) sub-block. The duty of IMRI is to exploit the hidden relations between cross-modal data, while the responsibility of VFE is to improve the visual features. By combining them, semantic bias can be mitigated, and the complex contents of RS images can be studied. Finally, the retrieval module (RM) is constructed to generate the matching scores for deciding the search results. Extensive experiments are conducted on four public RS data sets. The positive results demonstrate that our IEFT can achieve superior retrieval performance compared with many existing methods. Our source codes are available at https://github.com/TangXu-Group/Cross-modal-remote-sensing-image-and-text-retrieval-models/tree/main/IEFT.

Change detection (CD) is a hot research topic in the remote sensing (RS) community. With the increasing availability of high-resolution (HR) RS images, there is a growing demand for CD models with high detection accuracy and generalization ability. In other words, the CD models are expected to work well for various HRRS images. Convolutional neural networks (CNNs) have been dominated in HRRS image CD due to their excellent information extraction and nonlinear fitting capabilities. However, they are not skilled in modeling long-range contexts hidden in HRRS images, which limits their performance in CD tasks more or less. Recently, the Transformer, which is good at extracting global context dependencies, has become popular in the RS community. Nevertheless, detailed local knowledge receives insufficient emphasis in common Transformers. Considering the above discussion, we combine CNN and Transformer and propose a new W-shaped dual Siamese branch hierarchical network for HRRS image CD named WNet. WNet first incorporates a Siamese CNN and a Siamese Transformer into a dual-branch encoder to extract multi-level local fine-grained features and global long-range contextual dependencies. Also, we introduce deformable ideas into the Siamese CNN and Transformer to make WNet understand the critical and irregular areas within HRRS images. Second, the difference enhancement module (DEM) is developed and embedded into the encoder to produce the difference feature maps at different levels. Using simple pixel-wise subtraction and channel-wise concatenation, the changes of interest and irrelevant changes can be highlighted and suppressed in a learnable manner. Next, the multi-level difference feature maps are fused stage by stage by CNN-Transformer fusion modules (CTFMs), which are the basic units of the decoder in WNet. In CTFM, the local, global, and cross-scale clues are taken into account to ensure the integrity of information. Finally, a simple classifier is constructed and added at the top of the decoder to predict the change maps. Positive experimental results counted on four public datasets demonstrate that the proposed WNet is helpful in HRRS image CD tasks. Our source codes are available at https://github.com/TangXu-Group/Remote-Sensing-Image-Change-Detection/tree/main/WNet.

Oriented object detection is a challenging task in remote sensing, where the detected objects can be represented by oriented bounding boxes (OBBs). Angle prediction in oriented object detection has been widely studied, due to its crucial role in object detection. However, the precision of angle prediction is severely limited by misalignments in most of the existing methods, including representation-, evaluation-, and optimization-based misalignments. To alleviate these misalignments, this paper presents a novel angle prediction method, called Angle Quality Estimation (AQE). Specifically, our proposed AQE transforms the angle prediction task into a distribution estimation task to address the representation misalignment problem and implicitly measure the quality of the predicted angles. Based on the estimated angle quality, we then propose a new metric to comprehensively evaluate the quality of OBBs. Then we propose an object aspect ratio based loss function to optimize angle prediction for addressing the optimization misalignment. Our proposed AQE is a plug-and-play method, which can be embedded on any existing oriented object detector. Experimental results on three public benchmarks, including DOTA, HRSC2016, and ICDAR2015 datasets, show that our method achieves better performance than the other state-of-the-art.

Few-shot remote sensing scene classification aims to identify unseen classes using only a small number of labeled samples. Considering the large intra-class variances and inter-class similarity of remote sensing scenes, most existing methods focus on feature extraction, ignoring the overfitting problem caused by insufficient samples. To this end, we propose a novel few-shot learning framework, called multibranch feature generation networks (MFGNets), which solves the few-shot scene classification from the source by online sample generation at the representation space. Specifically, we first build a feature generation net to transform the few-shot classification into a regular classification problem, in which the generated samples are achieved by combining the class-specific features with the sampled intra-class features. Then, to ensure the quality of the generated samples, we introduce two novel regularization terms: the intra-class diversity loss (ID-Loss) and the inter-class consistency loss (IC-Loss), which aid the model in generating more diverse samples. Furthermore, we introduce a scale-angle aware self-supervised pretext to learn scale-invariant and rotation-invariant features, improving the model’s feature representation capability in remote sensing scenes. We evaluate the proposed method on three publicly available datasets, namely UC_Merced, NWPU-RESISC45, and AID. Our approach has achieved state-of-the-art performance, with an improvement of more than 3.31%, 2.64%, and 6.86% on the most challenging 1-shot tasks, respectively.

Hyperspectral image change detection (HSI-CD) aims to detect subtle changes on the Earth’s surface through approximately continuous spectral information, which has gradually become a very important research hotspot in the field of remote sensing (RS). In recent years, convolutional neural networks (CNNs) based HSI-CD methods have shown strong feature extraction capabilities. However, due to the simple fusion of spectral information in the channel dimension by CNN, the medium and long-term sequence properties of spectral features cannot be well mined and represented. Most previous studies mainly extract semantic features from images at different times, ignoring the temporal correlation between features, which cannot fully extract and effectively utilize temporal-spatial-spectral features. To this end, this paper proposes a cascade spectral aware transformer (CAST) for HSI-CD. First, we propose a temporal-spatial transformer (TS-Former) to enhance the temporal correlation and spatial global relationship of extracted features, thereby addressing the insufficient consideration of temporal correlation. Second, a spectral awareness transformer (SA-Former) is designed to better mine and represent the sequence properties of spectral features, especially the medium and long-term dependencies. Finally, we observe a spectral distortion in the process of extracting temporal-spatial features and based on this present a spectral constraint module (SCM) to preserve the sequence properties of spectral features and reduce the distortion of the spectrum. Extensive experiments on three challenging hyperspectral datasets demonstrate that our method achieves state-of-the-art results. The code is available at: https://github.com/tianshunli/CAST.

Cross-domain networks can solve the problem of insufficient labeled samples, especially for hyperspectral images (HSIs) where obtaining labeled samples is time-consuming and laborious. Most of the current methods rely on the spatial information to achieve domain alignment, without considering the rich spectral information of HSIs. Furthermore, the methods based on convolutional neural network (CNN) cannot get the spatial information of irregular image regions, resulting in poor classification results of object edges. Therefore, we design a spectral-spatial distribution consistent network (SSDC) based on meta-learning. Firstly, to improve the feature extraction ability of the cross-domain classification model, we introduce a feature pre-extraction module, which uses the spectral attention mechanism and the alternating meta-learning method to obtain the general features of the source domain and the discriminative features of the target domain, so as to obtain the spectral weight matrix for subsequent processing. Secondly, we propose a spectral consistent module based on singular value decomposition, which increases the difference between different classes of features by penalizing the singular values of the feature matrix to achieve data distribution alignment in the spectral dimension. Finally, aiming at the low classification accuracy of irregular image regions, we propose a spatial consistent module to obtain non-local spatial topological information through stacked cross modules and graph sample and aggregate networks, which can reduce domain shift. The experiments of SSDC on four classical HSI datasets show that the proposed method can obtain competitive results with other methods based on CNN and cross-domain.

Building footprint extraction plays an important role in the analysis of remote sensing images and has an extensive range of applications. Obtaining precise boundaries of buildings remains a challenge in existing building extraction methods. Some previous works have made notable efforts to address this concern. However, most of these methods require cumbersome and expensive post-processing steps. Moreover, they ignored the correlation between building semantics and contours, which we believe is crucial for building footprint extraction. To mitigate this issue, our paper presents an intuitive and effective framework that explores semantic and contour cues of buildings and fully excavates their correlation. Specifically, we construct an interactive dual-stream decoder. The Intermediate connections within this decoder interactively transmit features between branches, contributing to learning correlations between semantics and contours. We propose the Semantic Collaboration Module (SCM) to strengthen the connection between the two branches. To further boost performance, we build the Multi-Scale Semantic Context Fusion Module (MSCF) to fuse semantic information from the higher and lower layers of the network, allowing the network to obtain superior feature representations. The experimental results on the WHU, INRIA, and Massachusetts building datasets demonstrate the superior performance of our method.

The scene classification of remote sensing (RS) images plays an essential role in the RS community, aiming to assign the semantics to different RS scenes. With the increase of spatial resolution of RS images, high-resolution RS (HRRS) image scene classification becomes a challenging task because the contents within HRRS images are diverse in type, various in scale, and massive in volume. Recently, deep convolution neural networks (DCNNs) provide the promising results of the HRRS scene classification. Most of them regard HRRS scene classification tasks as single-label problems. In this way, the semantics represented by the manual annotation decide the final classification results directly. Although it is feasible, the various semantics hidden in HRRS images are ignored, thus resulting in inaccurate decision. To overcome this limitation, we propose a semantic-aware graph network (SAGN) for HRRS images. SAGN consists of a dense feature pyramid network (DFPN), an adaptive semantic analysis module (ASAM), a dynamic graph feature update module, and a scene decision module (SDM). Their function is to extract the multi-scale information, mine the various semantics, exploit the unstructured relations between diverse semantics, and make the decision for HRRS scenes, respectively. Instead of transforming single-label problems into multi-label issues, our SAGN elaborates the proper methods to make full use of diverse semantics hidden in HRRS images to accomplish scene classification tasks. The extensive experiments are conducted on three popular HRRS scene data sets. Experimental results show the effectiveness of the proposed SAGN. Our source codes are available at https://github.com/TangXu-Group/SAGN.

Road extraction faces the great challenges of occlusion, large span, and complex backgrounds in remote sensing images. Many existing methods receive context from regions near the road non-differently, and the context from irrelevant regions instead harms the semantics of features and leads to the mis-classification of the network. To address the above problem, we propose a Semantic-Aware Context Module (SACM) that encourages the network to model the context of different se-mantics supervised by a soft foreground map. And Strip Pooling Module (SPM) is introduced to match the fact that roads tend to be strip-shaped, contributing to the suppression of contamination information in irrelevant regions. Both SACM and SPM enable the network to obtain more specific semanti-cally relevant context. The experimental results on the Deep-Globe dataset show that the proposed method tremendously improves the performance of the network.

Remote sensing (RS) scene classification is a crucial research topic in the RS community, and many convolutional neural networks (CNNs)-based methods have been proposed to improve classification performance. Due to the intrinsic locality of convolution operations, CNNs are good at extracting local information but are not easy to capture global contextual information which is also important to fully interpret RS scenes. Recently, transformer has shown the potential for learning global contextual information, but it pays less attention to local information. In this paper, we propose a new interactive dual-branch network for RS scene classification, named Resformer, which can use CNNs's efficiency in extracting local information as well as transformer's power in capturing global information. Besides, we propose a two-way feature interaction module (TFIM), which can not only efficiently fuse CNNs-based local features with transformer-based global fetures, but also extract multi-scale information from RS scenes. Finally, we use a class score fusion strategy to integrate the features extracted from the two branches. Encouraging experimental results counted on two public RS scene data sets demonstrate that our Resformer is effective in RS scene classification task.

Few-shot remote sensing scene classification, which aims to recognize unseen classes given only a few labeled samples, is a challenge task due to the complex content contained in remote sensing scenes. In this end, we propose a noisy query prototypical network (NQ-ProtoNet), which uses query-mix module (QM) to produce extra query samples with inter-ference information for classification and thus implicitly enhance the feature learning ability of model. Our method alleviates the problem of large intraclass variances and inter-class similarity of remote sensing scenes to some extent, and the positive experimental results on UC Merced and NWPU data sets show that it outperforms several few-shot learning methods.

Change detection (CD) plays an increasingly important role in earth observation and reveals surface changes according to multi-temporal images. Although deep learning-based CD methods work well for their excellent modeling ability, objects in different size and shape are generally processed by the same filter kernels in feature extraction, which leads to spatial blurring and degrades the CD performance. In this paper, a spatial adaptive and feature enhanced (SAFE) siamese network is proposed to tackle this problem, where the SAFE consists of a spatial-adaptive (SA) part and a feature-enhanced (FE) part. Specifically, pixel belonging to different objects possesses its own spatial knowledge, which is captured by a soft fusion of multi-scale difference images (DIs) called SA part. Changed and unchanged areas are strengthened or weakened by the FE, which combines object features with each DI accordingly. Moreover, since there are more unchanged pixels than changed pixels, a weight-pair is introduced to balance changed and unchanged objects in the training process. The experimental results verify that compared with four representative CD algorithms, our proposed method performs best on the Change Detection Dataset (CDD).

Cross-modal Remote sensing (RS) image-text retrieval (CMR-SITR) plays a crucial role in the RS community. A common way for CMRSITR is to extract texts and RS images' feature representations separately and then measure their similarities in the specific or common feature space. Recently, along with the booming of deep convolutional neural networks (DCNNs), these kinds of methods are vivid and achieve successes in their own applications. However, they neglect the inherent relationships between different features, and they are always heavy. To overcome the limitations mentioned above, we propose a new model for CMRSITR in this paper, named multi-scale interactive transformer (MSIT). MSIT first adopts simple feature learning models for texts and RS images which could ensure the whole model is not heavy. Then, MSIT introduces transformer encoders to enhance features' usefulness by considering the potential relations between different representations. Also, a lightweight multi-scale feature learning module is proposed to mine more plentiful contents from RS images. Finally, instead of outputting the features, MSIT produces matching scores for texts and RS images, which can be used to decide the retrieval results directly. The experimental results on two RS datasets indicate our modal is effective for CMRSITR.

Instance segmentation predicts the categories of all instances and locates them using pixel-level masks. Although existing methods have shown exemplary performance, the poor boundaries due to the lack of fine-grained information re-mains a challenge for RSIs instance segmentation. In this paper, to address the problem, we propose a novel instance segmentation branch, namely Mask Decoupled Head, which is mainly composed of a Feature Enhance Module (FEM) and a Feature Decoupled Module (FDM). FEM enhances the rep-resentation of the body features through the low-frequency component of images. FDM decouples the segmentation task by supervising body and edge separately and leverages fine-grained information to complement the boundary details. We performed comprehensive experiments on NWPU VHR -10 and HRSID datasets to evaluate the effectiveness of our pro-posed method and achieved good performance.

As a hot topic in the field of remote sensing (RS), change detection aims to identify the semantic change between bitemporal RS images. Due to the semantic complexity of RS images, how to accurately detect the semantic change has become a challenging problem. Recently, many deep-based methods are proposed to solve this issue. However, ignoring the representation difference of same semantics in different periods limits their performance, such as river is liquid in summer and solid in winter. Therefore, a new method is presented, named dictionary learning based change detector (DLCDet), which consists of feature pyramid network, deep dictionary learning and dual supervision modules. In DLCDet, the deep dictionary learning is proposed to reduce the representation difference so that DLCDet identifies the potential semantic change more accurately. Experiments are conducted on two public datasets change detection dataset (CDD) and building change detection dataset (BCDD), which demonstrates the effectiveness of the proposed method.

Hyperspectral image (HSI) classification has attracted ever-rising attention to better performance based on limited labeled data. In this paper, a domain adaptation method of class balance based on few-shot learning is proposed, which obtains the classification results of target HSI by training the dataset in the source domain containing sufficient labeled data. We use a random weighted sampling strategy in the source domain and the generative adversarial network (GAN) in the target domain to reduce the label distribution shift caused by unbalanced classes. Then, the conditional maximum mean discrepancy (CMMD) is presented for a more comprehensive domain alignment by considering the posterior data distribution. In addition, the double cross non-local block and multi-scale strategy are adopted in the feature extraction stage to get a refined classification result. Experimental results on public HSI datasets demonstrate that our method is efficient and outperforms other baselines.

Weakly supervised object detection (WSOD) is a challenging task, in which image-level labels (e.g., categories of the instances in the whole image) are used to train an object detector. Many existing methods follow the standard multiple instance learning (MIL) paradigm and have achieved promising performance. However, the lack of deterministic information leads to part domination and missing instances. To address these issues, this paper focuses on identifying and fully exploiting the deterministic information in WSOD. We discover that negative instances (i.e. absolutely wrong instances), ignored in most of the previous studies, normally contain valuable deterministic information. Based on this observation, we here propose a negative deterministic information (NDI) based method for improving WSOD, namely NDI-WSOD. Specifically, our method consists of two stages: NDI collecting and exploiting. In the collecting stage, we design several processes to identify and distill the NDI from negative instances online. In the exploiting stage, we utilize the extracted NDI to construct a novel negative contrastive learning mechanism and a negative guided instance selection strategy for dealing with the issues of part domination and missing instances, respectively. Experimental results on several public benchmarks including VOC 2007, VOC 2012 and MS COCO show that our method achieves satisfactory performance.

Hyperspectral anomaly detection is a popular topic in remote sensing image intelligent interpretation. To detect anomaly, many methods for background representation have been proposed. However, the prior information of background and anomaly is not fully explored in these methods. To tackle this issue, we combine low-rank dictionary learning (LRDL) with total variation (TV) constraint for hyperspectral anomaly detection. To be specific, the LRDL is introduced for background representation to explore the low-rank priori of background. Considering the smooth structural characteristic of background in spatial, we introduce the TV constraint on coefficients matrix for better background representation learning. Then the residual part is used to discriminate anomaly. The experiments on three real data sets demonstrate the effectiveness of the proposed method compared with state-of-the-art algorithms in hyperspectral anomaly detection.

Recently, hyperspectral image (HSI) classification by deep learning is flourishing. However, only a few labeled samples are available in practice since it is time-and-labor-consuming to label pixels in HSI (called target domain). This paper proposes a domain-adaptive few-shot learning (DAFSL) method to tackle this problem. Specifically, some other HSIs (called source domain) with large labeled samples are fully used as complementary information and a generative architecture is employed to adapt embedded features in source domain to that of target domain. We first perform domain adaptation with unsupervised learning. In details, the embedded features are generated by the encoder of an autoencoder, where both source and target samples could be well recovered and the reconstruction loss is used to measure the gap between source domain and target domain. At the same time, the embedded features are put into a metric space for classification in source domain and the encoder parameter is fine-tuned together with the classifier in target domain with few labels, so that both general and discriminative features are well captured. The experiment results show that DAFSL outperforms the other mainstream methods with limited labeled samples.

In this article, we focus on the challenging multicategory instance segmentation problem in remote sensing images (RSIs), which aims at predicting the categories of all instances and localizing them with pixel-level masks. Although many landmark frameworks have demonstrated promising performance in instance segmentation, the complexity in the background and scale variability instances still remain challenging, for instance, segmentation of RSIs. To address the above problems, we propose an end-to-end multicategory instance segmentation model, namely, the semantic attention (SEA) and scale complementary network, which mainly consists of a SEA module and a scale complementary mask branch (SCMB). The SEA module contains a simple fully convolutional semantic segmentation branch with extra supervision to strengthen the activation of interest instances on the feature map and reduce the background noise's interference. To handle the undersegmentation of geospatial instances with large varying scales, we design the SCMB that extends the original single mask branch to trident mask branches and introduces complementary mask supervision at different scales to sufficiently leverage the multiscale information. We conduct comprehensive experiments to evaluate the effectiveness of our proposed method on the iSAID dataset and the NWPU Instance Segmentation dataset and achieve promising performance.

The remote sensing image captioning has attracted wide spread attention in remote sensing field due to its application potentiality. However, most existing approaches model limited interactions between image content and sentence and fail to exploit special characteristics of the remote sensing images. We introduce a novel recurrent attention and semantic gate (RASG) framework to facilitate the remote sensing image captioning in this article, which integrates competitive visual features and a recurrent attention mechanism to generate a better context vector for the images every time as well as enhances the representations of the current word state. Specifically, we first project each image into competitive visual features by taking the advantage of both static visual features and multiscale features. Then, a novel recurrent attention mechanism is developed to extract the high-level attentive maps from encoded features and nonvisual features, which can help the decoder recognize and focus on the effective information for understanding the complex content of the remote sensing images. Finally, the hidden states from the long short-term memory (LSTM) and other semantic references are incorporated into a semantic gate, which contributes to more comprehensive and precise semantic understanding. Comprehensive experiments on three widely used datasets, Sydney-Captions, UCM-Captions, and Remote Sensing Image Captioning Dataset, have demonstrated the superiority of the proposed RASG over a series of attentive models based on image captioning methods.

With the rapid development of remote sensing technology, satellites can easily obtain multispectral (MS) and panchromatic (PAN) images. How to mine the essence and peculiarity of the MS and PAN images and utilize their complementary to improve classification performance is still a challenge. This paper designs a two-stage mutual fusion network (TSMF-Net) for MS and PAN image classification. The network can be divided into two stages: data fusion and feature fusion. In the data fusion stage, we propose an adaptive twin intensity-hue-saturation (ATIHS) strategy. It not only aligns the size and channels of the MS and PAN images by a novel q-Split operation, but also introduces an adaptive soft-average mask to reduce the differences between replacement components, effectively mitigating spectral distortion and paving the way for the next stage. In the feature fusion stage, we propose a feature graft block (FG-Block) in which we introduce triplet loss and design an interlaced channel addition (ICA) module. Under the supervision of triplet loss, the FG-Block separates and hauls each branch’s essential and peculiar features. With the help of the ICA module, it can effectively graft the essential feature between branches and retain the peculiar feature of each branch, improving the utilization and discrimination of features. Finally, composed of the ATIHS, FG-Blocks, and output layers, our TSMF-Net is proven to improve the accuracy of the remote sensing classification task. The experimental results on multiple datasets verify the effectiveness of our proposed algorithms. Our code is available at: https://github.com/liaoyinuo/TSMF-Net.

Polarimetric synthetic aperture radar (PolSAR) image classification is one of the hottest issues in remote sensing, where studies on pixel-level information and relationship are of great significance. In this article, graph convolutional network (GCN) is employed to accomplish this pixel-level task benefiting from its excellent capability in structure exploration and information propagation between different pixels. To reduce the communication burden between various PolSAR pixels and high computational cost for the whole PolSAR image, an adaptive GCN (AdapGCN) consisting of pixel-centered subgraphs is proposed in this article. In the AdapGCN, a data-adaptive kernel and a spatial-adaptive kernel are introduced to, respectively, model data structure and spatial structure for PolSAR image. Moreover, a multiscale learning structure is integrated to further explore complicated relations between pixels. Extensive comparative evaluations validate the superiority of our new AdapGCN model for PolSAR image classification over a wide range of state-of-the-art methods on three challenging benchmarks.

Recently, multicategory object detection in high-resolution remote sensing images is still a challenge. First, objects with significant scale differences exist in one scene simultaneously, so it is generally difficult for the detectors to balance the detection performance of large and small objects. Second, because of the complex background and the objects’ densely distributed characteristics in the remote sensing images, the extracted features usually have noise and blurred boundaries, which interfere with the detection performance of the object detectors. With this observation, we propose an end-to-end scale-aware network called feature split–merge–enhancement network (SME-Net) for remote sensing object detection, composed of the feature split-and-merge (FSM) module, the offset-error rectification (OER) module, and the object saliency enhancement (OSE) strategy. FSM eliminates salient information of large objects to highlight the features of small objects in the shallow feature maps. It also transmits the effective detailed features of large objects to the deep feature maps, alleviating feature confusion between multiscale objects. OER corrects the inconsistency of the features spatial layout among the multilayer feature maps by the proposed offset loss, so as to achieve supervised elimination and transmission in FSM. OSE enhances the features of interests and suppresses the background information by the proposed membership function, thus preventing false detection and missed detection caused by noise and blurred boundaries. The effectiveness of the proposed algorithm has been verified on multiple datasets. Our code is available at:https://github.com/Momuli/SMENet.git

Recently, with the increasing availability of the high-quality panchromatic (PAN) and multispectral (MS) remote sensing (RS) images, the inherent complementarity between PAN and MS images provides a wide development prospect for the multimodality RS image classification task. However, how to cleverly relieve the modal differences and effectively integrate the single-modality PAN and MS features is still a challenge. In this article, we design a collaborative correlation-matching network (CCM-Net) for multimodality RS image classification. Concretely, we first propose a bidirectional dominant feature supervision (Bi-DFS) learning, it utilizes single-modality dominant features as supplementary supervision information to establish the joint optimization loss function, thereby adaptively narrowing the differences between modalities before the feature extraction. In the feature extraction stage, the interactive correlation feature matching (ICFM) learning, composing the spatial feature matching (Spa-FM) and spectral feature matching (Spe-FM) strategies, is proposed to establish interactive matching and enhancement between multimodality strong correlation features from the perspective of spatial and spectral, respectively, thereby effectively alleviating the semantic deviation of multimodality features. Finally, we aggregate finer multilevel multimodality features to obtain top-level features with high discrimination. The effectiveness of the proposed algorithm has been verified on multiple datasets. Our code is available at:https://github.com/Momuli/CCM-Net.git.

Remote sensing scene classification (RSSC) is an open and challenging research topic in the remote sensing (RS) community. It aims to define semantic labels for RS scenes according to their contents. Recently, with the development of deep convolutional neural networks (DCNNs), the results of RSSC have been enhanced to a large extent. However, the cracking performance of these DCNN-based models depends on a large number of labeled data. Once the volume of the labeled data is decreased, their behavior would be weakened dramatically. In this article, we propose a new training algorithm that can work smoothly with a few labeled samples to address this limitation. Along with the introduced DCNN, the presented methods perform satisfactorily. In particular, we first construct a dual-branch network (DBNet) to mine the multiscale and multiangle information from RS scenes. Thus, the abundant land covers with diverse sizes, directions, and shapes can be captured simultaneously. Then, to train DBNet using scarce semantic labels, a class-level prototype guided learning (CPGL) algorithm is developed based on the meta-learning paradigm. Besides the usual episode training manner, a prototype refinement module (PRM) and a prototype discrimination module (PDM) are designed with the help of metric learning theory to ensure the effectiveness of our CPGL. The comprehensive experiments are conducted on four public RS scene datasets, and the encouraging results imply that our DBNet and CPGL can copy with RSSC tasks with small labeled data. Our source codes are available athttps://github.com/TangXu-Group/Remote-Sensing-Images-Classification/tree/main/CPGL.

In recent years, convolutional neural network (CNN)-based methods have been widely used for remote sensing (RS) scene classification tasks and achieved excellent results. However, CNNs are not good at exploring contextual information, which is essential for fully understanding RS scenes. A new model named transformer attracts researchers’ attention to address this problem, which is skilled in mining the latent contextual information in RS scenes. Nevertheless, since the contents of RS scenes are diverse in type and various in scale, the performance of the original transformer in RS scene classification cannot reach what we expect. In addition, due to the specific self-attention mechanism, the time costs of the transformer are high, which hinders its practicability in the RS community. To overcome the above limitations, we propose a new model named efficient multi-scale transformer and cross-level attention learning (EMTCAL) for RS scene classification in this paper. EMTCAL combines the advantages of CNN and transformer to mine information within RS scenes fully. First, it uses a multi-layer feature extraction module (MFEM) to acquire global visual features and multi-level convolutional features from RS scenes. Second, a contextual information extraction module (CIEM) is proposed to capture rich contextual information from multi-level features. In CIEM, taking the characteristics of RS scenes and the computational complexity into account, we propose an efficient multi-scale transformer (EMST). EMST can mine the abundant knowledge with various scales hidden in RS scenes and model their inherent relations at small-time costs. Third, a cross-level attention module (CLAM) is developed to aggregate and explore correlations of multi-level features. Finally, a class score fusion module (CSFM) is designed to integrate the contributions of global and aggregated multi-level features for the discriminative scene representations. Extensive experiments are conducted on three public RS scene data sets. The positive results demonstrate that our EMTCAL can achieve superior classification performance and outperform many state-of-the-art methods.

With the explosive growth of the volume and resolution of high-resolution remote-sensing (HRRS) images, the management of them becomes a challenging task. The traditional content-based remote-sensing image retrieval (CBRSIR) technologies cannot meet what we expect due to the large volume of image archives and complex contents within HRRS images. As a successful approximate nearest neighborhood (ANN) search technique, Hash learning has received wide attention, especially when deep convolutional neural networks (DCNNs) appear. Due to DCNNs’ strong capacity of feature learning, many DCNN-based hashing methods have been proposed and achieved good performance for large-scale CBRSIR tasks. Nevertheless, their limitation is that a large of labeled training samples should be collected for training the deep models. To overcome this limitation, this article, therefore, develops a new supervised hash learning method for the large-scale HRRS CBRSIR task based on meta-learning, which could achieve well-retrieval performance with a few labeled training samples. First, taking the characteristics of HRRS into account, we develop a self-adaptive convolution (SAP-Conv) block and design a hashing net based on the block. SAP-Conv can learn robust features from HRRS images by exploring their multiscale information. Second, to enhance the generalization of the hashing net under a few labeled training samples, the hash learning is formulated in a meta-way, and we name it meta-hashing. Meta-hashing can effectively preserve the similarities between support and query set, and the similarities between samples within support set by the developed loss function. To further improve the performance of meta-hashing, we expand it to a dynamic version named dynamic-meta-hashing, in which the numbers of support and query are changeable in the training phase. Experimental results counted on the three widely used HRRS datasets demonstrate our dynamic-meta-hashing and meta-hashing can achieve promising performance in large-scale HRRS CBRSIR tasks based on a few training samples. Our source codes are available athttps://github.com/TangXu-Group/Meta-hashing.

As a fundamental application, change detection (CD) is widespread in the remote sensing (RS) community. With the increase in the spatial resolution of RS images, high-resolution remote sensing (HRRS) image CD tasks receive growing attention. The change information hidden in multitemporal HRRS images could help discover our planet comprehensively. In the current deep learning era, convolutional neural networks (CNNs) have become one of the most powerful tools for a wide range of RS tasks including HRRS image CD, due to their superb feature learning capacity. However, most of them need a large amount of labeled data to accomplish the CD process, which is challenging or even impractical in many RS applications. Also, given the limited valid receptive field, CNNs can only capture short-range context within HRRS images, which is probably not enough to fully explore change information from the images. To overcome these limitations, in this article, we propose an unsupervised CD method, termed GMCD, based on graph convolutional network (GCN) and metric learning. GMCD consists of a Siamese fully convolution network (FCN), a multiscale dynamic GCN (Mlt-GCN), and a pseudolabel generation mechanism based on metric learning. The Siamese FCN contains a Siamese encoder and a pyramid-shaped decoder, aiming to extract multiscale features and integrate them to generate reliable difference images (DIs). Mlt-GCN focuses on capturing the short- and long-range contextual patterns at feature map level to extract changed and unchanged areas completely. The pseudolabel generation mechanism aims to produce reliable pseudolabels (changed, unchanged, and uncertain) to help accomplish the model training in an unsupervised way. Experiments on four HRRS image CD datasets demonstrate that GMCD outperforms the existing state-of-the-art methods.

Ship detection plays a significant role in the high-resolution remote sensing (HRRS) community, but it is a challenging task due to the complex contents within HRRS images and the diverse orientation of ships. Recently, with the development of deep learning, the performance of the HRRS ship detection model has been improved greatly. Most of them employ deep networks and complicate anchor mechanism to get well ship detection results. Nevertheless, this kind of combination limits the detection efficiency. To address this problem, a new approach named accurate and real-time rotational ship detector (AR2Det) is proposed in this article to detect ships without the anchor mechanism. Based on the extracted features by the feature extraction module (FEM) and the central information of ships, AR2Det adopts two simple modules, ship detector (SDet) and center detector (CDet), to generate and improve the detection results, respectively. AR2Det is efficient due to the simple postprocessing and the lightweight network. Also, AR2Det performs satisfactorily due to the effective generation and enhancement strategy of bounding boxes. The extensive experiments are conducted on a public HRRS image ship detection dataset HRSC2016. The promising results show that our method outperforms the state-of-the-art approaches in terms of both accuracy and speed.

Graph convolution networks (GCNs) have been applied in a variety of fields due to their powerful ability in processing graph-like data. However, the massive number of hyperspectral pixels makes it challenging to define general graph structures on hyperspectral images (HSIs). On the other hand, convolutional neural networks (CNNs) take in regular image regions with fixed square size, and have demonstrated impressive accuracy while being efficient in computation. Inspired by the classification framework of CNNs, we develop a GCN-based model that generates effective local spectral–spatial features for HSI classification. Specifically, graph convolutions are performed separately on every local region, which significantly limits the graph’s size. While graph convolution extracts features of every pixel, it does not reduce the number of them. To fuse suitable representations for the classification task, we develop a graph pooling operation to preserve classification-specific features and reduce redundant pixels. Based on local regions of HSIs, pooling in the graph domain is equivalent to spatial pooling in the spatial domain. The proposed method is thus named the spatial pooling graph convolutional network (SPGCN). Experimental results on several typical datasets demonstrated that the proposed SPGCN provides competitive results compared with other state-of-the-art CNN-based methods.

Limited by hardware conditions and complex degradation processes, the obtained remote sensing images (RSIs) are often low-resolution (LR) data with insufficient high-frequency information. Image super-resolution (SR) aims to improve the spatial resolution of images and add reasonable detailed information. Although existing convolutional neural network (CNN)-based methods achieve good performance by adding residual structure and attention mechanism to the network, simply stacking the residual structure and embedding the attention module directly on the residual branch lead to localized use of features and information loss. To address the above problems, we propose a dual-resolution connected attention network (DRCAN). Specifically, a high-resolution (HR) learning branch is constructed to complement the mapping learning between LR images and HR images, and a connected attention module with residual learning is introduced to make full use of the different levels of intermediate layer features. Besides, we collect data at different resolutions from Google Earth to form a dataset named XD IPIU for RSIs SR. Extensive experiments demonstrate the effectiveness of the proposed model and DRCAN shows the state-of-the-art performance in terms of quantitative evaluation and visual quality.

Hyperspectral image (HSI) classification is one of the most important tasks in hyperspectral data analysis. Convolutional neural networks (CNN) have been introduced to HSI classification and achieved good performance. In this article, an effective and efficient CNN-based spectral partitioning residual network (SPRN) is proposed for HSI classification. The SPRN splits the input spectral bands into several nonoverlapping continuous subbands and uses cascaded parallel improved residual blocks to extract spectral–spatial features from these subbands, respectively. Finally, the features are fused and fed into a classifier. By equivalently using grouped convolutions, the spectral partition and feature extraction are embedded into an end-to-end network. Experimental results show that the proposed SPRN achieves state-of-the-art performance, meanwhile, with relatively fewer parameters and computational costs. Usually, the CNN takes a patch that contains continuous spatial information as the input and results in a class label of the center pixel. The large size of the input patch includes more spatial information, whereas also introduces interfering pixels that may lead to a degradation of classification accuracies. For that reason, we propose a novel spatial attention module named homogeneous pixel detection module (HPDM). The module alleviates the degradation of performance as the input patch size increases by capturing the homogeneous pixels in the input patch. The module can be integrated into any CNN-based HSI classification framework.

Object detection has been a fundamental task in the field of remote sensing and has made considerable progress in recent years. However, the high background complexity in remote sensing images (RSIs) remains challenging. In this article, we propose a refined rotation detector, namely, the Foreground Refinement Network (FoRDet), to alleviate the above problem by leveraging the information of foreground regions from the perspectives of feature and optimization. Specifically, we propose a foreground relation module (FRL) that aggregates the foreground-contextual representations from the coarse stage and improves the discrimination of foreground regions on feature maps in the refined stage. Besides, considering the risk of the potential foreground anchors being overwhelmed in the training phase, we design a foreground anchor reweighting (FRW) loss that integrates the classification confidence and localization accuracy of each foreground anchor from the coarse stage to dynamically regulate their contributions in the refined stage, which highlights the potential foreground anchors. The comprehensive experimental results on three public datasets for rotated object detection DOTA, HRSC2016, and UCAS-AOD demonstrate the effectiveness of our proposed method.

Video object detection, a basic task in the computer vision field, is rapidly evolving and widely used. In recent years, deep learning methods have rapidly become widespread in the field of video object detection, achieving excellent results compared with those of traditional methods. However, the presence of duplicate information and abundant spatiotemporal information in video data poses a serious challenge to video object detection. Therefore, in recent years, many scholars have investigated deep learning detection algorithms in the context of video data and have achieved remarkable results. Considering the wide range of applications, a comprehensive review of the research related to video object detection is both a necessary and challenging task. This survey attempts to link and systematize the latest cutting-edge research on video object detection with the goal of classifying and analyzing video detection algorithms based on specific representative models. The differences and connections between video object detection and similar tasks are systematically demonstrated, and the evaluation metrics and video detection performance of nearly 40 models on two data sets are presented. Finally, the various applications and challenges facing video object detection are discussed.

Hyperspectral image (HSI) classification is a hot research direction in remote sensing community. In HSIs, there are hundreds of narrow and continuous spectral bands. Due to powerful sequence data processing ability, recurrent neural network (RNN) has shown great potential in HSI classification in recent years. In order to ensure a good classification performance, how to design a proper RNN structure is of great importance. In this paper, an automatic RNN (Auto-RNN) for HSI classification is proposed. Firstly, a number of candidate modules, including ReLU, tanh, sigmoid, and identity, are provided. Then, a policy gradient-based reinforcement learning is us ed to search the feasible deep architecture. Finally, the best RNN structure is selected through evaluating on the validation set. The experimental results demonstrate that the proposed algorithm can yield promising classification performance compared with some existing methods.

As different kinds of high-resolution remote sensing (HRRS) image data sources increase, the cross-source content-based image retrieval (CS-CBRSIR) is becoming an important and urgent task to be solved. Most existing methods focus on optimizing the common space features for dual-source effectively. The source discrepancy in classifier level, however, has been ignored. To handle this problem, we propose teacher-ensemble learning with the knowledge distillation method in this paper. The ensemble of source-shared and source-specific classifiers could construct an effective teacher model. The useful information can be transferred back with the knowledge distillation. Besides, the feature pyramid network is introduced to learn the multi-scale features from HRRS images, which can describe the complex contents of HRRS images well. The positive experimental results conducted on DSRSID illustrates the effectiveness of the proposed method.

High-resolution remote sensing (HRRS) image scene classification based on limited data set is challenging in practical application. Although convolutional neural networks have shown powerful feature representation capability, they cannot perform well in the absence of rich label information in general. This paper proposes a multi-scale meta-learning-based (MSML) model to complete the HRRS scene classification with a little labeled data. First, we develop a multi-scale feature learning strategy to explore the rich information from HRRS scenes. Then, to use small data to train our network, we formulate the meta-learning as a regularization term and embed it into the classification loss function. By optimizing the proposed loss function, we can obtain a robust and generalized scene classification model. The positive experimental results counted on a public HRRS scene data set show that our MSML model is useful in HRRS scene classification tasks.

Convolutional neural networks (CNNs) have achieved cracking performance in the hyperspectral image (HSI) classification task. Nevertheless, most of them cannot meet what we expect when the numbers of labeled samples are small. Also, due to the rectangular convolution kernels, the long-range context information within HSIs is cannot fully be explored. To solve these problems, we propose a semi-supervised method based on the graph convolutional network (GCN) and bidirectional Long Short-Term Memory (Bi-LSTM). First, HSIs are over segmented into various superpixels and GCN is employed for mining the advanced spectral features. Second, we input the obtained spectral features to the Bi-LSTM model for exploring global spatial features. Due to the diverse receptive fields, the short-and long-range spatial relations can be discovered simultaneously. Finally, we map the features from region-level to pixel-level for classifying HSIs. The positive experimental results counted on two HSIs demonstrate that our method is superior to some popular methods.

As an important research topic in the remote sensing (RS) community, RS image scene classification is a challenging task due to the complex contents of RS images. In general, RS image scene classification is a single-label problem. Nevertheless, it is known that the contents within RS are huge in volume and diverse in type. Only a single semantic label cannot describe an RS scene completely, especially when the resolution of RS images is increased recently. The various semantics hidden in the high-resolution RS (HRRS) images are also important to the scene classification task. Taking the issues mentioned above into account, we develop a new scene classifier named graph scene classifier (GSCer) for HRRS images with the help of the deep convolution neural network (DCNN) and dynamic graph convolution (DGCN). Not only the global semantic but also the diverse hidden local semantics within an HRRS image can be fully explored. The encouraging experimental results counted on two public HRRS data sets demonstrate that our GSCer is effective in HRRS scene classification tasks.

Change Detection is an important and challenging task in the remote sensing (RS) field, especially when the resolution of RS images is getting higher. The appearance of the deep convolutional neural network (DCNN) provides new opportunities for the high-resolution RS (HRRS) image processing as well as the HRRS CD task. In this paper, we proposed a Siamese holistically-guided FCN (SHG-FCN) model to fully mine the low- and high-level features from HRRS images for completing the CD task. SHG-FCN consists of a Siamese encoder and a holistically-guided decoder. The Siamese encoder adopts five layers of convolution for feature extraction and generates multi-scale difference maps. The decoder employs the holistically-guided architecture, which uses the deep semantic feature as codewords to guide the up-sample of feature map and achieve multi-scale feature fusion. Our model is testified on two public HRRS datasets, and the obtained encouraging CD results illustrate that our method is effective in HRRS CD tasks.

The content-based remote sensing image retrieval (CBRSIR) has attracted increasing attention with the number of remote sensing (RS) images growing explosively. Benefiting from the strong capacity of the deep convolutional neural network (DCNN), the performance of CBRSIR has been improved in recent years. Although great successes have been obtained, learning the RS images' representative features and enhancing the retrieval efficiency for the large-scale CBRSIR tasks are still two challenging problems. In this article, we propose a new CBRSIR method named feature and hash (FAH) learning, which consists of a deep feature learning model (DFLM) and an adversarial hash learning model (AHLM). The DFLM aims at learning the RS images' dense features to guarantee the retrieval precision. In the DFLM, the DCNN and the proposed feature aggregation are integrated to capture the multiscale features. Then, the discrimination of the obtained features can be highlighted by the attention map in the developed attention branch. The AHLM maps the dense features onto the compact hash codes so that the retrieval efficiency can be improved. The AHLM contains a hash learning submodel and an adversarial regularization submodel. In particular, the hash learning submodel learns the real-valued hash codes that are similarity preserved by semantic supervisions. The adversarial regularization submodel regularizes the real-valued hash codes to learn the discrete uniform distribution with possible values 0 and 1. In this way, the hash codes are coding-balanced and the quantization errors are reduced. Encouraging experimental results counted on three public benchmark data sets demonstrate that our FAH can achieve competitive performance in the CBRSIR task compared with many existing hash learning methods.

A new method based on the Looking-Around-and-Into (LAaI) mode is proposed for the task of change detection in large-scope Polarimetric Synthetic Aperture Radar (PolSAR) image. Specifically, the LAaI mode consists of two processes named Look-Around and Look-Into, which are accomplished by attention proposal network (APN) and recurrent convolutional neural network (CNN) (Recurrent CNN), respectively. The former provides certain subregions efficiently, and the latter detects changes in subregions accurately. In Look-Around, difference image (DI) of whole PolSAR images is calculated first to get global information; then, APN is established to locate the position of interested subregions intentionally by paying special attention to; next interested subregions that contain changed area in high probability are picked out as candidate-regions. Moreover, candidate-regions are sorted in importance descending order so that highly interested regions have priority to be detected. In Look-Into, candidate-regions of different scales are selected at first; then, Recurrent CNN is constructed and employed to deal with multiscale PolSAR subimages so that clearer and finer change detection results are generated. The process is repeated until all candidate-regions are detected. As a whole, the proposed algorithm based on the LAaI mode looks around whole images first to find out the possible position of changes (candidate-regions generation in Look-Around) and then reveal the exact shape of changes in different scales (multiscale change detection in Look-Into). The effect of APN and Recurrent CNN is verified in experiments, and it shows that the proposed method performs well in the task of change detection in the large-scope PolSAR image.

In recent years, with the development of deep learning (DL), the hyperspectral image (HSI) classification methods based on DL have shown superior performance. Although these DL-based methods have great successes, there is still room to improve their ability to explore spatial-spectral information. In this article, we propose a 3-D octave convolution with the spatial-spectral attention network (3DOC-SSAN) to capture discriminative spatial-spectral features for the classification of HSIs. Especially, we first extend the octave convolution model using 3-D convolution, namely, a 3-D octave convolution model (3D-OCM), in which four 3-D octave convolution blocks are combined to capture spatial-spectral features from HSIs. Not only the spatial information can be mined deeply from the high- and low-frequency aspects but also the spectral information can be taken into account by our 3D-OCM. Second, we introduce two attention models from spatial and spectral dimensions to highlight the important spatial areas and specific spectral bands that consist of significant information for the classification tasks. Finally, in order to integrate spatial and spectral information, we design an information complement model to transmit important information between spatial and spectral attention features. Through the information complement model, the beneficial parts of spatial and spectral attention features for the classification tasks can be fully utilized. Comparing with several existing popular classifiers, our proposed method can achieve competitive performance on four benchmark data sets.

Anomaly detection of a hyperspectral image without any prior information has attracted much more attention in remote sensing image understanding and interpretation, which aims at determining whether a sample belongs to background or anomaly. Low-rank dictionary learning plays an important role in exploiting the low-rank prior of background for hyperspectral image (HSI) anomaly detection. In this article, the low-rank dictionary learning is introduced to learn a dictionary which can reconstruct the background positively, while anomaly cannot. Considering the high correlation of data especially between the adjacent bands, we resort to spectral-difference low-rank dictionary representation learning for global background modeling which can fully exploit the low-rank prior of background. Then, the residual matrix is used to distinguish anomaly. Different from the existing anomaly detection methods based on dictionary which is constructed or learned in a separated step, our proposed model can simultaneously learn the dictionary and separate anomaly by iterative learning. The experimental results on five real data sets demonstrate the superior performance of the proposed method for hyperspectral anomaly detection compared with other state-of-the-art algorithms.

Deep learning (DL) based object detection methods have been making great achievements for natural images, which guides the vehicle detection of optical remote sensing videos (ORSV). Compared with natural images, objects in ORSV are smaller and blurrier, and most of vehicles are crowded. Thus, it is difficult for DL to detect these small objects only using the single-frame image. To address this problem, a motion guided R-CNN (MG-RCNN) is proposed. In MG-RCNN, motion information from consecutive frames is extracted by the mean differencing method and merged into apparent information to obtain motion-related discriminative features. Then, high-quality proposals are generated on the feature maps by mini-region proposal network (MRPN). For small targets, an improved loss function is defined by incorporating smooth factor, which makes the regression of shapes more stable. Experiments on ORSV demonstrate the proposed method shows superior detection performance over state-of-the-art deep learning methods.

Deep learning method shows its powerful classification performance with sufficient available data. However, the labeled data is limited in hyperspectral images (HSIs). Semi-supervised algorithms have unique advantages on dealing with this problem. Therefore, a semi-supervised convolutional neural network is proposed in this paper. It consists of two branches, which use limited labeled samples and a large number of unlabeled samples, respectively. The first branch includes an encoder-decoder model to extract contextual information of unlabeled samples. The other one uses the similar construction except extra classification layers to extract discriminative features of labeled samples. In order to fuse contextual and discriminative information, we cascade the features of low-level layers from different branches. Furthermore, self-attention is added to the first branch, which focuses more on the global information for classification. The experiment results show that the proposed model provides a competitive result compared with state-of-the-art methods.

Remote sensing (RS) images feature learning, plays a crucial role in many RS images application, and attracts scholars' attention. However, since RS images contain complex contents, how to extract robust features that can fully represent RS images becomes an important and tough task. In this paper, we develop a feature learning method based on multi-branch networks, named M-Net, which consists of fine-grained branch and coarse branch. Considering the objects within RS images are diverse in type and resolution, the fine-grained branch is developed to capture rich object-level information. First, the RS images convolutional features are extracted by fine-grained branch. Second, through encoding the score maps which can highlight the important regions, the fine-grained structure mapping are obtained. Finally, the object-level features are generated by transforming the convolutional features through mapping. The coarse branch is developed to transform the obtained object-level features into global structure for representing images. The positive experimental results counted on RS benchmark data set demonstrate that the proposed M-Net can learn more powerful features.

Scene classification of remote sensing (RS) images has attracted increasing attention due to its wide applications. Recently, with the advances of deep learning models, especially convolutional neural networks (CNNs), the performance of remote sensing image scene classification has been significantly improved. In this paper, based on the popular CNN, we develop a new scene classification network, named the Global and Local Consistent (GLC) network, to deeply explore useful information from the RS images. First, we adopt a pre-trained CNN to learn the intermediate feature maps from the RS image pairs. Second, by introducing the visual attention mechanism, the global and local integration model is developed to mine the rich information from the obtained feature maps. Third, the attention consistent model is designed to eliminate the negative influence of the issue of attention inconsistency on the classification. To verify the effectiveness of the proposed method, we select two popular RS image data sets. Compared with some existing classification models, our network can achieve competitive results, which illustrates that our method is useul to the RS scene classification.

Hyperspectral image (HSI) classification is an important task in the remote sensing community. In general, many hyperspectral classification methods are based on pixel patch, which leads to information redundancy. In this paper, we propose a multi-scale encoder-decoder network for HSI classification. First, we adapt an encoder-decoder framework as the backbone network and use a skip connection between the encoder and decoder, the spatial information is obtained by this network. Second, we develop a multi-scale block to get the multi-scale information. Third, we retain complete spectral information through the constant number of spectral channels. Finally, an optimizer strategy is designed to achieve our model for the HSI classification task. We experiment with our method and other methods on two public datasets, and the results denote our model is useful for HSI classification task.

With the explosive increase of remote sensing images, content-based remote sensing image retrieval (CBRSIR) has aroused widespread attention. Convolutional Neural Network (CNN) based methods are widely used in CBRSIR due to the development of deep learning. However, common used CNN models have difficulties in holding shift-invariant property due to the widely used down-sampling method, which means a little shift of input may cause a mutation of feature representation. To mitigate the absence of shift-invariant in down-sampling, we propose the learnable blur kernel (LBK), that can enhance the feature extraction capability by leveraging more context information. We build on this concept without extra cost, which can be simply integrated with modern CNNs architecture. Our method is validated on the public remote sensing dataset and compared with other retrieval methods. The overall experimental results show that the proposed method achieves outstanding performance.

Object detection in remote sensing images is a challenging task because of the large scale variations across the geospatial objects. The feature pyramid network (FPN) is widely used to alleviate the scale variations problem, however, it only fuses the features from adjacent levels and lacks the information of the entire feature hierarchy. In this paper, we propose a novel and effective feature pyramid aggregation network, called Adaptive Feature Aggregation Network (AFANet). Specifically, we propose the Adaptive Feature Aggregation (AFA) module to adaptively aggregate multi-level features of FPN and introduce the Bottom-up Path to enhance the location information of the entire feature levels. In addition, we use the Receptive Field Block (RFB) module to capture different receptive field features for each level feature map. We evaluate the effectiveness of our AFANet on the DOTA dataset and achieves noticeable performance compared with the baseline.

With the development of deep learning, hyperspectral image (HSI) classification tasks have developed rapidly, the classification performance is improved in a big degree. Despite the great success of the existing methods, there is still room for improvement to extract features from spatial and spectral dimensions. In this paper, we propose a multiscale spatial and spectral feature network (MSSFN) to capture discriminative features for the classification of HSIs. Specifically, we first use three convolution layers to extract the features of original HSI data. Second, combining the spatial masks model and spectral attention model to build multiscale spatial and spectral model (MSSM). Through the MSSM model, the spatial information of different scales can be obtained and the useful spectral bands can be emphasized. Finally, in order to reduce the computation complexity and simplify network, the other three convolution layers with a small number of convolution kernels are adopted in our method. The experimental results demonstrate that our method is superior to most existing methods on two public HSI datasets.

Object detection is one of the most important tasks in the field of very high resolution (VHR) remote sensing (RS) images understanding. Due to the characteristics of VHR RS images, the detection performance is always limited by class imbalance and intersection-over-union (IoU) distribution imbalance. To mitigate the adverse effects caused thereby, we propose a supervised adaptive-RPN (SA-RPN) model with the help of deep learning in this paper. First, we introduced a supervised multi-dimensional attention network to overcome the foreground-background class imbalance. It can help the network to highlight the foreground and suppress the background effectively. Second, we develop the adaptive-RPN to reduce the negative impact of the IoU distribution imbalance by adaptively selecting the size of the anchor. The positive experimental results on the public data set validate the usefulness of our SA-RPN model. Compared with the popular deep learning RS object detection methods, our method achieves improved performance.

Multi-class geospatial object detection in remote sensing images suffer great challenges, such as large scales variability and complex background. Although feature pyramid network (FPN) can alleviate the problem of scale variation to some extent, it causes the loss of spatial and semantic information which is not conducive to object location. To address the above problem, this paper proposes a discriminative feature pyramid network (DFPN) by introducing a global guidance module (GGM) and a feature aggregation module (FAM). Specifically, the global guidance module delivers the high-level semantic information to lower layers, so as to obtain feature maps with stronger semantic information to eliminate the interference caused by complex background. The feature aggregation module enhances the interflow of information between different layers and better captures the discrimination information at each layer. We validate the effectiveness of our method on the NWPU VHR-10 and RSOD datasets, the results outperform baseline by 2.06 and 3.88 points respectively.

This letter proposed a road extraction method based on fully convolutional networks (FCNs) with an ensemble strategy in order to solve the imbalance of road and background areas in aerial images. By utilizing the FCN, we consider road extraction as a semantic segmentation problem. In the network, the weight of the loss function is modified because of the imbalance between the roads and backgrounds, and there will be a larger punishment if roads are wrongly classified as background. Since it is difficult to determine an appropriate weight of the loss function for a given image, an ensemble method based on spatial consistency (SC) is proposed. The result maps that are obtained from the FCNs with different loss functions are fused in our proposed ensemble strategy, which also avoids the determination of weights. Our method is tested using the Massachusetts road data set, and it was proven to be effective compared with the base fully convolutional model according to our experimental result.

Sparse representation (SR) method has the advantages of good category distinguishing performance, noise robustness, and data adaptiveness. In this article, a multi-feature weighted sparse graph (MWSG) is presented for synthetic aperture radar (SAR) image analysis. First, multiple types of features are extracted to fully describe the characteristics of SAR image. Then, multiple SRs of samples in multiple feature spaces are obtained by solving a weighted joint SR model, in which the weight is the Gaussian kernel distance among samples. Moreover, a new fusion mechanism is given to integrate multiple weighted SRs, which aims to eliminate the negative influence of the singular data, so the MWSG is obtained. Afterward, the brief steps of the SAR image segmentation and semisupervised classification based on MWSG are stated. A series of experiments on the simulated and real SAR images shows that the MWSG has better performance than other existing relevant methods.

Hyperspectral image (HSI) unmixing is an important issue of research due to its effect on the subsequent processing of HSIs. Recently, the sparse regression method with spatial information has been successfully applied in hyperspectral unmixing (HU). However, most sparse regression methods ignore the difference in spatial structure handling with only one sparse constraint. In fact, the pixels in detail regions are more likely to be severely mixed with more endmembers participated, and the sparsity degree of its corresponding abundances is relatively low. Considering the sparsity difference of abundances, a sketchbased region adaptive sparse unmixing applied to HSI is proposed in this article. Inspired by the vision computing theory, we use the region generation algorithm based on a sketch map to differentiate the homogeneous regions and detail regions. Then, the abundances of these two kind regions in HSIs are separately constrained by sparse regularizers of L1/2and L1with a proposed manifold constraint. Our method not only makes full use of the spatial information in HSIs but also exploits the latent structure of data. The encouraging experimental results on three data sets validate the effectiveness of our method for HU.

For an image retrieval system, the metric to measure the similarity between target images and queries greatly affects its retrieval performance. However, most of the existing metrics are based on single distance metric, which has been shown lack of robustness for different kinds of queries. In this work, we view the metric learning task as an optimization problem for a robust combination of existing metrics, where the objective function is data-driven rather than analytical. Our contribution is two-fold. Firstly, considering the robustness of image retrieval systems, we formulate the optimization problem as a multiobjective optimization with both the average and worst retrieval performance (precision) for different kinds of queries as objectives. Secondly, we apply a popular multi-objective evolutionary algorithm, NSGA-II, to search the optimal combined metric. With the experiment on two different datasets for image retrieval, we find that the proposed algorithm can find combined metrics with better and more robust retrieval performance than other existing single metrics.

In this paper, a stepwise method which consists of two main steps is proposed to tackle change detection in large-scale polarimetric Synthetic Aperture Radar (SAR) images. First, down-sample two registered polarimetric SAR images and calculate the corresponding Difference Image (DI), then spatial localization is conducted to orient sub-region which contains changed area in high probability. Second, polarimetric SAR sub-images are selected and they are trained together with down-sampled whole images by Convolutional Neural Network (CNN), where changed areas in sub-region are revealed and the next sub-region is generated accordingly. Repeat these two steps until all interested regions are detected. In general, it collects the whole but coarse information at first to locate important domains with changed areas and then analyzes them for accurate detection result and generate the next sub-region for further detection. Experiment results show that the proposed method performs well in detecting changed areas in large-scale polarimetric SAR images.

In this paper, a novel ternary weight convolution neural network (TWCNN) is proposed for band selection of hyperspectral images. TWCNN constructs deep-wise convolution layer with 1×1 filters as the first layer of the network, which is used for band selection. In the deep-wise convolution layer, weights are constrained to -1, 0, or 1. -1 and 1 represent that the corresponding band is selected, while 0 indicates it's not. TWCNN constructs subsequent layers to extract features and classify for selected spectral bands. It combines band selection, feature extraction and classification into a unified optimization procedure, which makes it to achieve end-to-end band selection and classification. Furthermore, the constraints of the number of spectral bands is added to the cost function of TWCNN. The specific number of spectral bands can be selected. The experiment results show that the proposed model provides a competitive result to state-of-the-art methods.

The following topics are dealt with: remote sensing; geophysical image processing; synthetic aperture radar; radar imaging; remote sensing by radar; image classification; learning (artificial intelligence); feature extraction; vegetation; image resolution.

Object detection in optical remote sensing video (ORSV) is a new trend which makes it possible for obtaining richer information in more complicated and diverse situations. However, the small objects are blurred in the videos captured from optical sensor assembled in satellite, limited by the devices and natural weather. The concept of weak object is defined in this situation that the objects are extremely small and hardly detected with only one static image. Therefore, we propose a simple but efficient method for weak moving object detection in ORSV by combining the temporal information from neighbor frames and spatial features from image pixels. First, we compute the difference map between two adjacent frames, and stack it with original RGB channel so that a (1+3)-channel input data is made. Then, a motion-drive D-RGB (difference map with RGB image) fusion network is developed to obtain the feature map of this (1+3)-channel data. To adapt unusual scale in ORSV images, based on statistical prior objects size, we change the size of anchor box in original Faster R-CNN. The proposed method is demonstrated to improve the mean average precision on detecting weak moving remote sensing objects.

Hashing, a useful solution for approximate nearest neighbor (ANN) search, is popular for large-scale image retrieval. In this paper, we presents a deep supervised hashing model for remote sensing image retrieval (RSIR) in the framework of generative adversarial networks (GAN), named GAN-assist Hashing (GAAH). First, to learn the compact and useful hash codes from the images, we define a novel loss function for the generator. The loss function mainly consists of classification, similarity, and bits entropy terms. The classification term makes the hash code is discriminative, the similarity term constrains the binary code is similarity preserving, and the bits entropy term assures the learned code is low-error in the quantization. Second, we construct the unique "true" matrix with the uniform distribution as the input of discriminator to limit the leaned hash codes are bit balanced. The final hash code is learned by a minimax optimization. The positive experimental results on a ground-truth remote sensing image archive validate the usefulness of our GAAH model. Compare with the popular deep hashing methods, our GAAH achieves improved performance.

As an useful solution of the approximate nearest neighbor (ANN) search, hashing attracts growing attention in the topic of large-scale image retrieval. In this paper, we propose a semi-supervised deep hashing method based on the adversarial autoencoder (AAE) network for remote sensing image retrieval (RSIR), and we name it SSHAAE. Here, we assume the RS images have been represented by the visual features, and the target of our SSHAAE is mapping those features into the binary codes. First, a hashing layer is adopted to replace the part of original latent layer in AAE. In addition, the classical reconstruction loss function is selected to generate the hash code. Second, two discriminators are added simultaneously to make sure the hash code is bit balanced and the generated label variable is one-hot. Third, we design the hash loss function to guarantee the obtained hash code is discriminative, similarity persevering, and low quantization error. The presented SSHAAE model can be trained by the minimax optimization. The encouraging experimental results counted on a high-resolution RS image archive demonstrate our SSHAAE model is effective to RSIR.

Multiple instance learning (MIL) can be used for solving the imprecisely labeled hyperspectral target detection problems, which only needs the label of an area containing some targets. Furthermore, existing methods decompose this task into a target signature learning task and a follow-on similarity measurement between the estimated signature and the test points. In this paper, we propose a deep multiple instance learning method based on self-attention mechanism, in which the max operation and 1D convolution neural network (1D CNN) are adopted to realize an end-to-end hyperspectral target detection structure without learning the target signature. In the proposed deep MIL target detection method, self-attention mechanism with max operation has advantage in estimating the labels of instances from the positive bag via calculating the contribution of each instance to the bag-level classification. The simulated and real hyperspectral target detection experiments are shown to illustrate the performance of the method.

The approaches for analyzing the polarimetric scattering matrix of polarimetric synthetic aperture radar (PolSAR) data have always been the focus of PolSAR image classification. Generally, the polarization coherent matrix and the covariance matrix obtained by the polarimetric scattering matrix are used as the main research object to extract features. In this paper, we focus on the original polarimetric scattering matrix and propose a polarimetric scattering coding way to deal with polarimetric scattering matrix and obtain a close complete feature. This encoding mode can also maintain polarimetric information of scattering matrix completely. At the same time, in view of this encoding way, we design a corresponding classification algorithm based on the convolution network to combine this feature. Based on the polarimetric scattering coding and convolution neural network, the polarimetric convolutional network is proposed to classify PolSAR images by making full use of polarimetric information. We perform the experiments on the PolSAR images acquired by AIRSAR and RADARSAT-2 to verify the proposed method. The experimental results demonstrate that the proposed method get better results and has huge potential for PolSAR data classification. Source code for polarimetric scattering coding is available at https://github.com/liuxuvip/Polarimetric-Scattering-Coding.

Based on a generative adversarial network (GAN), a novel version named Task-Oriented GAN is proposed to tackle difficulties in PolSAR image interpretation, including PolSAR data analysis and small sample problem. Besides two typical parts in GAN, i.e., generator (G-Net) and discriminator (D-Net), there is a third part named TaskNet (T-Net) in the Task-Oriented GAN, where T-Net is employed to accomplish a certain task. Two tasks, PolSAR image classification and clustering, are studied in this paper, where T-Net acts as a Classifier and a Clusterer, respectively. The learning procedure of Task-Oriented GAN consists of two main stages. In the first stage, G-Net and D-Net vie with each other like that in a general GAN; in the second stage, G-Net is adjusted and oriented by T-Net so that more samples, which are benefit for the task and called fake data, are generated. As a result, Task-Oriented GAN not only has the advantage of GAN (no-assumption data modeling) but also overcomes the disadvantage of GAN (task-free). After learning, fake data are employed to enrich training set and avoid overfitting; so Task-Oriented GAN performs well even if the manual-labeled data are small. To verify the effectiveness of T-Net, a visualized comparison is provided, where some fake digits generated from Task-Oriented GAN are illustrated along with that from GAN. What is more, considering that there is a great difference between PolSAR data and general data, in our PolSAR image classification and clustering tasks, the specific PolSAR information is inserted into the structure of the Task-Oriented GAN. This enables researchers to mine inherent information in PolSAR data without any data hypothesis and find ways for small sample problem at the same time. Experiment results tested on three PolSAR images show that the proposed method performs well in dealing with PolSAR image classification and clustering.

To detect changed areas in multitemporal polarimetric synthetic aperture radar (SAR) images, this paper presents a novel version of convolutional neural network (CNN), which is named local restricted CNN (LRCNN). CNN with only convolutional layers is employed for change detection first, and then LRCNN is formed by imposing a spatial constraint called local restriction on the output layer of CNN. In the training of CNN/LRCNN, the polarimetric property of SAR image is fully used instead of manual labeled pixels. As a preparation, a similarity measure for polarimetric SAR data is proposed, and several layered difference images (LDIs) of polarimetric SAR images are produced. Next, the LDIs are transformed into discriminative enhanced LDIs (DELDIs). CNN/LRCNN is trained to model these DELDIs by a regression pretraining, and then a classification fine-tuning is conducted with some pseudolabeled pixels obtained from DELDIs. Finally, the change detection result showing changed areas is directly generated from the output of the trained CNN/LRCNN. The relation of LRCNN to the traditional way for change detection is also discussed to illustrate our method from an overall point of view. Tested on one simulated data set and two real data sets, the effectiveness of LRCNN is certified and it outperforms various traditional algorithms. In fact, the experimental results demonstrate that the proposed LRCNN for change detection not only recognizes different types of changed/unchanged data, but also ensures noise insensitivity without losing details in changed areas.

This paper presents a novel method for hyperspectral anomaly detection considering the spectral redundancy and exploiting spectral-spatial information at the same time. We proposed a Hyper-Laplacian regularized low-rank tensor decomposition method combing with dimensionality reduction framework. Firstly, k-means++ algorithm is implemented to spectral bands and centers of each group are selected to reduce the HSI dimensionality in spectral direction. To jointly utilize spectral-spatial information, the cubic data (two spatial dimensions and one spectral dimension) is treated as a 3-order tensor. Then the non-local self-similarity is fully explored in our method. For the reason to reduce the ringing artifacts caused by over-lapped segmentation in exploring the non-local self-similarity, we introduce the hyper-Laplacian constrained low-rank tensor decomposition and we get the separated background and residual parts. Finally, to eliminate the effect of Gaussian noise, we use local-Rx basic detector to detect the residual matrix. Experimental results on two real hyperspectral data sets verified the effectiveness of the proposed algorithms for HSI anomaly detection.

Relevance feedback (RF) is a popular reranking technique, which aims at improving the performance of image retrieval by taking the user's opinions into account. In this paper, we introduce a new RF method, named circular relevance feedback (CRF), to enhance the behavior of remote sensing image retrieval (RSIR). Instead of the manual selection used in the common RF method, we adopt the active learning (AL) algorithm to select the samples from the initial results automatically in each RF iteration. Moreover, to ensure the selected images are representative and informative enough, we choose different AL algorithms to complete the different RF processes. Finally, the contributions of all AL-driven RF methods are integrated using a circular fusion scheme. The encouraging experimental results on the ground truth RS image archive illustrate that our CRF is useful for enhancing the performance of RSIR. In addition, compared with many existing RF methods, our CRF achieves improved behavior.

Panchromatic (PAN) and multispectral (MS) imagery classification is one of the hottest topics in the field of remote sensing. In recent years, deep learning techniques have been widely applied in many areas of image processing. In this paper, an end-to-end learning framework based on deep multiple instance learning (DMIL) is proposed for MS and PAN images’ classification using the joint spectral and spatial information based on feature fusion. There are two instances in the proposed framework: one instance is used to capture the spatial information of PAN and the other is used to describe the spectral information of MS. The features obtained by the two instances are concatenated directly, which can be treated as simple fusion features. To fully fuse the spatial–spectral information for further classification, the simple fusion features are fed into a fusion network with three fully connected layers to learn the high-level fusion features. Classification experiments carried out on four different airborne MS and PAN images indicate that the classifier provides feasible and efficient solution. It demonstrates that DMIL performs better than using a convolutional neural network and a stacked autoencoder network separately. In addition, this paper shows that the DMIL model can learn and fuse spectral and spatial information effectively, and has huge potential for MS and PAN imagery classification.

A new reranking method, fusion similarity-based reranking, is proposed in this letter to improve the performance of synthetic aperture radar (SAR) image retrieval. First, the top ranked SAR images within the initial retrieval results are picked for reranking. Considering the negative influence of the speckle noise, three SAR-oriented visual features are selected to represent them. In addition, the different relevance scores corresponding to an SAR image are estimated in various modalities (i.e., different feature spaces). Second, a fusion similarity is defined under the relevance score space to measure the resemblance between two SAR images. This fusion similarity is calculated using the modal-image matrix, which is construed by the estimated scores to integrate the contributions of all modalities. Finally, an existing reranking function is adopted to rerank the SAR images with the help of the estimated scores and calculated fusion similarities. The positive experimental results demonstrate that our reranking method is effective and efficient.

Image reranking is a popular postprocessing method for remote sensing image retrieval (RSIR), which aims at enhancing the initial retrieval performance. In general, it takes either users' opinions or the relationships between images into consideration to find an optimal reranked list based on the initial retrieved results. In this paper, we present a reranking method for improving RSIR, which is named two-stage reranking (TSR). Suppose the k-nearest neighbors of a query RS image have been obtained by the initial retrieval. The first step of our TSR is to edit these neighbors using the editing scheme. A handful of informative and representative RS images are selected by the active learning algorithm, and their binary labels are provided by the users relative to the query image. Then, a binary classifier is trained using the selected RS images and their labels to classify the rest of the neighbors. Finally, both classification results and rank information in the initial retrieval results are considered to decide which neighbor should be excluded. In the next step, the remaining RS images are reranked by the proposed reranking scheme, i.e., multisimilarity fusion reranking. Both the user's experience and image relationships are taken into account in TSR to ensure the performance of the reranking. The efficiency and the robustness of our method are validated by experiments conducted on two different types of RS images. Compared with the existing visual reranking approaches, our method achieves improved performance.