The rapid advancement of foundation models has revolutionized visual representation learning in a self-supervised manner. However, their application in remote sensing (RS) remains constrained by a fundamental gap: existing models predominantly handle single or limited modalities, overlooking the inherently multi-modal nature of RS observations. Optical, synthetic aperture radar (SAR), and multi-spectral data offer complementary insights that significantly reduce the inherent ambiguity and uncertainty in single-source analysis. To bridge this gap, we introduce RingMoE, a unified multi-modal RS foundation model with 14.7 billion parameters, pre-trained on 400 million multi-modal RS images from nine satellites. RingMoE incorporates three key innovations: 1) A hierarchical Mixture-of-Experts (MoE) architecture comprising modal-specialized, collaborative, and shared experts, effectively modeling intra-modal knowledge while capturing cross-modal dependencies to mitigate conflicts between modal representations; 2) Physics-informed self-supervised learning, explicitly embedding sensor-specific radiometric characteristics into the pre-training objectives; 3) Dynamic expert pruning, enabling adaptive model compression from 14.7B to 1B parameters while maintaining performance, facilitating efficient deployment in Earth observation applications. Evaluated across 23 benchmarks spanning six key RS tasks (i.e., classification, detection, segmentation, tracking, change detection, and depth estimation), RingMoE outperforms existing foundation models and sets new SOTAs, demonstrating remarkable adaptability from single-modal to multi-modal scenarios. Beyond theoretical progress, it has been deployed and trialed in multiple sectors, including emergency response, land management, marine sciences, and urban planning.

Vision foundation models in remote sensing have been extensively studied due to their superior generalization on various downstream tasks. Synthetic Aperture Radar (SAR) offers all-day, all-weather imaging capabilities, providing significant advantages for Earth observation. However, establishing a foundation model for SAR image interpretation inevitably encounters the challenges of insufficient information utilization and poor interpretability. In this paper, we propose a remote sensing foundation model based on complex-valued SAR data, which simulates the polarimetric decomposition process for pre-training, i.e., characterizing pixel scattering intensity as a weighted combination of scattering bases and scattering coefficients, thereby endowing the foundation model with physical interpretability. Specifically, we construct a series of scattering queries, each representing an independent and meaningful scattering basis, which interact with SAR features in the scattering query decoder and output the corresponding scattering coefficient. To guide the pre-training process, polarimetric decomposition loss and power self-supervised loss are constructed. The former aligns the predicted coefficients with Yamaguchi coefficients, while the latter reconstructs power from the predicted coefficients and compares it to the input image's power. The performance of our foundation model is validated on nine typical downstream tasks, achieving state-of-the-art results. Notably, the foundation model can extract stable feature representations and exhibits strong generalization, even in data-scarce conditions.

Remote sensing foundation models largely break away from the traditional paradigm of designing task-specific models, offering greater scalability across multiple tasks. However, they face challenges such as low computational efficiency and limited interpretability, especially when dealing with large-scale remote sensing images. To overcome these, we draw inspiration from heat conduction, a physical process modeling local heat diffusion. Building on this idea, we are the first to explore the potential of using the parallel computing model of heat conduction to simulate the local region correlations in high-resolution remote sensing images, and introduce RS-vHeat, an efficient multi-modal remote sensing foundation model. Specifically, RS-vHeat 1) applies the Heat Conduction Operator (HCO) with a complexity of $O(N^{1.5})$ and a global receptive field, reducing computational overhead while capturing remote sensing object structure information to guide heat diffusion; 2) learns the frequency distribution representations of various scenes through a self-supervised strategy based on frequency domain hierarchical masking and multi-domain reconstruction; 3) significantly improves efficiency and performance over state-of-the-art techniques across 4 tasks and 10 datasets. Compared to attention-based remote sensing foundation models, we reduce memory usage by 84\%, FLOPs by 24\% and improves throughput by 2.7 times. The code will be made publicly available.

For more efficient generalization to unseen domains (classes), most Few-shot Segmentation (FSS) would directly exploit pre-trained encoders and only fine-tune the decoder, especially in the current era of large models. However, such fixed feature encoders tend to be class-agnostic, inevitably activating objects that are irrelevant to the target class. In contrast, humans can effortlessly focus on specific objects in the line of sight. This paper mimics the visual perception pattern of human beings and proposes a novel and powerful prompt-driven scheme, called "Prompt and Transfer" (PAT), which constructs a dynamic class-aware prompting paradigm to tune the encoder for focusing on the interested object (target class) in the current task. Three key points are elaborated to enhance the prompting: 1) Cross-modal linguistic information is introduced to initialize prompts for each task. 2) Semantic Prompt Transfer (SPT) that precisely transfers the class-specific semantics within the images to prompts. 3) Part Mask Generator (PMG) that works in conjunction with SPT to adaptively generate different but complementary part prompts for different individuals. Surprisingly, PAT achieves competitive performance on 4 different tasks including standard FSS, Cross-domain FSS (e.g., CV, medical, and remote sensing domains), Weak-label FSS, and Zero-shot Segmentation, setting new state-of-the-arts on 11 benchmarks.

Aerial Remote Sensing (ARS) vision tasks present significant challenges due to the unique viewing angle characteristics. Existing research has primarily focused on algorithms for specific tasks, which have limited applicability in a broad range of ARS vision applications. This paper proposes RingMo-Aerial, aiming to fill the gap in foundation model research in the field of ARS vision. A Frequency-Enhanced Multi-Head Self-Attention (FE-MSA) mechanism is introduced to strengthen the model's capacity for small-object representation. Complementarily, an affine transformation-based contrastive learning method improves its adaptability to the tilted viewing angles inherent in ARS tasks. Furthermore, the ARS-Adapter, an efficient parameter fine-tuning method, is proposed to improve the model's adaptability and performance in various ARS vision tasks. Experimental results demonstrate that RingMo-Aerial achieves SOTA performance on multiple downstream tasks. This indicates the practicality and efficacy of RingMo-Aerial in enhancing the performance of ARS vision tasks.

Siamese network based trackers develop rapidly in the field of visual object tracking in recent years. The majority of Siamese network based trackers now in use treat each channel in the feature maps generated by the backbone network equally, making the similarity response map sensitive to background influence and hence challenging to focus on the target region. Additionally, there are no structural links between the classification and regression branches in these trackers, and the two branches are optimized separately during training. Therefore, there is a misalignment between the classification and regression branches, which results in less accurate tracking results. In this paper, a Target Highlight Module is proposed to help the generated similarity response maps to be more focused on the target region. To reduce the misalignment and produce more precise tracking results, we propose a corrective loss to train the model. The two branches of the model are jointly tuned with the use of corrective loss to produce more reliable prediction results. Experiments on 5 challenging benchmark datasets reveal that the method outperforms current models in terms of performance, and runs at 38 fps, proving its effectiveness and efficiency.

Instance segmentation of remote sensing images (RSIs) is an essential task for a wide range of applications such as land planning and intelligent transport. Instance segmentation of RSIs is constantly plagued by the unbalanced ratio of foreground and background and limited instance size. And most of the instance segmentation models are based on deep feature learning and contain operations such as multiple downsampling, which is harmful to instance segmentation of RSIs, and thus the performance is still limited. Inspired by the recent superior performance of prompt learning in visual tasks, we propose a new prompt paradigm to address the above issues. Based on the existing instance segmentation model, first, a local prompt module is designed to mine local prompt information from original local tokens for specific instances; second, a global-to-local prompt module is designed to model the contextual information from the global tokens to the local tokens where the instances are located for specific instances. Finally, a proposal’s area loss function (PAreaLoss) is designed to add a decoupling dimension for proposals on the scale to better exploit the potential of the above two prompt modules. It is worth mentioning that our proposed approach can extend the instance segmentation model to a promptable instance segmentation model, i.e., to segment the instances with the specific boxes’ prompt. The time consumption for each promptable instance segmentation process is only 40 ms. This article evaluates the effectiveness of our proposed approach based on several existing models in four instance segmentation datasets of RSIs, and thorough experiments prove that our proposed approach is effective for addressing the above issues and is a competitive model for instance segmentation of RSIs.

Aircraft detection in synthetic aperture radar (SAR) imagery is significant due to its critical role in various applications, including surveillance, reconnaissance, and security. However, given the background interference and discreteness of aircraft scattering, detectors are prone to acquire unremarkable aircraft features. These factors lead to false alarms and present difficulties in locating aircraft accurately. This article proposes an innovative scattering topology cue-based network (STC-Net), which enhances aircraft discriminability and more accurately evaluates the quality of the prediction results. We model the aircraft with the star topology (ST), which not only emphasizes critical components like the nose and wings but also explicitly links them as a cohesive unit. Based on the cue of ST, the ST space fusion module (ST-SFM) and the ST channel attention module (ST-CAM) are designed. The former integrates discrete components to reestablish the aircraft features based on neighboring information of ST, while the latter suppresses background interference to highlight the aircraft by exploiting node information of ST. In addition, completeness and consistency loss (CCLoss) function that includes the completeness-aware label and the positive sample weighting function is introduced. The completeness-aware label describe the localization accuracy by incorporating the degree of overlap of predicted results on ST, while the positive sample weighting function enhances the consistency of the classification and localization branches. Furthermore, experiments conducted on the Gaofen-3 SAR aircraft detection dataset (GF3ADD) and the publicly available SAR-AIRcraft-1.0 dataset demonstrate the effectiveness and generalizability of STC-Net, with our method achieving state-of-the-art performance.

Recently, multimodal large language models (MLLMs) have shown excellent reasoning capabilities in various fields. Most of the existing remote sensing (RS) MLLMs solve image-level text generation problems (e.g., image captioning), but ignore the core issues of object-level recognition, location, and multitemporal changes in the field of RS. In this article, we propose RingMoGPT, a multimodal foundation model that unifies vision, language, and localization. Based on the idea of domain adaption, RingMoGPT can complete training by fine-tuning only a few parameters. To make the model capable of object detection and change captioning, we further propose a location- and instruction-aware querying transformer (Q-Former) and a change detection module, respectively. To improve the performance of RingMoGPT, we carefully design the pretraining dataset and the instruction-tuning dataset. The pretraining dataset contains over a half million high-quality image and text pairs, which are generated through a low-cost and efficient data generation paradigm. The instruction-tuning dataset contains more than 1.6 million question-answer pairs, including six downstream tasks: scene classification, object detection, visual question answering (VQA), image captioning, grounded image captioning, and change captioning. Our experiments show that RingMoGPT performs well on six tasks, especially its ability to analyze multitemporal data changes and identify dense objects. We also verified the model under a zero-shot setting, and the results show that the proposed RingMoGPT also has good generalization ability in the face of new data.

Object detection and recognition (OD&R) based on deep learning is a hot topic in the application of synthetic aperture radar (SAR). These methodologies based on deep learning are inherently data-driven, which means that their performance is subjected to the corresponding datasets. Although existing datasets have included some common targets collected from real-valued intensity SAR images, there still exist some limitations in terms of quantity, categories, diversities, and data domain. Hence, it is urgent to establish a large-quantity benchmark for fine-grained OD&R on complex-valued SAR images, which contains rich signal-domain features well coupled with classical physical modeling. In addition, considering the unique imaging characteristics and diverse imaging conditions, some important attribute information, such as incidence and attitude angles, is necessary to be attached. In this article, we propose a novel benchmark dataset with more than 340k instances for fine-grained OD&R based on single-look complex (SLC) SAR images, which is named FAIR-CSAR. We collected complex-valued SAR images with a resolution of 1–5 m from 175 entire images of Gaofen-3 covering 32 cities and multiple sea areas worldwide. All instances in the FAIR-CSAR are annotated by oriented bounding boxes (OBBs), covering five major categories and 22 subcategories. Compared with existing datasets dedicated to OD&R, the FAIR-CSAR dataset has four particular advantages: 1) it contains complex-valued SAR images from various acquisition modes and polarization modes, including full-scale signal-domain features for object recognition; 2) it is much larger than other existing OD&R datasets in terms of quantity of instances; 3) it provides more fine-grained category annotation and more detailed attribute information; and 4) it provides more challenging images with some common imaging phenomena, such as speckle noise and azimuth ambiguities. To establish a baseline adapted for SLC SAR images, a multidomain feature extraction and fusion network (MDNet) is proposed as a novel framework to mine detailed information underlying various domains. A series of state-of-the-art (SOTA) algorithms are applied on the FAIR-CSAR to build the fine-grained OD&R benchmark. Experimental results indicate that FAIR-CSAR is closer to practical application and more challenging than existing datasets for SAR images.

Currently, the remote sensing fine-grained rigid object detectors mainly face two challenges: fuzzy localization and inaccurate classification (including misclassifying and multi-classifying). Firstly, mainstream detectors based on the “dense prediction” paradigm suffer from a misalignment between anchor points (APs) and ground truths (GTs). Commonly, they adopt multi-stage regression as a solution, which brings an ambiguous sample definition problem and redundant computational costs. Secondly, two factors seriously affect the classification. They are the insufficient sample learning caused by the long-tail distribution of categories, and the difficulty in extracting discriminative features caused by slight inter-class variance. To address the issues above, we propose an efficient aligning and completing detector (ACDet) based on a single-stage structure. Firstly, the Adaptive Anchor Alignment Mechanism decouples the centripetal sampling bias in the categorical features and leverages it to learn the APs aligned with GTs. It is plug-and-play, requiring no additional supervision annotations. Secondly, a novel Online Tail-sample Supplementation algorithm is proposed. It dynamically maintains class balance during training and can be easily added as post-processing behind existing sample assignment strategies. Thirdly, an Adaptive Group Perceptron is designed to effectively enrich the diversity of features and enhance the model’s ability to extract discriminative features. Experiments on four public datasets demonstrate that the proposed ACDet achieves the state-of-the-art (SOTA) level, even surpassing competitive multi-stage detectors, with fewer computing resources and a faster inference speed. The code will be public after the paper is published.

Few-shot object detection (FSOD) has garnered widespread attention in recent years, which makes it possible to learn novel classes with only a handful of labeled samples. Due to the obvious long-tail distribution of remote sensing (RS) data and serious challenges in data labeling, FSOD holds greater practical application value in RS. At present, the FSOD algorithms with fine-tuning of some parameters have attracted much attention due to their stronger incremental learning capacity. However, given large intraclass scale variations and small interclass feature differences of RS objects, we still need to place greater emphasis on the localization and classification of objects. In this article, we introduce the RoI feature refinement (RIFR) method for FSOD in RS imagery, which adopts a powerful training pipeline to better balance attention to the performance of base and novel classes. Aiming at large intraclass scale variations of RS objects, we design a scale-aware feature compensation module (SAFCM). By compensating for insufficient scale information, the model’s ability to perceive scale variations of the same class objects has been enhanced. Considering the high similarity among RS classes, we come up with a prototype trihard (PT) loss. It achieves the effect of interclass separability by constraining the relationship between samples and prototypes. Thus, the issue of interclass confusion has been resultfully solved. Comprehensive experiments on three datasets, DIOR, NWPU VHR-10.v2, and FAIR1M-Airplane, can showcase the efficacy of our RIFR method, and it can implement the most outstanding performance currently. The code will be available at:https://github.com/ningerhhh/RIFR.

With the development of large language models (LLMs) and remote sensing technology, visual language (VL) tasks in the field of remote sensing have attracted more and more research attention. Commonly used VL datasets currently usually focus on the overall scene of the image, lacking the description of instance-level details such as location, size, category and so on. More importantly, users are usually not allowed to directly intercept regions in the image to ask questions through these datasets. However, the instance-level question answering based on these information is of great significance for target extraction in practical applications. In this manuscript, we build an innovative and challenging dataset FAIR1M-GQA. It unlocks the ability of the model to learn directly from text input and text output both with region coordinates, which are directly linked to fine-grained objects in remote sensing images. We experiment our dataset to verify the feasibility of the relevant task and provide the benchmark results.

Compared with binary change detection (BCD), semantic change detection (SCD) further provides the category information of bitemporal changed regions which is significant for the practical application of Earth Observation. Although the recently proposed triple-branch structures including one BCD branch and two classification branches can effectively achieve the task balance, they still need to employ the carefully designed difference extraction module and branch interactions to capture the bitemporal correlations, which increases the complexity of the semantic information utilization. In this paper, we propose a new triple-branch network named JFRNet to tackle this challenge. From the perspective of the SCD process, because the category information and the change information are both derived from bitemporal images, we take the joint bitemporal features as the unified input, which can help each branch perceive the bitemporal semantic correlations without any additional interaction operations. From the perspective of the SCD structure, we introduce the convolutional attention fusion module (CAFM) and the convolutional attention refinement module (CARM) to unify the branch structure, which can help our model refine the unique semantic information without any specially designed difference extraction modules. Extensive experiment results on three available datasets indicate that compared with the baseline methods, our proposed JFRNet successfully simplifies the reasoning process and obtains the better SCD performance.

A weakly supervised object detection (WSOD) task is to train a detector with only image-level labels provided. Except for the training difficulty introduced by weaker annotations, the inherent complexity of the remote-sensing images (RSIs) also adds to the challenge. To boost the detector’s localization accuracy, we aim to exploit more semantic information contained in images and help improve the general robustness of the model. Noticing previous methods tend to focus on the most discriminative part of an object, we design a self-training-based network that leverages local semantic features. To this end, we develop a semantic-balanced localization module (SBLM) that distinguishes foreground from background and accurate proposals from incomplete ones, by leveraging a balance of region of interest (ROI) and its context information. Moreover, we find that the self-training strategy highly relies on the quality of pseudo-ground-truth boxes. Motivated by this possible lack of robustness, we design a comprehensive clustering module (CCM) and saliency-based proposal filtering (SPF) module that select pseudo-ground truth more comprehensively under supervision. To be more specific, CCM aims to reduce the arbitrariness during assigning pseudo-labels by considering multiple categorical vectors simultaneously. Salient object detection (SOD) is applied in the SPF module to help evaluate the quality of the chosen pseudo-ground-truth boxes. The detection performance is significantly boosted with the proposed method. Extensive experiments conducted on the NWPU VHR-10.v2 dataset and the DIOR dataset validate that the proposed model outperforms the previous state-of-the-art methods favorably with an mAP of 64.9% and 28.1%, respectively.

Due to the diversity of acquisition conditions and imaging mechanisms in remote sensing, the generalization of semantic segmentation models trained with labeled data in the source domain to other unlabeled target domains is hindered. Existing mainstream self-training-based methods provide pseudo-labels to target data as ground truth to utilize target domain evidence for unsupervised domain adaptation (UDA). However, the label shift and domain gap between different domains inevitably introduce noise into pseudo-labeled target data, that is, misclassified pixels. As a consequence, we present a dynamic loss correction (DLC) framework for cross-domain semantic segmentation, which mitigates domain discrepancy by formally modeling the noise distribution of pseudo-labels in the target domain with noise transition matrix (NTM). Specifically, to promote the model output to fit the true label distribution, we employ the high-order consistency information of neighbor representations to estimate NTM and correct the supervision signal without heuristically setting anchors. Furthermore, smooth geometric constraints are introduced to regularize the mutual improvement of NTM derivation and segmentation model optimization in a data-driven manner, thereby compensating for the lack of target domain knowledge. Extensive experimental results on four cross-domain remotely sensed segmentation tasks highlight the generalization capability and competitiveness of the presented method, including cross-scene, cross-band, and cross-modal transfer. Our results and code are available athttps://github.com/heqibin/dlc.

Most recent unmanned aerial vehicle (UAV) detectors focus primarily on general challenges such as uneven distribution and occlusion. However, the neglect of scale challenges, which encompass scale variation and small objects, continues to hinder object detection in UAV images. Although existing works propose solutions, they are implicitly modeled and have redundant steps, so detection performance remains limited. One specific work addressing the above scale challenges can help improve the performance of UAV image detectors. Compared to natural scenes, scale challenges in UAV images happen with problems of limited perception in comprehensive scales and poor robustness to small objects. We found that complementary learning is beneficial for the detection model to address the scale challenges. Therefore, the article introduces it to form our scale-robust complementary learning network (SCLNet) in conjunction with the object detection model. The SCLNet consists of two implementations and a cooperation method. In detail, one implementation is based on our proposed scale-complementary decoder and scale-complementary loss function to explicitly extract complementary information as a complement, named comprehensive-scale complementary learning (CSCL). Another implementation is based on our proposed contrastive complement network and contrastive complement loss function to explicitly guide the learning of small objects with the rich texture detail information of the large objects, named interscale contrastive complementary learning (ICCL). In addition, an end-to-end cooperation (ECoop) between two implementations and with the detection model is proposed to exploit each potential. In short, SCLNet forms a more comprehensive representation through feature complementary and improves the representation of small objects through interscale contrast, which in turn comes to improve scale robustness and detection performance. Thorough experiments prove the effectiveness of our SCLNet on Visdrone and UAVDT datasets, including the fact that the novel components included in SCLNet are effective and competitive with many CNN-based and transformer-based methods, among other aspects. In general, our SCLNet can effectively address scale challenges and is a competitive model in UAV image object detection.

Recently, there has been a growing interest in few-shot incremental object detection (FSIOD). It learns new tasks with limited data while mitigating catastrophic forgetting on previous tasks. However, existing FSIOD methods experience parameter changes after training on new tasks, causing a parameter competition issue among tasks. Additionally, the background information differs among various tasks, and using a common background weight for all tasks results in the background shift. Constrained by these two issues, existing methods only alleviate catastrophic forgetting and cannot wholly prevent the performance decline on previous tasks. Especially for complex remote sensing images with messy background, the models trained on new tasks exhibit noticeable performance drops on previous tasks. In this paper, we propose a novel FSIOD method via dual-frequency prompt to address these challenges, named FSIOD-DFP. It can completely eliminate catastrophic forgetting while mitigating over-fitting. Specifically, a dual-frequency prompt generator is designed to tackle the parameter competition issue. It decouples the frequency components of images to produce prompts that modify the images to adapt to the base model trained on previous tasks. Compared to traditional prompts, our generator introduces fewer parameters to address over-fitting for limited data and allows freezing the base model to maintain the performance of previous data. Besides, a self-regularization loss is introduced to guide the prompt-modified images to leverage the knowledge of the base model effectively. Furthermore, we propose a task-decoupled detection head to address the background shift problem. It separates the detection heads for new and previous tasks to resolve the conflict in the background between different tasks. In FSIOD-DFP, only a prompt generator and a novel detection head are added and fine-tuned when learning a new task. Experiments on three remote sensing object detection datasets demonstrate that our method achieves state-of-the-art performance on both new and previous tasks in all few-shot incremental settings.

With the continuous improvement of remote sensing satellite technology for earth observation, satellite video object tracking has become a new research field. Tracking aerial objects in satellite video facilitates highly accurate positioning over a wide coverage area which plays a crucial role in remote sensing scenes such as military analysis and disaster prevention. However, for such a data-driven task, most previous satellite video datasets focus on ground object tracking and lack attention to high-speed aerial objects. In this regard, a high-resolution satellite video dataset for detecting and tracking high-speed aerial objects (AIR-HSAO) is built including satellite videos from over 20 regions worldwide, with detailed annotations for approximately 1400 airplane trajectories. Based on the AIR-HSAO, an end-to-end online multi-object tracking method called MGTrack is proposed for high-speed aerial objects. Specifically, the motion prediction module is proposed to implicitly model object motion and associate objects. Guided by the predicted offsets, the feature warping and fusion module aligns and fuses temporal features, fully exploiting the temporal correlations of the video sequence to obtain more accurate feature representations for high-speed objects. Additionally, a trajectory-aware size-enhanced loss is designed to handle the problem of discontinuous trajectories and small-scale objects. Experimental results on AIR-HSAO demonstrate that the proposed MGTrack improves the tracking accuracy by 2.3 MOTA and 1.6 IDF1 compared to the state-of-the-art tracking methods. The code is released at https://github.com/RenLibo-aircas/AIR-HSAT-dataset.

Sidelobes commonly disturb synthetic aperture radar (SAR) image understanding and interpretation. Traditional spatially variant filtering algorithms achieve a superior tradeoff between sidelobe suppression and resolution preservation by means of adaptively calculating filtering parameters under some specific restrictions, such as filter design restriction and minimum amplitude constraint (MAC). These restriction aims to obtain an efficient analytical solution for filters, which is easy to calculate under unsupervised conditions. However, the restriction scope is so narrow that the suppression performance achieved by these filters is limited. Also, since the unsupervised optimization based on MAC indiscriminately minimizes amplitude, the main-lobe loss is unavoidable. To further improve the performance, a spatially variant convolution neural network (SVNN) is proposed, which consists of two core modules. One is the spatially variant filter generation (SVFG) module, adaptively generating superior spatially variant filters under more relaxed restrictions. The other is a paralleled shifted convolution (PSC) module, converting the signal format to achieve a fast and parallel spatially variant filtering process. Benefiting from more relaxed filter restrictions, the novel network successfully achieves better performance on sidelobe suppression. In addition, with supervised optimization based on another more accurate restriction, namely, minimum error constraint (MEC), the proposed algorithm also achieves superior main-lobe maintenance. All of them are validated by comparative experiments based on satellite data from GaoFen-3 and TerraSAR-X, and our proposed method achieves state-of-the-art performance. The entire project is available athttps://github.com/suoyuxi/SVNN.

Synthetic aperture radar (SAR) possesses significant advantages in aircraft detection due to its all-day and all-weather monitoring capability, but some unique problems in SAR images decrease the performance of aircraft detection. The speckle effect and excessive dynamic range are the most common problems that interfere with the visual features in SAR images and deteriorate detection performance. However, there lacks a detection-oriented image enhancement algorithm to collaboratively solve these two problems. An adaptive image enhancement algorithm is proposed to improve the performance of aircraft detection in SAR images. The proposed image enhancement algorithm provides a pseudocolor image through speckle suppression and channel combination, which consists of the speckle noise suppression channel, strong scattering feature enhancement channel, and weak scattering feature enhancement channel. The speckle noise suppression is achieved by a despeckle network, and the radiational feature enhancement channels are derived from an adaptive quantization method based on the characteristics of amplitude distribution. By optimizing the quality of the input image, the proposed image enhancement algorithm improves the performance of aircraft detection. Experiments based on datasets acquired by GaoFen-3 satellites indicate that the proposed algorithms significantly improve the detection performance of various types of detectors. The source project is available athttps://github.com/suoyuxi/ChannelEnhancement.

Existing aircraft detection methods primarily rely on loss function constraints to guide the learning of end-to-end detectors, which significantly diverges from the judgment logic of human experts. Inspired by how human experts make decisions based on cognitive features, this article introduces the concept of physical knowledge features. Leveraging three properties of physical knowledge features, we identify the circle grayscale (CG) feature of aircraft and propose a physical knowledge-guided network (PKG-Net). By embedding CG features into the supervised learning process, the network improves its proficiency in learning stable features, thereby enhancing detection accuracy. Within this network, a multiscale circular frequency filter module (MS-CFFM) is responsible for extracting and integrating aircraft CG features across different scales. Adaptive channel selection module (ACSM) selectively activates channels for learning CG features. The hybrid attention feature fusion module (HAFFM) focuses intensively on the central localization of aircraft and deep reinforcement of channels. Experimental results on the RSOD and UCAS-AOD datasets demonstrate that the proposed method surpasses existing techniques in accuracy, achieving state-of-the-art performance.

The essence of improving the effect of cross-modal image-text retrieval (CIR) lies in the finer-grained modeling of homogeneous features between modalities. However, in remote sensing (RS) scenarios, existing methods usually apply the image-sentence granular feature alignment paradigm, bringing significant difficulties to the fine-grained representation of homogeneous features between modalities. Besides, more complex background noise and extreme scale ranges of foreground targets are hard to distinguish, causing the feature mottle problem. To address the above issues, we propose a novel Semantic-guided Image-text Retrieval framework with Segmentation (SIRS). It is a multi-task joint learning framework for plug-and-play and end-to-end training RS CIR models efficiently, including Semantic-guided Spatial Attention (SSA) and Adaptive Multi-scale Weighting (AMW) modules. First, SSA introduces a background reconstruction branch based on noise perception and a semantic segmentation branch based on pixel-level prediction. It explores a joint learning strategy that concisely filters background noise and refines foreground features considerably. Secondly, AMW performs multi-scale weighting on various layers of feature map output by the encoder, effectively improving the learning efficiency of foreground targets at different scales. It is worth mentioning that SIRS outputs combination results with image and segmentation mask, which is not available in other methods. Based on the RSITMD dataset, we complete the semantic segmentation annotation RSITMD-SS to verify the performance of the proposed method. Sufficient and complete experiments verify the effectiveness of the proposed method. With SIRS, the mainstream SVP and CLIP-based methods improve about 7 mR and derive segmentation prediction with acceptable computational cost optionally. The code and associated dataset will be available on https://github.com/StarBurstStream0/SIRS.

Recently, the transformer has achieved notable success in remote sensing (RS) change detection (CD). Its outstanding long-distance modeling ability can effectively recognize the change of interest (CoI). However, in order to obtain the precise pixel-level change regions, many methods directly integrate the stacked transformer blocks into the UNet-style structure, which causes the high computation costs. Besides, the existing methods generally consider bitemporal or differential features separately, which makes the utilization of ground semantic information still insufficient. In this paper, we propose the multiscale dual-space interactive perception network (MDIPNet) to fill these two gaps. On the one hand, we simplify the stacked multi-head transformer blocks into the single-layer single-head attention module and further introduce the lightweight parallel fusion module (LPFM) to perform the efficient information integration. On the other hand, based on the simplified attention mechanism, we propose the cross-space perception module (CSPM) to connect the bitemporal and differential feature spaces, which can help our model suppress the pseudo changes and mine the more abundant semantic consistency of CoI. Extensive experiment results on three challenging datasets and one urban expansion scene indicate that compared with the mainstream CD methods, our MDIPNet obtains the state-of-the-art (SOTA) performance while further controlling the computation costs.

Few-shot object detection, expecting detectors to detect novel classes with a few instances, has made conspicuous progress. However, the prototypes extracted by existing meta-learning based methods still suffer from insufficient representative information and lack awareness of query images, which cannot be adaptively tailored to different query images. Firstly, only the support images are involved for extracting prototypes, resulting in scarce perceptual information of query images. Secondly, all pixels of all support images are treated equally when aggregating features into prototype vectors, thus the salient objects are overwhelmed by the cluttered background. In this paper, we propose an Information-Coupled Prototype Elaboration (ICPE) method to generate specific and representative prototypes for each query image. Concretely, a conditional information coupling module is introduced to couple information from the query branch to the support branch, strengthening the query-perceptual information in support features. Besides, we design a prototype dynamic aggregation module that dynamically adjusts intra-image and inter-image aggregation weights to highlight the salient information useful for detecting query images. Experimental results on both Pascal VOC and MS COCO demonstrate that our method achieves state-of-the-art performance in almost all settings. Code will be available at: https://github.com/lxn96/ICPE.

Knowledge distillation (KD) is a promising approach that facilitates the compact student model to learn dark knowledge from the huge teacher model for better results. Although KD methods are well explored in the 2D detection task, existing approaches are not suitable for 3D monocular detection without considering spatial cues. Motivated by the potential of depth information, we propose a novel distillation framework that validly improves the performance of the student model without extra depth labels. Specifically, we first put forward a perspective-induced feature imitation, which utilizes the perspective principle (the farther the smaller) to facilitate the student to imitate more features of farther objects from the teacher model. Moreover, we construct a depth-guided matrix by the predicted depth gap of teacher and student to facilitate the model to learn more knowledge of farther objects in prediction level distillation. The proposed method is available for advanced monocular detectors with various backbones, which also brings no extra inference time. Extensive experiments on the KITTI and nuScenes benchmarks with diverse settings demonstrate that the proposed method outperforms the state-of-the-art KD methods.

The construction of a basic model to extract generalized features from a large number of multimodal data is a new challenge in the field of remote sensing. Compared with natural scene images, When faced with a complex application scenario of remote sensing of multi-sensor acquisition, models that are suitable for a specific task are difficult to generalize to new scenarios. In this paper, we propose a model architecture based on the concepts of multi-domain representation and cross-domain fusion. By extracting strong generalization features from massive multi-modal data, a single foundation model can accomplish generalization interpretation for multiple downstream tasks. Experimental results show that the proposed model performs well on multiple downstream tasks, which validates the feasibility of the remote sensing cross-modal foundation model in the interpretation task.

Welcome to the concluding issue of IEEE Transactions on Computational Social Systems (TCSS) for the year 2023. We would like to seize this opportunity to extend our heartfelt appreciation and congratulations to all for your exceptional dedication and unwavering support. We eagerly anticipate further collaboration to enhance the publication quality and expedite the review process of TCSS in the upcoming year 2024.

Few-shot segmentation (FSS) is proposed to segment unknown class targets with just a few annotated samples. Most current FSS methods follow the paradigm of mining the semantics from the support images to guide the query image segmentation. However, such a pattern of ‘learning from others’ struggles to handle the extreme intra-class variation, preventing FSS from being directly generalized to remote sensing scenes. To bridge the gap of intra-class variance, we develop a Dual-Mining network named DMNet for cross-image mining and self-mining, meaning that it no longer focuses solely on support images but pays more attention to the query image itself. Specifically, we propose a Class-public Region Mining (CPRM) module to effectively suppress irrelevant feature pollution by capturing the common semantics between the support-query image pair. The Class-specific Region Mining (CSRM) module is then proposed to continuously mine the class-specific semantics of the query image itself in a ‘filtering’ and ‘purifying’ manner. In addition, to prevent the co-existence of multiple classes in remote sensing scenes from exacerbating the collapse of FSS generalization, we also propose a new Known-class Meta Suppressor (KMS) module to suppress the activation of known-class objects in the sample. Extensive experiments on the iSAID and LoveDA remote sensing datasets have demonstrated that our method sets the state-of-the-art with a minimum number of model parameters. Significantly, our model with the backbone of Resnet-50 achieves the mIoU of 49.58% and 51.34% on iSAID under 1-shot and 5-shot settings, outperforming the state-of-the-art method by 1.8% and 1.12%, respectively. The code is publicly available at https://github.com/HanboBizl/DMNet/.

Intelligent ship detection algorithms for synthetic aperture radar (SAR) images have achieved significant results in Earth observation applications. By learning features such as scale, shape and texture from samples, they can quickly locate and recognize ships in complex backgrounds. However, due to the lack of use of polarization features, the upper bound of detection performance is still limited, especially under poor image quality conditions such as ambiguous interference. To solve this, the dual-polarization image feature fusion network (DPFF-Net) is proposed. The key of it lies in adaptive mining, enhancement and fusion of polarization features through the designed siamese structure, polarization-aware enhancement block (PAEB) and dynamic gated fusion block (DGFB). With fully utilizing complementary information hidden between co-polarization and cross-polarization data, more comprehensive and accurate features are obtained and used as the detect head input. Thus, the proposed algorithm achieves state-of-the-art performance, and its effectiveness are validated by experiments on dual-polarization SAR datasets.

Multi-object tracking (MOT) in satellite videos is an essential topic with many applications, such as traffic monitoring and disaster response. However, many multi-object trackers that perform well in natural scenes show weak generalization in satellite videos due to low object discrimination caused by low spatial resolution and the widespread indistinguishable background, such as clouds and reflections. In this paper, we design a novel multi-object tracking framework called CFTracker (Cross-Frame Tracker) for satellite videos from the point of both network structure and training method. On the one hand, in network structure design, a cross-frame feature update module (CFU) is proposed to enhance object recognition and reduce the response to background noises by using rich temporal semantic information. On the other hand, we reveal that the picture-pair training approach used by the mainstream MOT network is not entirely conducive to the network learning temporal semantic information. To better grasp the cross-frame feature connections and output time-consistent motion predictions, we train CFTracker by a novel cross-frame training flow (CT). Experiments demonstrate the effectiveness of our CFTracker and obtain state-of-the-art tracking accuracy and precision of 72.9% score on the AIR-MOT dataset and 57.1% score on the VISO dataset. The code will be available online.

Few-shot object detection (FSOD) has received numerous attention due to the difficulty and time-consuming of labeling objects. Recent researches achieve excellent performance in a natural scene by only using a few instances of novel classes to fine-tune the last prediction layer of the model well-trained on plentiful base data. However, compared with natural scene objects with a single direction and small size variety, the direction and size of the objects in remote sensing images (RSIs) vary greatly. The methods proposed for the natural scene cannot be directly applied to RSIs. In this article, we first propose a strong baseline for RSIs. It fine-tunes all detector components acting on high-level features and effectively improves the performance of novel classes. Further analyzing the results of the baseline, we find that the error for novel classes is mainly concentrated in classification. It misclassifies novel classes as confusable base classes or backgrounds due to the difficulty in extracting generalized information from limited instances. As is well-known, text-modal knowledge can highly summarize the generalized and unique characteristics of categories. Thus, we introduce text-modal descriptions for each category and propose an FSOD method guided by TExt-MOdal knowledge, called TEMO. Specifically, a text-modal knowledge extractor and a cross-modal assembly module are proposed to extract text features and fuse the text-modal features into visual-modal features. The fused features greatly reduce the classification confusion of novel classes. Furthermore, we introduce a mask strategy and a separation loss to avoid over-fitting and ambiguity of text-modal features. Experimental results on detection in optical remote sensing images (DIOR), Northwestern Polytechnical University (NWPU), and fine-grained object recognition in high-resolution remote sensing imagery (FAIR1M) illustrate that our TEMO achieves state-of-the-art performance in all settings.

Building 3D reconstruction from remote sensing images has a wide range of applications in smart cities, photogrammetry and other fields. Methods for automatic 3D urban building modeling typically employ multi-view images as input to algorithms to recover point clouds and 3D models of buildings. However, such models rely heavily on multi-view images of buildings, which are time-intensive and limit the applicability and practicality of the models. To solve these issues, we focus on designing an efficient DSM estimation-driven reconstruction framework (Building3D), which aims to reconstruct 3D building models from the input single-view remote sensing image. Existing DSM estimation networks suffer from the imbalance between local features and global features, which leads to over-smooth DSM estimates at instance boundaries. To address this issue, we propose a Semantic Flow Field-guided DSM Estimation (SFFDE) network, which utilizes the proposed concept of elevation semantic flow to achieve the registration of local and global features. First, in order to make the network semantics globally aware, we propose an Elevation Semantic Globalization (ESG) module to realize the semantic globalization of instances. Further, in order to alleviate the semantic span of global features and original local features, we propose a Local-to-Global Elevation Semantic Registration (L2G-ESR) module based on elevation semantic flow. Our Building3D is rooted in the SFFDE network for building elevation prediction, synchronized with a building extraction network for building masks, and then sequentially performs point cloud reconstruction and surface reconstruction (or CityGML model reconstruction). On this basis, our Building3D can optionally generate CityGML models or surface mesh models of the buildings. Extensive experiments on ISPRS Vaihingen and DFC2019 datasets on the DSM estimation task show that our SFFDE significantly improves upon state-of-the-art and δ1, δ2and δ3metrics of our SFFDE are improved to 0.595, 0.897 and 0.970. Furthermore, our Building3D achieves impressive results in the 3D point cloud and 3D model reconstruction process.

Continual learning is an effective way to overcome catastrophic forgetting (CF) in incremental learning for semantic segmentation. The existing continual semantic segmentation (CSS) methods of remote sensing (RS) ignore the semantic relationships among pixels across different images, which will lead to disappointing segmentation results, such as edge pixel misclassification and small object omission. In this paper, we propose a framework for modeling cross-image semantic relationship dependencies (MiCro), which aims to learn an inter-class separable and intra-class cohesive feature space from the pixel relationships across various images to ensure that learned categories can prevent CF in the incremental process. Specifically, we exploit the relationships among pixels of images in mini-batch to construct three losses: (a) Cross-image feature relationship distillation (CFRD) loss, which builds a well-structured feature space; (b) Cross-image intra-class feature cohesion (CIFC) loss, which is devised to make intra-class features more cohesive; and (c) Cross-image class-area weighted cross-entropy (CCWCE) loss, which is mainly employed to inversely weight the proportion of category area in mini-batch. The effectiveness of the proposed approach is demonstrated by extensive experiments on three RS semantic segmentation datasets from ISPRS Vaihingen, ISPRS Potsdam, and iSAID. MiCro is superior to the current most advanced methods in most incremental settings, especially improving mIoU by 11.59% on ISPRS Vaihingen, 13.17% on ISPRS Potsdam, and 15.01% on iSAID in the most difficult incremental settings, which promotes the CSS to a state-of-the-art (SOTA) level. The code will be available at https://github.com/RongXueE/MiCro.

Aliasing artifacts introduced by azimuth ambiguity seriously impact the interpretation of synthetic aperture radar images. To achieve parameter-free and fast azimuth ambiguity suppression, a novel deep learning model is designed to estimate the ambiguous signal intensity to total signal intensity ratio in the range-Doppler domain. This model does not depend on processing parameters and can be applied in any acquisition mode. The mean shift algorithm is applied to select less ambiguous subspectra according to the estimation result. The selected subspectra are restored to a full spectrum with an energy concentrated extrapolation method to preserve the resolution. The enhanced spectral selection and extrapolation algorithm overcomes the dependence on processing parameters, and experiments based on TerraSAR-X and Radarsat-2 images indicate that the proposed algorithm suppresses the azimuth ambiguity significantly.

The rapid development of satellite platforms has yielded copious and diverse multi-source data for earth observation, greatly facilitating the growth of multimodal semantic segmentation (MSS) in remote sensing. However, MSS also suffers from numerous challenges: 1) Existing inherent defects in each modality due to the different imaging mechanisms. 2) Insufficient exploration of the intrinsic characteristics of modalities. 3) The existence of the huge semantic gap between heterogeneous data causes difficulties in feature fusion. The inability to effectively utilize the rich and diverse information provided by each modality and ignorance of the heterogeneity between modalities will hinder the feature enhancement, and further significantly impacts the semantic segmentation accuracy. Furthermore, neglecting the huge gap makes feature fusion challenging. In this study, we introduce a novel framework for multimodal semantic segmentation that effectively mitigates the aforementioned problems. Our approach employs a pseudo-siamese structure for feature extraction. Specifically, we propose a simple yet effective geometric topology structure modeling (GTSM) module to extract geometric relationships and texture information from optical data. Additionally, we present a modality intrinsic noise suppression (MINS) module to fully exploit radiation information and alleviate the effects of unique geometric distortions for SAR. Furthermore, we present an adaptive multimodal feature fusion (AMFF) module for fully fusing different modality features. Extensive experiments on both WHU-OPT-SAR and DFC23 datasets validate the robustness and effectiveness of the proposed Modality Characteristics-Guided Semantic Segmentation (MoCG) network compared to other state-of-the-art semantic segmentation methods, including multimodal and single-modal approaches. Our approach achieves the best performance on both datasets, resulting in mIoU/OA gains 69.1%/87.5% on WHU-OPT-SAR and 86.7%/97.3% on DFC23.

The proposal of Segment Anything Model (SAM) has created a new paradigm for deep learning-based semantic segmentation field, and has shown amazing generalization performance. However, we find it may fail or perform poorly on multimodal remote sensing scenarios, especially the Synthetic Aperture Radar (SAR) images. Besides, SAM does not provide category information of objects. In this paper, we propose a foundation model for multimodal remote sensing image segmentation called RingMo-SAM, which can not only segment anything in optical and SAR remote sensing data, but also identify object categories. First, a large-scale dataset containing millions of segmentation instances is constructed by collecting multiple open-source datasets in this field to train the model. Then, by constructing an instance-type and terrain-type category-decoupling mask decoder, the category-wise segmentation of various objects is achieved. In addition, a prompt encoder embedded with the characteristics of multimodal remote sensing data is designed. It not only supports multi-box prompts to improve the segmentation accuracy of multi-objects in complicated remote sensing scenes, but also supports SAR characteristics prompts to improve the segmentation performance on SAR images. Extensive experimental results on several datasets including iSAID, ISPRS Vaihingen, ISPRS Potsdam, AIR-PolSAR-Seg, etc. have demonstrated the effectiveness of our method.

Due to variable imaging conditions, the resolution degradation often occurs in aerial images, which in turn impairs the performance upper bound of semantic segmentation. To solve this problem, super-resolution is placed before semantic segmentation as a pre-processing sub-task in most existing methods. The above two sub-tasks often form a unidirectional open-loop structure for relatively independent optimization, which constrains the ultimate segmentation performance improvement. To break down information barriers among them and form a more compact overall optimization, we propose an effective learning method named as Pixel-Wise Mutual-Feed Learning (PW-MFL) for segmenting images with resolution degradation. The key is to build auxiliary bidirectional connections, which contribute to the mutual pixel-wise spatial and feature information guidance during training. The feed-forward connection is realized by the Self-Attention Context Correlation (SACC) module, which enhances the intra-class semantic features of pixel positions with poor super-resolution performance by the fusion of that with superior performance. The feed-back connection is achieved by the Semantic Weighted Mapping (SWM) module, which aims to activate and adjust the detailed features of super-resolution in incorrectly segmented pixel positions via the semantic feature information. In addition, the Pixel-Aware Optimization (PAO) strategy is proposed to give more attention to optimizing specific pixel positions based on spatial information. Extensive experiments are conducted on three representative remote sensing segmentation benchmarks, ISPRS Vaihingen, ISPRS Potsdam, and iSAID datasets. The state-of-the-art segmentation level in resolution-degraded aerial images is achieved through the proposed learning method.

Remarkable progress based on deep neural networks has been achieved on the semantic segmentation in remote sensing images. However, pixel-level labeling is expensive for remote sensing images. Semi-supervised semantic segmentation becomes an alternative approach to reduce the cost of annotation, and it is crucial to utilize efficiently a large number of unlabeled data. Nevertheless inevitably, there is the unbalanced class distribution between labeled and unlabeled data of remote sensing scene. Existing semi-supervised methods train unlabeled images in isolation from labeled images and only learn reliable pixel pseudo-labels, leading to underutilization of unlabeled images. This article proposes a novel semi-supervised semantic segmentation approach based on label propagation and contrastive regularization for remote sensing images. Specifically, the unlabeled images are augmented by randomly copy-pasting the class regions from labeled images. A prototype feature constraint module is used to enforce the constraint on the pixel features of unlabeled images relying on the prototype features from labeled images, achieving feature alignment on the entire dataset. Furthermore, we present the region contrastive learning module that guides the model to learn feature consistency under different perturbations and compact feature representations over class regions on unlabeled images. Extensive experimental results on multiple remote sensing datasets demonstrate that our proposed approach achieves superior performance compared with state-of-the-art semi-supervised semantic segmentation methods.

Deep learning technology has promoted the object detection task in the remote sensing (RS) field to move toward better performance and more demanding requirements. Except for rigid body objects, component objects (COs) with more complex characteristics remain a detection challenge. Its “partial rules and overall disorder” characteristic limits the model learning ability to the structural features. And the internal noise and relatively sparse arrangement are not conducive to optimizing the model by the existing sample assignment strategies. We propose CODet to detect COs in RS scenes. It consists of a cross-hierarchy feature fusion module (CFM) and a noise-sparse sample assignment (NSA) strategy. CFM learns the potential representation and relative position relationship of components by fusing different level features. NSA redefines the optimization process of sample assignment. It aims to alleviate the problems of classification–localization misalignment (CLM) and the positive–negative sample imbalance (PNI) caused by the object’s internal noise and sparse arrangement. The method is verified on the proposed COD dataset of six categories of COs, reaching an average mAP/mAP50of 54.3/86.0. To be closer to the task requirements of the practical RS scene, we also propose a RS large-scale images inference framework. It includes a dataset (APRoI, labeled with COs and rigid body objects), a large-scale image inference strategy, and a set of evaluation metrics. With CODet as the core, the framework can effectively reduce the inference time by three to four times on images with an average of more than 100 million pixels.

Object detectors of remote sensing (RS) imagery with deep learning have become increasingly popular and rely heavily on extensive labeled data. The source-only detectors, which are trained on massive labeled data in a source domain, in some cases fail to get satisfactory performance on a target domain due to the domain shift. To alleviate the domain shift, popular approaches consider feature distribution alignment, but the target domain with massive unlabeled data is under-utilized. Some methods use the source-only model to generate pseudo labels for target domain data, but the variation of different remote sensing scenarios produces the domain shift, which is injected directly into the pseudo labels. Therefore, We propose the Dual-head rectification Domain Adaptation network (DualDA-Net) to alleviate the domain shift and exploit the potential of unlabeled target domain data. DualDA-Net cooperates the coarse-to-fine consistency alignment (CCA) with dual-head co-training (DHCT) to align the distribution and generate pseudo labels progressively. Specifically, the CCA focuses on source and target domain feature distribution alignment via coarse-to-fine consistency alignment on multi-level features. Moreover, the DHCT with dual detection heads is deployed in the teacher-student framework, where one of the heads complements the other with high-quality predictions to rectify the pseudo labels as supervision and alleviate the biased information. Sufficient experiments have been conducted on several domain adaptation settings. The experimental results demonstrate that our DualDA-Net achieves success in the target domain for cross domain object detection of RS imagery.

Multimodal remote sensing (RS) image segmentation aims to comprehensively utilize multiple RS modalities to assign pixel-level semantics to the studied scenes, which can provide a new perspective for global city understanding. Multimodal segmentation inevitably encounters the challenge of modeling intra- and inter-modal relationships, $i.e$ ., object diversity and modal gaps. However, the previous methods are usually designed for a single RS modality, limited by the noisy collection environment and poor discrimination information. Neuropsychology and neuroanatomy confirm that the human brain performs the guiding perception and integrative cognition of multimodal semantics through intuitive reasoning. Therefore, establishing a semantic understanding framework inspired by intuition to realize multimodal RS segmentation becomes the main motivation of this work. Drived by the superiority of hypergraphs in modeling high-order relationships, we propose an intuition-inspired hypergraph network ( $I^{2}HN$ ) for multimodal RS segmentation. Specifically, we present a hypergraph parser to imitate guiding perception to learn intra-modal object-wise relationships. It parses the input modality into irregular hypergraphs to mine semantic clues and generate robust mono-modal representations. In addition, we also design a hypergraph matcher to dynamically update the hypergraph structure from the explicit correspondence of visual concepts, similar to integrative cognition, to improve cross-modal compatibility when fusing multimodal features. Extensive experiments on two multimodal RS datasets show that the proposed $I^{2}HN$ outperforms the state-of-the-art models, achieving F1/mIoU accuracy 91.4%/82.9% on the ISPRS Vaihingen dataset, and 92.1%/84.2% on the MSAW dataset.

The label assignment problem is a core task in object detection, which mainly focuses on how to define the$positive/negative$samples during the training phase. Recent works have proved that label assignment is significant for performance improvement of the detector. In this article, we propose an exquisite strategy that can dynamically assign labels according samples' joint scores (classification and location). Moreover, our strategy can apply to both 2D and 3D monocular detectors. In our strategy, we formulate label assignment as an optimization problem. Concretely, we first calculate the classification and location costs of each sample, which are treated as points in a 2-D coordinate system. Then an optimal divider line that minimizes the sum of point-to-line distances is designed to separate the$positive/negative$samples. An iterative Genetic Algorithm is employed in acquiring the optimal solution. Furthermore, a GIoU auxiliary branch is devised to keep sample selection consistent during the training and testing phase. Benefitting from the non-maximum suppression (NMS) that utilizes the joint scores of classification and location, excellent detection performance is achieved. Extensive experiments conducted on MS COCO, PASCAL VOC (2D object detection), and KITTI (3D object detection) verify the effectiveness and universality of our proposed Optimal Partition Assignment (OPA).

Estimating height information from a single remote sensing image is a critical component for 3D perception. Recent methods formulate it as a dense height prediction task based on regression loss functions. However, the regression accuracy is limited by the infinite continuous solution space. In this paper, we propose the soft weighted ordinal (SWO) classification loss for height prediction model to convert the regression problem with infinite continuous values into the classification problem with finite discrete values. which greatly improves the accuracy of high estimation. Specifically, we first define the discrete height rule and introduce the distance penalty metric to transform the continuous ground truth height value to the soft probability distributions. This is then used as supervised information to optimize the pixel-wise classification model. Finally, we utilize soft weighted summation to generate continuous height values in the inference phase. The proposed SWO classification loss can be used directly with existing dense prediction structures whose performance can be strengthened by direct replacement of the loss functions. Comprehensive experiments on the IS-PRS Vaihingen dataset show that the proposed method has achieved promising results.

With the rapid update and iteration of current aerial image data, the continual learning scenarios and catastrophic forgetting problem attracted increased attention, especially in the semantic segmentation task. However, the existing methods mainly focus on the class continual learning in a single task and are not satisfactory when extended to multiple tasks. In this article, we consider more realistic and complicated settings, namely task continual learning. We revisit the characteristics of semantic segmentation and knowledge distillation (KD) strategy, then propose a general and effective framework, named structured inheritance, to learn new tasks while retaining high performance on old tasks. Specifically, we present two structure-preserving penalties: pixel affinity structure loss and representation consistency structure loss. The former breaks the isolation of pixels and retains the pixel interactive information learned by the old tasks. At the same time, the latter protects high-frequency stationary information between sequence semantic segmentation tasks. Our approach does not need to add extra parameters nor does it need to access the data stream of the old tasks. Therefore, it can be applied in practical applications with strict computational burden, memory cost, and storage budget. Extensive continual learning experiments on four semantic segmentation datasets of Vaihingen, Potsdam, DeepGlobe, and Gaofen challenge semantic segmentation dataset (GCSS) prove the effectiveness of our proposed framework, which outperforms the current state-of-the-art methods and even exceeds the theoretical upper-bound performance of multitask learning. The code and models will be made publicly available.

Boundary information is essential for the semantic segmentation of remote sensing images. However, most existing methods were designed to establish strong contextual information while losing detailed information, making it challenging to extract and recover boundaries accurately. In this paper, a boundary-sensitive network (BSNet) is proposed to address this problem via dynamic hybrid gradient convolution (DHGC) and coordinate sensitive attention (CSA). Specifically, in the feature extraction stage, we propose dynamic hybrid gradient convolution (DHGC) to replace vanilla convolution, which adaptively aggregates one vanilla convolution kernel and two gradient convolution kernels (GCKs) into a new operator to enhance boundary information extraction. The GCKs are proposed to explicitly encode boundary information, which are inspired by traditional Sobel operators. In the feature recovery stage, the coordinate sensitive attention (CSA) is introduced. This module is used to reconstruct the sharp and detailed segmentation results by adaptively modeling the boundary information and long-range dependencies in the low-level features as the assistance of high-level features. Note that DHGC and CSA are plug-and-play modules. We evaluate the proposed BSNet on three public data sets: the ISPRS 2-D semantic labeling Vaihingen, Potsdam benchmark and iSAID data set. The experimental results indicate that BSNet is a highly effective architecture that produces sharper predictions around object boundaries and significantly improves the segmentation accuracy. Our method demonstrates superior performance on the Vaihingen, Potsdam benchmark and iSAID data set, in terms of the mean F1, with improvements of 4.6%, 2.3% and 2.4% over strong baselines, respectively. The code and models will be made publicly available.

Segmentation incremental learning has received a lot of attention in recent years due to the ability to overcome the problem of catastrophic forgetting. Our study found that differences in label number distribution affect the performance of segmentation incremental learning. Because the labels for pixels of the old category are marked as background when the model is trained on the new tasks, the label number distribution is inconsistent with static learning that is considered to be the upper bound on incremental learning, which hinders the mitigation of the catastrophic forgetting problem. In response to the above problems, we propose an incremental learning method named SIL-LAND, which improves the accuracy by making the label number distribution of our method close to that of static learning. From the perspective of high-level semantic labels, we propose the prototype update mechanism for the problem that non-adaptive representative prototypes ignore the sample diversity of semantic categories in remote sensing images. By compensating for the difference in label number distribution at the feature level, the distance between the prototype and the actual class center is reduced; Aiming at the lack of semantic consistency between feature vectors and prototypes, we propose a similarity measure module to increase the intra-class similarity between the prototype and corresponding feature vectors. From the perspective of one-hot labels, we propose label reconstruction, including foreground screening and background padding to make the number distribution of one-hot labels as close as possible to that of static learning. A series of experimental results demonstrate the effectiveness of our method.

Due to the differences in the feature distribution between classes, when the model learns in a continuous data stream, it will encounter catastrophic forgetting. The incremental learning methods have shown great potential to solve this problem. However, most existing methods based on task-incremental learning are difficult to adapt to characteristics of remote sensing scenes with few differences in appearance but large differences in features, which is not conducive to artificially distinguish task-identity document (ID). Thus, we propose a class-incremental learning (CIL) network for small objects enhancing semantic segmentation in aerial imagery. Specifically, considering the superior accuracy of the binary classifier, we propose a twin-auxiliary (TA) model that adds an auxiliary binary classification task. Then, for expansion and contraction at the edge and small object confusion problems, we introduce a diversity distillation loss, using the results of binary-classifier to constrain the multiclass segmentation results and strengthen the attention to the locations of the segmentation results that have changed. Finally, we design a conflict reduction mechanism for multihead classifier to achieve single-head prediction for CIL. Experiments demonstrate that our method has good performance on the Vaihingen and Potsdam datasets by the International Society for Photogrammetry and Remote Sensing (ISPRS), outperforming state-of-the-art (SOTA) incremental learning methods. The code will be available soon.

With the proposal of neural architecture search (NAS), automated network architecture design gradually becomes a new way in deep learning research. Due to its high capability regarding automated design, some pioneers have made an attempt to apply NAS in remote sensing and made some achievements, like 1-D/3-D Auto-convolutional neural network (CNN) and polarimetric synthetic aperture radar (PolSAR)-tailored Differentiable Architecture Search (PDAS). However, there are still some areas to be improved for existing NAS in remote-sensing field. In this article, we propose a random topology and random multiscale mapping (RTRMM) method to generate a multiscale and lightweight architecture for remote-sensing image recognition. First, a random topology generator generates the topology through random graph. Second, during the experiment, we find remote-sensing image features extracted by a multiscale network are more appropriate, compared with features extracted by a single-scale model. Nevertheless, the complexity inevitably increases with the introduction of a multiscale concept. Consequently, we design a variable search space consisting of decomposition convolution units under the guidance of mathematical analysis. The mapping of each neuron is then determined by a random multiscale mapping sampler. After that, we assemble the topology and mappings into blocks and construct three RTRMM models. Experiments on four scene classification datasets confirm the feature extraction capability and lightweight performance of RTRMM models. Moreover, we also observe that our approach achieves a better tradeoff between floating-point operations (FLOPs) and accuracy than some current well-behaved methods. Furthermore, the results on Vaihingen dataset verify the high feature-transfer capability.

Existing deep learning models usually assume that all data obeys independent identically distribution, which is unreasonable in remote sensing. Due to the differences in camera parameters, spectral ranges, resolutions, and so on, the images acquired by remote sensing sensors may be greatly diverse, causing models to face catastrophic forgetting when they are trained on new data only. Thus, incremental learning is introduced. An ideal incremental learning model should be expanded as the number of tasks increases, so as to have enough ability to adapt to the changes in data. However, existing approaches normally expand heavy modules for each task, making the holistic models cumbersome. In this article, a lightweight incremental learning approach (LIL) is proposed for remote sensing image scene classification. We replace the role of the feature extractor with extracting features of a single task instead of task-sharing features of all tasks to lighten the backbone. In addition, we propose a light feature transfer module (FTM) to realize the alignment of data distributions between different tasks in the feature domain. Furthermore, dual-constraint loss with knowledge distillation and adversarial learning is introduced to promote the mapping and alignment of data distributions at both the feature level and the semantic level. In LIL, only a tiny FTM and a classifier are added to the model when the model learns a new task. Experimental results show that our approach with a small number of parameters outperforms state-of-the-art approaches for incremental learning on both a single dataset and a sequence of multiple datasets.

Semantic segmentation in very-high-resolution (VHR) aerial images is one of the most challenging tasks in remote sensing image understanding. Most of the current approaches are based on deep convolutional neural networks (DCNNs). However, standard convolution with local receptive fields fails in modeling global dependencies. Prior research works have indicated that attention-based methods can capture long-range dependencies and further reconstruct the feature maps for better representation. Nevertheless, limited by the mere perspective of spatial and channel attention and huge computation complexity of self-attention (SA) mechanism, it is unlikely to model the effective semantic interdependencies between each pixel pair of remote sensing data with complex spectra. In this work, we propose a novel attention-based framework named hybrid multiple attention network (HMANet) to adaptively capture global correlations from the perspective of space, channel, and category in a more effective and efficient manner. Concretely, a class augmented attention (CAA) module embedded with a class channel attention (CCA) module can be used to compute category-based correlation and recalibrate the class-level information. In addition, we introduce a simple yet effective region shuffle attention (RSA) module to reduce feature redundant and improve the efficiency of SA mechanism via regionwise representations. Extensive experimental results on the ISPRS Vaihingen, Potsdam benchmark, and iSAID data set demonstrate the effectiveness and efficiency of our HMANet over other state-of-the-art methods.

Semantic segmentation in aerial imagery is still an important, yet challenging task due to the complex characteristics of remote-sensing data. The critical issues consist of: 1) extreme foreground–background imbalance; 2) large intra-class variance; and 3) arbitrary-oriented, dense, and small objects. The above challenges make it unlikely to model the effective global interdependencies of semantic heterogeneous regions. Besides, general semantic segmentation methods suffer from feature ambiguity due to the joint feature learning paradigm, leading to inferior detail information. In this article, we propose an improved semantic segmentation framework to tackle these problems via graph reasoning (GR) and disentangled learning. On the one hand, a simple, yet effective GR unit is introduced to implement coordinate-interaction space mapping and perform relation reasoning over the graph. It can be deployed on the feature pyramid network (FPN) to exploit cross-stage multi-scale information. On the other hand, we propose a so- called disentangled learning paradigm to explicitly model the foreground and boundary objects, instantiated as foreground prior estimation (FPE) and boundary alignment (BA). The indication of the intermediate feature can be effectively emphasized to enhance the discriminative abilities of the network. Extensive experiments over iSAID, ISPRS Vaihingen, and the general Cityscapes datasets demonstrate the effectiveness and efficiency of the proposed framework over other state-of-the-art semantic segmentation methods.

Despite the extraordinary success of the deep architectures on semantic segmentation for remote sensing (RS) images, they have difficulties in learning new classes from a sequential data stream because of catastrophic forgetting. Continual learning for semantic segmentation (CSS) is an emerging trend for its capability to cope with the above problems effectively. However, old classes from previous steps are collapsed into the background, which further aggravates the challenge of CSS in the RS scene. In this article, we revisit the knowledge distillation (KD) strategy and the characteristics of class-incremental semantic segmentation (CISS) and then present a generalized and effective framework to learn new classes while preserving knowledge of the learned classes. In particular, we propose two novel historical information-guided modules: the feature global perception module and the label reconstruction (LR) module. The former enables the current model to pay more attention to the region related to the old categories identified by the historical information when learning new classes. Meanwhile, the latter retrieves pixels belonging to the learned classes from the background to handle the background shift problem and maintain the high performance of old classes. We have conducted comprehensive experiments on two RS semantic segmentation datasets of Instance Segmentation in Aerial Images Dataset (iSAID) and Gao Fen (GF) challenge semantic segmentation dataset (GCSS). The experimental results outperform the current state-of-the-art methods in most incremental settings, which demonstrates the effectiveness of the proposed framework.

Range sidelobe is very common in synthetic aperture radar (SAR) images, particularly when imaging scene includes strongly scattering targets such as ships or complex buildings. As a kind of interference, it may reduce the image quality and hinder the image interpretation. Hence, range sidelobe suppression is an important mission for SAR images. The main task of mitigating the sidelobe is how to achieve the most effective suppression with the minimal resolution loss and signal-to-noise ratio (SNR) loss. However, the widely recognized classic method, spatially variant apodization (SVA), still has a lot of residual sidelobe energy and other problems. This article proposes a novel suppression approach based on time-variant transmission of chaotic frequency modulation (CFM) signals. The key is to build an appropriate transmitted signal set, where the signals are generated by various chaotic initial states and the same special map with low mixing rate and uniform invariant probability density (IPD). Due to their beneficial autocorrelation properties, the proposed approach achieves superior performance in range sidelobe suppression and resolution preservation. More importantly, it maintains the energy of the signals and overcomes the SNR loss that occurs in some classic methods, such as spectral weighting (SW) and SVA. In addition, it is suitable for both vertical and squint side-looking mode and can well reconstruct the weakly scattering targets which are severely disturbed by range sidelobe. All of them are validated by comparative experiments.

Object detection is a fundamental part of the interpretation of remote sensing imagery. The one-stage object detector has been adopted into this field because of its high computational efficiency. However, this detector suffers from the misalignment among predefined anchor, object, and feature extracted by standard convolution kernel both in spatial and scale. It limits the further improvement of performance, especially for the long-narrow and multiscale geospatial objects. In this article, the problem is defined asthe feature misalignmentproblem. To deal with this issue, an efficient feature aligned single-shot detector (ASSD) is proposed, which consists of two modules: a novel pseudo anchor proposal module (PAPM) and a flexible context-based feature alignment module (CFAM). The PAPM replaces the regular anchor group with the proposed core anchor and refines it to get aligned locations. It can tackle the spatial misalignment between anchors and their corresponding objects and alleviate the negative/positive imbalance problem. Then, the CFAM adaptively adjusts the sampling points of the convolution kernel and collects the context information according to the aligned core anchor. This plug-and-play module can effectively rectify the misalignment between kernel and objects and extract aligned and robust features. A series of comprehensive experiments are conducted on two large-scale public remote sensing object detection datasets. Experiment results suggest that the proposed method is effective to alleviate the misalignment problem. Compared with the baseline model, the detection accuracy is improved by 8.5% mAP and 11.0% mAP on the challenging benchmark for object detection in optical remote sensing image (DIOR) and a large-scale dataset for object detection in aerial image (DOTA) dataset, respectively. Our best-resulting model achieves the state-of-the-art performance, surpassing other one-stage detectors both on the two datasets at a high detection speed of 21 FPS.

Deep learning-based object detectors have been widely adopted in the field of remote sensing imagery interpretation. These detectors heavily depend on the expensive large-scale labeled datasets, while the scarce remote sensing datasets limit the performance. The domain adaptive object detection can alleviate this problem. However, it struggles with the confusing feature’s alignment, damaging the domain generalization performance, especially for the remote sensing scene with sparse objects and diverse backgrounds. For that reason, a semisynthetic data generator (SDG) is proposed to automatically generate the remote sensing dataset with low cost and replace the real-world training dataset, afeature aligned domain adaptive object detector(FADA) is proposed to enhance the domain adaptation among the cross-domain remote sensing images. The FADA contains two proposed modules in addition to the base detector: an adversarial-based foreground alignment (AFA) and a prototype-based confusing feature alignment (PCFA). The AFA aligns the cross-domain foreground feature by adversarial training (AT), and it can filter the noisy background feature that is not suitable to transfer. Then, the PCFA adaptively aligns the confusing background and foreground feature, further promoting the domain adaptation performance. Comprehensive experiments validate the effectiveness of the proposed method. Compared with the baseline model trained on the semisynthetic source dataset, our FADA improves the generalized performance on the real-world target dataset a large-scale Dataset for Object deTection in Aerial images (DOTA) by 15.7% average precision (AP) and achieves state-of-the-art results.

The lightweight model has played an important role in the remote sensing (RS) realm. The existing researchers have proposed many models with lightweight structures, but their performance still has a gap compared with the deep model. A promising approach to optimize the lightweight model is knowledge distillation (KD), which can be viewed as knowledge transfer from the teacher model. However, the existing KD approaches have some issues. On one hand, offline distillation methods usually ignore the interactive learning between the student model and the teacher model. On the other hand, knowledge transfer does not consider instance property. This offline distillation strategy without property perception may not suitable for multiscale, diverse, and complex RS instances and results in a suboptimum training status. In this article, we propose a dynamic interactive learning (DIL) framework for optimizing RS lightweight detectors. First, we propose an instance interaction learning module. It calculates the value of every instance in the batch of the teacher and student prediction by each model’s real-time state and instance property. Then according to the DIL thought, we facilitate the low-quality instance to learn from the high-quality one whether it is from the teacher or student model. Moreover, we also propose the instance property perception (IPP) strategy that weighs the distillation knowledge of instances according to their feature, category, and location property. In the proposed DIL framework, both the teacher and student models are trained together and it is cost-free in the testing phase. Extensive experiments on three RS datasets demonstrate the effectiveness of the DIL.

The resolution degradation poses a huge challenge for object detection (OD) in the aerial imagery. Existing methods utilize super resolution (SR) based on Generative Adversarial Network (GAN) to restore texture details in degraded images. However, constrained detection results are still acquired due to the object feature difference between restored and clear images. Therefore, we propose a simple-yet-effective learning method called Mutual-Feed Learning (MFL) to solve the problem in this paper. A closed-loop structure is designed via building the feedback connection based on the feedforward connection between the two tasks. It effectively delivers the object spatial and feature information from OD to SR, and provides restoration-enhanced images from SR to OD. Specifically, a Feedback of Region of Interest (FROI) module is introduced to realize a region-level discrimination under the guidance of object information. It guides the discrimination process of super resolution. Furthermore, a Multi-Scale Object Information (MSOI) module is developed to implement a feature-level restoration by narrowing differences in object-related features. It improves the generation process of super resolution. Then object detection can be performed in restoration-enhanced images to obtain more accurate results. Extensive experiments over NWPU VHR-10, COWC, and FAIR1M dataset show that the method can achieve state-of-the-art results.

Lightweight object detector is currently gaining more and more popularity in remote sensing. In general, it’s hard for lightweight detectors to achieve competitive performance compared to traditional deep models, while knowledge distillation is a promising training method to tackle the issue. Since the background is more complicated and the object size varies extremely in remote sensing images, it will deliver lots of noise and affect the training performance when directly applying the existing knowledge distillation methods. To tackle the above problems, we propose an Adaptive Reinforcement Supervision Distillation (ARSD) framework to promote the detection capability of the lightweight model. Firstly, we put forward a multiscale core features imitation (MCFI) module for transferring the knowledge of features, which can adaptively select the multiscale core features of objects for distillation and focus more on the features of small objects by an area-weighted strategy. In addition, a strict supervision regression distillation (SSRD) module is designed to select the optimal regression results for distillation, which facilitates the student to effectively imitate the more precise regression output of the teacher network. Massive experiments on the DOTA, DIOR, and NWPU VHR-10 datasets prove that ARSD outperforms the existing distillation SOTA methods. Moreover, the performance of lightweight model trained with our method transcends other classic heavy and lightweight detectors, which beneficiates the development of lightweight models.

In recent years, more concerns are shed on the lightweight detection model in remote sensing (RS), but it is difficult to reach a competitive performance relative to the deep model. Knowledge distillation has been verified as a promising method, which can promote the performance of the lightweight model without extra parameters. While there are two key issues of detection distillation, one is the sample selection, the other is the knowledge selection. Since the varying object size and complex features in RS, the existing methods based on the fixed threshold are incapable of selecting the optimal distillation samples and they also ignore the potential multivariate knowledge among RS samples simultaneously. In this paper, we propose a statistical sample selection and multivariate knowledge mining framework. The statistical sample selection module formulates the task as the modeling and splitting the probability distribution of sample selection cost, which is more suitable for dynamically choosing multiscale samples in RS and eliminates the distortion of previous static distillation selection. Furthermore, to mine the complex feature knowledge of samples in RS, we design a multivariate knowledge mining module, in which knowledge includes explicit and implicit knowledge. The proposed module validly deliver the core knowledge from the teacher model to the lightweight model. Massive experiments on three challenging RS datasets (DOTA, NWPU VHR-10, DIOR) prove that our method achieves state-of-the-art performance.

Cloud detection relying on deep convolutional neural networks obtains remarkable accuracy gains at the expense of high computation and storage costs, which are difficult to deploy to resource-constrained devices, such as intelligent satellites. Recently, knowledge distillation (KD) has been a promising solution for compact model. However, most existing KD methods only transfer the feature relationship of pairwise pixel which fails to cope with thin clouds and cloud-like objects in complex scenes. Furthermore, those KD methods directly imitate the output of complicated model regardless of the correctness. In this article, we propose a novel Category Correlation and Adaptive Knowledge Distillation (CAKD) framework for the lightweight cloud detection network. We design a category relational context (CRC) module to refine the structured pixel-category correlation from the teacher and student network. Then, we perform the category correlation distillation (CCD) to make the student model better address the intra-class consistency and the inter-class difference, thus reducing the category confusion. Besides, a pixel-adaptive distillation (PAD) module is utilized to adaptively transfer the soft-output knowledge of teacher model by extracting the teacher’s pixel prediction probability. Extensive experiments on Landsat 8, Landsat 7, Gaofen-2, Gaofen-1 and Google Earth dataset report the effectiveness and universality of our distillation method. The CAKD allows MobileNetV2 with 2.31M parameters and 4.63G FLOPs to outperform advanced cloud detection methods without the added inference overhead.

Deep convolutional neural networks (CNNs) have brought a tremendous increase in detection accuracy, but too cumbersome model makes them hard to deploy on low computation edge devices, such as satellites and unmanned aerial vehicles. A promising method to tackle this problem is knowledge distillation (KD), which makes models lightweight with satisfactory accuracy. For remote sensing images, the objects are usually environment-related and located in a cluttered scene. The features that objects’ semantic information relies on are tangled. However, existing distillation methods only imitate feature distribution derived from regions, including objects resulting in poor performance. Furthermore, masses of instances generated by teachers are blindly inherited, even if some of them are outliers. In this article, we propose a general and effective KD framework called dynamic knowledge distillation (DKD). First, our framework leverages the dynamic global distillation (GD) module to discover valuable regions from the foreground and background for multiscale features imitation, avoiding ignoring the potential geographical spatial relationship. Second, we propose a dynamic instance selection distillation (ISD) module to give students the ability of self-judgment through the magnitude of detection loss. Third, toward more accurate handling of hard samples in regression, a training-status-aware loss is tailored to guide students mine knowledge about objects with large aspect ratio or small size. Extensive experiments are conducted to show the effectiveness of DKD framework. The detection results on DOTA and NWPU VHR-10 dataset illustrate that our method is suitable for single-stage, two-stage and even anchor-free detectors. It shows the state-of-the-art performance. The code will be publicly available.

With the development of deep convolutional neural networks, detecting rotating objects in remote-sensing images is of great significance in various fields. Existing rotating object detectors most suffer the problem of ambiguous supervision caused by inappropriate rotating object representations. This problem may result in fuzzy object localization and further lead to misclassification. In this article, we propose an Automatic Organized Points Detector (AOPDet), which derives precise localization results by applying a novel rotating object representation called nonsequential corners representation. To achieve the proposed representation, an Automatic Organization Mechanism (AOM) technique is designed to guide the model to organize points to object corners automatically. An Automatic-Organized-Points-specific (AOP-specific) head structure is also designed and equipped in the model to better focus on the rotating object detection task. On public aerial datasets, experiments show that the AOPDet achieves 17.0 mAP higher than the compared baseline model, reaching the state-of-the-art (SOTA) level. Detailed ablation experiments and error analysis strongly reveal the effectiveness of the proposed model.

Due to the characteristics of vertical orthophoto imaging, the apparent structural features of the object in the remote sensing image are relatively stable, such as the cross-shaped structure of the aircraft, the rectangular structure of the vehicle, etc. Compared with the traditional visual features, using these features is conducive to improving the accuracy of object detection. However, there are few studies on such characteristics. In this paper, we systematically study the invariant structural features of remote sensing objects and propose a Graph Focusing Aggregation Network (GFA-Net) to represent the structural features of remote sensing objects. Among them, in view of the problem that traditional convolutional neural networks (CNNs) are sensitive to the changes in rotation, scale, and other factors, which makes it difficult to extract structural features, we propose the Graph Focusing Process (GFP) based on the idea of graph convolution. Analysis and experiments show that graph structure has significant advantages over Euclidean feature space under CNN in expressing such structural features. In order to realize the end-to-end efficient training of the above model, we design Graph Aggregation Network (GAN) to update the weight of nodes. We verify the effectiveness of our method on the proposed multi-task datasets ACSD and large-scale fine-grained remote sensing dataset FAIR1M. Experiments conducted on the object detection data sets of DOTA and HRSC2016 prove that the proposed method is superior to the current state-of-the-art method.

The detection of garbage dumps is of great significance for environmental protection. Recently, deep learning algorithms have brought impressive improvements for regular object detection. Different from conventional objects, garbage dumps are more inconspicuous and irregular and have the problem of blurred boundaries. To solve these problems, we propose a shape robust anchor-free network (SRAF-Net) that consists of feature extraction, multitask detection, and postprocessing. First, our network leverages the context-based deformable (CBD) module to combine context attention and deformable convolution. The contextual information obtained by context attention enables the network to focus on objects with inconspicuous appearance, while the deformable convolution enhances the feature representation. Then, we propose a multitask detection head to regress irregular garbage dumps in a more accurate and efficient way. The anchor-based methods need to define some anchors with a fixed shape. However, our detection method is anchor-free that learns the shapes of objects from training data. The detection head adaptively generates various shapes of bounding boxes with their classification confidences and localization confidences. Weighted by the localization confidences, we merge bounding boxes during postprocessing, which alleviates the blurred boundaries. In addition, we build a new public data set named garbage dumps data set (GDD) to verify the effectiveness of our method. Extensive experiments on GDD indicate that our method surpasses the existing detection methods in terms of speed and accuracy for the garbage dumps detection task.

The balance between high accuracy and high speed has always been a challenging task in semantic image segmentation. Compact segmentation networks are more widely used in the case of limited resources, while their performances are constrained. In this paper, motivated by the residual learning and global aggregation, we propose a simple yet general and effective knowledge distillation framework called double similarity distillation (DSD) to improve the classification accuracy of all existing compact networks by capturing the similarity knowledge in pixel and category dimensions, respectively. Specifically, we propose a pixel-wise similarity distillation (PSD) module that utilizes residual attention maps to capture more detailed spatial dependencies across multiple layers. Compared with exiting methods, the PSD module greatly reduces the amount of calculation and is easy to expand. Furthermore, considering the differences in characteristics between semantic segmentation task and other computer vision tasks, we propose a category-wise similarity distillation (CSD) module, which can help the compact segmentation network strengthen the global category correlation by constructing the correlation matrix. Combining these two modules, DSD framework has no extra parameters and only a minimal increase in FLOPs. Extensive experiments on four challenging datasets, including Cityscapes, CamVid, ADE20K, and Pascal VOC 2012, show that DSD outperforms current state-of-the-art methods, proving its effectiveness and generality. The code and models will be publicly available.

Feature pyramid is the mainstream method for multi-scale object detection. In most detectors with feature pyramid, each proposal is predicted based on feature grids pooled from only one feature level, which is assigned heuristically. Recent studies report that the feature representation extracted using this method is sub-optimal, since they ignore the valid information exists on other unselected layers of the feature pyramid. To address this issue, researchers present to fuse valid information across all feature levels. However, these methods can be further improved: the feature fusion strategies, which use common operation (element-wise max or sum) in most detectors, should be replaced by a more flexible way. In this work, a novel method called feature adaptive selection subnetwork (FAS-Net) is proposed to construct effective features for detecting objects of different scales. Particularly, its adaption consists of two level: global attention and local adaptive selection. First, we model the global context of each feature map with global attention based feature selection module (GAFSM), which can strengthen the effective features across each layer adaptively. Then we extract the features of each region of interest (RoI) on the entire feature pyramid to construct a RoI feature pyramid. Finally, the RoI feature pyramid is sent to the feature adaptive selection module (FASM) to integrate the strengthened features according to the input adaptively. Our FAS-Net can be easily extended to other two-stage object detectors with feature pyramid, and supports to analyze the importance of different feature levels for multi-scale objects quantitatively. Besides, FAS-Net can also be further applied to instance segmentation task and get consistent improvements. Experiments on PASCAL07/12 and MSCOCO17 demonstrate the effectiveness and generalization of the proposed method.

Considering the lack of labeled training data sets for the supervised change detection task, in this letter, we try to relieve this problem by proposing a convolutional neural network (CNN)-based change detection method with a newly designed loss function to achieve transfer learning among different data sets. To reach this goal, we first pretrain a U-Net model on an open source data set by taking advantages of the relatively sufficient training data used for the supervised semantic segmentation task. Then, we minimize a skillfully designed loss function to combine the high-level features extracted from the pretrained model and the semantic information contained in the change detection data set, by which a transfer learning is achieved. Third, we compute the distance between the feature vectors obtained from the above step and produce a difference map. Finally, a simple clustering method used on the difference map can even obtain satisfied change map. Experiments carried out on typical optical aerial image data sets validate that the proposed approach compares favorably to the state-of-the-art unsupervised methods.

Recently, the deep convolutional neural network has brought great improvements in object detection. However, the balance between high accuracy and high speed has always been a challenging task in multiclass object detection for large-scale remote sensing imagery. One-stage methods are more widely used because of their high efficiency but are limited by their performances on small object detection. In this article, we propose a unified framework called feature-merged single-shot detection (FMSSD) network, which aggregates the context information both in multiple scales and the same scale feature maps. First, our network leverages the atrous spatial feature pyramid (ASFP) module to fuse the context information in multiscale features by using feature pyramid and multiple atrous rates. Second, we propose a novel area-weighted loss function to pay more attention to small objects, while the replaced original loss treats all objects equally. We believe that small objects should be given more weight than large objects because they lose more information during training. Specifically, a monotonic decreasing function about the area is designed to add weights on the loss function. Extensive experiments on the DOTA data set and NWPU VHR-10 data set demonstrate that our method achieves state-of-the-art detection accuracy with high efficiency. We also build a new large-scale data set called AIR-OBJ data set from Google Earth and show the detection results of small objects, which validates the effectiveness on large-scale remote sensing imagery.

Semantic labeling in high-resolution aerial images is important for its wide range of applications. In this paper, we propose an end-to-end dual multi-scale feature fusion network (DMFFNet) for high-resolution aerial multi-source images. DMFFNet aims to further improve the semantic labeling results of the region where the multispectral features are indistinguishable. Specifically, we design a channel fusion strengthen (CFS) module, which can fuse features adaptively by modelling interdependencies between channels. Furthermore, a multiscale context aggregation (MCA) module is utilized to obtain larger receptive field and more contextual information. The experiment results confirm the DMFFNet with CFS and MCA improve the semantic labeling performance by utilizing multi-source data.

The performance of object instance segmentation in remote sensing images has been greatly improved through the introduction of many landmark frameworks based on convolutional neural network. However, the object densely issue still affects the accuracy of such segmentation frameworks. Objects of the same class are easily confused, which is most likely due to the close docking between objects. We think context information is critical to address this issue. So, we propose a novel framework called SLCMASK-Net, in which a sequence local context module (SLC) is introduced to avoid confusion between objects of the same class. The SLC module applies a sequence of dilation convolution blocks to progressively learn multi-scale context information in the mask branch. Besides, we try to add SLC module to different locations in our framework and experiment with the effect of different parameter settings. Comparative experiments are conducted on remote sensing images acquired by QuickBird with a resolution of 0.5m - 1m and the results show that the proposed method achieves state-of-the-art performance.

In this paper, we present a semi-supervised remote sensing change detection method based on graph model with Generative Adversarial Networks (GANs). Firstly, the multi-temporal remote sensing change detection problem is converted as a problem of semi-supervised learning on graph where a majority of unlabeled nodes and a few labeled nodes are contained. Then, GANs are adopted to generate samples in a competitive manner and help improve the classification accuracy. Finally, a binary change map is produced by classifying the unlabeled nodes to a certain class with the help of both the labeled nodes and the unlabeled nodes on graph. Experimental results carried on several very high resolution remote sensing image data sets demonstrate the effectiveness of our method.

Object detection has been playing a significant role in the field of remote sensing for a long period while it is still full of challenges. The biggest one is how to detect multi-scale objects with high accuracy and fast speed in remote sensing images. One-stage object detectors have been achieving relatively high accuracy and efficiency with small memory footprint. However, they have a not very well performance on small objects. In this paper, we discuss the importance of the context information between feature maps in different scales which is helpful for detecting small objects. Especially, we propose a Feature-merged detection networks (MergeNet), which can be inserted into the one-stage detectors easily, to unify the multi-scale feature and context information effectively. Experiments on DOTA dataset demonstrate that our model can significantly improve the performance of the one-stage method.

Semantic segmentation in high-resolution aerial images is a fundamental research problem in remote sensing field for its wide range of applications. However, it is difficult to distinguish regions with similar spectral features using only multispectral data. Recent research studies have indicated that the introduction of multisource information can effectively improve the robustness of segmentation method. In this letter, we use digital surface models (DSMs) information as a complementary feature to further improve the semantic segmentation results. To this end, we propose a lightweight and simple DSM fusion (DSMF) branch structure module. Compared with the existing feature extraction structures, proposed DSMF module is simple and can be easily applied to other networks. In addition, we investigate four fusion strategies based on DSMF module to explore the optimal feature fusion strategy and four end-to-end DSMFNets are designed according to the corresponding strategies. We evaluate our models on International Society for Photogrammetry and Remote Sensing Vaihingen data set and all DSMFNets achieve promising results. In particular, DSMFNet-1 achieves an overall accuracy of 91.5% on the test data set.

An algorithm based on optical flow is proposed to detect a ground moving target via the single-channel synthetic aperture radar. First, the signal models of uniform moving targets are established and classified into three types. Next, the Doppler spectrum is divided to generate a multilook image sequence. Then, the motion feature of a moving target response is described in the image sequence, in which the optical flow is introduced to realize the moving target detection. The detection results of real moving targets are obtained after the false alarm elimination based on the response motion relevance. This algorithm has a large range of detectable velocity and can even be applied to detect the moving targets with acceleration. In addition, compared with constant false alarm rate method, the optical flow has a better anti-interference performance against the strong static scatters. Finally, some numerical experiments are provided to demonstrate the effectiveness of the proposed method.

Object detection has been playing a significant role in the field of remote sensing for a long time but it is still full of challenges. In this paper, we propose a novel detection framework based on rotational region convolution neural network to cope with the problem of non-maximum suppression in dense objects detection. The bounding boxes obtained by adopting our method is the minimum bounding rectangle of object with less redundant regions. Furthermore, we find the head direction of the object through prediction. There are three important changes to our framework over traditional detection methods, representation and regression of rotational bounding box, head direction prediction and rotational non-maximal suppression. Experiments based on remote sensing images from Google Earth for Object detection show that our detection method based on rotational region CNN has a competitive performance.

Deeper architectures are proven to be beneficial for the classification performance obviously in computer vision field. Inspired by this, deep CNN s are expected to make progress in the SAR target classification problem as well. However, it is hard to train deeper CNNs for SAR images. Such CNNs have millions of parameters to be determined in the network (for example the VGGNet has more than 130 million parameters), hence large-scale dataset is indispensable when training a deep CNN. But there is no large-scale annotated SAR target dataset, and data acquisition and annotation is much more costly for SAR images. With inadequate data, the network is easy to be overfitting. Several methods based on deep learning have been proposed for SAR image classifications, but they cannot get rid of the aforementioned data limitation of labelled SAR images. To solve this problem, this paper proposes a microarchitecture called CompressUnit (CU). With CU, we design a deeper CNN. Compared with the network with the fewest parameters for SAR image classification in literature so far, our network is 2X deeper with only about 10% of parameters. In this way, we get a deeper network with much fewer parameters. This network is easier to be trained with limited SAR data and is more likely to get rid of overfitting.

In this paper, a novel framework for aircraft detection in high resolution apron area in Synthetic Aperture Radar (SAR) images is proposed, which combines the strength of location regression based convolutional neural network (CNN) framework and the salient features of target in SAR images. Specifically, a Constant False Alarm Rate (CFAR) based target pre-locating algorithm is introduced, which can match the scale of target in SAR images more accurate compared to the existing region proposal method. In addition, in order to eliminate the fact of overfitting, we explore several strategies for SAR data augmentation, including translation, adding noise and rotation within a small range. Experiments are conducted on the data set acquired by the TerraSAR-X satellite in a resolution of 3.0 meters. The results show that the proposed detection framework could effectively obtain a more accurate detection result.

Scattering structure features of targets is of great importance for Synthetic Aperture Radar (SAR) image analysis. In this paper, a novel algorithm for aircraft recognition in high resolution apron area of SAR images is proposed. The algorithm combines the strength of gradient saliency map and scattering structure features to improve accuracy and efficiency. Specially, Constant False-Alarm Rate (CFAR) algorithm is carried out to segment images. Then, a new efficient object locating method based on directional local gradient map is proposed to detect aircraft targets. Then, the candidate slices as well as template slices are modeled using Gaussian Mixture Model (GMM), which will be treated as structure features. In the recognition stage, a novel similarity measurement algorithm based on Kullback-Leibler Divergence for GMM models is proposed for classification. We conduct experiments on the dataset with 3.0m resolution and the recognition results demonstrate the accuracy of our proposed method.

In this paper, a fully automatic building reconstruction method for high resolution interferometric synthetic aperture radar (InSAR) data is presented. This method is based on stochastic geometrical model. Firstly, a building detection procedure is implemented on the big image and the entire scene is divided into building clips. After that, the reconstruction process is utilized for each building clip. In the reconstruction process, a building in 3D space is projected to the image plane and then decomposed to feature regions including layover, corner line, roof and shadow. We explore the statistic properties of the each region, and include it in the posterior function, together with the edge term and the prior we defined. Finally, in order to overcome local optima, a group of special transmission kernels are designed. The experimental results on TanDEM-X data demonstrate the effectiveness of our method.

The interferometric coherence map is derived from the cross-correlation of two registered synthetic aperture radar (SAR) images. It can give additional information complementary to the intensity image, or act as an independent information source in many applications. Compared to the plenty of work on SAR intensity statistics, there are quite fewer researches on the statistical characters of interferometric SAR (InSAR) coherence. And to our knowledge, all of the existing work that related to InSAR coherence statistics, models the coherence with Gaussian distribution with no discrimination on data resolutions or scene types. Our main contribution is the investigation on the accuracies of several typical models for high resolution coherence statistics over urban areas. We select three typical land classes including trees, buildings, and shadow, as the representatives of urban areas. And different models including Gaussian, Weibull, Rayleigh, Nakagami and Beta are evaluated. Experiment results on TanDEM-X data illustrate that the Beta model reveals a better performance than other distributions. Finally, the Beta model is used in the detection of buildings.

Object detection has been one of the hottest issues in the field of remote sensing image analysis. In this letter, an efficient object detection framework is proposed, which combines the strength of the unsupervised feature learning of deep belief networks (DBNs) and visual saliency. In particular, we propose an efficient coarse object locating method based on a saliency mechanism. The method could avoid an exhaustive search across the image and generate a small number of bounding boxes, which can locate the object quickly and precisely. After that, the trained DBN is used for feature extraction and classification on subimages. The feature learning of the DBN is operated by pretraining each layer of restricted Boltzmann machines (RBMs) using the general layerwise training algorithm. An unsupervised blockwise pretraining strategy is introduced to train the first layer of RBMs, which combines the raw pixels with a saliency map as inputs. This makes an RBM generate local and edge filters. The precise edge position information and pixel value information are more efficient to build a good model of images. Comparative experiments are conducted on the data set acquired by QuickBird with a 60-cm resolution. The results demonstrate the accuracy and efficiency of our method.