The rapid advancement of foundation models has revolutionized visual representation learning in a self-supervised manner. However, their application in remote sensing (RS) remains constrained by a fundamental gap: existing models predominantly handle single or limited modalities, overlooking the inherently multi-modal nature of RS observations. Optical, synthetic aperture radar (SAR), and multi-spectral data offer complementary insights that significantly reduce the inherent ambiguity and uncertainty in single-source analysis. To bridge this gap, we introduce RingMoE, a unified multi-modal RS foundation model with 14.7 billion parameters, pre-trained on 400 million multi-modal RS images from nine satellites. RingMoE incorporates three key innovations: 1) A hierarchical Mixture-of-Experts (MoE) architecture comprising modal-specialized, collaborative, and shared experts, effectively modeling intra-modal knowledge while capturing cross-modal dependencies to mitigate conflicts between modal representations; 2) Physics-informed self-supervised learning, explicitly embedding sensor-specific radiometric characteristics into the pre-training objectives; 3) Dynamic expert pruning, enabling adaptive model compression from 14.7B to 1B parameters while maintaining performance, facilitating efficient deployment in Earth observation applications. Evaluated across 23 benchmarks spanning six key RS tasks (i.e., classification, detection, segmentation, tracking, change detection, and depth estimation), RingMoE outperforms existing foundation models and sets new SOTAs, demonstrating remarkable adaptability from single-modal to multi-modal scenarios. Beyond theoretical progress, it has been deployed and trialed in multiple sectors, including emergency response, land management, marine sciences, and urban planning.

Vision foundation models in remote sensing have been extensively studied due to their superior generalization on various downstream tasks. Synthetic Aperture Radar (SAR) offers all-day, all-weather imaging capabilities, providing significant advantages for Earth observation. However, establishing a foundation model for SAR image interpretation inevitably encounters the challenges of insufficient information utilization and poor interpretability. In this paper, we propose a remote sensing foundation model based on complex-valued SAR data, which simulates the polarimetric decomposition process for pre-training, i.e., characterizing pixel scattering intensity as a weighted combination of scattering bases and scattering coefficients, thereby endowing the foundation model with physical interpretability. Specifically, we construct a series of scattering queries, each representing an independent and meaningful scattering basis, which interact with SAR features in the scattering query decoder and output the corresponding scattering coefficient. To guide the pre-training process, polarimetric decomposition loss and power self-supervised loss are constructed. The former aligns the predicted coefficients with Yamaguchi coefficients, while the latter reconstructs power from the predicted coefficients and compares it to the input image's power. The performance of our foundation model is validated on nine typical downstream tasks, achieving state-of-the-art results. Notably, the foundation model can extract stable feature representations and exhibits strong generalization, even in data-scarce conditions.

Semantic segmentation and 3D reconstruction are two fundamental tasks in remote sensing, typically treated as separate or loosely coupled tasks. Despite attempts to integrate them into a unified network, the constraints between the two heterogeneous tasks are not explicitly modeled, since the pioneering studies either utilize a loosely coupled parallel structure or engage in only implicit interactions, failing to capture the inherent connections. In this work, we explore the connections between the two tasks and propose a new network that imposes semantic constraints on the stereo matching task, both implicitly and explicitly. Implicitly, we transform the traditional parallel structure to a new cascade structure termed Semantic-Guided Cascade structure, where the deep features enriched with semantic information are utilized for the computation of initial disparity maps, enhancing semantic guidance. Explicitly, we propose a Semantic Selective Refinement (SSR) module and a Left-Right Semantic Consistency (LRSC) module. The SSR refines the initial disparity map under the guidance of the semantic map. The LRSC ensures semantic consistency between two views via reducing the semantic divergence after transforming the semantic map from one view to the other using the disparity map. Experiments on the US3D and WHU datasets demonstrate that our method achieves state-of-the-art performance for both semantic segmentation and stereo matching.

Long-term Multivariate Time Series (LMTS) forecasting aims to predict extended future trends based on channel-interrelated historical data. Considering the elusive channel correlations, most existing methods compromise by treating channels as independent or tentatively modeling pairwise channel interactions, making it challenging to handle the characteristics of both higher-order interactions and time variation in channel correlations. In this paper, we propose HyperMixer, a novel specializable hypergraph channel mixing plugin which introduces versatile hypergraph structures to capture group channel interactions and time-varying patterns for long-term multivariate time series forecasting. Specifically, to encode the higher-order channel interactions, we structure multiple channels into a hypergraph, achieving a two-phase message-passing mechanism: channel-to-group and group-to-channel. Moreover, the functionally specializable hypergraph structures are presented to boost the capability of hypergraph to capture the time-varying patterns across periods, further refining modeling of channel correlations. Extensive experimental results on seven available benchmark datasets demonstrate the effectiveness and generalization of our plugin in LMTS forecasting. The visual analysis further illustrates that HyperMixer with specializable hypergraphs tailors channel interactions specific to certain periods.

Existing vision-based 3D occupancy prediction methods are inherently limited in accuracy due to their exclusive reliance on street-view imagery, neglecting the potential benefits of incorporating satellite views. We propose SA-Occ, the first Satellite-Assisted 3D occupancy prediction model, which leverages GPS & IMU to integrate historical yet readily available satellite imagery into real-time applications, effectively mitigating limitations of ego-vehicle perceptions, involving occlusions and degraded performance in distant regions. To address the core challenges of cross-view perception, we propose: 1) Dynamic-Decoupling Fusion, which resolves inconsistencies in dynamic regions caused by the temporal asynchrony between satellite and street views; 2) 3D-Proj Guidance, a module that enhances 3D feature extraction from inherently 2D satellite imagery; and 3) Uniform Sampling Alignment, which aligns the sampling density between street and satellite views. Evaluated on Occ3D-nuScenes, SA-Occ achieves state-of-the-art performance, especially among single-frame methods, with a 39.05% mIoU (a 6.97% improvement), while incurring only 6.93 ms of additional latency per frame. Our code and newly curated dataset are available at https://github.com/chenchen235/SA-Occ.

Remote sensing foundation models largely break away from the traditional paradigm of designing task-specific models, offering greater scalability across multiple tasks. However, they face challenges such as low computational efficiency and limited interpretability, especially when dealing with large-scale remote sensing images. To overcome these, we draw inspiration from heat conduction, a physical process modeling local heat diffusion. Building on this idea, we are the first to explore the potential of using the parallel computing model of heat conduction to simulate the local region correlations in high-resolution remote sensing images, and introduce RS-vHeat, an efficient multi-modal remote sensing foundation model. Specifically, RS-vHeat 1) applies the Heat Conduction Operator (HCO) with a complexity of $O(N^{1.5})$ and a global receptive field, reducing computational overhead while capturing remote sensing object structure information to guide heat diffusion; 2) learns the frequency distribution representations of various scenes through a self-supervised strategy based on frequency domain hierarchical masking and multi-domain reconstruction; 3) significantly improves efficiency and performance over state-of-the-art techniques across 4 tasks and 10 datasets. Compared to attention-based remote sensing foundation models, we reduce memory usage by 84\%, FLOPs by 24\% and improves throughput by 2.7 times. The code will be made publicly available.

Large models have accelerated the development of intelligent interpretation in remote sensing. Many remote sensing foundation models (RSFM) have emerged in recent years, sparking a new wave of deep learning in this field. Fine-tuning techniques serve as a bridge between remote sensing downstream tasks and advanced foundation models. As RSFMs become more powerful, fine-tuning techniques are expected to lead the next research frontier in numerous critical remote sensing applications. Advanced fine-tuning techniques can reduce the data and computational resource requirements during the downstream adaptation process. Current fine-tuning techniques for remote sensing are still in their early stages, leaving a large space for optimization and application. To elucidate the current development and future trends of remote sensing fine-tuning techniques, this survey offers a comprehensive overview of recent research. Specifically, this survey summarizes the applications and innovations of each work and categorizes recent remote sensing fine-tuning techniques into six types: adapter-based, prompt-based, reparameterization-based, hybrid methods, partial tuning, and improved tuning. In the final section, this survey suggests nine areas worth exploring in this field. Remote sensing fine-tuning methods in this survey can be found at https://github.com/DongshuoYin/Remote-Sensing-Tuning-A-Survey.

In the era of smart cyber-physical grid, dynamic insecurity risk has become a significant concern due to the increasing integration of renewable energy sources and the inherent uncertainties in smart grid. Dynamic security assessment (DSA) has been adopted to hedge against such risks by estimating the stability of large-scale smart grids. Existing DSA approaches often involve complex high dimensional models which incur high communication and computational costs, hindering their practical adoption. In this paper, we address these limitations with the Quantum Federated Ensembled Variational Adaptive Learning (QFEVAL) approach for smart grid DSA. QFEVAL is designed to combine quantum machine learning and federated learning to handle the differential-algebraic equations that describe smart grid stability, providing an efficient way to deal with high-dimensional data and uncertainties. QFEVAL enables the training of the hybrid quantum-classical neural networks on distributed DSA datasets located at different nodes in smart grids, without requiring large numbers of parameters to be transmitted. QFEVAL accurately predicts the stability of the smart grid under various conditions, enabling the implementation of preventive stability control measures. Through extensive experiments, we demonstrate that QFEVAL achieves comparable performance to 9 state-of-the-art DSA approaches with more than 2 orders of magnitude fewer model parameter transmissions. QFEVAL paves the way for reliable, secure, and continuous electricity supply, offering a robust solution to the challenges of DSA in smart grids.

Remote sensing change detection (RSCD) has achieved creditable success in recent years. However, the challenge of identifying changed objects with shape details persists in RSCD. In this letter, we proposed a cross-temporal interaction with difference refinement network (CTIDRNet) to solve interference-caused fake change and incomplete irregular change shape in RSCD tasks. Specifically, by combining cross-attention and self-attention to steer the temporal feature interaction of each input, we design a temporal feature attention (TFA) module to excavate the potential relation of change areas and suppress the unchanged object interference. Afterward, a deformable convolution is used to design a difference feature refinement (DFR) architecture to capture temporal difference information at diverse feature levels. At last, we proposed a multiscale-guided fusion (MGF) module to fuse pyramid features, thereby dealing with scaling changes. Experimental results on three datasets show that CTIDRNet can extract irregularly changed areas effectively, and the evaluation result outperforms other SOTA methods, with an improvement of 1.79%–19.82%, 2.9%–11.07%, and 0.97%–8.91% in terms of F1 for CDD, SYSU, and LEVIR datasets, respectively. The demo code of this work is publicly available athttps://github.com/lucyjiong/CTIDR.

For more efficient generalization to unseen domains (classes), most Few-shot Segmentation (FSS) would directly exploit pre-trained encoders and only fine-tune the decoder, especially in the current era of large models. However, such fixed feature encoders tend to be class-agnostic, inevitably activating objects that are irrelevant to the target class. In contrast, humans can effortlessly focus on specific objects in the line of sight. This paper mimics the visual perception pattern of human beings and proposes a novel and powerful prompt-driven scheme, called "Prompt and Transfer" (PAT), which constructs a dynamic class-aware prompting paradigm to tune the encoder for focusing on the interested object (target class) in the current task. Three key points are elaborated to enhance the prompting: 1) Cross-modal linguistic information is introduced to initialize prompts for each task. 2) Semantic Prompt Transfer (SPT) that precisely transfers the class-specific semantics within the images to prompts. 3) Part Mask Generator (PMG) that works in conjunction with SPT to adaptively generate different but complementary part prompts for different individuals. Surprisingly, PAT achieves competitive performance on 4 different tasks including standard FSS, Cross-domain FSS (e.g., CV, medical, and remote sensing domains), Weak-label FSS, and Zero-shot Segmentation, setting new state-of-the-arts on 11 benchmarks.

Aerial Remote Sensing (ARS) vision tasks present significant challenges due to the unique viewing angle characteristics. Existing research has primarily focused on algorithms for specific tasks, which have limited applicability in a broad range of ARS vision applications. This paper proposes RingMo-Aerial, aiming to fill the gap in foundation model research in the field of ARS vision. A Frequency-Enhanced Multi-Head Self-Attention (FE-MSA) mechanism is introduced to strengthen the model's capacity for small-object representation. Complementarily, an affine transformation-based contrastive learning method improves its adaptability to the tilted viewing angles inherent in ARS tasks. Furthermore, the ARS-Adapter, an efficient parameter fine-tuning method, is proposed to improve the model's adaptability and performance in various ARS vision tasks. Experimental results demonstrate that RingMo-Aerial achieves SOTA performance on multiple downstream tasks. This indicates the practicality and efficacy of RingMo-Aerial in enhancing the performance of ARS vision tasks.

Scene graph generation (SGG) in satellite imagery (SAI) benefits promoting understanding of geospatial scenarios from perception to cognition. In SAI, objects exhibit great variations in scales and aspect ratios, and there exist rich relationships between objects (even between spatially disjoint objects), which makes it attractive to holistically conduct SGG in large-size very-high-resolution (VHR) SAI. However, there lack such SGG datasets. Due to the complexity of large-size SAI, mining triplets subject, relationship, object heavily relies on long-range contextual reasoning. Consequently, SGG models designed for small-size natural imagery are not directly applicable to large-size SAI. This paper constructs a large-scale dataset for SGG in large-size VHR SAI with image sizes ranging from 512 × 768 to 27,860 × 31,096 pixels, named STAR (Scene graph generaTion in lArge-size satellite imageRy), encompassing over 210K objects and over 400K triplets. To realize SGG in large-size SAI, we propose a context-aware cascade cognition (CAC) framework to understand SAI regarding object detection (OBD), pair pruning and relationship prediction for SGG. We also release a SAI-oriented SGG toolkit with about 30 OBD and 10 SGG methods which need further adaptation by our devised modules on our challenging STAR dataset. The dataset and toolkit are available at: https://linlin-dev.github.io/project/STAR.

Siamese network based trackers develop rapidly in the field of visual object tracking in recent years. The majority of Siamese network based trackers now in use treat each channel in the feature maps generated by the backbone network equally, making the similarity response map sensitive to background influence and hence challenging to focus on the target region. Additionally, there are no structural links between the classification and regression branches in these trackers, and the two branches are optimized separately during training. Therefore, there is a misalignment between the classification and regression branches, which results in less accurate tracking results. In this paper, a Target Highlight Module is proposed to help the generated similarity response maps to be more focused on the target region. To reduce the misalignment and produce more precise tracking results, we propose a corrective loss to train the model. The two branches of the model are jointly tuned with the use of corrective loss to produce more reliable prediction results. Experiments on 5 challenging benchmark datasets reveal that the method outperforms current models in terms of performance, and runs at 38 fps, proving its effectiveness and efficiency.

Remote sensing scene graph generation aims to capture both objects and their semantic relationships, offering a comprehensive understanding of complex scenes. However, two major challenges hinder the performance of existing methods. First, remote sensing images often contain a large number of objects, many of which are unrelated. Performing global feature interactions across all objects introduces noise from irrelevant pairs, degrading feature quality. Second, relationship categories in remote sensing scenes exhibit significant intra-class variation across different contexts, and long-tailed distribution further complicates learning due to limited samples for tail classes. To address these issues, we propose the Scene-specific Multi-Prototype Network (SSMP). Our method performs contextual interactions selectively based on object and relationship categories, reducing interference from irrelevant features. Moreover, we introduce a scene-specific multi-prototype classification framework that better captures the diverse visual manifestations of each relationship class, while also improving discrimination under long-tailed distributions. Experimental results demonstrate that the proposed model achieves state-of-the-art (SOTA) performance, with a minimum improvement of 4.5% and a maximum improvement of 21.3% in mR@20 on the PredCls task over baseline models.

Instance segmentation of remote sensing images (RSIs) is an essential task for a wide range of applications such as land planning and intelligent transport. Instance segmentation of RSIs is constantly plagued by the unbalanced ratio of foreground and background and limited instance size. And most of the instance segmentation models are based on deep feature learning and contain operations such as multiple downsampling, which is harmful to instance segmentation of RSIs, and thus the performance is still limited. Inspired by the recent superior performance of prompt learning in visual tasks, we propose a new prompt paradigm to address the above issues. Based on the existing instance segmentation model, first, a local prompt module is designed to mine local prompt information from original local tokens for specific instances; second, a global-to-local prompt module is designed to model the contextual information from the global tokens to the local tokens where the instances are located for specific instances. Finally, a proposal’s area loss function (PAreaLoss) is designed to add a decoupling dimension for proposals on the scale to better exploit the potential of the above two prompt modules. It is worth mentioning that our proposed approach can extend the instance segmentation model to a promptable instance segmentation model, i.e., to segment the instances with the specific boxes’ prompt. The time consumption for each promptable instance segmentation process is only 40 ms. This article evaluates the effectiveness of our proposed approach based on several existing models in four instance segmentation datasets of RSIs, and thorough experiments prove that our proposed approach is effective for addressing the above issues and is a competitive model for instance segmentation of RSIs.

Noticeable achievements have been made in entity-level perception tasks (e.g., object detection) in remote sensing (RS) image interpretation. But for RS images carrying rich content, individual perception cannot well obtain the interaction patterns between entities. The recognition of relationships between entities is the key to deeply understanding RS scenes. In this article, we propose a hypergraph-guided multimodal prototype network (HMPNet), which performs relation recognition by matching relation representations with multimodal predicate prototypes. To overcome the imbalance of modal information in the matching process, a multimodal calibration strategy is devised, taking into account the image subprototype and text subprototype, which makes prediction results more reliable. Meanwhile, to align image and text subprototypes and explore relevant semantic patterns, the multimodal hypergraph is constructed to efficiently capture the associations between heterogeneous prototypes. Experimental results show that the performance of our model can reach the state-of-the-art (SOTA) level on the RS scene graph generation (SGG) task.

Aircraft detection in synthetic aperture radar (SAR) imagery is significant due to its critical role in various applications, including surveillance, reconnaissance, and security. However, given the background interference and discreteness of aircraft scattering, detectors are prone to acquire unremarkable aircraft features. These factors lead to false alarms and present difficulties in locating aircraft accurately. This article proposes an innovative scattering topology cue-based network (STC-Net), which enhances aircraft discriminability and more accurately evaluates the quality of the prediction results. We model the aircraft with the star topology (ST), which not only emphasizes critical components like the nose and wings but also explicitly links them as a cohesive unit. Based on the cue of ST, the ST space fusion module (ST-SFM) and the ST channel attention module (ST-CAM) are designed. The former integrates discrete components to reestablish the aircraft features based on neighboring information of ST, while the latter suppresses background interference to highlight the aircraft by exploiting node information of ST. In addition, completeness and consistency loss (CCLoss) function that includes the completeness-aware label and the positive sample weighting function is introduced. The completeness-aware label describe the localization accuracy by incorporating the degree of overlap of predicted results on ST, while the positive sample weighting function enhances the consistency of the classification and localization branches. Furthermore, experiments conducted on the Gaofen-3 SAR aircraft detection dataset (GF3ADD) and the publicly available SAR-AIRcraft-1.0 dataset demonstrate the effectiveness and generalizability of STC-Net, with our method achieving state-of-the-art performance.

Radio frequency interference (RFI) is an important factor affecting microwave remote sensing observations, causing random degradation of synthetic aperture radar (SAR) images. Due to the huge amount of raw echo and single-look complex (SLC) data, there are some SAR interpretation scenarios when only amplitude images can be obtained, and traditional signal transformation and matrix operations can hardly meet the suppression requirements in the absence of phase information at this time. Although deep learning algorithms have made some progresses on this issue, they still suffer from the following limitations: (i) There are few models dedicated to SAR RFI in the image domain; (ii) The network structure is relatively complex due to not fully exploit the physical characteristics of SAR and RFI. To this end, we propose an end-to-end suppression network (PMNet), which includes a novel explainable feature decomposition module (FDM) based on the idea of non-negative matrix factorization and a composite loss function to achieve dynamic separation of foreground and background features of supervised RFI-contaminated images. The ablation experiment proves that compared with the baseline algorithm, the visual similarity of the proposed PMNet on the test set can be improved by up to 14.15%. The suppression result on Sentinel-1 and Gaofen-3 real data also verifies the effectiveness of the PMNet in different SAR platforms.

Remote sensing change detection (RSCD) has become an essential tool in observing and analyzing geographical information. However, existing deep learning approaches dependent solely on visual modalities may encounter challenges in discerning subtle variations amidst noise interference. To overcome these issues, we propose a multiscale semantic-guided synergistic interaction network (MSSI-Net), which utilizes the advanced multimodal semantic representations for enhancing the capacity to perceive hierarchical changes. Specifically, we first devise a multiscale interaction module (MIM) which leverages multiscale attention mechanism to guide the interaction between the coarse and fine stages of different visual features. The fine-grained visual features subsequently complement the semantic features through scale weight reassignment to enhance the discriminative capability of vision-language features. Furthermore, driven by the semantic-guided synergistic interaction mechanism, our developed cross-modal feature fusion module (CFFM) exploits both homogeneous and heterogeneous features among modalities. This ensures that the generated vision-language features are semantically representative. Finally, we formulate a manifold differential perception head (MDPH) to optimize the detection of changes by efficiently fusing diverse differential feature representations, achieving comprehensive performance enhancement. Extensive experiments conducted on four benchmark datasets (LEVIR-CD, CDD, SYSU-CD and WHU-CD) indicate that the designed MSSI-Net achieves state-of-the-art performance compared to existing methods.

Multi-unmanned aerial vehicle (UAV) collaborative 3-D object detection can comprehend complex environments by integrating complementary information, with applications encompassing traffic monitoring, delivery services, and agricultural management. However, the extremely broad observations in aerial remote sensing and significant perspective differences across multiple UAVs make it challenging to achieve precise and consistent feature mapping from 2-D images to 3-D space in multi-UAV collaborative 3-D object detection paradigm. To address the problem, we propose an unparalleled camera-based multi-UAV collaborative 3-D object detection paradigm called UCDNet. Specifically, the depth information from the UAVs to the ground is explicitly utilized as a strong prior to provide a reference for more accurate and generalizable feature mapping. Additionally, we design a homologous point geometric consistency loss as an auxiliary self-supervision, which directly influences the feature mapping module, thereby strengthening the global consistency of multiview perception. Experiments on AeroCollab3D and CoPerception-UAVs datasets show that our method increases 4.7% and 10% mean Average Precision (mAP) respectively compared to the baseline, which demonstrates the superiority of UCDNet.

Recently, multimodal large language models (MLLMs) have shown excellent reasoning capabilities in various fields. Most of the existing remote sensing (RS) MLLMs solve image-level text generation problems (e.g., image captioning), but ignore the core issues of object-level recognition, location, and multitemporal changes in the field of RS. In this article, we propose RingMoGPT, a multimodal foundation model that unifies vision, language, and localization. Based on the idea of domain adaption, RingMoGPT can complete training by fine-tuning only a few parameters. To make the model capable of object detection and change captioning, we further propose a location- and instruction-aware querying transformer (Q-Former) and a change detection module, respectively. To improve the performance of RingMoGPT, we carefully design the pretraining dataset and the instruction-tuning dataset. The pretraining dataset contains over a half million high-quality image and text pairs, which are generated through a low-cost and efficient data generation paradigm. The instruction-tuning dataset contains more than 1.6 million question-answer pairs, including six downstream tasks: scene classification, object detection, visual question answering (VQA), image captioning, grounded image captioning, and change captioning. Our experiments show that RingMoGPT performs well on six tasks, especially its ability to analyze multitemporal data changes and identify dense objects. We also verified the model under a zero-shot setting, and the results show that the proposed RingMoGPT also has good generalization ability in the face of new data.

Remote sensing lightweight foundation models have successfully achieved online perception, providing real-time intelligent interpretation. However, their capabilities are restricted to inferences solely based on their respective observations and models, thus lacking a comprehensive understanding of large-scale remote sensing scenarios. To address this limitation, we propose RingMo-Galaxy, a remote sensing distributed foundation model based on generalized information mapping and interaction. RingMo-Galaxy can realize online collaborative perception across multiple platforms and diverse downstream tasks by mapping observations into a unified space and implementing a task-agnostic information interaction strategy. Specifically, we leverage the ground-based geometric prior of remote sensing oblique observations to change feature mapping from absolute to relative depth estimation, thereby enhancing the model’s ability to extract generalized features across diverse heights and perspectives. In addition, we present a dual-branch information compression module to decouple high-frequency and low-frequency features, achieving feature-level compression while preserving critical task-agnostic details. To support our research, we collect a multitask simulation dataset named AirCo-MultiTasks, specifically designed for multi-unmanned aerial vehicle (UAV) collaborative observation. We also conduct extensive experiments, including 3-D object detection, instance segmentation, and trajectory prediction. The numerous results demonstrate that our proposed RingMo-Galaxy achieves state-of-the-art performance across various downstream tasks.

Object detection and recognition (OD&R) based on deep learning is a hot topic in the application of synthetic aperture radar (SAR). These methodologies based on deep learning are inherently data-driven, which means that their performance is subjected to the corresponding datasets. Although existing datasets have included some common targets collected from real-valued intensity SAR images, there still exist some limitations in terms of quantity, categories, diversities, and data domain. Hence, it is urgent to establish a large-quantity benchmark for fine-grained OD&R on complex-valued SAR images, which contains rich signal-domain features well coupled with classical physical modeling. In addition, considering the unique imaging characteristics and diverse imaging conditions, some important attribute information, such as incidence and attitude angles, is necessary to be attached. In this article, we propose a novel benchmark dataset with more than 340k instances for fine-grained OD&R based on single-look complex (SLC) SAR images, which is named FAIR-CSAR. We collected complex-valued SAR images with a resolution of 1–5 m from 175 entire images of Gaofen-3 covering 32 cities and multiple sea areas worldwide. All instances in the FAIR-CSAR are annotated by oriented bounding boxes (OBBs), covering five major categories and 22 subcategories. Compared with existing datasets dedicated to OD&R, the FAIR-CSAR dataset has four particular advantages: 1) it contains complex-valued SAR images from various acquisition modes and polarization modes, including full-scale signal-domain features for object recognition; 2) it is much larger than other existing OD&R datasets in terms of quantity of instances; 3) it provides more fine-grained category annotation and more detailed attribute information; and 4) it provides more challenging images with some common imaging phenomena, such as speckle noise and azimuth ambiguities. To establish a baseline adapted for SLC SAR images, a multidomain feature extraction and fusion network (MDNet) is proposed as a novel framework to mine detailed information underlying various domains. A series of state-of-the-art (SOTA) algorithms are applied on the FAIR-CSAR to build the fine-grained OD&R benchmark. Experimental results indicate that FAIR-CSAR is closer to practical application and more challenging than existing datasets for SAR images.

The rapid development of Remote Sensing (RS) technology has generated a vast amount of heterogeneous time series data from various sources, including drone videos, satellite time-series images, and multi-object trajectories. Effectively processing and analyzing this multi-source heterogeneous data for accurate spatiotemporal prediction is crucial in fields such as environmental protection and disaster response. In this paper, we propose a universal predictive foundation model named Ringmo-SenseV2 to learn the general evolutionary patterns of RS elements from massive heterogeneous data. Ringmo- SenseV2 features a Mixture-of-Heterogeneous-Experts (MoHE) Transformer, which unifies the modeling of multi-source heterogeneous time-series data. Additionally, to better capture the complex dependencies across different spatiotemporal locations, we introduce a hypergraph translator, treating embeddings of different spatiotemporal locations as nodes and employing hypergraph convolution for information propagation. Furthermore, to enhance the model’s adaptability to different evolution speeds during pre-training, we implement the Adaptive tube Masking (AM) strategy, which controls prediction difficulty by adaptively setting mask proportions for sequences with varying evolution speeds. Extensive experiments demonstrate that Ringmo-SenseV2 exhibits outstanding performance across various RS prediction tasks. Further tests on scene graph generation for RS images showcase the model’s ability to extract image features, thereby enhancing image perception tasks.

Scene graph generation (SGG) is a high-level visual understanding and reasoning task aimed at extracting entities (such as objects) and their interrelationships from images. Significant progress has been made in the study of SGG in natural images in recent years, but its exploration in the domain of remote sensing images remains very limited. The complex characteristics of remote sensing images necessitate higher time and manual interpretation costs for annotation compared to natural images. The lack of a large-scale public SGG benchmark is a major impediment to the advancement of SGG-related research in aerial imagery. In this article, we introduce the first publicly available large-scale, million-level relation dataset in the field of remote sensing images, which is named ReCon1M. Specifically, our dataset is built upon FAIR1M and comprises 22 262 images. It includes annotations for 873 761 object bounding boxes across 60 categories and 1 052 223 relation triplets across 59 categories based on these bounding boxes. We provide a detailed description of the dataset’s characteristics and statistical information. In addition, an efficient global context-aware network (EGCAN) is proposed to improve inference efficiency in dense relation prediction through an object-pair pre-screening mechanism. By integrating visual, spatial, and semantic features, EGCAN captures fine-grained pairwise features and object-level contextual information to enhance its ability to discriminate relation. We conduct two object detection tasks and three subtasks within SGG on this dataset, assessing the performance of mainstream methods on these tasks. The experimental results show that the proposed EGCAN achieves state-of-the-art (SOTA) performance in 17 out of 24 accuracy metrics across three tasks and delivers the best performance in frames per second (FPS) for model inference. The ReCon1M dataset and related resources are available athttps://recon1m-dataset.github.io/

Trajectory prediction is an essential task within various automation systems. Recent studies have highlighted that the social interactions among multiple agents are crucial for accurate predictions, relying on empirically derived human-imposed constraints to model these interactions. However, from a sociological perspective, agents’ interactions exhibit significant inherent randomness. Dependence on a priori knowledge may lead to biased estimations of data distributions across different scenarios, failing to account for this randomness. Consequently, such methodologies often do not comprehensively capture the full spectrum of social influences, thus limiting the models’ predictive efficacy. To address these issues, we propose a novel multi-agent trajectory prediction framework, SoPerModel, which incorporates a freeform social evolution module (FSEM) and a local perception attention mechanism (LPA). The FSEM enables SoPerModel to naturally capture representative social interactions among agents without the reliance on additional human-derived priors. Through LPA, the model integrates both local and global social interaction information and leverages them to enhance trajectory prediction performance. Our framework is empirically evaluated on real-world trajectory prediction datasets, and the results demonstrate that our approach achieves a highly competitive performance compared with state-of-the-art models.

The terminal-cloud collaborative framework boosts precision and efficiency by integrating cloud computing power with low-latency terminal responsiveness, offering a suitable solution for the growing demands of multiplatform remote sensing (RS) image interpretation. However, the significant differences in data distribution across various RS platforms present a great challenge in balancing the cloud’s centralized processing capabilities with the local interpretation abilities of different terminals. To address this challenge, we propose a terminal-cloud cross-domain collaborative (TeCCo) framework that inherits the efficiency advantages of multiple platforms while ensuring high-accuracy interpretation of diverse data distributions from different terminals. First, the dual classifier co-learning (DCCL) module is designed to enhance cloud robustness. By combining a multilayer perceptron for instance-level classification and a graph convolutional network (GCN) for feature-level aggregation, it achieves mutual supervision and improves feature alignment across different data distributions. Second, the hypernetwork personalization (HNP) module is introduced to generate personalized classifier parameters for each terminal with little fine-tuning cost, allowing terminals to maintain their uniqueness while benefiting from the generalization advantages of collaborative training. Finally, a data-assisted progressive inference mechanism is proposed to enhance accuracy by jointly clustering the features transmitted from terminals and the features of supervised data in the cloud. Extensive experiments demonstrate that TeCCo effectively addresses data distribution challenges, enhancing both the generalization of the cloud model and the personalization of terminal models, achieving state-of-the-art (SOTA) performance in cross-domain and multiplatform RS image classification.

Few-shot object detection (FSOD) has garnered widespread attention in recent years, which makes it possible to learn novel classes with only a handful of labeled samples. Due to the obvious long-tail distribution of remote sensing (RS) data and serious challenges in data labeling, FSOD holds greater practical application value in RS. At present, the FSOD algorithms with fine-tuning of some parameters have attracted much attention due to their stronger incremental learning capacity. However, given large intraclass scale variations and small interclass feature differences of RS objects, we still need to place greater emphasis on the localization and classification of objects. In this article, we introduce the RoI feature refinement (RIFR) method for FSOD in RS imagery, which adopts a powerful training pipeline to better balance attention to the performance of base and novel classes. Aiming at large intraclass scale variations of RS objects, we design a scale-aware feature compensation module (SAFCM). By compensating for insufficient scale information, the model’s ability to perceive scale variations of the same class objects has been enhanced. Considering the high similarity among RS classes, we come up with a prototype trihard (PT) loss. It achieves the effect of interclass separability by constraining the relationship between samples and prototypes. Thus, the issue of interclass confusion has been resultfully solved. Comprehensive experiments on three datasets, DIOR, NWPU VHR-10.v2, and FAIR1M-Airplane, can showcase the efficacy of our RIFR method, and it can implement the most outstanding performance currently. The code will be available at:https://github.com/ningerhhh/RIFR.

Multimodal hate detection plays a crucial role in maintaining harmonious online environments by identifying harmful content, such as hateful memes. Although previous research has made significant progress in detecting explicit hate speech, there remains a critical gap in analyzing implicit hate, which is particularly challenging due to the absence of explicit harmful text claims or demographic visual cues. Despite the promising results based on cross-modal attention, previous methods may suffer from the distributional modality gap caused by the non-literal associations between multimodal elements, which lacks apparent alignment in implicit hateful contents. In this work, we propose a novel framework: Flexible Optimal Transport (FLOT) to capture the non-literal cross-modal alignment for multimodal hate in the context of memes. FLOT formulates the problem of cross-modal alignment as finding optimal transportation plans, which leverages a kernel method to capture complementary information from multiple modalities. The kernel embeddings reproduce a kernel Hilbert space (RKHS) to serve as a non-linear transformation of alignment, which effectively reduces the distributional modality gap with more interpretability. Moreover, we established topological structures with contrastive modeling for the aligned representations, which are optimized to achieve comprehensive alignment between different modalities, and facilitate local reasoning based on multimodal elements. Experimental results have demonstrated that our FLOT achieved state-of-the-art performance on three publicly available benchmark datasets. Furthermore, extensive qualitative analysis confirms the superior ability of FLOT in capturing implicit cross-modal alignment.

In this paper, we propose a new large scale dataset dedicated to fine-grained building extraction in Chinese regions. To ensure the representativeness of the dataset samples, various factors are taken into consideration, encompassing geographical distribution, appearance variety, rural areas and spatial layout. Consequently, the dataset is composed of 131 K building instances, covering an expansive area of 784.3 square kilometers across 34 cities. Each building is represented with a polygon and a corresponding rooftop type. The rooftops are categorized into 12 fine-grained types. Moreover, we conduct experiments using four classical methods on this dataset to establish baselines for future studies. This dataset will promote the refined development of the city management and planning by providing essential resources for developing the state-of-the-art methods to accurately identify building instances on a city or national scale.

Floods represent a pervasive natural hazard with global ramifications, impacting a vast population and resulting in substantial property damage and severe mortality. Particularly worrisome is their disproportionate effect on the least developed countries, which exacerbates developmental imbalances, posing a significant obstacle to the attainment of the United Nations Sustainable Development Goals (UN SDGs). This paper introduces the AI-powered Flood Mapathon activity, co-organized by the Aerospace Information Research Institute under the Chinese Academy of Sciences, in partnership with GEOVIS Technology Co., Ltd., GEOVIS Earth Technology Co., Ltd., and IEEE GRSS IADF. The activity seeks to mobilize individuals worldwide to address the most prevalent natural hazard-floods by collaboratively mapping inundated regions through the analysis of satellite imagery. Gaining widespread attention, the activity has garnered 30,755 submissions from 310 participants across 34 countries. Through collective efforts, participants have curated a semantic segmentation dataset focusing on floods, incorporating annotations of pertinent features related to both floods and human activities. Additionally, the paper elucidates the custom crowdsourcing mapping system, which seamlessly integrates cutting-edge AI technologies to alleviate mapping complexities. The activity contributes to sustainability by drawing extensive public attention, creating a public flood dataset for academic research, and establishing an efficient and intelligent mapping system.

With the development of large language models (LLMs) and remote sensing technology, visual language (VL) tasks in the field of remote sensing have attracted more and more research attention. Commonly used VL datasets currently usually focus on the overall scene of the image, lacking the description of instance-level details such as location, size, category and so on. More importantly, users are usually not allowed to directly intercept regions in the image to ask questions through these datasets. However, the instance-level question answering based on these information is of great significance for target extraction in practical applications. In this manuscript, we build an innovative and challenging dataset FAIR1M-GQA. It unlocks the ability of the model to learn directly from text input and text output both with region coordinates, which are directly linked to fine-grained objects in remote sensing images. We experiment our dataset to verify the feasibility of the relevant task and provide the benchmark results.

Recently, large-scale Vision-Language Models (VLMs) have gained widely attention in the field of remote sensing. However, the researching on VLM requires a substantial amount of data, which is relatively scarce in the remote sensing domain. To overcome this limitation, in this paper, we present ViGen500K, a larger and more challenging image-text dataset. Nearly 500,000 images have been collected, accompanied by over 1 million annotations to adapt to the diverse requirements of various image-text tasks in remote sensing. Besides, a promising, efficient, low-cost, and highly automated data annotation method is proposed to make our dataset could be easily extensive by keeping adding extra unlabeled remote sensing images. Theoretically, ViGen500K is an infinitely large dataset. From a quantitative point of view, compared with traditional image caption datasets, ViGen500K not only has more images but also covers more object categories, which enables the model trained on our dataset could have a wider range of target-text alignment capabilities. Several experiments have been conducted to provide benchmarks for our dataset.

Collaborative trajectory prediction can comprehensively forecast the future motion of objects through multi-view complementary information. However, it encounters two main challenges in multi-drone collaboration settings. The expansive aerial observations make it difficult to generate precise Bird's Eye View (BEV) representations. Besides, excessive interactions can not meet real-time prediction requirements within the constrained drone-based communication bandwidth. To address these problems, we propose a novel framework named "Drones Help Drones" (DHD). Firstly, we incorporate the ground priors provided by the drone's inclined observation to estimate the distance between objects and drones, leading to more precise BEV generation. Secondly, we design a selective mechanism based on the local feature discrepancy to prioritize the critical information contributing to prediction tasks during inter-drone interactions. Additionally, we create the first dataset for multi-drone collaborative prediction, named "Air-Co-Pred", and conduct quantitative and qualitative experiments to validate the effectiveness of our DHD framework. The results demonstrate that compared to state-of-the-art approaches, DHD reduces position deviation in BEV representations by over 20\% and requires only a quarter of the transmission ratio for interactions while achieving comparable prediction performance. Moreover, DHD also shows promising generalization to the collaborative 3D object detection in CoPerception-UAVs.

With the development of on-orbit hardware and distributed multiplatform observation systems in satellite remote sensing (RS) scenario, on-orbit collaborative model updating has become a promising trend. Due to restrictions of imaging conditions and storage resources, on-orbit updating is usually carried out with limited samples. However, existing collaborative learning methods rarely consider the few-shot problem. To address this issue, this letter innovatively proposes a distributed collaborative learning method for few-shot RS image classification (FS-DCL), which encourages the collaboration between satellites with similar data distribution to supplement useful information for each satellite, and design on-orbit models to extract more discriminative features. Specifically, a personalized parameter aggregation strategy (PPAS) is proposed to generate personalized parameters for each satellite based on information from satellites with similar data distributions, providing information gain to alleviate problems of insufficient samples. Besides, a feature enhancement method (FEM) is applied to on-orbit models to enhance the feature representation and produce a more discriminative feature space, thus improving the accuracy of few-shot metric classification. Extensive experiments on two RS datasets demonstrate the superiority of FS-DCL.

Multimodal summarization (MS) for videos aims to generate summaries from multi-source information (e.g., video and text transcript), showing promising progress recently. However, existing works are limited to monolingual scenarios, neglecting non-native viewers' needs to understand videos in other languages. It stimulates us to introduce multimodal cross-lingual summarization for videos (MCLS), which aims to generate cross-lingual summaries from multimodal input of videos. Considering the challenge of high annotation cost and resource constraints in MCLS, we propose a knowledge distillation (KD) induced triple-stage training method to assist MCLS by transferring knowledge from abundant monolingual MS data to those data with insufficient volumes. In the triple-stage training method, a video-guided dual fusion network (VDF) is designed as the backbone network to integrate multimodal and cross-lingual information through diverse fusion strategies in the encoder and decoder; What's more, we propose two cross-lingual knowledge distillation strategies: adaptive pooling distillation and language-adaptive warping distillation (LAWD), designed for encoder-level and vocab-level distillation objects to facilitate effective knowledge transfer across cross-lingual sequences of varying lengths between MS and MCLS models. Specifically, to tackle lingual sequences of varying lengths between MS and MCLS models. Specifically, to tackle the challenge of unequal length of parallel cross-language sequences in KD, LAWD can directly conduct cross-language distillation while keeping the language feature shape unchanged to reduce potential information loss. We meticulously annotated the How2-MCLS dataset based on the How2 dataset to simulate MCLS scenarios. Experimental results show that the proposed method achieves competitive performance compared to strong baselines, and can bring substantial performance improvements to MCLS models by transferring knowledge from the MS model.

Change captioning aims to describe the semantic change between a pair of images with natural language while remaining immune to viewpoint change. Based on the encoder-decoder architecture, most existing methods primarily focus on encoding effective change representations for transmission to the decoder. However, they suffer from an insufficient understanding of visual semantics, inadequate single-pass feature comparison, and a confounding bias caused by imbalanced viewpoint change data. These impair change representations and hinder unbiased caption generation. In this paper, we analyze and identify the confounding bias from a causality perspective and propose a Relation-aware Multi-pass Comparison Deconfounded (RMCD) network for change captioning, which elevates the encoding of change representations and mitigates the bias. Specifically, in the encoding stage, to sufficiently understand visual semantics, a position-guided context aggregating module is presented to capture the positional and contextual relations among objects in the image. Then, to achieve comprehensive change representations, we present a multi-pass feature comparison module to recognize semantic differences at various feature levels and progressively integrate them. In the decoding stage, to generate de-biased captions, the causal intervention is employed to remove the confounding bias which introduces spurious correlations between encoded change representations and captions. The newly achieved state-of-the-art performance on four publicly available benchmark datasets and further visual analysis demonstrate the superiority of our method.

Compared with binary change detection (BCD), semantic change detection (SCD) further provides the category information of bitemporal changed regions which is significant for the practical application of Earth Observation. Although the recently proposed triple-branch structures including one BCD branch and two classification branches can effectively achieve the task balance, they still need to employ the carefully designed difference extraction module and branch interactions to capture the bitemporal correlations, which increases the complexity of the semantic information utilization. In this paper, we propose a new triple-branch network named JFRNet to tackle this challenge. From the perspective of the SCD process, because the category information and the change information are both derived from bitemporal images, we take the joint bitemporal features as the unified input, which can help each branch perceive the bitemporal semantic correlations without any additional interaction operations. From the perspective of the SCD structure, we introduce the convolutional attention fusion module (CAFM) and the convolutional attention refinement module (CARM) to unify the branch structure, which can help our model refine the unique semantic information without any specially designed difference extraction modules. Extensive experiment results on three available datasets indicate that compared with the baseline methods, our proposed JFRNet successfully simplifies the reasoning process and obtains the better SCD performance.

A weakly supervised object detection (WSOD) task is to train a detector with only image-level labels provided. Except for the training difficulty introduced by weaker annotations, the inherent complexity of the remote-sensing images (RSIs) also adds to the challenge. To boost the detector’s localization accuracy, we aim to exploit more semantic information contained in images and help improve the general robustness of the model. Noticing previous methods tend to focus on the most discriminative part of an object, we design a self-training-based network that leverages local semantic features. To this end, we develop a semantic-balanced localization module (SBLM) that distinguishes foreground from background and accurate proposals from incomplete ones, by leveraging a balance of region of interest (ROI) and its context information. Moreover, we find that the self-training strategy highly relies on the quality of pseudo-ground-truth boxes. Motivated by this possible lack of robustness, we design a comprehensive clustering module (CCM) and saliency-based proposal filtering (SPF) module that select pseudo-ground truth more comprehensively under supervision. To be more specific, CCM aims to reduce the arbitrariness during assigning pseudo-labels by considering multiple categorical vectors simultaneously. Salient object detection (SOD) is applied in the SPF module to help evaluate the quality of the chosen pseudo-ground-truth boxes. The detection performance is significantly boosted with the proposed method. Extensive experiments conducted on the NWPU VHR-10.v2 dataset and the DIOR dataset validate that the proposed model outperforms the previous state-of-the-art methods favorably with an mAP of 64.9% and 28.1%, respectively.

With the recent development of remote sensing (RS) technology, the amount of RS platforms has witnessed a substantial increase, and the capacity of Earth observation has been greatly enhanced. The interpretation of RS images has also gradually evolved from traditional centralized ground processing to on-orbit processing. However, the traditional single-platform on-orbit processing is limited to a single source of information, which results in the underutilization of the advantages of multiplatform observation in the current RS field, and restricts the accuracy of inference tasks. To tackle the aforementioned problem, we propose a multiplatform distributed collaborative inference network, which can combine the intermediate features from multiple platforms to improve the accuracy of inference tasks. First, we proposed the collaboration map generator, which generates the collaboration map for optimal collaborator selection autonomously. Second, a spatial feature compression (SFC) module is designed to compress the interplatform transmission features, adapting spatially sparse distribution characteristics of RS objects. Finally, a feature fusion module containing spatial priors is proposed to fuse the features collected from multiple platforms to obtain more precise inference results. We conducted extensive experiments on three public datasets and verified the effectiveness of the proposed framework. On the NWPU VHR-10 dataset, for example, the proposed method improves the detection accuracy by 13.7% and 10.3% under two experimental settings compared with a single platform and compresses the intermediate data transmission between platforms by more than 80%.

Due to the diversity of acquisition conditions and imaging mechanisms in remote sensing, the generalization of semantic segmentation models trained with labeled data in the source domain to other unlabeled target domains is hindered. Existing mainstream self-training-based methods provide pseudo-labels to target data as ground truth to utilize target domain evidence for unsupervised domain adaptation (UDA). However, the label shift and domain gap between different domains inevitably introduce noise into pseudo-labeled target data, that is, misclassified pixels. As a consequence, we present a dynamic loss correction (DLC) framework for cross-domain semantic segmentation, which mitigates domain discrepancy by formally modeling the noise distribution of pseudo-labels in the target domain with noise transition matrix (NTM). Specifically, to promote the model output to fit the true label distribution, we employ the high-order consistency information of neighbor representations to estimate NTM and correct the supervision signal without heuristically setting anchors. Furthermore, smooth geometric constraints are introduced to regularize the mutual improvement of NTM derivation and segmentation model optimization in a data-driven manner, thereby compensating for the lack of target domain knowledge. Extensive experimental results on four cross-domain remotely sensed segmentation tasks highlight the generalization capability and competitiveness of the presented method, including cross-scene, cross-band, and cross-modal transfer. Our results and code are available athttps://github.com/heqibin/dlc.

With well-pretrained foundation models (FMs), the performance of almost every remote sensing interpretation task has been boosted. The parameter volume of FMs increases with their continuously enhanced capabilities, leading to increased costs of fine-tuning. To apply FMs more effectively and efficiently, there are already some arts that introduce the parameter-efficient fine-tuning (PEFT) concept into remote sensing and achieve competitive performance with much lower parameter cost. However, the training efficiency of most PEFT frameworks may be not satisfactory. To make tuning FMs for remote sensing applications more efficient, we propose a training-efficient adapting (TEA) framework. Specifically, we attach a SIDE adapter network (SIDEAN) to the frozen powerful FMs and only update the SIDEAN to perform the downstream tasks. Moreover, to make TEA perceive remote sensing scenes from a macroscopic perspective and boost the performance, we propose a top-down guidance mechanism to inject macro scene information into the SIDEAN during adapting. TEA is also parameter-efficient, as SIDEAN is designed to be lightweight. We conduct extensive experiments to demonstrate the effectiveness and efficiency of TEA on ten widely adopted datasets covering four primary remote sensing tasks, e.g., object detection, orientated object detection, semantic segmentation, and scene classification. By training only 5.43% of the frozen FM parameters, TEA can save more than 57% of training memory footprint and up to 15% of time cost on average while achieving competitive performance on all datasets. Furthermore, TEA can surpass full fine-tuning on several datasets.

Remote sensing is stepping into the era of the foundation model, where the fine-tuning paradigm is widely adopted to transfer the profound knowledge of pretrained foundation models to downstream tasks. However, the full fine-tuning method would become inefficient in terms of training and storage, as the foundation models are getting larger and larger. Recently, a lot of deep learning research has proposed various parameter-efficient fine-tuning (PEFT) methods that perform well with a few trainable parameters. However, most of them focus on fine-tuning general foundation models without considering the special properties of remote sensing. In this article, we propose an adapter in remote sensing (AiRs) to fine-tune large foundation models for remote sensing downstream tasks by introducing the adapter-tuning framework. Specifically, we construct AiRs from two aspects: more expressive adaptation modules and a more efficient integration strategy. Specialized adaptation modules are applied to different functional layers in AiRs, which encode the inductive bias of remote sensing images and enhance the semantic concepts of geography. Moreover, AiRs establishes pathways between trainable modules with residual connections, which reduces training difficulty and improves performance. We conduct extensive experiments on object detection, semantic segmentation, and scene classification tasks. By training only 4.4% parameters of the pretrained backbone, AiRs surpasses the previous state-of-the-art (SOTA) PEFT competitors on all experimental datasets and outperforms the full fine-tuning on six out of ten datasets.

Remote sensing technology has made remarkable progress, providing a wealth of data for various applications, such as ecological conservation and urban planning. However, the meticulous annotation of this data is labor-intensive, leading to a shortage of labeled data, particularly in tasks like semantic segmentation. Semi-supervised methods, combining consistency regularization with self-training, offer a solution to efficiently utilize labeled and unlabeled data. However, these methods encounter challenges due to imbalanced data ratios. To tackle these challenges, we introduce a self-training approach namedDAST(Dynamic andAdaptiveSelf-Training), which is combined with dynamic pseudo-label sampling, distribution matching, and adaptive threshold updating. Dynamic pseudo-label sampling is tailored to address the issue of class distribution imbalance by giving priority to classes with fewer samples. Meanwhile, distribution matching and adaptive threshold updating aim to reduce distribution disparities by adjusting model predictions across augmented images within the framework of consistency regularization, ensuring they align with the actual data distribution. Experiment results on the Potsdam and iSAID datasets demonstrate that DAST effectively balances class distribution, aligns model predictions with data distribution, and stabilizes pseudo-labels, leading to state-of-the-art performance on both datasets. These findings highlight the potential of DAST in overcoming the challenges associated with significant disparities in labeled-to-unlabeled data ratios.

Most recent unmanned aerial vehicle (UAV) detectors focus primarily on general challenges such as uneven distribution and occlusion. However, the neglect of scale challenges, which encompass scale variation and small objects, continues to hinder object detection in UAV images. Although existing works propose solutions, they are implicitly modeled and have redundant steps, so detection performance remains limited. One specific work addressing the above scale challenges can help improve the performance of UAV image detectors. Compared to natural scenes, scale challenges in UAV images happen with problems of limited perception in comprehensive scales and poor robustness to small objects. We found that complementary learning is beneficial for the detection model to address the scale challenges. Therefore, the article introduces it to form our scale-robust complementary learning network (SCLNet) in conjunction with the object detection model. The SCLNet consists of two implementations and a cooperation method. In detail, one implementation is based on our proposed scale-complementary decoder and scale-complementary loss function to explicitly extract complementary information as a complement, named comprehensive-scale complementary learning (CSCL). Another implementation is based on our proposed contrastive complement network and contrastive complement loss function to explicitly guide the learning of small objects with the rich texture detail information of the large objects, named interscale contrastive complementary learning (ICCL). In addition, an end-to-end cooperation (ECoop) between two implementations and with the detection model is proposed to exploit each potential. In short, SCLNet forms a more comprehensive representation through feature complementary and improves the representation of small objects through interscale contrast, which in turn comes to improve scale robustness and detection performance. Thorough experiments prove the effectiveness of our SCLNet on Visdrone and UAVDT datasets, including the fact that the novel components included in SCLNet are effective and competitive with many CNN-based and transformer-based methods, among other aspects. In general, our SCLNet can effectively address scale challenges and is a competitive model in UAV image object detection.

The remote sensing visual grounding (RSVG) task focuses on accurately identifying and localizing specific targets in remote sensing (RS) images using descriptive query expressions. Existing methods independently extract visual and textual features, ignoring early complementary information between image and text. This leads to information loss and misalignment, limiting the model’s ability to distinguish similar targets. To address this challenge, we propose the query-aware multimodal fusion network (QAMFN), which introduces an innovative query-guided visual attention (QGVA) mechanism in the early stages of the visual encoder. This mechanism integrates textual information during the early visual feature extraction process, thereby resolving the issue of missing image-text complementary information. QGVA ensures that the visual backbone accurately focuses on local features highly relevant to the query by injecting textual information into the visual encoding process. Additionally, to enhance the model’s ability to integrate multimodal information and adapt to more complex RS images, we introduce the text-semantic attention-guided masking (TAM) module. TAM aggregates multimodal features processed by the backbones and filters out redundant information, producing high-quality fused features. Experiments demonstrate that our approach sets a new record on the DIOR-RSVG dataset, improving accuracy to 81.67% (an absolute increase of 4.98%).

Recently, there has been a growing interest in few-shot incremental object detection (FSIOD). It learns new tasks with limited data while mitigating catastrophic forgetting on previous tasks. However, existing FSIOD methods experience parameter changes after training on new tasks, causing a parameter competition issue among tasks. Additionally, the background information differs among various tasks, and using a common background weight for all tasks results in the background shift. Constrained by these two issues, existing methods only alleviate catastrophic forgetting and cannot wholly prevent the performance decline on previous tasks. Especially for complex remote sensing images with messy background, the models trained on new tasks exhibit noticeable performance drops on previous tasks. In this paper, we propose a novel FSIOD method via dual-frequency prompt to address these challenges, named FSIOD-DFP. It can completely eliminate catastrophic forgetting while mitigating over-fitting. Specifically, a dual-frequency prompt generator is designed to tackle the parameter competition issue. It decouples the frequency components of images to produce prompts that modify the images to adapt to the base model trained on previous tasks. Compared to traditional prompts, our generator introduces fewer parameters to address over-fitting for limited data and allows freezing the base model to maintain the performance of previous data. Besides, a self-regularization loss is introduced to guide the prompt-modified images to leverage the knowledge of the base model effectively. Furthermore, we propose a task-decoupled detection head to address the background shift problem. It separates the detection heads for new and previous tasks to resolve the conflict in the background between different tasks. In FSIOD-DFP, only a prompt generator and a novel detection head are added and fine-tuned when learning a new task. Experiments on three remote sensing object detection datasets demonstrate that our method achieves state-of-the-art performance on both new and previous tasks in all few-shot incremental settings.

Research on multiview stereo (MVS) based on remote sensing images has promoted the development of large-scale urban 3-D reconstruction. However, remote sensing multiview image data suffer from the problems of occlusion and uneven brightness between views during acquisition, which leads to the problem of blurred details in depth estimation. To solve the above problem, we reexamine the deformable learning method in the MVS task and propose a novel paradigm based on view space and depth deformable learning (SDL-MVS), aiming to learn deformable interactions of features in different view spaces and deformably model the depth ranges and intervals to enable high accurate depth estimation. Specifically, to solve the problem of view noise caused by occlusion and uneven brightness, we propose a progressive space deformable sampling (PSS) mechanism, which performs deformable learning of sampling points in the 3-D frustum space and the 2-D image space in a progressive manner to embed source features to the reference feature adaptively. To further optimize the depth, we introduce depth hypothesis deformable discretization (DHD), which achieves precise positioning of the depth prior by adaptively adjusting the depth range hypothesis and performing deformable discretization of the depth interval hypothesis. Finally, our SDL-MVS achieves explicit modeling of occlusion and uneven brightness faced in MVS through the deformable learning paradigm of view space and depth, achieving accurate multiview depth estimation. Extensive experiments on LuoJia-MVS and WHU datasets show that our SDL-MVS reaches state-of-the-art performance. It is worth noting that our SDL-MVS achieves a mean absolute error (MAE) error of 0.086 and an accuracy of 98.9% for Acc$_{\lt 0.6\,\text {m}}$and 98.9% for Acc$_{\lt 3-\text {interval}}$on the LuoJia-MVS dataset under the premise of three views as input.

Multi-satellite networks have become the prevalent mode for remote sensing intelligent interpretation, with the onboard models requiring class-incremental updates to accommodate the new categories emerging in evolving data and tasks. Traditional model updating methods, which involve uploading models separately after ground-based updating, are inefficient due to limited uplink bandwidth and cumbersome ground update processes while underutilizing potential computing resources on satellites. To address the aforementioned problems, this paper innovatively proposes a collaborative in-orbit incremental update method termed FCIL-MSN, which leverages observational information and computing resources from multi-satellite networks. Firstly, FCIL-MSN achieves collaborative onboard model updates by introducing federated class-incremental learning into multi-satellite networks. Secondly, a bias calibration-guided relationship distillation module constructs a pseudo-feature set by collaborative multi-satellite networks, which alleviates the model bias caused by class imbalance from a global perspective, thereby enhancing model performance. Finally, a gradient information aggregation module is designed to facilitate the exclusion of unfavorable local updates by measuring the contribution of each terminal, thereby accelerating the convergence while obtaining the global model. We conduct extensive experiments on two datasets for scene classification tasks to verify the effectiveness of our proposed method. Experimental results demonstrate that FCIL-MSN outperforms existing general FCIL methods, improving average classification accuracy by 1.45% and decreasing the performance degradation rate by 6.40%.

Synthetic aperture radar (SAR) ship detection plays an important role in the field of maritime security. However, certain unique imaging properties make it challenging to extract the shape features of ships, such as speckle noise and strong scattering interference from irrelevant objects. These factors result in inaccurate ship localization and obvious false alarms under complex large-scale inshore scenes. To address this issue, we propose the scattering region topology network (SRT-Net), which can dynamically capture the comprehensive global context and enhance the ship saliency. This is achieved through two key modules, namely the scattering region topological structure pyramid (SRTP) and the ship saliency enhancement (SSE) module. The former provides richer semantic information to distinguish the object from the background, while the latter offers an extra pixel-level classification task to guide accurate bounding box regression. Thanks to the guidance of richer information, the proposed method can achieve fewer false alarms and enhance location accuracy. Additionally, we introduce a scale feature adaptive (SFA) loss to balance the attention to ships with various scales, which improves the robustness of multiscale ship detection. The proposed method achieves state-of-the-art performance under complex inshore scenes, and its effectiveness is verified by experiments on a large-scale SAR ship detection dataset (LSSDD) and a public SAR ship detection dataset (SSDD+).

With the continuous improvement of remote sensing satellite technology for earth observation, satellite video object tracking has become a new research field. Tracking aerial objects in satellite video facilitates highly accurate positioning over a wide coverage area which plays a crucial role in remote sensing scenes such as military analysis and disaster prevention. However, for such a data-driven task, most previous satellite video datasets focus on ground object tracking and lack attention to high-speed aerial objects. In this regard, a high-resolution satellite video dataset for detecting and tracking high-speed aerial objects (AIR-HSAO) is built including satellite videos from over 20 regions worldwide, with detailed annotations for approximately 1400 airplane trajectories. Based on the AIR-HSAO, an end-to-end online multi-object tracking method called MGTrack is proposed for high-speed aerial objects. Specifically, the motion prediction module is proposed to implicitly model object motion and associate objects. Guided by the predicted offsets, the feature warping and fusion module aligns and fuses temporal features, fully exploiting the temporal correlations of the video sequence to obtain more accurate feature representations for high-speed objects. Additionally, a trajectory-aware size-enhanced loss is designed to handle the problem of discontinuous trajectories and small-scale objects. Experimental results on AIR-HSAO demonstrate that the proposed MGTrack improves the tracking accuracy by 2.3 MOTA and 1.6 IDF1 compared to the state-of-the-art tracking methods. The code is released at https://github.com/RenLibo-aircas/AIR-HSAT-dataset.

Sidelobes commonly disturb synthetic aperture radar (SAR) image understanding and interpretation. Traditional spatially variant filtering algorithms achieve a superior tradeoff between sidelobe suppression and resolution preservation by means of adaptively calculating filtering parameters under some specific restrictions, such as filter design restriction and minimum amplitude constraint (MAC). These restriction aims to obtain an efficient analytical solution for filters, which is easy to calculate under unsupervised conditions. However, the restriction scope is so narrow that the suppression performance achieved by these filters is limited. Also, since the unsupervised optimization based on MAC indiscriminately minimizes amplitude, the main-lobe loss is unavoidable. To further improve the performance, a spatially variant convolution neural network (SVNN) is proposed, which consists of two core modules. One is the spatially variant filter generation (SVFG) module, adaptively generating superior spatially variant filters under more relaxed restrictions. The other is a paralleled shifted convolution (PSC) module, converting the signal format to achieve a fast and parallel spatially variant filtering process. Benefiting from more relaxed filter restrictions, the novel network successfully achieves better performance on sidelobe suppression. In addition, with supervised optimization based on another more accurate restriction, namely, minimum error constraint (MEC), the proposed algorithm also achieves superior main-lobe maintenance. All of them are validated by comparative experiments based on satellite data from GaoFen-3 and TerraSAR-X, and our proposed method achieves state-of-the-art performance. The entire project is available athttps://github.com/suoyuxi/SVNN.

Synthetic aperture radar (SAR) possesses significant advantages in aircraft detection due to its all-day and all-weather monitoring capability, but some unique problems in SAR images decrease the performance of aircraft detection. The speckle effect and excessive dynamic range are the most common problems that interfere with the visual features in SAR images and deteriorate detection performance. However, there lacks a detection-oriented image enhancement algorithm to collaboratively solve these two problems. An adaptive image enhancement algorithm is proposed to improve the performance of aircraft detection in SAR images. The proposed image enhancement algorithm provides a pseudocolor image through speckle suppression and channel combination, which consists of the speckle noise suppression channel, strong scattering feature enhancement channel, and weak scattering feature enhancement channel. The speckle noise suppression is achieved by a despeckle network, and the radiational feature enhancement channels are derived from an adaptive quantization method based on the characteristics of amplitude distribution. By optimizing the quality of the input image, the proposed image enhancement algorithm improves the performance of aircraft detection. Experiments based on datasets acquired by GaoFen-3 satellites indicate that the proposed algorithms significantly improve the detection performance of various types of detectors. The source project is available athttps://github.com/suoyuxi/ChannelEnhancement.

In recent years, remote sensing (RS) vision foundation models such as RingMo have emerged and achieved excellent performance in various downstream tasks. However, the high demand for computing resources limits the application of these models on edge devices. It is necessary to design a more lightweight foundation model to support on-orbit RS image interpretation. Existing methods face challenges in achieving lightweight solutions while retaining generalization in RS image interpretation. This is due to the complex high and low-frequency spectral components in RS images, which make traditional single CNN or Vision Transformer methods unsuitable for the task. Therefore, this paper proposes RingMo-lite, a RS lightweight network with a CNN-Transformer hybrid framework, which effectively exploits the frequency-domain properties of RS to optimize the interpretation process on several tasks like classification, object detection, semantic segmentation, and change detection. It is combined by the Transformer module as a low-pass filter to extract global features of RS images through a dual-branch structure, and the CNN module as a stacked high-pass filter to extract fine-grained details effectively. Furthermore, a novelty-designed frequency-domain masked image modeling (FD-MIM) is employed during the pretraining stage for self-supervised learning, which combines the high-frequency and low-frequency characteristics of each image patch. This approach effectively captures the latent feature representation in RS data. As shown in Fig. 1, compared with RingMo, the proposed RingMo-lite reduces the parameters over 60% in various RS image interpretation tasks, the average accuracy drops by less than 2% in most of the scenes and achieves SOTA performance compared to models of the similar size. In addition, our work will be integrated into the MindSpore computing platform in the near future.

Existing aircraft detection methods primarily rely on loss function constraints to guide the learning of end-to-end detectors, which significantly diverges from the judgment logic of human experts. Inspired by how human experts make decisions based on cognitive features, this article introduces the concept of physical knowledge features. Leveraging three properties of physical knowledge features, we identify the circle grayscale (CG) feature of aircraft and propose a physical knowledge-guided network (PKG-Net). By embedding CG features into the supervised learning process, the network improves its proficiency in learning stable features, thereby enhancing detection accuracy. Within this network, a multiscale circular frequency filter module (MS-CFFM) is responsible for extracting and integrating aircraft CG features across different scales. Adaptive channel selection module (ACSM) selectively activates channels for learning CG features. The hybrid attention feature fusion module (HAFFM) focuses intensively on the central localization of aircraft and deep reinforcement of channels. Experimental results on the RSOD and UCAS-AOD datasets demonstrate that the proposed method surpasses existing techniques in accuracy, achieving state-of-the-art performance.

Few-shot remote sensing image classification entails identifying images using a limited set of labeled data within remote sensing scenes, holding significant theoretical and practical implications. However, owing to the intricacy and variety of remote sensing images, traditional classification methods usually struggle to extract effective features and learn robust classifiers. To address this issue, an end-to-end metric learning framework named Attention-based Contrastive Learning Network is introduced in this paper. Specifically, the Attention-based Feature Optimization (ABFO) module is employed to align and enhance target image features, highlighting the target region and strengthening the network’s feature extraction capability. Additionally, the Dictionary-based Contrastive Loss (DBCL) module is assigned to optimize image feature vectors, improving category distinguishability and consequently enhancing classification accuracy. The experimental results on five publicly available Few-shot remote sensing classification datasets demonstrate the high competitiveness of our proposed method. Furthermore, it illustrates superior classification accuracy compared to other pertinent Few-shot learning algorithms in the 5-way 1-shot scenario.

As drone technology advances, using unmanned aerial vehicles for aerial surveys has become the dominant trend in modern low-altitude remote sensing. The surge in aerial video data necessitates accurate prediction for future scenarios and motion states of the interested target, particularly in applications like traffic management and disaster response. Existing video prediction methods focus solely on predicting future scenes (video frames), suffering from the neglect of explicitly modeling target’s motion states, which is crucial for aerial video interpretation. To address this issue, we introduce a novel task called Target-Aware Aerial Video Prediction, aiming to simultaneously predict future scenes and motion states of the target. Further, we design a model specifically for this task, named TAFormer, which provides a unified modeling approach for both video and target motion states. Specifically, we introduce Spatiotemporal Attention (STA), which decouples the learning of video dynamics into spatial static attention and temporal dynamic attention, effectively modeling the scene appearance and motion. Additionally, we design an Information Sharing Mechanism (ISM), which elegantly unifies the modeling of video and target motion by facilitating information interaction through two sets of messenger tokens. Moreover, to alleviate the difficulty of distinguishing targets in blurry predictions, we introduce Target-Sensitive Gaussian Loss (TSGL), enhancing the model’s sensitivity to both target’s position and content. Extensive experiments on UAV123VP and VisDroneVP (derived from single-object tracking datasets) demonstrate the exceptional performance of TAFormer in target-aware video prediction, showcasing its adaptability to the additional requirements of aerial video interpretation for target awareness.

Extrapolating future weather radar echoes from past observations is a complex task vital for precipitation nowcasting. The spatial morphology and temporal evolution of radar echoes exhibit a certain degree of correlation, yet they also possess independent characteristics. Existing methods learn unified spatial and temporal representations in a highly coupled feature space, emphasizing the correlation between spatial and temporal features but neglecting the explicit modeling of their independent characteristics, which may result in mutual interference between them. To effectively model the spatiotemporal dynamics of radar echoes, we propose a spatial-frequency-temporal correlation-decoupling transformer (SFTformer). The model leverages stacked multiple SFT-Blocks to not only mine the correlation of the spatiotemporal dynamics of echo cells but also avoid the mutual interference between the temporal modeling and the spatial morphology refinement by decoupling them. Furthermore, inspired by the practice that weather forecast experts effectively review historical echo evolution to make accurate predictions, SFTfomer incorporates a joint training paradigm for historical echo sequence reconstruction and future echo sequence prediction. Experimental results on the HKO-7 dataset and ChinaNorth-2021 dataset demonstrate the superior performance of SFTfomer in short-term (1 h), mid-term (2 h), and long-term (3 h) precipitation nowcasting.

Since polarimetric synthetic aperture radar (PolSAR) terrain segmentation is a dense prediction task, the disadvantage of inadequate labeled samples greatly limits its performance. In this article, we present a semi-supervised segmentation network called SemiPSCN to reduce the data reliance on label annotation, which integrates semi-supervised learning (SSL) paradigm and the characteristics of PolSAR data into a unified architecture. First, considering the unreliability of pseudolabels caused by noise interference in PolSAR data, a pseudolabel error localization (PEL) module is designed. By mapping the pixels that have mispredictions in pseudolabels, PEL can greatly enhance the confidence of pseudolabels. Then, SemiPSCN introduces a category representation constraint (CRC) module to explicitly boost the category consistency between labeled and unlabeled PolSAR data. Via explicit intracategory and intercategory constraints, CRC can guarantee the invariant representations on the same category region between labeled and unlabeled data. Furthermore, a region consistency constraint (RCC) module is designed to enhance the regional consistency in PolSAR data. RCC leverages the conception of graph to model the understanding of spatial relationships among terrain targets, thereby facilitating consistent spatial region expression in semi-supervised process. Finally, we build a challenging large-scale dataset called LSPolSAR-Seg and conduct abundant experiments on LSPolSAR-Seg. SemiPSCN exhibits superior performance when compared with other advanced approaches, especially improving mean intersection over union (mIoU) by 3.44%–12.77% under 20% split setting, which promotes the performance to a state-of-the-art level.

Recently, the transformer has achieved notable success in remote sensing (RS) change detection (CD). Its outstanding long-distance modeling ability can effectively recognize the change of interest (CoI). However, in order to obtain the precise pixel-level change regions, many methods directly integrate the stacked transformer blocks into the UNet-style structure, which causes the high computation costs. Besides, the existing methods generally consider bitemporal or differential features separately, which makes the utilization of ground semantic information still insufficient. In this paper, we propose the multiscale dual-space interactive perception network (MDIPNet) to fill these two gaps. On the one hand, we simplify the stacked multi-head transformer blocks into the single-layer single-head attention module and further introduce the lightweight parallel fusion module (LPFM) to perform the efficient information integration. On the other hand, based on the simplified attention mechanism, we propose the cross-space perception module (CSPM) to connect the bitemporal and differential feature spaces, which can help our model suppress the pseudo changes and mine the more abundant semantic consistency of CoI. Extensive experiment results on three challenging datasets and one urban expansion scene indicate that compared with the mainstream CD methods, our MDIPNet obtains the state-of-the-art (SOTA) performance while further controlling the computation costs.

Natural language moment localization aims to localize the target moment that matches a given natural language query in an untrimmed video. The key to this challenging task is to capture fine-grained video-language correlations to establish the alignment between the query and target moment. Most existing works establish a single-pass interaction schema to capture correlations between queries and moments. Considering the complex feature space of lengthy video and diverse information between frames, the weight distribution of information interaction flow is prone to dispersion or misalignment, which leads to redundant information flow affecting the final prediction. We address this issue by proposing a capsule-based approach to model the query-video interactions, termed the Multimodal, Multichannel, and Dual-step Capsule Network ( [Formula: see text]DCapsN), which is derived from the intuition that "multiple people viewing multiple times is better than one person viewing one time." First, we introduce a multimodal capsule network, replacing the single-pass interaction schema of "one person viewing one time" with the iterative interaction schema of "one person viewing multiple times," which cyclically updates cross-modal interactions and modifies potential redundant interactions via its routing-by-agreement. Then, considering that the conventional routing mechanism only learns a single iterative interaction schema, we further propose a multichannel dynamic routing mechanism to learn multiple iterative interaction schemas, where each channel performs independent routing iteration to collectively capture cross-modal correlations from multiple subspaces, that is, "multiple people viewing." Moreover, we design a dual-step capsule network structure based on the multimodal, multichannel capsule network, bringing together the query and query-guided key moments to jointly enhance the original video, so as to select the target moments according to the enhanced part. Experimental results on three public datasets demonstrate the superiority of our approach in comparison with state-of-the-art methods, and comprehensive ablation and visualization analysis validate the effectiveness of each component of the proposed model.

Semantic comprehension aims to reasonably reproduce people's real intentions or thoughts, e.g., sentiment, humor, sarcasm, motivation, and offensiveness, from multiple modalities. It can be instantiated as a multimodal-oriented multitask classification issue and applied to scenarios, such as online public opinion supervision and political stance analysis. Previous methods generally employ multimodal learning alone to deal with varied modalities or solely exploit multitask learning to solve various tasks, a few to unify both into an integrated framework. Moreover, multimodal-multitask cooperative learning could inevitably encounter the challenges of modeling high-order relationships, i.e., intramodal, intermodal, and intertask relationships. Related research of brain sciences proves that the human brain possesses multimodal perception and multitask cognition for semantic comprehension via decomposing, associating, and synthesizing processes. Thus, establishing a brain-inspired semantic comprehension framework to bridge the gap between multimodal and multitask learning becomes the primary motivation of this work. Motivated by the superiority of the hypergraph in modeling high-order relations, in this article, we propose a hypergraph-induced multimodal-multitask (HIMM) network for semantic comprehension. HIMM incorporates monomodal, multimodal, and multitask hypergraph networks to, respectively, mimic the decomposing, associating, and synthesizing processes to tackle the intramodal, intermodal, and intertask relationships accordingly. Furthermore, temporal and spatial hypergraph constructions are designed to model the relationships in the modality with sequential and spatial structures, respectively. Also, we elaborate a hypergraph alternative updating algorithm to ensure that vertices aggregate to update hyperedges and hyperedges converge to update their connected vertices. Experiments on the dataset with two modalities and five tasks verify the effectiveness of HIMM on semantic comprehension.

Few-shot object detection, expecting detectors to detect novel classes with a few instances, has made conspicuous progress. However, the prototypes extracted by existing meta-learning based methods still suffer from insufficient representative information and lack awareness of query images, which cannot be adaptively tailored to different query images. Firstly, only the support images are involved for extracting prototypes, resulting in scarce perceptual information of query images. Secondly, all pixels of all support images are treated equally when aggregating features into prototype vectors, thus the salient objects are overwhelmed by the cluttered background. In this paper, we propose an Information-Coupled Prototype Elaboration (ICPE) method to generate specific and representative prototypes for each query image. Concretely, a conditional information coupling module is introduced to couple information from the query branch to the support branch, strengthening the query-perceptual information in support features. Besides, we design a prototype dynamic aggregation module that dynamically adjusts intra-image and inter-image aggregation weights to highlight the salient information useful for detecting query images. Experimental results on both Pascal VOC and MS COCO demonstrate that our method achieves state-of-the-art performance in almost all settings. Code will be available at: https://github.com/lxn96/ICPE.

Multimodal hate detection, which aims to identify the harmful content online such as memes, is crucial for building a wholesome internet environment. Previous work has made enlightening exploration in detecting explicit hate remarks. However, most of their approaches neglect the analysis of implicit harm, which is particularly challenging as explicit text markers and demographic visual cues are often twisted or missing. The leveraged cross-modal attention mechanisms also suffer from the distributional modality gap and lack logical interpretability. To address these semantic gap issues, we propose TOT: a topology-aware optimal transport framework to decipher the implicit harm in memes scenario, which formulates the cross-modal aligning problem as solutions for optimal transportation plans. Specifically, we leverage an optimal transport kernel method to capture complementary information from multiple modalities. The kernel embedding provides a non-linear transformation ability to reproduce a kernel Hilbert space (RKHS), which reflects significance for eliminating the distributional modality gap. Moreover, we perceive the topology information based on aligned representations to conduct bipartite graph path reasoning. The newly achieved state-of-the-art performance on two publicly available benchmark datasets, together with further visual analysis, demonstrate the superiority of TOT in capturing implicit cross-modal alignment.

Kaiwen Wei, Yiran Yang, Li Jin, Xian Sun, Zequn Zhang, Jingyuan Zhang, Xiao Li, Linhao Zhang, Jintao Liu, Guo Zhi. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Fine-tuning large-scale pretrained vision models to downstream tasks is a standard technique for achieving state-of-the-art performance on computer vision benchmarks. However, fine-tuning the whole model with millions of parameters is inefficient as it requires storing a same-sized new model copy for each task. In this work, we propose LoRand, a method for fine-tuning large-scale vision models with a better tradeoff between task performance and the number of trainable parameters. LoRand generates tiny adapter structures with low-rank synthesis while keeping the original backbone parameters fixed, resulting in high parameter sharing. To demonstrate LoRand's effectiveness, we implement extensive experiments on object detection, semantic segmentation, and instance segmentation tasks. By only training a small percentage (1% to 3%) of the pretrained backbone parameters, LoRand achieves comparable performance to standard fine-tuning on COCO and ADE20K and outperforms fine-tuning in low-resource PASCAL VOC dataset.

Event Causality Extraction (ECE) aims to extract the cause-effect event pairs from the given text, which requires the model to possess a strong reasoning ability to capture event causalities.However, existing works have not adequately exploited the interactions between the cause and effect event that could provide crucial clues for causality reasoning.To this end, we propose an Implicit Cause-Effect interaction (ICE) framework, which formulates ECE as a template-based conditional generation problem.The proposed method captures the implicit intra-and inter-event interactions by incorporating the privileged information (ground truth event types and arguments) for reasoning, and a knowledge distillation mechanism is introduced to alleviate the unavailability of privileged information in the test stage.Furthermore, to facilitate knowledge transfer from teacher to student, we design an event-level alignment strategy named Cause-Effect Optimal Transport (CEOT) to strengthen the semantic interactions of cause-effect event types and arguments.Experimental results indicate that ICE achieves state-of-the-art performance on the ECE-CCKS dataset.

Knowledge distillation (KD) is a promising approach that facilitates the compact student model to learn dark knowledge from the huge teacher model for better results. Although KD methods are well explored in the 2D detection task, existing approaches are not suitable for 3D monocular detection without considering spatial cues. Motivated by the potential of depth information, we propose a novel distillation framework that validly improves the performance of the student model without extra depth labels. Specifically, we first put forward a perspective-induced feature imitation, which utilizes the perspective principle (the farther the smaller) to facilitate the student to imitate more features of farther objects from the teacher model. Moreover, we construct a depth-guided matrix by the predicted depth gap of teacher and student to facilitate the model to learn more knowledge of farther objects in prediction level distillation. The proposed method is available for advanced monocular detectors with various backbones, which also brings no extra inference time. Extensive experiments on the KITTI and nuScenes benchmarks with diverse settings demonstrate that the proposed method outperforms the state-of-the-art KD methods.

The construction of a basic model to extract generalized features from a large number of multimodal data is a new challenge in the field of remote sensing. Compared with natural scene images, When faced with a complex application scenario of remote sensing of multi-sensor acquisition, models that are suitable for a specific task are difficult to generalize to new scenarios. In this paper, we propose a model architecture based on the concepts of multi-domain representation and cross-domain fusion. By extracting strong generalization features from massive multi-modal data, a single foundation model can accomplish generalization interpretation for multiple downstream tasks. Experimental results show that the proposed model performs well on multiple downstream tasks, which validates the feasibility of the remote sensing cross-modal foundation model in the interpretation task.

Semi-supervised remote sensing semantic segmentation is an efficient way to increase the use of unlabeled data and cut labelling costs. The unlabeled-to-labeled data ratio is employed in more recent methods, which is very different from what is really used in practise. In this paper, we propose a semi-supervised semantic generative network for remote sensing images, introducing a self-supervised learning method to enhance the feature representation of the model when the data ratio is high. Specifically, we design a new branch for unlabeled data, which includes modules for both semantic reconstruction and appearance reconstruction. It can effectively alleviate the category confusion in complicated remote sensing image when there are few labeled data. Comprehensive experiments on the ISPRS POTSDAM dataset demonstrate that the proposed method achieves promising results.

Building extraction and height estimation are two important basic tasks in remote sensing image interpretation, which are widely used in urban planning, real-world 3D construction, and other fields. Most of the existing research regards the two tasks as independent studies. Therefore the height information cannot be fully used to improve the accuracy of building extraction and vice versa. In this work, we combine the individuaL buIlding extraction and heiGHt estimation through a unified multiTask learning network (LIGHT) for the first time, which simultaneously outputs a height map, bounding boxes, and a segmentation mask map of buildings. Specifically, LIGHT consists of an instance segmentation branch and a height estimation branch. In particular, so as to effectively unify multi-scale feature branches and alleviate feature spans between branches, we propose a Gated Cross Task Interaction (GCTI) module that can efficiently perform feature interaction between branches. Experiments on the DFC2023 dataset show that our LIGHT can achieve superior performance, and our GCTI module with ResNet 101 as the backbone can significantly improve the performance of multitask learning by 2.8% AP50 and 6.5% δ1, respectively.

The front deployed deep learning is a promising technology of the next generation industrial applications, which can extract essential information from high dimension sensors. However, part of these heavy computation tasks at resource constrained front devices have to be offloaded to the edge or cloud devices, which forms the cooperative deep learning system through the exchange of intermediate data. The inference efficiency of cooperative deep learning system will then be highly correlated with the communication latency caused by the non-stationary industrial multipath-rich fading channel. This paper proposes a novel method to control the distribution of communications latency, which is able to support efficient cooperative deep learning architecture in the harsh industrial environment. The proposed method is essentially an inverse process of Gaussian Mixture Model (GMM), which adjusts latency samples to approach the given arbitrary shape function. To achieve this objective, a new variation of Expectation-Maximization (EM) algorithm in analytical domain is derived to decompose arbitrary distribution shape with multiple Gaussian kernels and an optimized stochastic resource allocation algorithm is proposed to approximate each Gaussian kernels. The performance of proposed method is verified by both classical Rician channel model and field measured industrial fading channel responses.

Recently, convolutional neural networks (CNNs) have been developed for remote sensing image fusion (RSIF). To obtain competitive fusion performance, network design becomes more complicated by stacking convolutional layers deeper and wider. However, problems still remain when applying existing networks in practical applications. On the one hand, researchers focus on improving spatial resolution but ignore that the fused images will be used in subsequent interpretation applications, e.g., objection detection. On the other hand, RSIF involves different tasks with different image sources e.g., pansharpening of the panchromatic and multispectral image, hypersharpening of the panchromatic and hyperspectral image, etc. However, existing networks only solve one of them, failing to be compatible with other tasks. To address the above problems, a convenient task-inspired multiscale nonlocal-attention network (MNAN) is proposed for RSIF. The proposed MNAN focuses more on enhancing the multi-scale targets in the scene when improving the resolution of the fused image. In addition, the proposed network can be applied to both pansharpening and hypersharpening tasks without any modification.

Welcome to the concluding issue of IEEE Transactions on Computational Social Systems (TCSS) for the year 2023. We would like to seize this opportunity to extend our heartfelt appreciation and congratulations to all for your exceptional dedication and unwavering support. We eagerly anticipate further collaboration to enhance the publication quality and expedite the review process of TCSS in the upcoming year 2024.

Few-shot segmentation (FSS) is proposed to segment unknown class targets with just a few annotated samples. Most current FSS methods follow the paradigm of mining the semantics from the support images to guide the query image segmentation. However, such a pattern of ‘learning from others’ struggles to handle the extreme intra-class variation, preventing FSS from being directly generalized to remote sensing scenes. To bridge the gap of intra-class variance, we develop a Dual-Mining network named DMNet for cross-image mining and self-mining, meaning that it no longer focuses solely on support images but pays more attention to the query image itself. Specifically, we propose a Class-public Region Mining (CPRM) module to effectively suppress irrelevant feature pollution by capturing the common semantics between the support-query image pair. The Class-specific Region Mining (CSRM) module is then proposed to continuously mine the class-specific semantics of the query image itself in a ‘filtering’ and ‘purifying’ manner. In addition, to prevent the co-existence of multiple classes in remote sensing scenes from exacerbating the collapse of FSS generalization, we also propose a new Known-class Meta Suppressor (KMS) module to suppress the activation of known-class objects in the sample. Extensive experiments on the iSAID and LoveDA remote sensing datasets have demonstrated that our method sets the state-of-the-art with a minimum number of model parameters. Significantly, our model with the backbone of Resnet-50 achieves the mIoU of 49.58% and 51.34% on iSAID under 1-shot and 5-shot settings, outperforming the state-of-the-art method by 1.8% and 1.12%, respectively. The code is publicly available at https://github.com/HanboBizl/DMNet/.

Intelligent ship detection algorithms for synthetic aperture radar (SAR) images have achieved significant results in Earth observation applications. By learning features such as scale, shape and texture from samples, they can quickly locate and recognize ships in complex backgrounds. However, due to the lack of use of polarization features, the upper bound of detection performance is still limited, especially under poor image quality conditions such as ambiguous interference. To solve this, the dual-polarization image feature fusion network (DPFF-Net) is proposed. The key of it lies in adaptive mining, enhancement and fusion of polarization features through the designed siamese structure, polarization-aware enhancement block (PAEB) and dynamic gated fusion block (DGFB). With fully utilizing complementary information hidden between co-polarization and cross-polarization data, more comprehensive and accurate features are obtained and used as the detect head input. Thus, the proposed algorithm achieves state-of-the-art performance, and its effectiveness are validated by experiments on dual-polarization SAR datasets.

Satisfactory extraction of buildings from aerial images has long been a challenging task. In the recently fully convolutional network (FCN)-based methods, the locality of the convolution operation is detrimental to handling global long-range dependencies for complex buildings, such as those shaded by trees, obscured by the shadow of high-rise ones, and blurred by the high-similarity pixels. Although graph neural networks (GNN) show clear-cut advantages in modeling semantic correlations between different segments or instances, existing FCN-GNN methods cannot directly perform graph reasoning for spatial features in an end-to-end framework due to two chief limitations: 1) it is costly to construct large fully-connected graphs; 2) there is not a standard pixel-based graph reasoning paradigm. We therefore developed a hybrid end-to-end FCN-like network to perform better building extraction on complex scenes, termed the Dual Spatial-Graph Refinement Network (DSRNet). Specifically, we first proposed a spatial-graph reasoning module (SR), which effectively constructs adjacent relations for dense pixels, to make fast graph reasoning for grid-based features possible. Considering SR as the basis, we further developed a dual spatial-graph refinement module (DSR), consisting of body and structure SRs (BSR and SSR), to make SR attentively perceive global buildings’ spatial-semantic relationships from two complementary perspectives. BSR was designed to enhance consistency within buildings and SSR models correlations of buildings’ structural information, to extract buildings with more coherent bodies and tight-fitting edge contours. Finally, we devised a Contour Alignment loss function (CA) to encourage the segmentation result to align correctly with the contours of the ground truths. DSRNet outperforms state-of-the-art FCN-based methods on Christchurch and Tokyo high-resolution building datasets and consistently shows improvements in building segmentation and contour extraction, especially in the case of complex scenes.

Datasets play a key role in developing superior building detection approaches. However, most of the previous work focuses on accurate building masks and scale expansion, while the categories are always missing, which hinders the further analysis of urban development and cultures. Therefore, we propose a benchmark for building detection and fine-grained classification from very high-resolution (VHR) satellite imagery. An extensive annotation is performed for about 0.5 million building instances with 12 fine-grained roof types and individual polygons. The annotation of building functions of two cities in the previous version (UBCv1) [1] is also integrated. To ensure the building variety, it consists of VHR optical images of 20 unique cities worldwide with various landforms and styles of architecture. Its variety and fine-grained categories pose great challenges and meanwhile provide a foundation for the building extraction and fine-grained classification on a global scale. Besides, 17 cities are provided with finely aligned Synthetic Aperture Radar (SAR) images, which can be employed for the development and evaluation of approaches optionally based on optical, SAR, or multi-modal images. Significantly, the proposed benchmark is used as the base of the 2023 IEEE GRSS Data Fusion Contest [2]. The dataset and codes of the baseline methods are available at: https://github.com/AICyberTeam/UBC-dataset/tree/UBCv2.

Aircraft classification in synthetic aperture radar (SAR) images plays a considerable role in global region management and surveillance. Recently, deep learning has been applied to solve the classification problem and made significant progress. Due to the imaging variability at different angles and component scattering discreteness in SAR images, previous works have had difficulty in achieving desirable classification results. To address these issues, we study the positional and semantic relationship between the scattering points and propose an innovative scattering topology network (ST-Net) in this article. First, considering the diversity of imaging results caused by different target attitude angles, we extract and transform the scattering cluster centers to update the information of various categories. It can guide the model to strengthen the discriminative features and mitigate the impact of imaging variability on classification performance. Second, a novel scattering topology module (STM) is introduced to model the spatial relationships and semantic information interaction of discrete scattering points. In this process, the topology relations and scattering characteristics are enhanced for further accurate classification. Third, context attention excitation (CAE) is designed to capture significant global and semantic information, which is conducive to suppressing background interference and reducing category confusion. In conclusion, the ST-Net is presented with the SAR imaging mechanism and the topology geometric representation of aircraft. We construct the SAR aircraft category dataset (SAR-ACD) and conduct extensive experiments on it to show the effectiveness of ST-Net, which illustrates that our method achieves superior classification performance.

Aircraft detection is a challenging task for remote sensing images. The anchor-based methods are of high complexity, and the keypoint-based detectors suffer the grouping difficulty. Some line-based models relying on local features are hindered by adhesion and disintegrity problems. Moreover, those detection representations rarely take into account the sword-shaped component geometric semantics (e.g., fuselage and the wing) of the aircraft itself, leading to being less robust and unfavorable for downstream tasks, such as ones needing the detailed size and shape of aircraft. Accordingly, we model the sword-shaped component geometry and propose S2CGNet, a more robust appearance-based aircraft detector. The sword attenuation mask (SAM) module is devised to encode a “sword-shaped mask” for each aircraft while exploring more robustness via the geometric surface embedding. The SAM can provide clearer borders to separate different aircraft more precisely. Besides, to address the instance disintegrity problem and further boost the quality of SAM, we propose an instance aware graph (IAG) module to jointly optimize the parameters of the fuselage/wing detection heads. Experimental results show that the performance of S2CGNet can reach the state-of-the-art (SOTA) level. Specifically, it achieves 98.5% in terms of AP50 on the combined dataset of Aircraft-KP and NWPU VHR-10, boosting 3.8% than the baseline. Besides, S2CGNet boosts the quality of detection results greatly, e.g., it yields a significant improvement of 21.3% on AP75 compared to the baseline. Furthermore, the generalization comparisons on the FAIR1M dataset strongly demonstrate the robustness of our model surpasses other oriented detectors by a large margin.

Few-shot object detection (FSOD) has received numerous attention due to the difficulty and time-consuming of labeling objects. Recent researches achieve excellent performance in a natural scene by only using a few instances of novel classes to fine-tune the last prediction layer of the model well-trained on plentiful base data. However, compared with natural scene objects with a single direction and small size variety, the direction and size of the objects in remote sensing images (RSIs) vary greatly. The methods proposed for the natural scene cannot be directly applied to RSIs. In this article, we first propose a strong baseline for RSIs. It fine-tunes all detector components acting on high-level features and effectively improves the performance of novel classes. Further analyzing the results of the baseline, we find that the error for novel classes is mainly concentrated in classification. It misclassifies novel classes as confusable base classes or backgrounds due to the difficulty in extracting generalized information from limited instances. As is well-known, text-modal knowledge can highly summarize the generalized and unique characteristics of categories. Thus, we introduce text-modal descriptions for each category and propose an FSOD method guided by TExt-MOdal knowledge, called TEMO. Specifically, a text-modal knowledge extractor and a cross-modal assembly module are proposed to extract text features and fuse the text-modal features into visual-modal features. The fused features greatly reduce the classification confusion of novel classes. Furthermore, we introduce a mask strategy and a separation loss to avoid over-fitting and ambiguity of text-modal features. Experimental results on detection in optical remote sensing images (DIOR), Northwestern Polytechnical University (NWPU), and fine-grained object recognition in high-resolution remote sensing imagery (FAIR1M) illustrate that our TEMO achieves state-of-the-art performance in all settings.

Building 3D reconstruction from remote sensing images has a wide range of applications in smart cities, photogrammetry and other fields. Methods for automatic 3D urban building modeling typically employ multi-view images as input to algorithms to recover point clouds and 3D models of buildings. However, such models rely heavily on multi-view images of buildings, which are time-intensive and limit the applicability and practicality of the models. To solve these issues, we focus on designing an efficient DSM estimation-driven reconstruction framework (Building3D), which aims to reconstruct 3D building models from the input single-view remote sensing image. Existing DSM estimation networks suffer from the imbalance between local features and global features, which leads to over-smooth DSM estimates at instance boundaries. To address this issue, we propose a Semantic Flow Field-guided DSM Estimation (SFFDE) network, which utilizes the proposed concept of elevation semantic flow to achieve the registration of local and global features. First, in order to make the network semantics globally aware, we propose an Elevation Semantic Globalization (ESG) module to realize the semantic globalization of instances. Further, in order to alleviate the semantic span of global features and original local features, we propose a Local-to-Global Elevation Semantic Registration (L2G-ESR) module based on elevation semantic flow. Our Building3D is rooted in the SFFDE network for building elevation prediction, synchronized with a building extraction network for building masks, and then sequentially performs point cloud reconstruction and surface reconstruction (or CityGML model reconstruction). On this basis, our Building3D can optionally generate CityGML models or surface mesh models of the buildings. Extensive experiments on ISPRS Vaihingen and DFC2019 datasets on the DSM estimation task show that our SFFDE significantly improves upon state-of-the-art and δ1, δ2and δ3metrics of our SFFDE are improved to 0.595, 0.897 and 0.970. Furthermore, our Building3D achieves impressive results in the 3D point cloud and 3D model reconstruction process.

Continual learning is an effective way to overcome catastrophic forgetting (CF) in incremental learning for semantic segmentation. The existing continual semantic segmentation (CSS) methods of remote sensing (RS) ignore the semantic relationships among pixels across different images, which will lead to disappointing segmentation results, such as edge pixel misclassification and small object omission. In this paper, we propose a framework for modeling cross-image semantic relationship dependencies (MiCro), which aims to learn an inter-class separable and intra-class cohesive feature space from the pixel relationships across various images to ensure that learned categories can prevent CF in the incremental process. Specifically, we exploit the relationships among pixels of images in mini-batch to construct three losses: (a) Cross-image feature relationship distillation (CFRD) loss, which builds a well-structured feature space; (b) Cross-image intra-class feature cohesion (CIFC) loss, which is devised to make intra-class features more cohesive; and (c) Cross-image class-area weighted cross-entropy (CCWCE) loss, which is mainly employed to inversely weight the proportion of category area in mini-batch. The effectiveness of the proposed approach is demonstrated by extensive experiments on three RS semantic segmentation datasets from ISPRS Vaihingen, ISPRS Potsdam, and iSAID. MiCro is superior to the current most advanced methods in most incremental settings, especially improving mIoU by 11.59% on ISPRS Vaihingen, 13.17% on ISPRS Potsdam, and 15.01% on iSAID in the most difficult incremental settings, which promotes the CSS to a state-of-the-art (SOTA) level. The code will be available at https://github.com/RongXueE/MiCro.

Deep learning approaches have contributed to the rapid development of remote sensing (RS) image interpretation. The most widely used training paradigm is to use ImageNet pretrained models to process RS data for specified tasks. However, there are issues such as domain gap between natural and RS scenes and the poor generalization capacity of RS models. It makes sense to develop a foundation model with general RS feature representation. Since a large amount of unlabeled data is available, the self-supervised method has more development significance than the fully supervised method in RS. However, most of the current self-supervised methods use contrastive learning, whose performance is sensitive to data augmentation, additional information, and selection of positive and negative pairs. In this article, we leverage the benefits of generative self-supervised learning (SSL) for RS images and propose an RS foundationmodel framework called RingMo, which consists of two parts. First, a large-scale dataset is constructed by collecting two million RS images from satellite and aerial platforms, covering multiple scenes and objects around the world. Second, we propose an RS foundation model training method designed for dense and small objects in complicated RS scenes. We show that the foundation model trained on our dataset with RingMo method achieves state-of-the-art (SOTA) on eight datasets across four downstream tasks, demonstrating the effectiveness of the proposed framework. Through in-depth exploration, we believe it is time for RS researchers to embrace generative SSL and leverage its general representation capabilities to speed up the development of RS applications.

Aliasing artifacts introduced by azimuth ambiguity seriously impact the interpretation of synthetic aperture radar images. To achieve parameter-free and fast azimuth ambiguity suppression, a novel deep learning model is designed to estimate the ambiguous signal intensity to total signal intensity ratio in the range-Doppler domain. This model does not depend on processing parameters and can be applied in any acquisition mode. The mean shift algorithm is applied to select less ambiguous subspectra according to the estimation result. The selected subspectra are restored to a full spectrum with an energy concentrated extrapolation method to preserve the resolution. The enhanced spectral selection and extrapolation algorithm overcomes the dependence on processing parameters, and experiments based on TerraSAR-X and Radarsat-2 images indicate that the proposed algorithm suppresses the azimuth ambiguity significantly.

As the number of aero and space remote sensing platforms increases, distributed observation and real-time terminal processing become mainstream in the future. However, most of the training methods for the multi-platform are still limited to centralized structures or independent training based on a single platform, which is inefficient or limited in accuracy. In order to solve this problem, we innovatively propose a distributed collaborative method (DCM) for remote sensing image classification training in this article. First, the proposed training method, which is based on one cloud and several terminals, can aggregate different parameters of the terminal network to the cloud to improve global accuracy. Second, a sample proximity network is designed to process the problem of data heterogeneity on different terminal networks, which further improves the accuracy during the model fusion on the cloud. Third, a multi-layer grouped concatenation module is applied after the model fusion to extract hierarchical features with different categories of remote sensing images. Experimental results on the challenging remote sensing image classification dataset FAIR1M show that the proposed training method has better collaborative learning ability than the centralized-based model or terminal-trained lightweight network under the heterogeneous data.

The rapid development of satellite platforms has yielded copious and diverse multi-source data for earth observation, greatly facilitating the growth of multimodal semantic segmentation (MSS) in remote sensing. However, MSS also suffers from numerous challenges: 1) Existing inherent defects in each modality due to the different imaging mechanisms. 2) Insufficient exploration of the intrinsic characteristics of modalities. 3) The existence of the huge semantic gap between heterogeneous data causes difficulties in feature fusion. The inability to effectively utilize the rich and diverse information provided by each modality and ignorance of the heterogeneity between modalities will hinder the feature enhancement, and further significantly impacts the semantic segmentation accuracy. Furthermore, neglecting the huge gap makes feature fusion challenging. In this study, we introduce a novel framework for multimodal semantic segmentation that effectively mitigates the aforementioned problems. Our approach employs a pseudo-siamese structure for feature extraction. Specifically, we propose a simple yet effective geometric topology structure modeling (GTSM) module to extract geometric relationships and texture information from optical data. Additionally, we present a modality intrinsic noise suppression (MINS) module to fully exploit radiation information and alleviate the effects of unique geometric distortions for SAR. Furthermore, we present an adaptive multimodal feature fusion (AMFF) module for fully fusing different modality features. Extensive experiments on both WHU-OPT-SAR and DFC23 datasets validate the robustness and effectiveness of the proposed Modality Characteristics-Guided Semantic Segmentation (MoCG) network compared to other state-of-the-art semantic segmentation methods, including multimodal and single-modal approaches. Our approach achieves the best performance on both datasets, resulting in mIoU/OA gains 69.1%/87.5% on WHU-OPT-SAR and 86.7%/97.3% on DFC23.

The proposal of Segment Anything Model (SAM) has created a new paradigm for deep learning-based semantic segmentation field, and has shown amazing generalization performance. However, we find it may fail or perform poorly on multimodal remote sensing scenarios, especially the Synthetic Aperture Radar (SAR) images. Besides, SAM does not provide category information of objects. In this paper, we propose a foundation model for multimodal remote sensing image segmentation called RingMo-SAM, which can not only segment anything in optical and SAR remote sensing data, but also identify object categories. First, a large-scale dataset containing millions of segmentation instances is constructed by collecting multiple open-source datasets in this field to train the model. Then, by constructing an instance-type and terrain-type category-decoupling mask decoder, the category-wise segmentation of various objects is achieved. In addition, a prompt encoder embedded with the characteristics of multimodal remote sensing data is designed. It not only supports multi-box prompts to improve the segmentation accuracy of multi-objects in complicated remote sensing scenes, but also supports SAR characteristics prompts to improve the segmentation performance on SAR images. Extensive experimental results on several datasets including iSAID, ISPRS Vaihingen, ISPRS Potsdam, AIR-PolSAR-Seg, etc. have demonstrated the effectiveness of our method.

Due to variable imaging conditions, the resolution degradation often occurs in aerial images, which in turn impairs the performance upper bound of semantic segmentation. To solve this problem, super-resolution is placed before semantic segmentation as a pre-processing sub-task in most existing methods. The above two sub-tasks often form a unidirectional open-loop structure for relatively independent optimization, which constrains the ultimate segmentation performance improvement. To break down information barriers among them and form a more compact overall optimization, we propose an effective learning method named as Pixel-Wise Mutual-Feed Learning (PW-MFL) for segmenting images with resolution degradation. The key is to build auxiliary bidirectional connections, which contribute to the mutual pixel-wise spatial and feature information guidance during training. The feed-forward connection is realized by the Self-Attention Context Correlation (SACC) module, which enhances the intra-class semantic features of pixel positions with poor super-resolution performance by the fusion of that with superior performance. The feed-back connection is achieved by the Semantic Weighted Mapping (SWM) module, which aims to activate and adjust the detailed features of super-resolution in incorrectly segmented pixel positions via the semantic feature information. In addition, the Pixel-Aware Optimization (PAO) strategy is proposed to give more attention to optimizing specific pixel positions based on spatial information. Extensive experiments are conducted on three representative remote sensing segmentation benchmarks, ISPRS Vaihingen, ISPRS Potsdam, and iSAID datasets. The state-of-the-art segmentation level in resolution-degraded aerial images is achieved through the proposed learning method.

Remarkable progress based on deep neural networks has been achieved on the semantic segmentation in remote sensing images. However, pixel-level labeling is expensive for remote sensing images. Semi-supervised semantic segmentation becomes an alternative approach to reduce the cost of annotation, and it is crucial to utilize efficiently a large number of unlabeled data. Nevertheless inevitably, there is the unbalanced class distribution between labeled and unlabeled data of remote sensing scene. Existing semi-supervised methods train unlabeled images in isolation from labeled images and only learn reliable pixel pseudo-labels, leading to underutilization of unlabeled images. This article proposes a novel semi-supervised semantic segmentation approach based on label propagation and contrastive regularization for remote sensing images. Specifically, the unlabeled images are augmented by randomly copy-pasting the class regions from labeled images. A prototype feature constraint module is used to enforce the constraint on the pixel features of unlabeled images relying on the prototype features from labeled images, achieving feature alignment on the entire dataset. Furthermore, we present the region contrastive learning module that guides the model to learn feature consistency under different perturbations and compact feature representations over class regions on unlabeled images. Extensive experimental results on multiple remote sensing datasets demonstrate that our proposed approach achieves superior performance compared with state-of-the-art semi-supervised semantic segmentation methods.

Remote sensing spatiotemporal prediction aims to infer future trends from historical spatiotemporal data, e.g., videos and time series images, has a broad application prospect in many fields. The foundation model is a promising research direction for spatiotemporal information mining because of its robust feature extraction capability, and has made rapid progress in natural scenes. Nevertheless, due to the spatially multi-scale and temporally multi-scale properties in remote sensing data, these methods still encounter bottlenecks when applied to remote sensing. Therefore, we propose a foundation model for remote sensing spatiotemporal prediction via spatiotemporal evolution decoupling, abbreviated as RingMo-Sense. Considering spatial affinity, temporal continuity, and spatiotemporal interaction, we construct spatial, temporal, and spatiotemporal triple-branch prediction networks. Specifically, we use parameter-sharing and progressive joint training strategies to achieve stable long-range prediction and parameter reduction simultaneously. In addition, we build a remote sensing spatiotemporal dataset by collecting various remote sensing videos and time series images. The experimental results on six downstream spatiotemporal tasks demonstrate that the proposed model yields competitive performance.

Remote sensing cross-modal text-image retrieval (RSCTIR) is a flexible and human-centered approach to retrieving rich information from different modalities, which has attracted plenty of attention in recent years. It remains challenging because the current methods usually ignore the varying difficulty levels of different sample pairs, stemming from the large image distribution difference and the high text similarity in the remote sensing (RS) field. Therefore, in this paper, we propose an innovative hypersphere-based visual semantic alignment (HVSA) network via curriculum learning. Specifically, we first design an adaptive alignment strategy based on curriculum learning, that aligns RS image-text pairs from easy to hard. Sample pairs with different levels of difficulty are treated unequally, and we obtain a better embedding representation when projecting the features onto the unit hypersphere. Then, to measure the robustness of cross-modal feature alignment on the unit hypersphere, we introduce the feature uniformity strategy. It reduces the occurrence of mismatching cases and improves generalization performance. Finally, we design the key-entity attention (KEA) mechanism to alleviate the problem of information imbalance among different modalities. KEA has the ability to extract information about the key entity which is aligned with textual information. Despite its conciseness, our framework achieves state-of-the-art performance on classical datasets of RSCTIR tasks while enjoying faster inference. The summed recall of HVSA on the RISCD and RSITMD is 120.97 and 198.94, 2.50 and 10.49 points ahead of the current best methods, respectively. Extensive experiments demonstrate the competitiveness of our method. The code has been released at https://github.com/ZhangWeihang99/HVSA.

With the development of information technology, multiplatform collaborative collection and processing of remote sensing (RS) images has become a significant trend. However, the existing models are challenging to achieve accurate and efficient image interpretation on RS multiplatform systems. To solve this problem, we propose a novel distributed convolutional neural network (DCNNet) and demonstrate the superiority of our method in RS image classification. First, a progressive inference mechanism is introduced to support most images to be classified in advance with satisfactory accuracy, which minimizes redundant cloud transmission and achieves higher inference acceleration. Meanwhile, a distributed self-distillation paradigm is designed to integrate and refine in-depth features, performing efficient knowledge transfer between the terminals and the cloud network. Second, a multiscale feature fusion (MSFF) module is presented to extract valid receptive fields and assign weights to crucial channel dimension features. Finally, a sampling augmentation (SA) attention is proposed to enhance the effective feature representation of RS images through a bottom-up and top-down feedforward structure. We conducted extensive experiments and visual analyses on three benchmark scene classification datasets and one fine-grained dataset. Compared with the existing methods, DCNNet consolidates several advantages in terms of accuracy, computation, transmission, and processing efficiency into a single framework for multiplatform RS image classification.

With urgent application requirements, such as satellite in-orbit processing and unmanned aerial vehicle tracking, knowledge distillation (KD) following the teacher–student teaching mechanism has shown great potential to obtain lightweight detectors. However, compact students have limited accuracy due to the interference of large-scale variations and blurred boundaries in remote sensing objects. Specifically, previous methods mostly force teacher–student responses from the layer of the same depth and scale to align. Stereotyped manual interlayer associations may cause discriminative features of multiscale objects to be incorrectly bundled. Furthermore, the regression branch follows the identical distillation paradigm as the classification branch, resulting in ambiguous object bounding box deviations. To solve the above two issues, we propose an effective KD framework called layer-calibration and task-disentangle distillation (LTD). First, the cross-layer calibration distillation (CCD) structure is innovatively proposed. It adaptively binds a student layer with several related target layers, rather than a fixed layer in the teacher model. Appropriate and clear knowledge of large and small objects is transmitted. Since the CCD structure requires explicit global inner product computation between multiple layers, the local implicit calibration (LIC) module is further proposed to reduce distilled convergence difficulty. Second, the task-aware spatial disentangle distillation (TASD) structure is devised to transfer task-decoupled semantics and localization knowledge in a divide-and-conquer manner, alleviating objects’ localization imprecision. Experiments demonstrate that our LTD achieves state-of-the-art performance on several datasets and is a plug-and-play approach to most detectors. The code will be available soon.

Deep learning technology has promoted the object detection task in the remote sensing (RS) field to move toward better performance and more demanding requirements. Except for rigid body objects, component objects (COs) with more complex characteristics remain a detection challenge. Its “partial rules and overall disorder” characteristic limits the model learning ability to the structural features. And the internal noise and relatively sparse arrangement are not conducive to optimizing the model by the existing sample assignment strategies. We propose CODet to detect COs in RS scenes. It consists of a cross-hierarchy feature fusion module (CFM) and a noise-sparse sample assignment (NSA) strategy. CFM learns the potential representation and relative position relationship of components by fusing different level features. NSA redefines the optimization process of sample assignment. It aims to alleviate the problems of classification–localization misalignment (CLM) and the positive–negative sample imbalance (PNI) caused by the object’s internal noise and sparse arrangement. The method is verified on the proposed COD dataset of six categories of COs, reaching an average mAP/mAP50of 54.3/86.0. To be closer to the task requirements of the practical RS scene, we also propose a RS large-scale images inference framework. It includes a dataset (APRoI, labeled with COs and rigid body objects), a large-scale image inference strategy, and a set of evaluation metrics. With CODet as the core, the framework can effectively reduce the inference time by three to four times on images with an average of more than 100 million pixels.

Object detectors of remote sensing (RS) imagery with deep learning have become increasingly popular and rely heavily on extensive labeled data. The source-only detectors, which are trained on massive labeled data in a source domain, in some cases fail to get satisfactory performance on a target domain due to the domain shift. To alleviate the domain shift, popular approaches consider feature distribution alignment, but the target domain with massive unlabeled data is under-utilized. Some methods use the source-only model to generate pseudo labels for target domain data, but the variation of different remote sensing scenarios produces the domain shift, which is injected directly into the pseudo labels. Therefore, We propose the Dual-head rectification Domain Adaptation network (DualDA-Net) to alleviate the domain shift and exploit the potential of unlabeled target domain data. DualDA-Net cooperates the coarse-to-fine consistency alignment (CCA) with dual-head co-training (DHCT) to align the distribution and generate pseudo labels progressively. Specifically, the CCA focuses on source and target domain feature distribution alignment via coarse-to-fine consistency alignment on multi-level features. Moreover, the DHCT with dual detection heads is deployed in the teacher-student framework, where one of the heads complements the other with high-quality predictions to rectify the pseudo labels as supervision and alleviate the biased information. Sufficient experiments have been conducted on several domain adaptation settings. The experimental results demonstrate that our DualDA-Net achieves success in the target domain for cross domain object detection of RS imagery.

Multimodal remote sensing (RS) image segmentation aims to comprehensively utilize multiple RS modalities to assign pixel-level semantics to the studied scenes, which can provide a new perspective for global city understanding. Multimodal segmentation inevitably encounters the challenge of modeling intra- and inter-modal relationships, $i.e$ ., object diversity and modal gaps. However, the previous methods are usually designed for a single RS modality, limited by the noisy collection environment and poor discrimination information. Neuropsychology and neuroanatomy confirm that the human brain performs the guiding perception and integrative cognition of multimodal semantics through intuitive reasoning. Therefore, establishing a semantic understanding framework inspired by intuition to realize multimodal RS segmentation becomes the main motivation of this work. Drived by the superiority of hypergraphs in modeling high-order relationships, we propose an intuition-inspired hypergraph network ( $I^{2}HN$ ) for multimodal RS segmentation. Specifically, we present a hypergraph parser to imitate guiding perception to learn intra-modal object-wise relationships. It parses the input modality into irregular hypergraphs to mine semantic clues and generate robust mono-modal representations. In addition, we also design a hypergraph matcher to dynamically update the hypergraph structure from the explicit correspondence of visual concepts, similar to integrative cognition, to improve cross-modal compatibility when fusing multimodal features. Extensive experiments on two multimodal RS datasets show that the proposed $I^{2}HN$ outperforms the state-of-the-art models, achieving F1/mIoU accuracy 91.4%/82.9% on the ISPRS Vaihingen dataset, and 92.1%/84.2% on the MSAW dataset.

Weakly Supervised 3D Object Detection (WS3DOD) aims to perform 3D object detection with little reliance on 3D labels, which greatly reduces the cost of 3D annotations. In recent literature, the pseudo-label-based approach brings impressive performance, which generates 3D pseudo-labels from 2D bounding boxes. Despite their success, two key issues remain unresolved that reduce the quality of 3D pseudo-labels: 1) the existing local object locating algorithm can not capture complete clusters of points globally, and 2) the existing algorithm can not capture sparse points caused by the unevenly distributed points obtained by LiDAR cameras. Hence, we propose GAL, a Graph-induced Adaptive Learning algorithm, to generate 3D pseudo-labels. First, we propose the Cluster Locating algorithm based on the Minimum Spanning Tree (MST) to globally locate the objects, which can leverage the characteristic that points inside an object are compact while points between objects are discrete. Second, we propose a density-guided adaptive learning algorithm to optimise the Cluster Locating algorithm, named Cuboid Drift. Cuboid Drift considers the inhomogeneous distribution of reflected points on different reflective surfaces of LiDAR imaging. Finally, 3D pseudo-labels generated by GAL are leveraged to train 3D detectors. Extensive experiments on the challenging KITTI and DAIR-V2X-V dataset demonstrate that GAL without 3D labels can be comparable with strongly supervised approaches and outperforms the previous state-of-the-art WS3DOD methods. Moreover, our method saves 88% of the time spent on pseudo-label generation.

As a challenging sub-task of event argument extraction, implicit event argument extraction seeks to identify document-level arguments that play direct or implicit roles in a given event. Prior work mainly focuses on capturing direct relations between arguments and the event trigger; however, the lack of reasoning ability imposes limitations to the extraction of implicit arguments. In this work, we propose anArgument-argumentRelation-enhancedEventArgument extraction (AREA) learning framework to tackle this issue through reasoning in event frame-level scope. The proposed method leverages related arguments of the expected one as clues, and utilizes such argument-argument dependencies to guide the reasoning process. To bridge the distribution gap between oracle knowledge used in the training phase and the imperfect related arguments in the test stage, we introduce a conventional knowledge distillation strategy to drive a final model that can work without extra inputs by mimicking the behaviour of a well-informed teacher model. In addition, considering that conventional knowledge distillation methods transfer knowledge individually, we integrate it with a novel relational knowledge distillation mechanism to explicitly capture the structural mutual argument-argument relation. Moreover, since the training process is not compatible with the real situation, a curriculum learning method is further introduced to make the training process smoother. Experimental results demonstrate that the learning framework obtains state-of-the-art performance on the RAMS and Wikievents datasets. Ablation study and further discussion also show it could handle long-range dependency and implicit argument problems effectively.

Multimodal abstractive summarization for videos is an emerging task that aims to generate a summary from multi-source information (i.e., video, audio transcript). The challenge is how to merge multimodal long sequences to capture rich semantic information without allowing possible noise from either lengthy modal sequence to degrade the other modality and thus hurt the entire model. To address the issues, we propose amultistagefusion network withforgetgate (MFFG), which selectively integrates multi-source information through the cross-fusion in encoding and hierarchical fusion in decoding between modalities, and design a fusion forget gate module to suppress the potential multimodal noise flow of multi-source long sequence. Meanwhile, considering that the source text in this task is lengthy and has the same distribution as the output summary text, we inherit the partial structure of the MFFG model and again propose its variant, single-stage fusion network with forget gate (SFFG), which simplifies the fusion schema, and leverages the long source text to enhance the representation of the target summary. Experimental results on How2 dataset and How2-300 dataset demonstrate the superiority of the two multimodal fusion methods. Further, we provide a version of ASR transcription data of How2 dataset to evaluate model performance under noisy scenarios, and experimental results show obvious advantages of our proposed models over prior systems.

The label assignment problem is a core task in object detection, which mainly focuses on how to define the$positive/negative$samples during the training phase. Recent works have proved that label assignment is significant for performance improvement of the detector. In this article, we propose an exquisite strategy that can dynamically assign labels according samples' joint scores (classification and location). Moreover, our strategy can apply to both 2D and 3D monocular detectors. In our strategy, we formulate label assignment as an optimization problem. Concretely, we first calculate the classification and location costs of each sample, which are treated as points in a 2-D coordinate system. Then an optimal divider line that minimizes the sum of point-to-line distances is designed to separate the$positive/negative$samples. An iterative Genetic Algorithm is employed in acquiring the optimal solution. Furthermore, a GIoU auxiliary branch is devised to keep sample selection consistent during the training and testing phase. Benefitting from the non-maximum suppression (NMS) that utilizes the joint scores of classification and location, excellent detection performance is achieved. Extensive experiments conducted on MS COCO, PASCAL VOC (2D object detection), and KITTI (3D object detection) verify the effectiveness and universality of our proposed Optimal Partition Assignment (OPA).

Sarcasm is a sophisticated construct to express contempt or ridicule. It is well-studied in multiple disciplines (e.g., neuroanatomy and neuropsychology) but is still in its infancy in computational science (e.g., Twitter sarcasm detection). In contrast to previous methods that are usually geared toward a single discipline, we focus on the multidisciplinary cross-innovation, i.e., improving embryonic sarcasm detection in computational science by leveraging the advanced knowledge of sarcasm cognition in neuroanatomy and neuropsychology. In this work, we are oriented toward sarcasm detection in social media and correspondingly propose a multimodal, multi-interactive, and multihierarchical neural network ($M_{3}N_{2} $). We select Twitter, image, text in image, and image caption as the input of$M_{3}N_{2} $since the brain’s perception of sarcasm requires multiple modalities. To reasonably address the multimodalities, we introduce singlewise, pairwise, triplewise, and tetradwise modality interactions incorporating gate mechanism and guide attention (GA) to simulate the interactions and collaborations of involved regions in the brain while perceiving multiple modes. Specifically, we exploit a multihop process for each modality interaction to extract modal information multiple times using GA for obtaining multiperspective information. Also, we adopt a two-hierarchical structure leveraging self-attention accompanied by attention pooling to integrate multimodal semantic information from different levels mimicking the brain’s first- and second-order comprehensions of sarcasm. Experimental results show that$M_{3}N_{2} $achieves competitive performance in sarcasm detection and displays powerful generalization ability in multimodal sentiment analysis and emotion recognition.

N-ary relational knowledge base (KBs) embedding aims to map binary and beyond-binary facts into low-dimensional vector space simultaneously. Existing approaches typically decompose n-ary relational facts into subtuples (entity pairs, triples or quintuples, etc.), and they generally model n-ary relational KBs in Euclidean space. However, n-ary relational facts are semantically and structurally intact, decomposition leads to the loss of global information and undermines the semantical and structural integrity. Moreover, compared to the binary relational KBs, n-ary ones are characterized by more abundant and complicated hierarchy structures, which could not be well expressed in Euclidean space. To address the issues, we propose a gyro-polygon embedding approach to realize n-ary fact integrity keeping and hierarchy capturing, termed as PolygonE. Specifically, n-ary relational facts are modeled as gyro-polygons in the hyperbolic space, where we denote entities in facts as vertexes of gyro-polygons and relations as entity translocation operations. Importantly, we design a fact plausibility measuring strategy based on the vertex-gyrocentroid geodesic to optimize the relation-adjusted gyro-polygon. Extensive experiments demonstrate that PolygonE shows SOTA performance on all benchmark datasets, generalizability to binary data, and applicability to arbitrary arity fact. Finally, we also visualize the embedding to help comprehend PolygonE's awareness of hierarchies.

Multimodal summarization for videos aims to generate summaries from multi-source information (videos, audio transcripts), which has achieved promising progress.However, existing works are restricted to monolingual video scenarios, ignoring the demands of non-native video viewers to understand the cross-language videos in practical applications.It stimulates us to propose a new task, named Multimodal Cross-Lingual Summarization for videos (MCLS), which aims to generate cross-lingual summaries from multimodal inputs of videos.First, to make it applicable to MCLS scenarios, we conduct a Video-guided Dual Fusion network (VDF) that integrates multimodal and cross-lingual information via diverse fusion strategies at both encoder and decoder.Moreover, to alleviate the problem of high annotation costs and limited resources in MCLS, we propose a triple-stage training framework to assist MCLS by transferring the knowledge from monolingual multimodal summarization data, which includes: 1) multimodal summarization on sufficient prevalent language videos with a VDF model; 2) knowledge distillation (KD) guided adjustment on bilingual transcripts; 3) multimodal summarization for cross-lingual videos with a KD induced VDF model.Experiment results on the reorganized How2 dataset show that the VDF model alone outperforms previous methods for multimodal summarization, and the performance further improves by a large margin via the proposed triple-stage training framework. * Equal contribution. † Corresponding author.Portuguese (Pt) Transcript: vamos falar hoje sobre o solo.em primeiro lugar, precisamos de uma grande quan dade de solo bom para transplantes na primavera.ela vai adicionar partes iguais de musgo de turfa e composto de jardinagem que extraímos do nosso sistema interno de compostagem, e então um agregado orgânico, uma pedra chamada perlite, que serve para adicionar volume e aumentar a capacidade de retenção de água e de aeração de sua mistura... English (En) Summary: mix sterile soil for plan ng greens in trays to keep in a hoop house.learn to mix soil for growing greens from an organic farmer in this free gardening video.

Estimating height information from a single remote sensing image is a critical component for 3D perception. Recent methods formulate it as a dense height prediction task based on regression loss functions. However, the regression accuracy is limited by the infinite continuous solution space. In this paper, we propose the soft weighted ordinal (SWO) classification loss for height prediction model to convert the regression problem with infinite continuous values into the classification problem with finite discrete values. which greatly improves the accuracy of high estimation. Specifically, we first define the discrete height rule and introduce the distance penalty metric to transform the continuous ground truth height value to the soft probability distributions. This is then used as supervised information to optimize the pixel-wise classification model. Finally, we utilize soft weighted summation to generate continuous height values in the inference phase. The proposed SWO classification loss can be used directly with existing dense prediction structures whose performance can be strengthened by direct replacement of the loss functions. Comprehensive experiments on the IS-PRS Vaihingen dataset show that the proposed method has achieved promising results.

With the development of remote sensing (RS) acquisition technology, a mass of RS images have been produced, which brings challenges to the traditional manual retrieval methods and gives birth to the automatic RS image retrieval methods. Cross-modal RS image retrieval allows the usage of text and other modalities to retrieve RS images. For its flexible and convenient advantages, it has become a research hotspot. However, cross-modal RS image retrieval encounters the information asymmetry between modalities, i.e., RS images possess multi-scale, multi-objective properties and own rich information. At the same time, the query text is usually short and with less information. To solve the issues above, a cross-modal feature matching network is proposed to learn the feature fusion intra-modalities and the feature association inter-modalities to avoid the poor retrieval performance caused by the information asymmetry. Specifically, for the feature fusion intra-modalities, relying on the powerful feature representation ability of graph network, text and RS image graph modules are designed to fuse the intra-modal features. In terms of the feature correlation between modalities, RS image-text association module is created to attend the parts in text related to RS images and vice versa. Extended experiments on two public standard datasets verify the effectiveness of the proposed model.

Forecasting the future trajectories of multiple agents is a core technology for human-robot interaction systems. To predict multi-agent trajectories more accurately, it is inevitable that models need to improve interpretability and reduce redundancy. However, many methods adopt implicit weight calculation or black-box networks to learn the semantic interaction of agents, which obviously lack enough interpretation. In addition, most of the existing works model the relation among all agents in a one-to-one manner, which might lead to irrational trajectory predictions due to its redundancy and noise. To address the above issues, we present Hypertron, a human-understandable and lightweight hypergraph-based multi-agent forecasting framework, to explicitly estimate the motions of multiple agents and generate reasonable trajectories. The framework explicitly interacts among multiple agents and learns their latent intentions by our coarse-to-fine hypergraph convolution interaction module. Our experiments on several challenging real-world trajectory forecasting datasets show that Hypertron outperforms a wide array of state-of-the-art methods while saving over 60% parameters and reducing 30% inference time.

Abstract The current image captioning methods usually integrate an object detection network to obtain image features at the level of objects and other salient regions. However, the detection network needs to be independently pre‐trained on additional data. Thus, mainly due to the demand for extra training data and computing resources, the detection network's utilization will impose higher training costs on the overall captioning model. In this work, the authors propose a local–global attention model based on two‐step clustering features for image captioning. The two‐step clustering features can be obtained at a relatively low cost and have the presentation ability in objects or other salient image regions. To make the model perceive the image better, the authors introduce a novel local–global attention mechanism. The model will analyse the clustering features from local perspectives to global ones at each time step, making the model better understand the image contents. The authors evaluate the proposed method on the MSCOCO test server, achieving BLEU‐4/METEOR/ROUGE‐L scores of 36.8, 27.4, and 57.2, respectively. With the benefit of reducing training costs, the authors' model also achieves closing results compared with the models using detection features.

Abstract The transformer architecture has been the dominant framework for today's image captioning tasks because of its superior performance. However, existing methods based on transformer often lack the integrated use of multi‐level semantic information and are weak in maintaining the relevance of captions to the image. In this paper, a semantic‐meshed and content‐guided transformer network is introduced for image captioning to solve these problems. The semantic‐meshed mechanism allows the model to generate words by selecting semantic information of multiple interaction levels adaptively through attention‐based reconstruction. And the content‐guided module guides the words generation by using attribute features that represent the image content, which aims to keep the generated caption consistent with the main content of the image. Experiments on dataset on the MSCOCO captioning dataset are conducted to validate the authors’ model and achieve superior results compared to other state‐of‐the‐art method approaches.

Most state-of-the-art deep learning-based methods for extraction of building footprints are aimed at designing proper convolutional neural network (CNN) architectures or loss functions able to effectively predict building masks from remote sensing (RS) images. To properly train such CNN models, large-scale and pixel-level building annotations are required. One common approach to obtain scalable benchmark data sets for the segmentation of buildings is to register RS images with auxiliary geospatial information data, such as those available from OpenStreetMaps (OSM). However, due to land-cover changes, urban construction, and delayed geospatial information updating, some building annotations may be missing in the corresponding ground-truth building mask layers. This will likely introduce confusion in the training of CNN models for discriminating between background and building pixels. To solve this important issue, we first formulate the problem as a long-tailed classification one. Then, we introduce a new joint loss function based on three terms: 1) logit adjusted cross entropy (LACE) loss, aimed at discriminating between building and background pixels from a long-tailed label distribution; 2) weighted dice loss, aimed at increasing the$F_{1}$scores of the predicted building masks; and 3) boundary (BD) alignment loss, which is optimized for preserving the fine-grained structure of building boundaries. Our experiments, conducted on two benchmark building segmentation data sets, validate the effectiveness of our newly proposed loss with respect to other state-of-the-art losses commonly used for extracting building footprints. The codes of this letter will be publicly available fromhttps://github.com/jiankang1991/GRSL_BFE_MA.

The terrain segmentation in polarimetric synthetic aperture radar (PolSAR) images is an important task for image interpretation. Since the speckle noise and complex scattering mechanism exist in SAR images, the classification results achieved by traditional methods appear fragmented. Gradually, deep-learning-based methods are proposed to solve this problem. However, only the amplitude data in the SAR image is utilized, which limits the classification precision. In this letter, a novel method based on a dual-attention fusion network (DAFN) is presented. DAFN is mainly composed of a two-way structure encoder for feature extraction and the attention-based fusion module. Considering the terrain characteristic and the SAR imaging mechanism, the introduction of the polarization information in DAFN increases the discrimination of different categories, which contributes to the consistent and accurate fine-grained classification results. To demonstrate the effectiveness of the proposed method, the corresponding experiments are done based on a GaoFen-3 satellite full-polarization SAR data set, in which the superior performance in terrain segmentation is obtained.

Facing the challenge of small open labeled data sets in remote sensing change detection, this letter proposes a novel supervised change detection method by taking advantages of capsule network which can reach the same performance as traditional convolutional neural networks (CNNs) but with less training data. To achieve this aim, we propose a pseudo-Siamese capsule network which takes both rotational invariance and spatial hierarchies between features into account for aerial images change detection. First, the features of image pairs are extracted by two identical nonshared weights convolutional capsule networks. Second, the extracted features are directly concatenated and sent to another convolutional capsule layer. The change probability map is obtained by calculating the length of the capsule vectors in the final layer. Additionally, to reduce the influence of imbalance samples when we optimize our network, we design a margin-focal loss function to pay more attention to the misclassified samples. Finally, binary change map can be produced by a simple threshold. Experimental results carried out on the SZTAKI AirChange Benchmark Set show that the proposed method achieves comparable and even better results with existing state-of-the-art methods in terms of F-measure.

With the rapid update and iteration of current aerial image data, the continual learning scenarios and catastrophic forgetting problem attracted increased attention, especially in the semantic segmentation task. However, the existing methods mainly focus on the class continual learning in a single task and are not satisfactory when extended to multiple tasks. In this article, we consider more realistic and complicated settings, namely task continual learning. We revisit the characteristics of semantic segmentation and knowledge distillation (KD) strategy, then propose a general and effective framework, named structured inheritance, to learn new tasks while retaining high performance on old tasks. Specifically, we present two structure-preserving penalties: pixel affinity structure loss and representation consistency structure loss. The former breaks the isolation of pixels and retains the pixel interactive information learned by the old tasks. At the same time, the latter protects high-frequency stationary information between sequence semantic segmentation tasks. Our approach does not need to add extra parameters nor does it need to access the data stream of the old tasks. Therefore, it can be applied in practical applications with strict computational burden, memory cost, and storage budget. Extensive continual learning experiments on four semantic segmentation datasets of Vaihingen, Potsdam, DeepGlobe, and Gaofen challenge semantic segmentation dataset (GCSS) prove the effectiveness of our proposed framework, which outperforms the current state-of-the-art methods and even exceeds the theoretical upper-bound performance of multitask learning. The code and models will be made publicly available.

The state-of-the-art deep neural networks have made a great breakthrough in remote sensing image classification. However, the heavy dependence on large-scale data sets limits the application of the deep learning to synthetic aperture radar (SAR) automatic target recognition (ATR) field where the target sample set is generally small. In this work, a metalearning framework named MSAR, consisting of a metalearner and a base-learner, is proposed to solve the sample restriction problem, which can learn a good initialization as well as a proper update strategy. After training, MSAR can implement fast adaptation with a few training images on new tasks. To the best of our knowledge, this is the first study to solve a few-shot SAR target classification via metalearning. In particular, the few-task problem is defined by analyzing the effect of available training classes on the performance of metalearning models. In order to reduce the metalearning difficulties caused by the few-task problem, three transfer-learning methods are employed, which can leverage the prior knowledge from the pretraining phase. Besides, we design a hard task mining method for effective metalearning. Based on the Moving and Stationary Target Acquisition and Recognition (MSTAR) data set, a specialized data set named NIST-SAR is devised to train and evaluate the proposed method. The experiments on NIST-SAR have shown that the proposed method yields better performances with the largest absolute improvements of 1.7% and 2.3% for 1-shot and 5-shot, respectively, over the next best, which indicates that the proposed method is promising and metalearning is a feasible solution for few-shot SAR ATR.

In recent years, deep convolutional neural networks (DCNNs) have made significant progress in cloud detection tasks, and the detection accuracy has been greatly improved. However, most existing CNN-based models have high computational complexity, which limits their practical application, especially for spaceborne optical remote sensing. In addition, most of the methods cannot make adaptive adjustments based on the structural information of the clouds, and blurred boundaries often occur in the detection results. In order to address these problems, this article proposes a lightweight network (DABNet) to achieve high-accuracy detection of complex clouds, not only a clearer boundary but also lower false-alarm rate. Specifically, a deformable context feature pyramid module is proposed to improve the adaptive modeling capability of multiscale features. Besides, a boundary-weighted loss function is designed to direct the network to focus on cloud boundary information and optimize the relevant detection results. The proposed method has been validated on two data sets: the public GF-1 WFV benchmark and our self-built GF-2 cloud detection data set with higher spatial resolution. The experimental results exhibit that DABNet achieves state-of-the-art performance while only using 4.12M parameters and 8.29G multiadds.

Recently, satellite video has become an emerging means of earth observation, providing the possibility of tracking moving objects. However, the existing multi-object trackers are commonly designed for natural scenes without considering the characteristics of remotely sensed data. In addition, most trackers are composed of two independent stages of detection and reidentification (ReID), which means that they cannot be mutually promoted. To this end, we propose an end-to-end online framework, which is called TGraM, for multi-object tracking in satellite videos. It models multi-object tracking as a graph information reasoning procedure from the multitask learning perspective. Specifically, a graph-based spatiotemporal reasoning module is presented to mine the potential high-order correlations between video frames. Furthermore, considering the inconsistency of optimization objectives between detection and ReID, a multitask gradient adversarial learning strategy is designed to regularize each task-specific network. In addition, aiming at the data scarcity in this field, a large-scale and high-resolution Jilin-1 satellite video dataset for multi-object tracking (AIR-MOT) is built for the experiments. Compared with state-of-the-art multi-object trackers, TGraM achieves efficient collaborative learning between detection and ReID, improving the tracking accuracy by 1.2 multiple object tracking accuracy. The code and dataset will be available online (https://github.com/HeQibin/TGraM).

Endowing convolutional neural networks (CNNs) with the rotation-invariant capability is important for characterizing the semantic contents of remote sensing (RS) images since they do not have typical orientations. Most of the existing deep methods for learning rotation-invariant CNN models are based on the design of proper convolutional or pooling layers, which aims at predicting the correct category labels of the rotated RS images equivalently. However, a few works have focused on learning rotation-invariant embeddings in the framework of deep metric learning for modeling the fine-grained semantic relationships among RS images in the embedding space. To fill this gap, we first propose a rule that the deep embeddings of rotated images should be closer to each other than those of any other images (including the images belonging to the same class). Then, we propose to maximize the joint probability of the leave-one-out image classification and rotational image identification. With the assumption of independence, such optimization leads to the minimization of a novel loss function composed of two terms: 1) a class-discrimination term and 2) a rotation-invariant term. Furthermore, we introduce a penalty parameter that balances these two terms and further propose a final loss to Rotation-invariant Deep embedding for RS images, termed RiDe. Extensive experiments conducted on two benchmark RS datasets validate the effectiveness of the proposed approach and demonstrate its superior performance when compared to other state-of-the-art methods. The codes of this article will be publicly available athttps://github.com/jiankang1991/TGRS_RiDe.

Aircraft detection in synthetic aperture radar (SAR) images plays a significant role in dynamic monitoring and national security. Previous methods have difficulty in obtaining the desirable detection performance due to the interference of complex scenes and diversity of aircraft sizes. In order to solve these problems, we propose an innovative scattering feature relation network (SFR-Net) in this article. First, considering that the strong scattering points of the aircraft in SAR images are usually discrete, we leverage the proposed scattering point relation module to fulfill the analysis and correlation of scattering points. By enhancing the characteristics and relationships among the scattering points, this method is beneficial to guarantee the completeness of aircraft detection results. Second, we design a salient fusion module to adaptively aggregate the features from different layers of SFR-Net with rich semantic information and plentiful details, which can highlight the significant objects with different sizes and enhance the distinguishable features. Third, to reduce the false alarm and improve the localization accuracy, the contextual feature attention is presented to capture the global spatial and semantic information with a large receptive field. Overall, the SFR-Net is designed based on the SAR imaging mechanism and the scattering characteristics of aircrafts. The extensive experiments are conducted on the SAR aircraft detection dataset (AIRD) from the Gaofen-3 satellite to demonstrate the effectiveness of the SFR-Net and also illustrate that our method achieves state-of-the-art performance.

Synthetic aperture radar (SAR) images provide all-weather and all-time capabilities for Earth observation, which becomes highly beneficial in the field of intelligent remote sensing (RS) image interpretation. Due to these advantages, SAR images have been widely exploited in automatic building segmentation tasks under poor weather conditions, especially when disasters happen. However, compared to optical images, the semantics inherent to SAR images are less rich and interpretable due to factors such as speckle noise and imaging geometry. In this scenario, most state-of-the-art methods are focused on designing advanced network architectures or loss functions for building footprint extraction. However, few works have been oriented toward improving segmentation performance through knowledge transfer from optical images. In this article, we propose a novel method based on theDisOptNetnetwork, which can distill the useful semantic knowledge from optical images into a network only trained with SAR data. Specifically, we first analyze the multilevel feature discrepancies between multiple stages of the networks pretrained on the two image modalities. We observe that feature discrepancies start to increase as the encoding stage gradually changes from low level to high level. Based on such observation, we reuse the early stage features and construct parallel convolutional neural network (CNN) branches that are responsible for capturing high-level domain-specific knowledge for each image modality. The optical branch is aimed at mimicking feature generation at the optical pretrained network given the input SAR images. Then, an aggregation module is introduced to calibrate and fuse the features from different modalities while generating the building segments. Extensive experiments were conducted on a large-scale multisensor all-weather building segmentation dataset with state-of-the-art methods used for comparison. Our experimental results validate the effectiveness ofDisOptNet, which demonstrates great potential in the task of weather-independent building footprint generation under real scenarios. The codes of this article will be made publicly available athttps://github.com/jiankang1991/TGRS_DisOptNet.

Segmentation incremental learning has received a lot of attention in recent years due to the ability to overcome the problem of catastrophic forgetting. Our study found that differences in label number distribution affect the performance of segmentation incremental learning. Because the labels for pixels of the old category are marked as background when the model is trained on the new tasks, the label number distribution is inconsistent with static learning that is considered to be the upper bound on incremental learning, which hinders the mitigation of the catastrophic forgetting problem. In response to the above problems, we propose an incremental learning method named SIL-LAND, which improves the accuracy by making the label number distribution of our method close to that of static learning. From the perspective of high-level semantic labels, we propose the prototype update mechanism for the problem that non-adaptive representative prototypes ignore the sample diversity of semantic categories in remote sensing images. By compensating for the difference in label number distribution at the feature level, the distance between the prototype and the actual class center is reduced; Aiming at the lack of semantic consistency between feature vectors and prototypes, we propose a similarity measure module to increase the intra-class similarity between the prototype and corresponding feature vectors. From the perspective of one-hot labels, we propose label reconstruction, including foreground screening and background padding to make the number distribution of one-hot labels as close as possible to that of static learning. A series of experimental results demonstrate the effectiveness of our method.

Due to the differences in the feature distribution between classes, when the model learns in a continuous data stream, it will encounter catastrophic forgetting. The incremental learning methods have shown great potential to solve this problem. However, most existing methods based on task-incremental learning are difficult to adapt to characteristics of remote sensing scenes with few differences in appearance but large differences in features, which is not conducive to artificially distinguish task-identity document (ID). Thus, we propose a class-incremental learning (CIL) network for small objects enhancing semantic segmentation in aerial imagery. Specifically, considering the superior accuracy of the binary classifier, we propose a twin-auxiliary (TA) model that adds an auxiliary binary classification task. Then, for expansion and contraction at the edge and small object confusion problems, we introduce a diversity distillation loss, using the results of binary-classifier to constrain the multiclass segmentation results and strengthen the attention to the locations of the segmentation results that have changed. Finally, we design a conflict reduction mechanism for multihead classifier to achieve single-head prediction for CIL. Experiments demonstrate that our method has good performance on the Vaihingen and Potsdam datasets by the International Society for Photogrammetry and Remote Sensing (ISPRS), outperforming state-of-the-art (SOTA) incremental learning methods. The code will be available soon.

With the proposal of neural architecture search (NAS), automated network architecture design gradually becomes a new way in deep learning research. Due to its high capability regarding automated design, some pioneers have made an attempt to apply NAS in remote sensing and made some achievements, like 1-D/3-D Auto-convolutional neural network (CNN) and polarimetric synthetic aperture radar (PolSAR)-tailored Differentiable Architecture Search (PDAS). However, there are still some areas to be improved for existing NAS in remote-sensing field. In this article, we propose a random topology and random multiscale mapping (RTRMM) method to generate a multiscale and lightweight architecture for remote-sensing image recognition. First, a random topology generator generates the topology through random graph. Second, during the experiment, we find remote-sensing image features extracted by a multiscale network are more appropriate, compared with features extracted by a single-scale model. Nevertheless, the complexity inevitably increases with the introduction of a multiscale concept. Consequently, we design a variable search space consisting of decomposition convolution units under the guidance of mathematical analysis. The mapping of each neuron is then determined by a random multiscale mapping sampler. After that, we assemble the topology and mappings into blocks and construct three RTRMM models. Experiments on four scene classification datasets confirm the feature extraction capability and lightweight performance of RTRMM models. Moreover, we also observe that our approach achieves a better tradeoff between floating-point operations (FLOPs) and accuracy than some current well-behaved methods. Furthermore, the results on Vaihingen dataset verify the high feature-transfer capability.

Image scene classification aiming to assign specific semantic labels for each image is vital important for the applications of remote sensing (RS) data. In real world, since the observation environment is open and dynamic, RS images are collected sequentially and the numbers of images and classes grow rapidly over time. Most existing scene classification methods are offline learning algorithms which are inefficient and unscalable for this scenario. In this paper, an incremental learning with open set recognition framework is proposed for RS image scene classification in the open and dynamic environment, called ILOSR, which can identify the unknown classes from a stream of data and learn these new classes incrementally. Specifically, a controllable convex hull-based exemplar selection strategy is designed to address the catastrophic forgetting issue in incremental learning, which can reduce training time and memory footprint effectively. In addition, a new loss function based on prototype learning and uncertainty measurement is proposed for open set recognition to enhance the inter-class discrimination and intra-class compactness of the learned deep features. Experimental results on real remote sensing datasets demonstrate that the proposed method can not only outperform the state-of-the-art approaches on offline classification, incremental learning and OSR problem separately, but also achieve better and more stable performance in the experiments for incremental learning with OSR.

Numerous deep-learning methods have been successfully applied to semantic segmentation and height estimation of remote-sensing imagery. It has also been proved that such framework can be reusable for multiple tasks to reduce computational resource overhead. However, there are still some technical limitations due to the semantic inconsistency between 3-D and 2-D features and strong interference of different objects with similar spectral-spatial properties. Previous works have sought to address these issues through hard parameter sharing or soft parameter sharing schemes. But due to unintentional integration, the specific information transmitted between multiple tasks is not clear or in a lot of redundancy. Furthermore, tuning the weights by hand between classification and regression loss function is challenging. In this paper, a novel multi-task learning method, termed ASSEH, is proposed to associatively segment semantics and estimate height from monocular remote-sensing imagery. First, considering semantic inconsistency across tasks, we design a task-specific distillation (TSD) module containing a set of task-specific gating units for each task at the cost of fewer parameters. The module allows for task-specific features to be tailored from backbone, whilst allowing for task-shared features to be transmitted. Second, we leverage the proposed cross-task propagation (CTP) module to construct and diffuse the local pattern graphlets at the common positions across tasks. Such a high-order recursive method can bridge two tasks explicitly to effectively settle semantic ambiguities caused by similar spectral characteristics with less computational burden and memory requirements. Third, a dynamic weighted geometric mean (DWGeoMean) strategy is introduced to dynamically learn the weights of each task and be more robust to the magnitude of the loss function. Finally, the results on ISPRS Vaihingen and Urban Semantic 3D data set well demonstrate that our ASSEH achieves the state-of-the-art performance.

Existing deep learning models usually assume that all data obeys independent identically distribution, which is unreasonable in remote sensing. Due to the differences in camera parameters, spectral ranges, resolutions, and so on, the images acquired by remote sensing sensors may be greatly diverse, causing models to face catastrophic forgetting when they are trained on new data only. Thus, incremental learning is introduced. An ideal incremental learning model should be expanded as the number of tasks increases, so as to have enough ability to adapt to the changes in data. However, existing approaches normally expand heavy modules for each task, making the holistic models cumbersome. In this article, a lightweight incremental learning approach (LIL) is proposed for remote sensing image scene classification. We replace the role of the feature extractor with extracting features of a single task instead of task-sharing features of all tasks to lighten the backbone. In addition, we propose a light feature transfer module (FTM) to realize the alignment of data distributions between different tasks in the feature domain. Furthermore, dual-constraint loss with knowledge distillation and adversarial learning is introduced to promote the mapping and alignment of data distributions at both the feature level and the semantic level. In LIL, only a tiny FTM and a classifier are added to the model when the model learns a new task. Experimental results show that our approach with a small number of parameters outperforms state-of-the-art approaches for incremental learning on both a single dataset and a sequence of multiple datasets.

Semantic segmentation in very-high-resolution (VHR) aerial images is one of the most challenging tasks in remote sensing image understanding. Most of the current approaches are based on deep convolutional neural networks (DCNNs). However, standard convolution with local receptive fields fails in modeling global dependencies. Prior research works have indicated that attention-based methods can capture long-range dependencies and further reconstruct the feature maps for better representation. Nevertheless, limited by the mere perspective of spatial and channel attention and huge computation complexity of self-attention (SA) mechanism, it is unlikely to model the effective semantic interdependencies between each pixel pair of remote sensing data with complex spectra. In this work, we propose a novel attention-based framework named hybrid multiple attention network (HMANet) to adaptively capture global correlations from the perspective of space, channel, and category in a more effective and efficient manner. Concretely, a class augmented attention (CAA) module embedded with a class channel attention (CCA) module can be used to compute category-based correlation and recalibrate the class-level information. In addition, we introduce a simple yet effective region shuffle attention (RSA) module to reduce feature redundant and improve the efficiency of SA mechanism via regionwise representations. Extensive experimental results on the ISPRS Vaihingen, Potsdam benchmark, and iSAID data set demonstrate the effectiveness and efficiency of our HMANet over other state-of-the-art methods.

Semantic segmentation in aerial imagery is still an important, yet challenging task due to the complex characteristics of remote-sensing data. The critical issues consist of: 1) extreme foreground–background imbalance; 2) large intra-class variance; and 3) arbitrary-oriented, dense, and small objects. The above challenges make it unlikely to model the effective global interdependencies of semantic heterogeneous regions. Besides, general semantic segmentation methods suffer from feature ambiguity due to the joint feature learning paradigm, leading to inferior detail information. In this article, we propose an improved semantic segmentation framework to tackle these problems via graph reasoning (GR) and disentangled learning. On the one hand, a simple, yet effective GR unit is introduced to implement coordinate-interaction space mapping and perform relation reasoning over the graph. It can be deployed on the feature pyramid network (FPN) to exploit cross-stage multi-scale information. On the other hand, we propose a so- called disentangled learning paradigm to explicitly model the foreground and boundary objects, instantiated as foreground prior estimation (FPE) and boundary alignment (BA). The indication of the intermediate feature can be effectively emphasized to enhance the discriminative abilities of the network. Extensive experiments over iSAID, ISPRS Vaihingen, and the general Cityscapes datasets demonstrate the effectiveness and efficiency of the proposed framework over other state-of-the-art semantic segmentation methods.

With the availability of very high-resolution remote sensing imagery, the fine-grained recognition of complex geospatial facilities has become possible. We can view these facilities as a combination of component objects with specific functions and distribution. However, the existing methods are insufficient in modeling spatial relations of component objects. In this article, we propose an instance-based multitask graph network (IBMG-Net) for complex facility recognition. Specifically, we perform pixel-level component objects prediction and facility recognition simultaneously and achieve performance improvement of both tasks by joint multitasking training. Given the component information, we build an instance-based graph neural network (IBGN) where components are defined as nodes and their spatial relations are encoded as edges. The IBGN module aims to flexibly model spatial relations of complex facility. To enhance the feature representation of component objects, we utilize the multiscale region of interest module (MS-ROI) to retain all scale-specific features and the sparse context information module (SCM) to aggregate long-range context information. In addition, we build a new multitask dataset for complex facility recognition in remote sensing (MCF dataset) to verify the effectiveness of our method and alleviate the lack of pixel-level labeled multitask datasets in remote sensing. Extensive experiments on MCF also indicate that the significant performance improvement of our approach to complex facility recognition.

Despite the extraordinary success of the deep architectures on semantic segmentation for remote sensing (RS) images, they have difficulties in learning new classes from a sequential data stream because of catastrophic forgetting. Continual learning for semantic segmentation (CSS) is an emerging trend for its capability to cope with the above problems effectively. However, old classes from previous steps are collapsed into the background, which further aggravates the challenge of CSS in the RS scene. In this article, we revisit the knowledge distillation (KD) strategy and the characteristics of class-incremental semantic segmentation (CISS) and then present a generalized and effective framework to learn new classes while preserving knowledge of the learned classes. In particular, we propose two novel historical information-guided modules: the feature global perception module and the label reconstruction (LR) module. The former enables the current model to pay more attention to the region related to the old categories identified by the historical information when learning new classes. Meanwhile, the latter retrieves pixels belonging to the learned classes from the background to handle the background shift problem and maintain the high performance of old classes. We have conducted comprehensive experiments on two RS semantic segmentation datasets of Instance Segmentation in Aerial Images Dataset (iSAID) and Gao Fen (GF) challenge semantic segmentation dataset (GCSS). The experimental results outperform the current state-of-the-art methods in most incremental settings, which demonstrates the effectiveness of the proposed framework.

Recently, deep learning in synthetic aperture radar (SAR) automatic target recognition (ATR) has made significant progress, but the sample limitation problem in the SAR field is still obvious. Compared with the optical remote sensing images, the SAR images are insufficient, especially those containing the geospatial targets with certain target attitude angles (TAAs). To solve these problems, a novel few-shot learning framework named scattering characteristics analysis network (SCAN) is proposed in this article. First, a scattering extraction module (SEM) is designed to combine the target imaging mechanism with the network, which learns the number and distribution of the scattering points for each target type via explicit supervision. Besides, considering the imaging variability of SAR targets, a TAA-guided metalearning network consisting of an angle self-adaption classifier (ASC) and a frequency embedded module (FEM) is designed. ASC guides the network to focus on the positive sample pairs with different TAAs. FEM combines pulse cosine transform (PCT) with the network training process effectively to enrich frequency-domain information. In addition, a new dataset named SAR aircraft category dataset is constructed for the experiments. Compared with other few-shot SAR target classification approaches, our model efficiently integrates the scattering characteristics with the learning process, and the test accuracy for 5-way 1-shot has been improved by 4.74%. Finally, the experimental results are provided to demonstrate the validity of the proposed method.

Ship detection has broad applications in many areas, including fishery management, maritime rescue, and maritime monitoring. Recently, numerous detectors based on deep learning have been carried in ship detection in synthetic aperture radar (SAR) images. However, detecting the inshore ships faces enormous challenges because of the strong scattering interference of the inland area. In order to address such issues, a novel method named strong scattering points network for ship detection is proposed in this article. First, according to the SAR imaging mechanism, the ships usually appear strong scattering phenomenon in the SAR images. Therefore, the proposed method detects the strong scattering points on the ship and then aggregates their positions to obtain the ship’s arbitrary orientation box. Second, our method designs an embedding vector to cluster these points as an individual object to regress the oriented bounding box. Third, in order to distinguish the strong scattering points on land, a ship attention module is employed to extract the image texture features and representations of local features. It can suppress the false alarm caused by land interference in the detection process. Furthermore, to demonstrate the effectiveness of the proposed algorithm, this article introduces a new ship dataset for oriented ship detection named large-scale dataset for ship detection in SAR images (LDSD). Moreover, the public SAR ship detection dataset (SSDD) is utilized to verify the robustness and generalization ability of the detector. The experimental results on two datasets show that our method has a strong anti-interference ability in the inshore background and achieves state-of-the-art detection performance.

Geographic object segmentation is a fundamental yet challenging problem for remote sensing image interpretation. The prevalent paradigm to solve this problem is to train deep neural networks on massive labeled samples. Although remarkable achievements have been attained, these methods suffer from the severe dependence on the large-scale dataset and require a long training process with high computation burden. To address these issues, a deep metametric learning framework, named DMML-Net, consisting of the metametric learner and the base-metric learner, is proposed for few-shot geographic object segmentation. First, DMML-Net formulates the segmentation as the metric-based pixel classification and develops a deep feature pyramid comparison network as the architecture of the metric learner for multiscale metric learning. Benefiting from this design, the segmentation can be efficiently solved, as well as being robust to deal with the scale variations of geographic objects. Second, an affinity-based fusion mechanism is introduced to adaptively reweight and fuse the semantic information across samples, effectively calibrating the deviation of prototypes induced by the intraclass variations. Third, considering the impact of the large interclass distribution divergences, DMML-Net presents a metametric training paradigm to provide the metric model with flexible scalability for fast adaptation to novel tasks. After metatraining, DMML-Net can be applied for the few-shot segmentation tasks of novel geographic objects with only a few gradient steps on the small training set. Experimental results on two benchmark remote sensing datasets demonstrate the validity and the superiority of our method in low-shot conditions where there are only one to ten labeled samples.

Range sidelobe is very common in synthetic aperture radar (SAR) images, particularly when imaging scene includes strongly scattering targets such as ships or complex buildings. As a kind of interference, it may reduce the image quality and hinder the image interpretation. Hence, range sidelobe suppression is an important mission for SAR images. The main task of mitigating the sidelobe is how to achieve the most effective suppression with the minimal resolution loss and signal-to-noise ratio (SNR) loss. However, the widely recognized classic method, spatially variant apodization (SVA), still has a lot of residual sidelobe energy and other problems. This article proposes a novel suppression approach based on time-variant transmission of chaotic frequency modulation (CFM) signals. The key is to build an appropriate transmitted signal set, where the signals are generated by various chaotic initial states and the same special map with low mixing rate and uniform invariant probability density (IPD). Due to their beneficial autocorrelation properties, the proposed approach achieves superior performance in range sidelobe suppression and resolution preservation. More importantly, it maintains the energy of the signals and overcomes the SNR loss that occurs in some classic methods, such as spectral weighting (SW) and SVA. In addition, it is suitable for both vertical and squint side-looking mode and can well reconstruct the weakly scattering targets which are severely disturbed by range sidelobe. All of them are validated by comparative experiments.

Object detection is a fundamental part of the interpretation of remote sensing imagery. The one-stage object detector has been adopted into this field because of its high computational efficiency. However, this detector suffers from the misalignment among predefined anchor, object, and feature extracted by standard convolution kernel both in spatial and scale. It limits the further improvement of performance, especially for the long-narrow and multiscale geospatial objects. In this article, the problem is defined asthe feature misalignmentproblem. To deal with this issue, an efficient feature aligned single-shot detector (ASSD) is proposed, which consists of two modules: a novel pseudo anchor proposal module (PAPM) and a flexible context-based feature alignment module (CFAM). The PAPM replaces the regular anchor group with the proposed core anchor and refines it to get aligned locations. It can tackle the spatial misalignment between anchors and their corresponding objects and alleviate the negative/positive imbalance problem. Then, the CFAM adaptively adjusts the sampling points of the convolution kernel and collects the context information according to the aligned core anchor. This plug-and-play module can effectively rectify the misalignment between kernel and objects and extract aligned and robust features. A series of comprehensive experiments are conducted on two large-scale public remote sensing object detection datasets. Experiment results suggest that the proposed method is effective to alleviate the misalignment problem. Compared with the baseline model, the detection accuracy is improved by 8.5% mAP and 11.0% mAP on the challenging benchmark for object detection in optical remote sensing image (DIOR) and a large-scale dataset for object detection in aerial image (DOTA) dataset, respectively. Our best-resulting model achieves the state-of-the-art performance, surpassing other one-stage detectors both on the two datasets at a high detection speed of 21 FPS.

Deep learning-based object detectors have been widely adopted in the field of remote sensing imagery interpretation. These detectors heavily depend on the expensive large-scale labeled datasets, while the scarce remote sensing datasets limit the performance. The domain adaptive object detection can alleviate this problem. However, it struggles with the confusing feature’s alignment, damaging the domain generalization performance, especially for the remote sensing scene with sparse objects and diverse backgrounds. For that reason, a semisynthetic data generator (SDG) is proposed to automatically generate the remote sensing dataset with low cost and replace the real-world training dataset, afeature aligned domain adaptive object detector(FADA) is proposed to enhance the domain adaptation among the cross-domain remote sensing images. The FADA contains two proposed modules in addition to the base detector: an adversarial-based foreground alignment (AFA) and a prototype-based confusing feature alignment (PCFA). The AFA aligns the cross-domain foreground feature by adversarial training (AT), and it can filter the noisy background feature that is not suitable to transfer. Then, the PCFA adaptively aligns the confusing background and foreground feature, further promoting the domain adaptation performance. Comprehensive experiments validate the effectiveness of the proposed method. Compared with the baseline model trained on the semisynthetic source dataset, our FADA improves the generalized performance on the real-world target dataset a large-scale Dataset for Object deTection in Aerial images (DOTA) by 15.7% average precision (AP) and achieves state-of-the-art results.

The lightweight model has played an important role in the remote sensing (RS) realm. The existing researchers have proposed many models with lightweight structures, but their performance still has a gap compared with the deep model. A promising approach to optimize the lightweight model is knowledge distillation (KD), which can be viewed as knowledge transfer from the teacher model. However, the existing KD approaches have some issues. On one hand, offline distillation methods usually ignore the interactive learning between the student model and the teacher model. On the other hand, knowledge transfer does not consider instance property. This offline distillation strategy without property perception may not suitable for multiscale, diverse, and complex RS instances and results in a suboptimum training status. In this article, we propose a dynamic interactive learning (DIL) framework for optimizing RS lightweight detectors. First, we propose an instance interaction learning module. It calculates the value of every instance in the batch of the teacher and student prediction by each model’s real-time state and instance property. Then according to the DIL thought, we facilitate the low-quality instance to learn from the high-quality one whether it is from the teacher or student model. Moreover, we also propose the instance property perception (IPP) strategy that weighs the distillation knowledge of instances according to their feature, category, and location property. In the proposed DIL framework, both the teacher and student models are trained together and it is cost-free in the testing phase. Extensive experiments on three RS datasets demonstrate the effectiveness of the DIL.

The resolution degradation poses a huge challenge for object detection (OD) in the aerial imagery. Existing methods utilize super resolution (SR) based on Generative Adversarial Network (GAN) to restore texture details in degraded images. However, constrained detection results are still acquired due to the object feature difference between restored and clear images. Therefore, we propose a simple-yet-effective learning method called Mutual-Feed Learning (MFL) to solve the problem in this paper. A closed-loop structure is designed via building the feedback connection based on the feedforward connection between the two tasks. It effectively delivers the object spatial and feature information from OD to SR, and provides restoration-enhanced images from SR to OD. Specifically, a Feedback of Region of Interest (FROI) module is introduced to realize a region-level discrimination under the guidance of object information. It guides the discrimination process of super resolution. Furthermore, a Multi-Scale Object Information (MSOI) module is developed to implement a feature-level restoration by narrowing differences in object-related features. It improves the generation process of super resolution. Then object detection can be performed in restoration-enhanced images to obtain more accurate results. Extensive experiments over NWPU VHR-10, COWC, and FAIR1M dataset show that the method can achieve state-of-the-art results.

Lightweight object detector is currently gaining more and more popularity in remote sensing. In general, it’s hard for lightweight detectors to achieve competitive performance compared to traditional deep models, while knowledge distillation is a promising training method to tackle the issue. Since the background is more complicated and the object size varies extremely in remote sensing images, it will deliver lots of noise and affect the training performance when directly applying the existing knowledge distillation methods. To tackle the above problems, we propose an Adaptive Reinforcement Supervision Distillation (ARSD) framework to promote the detection capability of the lightweight model. Firstly, we put forward a multiscale core features imitation (MCFI) module for transferring the knowledge of features, which can adaptively select the multiscale core features of objects for distillation and focus more on the features of small objects by an area-weighted strategy. In addition, a strict supervision regression distillation (SSRD) module is designed to select the optimal regression results for distillation, which facilitates the student to effectively imitate the more precise regression output of the teacher network. Massive experiments on the DOTA, DIOR, and NWPU VHR-10 datasets prove that ARSD outperforms the existing distillation SOTA methods. Moreover, the performance of lightweight model trained with our method transcends other classic heavy and lightweight detectors, which beneficiates the development of lightweight models.

In recent years, more concerns are shed on the lightweight detection model in remote sensing (RS), but it is difficult to reach a competitive performance relative to the deep model. Knowledge distillation has been verified as a promising method, which can promote the performance of the lightweight model without extra parameters. While there are two key issues of detection distillation, one is the sample selection, the other is the knowledge selection. Since the varying object size and complex features in RS, the existing methods based on the fixed threshold are incapable of selecting the optimal distillation samples and they also ignore the potential multivariate knowledge among RS samples simultaneously. In this paper, we propose a statistical sample selection and multivariate knowledge mining framework. The statistical sample selection module formulates the task as the modeling and splitting the probability distribution of sample selection cost, which is more suitable for dynamically choosing multiscale samples in RS and eliminates the distortion of previous static distillation selection. Furthermore, to mine the complex feature knowledge of samples in RS, we design a multivariate knowledge mining module, in which knowledge includes explicit and implicit knowledge. The proposed module validly deliver the core knowledge from the teacher model to the lightweight model. Massive experiments on three challenging RS datasets (DOTA, NWPU VHR-10, DIOR) prove that our method achieves state-of-the-art performance.

Cloud detection relying on deep convolutional neural networks obtains remarkable accuracy gains at the expense of high computation and storage costs, which are difficult to deploy to resource-constrained devices, such as intelligent satellites. Recently, knowledge distillation (KD) has been a promising solution for compact model. However, most existing KD methods only transfer the feature relationship of pairwise pixel which fails to cope with thin clouds and cloud-like objects in complex scenes. Furthermore, those KD methods directly imitate the output of complicated model regardless of the correctness. In this article, we propose a novel Category Correlation and Adaptive Knowledge Distillation (CAKD) framework for the lightweight cloud detection network. We design a category relational context (CRC) module to refine the structured pixel-category correlation from the teacher and student network. Then, we perform the category correlation distillation (CCD) to make the student model better address the intra-class consistency and the inter-class difference, thus reducing the category confusion. Besides, a pixel-adaptive distillation (PAD) module is utilized to adaptively transfer the soft-output knowledge of teacher model by extracting the teacher’s pixel prediction probability. Extensive experiments on Landsat 8, Landsat 7, Gaofen-2, Gaofen-1 and Google Earth dataset report the effectiveness and universality of our distillation method. The CAKD allows MobileNetV2 with 2.31M parameters and 4.63G FLOPs to outperform advanced cloud detection methods without the added inference overhead.

Remote sensing (RS) cross-modal text–image retrieval has attracted extensive attention for its advantages of flexible input and efficient query. However, traditional methods ignore the characteristics of multiscale and redundant targets in RS image, leading to the degradation of retrieval accuracy. To cope with the problem of multiscale scarcity and target redundancy in RS multimodal retrieval task, we come up with a novel asymmetric multimodal feature matching network (AMFMN). Our model adapts to multiscale feature inputs, favors multisource retrieval methods, and can dynamically filter redundant features. AMFMN employs the multiscale visual self-attention (MVSA) module to extract the salient features of RS image and utilizes visual features to guide the text representation. Furthermore, to alleviate the positive samples ambiguity caused by the strong intraclass similarity in RS image, we propose a triplet loss function with dynamic variable margin based on prior similarity of sample pairs. Finally, unlike the traditional RS image-text dataset with coarse text and higher intraclass similarity, we construct a fine-grained and more challenging Remote sensing Image-Text Match dataset (RSITMD), which supports RS image retrieval through keywords and sentence separately and jointly. Experiments on four RS text–image datasets demonstrate that the proposed model can achieve state-of-the-art performance in cross-modal RS text–image retrieval task.

Semantic localization (SeLo) refers to the task of obtaining the most relevant locations in large-scale remote sensing (RS) images using semantic information such as text. As an emerging task based on cross-modal retrieval, SeLo achieves semantic-level retrieval with only caption-level annotation, which demonstrates its great potential in unifying downstream tasks. Although SeLo has been carried out successively, but there is currently no work has systematically explores and analyzes this urgent direction. In this paper, we thoroughly study this field and provide a complete benchmark in terms of metrics and testdata to advance the SeLo task. Firstly, based on the characteristics of this task, we propose multiple discriminative evaluation metrics to quantify the performance of the SeLo task. The devised significant area proportion, attention shift distance, and discrete attention distance are utilized to evaluate the generated SeLo map from pixel-level and region-level. Next, to provide standard evaluation data for the SeLo task, we contribute a diverse, multi-semantic, multi-objective Semantic Localization Testset (AIR-SLT). AIR-SLT consists of 22 large-scale RS images and 59 test cases with different semantics, which aims to provide a comprehensive evaluations for retrieval models. Finally, we analyze the SeLo performance of RS cross-modal retrieval models in detail, explore the impact of different variables on this task, and provide a complete benchmark for the SeLo task. We have also established a new paradigm for RS referring expression comprehension, and demonstrated the great advantage of SeLo in semantics through combining it with tasks such as detection and road extraction. The proposed evaluation metrics, semantic localization testsets, and corresponding scripts have been open to access at https://github.com/xiaoyuan1996/SemanticLocalizationMetrics.

Remote sensing (RS) crossmodal text-image retrieval has become a research hotspot in recent years for its application in semantic localization. However, since multiple inferences on slices are demanded in semantic localization, designing a crossmodal retrieval model with less computation but well performance becomes an emergent and challenging task. In this article, considering the characteristics of multi-scale and target redundancy in RS, a concise but effective crossmodal retrieval model (LW-MCR) is designed. The proposed model incorporates multi-scale information and dynamically filters out redundant features when encoding RS image, while text features are obtained via lightweight group convolution. To improve the retrieval performance of LW-MCR, we come up with a novel hidden supervised optimization method based on knowledge distillation. This method enables the proposed model to acquire dark knowledge of the multi-level layers and representation layers in the teacher network, which significantly improves the accuracy of our lightweight model. Finally, on the basis of contrast learning, we present a method employing unlabeled data to boost the performance of RS retrieval model further. The experiment results on four RS image-text datasets demonstrate the efficiency of LW-MCR in RS crossmodal retrieval (RSCR) tasks. We have released some codes of the semantic localization and made it open to access athttps://github.com/xiaoyuan1996/retrievalSystem.

Cross-modal remote sensing text-image retrieval (RSCTIR) has recently become an urgent research hotspot due to its ability of enabling fast and flexible information extraction on remote sensing (RS) images. However, current RSCTIR methods mainly focus on global features of RS images, which leads to the neglect of local features that reflect target relationships and saliency. In this article, we first propose a novel RSCTIR framework based on global and local information (GaLR), and design a multi-level information dynamic fusion (MIDF) module to efficaciously integrate features of different levels. MIDF leverages local information to correct global information, utilizes global information to supplement local information, and uses the dynamic addition of the two to generate prominent visual representation. To alleviate the pressure of the redundant targets on the graph convolution network (GCN) and to improve the model’s attention on salient instances during modeling local features, the denoised representation matrix and the enhanced adjacency matrix (DREA) are devised to assist GCN in producing superior local representations. DREA not only filters out redundant features with high similarity, but also obtains more powerful local features by enhancing the features of prominent objects. Finally, to make full use of the information in the similarity matrix during inference, we come up with a plug-and-play multivariate rerank (MR) algorithm. The algorithm utilizes the$k$nearest neighbors of the retrieval results to perform a reverse search, and improves the performance by combining multiple components of bidirectional retrieval. Extensive experiments on public datasets strongly demonstrate the state-of-the-art performance of GaLR methods on the RSCTIR task. The code of GaLR method, MR algorithm, and corresponding files have been made available at:https://github.com/xiaoyuan1996/GaLR.

Convolutional neural network (CNN) has attracted extensive attention in the research field of polarimetric synthetic aperture radar (PolSAR) terrain segmentation. However, directly using CNN in PolSAR terrain segmentation while ignoring the characteristics of PolSAR images has become the main factor restricting the performance of algorithms. In this article, we propose an efficient PolSAR terrain segmentation algorithm called TS-SHES, which integrates the polarization scattering characteristics of PolSAR images and the CNN learning process into a unified architecture. First, considering the intrinsic structure of complex-valued PolSAR data, TS-SHES transforms the scattering matrix into the form of amplitude and phase components, which preserves the original information maximally. Then, TS-SHES introduces a scattering harmonized encoding method (SH-Enc) to balance the feature contributions of weak and strong scattering regions as well as map the two components into the same representation space. Through the above scattering harmonization operations, the segmentation performance of CNN on weak scattering regions can be improved, and the feature imbalance in amplitude and phase can be alleviated. Furthermore, in view of the implicit states of CNN feature construction, a scattering explicit learning network (SEL-Net) is presented to collect the scattering features of amplitude and phase. Via explicit supervision, SEL-Net avoids the incomplete collection of scattering information caused by implicit feature construction, thereby improving the segmentation accuracy. Abundant experiments are conducted on two PolSAR images acquired by the GaoFen-3 satellite, which demonstrates the superiority of our proposed algorithm.

Recently, many studies exploit deep neural networks to promote terrain segmentation in polarimetric synthetic aperture radar (PolSAR) images. However, these works usually inherit the nature-scene approaches directly and may not be robust for the PolSAR image segmentation task. The main limitations include single-type feature construction, weak feature consistency, and geometry-agnostic collection of scattering information. In this article, we present the DENet, a double-encoder network with feature refinement and region adaption for the terrain segmentation in PolSAR images. First, a double-encoder architecture is proposed to leverage the multitype information of PolSAR images, which can provide more discriminative features than the previous methods using the single-type feature. Second, considering that the polarization information has strong consistency over the category-identical regions, a polarization-guided refinement module is proposed to maintain the feature consistency in the PolSAR segmentation model. This design alleviates the phenomenon of incomplete and fragmented segmentation results. Third, in view of the rich targets’ characteristics in the scattering information, a region-adaptive convolution module is developed to facilitate the scattering information collected over the geometry-irregular regions. This design can improve the segmentation accuracy on the geometry-irregular regions. Extensive experiments are conducted on six PolSAR images to verify the effectiveness of the DENet. Compared with the previous works, our method achieves competitive performance.

Deep convolutional neural networks (CNNs) have brought a tremendous increase in detection accuracy, but too cumbersome model makes them hard to deploy on low computation edge devices, such as satellites and unmanned aerial vehicles. A promising method to tackle this problem is knowledge distillation (KD), which makes models lightweight with satisfactory accuracy. For remote sensing images, the objects are usually environment-related and located in a cluttered scene. The features that objects’ semantic information relies on are tangled. However, existing distillation methods only imitate feature distribution derived from regions, including objects resulting in poor performance. Furthermore, masses of instances generated by teachers are blindly inherited, even if some of them are outliers. In this article, we propose a general and effective KD framework called dynamic knowledge distillation (DKD). First, our framework leverages the dynamic global distillation (GD) module to discover valuable regions from the foreground and background for multiscale features imitation, avoiding ignoring the potential geographical spatial relationship. Second, we propose a dynamic instance selection distillation (ISD) module to give students the ability of self-judgment through the magnitude of detection loss. Third, toward more accurate handling of hard samples in regression, a training-status-aware loss is tailored to guide students mine knowledge about objects with large aspect ratio or small size. Extensive experiments are conducted to show the effectiveness of DKD framework. The detection results on DOTA and NWPU VHR-10 dataset illustrate that our method is suitable for single-stage, two-stage and even anchor-free detectors. It shows the state-of-the-art performance. The code will be publicly available.

The encoder–decoder framework is prevalent in existing remote-sensing image captioning (RSIC) models. The appearance of attention mechanisms brings significant results. However, current attention-based caption models only build up the relationships between the local features without introducing the global visual feature and removing redundant feature components. It will cause caption models to generate descriptive sentences that are weakly related to the scene of images. To solve the problems, this article proposed a global visual feature-guided attention (GVFGA) mechanism. First, GVFGA introduces the global visual feature and fuses them with local visual features to build up their relationships between them. Second, an attention gate utilizing the global visual feature is proposed in GVFGA to filter out redundant feature components in the fused image features and provide more salient image features. In addition, to relieve the hidden state’s burden, a linguistic state (LS) is proposed to specifically provide textual features, making the hidden state only guiding visual–textual attention process. What’s more, to further refine the fusion of visual features and textual features, a LS-Guided Attention (LSGA) mechanism is proposed. It can also filter out the irrelevant information in the fused visual–textual feature with the help of an attention gate. The experimental results show that this proposed image captioning model can achieve better results on three RSIC datasets, UCM-Captions, Sydney-Captions, and RSICD datasets.

In recent years, image-level weakly supervised semantic segmentation (WSSS) has developed rapidly in natural scenes due to the easy availability of classification tags. However, limited to complex backgrounds, multi-category scenes, and dense small targets in remote sensing (RS) images, relatively little research has been conducted in this field. To alleviate the impact of the above problems in RS scenes, a self-supervised Siamese network based on an explicit pixel-level constraints framework is proposed, which greatly improves the quality of class activation maps and the positioning accuracy in multi-category RS scenes. Specifically, there are three novel devices in this paper to promote performance to a new level: (a) A pixel-soft classification loss is proposed, which realizes explicit constraints on pixels during the image-level training; (b) A pixel global awareness module, which captures high-level semantic context and low-level pixel spatial information, is constructed to improve the consistency and accuracy of RS object segmentation; (c) A dynamic multi-scale fusion module with a gating mechanism is devised, which enhances feature representation and improves the positioning accuracy of RS objects, particularly on small and dense objects. Experiments on two RS challenge datasets demonstrate that these proposed modules achieve new state-of-the-art results by only using image-level labels, which improve mIoU to 36.79% on iSAID and 45.43% on ISPRS in the WSSS task. To the best of our knowledge, this is the first work to perform image-level WSSS on multi-class RS scenes.

With the development of deep convolutional neural networks, detecting rotating objects in remote-sensing images is of great significance in various fields. Existing rotating object detectors most suffer the problem of ambiguous supervision caused by inappropriate rotating object representations. This problem may result in fuzzy object localization and further lead to misclassification. In this article, we propose an Automatic Organized Points Detector (AOPDet), which derives precise localization results by applying a novel rotating object representation called nonsequential corners representation. To achieve the proposed representation, an Automatic Organization Mechanism (AOM) technique is designed to guide the model to organize points to object corners automatically. An Automatic-Organized-Points-specific (AOP-specific) head structure is also designed and equipped in the model to better focus on the rotating object detection task. On public aerial datasets, experiments show that the AOPDet achieves 17.0 mAP higher than the compared baseline model, reaching the state-of-the-art (SOTA) level. Detailed ablation experiments and error analysis strongly reveal the effectiveness of the proposed model.

Due to the characteristics of vertical orthophoto imaging, the apparent structural features of the object in the remote sensing image are relatively stable, such as the cross-shaped structure of the aircraft, the rectangular structure of the vehicle, etc. Compared with the traditional visual features, using these features is conducive to improving the accuracy of object detection. However, there are few studies on such characteristics. In this paper, we systematically study the invariant structural features of remote sensing objects and propose a Graph Focusing Aggregation Network (GFA-Net) to represent the structural features of remote sensing objects. Among them, in view of the problem that traditional convolutional neural networks (CNNs) are sensitive to the changes in rotation, scale, and other factors, which makes it difficult to extract structural features, we propose the Graph Focusing Process (GFP) based on the idea of graph convolution. Analysis and experiments show that graph structure has significant advantages over Euclidean feature space under CNN in expressing such structural features. In order to realize the end-to-end efficient training of the above model, we design Graph Aggregation Network (GAN) to update the weight of nodes. We verify the effectiveness of our method on the proposed multi-task datasets ACSD and large-scale fine-grained remote sensing dataset FAIR1M. Experiments conducted on the object detection data sets of DOTA and HRSC2016 prove that the proposed method is superior to the current state-of-the-art method.

The development of social media enables fake news to be expressed in a multi-modal form, which is disseminated on various social platforms and brings harmful social impacts. To handle this challenge, the fake news detection task was proposed to examine whether false information is contained in multi-modal news. Existing methods exploit various approaches with cross-modal interaction and fusion, which have proven to be effective in detecting common fake news. However, although the description of multi-modal news is narrated around entities, the previously developed methods pay less attention to this characteristic. They do not explore its benefits to the detection task and underperform with respect to the detection of fake news that requires entity-centric comparisons. To make up for this omission, we explore a novel paradigm to detect fake news by aligning and fusing multi-modal entities and propose the Entity-oriented Multi-modal Alignment and Fusion network (EMAF). Our work adopts entity-centric cross-modal interaction, which can reserve semantic integrity and capture the details of multi-modal entities. Specifically, we design an Alignment module with the improved dynamic routing algorithm and introduce a Fusion module based on the comparison, the former aligns and captures the important entities and the latter compares and aggregates entity-centric features. Comparative experiments conducted on multiple public datasets, including Weibo, Twitter, and Reddit, reveal the superiority of the proposed EMAF method, and extensive analytical experiments demonstrate the effectiveness of our proposed modules.

Kaiwen Wei, Xian Sun, Zequn Zhang, Jingyuan Zhang, Guo Zhi, Li Jin. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Object detection in aerial images is important for a wide range of applications. The most challenging dilemma in this task is the arbitrary orientation of objects, and many deep-learning-based methods are proposed to address this issue. In previous works on oriented object detection, the regression-based method for object localization has limited performance due to the shortage of spatial information. And the models suffer from the divergence of feature construction for object recognition and localization. In this article, we propose a novel architecture, i.e., point-based estimator to remedy these problems. To utilize the spatial information explicitly, the detector encodes an oriented object with a point-based representation and operates a fully convolutional network for point localization. To improve localization accuracy, the detector takes the manner of coarse-to-fine to lessen the quantization error in point localization. To avoid the discrepancy of feature construction, the detector decouples localization and recognition with individual pathways. In the pathway of object recognition, the instance-alignment block is involved to ensure the alignment between the feature map and oriented region. Overall, the point-based estimator can be easily embedded into the region-based detector and leads to significant improvement on oriented object detection. Extensive experiments have demonstrated the effectiveness of our point-based estimator. Compared with existing works, our method shows state-of-the-art performance on oriented object detection in aerial images.

Recently, deep-learning methods have been successfully applied to the ship detection in the synthetic aperture radar (SAR) images. It is still a great challenge to detect multiscale SAR ships due to the broad diversity of the scales and the strong interference of the inshore background. Most prevalent approaches are based on the anchor mechanism that uses the predefined anchors to search the possible regions containing objects. However, the anchor settings have a great impact on their detection performance as well as the generalization ability. Furthermore, considering the sparsity of the ships, most anchors are redundant and will lead to the computation increase. In this article, a novel detection method named feature balancing and refinement network (FBR-Net) is proposed. First, our method eliminates the effect of anchors by adopting a general anchor-free strategy that directly learns the encoded bounding boxes. Second, we leverage the proposed attention-guided balanced pyramid to balance semantically the multiple features across different levels. It can help the detector learn more information about the small-scale ships in complex scenes. Third, considering the SAR imaging mechanism, the interference near the ship boundary with the similar scattering power probably affects the localization accuracy because of feature misalignment. To tackle the localization issue, a feature-refinement module is proposed to refine the object features and guide the semantic enhancement. Finally, extensive experiments are conducted to show the effectiveness of our FBR-Net compared with the general anchor-free baseline. The detection results on the SAR ship detection dataset (SSDD) and AIR-SARShip-1.0 dataset illustrate that our method achieves the state-of-the-art performance.

The detection of garbage dumps is of great significance for environmental protection. Recently, deep learning algorithms have brought impressive improvements for regular object detection. Different from conventional objects, garbage dumps are more inconspicuous and irregular and have the problem of blurred boundaries. To solve these problems, we propose a shape robust anchor-free network (SRAF-Net) that consists of feature extraction, multitask detection, and postprocessing. First, our network leverages the context-based deformable (CBD) module to combine context attention and deformable convolution. The contextual information obtained by context attention enables the network to focus on objects with inconspicuous appearance, while the deformable convolution enhances the feature representation. Then, we propose a multitask detection head to regress irregular garbage dumps in a more accurate and efficient way. The anchor-based methods need to define some anchors with a fixed shape. However, our detection method is anchor-free that learns the shapes of objects from training data. The detection head adaptively generates various shapes of bounding boxes with their classification confidences and localization confidences. Weighted by the localization confidences, we merge bounding boxes during postprocessing, which alleviates the blurred boundaries. In addition, we build a new public data set named garbage dumps data set (GDD) to verify the effectiveness of our method. Extensive experiments on GDD indicate that our method surpasses the existing detection methods in terms of speed and accuracy for the garbage dumps detection task.

The balance between high accuracy and high speed has always been a challenging task in semantic image segmentation. Compact segmentation networks are more widely used in the case of limited resources, while their performances are constrained. In this paper, motivated by the residual learning and global aggregation, we propose a simple yet general and effective knowledge distillation framework called double similarity distillation (DSD) to improve the classification accuracy of all existing compact networks by capturing the similarity knowledge in pixel and category dimensions, respectively. Specifically, we propose a pixel-wise similarity distillation (PSD) module that utilizes residual attention maps to capture more detailed spatial dependencies across multiple layers. Compared with exiting methods, the PSD module greatly reduces the amount of calculation and is easy to expand. Furthermore, considering the differences in characteristics between semantic segmentation task and other computer vision tasks, we propose a category-wise similarity distillation (CSD) module, which can help the compact segmentation network strengthen the global category correlation by constructing the correlation matrix. Combining these two modules, DSD framework has no extra parameters and only a minimal increase in FLOPs. Extensive experiments on four challenging datasets, including Cityscapes, CamVid, ADE20K, and Pascal VOC 2012, show that DSD outperforms current state-of-the-art methods, proving its effectiveness and generality. The code and models will be publicly available.

Feature pyramid is the mainstream method for multi-scale object detection. In most detectors with feature pyramid, each proposal is predicted based on feature grids pooled from only one feature level, which is assigned heuristically. Recent studies report that the feature representation extracted using this method is sub-optimal, since they ignore the valid information exists on other unselected layers of the feature pyramid. To address this issue, researchers present to fuse valid information across all feature levels. However, these methods can be further improved: the feature fusion strategies, which use common operation (element-wise max or sum) in most detectors, should be replaced by a more flexible way. In this work, a novel method called feature adaptive selection subnetwork (FAS-Net) is proposed to construct effective features for detecting objects of different scales. Particularly, its adaption consists of two level: global attention and local adaptive selection. First, we model the global context of each feature map with global attention based feature selection module (GAFSM), which can strengthen the effective features across each layer adaptively. Then we extract the features of each region of interest (RoI) on the entire feature pyramid to construct a RoI feature pyramid. Finally, the RoI feature pyramid is sent to the feature adaptive selection module (FASM) to integrate the strengthened features according to the input adaptively. Our FAS-Net can be easily extended to other two-stage object detectors with feature pyramid, and supports to analyze the importance of different feature levels for multi-scale objects quantitatively. Besides, FAS-Net can also be further applied to instance segmentation task and get consistent improvements. Experiments on PASCAL07/12 and MSCOCO17 demonstrate the effectiveness and generalization of the proposed method.

Multimodal summarization for open-domain videos is an emerging task, aiming to generate a summary from multisource information (video, audio, transcript).Despite the success of recent multiencoder-decoder frameworks on this task, existing methods lack finegrained multimodality interactions of multisource inputs.Besides, unlike other multimodal tasks, this task has longer multimodal sequences with more redundancy and noise.To address these two issues, we propose a multistage fusion network with the fusion forget gate module, which builds upon this approach by modeling fine-grained interactions between the multisource modalities through a multistep fusion schema and controlling the flow of redundant information between multimodal long sequences via a forgetting module.Experimental results on the How2 dataset show that our proposed model achieves a new state-of-the-art performance.Comprehensive analysis empirically verifies the effectiveness of our fusion schema and forgetting module on multiple encoder-decoder architectures.Specially, when using high noise ASR transcripts (W ER>30%), our model still achieves performance close to the ground-truth transcript model, which reduces manual annotation cost.

It is widely shared that capturing relationships among multi-modality features would be helpful for representing and ultimately describing an image. In this paper, we present a novel Intra- and Inter-modality visual Relation Transformer to improve connections among visual features, termed I2RT. Firstly, we propose Relation Enhanced Transformer Block (RETB) for image feature learning, which strengthens intra-modality visual relations among objects. Moreover, to bridge the gap between inter-modality feature representations, we align them explicitly via Visual Guided Alignment (VGA) module. Finally, an end-to-end formulation is adopted to train the whole model jointly. Experiments on the MS-COCO dataset show the effectiveness of our model, leading to improvements on all commonly used metrics on the "Karpathy" test split. Extensive ablation experiments are conducted for the comprehensive analysis of the proposed method.

Considering the lack of labeled training data sets for the supervised change detection task, in this letter, we try to relieve this problem by proposing a convolutional neural network (CNN)-based change detection method with a newly designed loss function to achieve transfer learning among different data sets. To reach this goal, we first pretrain a U-Net model on an open source data set by taking advantages of the relatively sufficient training data used for the supervised semantic segmentation task. Then, we minimize a skillfully designed loss function to combine the high-level features extracted from the pretrained model and the semantic information contained in the change detection data set, by which a transfer learning is achieved. Third, we compute the distance between the feature vectors obtained from the above step and produce a difference map. Finally, a simple clustering method used on the difference map can even obtain satisfied change map. Experiments carried out on typical optical aerial image data sets validate that the proposed approach compares favorably to the state-of-the-art unsupervised methods.

Recently, the deep convolutional neural network has brought great improvements in object detection. However, the balance between high accuracy and high speed has always been a challenging task in multiclass object detection for large-scale remote sensing imagery. One-stage methods are more widely used because of their high efficiency but are limited by their performances on small object detection. In this article, we propose a unified framework called feature-merged single-shot detection (FMSSD) network, which aggregates the context information both in multiple scales and the same scale feature maps. First, our network leverages the atrous spatial feature pyramid (ASFP) module to fuse the context information in multiscale features by using feature pyramid and multiple atrous rates. Second, we propose a novel area-weighted loss function to pay more attention to small objects, while the replaced original loss treats all objects equally. We believe that small objects should be given more weight than large objects because they lose more information during training. Specifically, a monotonic decreasing function about the area is designed to add weights on the loss function. Extensive experiments on the DOTA data set and NWPU VHR-10 data set demonstrate that our method achieves state-of-the-art detection accuracy with high efficiency. We also build a new large-scale data set called AIR-OBJ data set from Google Earth and show the detection results of small objects, which validates the effectiveness on large-scale remote sensing imagery.

Object detection has been a building block in computer vision. Though considerable progress has been made, there still exist challenges for objects with small size, arbitrary direction, and dense distribution. Apart from natural images, such issues are especially pronounced for aerial images of great importance. This paper presents a novel multi-category rotation detector for small, cluttered and rotated objects, namely SCRDet. Specifically, a sampling fusion network is devised which fuses multi-layer feature with effective anchor sampling, to improve the sensitivity to small objects. Meanwhile, the supervised pixel attention network and the channel attention network are jointly explored for small and cluttered object detection by suppressing the noise and highlighting the objects feature. For more accurate rotation estimation, the IoU constant factor is added to the smooth L1 loss to address the boundary problem for the rotating bounding box. Extensive experiments on two remote sensing public datasets DOTA, NWPU VHR-10 as well as natural image datasets COCO, VOC2007 and scene text data ICDAR2015 show the state-of-the-art performance of our detector. The code and models will be available at https://github.com/DetectionTeamUCAS.

Building reconstruction from SAR images is a hot topic in recent years. Currently, related methods mainly deal with on flat-roofed buildings. In this paper, we extend the research scope to gable-roofed buildings, and try to present a parameterized geometrical model for the layover of gable-roofed buildings. Based on this model, a top-down building reconstruction technique based on MCMC method is proposed. Through representing the layover with parameterized geometrical models, building reconstruction is converted into an optimization problem under the Bayesian scheme. In order to obtain global optima, simulated annealing algorithm with MCMC is used in the optimization stage. Two groups of transmission kernels which are responsible for model updates are designed according to the model. Experiments show that the layover model is accurate and the reconstruction method is effective.

Semantic labeling in high-resolution aerial images is important for its wide range of applications. In this paper, we propose an end-to-end dual multi-scale feature fusion network (DMFFNet) for high-resolution aerial multi-source images. DMFFNet aims to further improve the semantic labeling results of the region where the multispectral features are indistinguishable. Specifically, we design a channel fusion strengthen (CFS) module, which can fuse features adaptively by modelling interdependencies between channels. Furthermore, a multiscale context aggregation (MCA) module is utilized to obtain larger receptive field and more contextual information. The experiment results confirm the DMFFNet with CFS and MCA improve the semantic labeling performance by utilizing multi-source data.

In this paper, we achieve a semantic segmentation of aerial imagery based on the fusion of multi-modal data in an effective way. The multi-modal data contains a true orthophoto and the corresponding normalized Digital Surface Model (nDSM), which are stacked together before they are fed into a Convolutional Neural Network (CNN). Though the two modalities are fused at the early stage, their features are learned independently with group convolutions firstly and then the learned features of different modalities are fused at multiple scales with standard convolutions. Therefore, the multi-scale fusion of multi-modal features is completed in a single-branch convolutional network. In this way, the computational cost is reduced while the experimental results reveal that we can still get promising results.

The performance of object instance segmentation in remote sensing images has been greatly improved through the introduction of many landmark frameworks based on convolutional neural network. However, the object densely issue still affects the accuracy of such segmentation frameworks. Objects of the same class are easily confused, which is most likely due to the close docking between objects. We think context information is critical to address this issue. So, we propose a novel framework called SLCMASK-Net, in which a sequence local context module (SLC) is introduced to avoid confusion between objects of the same class. The SLC module applies a sequence of dilation convolution blocks to progressively learn multi-scale context information in the mask branch. Besides, we try to add SLC module to different locations in our framework and experiment with the effect of different parameter settings. Comparative experiments are conducted on remote sensing images acquired by QuickBird with a resolution of 0.5m - 1m and the results show that the proposed method achieves state-of-the-art performance.

In this paper, we present a semi-supervised remote sensing change detection method based on graph model with Generative Adversarial Networks (GANs). Firstly, the multi-temporal remote sensing change detection problem is converted as a problem of semi-supervised learning on graph where a majority of unlabeled nodes and a few labeled nodes are contained. Then, GANs are adopted to generate samples in a competitive manner and help improve the classification accuracy. Finally, a binary change map is produced by classifying the unlabeled nodes to a certain class with the help of both the labeled nodes and the unlabeled nodes on graph. Experimental results carried on several very high resolution remote sensing image data sets demonstrate the effectiveness of our method.

Object detection has been playing a significant role in the field of remote sensing for a long period while it is still full of challenges. The biggest one is how to detect multi-scale objects with high accuracy and fast speed in remote sensing images. One-stage object detectors have been achieving relatively high accuracy and efficiency with small memory footprint. However, they have a not very well performance on small objects. In this paper, we discuss the importance of the context information between feature maps in different scales which is helpful for detecting small objects. Especially, we propose a Feature-merged detection networks (MergeNet), which can be inserted into the one-stage detectors easily, to unify the multi-scale feature and context information effectively. Experiments on DOTA dataset demonstrate that our model can significantly improve the performance of the one-stage method.

Deep learning based object detection has achieved great success. However, these supervised learning methods are data-hungry and time-consuming. This restriction makes them unsuitable for limited data and urgent tasks, especially in the applications of remote sensing. Inspired by the ability of humans to quickly learn new visual concepts from very few examples, we propose a training-free, one-shot geospatial object detection framework for remote sensing images. It consists of (1) a feature extractor with remote sensing domain knowledge, (2) a multi-level feature fusion method, (3) a novel similarity metric method, and (4) a 2-stage object detection pipeline. Experiments on sewage treatment plant and airport detections show that proposed method has achieved a certain effect. Our method can serve as a baseline for training-free, one-shot geospatial object detection.

Image collection summarisation aims to represent a large‐scale multi‐modal collection with a small subset of images and tags, helping navigate a large image dataset. Most extant methods leverage the contributions of text‐to‐visual summaries, ignoring the visual contribution to the textual topic. When the tags are weakly labelled, the textual topic cannot accurately reflect the visual summary. To solve this, the authors propose a novel model, joint optimisation of convex non‐negative matrix factorisation, which incorporates images and tags in a beneficial way. The objective function contains visual and textual error functions, sharing the same indicator matrix, connecting different modal relations. Then, they propose an iterative algorithm to optimise the proposed model. Finally, they explore the effects of different visual feature representations (e.g. bag‐of‐words and deep learning) on multi‐modal collection summary. Our proposed method is then compared with state‐of‐the‐art algorithms using two multi‐modal datasets (i.e. MIRFlickr and NUS‐WIDE‐SCENE). Experimental results demonstrate the effectiveness of their proposed approach.

Semantic segmentation in high-resolution aerial images is a fundamental research problem in remote sensing field for its wide range of applications. However, it is difficult to distinguish regions with similar spectral features using only multispectral data. Recent research studies have indicated that the introduction of multisource information can effectively improve the robustness of segmentation method. In this letter, we use digital surface models (DSMs) information as a complementary feature to further improve the semantic segmentation results. To this end, we propose a lightweight and simple DSM fusion (DSMF) branch structure module. Compared with the existing feature extraction structures, proposed DSMF module is simple and can be easily applied to other networks. In addition, we investigate four fusion strategies based on DSMF module to explore the optimal feature fusion strategy and four end-to-end DSMFNets are designed according to the corresponding strategies. We evaluate our models on International Society for Photogrammetry and Remote Sensing Vaihingen data set and all DSMFNets achieve promising results. In particular, DSMFNet-1 achieves an overall accuracy of 91.5% on the test data set.

An algorithm based on optical flow is proposed to detect a ground moving target via the single-channel synthetic aperture radar. First, the signal models of uniform moving targets are established and classified into three types. Next, the Doppler spectrum is divided to generate a multilook image sequence. Then, the motion feature of a moving target response is described in the image sequence, in which the optical flow is introduced to realize the moving target detection. The detection results of real moving targets are obtained after the false alarm elimination based on the response motion relevance. This algorithm has a large range of detectable velocity and can even be applied to detect the moving targets with acceleration. In addition, compared with constant false alarm rate method, the optical flow has a better anti-interference performance against the strong static scatters. Finally, some numerical experiments are provided to demonstrate the effectiveness of the proposed method.

This letter presents a novel supervised change detection method based on a deep siamese semantic network framework, which is trained by using improved triplet loss function for optical aerial images. The proposed framework can not only extract features directly from image pairs which include multiscale information and are more abstract as well as robust, but also enhance the interclass separability and the intraclass inseparability by learning semantic relation. The feature vectors of the pixels pair with the same label are closer, and at the same time, the feature vectors of the pixels with different labels are farther from each other. Moreover, we use the distance of the feature map to detect the changes on the difference map between the image pair. Binarized change map can be obtained by a simple threshold. Experiments on optical aerial image data set validate that the proposed approach produces comparable, even better results, favorably to the state-of-the-art methods in terms of F-measure.

Object detection has been playing a significant role in the field of remote sensing for a long time but it is still full of challenges. In this paper, we propose a novel detection framework based on rotational region convolution neural network to cope with the problem of non-maximum suppression in dense objects detection. The bounding boxes obtained by adopting our method is the minimum bounding rectangle of object with less redundant regions. Furthermore, we find the head direction of the object through prediction. There are three important changes to our framework over traditional detection methods, representation and regression of rotational bounding box, head direction prediction and rotational non-maximal suppression. Experiments based on remote sensing images from Google Earth for Object detection show that our detection method based on rotational region CNN has a competitive performance.

Deeper architectures are proven to be beneficial for the classification performance obviously in computer vision field. Inspired by this, deep CNN s are expected to make progress in the SAR target classification problem as well. However, it is hard to train deeper CNNs for SAR images. Such CNNs have millions of parameters to be determined in the network (for example the VGGNet has more than 130 million parameters), hence large-scale dataset is indispensable when training a deep CNN. But there is no large-scale annotated SAR target dataset, and data acquisition and annotation is much more costly for SAR images. With inadequate data, the network is easy to be overfitting. Several methods based on deep learning have been proposed for SAR image classifications, but they cannot get rid of the aforementioned data limitation of labelled SAR images. To solve this problem, this paper proposes a microarchitecture called CompressUnit (CU). With CU, we design a deeper CNN. Compared with the network with the fewest parameters for SAR image classification in literature so far, our network is 2X deeper with only about 10% of parameters. In this way, we get a deeper network with much fewer parameters. This network is easier to be trained with limited SAR data and is more likely to get rid of overfitting.

In this paper, we focus on the use of multi-modal data to achieve a semantic segmentation of aerial imagery. Thereby, the multi-modal data is composed of a true orthophoto, the Digital Surface Model (DSM) and further representations derived from these. Taking data of different modalities separately and in combination as input to a Residual Shuffling Convolutional Neural Network (RSCNN), we analyze their value for the classification task given with a benchmark dataset. The derived results reveal an improvement if different types of geometric features extracted from the DSM are used in addition to the true orthophoto.

In this paper, a novel framework for aircraft detection in high resolution apron area in Synthetic Aperture Radar (SAR) images is proposed, which combines the strength of location regression based convolutional neural network (CNN) framework and the salient features of target in SAR images. Specifically, a Constant False Alarm Rate (CFAR) based target pre-locating algorithm is introduced, which can match the scale of target in SAR images more accurate compared to the existing region proposal method. In addition, in order to eliminate the fact of overfitting, we explore several strategies for SAR data augmentation, including translation, adding noise and rotation within a small range. Experiments are conducted on the data set acquired by the TerraSAR-X satellite in a resolution of 3.0 meters. The results show that the proposed detection framework could effectively obtain a more accurate detection result.

Road extraction from high-resolution remote sensing images has been applied in many domains, but it is still full of challenges. We focus on the problem of slender roads, proposing a new multiple feature pyramid network (MFPN), which is composed of an effective feature pyramid and the tailored pyramid pooling module based on PSPNet. These two designs can address the sparsity of roads in remote sensing images via using multi-level semantic features. Experiments on remote sensing images from Quick Bird show that our MFPN model achieves competitive performance, especially for slender roads.

Through the recent performance of convolutional neural networks in image processing tasks, we propose a deep fully convolutional network for remote sensing image inpainting. The proposed Dense-Add Net (Dense-Add Network) can alleviate the vanishing-gradient problem, strengthen feature reuse, and substantially reduce the memory usage. We apply residual learning to learn the mappings from corrupted image to recovered image directly; it will back-propagate gradient to the bottom layers and accelerate the training process. We train the proposed Dense-Add Net with a robust Charbonnier loss function which can achieve high-quality reconstruction. The experimental verify the efficacy of our proposed Dense-Add Net.

Single image super-resolution is an effective way to enhance the spatial resolution of remote sensing image, which is crucial for many applications such as target detection and image classification. However, existing methods based on the neural network usually have small receptive fields and ignore the image detail. We propose a novel method named deep memory connected network (DMCN) based on a convolutional neural network to reconstruct high-quality super-resolution images. We build local and global memory connections to combine image detail with environmental information. To further reduce parameters and ease time-consuming, we propose downsampling units, shrinking the spatial size of feature maps. We test DMCN on three remote sensing datasets with different spatial resolution. Experimental results indicate that our method yields promising improvements in both accuracy and visual performance over the current state-of-the-art.

Semantic segmentation of aerial images refers to assigning one land cover category to each pixel. This is a challenging task due to the great differences in the appearances of ground objects. Many attempts have been made during the past decades. In recent years, convolutional neural networks (CNNs) have been introduced in the remote sensing field, and various solutions have been proposed to realize dense semantic labeling with CNNs. In this letter, we propose shuffling CNNs to realize semantic segmentation of aerial images in a periodic shuffling manner. This approach is a supplement to current methods for semantic segmentation of aerial images. We propose a naive version and a deeper version of this method, and both are adept at detecting small objects. Additionally, we propose a method called field-of-view (FoV) enhancement that can enhance the predictions. This method can be applied to various networks, and our experiments verify its effectiveness. The final results are further improved through an ensemble method that averages the score maps generated by the models at different checkpoints of the same network. We evaluate our models using the ISPRS Vaihingen and Potsdam data sets, and we acquire promising results using these two data sets.

Water-body segmentation is an important issue in remote sensing and image interpretation. Classic methods for counteracting this problem usually include the construction of index features by combining different spectra, however, these methods are essentially rule-based and fail to take advantage of context information. Additionally, as the quality of image resolution improves, these methods are proved to be inadequate. With the rise of convolutional neural networks (CNN), the level of research about segmentation has taken a huge leap, but the field is still facing an increasing demand for data and the problem of blurring boundaries. In this letter, a new segmentation network called restricted receptive field deconvolution network (RRF DeconvNet) is proposed, with which to extract water bodies from high-resolution remote sensing images. Compared with natural images, remote sensing images have a weaker pixel neighborhood relativity; in consideration of this challenge, an RRF DeconvNet compresses the redundant layers in the original DeconvNet and no longer relies on a pretrained model. In addition, to tackle the blurring boundaries that occur in CNN, a new loss function called edges weighting loss is proposed to train segmentation networks, which has been shown to significantly sharpen the segmentation boundaries in results. Experiments, based on Google Earth images for water-body segmentation, are presented in this letter to prove our method.

Cloud and cloud shadow detection in remote sensing imagery is important for its wide range of applications. Traditionally, the detection is usually based on the manually designed thresholds from multiband, which is complicated and of multistage. To simplify the process of cloud and cloud shadow detection and improve the performance, we propose a multilevel feature fused segmentation network (MFFSNet), which can be trained end-to-end without any hand-tuned parameters. Specifically, a fully convolutional network is proposed for cloud and cloud shadow features learning. Then, we utilize a novel pyramid pooling module to extract contextual relation between cloud and shadow. Furthermore, a special multilevel feature fused structure is designed to combine semantic information with spatial information from different levels, so that we can better handle the multiscale objects and produce detailed segmentation boundaries. Experiments show that the MFFSNet outperforms the state-of-the-art methods and achieves high accuracies of 98.69% and 98.92% for cloud and cloud shadow detection.

Aircraft type recognition in remote sensing images is a meaningful task. It remains challenging due to the difficulty of obtaining appropriate representation of aircrafts for recognition. To solve this problem, we propose a novel aircraft type recognition framework based on deep convolutional neural networks. First, an aircraft segmentation network is designed to obtain refined aircraft segmentation results which provide significant details to distinguish different aircrafts. Then, a keypoints' detection network is proposed to acquire aircrafts' directions and bounding boxes, which are used to align the segmentation results. A new multirotation refinement method is carefully designed to further improve the keypoints' precision. At last, we apply a template matching method to identify aircrafts, and the intersection over union is adopted to evaluate the similarity between segmentation results and templates. The proposed framework takes advantage of both shape and scale information of aircrafts for recognition. Experiments show that the proposed method outperforms the state-of-the-art methods and can achieve 95.6% accuracy on the challenging data set.

Recently, convolutional neural networks (CNN) have achieved impressive breakthroughs in single image superresolution. In particular, an efficient nonlinear mapping by increasing the depth and width of the network can be learned between the low-resolution input image and the high-resolution target image. However, this will lead to a substantial increase in network parameters, requiring the massive amount of training data to prevent overfitting. Besides, most CNN-based methods ignore the full use of different levels of features and, therefore, achieve relatively low performance. In this letter, we propose a deep convolutional network named densely connected residual networks (DRNet). Our proposed DRNet can reach very deep and wide while requiring fewer parameters. The significant performance improvement of our model is mainly due to the integration of dense skip connection and residual learning. In this way, DRNet mitigates the problems of overfitting, vanishing gradient, and training instability during training very deep and wide networks. Moreover, it can improve the propagation and reuse of features by creating direct connections from the previous layers to the subsequent layers. We evaluate the proposed method using images from four benchmark datasets and set a new state of the art.

In this paper, we propose a top-down building reconstruction technique based on layover modelling and MCMC method. Through representing the layover with parameterized geometrical models, the problem is converted into an optimization problem under the Bayesian scheme. The energy function consists of two parts: region part and edge part. In order to obtain global optima, simulated annealing algorithm with MCMC is used in the optimization stage. Two groups of transmission kernels which are responsible for model updates are designed according to the model. This method is tested both on simulated SAR image and HR TanDEM-X data. At this moment, only qualitative analysis for this method is provided. It proves the effectiveness of the presented method. Detailed quantitative evaluation will be added when we submit the final version of this paper.

Building extraction from remote sensing images is a longstanding topic in land use analysis and applications of remote sensing. Variations in shape and appearance of buildings, occlusions and other unpredictable factors increase the hardness of automatic building extraction. Numerous methods have been proposed during the last several decays, but most of these works are task oriented and lack of generalization. This paper applys deep learning to building extraction in a supervised manner. A deep deconvolution neural network with 27 Convolution/Deconvolution weight layers is designed to realize building extraction in pixel level. As such a deep network is prone to overfitting, a data augment method that suits pixel-wise prediction tasks in remote sensing is suggested. Moreover, an overall training and inferencing architecture is proposed. Our methods are finally applied to building extraction tasks and get competitive results with other methods published.

Semantic segmentation for remote sensing images is a critical process in the workflow of object-based image analysis. Recently, convolutional neural networks(CNNs) are powerful visual models that yield hierarchies of features. In this paper, we propose a deep convolutional encoder-decoder model for remote sensing images segmentation. Specifically, we rely on the encoder network to extract the high-level semantic feature of ultra-high resolution images and the decoder network is employed to map the low resolution encoder feature maps to full input resolution feature maps for pixel-wise labeling. Also the fully connected conditional random field (CRF) is integrated into the model so that the network can be trained end-to-end. Experiments on the Vaihingen dataset demonstrate that our model can make promising performance.

Automatic inshore ship recognition, which includes target localization and type recognition, is an important and challenging task. However, existing ship recognition methods mainly focus on the classification of ship samples or clips. These methods rely deeply on the detection algorithm to complete localization and recognition in large scene images. In this letter, we present an integrated framework to automatically locate and recognize inshore ships in large scene satellite images. Different from traditional object recognition methods using two steps of detection-classification, the proposed framework could locate inshore ships and identify types without the detection step. Considering ship size is a useful feature, a novel multimodel method is proposed to utilize this feature. And an Euclidean-distance-based fusion strategy is used to combine candidates given by models. This fusion strategy could effectively separate side-by-side ships. To handle large scene images efficiently, scale-invariant feature transform registration is also integrated into the framework to utilize geographic information. All of these make the framework an end-to-end fashion which could automatically recognize inshore ships in large scene satellite images. Experiments on Quickbird images show that this framework could achieve the actual applied requirements.

With the development of deep learning, supervised learning has frequently been adopted to classify remotely sensed images using convolutional networks. However, due to the limited amount of labeled data available, supervised learning is often difficult to carry out. Therefore, we proposed an unsupervised model called multiple-layer feature-matching generative adversarial networks (MARTA GANs) to learn a representation using only unlabeled data. MARTA GANs consists of both a generative model G and a discriminative model D. We treat D as a feature extractor. To fit the complex properties of remote sensing data, we use a fusion layer to merge the mid-level and global features. G can produce numerous images that are similar to the training data; therefore, D can learn better representations of remotely sensed images using the training data provided by G. The classification results on two widely used remote sensing image databases show that the proposed method significantly improves the classification performance compared with other state-of-the-art methods.

Polarimetric synthetic aperture radar (PolSAR) is an advanced imaging radar system, for which the acquired data provide not only the information of each channel but also the correlation between channels. To fully utilize and accurately model the multilook PolSAR data, a novel compound distribution, named the H distribution, is proposed based on the generalized Fisher distribution (GFD). Specifically, the GFD introduces a power parameter to the ordinary Fisher distribution. With one more free parameter, the GFD is flexible and versatile enough to characterize different kinds of texture. Then, by assuming the generalized-Fisher-distributed texture and the Wishart-distributed speckle, the H distribution is derived, whose closed-form expression is obtained with the help of Fox's H-function. As such, the H distribution has a compact form and is conveniently applied to practical problems, such as modeling and classification of PolSAR data. The effectiveness of this method is tested by modeling the multilook PolSAR data and performing image classification. The experimental results demonstrate that the H distribution is a flexible and effective way to model multilook PolSAR data.

In this letter, we propose a novel supervised change detection method based on a deep siamese convolutional network for optical aerial images. We train a siamese convolutional network using the weighted contrastive loss. The novelty of the method is that the siamese network is learned to extract features directly from the image pairs. Compared with hand-crafted features used by the conventional change detection method, the extracted features are more abstract and robust. Furthermore, because of the advantage of the weighted contrastive loss function, the features have a unique property: the feature vectors of the changed pixel pair are far away from each other, while the ones of the unchanged pixel pair are close. Therefore, we use the distance of the feature vectors to detect changes between the image pair. Simple threshold segmentation on the distance map can even obtain good performance. For improvement, we use a k-nearest neighbor approach to update the initial result. Experimental results show that the proposed method produces results comparable, even better, with the two state-of-the-art methods in terms of F-measure.

Detecting artificial targets, such as aircraft, in satellite images is significant in military and civil applications. Although the performance has improved with the use of more complicated features and better learning methods, effectively handling aircraft with variations of type, pose, and size is still very challenging. To solve this problem, we propose a multiscale sliding-window framework based on aggregate channel features, well-designed features that contain rich information. We also employ a fast feature pyramids algorithm to accelerate multiscale aircraft detection. In this framework, features are trained by Cascade AdaBoost including multiple rounds of bootstrapping that leads to improved overall accuracy. A two-step nonmaximum suppression algorithm is carefully designed based on a given set of detections. Our method shows a competitive performance on the QuickBird images of 0.6 m resolution.

Scattering structure features of targets is of great importance for Synthetic Aperture Radar (SAR) image analysis. In this paper, a novel algorithm for aircraft recognition in high resolution apron area of SAR images is proposed. The algorithm combines the strength of gradient saliency map and scattering structure features to improve accuracy and efficiency. Specially, Constant False-Alarm Rate (CFAR) algorithm is carried out to segment images. Then, a new efficient object locating method based on directional local gradient map is proposed to detect aircraft targets. Then, the candidate slices as well as template slices are modeled using Gaussian Mixture Model (GMM), which will be treated as structure features. In the recognition stage, a novel similarity measurement algorithm based on Kullback-Leibler Divergence for GMM models is proposed for classification. We conduct experiments on the dataset with 3.0m resolution and the recognition results demonstrate the accuracy of our proposed method.

In this paper, a fully automatic building reconstruction method for high resolution interferometric synthetic aperture radar (InSAR) data is presented. This method is based on stochastic geometrical model. Firstly, a building detection procedure is implemented on the big image and the entire scene is divided into building clips. After that, the reconstruction process is utilized for each building clip. In the reconstruction process, a building in 3D space is projected to the image plane and then decomposed to feature regions including layover, corner line, roof and shadow. We explore the statistic properties of the each region, and include it in the posterior function, together with the edge term and the prior we defined. Finally, in order to overcome local optima, a group of special transmission kernels are designed. The experimental results on TanDEM-X data demonstrate the effectiveness of our method.

The interferometric coherence map is derived from the cross-correlation of two registered synthetic aperture radar (SAR) images. It can give additional information complementary to the intensity image, or act as an independent information source in many applications. Compared to the plenty of work on SAR intensity statistics, there are quite fewer researches on the statistical characters of interferometric SAR (InSAR) coherence. And to our knowledge, all of the existing work that related to InSAR coherence statistics, models the coherence with Gaussian distribution with no discrimination on data resolutions or scene types. Our main contribution is the investigation on the accuracies of several typical models for high resolution coherence statistics over urban areas. We select three typical land classes including trees, buildings, and shadow, as the representatives of urban areas. And different models including Gaussian, Weibull, Rayleigh, Nakagami and Beta are evaluated. Experiment results on TanDEM-X data illustrate that the Beta model reveals a better performance than other distributions. Finally, the Beta model is used in the detection of buildings.

Object detection has been one of the hottest issues in the field of remote sensing image analysis. In this letter, an efficient object detection framework is proposed, which combines the strength of the unsupervised feature learning of deep belief networks (DBNs) and visual saliency. In particular, we propose an efficient coarse object locating method based on a saliency mechanism. The method could avoid an exhaustive search across the image and generate a small number of bounding boxes, which can locate the object quickly and precisely. After that, the trained DBN is used for feature extraction and classification on subimages. The feature learning of the DBN is operated by pretraining each layer of restricted Boltzmann machines (RBMs) using the general layerwise training algorithm. An unsupervised blockwise pretraining strategy is introduced to train the first layer of RBMs, which combines the raw pixels with a saliency map as inputs. This makes an RBM generate local and edge filters. The precise edge position information and pixel value information are more efficient to build a good model of images. Comparative experiments are conducted on the data set acquired by QuickBird with a 60-cm resolution. The results demonstrate the accuracy and efficiency of our method.

Layover in slant range synthetic aperture radar (SAR) images contains rich 3-D information of building geometry. In this letter, a coarse-to-fine method for building reconstruction from high-resolution (HR) SAR layover map using restricted parametric geometrical models is presented. First, we propose a new restricted parametric geometrical model for building layover and for corner line, respectively. Under the guidance of these models, a hierarchical coarse-to-fine layover estimation scheme is designed. Owning to the coarse-to-fine scheme, this method is resistant to various flaws of layover. At last, the building is reconstructed from the well-estimated layover. Experiments on HR TanDEM-X data demonstrate the effectiveness and precision of our method.

Automatic aircraft recognition is a challenging task. Conventional methods always extract the overall shapes of aircraft at first and then represent the aircraft based on the extracted shape with different features for recognition. The major problem of these methods is that they have a high requirement on shape extraction, which is too idealistic for targets in satellite images. In this letter, we propose a new aircraft recognition approach that can recognize aircraft robustly without perfect extraction of silhouette or shape of aircraft as a precondition, and can deal with the situation of parts missing and shadow disturbance. Specifically, a direction estimation method is proposed first to align aircraft to a same direction. Then, a reconstruction-based similarity measure is proposed, which transforms the type recognition problem into a reconstruction problem. Finally, a jigsaw matching pursuit algorithm is proposed to solve the reconstruction problem. We use panchromatic Quickbird imagery for evaluation, and the experimental results illuminate that the proposed method is effective and accurate.

In this letter, we propose a novel method to reconstruct accurate building structures from high-resolution multiview aerial imagery, using layered contour fitting (LCF) with a density-based clustering algorithm. Initially, the complicated 3-D scene is reconstructed by a probabilistic volumetric modeling algorithm. Subsequently, the reconstructed 3-D scene model is projected into layer images based on the height information. At last, we combine an extended layered density-based clustering approach with a generative LCF approach to remove noise and extract accurate building contours in every layer image at the same time. The final accurate 3-D building model is generated from these contours in layer images with a smoothing operation. Experiments on the aerial image sets demonstrate effectiveness and precision of our method.

Image quality assessment (IQA) aims to devise perceptual models to predict the image quality consistently with human subjective evaluation. The representative metrics focus on measuring the image quality with low-level features. In this letter, we assumed that the distortion in specific regions containing semantically significant objects would be enhanced by HVS significantly. According to this hypothesis, a novel IQA metric based on a commonly used object-detecting feature, Speed Up Robust Features (SURF), was proposed. First, it determined the interest points which represented significant objects through the SURF features both on the reference image and distorted image. Then it computed the multilevel SURF descriptors differences between the reference image and the distorted one. Finally, all the difference results were combined with a suitable pooling strategy. Comparing with other nine state-of-the-art IQA models on three biggest IQA databases, SURF-SIM demonstrated its highly competitive prediction accuracy especially on complicated applications and excellent robustness across different distortion types.

In this letter, we present a new method to detect inshore ships using shape and context information. We first propose a new energy function based on an active contour model to segment water and land and minimize it with an iterative global optimization method. The proposed energy performs well on the different intensity distributions between water and land and produces a result that can be well used in shape and context analyses. In the segmented image, ships are detected with successive shape analysis, including shape analysis in the localization of ship head and region growing in computing the width and length of ship. Finally, to locate ships accurately and remove the false alarms, we unify them with a binary linear programming problem by utilizing the context information. Experiments on QuickBird images show the robustness and precision of our method.

In this letter, we propose a new detection framework based on robust invariant generalized Hough transform (RIGHT) to solve the problem of detecting inshore ships in high-resolution remote sensing imagery. The invariant generalized Hough transform is an effective shape extraction technique, but it is not adaptive to shape deformation well. In order to improve its adaptability, we use an iterative training method to learn a robust shape model automatically. The model could capture the shape variability of the target contained in the training data set, and every point in the model is equipped with an individual weight according to its importance, which greatly reduces the false-positive rate. Through the iteration process, the model performance is gradually improved by extending the shape model with these necessary weighted points. Experimental result demonstrates the precision, robustness, and effectiveness of our detection framework based on RIGHT.

In this letter, we propose a rotation invariant parts-based model to detect objects with complex shape in high-resolution remote sensing images. Specifically, the geospatial objects with complex shape are firstly divided into several main parts, and the structure information among parts is described and regulated in polar coordinates to achieve the rotation invariance on configuration. Meanwhile, the pose variance of each part relative to the object is also defined in our model. In encoding the features of the rotated parts and objects, a new rotation invariant feature is proposed by extending histogram oriented gradients. During the final detection step, a clustering method is introduced to locate the parts in objects, and that method can also be used to fuse the detection results. By this way, an efficient detection model is constructed and the experimental results demonstrate the robustness and precision of our proposed detection model.

For high-resolution remote sensing image classification tasks, multiple features are usually required for better performances since single visual feature is valid only in describing one pattern of images. In this letter, we propose a novel Semi-Supervised Manifold learning based Multigraph Fusion framework (SSM-MF), in which multiple features are combined to learn a low-dimensional subspace. The obtained subspace can effectively characterize the semantic information of the features and thus benefits classification. Our framework employs a semi-supervised manner by exploiting labeled and unlabeled data and therefore enjoy three advancements: 1) discriminative information and geometric information in labeled data and the structural information in unlabeled data can be jointly utilized to enhance manifold learning; 2) our framework explores the complementary of multiple features and meanwhile avoids the curse of dimensionality; and 3) our semi-supervised learning mode makes use of information in abundant unlabeled data in real-world applications. Experiments on a remote sensing image data set validate the effectiveness of our proposed method.

In this letter, we propose a semisupervised context-sensitive technique for change detection in high-resolution multitemporal remote sensing images. This is achieved by analyzing the posterior probability of probabilistic Gaussian process (GP) classifier within a Markov random field (MRF) model. In particular, the method consists of two steps: 1) A semisupervised initialization exploits both labeled and unlabeled data based on a probabilistic GP classifier, and 2) an MRF regularization aims at refining the posterior probability by employing the spatial context information. In particular, both edge information and high-order potential are utilized in MRF energy function formulation. Experimental results obtained on real remote sensing multitemporal imagery data sets confirm the effectiveness of the proposed approach.

Automatic aircraft recognition in high-resolution satellite images has many important applications. Due to the diversity and complexity of fore-/background, recognition using pixel-based methods usually does not perform well. In this letter, we propose a new method integrating the high-level information of a shape prior, which is considered as a coarse-to-fine process. In the coarse stage, the pose of an aircraft is roughly estimated by a single template matching with a defined score criterion. In the fine stage, we derive a parametric shape model by applying principal component analysis and kernel density function, which have good effects on both dimension reduction and sample space description; then, a new variational formulation combining region information and a shape prior is proposed to segment the object using a level set method. Finally, the parameters of the segmentation result are directly applied to verify aircraft type with two$k$-nearest neighbor steps. Experiments on QuickBird images demonstrate the robustness and accuracy of the proposed method.

High-resolution remote-sensing image classification is a challenging task. In this letter, we first propose a bag-of-features (BOF) model-based classification framework for high-resolution remote-sensing images via Earth mover's distance (EMD) to perform histogram matching. Compared with conventional BOF, EMD-based BOF is insensitive to vector quantization and can explore the relations among visual codes. In addition, such relations can be utilized as a key discriminative feature for image classification task. However, EMD is not practically utilized because of expensive computational cost. Motivated by Pele and Werman, we propose a faster approximate EMD (AEMD), and our AEMD-based BOF can inherit the advantages of EMD. Experimental results on a multicategory remote-sensing image data set demonstrate the effectiveness of our classification framework.

In this letter, we propose a novel framework for large-satellite-image annotation using multifeature joint sparse coding (MFJSC) with spatial relation constraint. The MFJSC model imposes anl1, 2-mixed-norm regularization on encoded coefficients of features. The regularization will encourage the coefficients to share a common sparsity pattern, which will preserve the cross-feature information and eliminate the constraint that they must have identical coefficients. Spatial dependences between patches of large images are useful for the annotation task but are usually ignored or insufficiently exploited in other methods. In this letter, we design a spatial-relation-constrained classifier to utilize the output of MFJSC and the spatial dependences to annotate images more precisely. Experiments on a data set of 21 land-use classes and QuickBird images show the discriminative power of MFJSC and the effectiveness of our annotation framework.

In this letter, we propose a contour-based spatial model which can detect geospatial targets accurately in high-resolution remote sensing images. To detect the geospatial targets with complex structures, each image was partitioned into pieces as target candidate regions using multiple segmentations at first. Then, the automatic identification of target seed regions is achieved by computing the similarity of the contour information with the target template using dynamic programming. Finally, the contour-based similarity was further updated and combined with spatial relationships to figure out the missing parts. In this way, a more accurate target detection result can be achieved. The precision, robustness, and effectiveness of the proposed method were demonstrated by the experimental results.

Automatic detection for targets with complex shape in high-resolution remote sensing images is a challenging task. In this letter, we propose a new detection framework based on spatial sparse coding bag-of-words (BOW) (SSCBOW) model to solve this problem. Specifically, after selecting a processing unit by the sliding window and extracting features, a new spatial mapping strategy is used to encode the geometric information, which not only represents the relative position of the parts of a target but also has the ability to handle rotation variations. Moreover, instead ofK-means for visual-word encoding in the traditional BOW model, sparse coding is introduced to achieve a much lower reconstruction error. Finally, the SSCBOW representation is combined with linear support vector machine for target detection. The experimental results demonstrate the precision and robustness of our detection method based on the SSCBOW model.

Target recognition in high-resolution synthetic aperture radar (SAR) images is a challenging task. In this letter, a novel geometrical-based SAR image simulator is proposed to assist target recognition. Specifically, in addition to the using of Lamberitian-specular mixed model for single-bounce simulation, we propose a dihedral corner model for double-bounce simulation, which allows directly retrieving structure information from scattering patterns. In addition, a dihedral tracing technique is taken instead of ray tracing for double-bounce detection to reduce computation time. Also, a new primitive-by-primitive visualization approach which is well combined with our scattering model is proposed to obtain high efficiency. The efficiency of the simulator is demonstrated by the comparison between the simulated results and MiniSAR images and by the application in vehicle recognition.

In this letter, we propose a novel method to solve the problem of detecting geospatial objects present in high-resolution remote sensing images automatically. Each image is represented as a segmentation tree by applying a multiscale segmentation algorithm at first, and all of the tree nodes are described as coherent groups instead of binary classified values. The trees are matched to select the maximally matched subtrees, denoted as common subcategories. Then, we organize these subcategories to learn the embedded taxonomic semantics of objects categories, which allow categories to be defined recursively, and express both explicit and implicit spatial configuration of categories. Detection, recognition, and segmentation of the geospatial objects in a new image can be simultaneously conducted by using the learned taxonomic semantics. This procedure also provides a meaningful explanation for image understanding. Experiments for complex and compound objects demonstrate the precision, robustness, and effectiveness of the proposed method.

Many traditional target extraction methods encountered new challenges as the spatial resolution is increasing quickly. For the purpose of extracting buildings in that circumstance, a new method combing both the object-based approach and boosting algorithm is proposed in this paper. The method associates segmentation with recognition by constructing a hierarchical object network, which effectively improves the problem of detecting targets with a modifiable sliding window existed in other methods. And some useful features are selected automatically to train a validate classifier. Then the label confidence of each object is computed using contextual models to complete the extraction procedure. Competitive results for both multiform and complicated buildings demonstrate the precision, robustness and effectiveness of the proposed method.