The rapid advancement of foundation models has revolutionized visual representation learning in a self-supervised manner. However, their application in remote sensing (RS) remains constrained by a fundamental gap: existing models predominantly handle single or limited modalities, overlooking the inherently multi-modal nature of RS observations. Optical, synthetic aperture radar (SAR), and multi-spectral data offer complementary insights that significantly reduce the inherent ambiguity and uncertainty in single-source analysis. To bridge this gap, we introduce RingMoE, a unified multi-modal RS foundation model with 14.7 billion parameters, pre-trained on 400 million multi-modal RS images from nine satellites. RingMoE incorporates three key innovations: 1) A hierarchical Mixture-of-Experts (MoE) architecture comprising modal-specialized, collaborative, and shared experts, effectively modeling intra-modal knowledge while capturing cross-modal dependencies to mitigate conflicts between modal representations; 2) Physics-informed self-supervised learning, explicitly embedding sensor-specific radiometric characteristics into the pre-training objectives; 3) Dynamic expert pruning, enabling adaptive model compression from 14.7B to 1B parameters while maintaining performance, facilitating efficient deployment in Earth observation applications. Evaluated across 23 benchmarks spanning six key RS tasks (i.e., classification, detection, segmentation, tracking, change detection, and depth estimation), RingMoE outperforms existing foundation models and sets new SOTAs, demonstrating remarkable adaptability from single-modal to multi-modal scenarios. Beyond theoretical progress, it has been deployed and trialed in multiple sectors, including emergency response, land management, marine sciences, and urban planning.

Remote sensing foundation models largely break away from the traditional paradigm of designing task-specific models, offering greater scalability across multiple tasks. However, they face challenges such as low computational efficiency and limited interpretability, especially when dealing with large-scale remote sensing images. To overcome these, we draw inspiration from heat conduction, a physical process modeling local heat diffusion. Building on this idea, we are the first to explore the potential of using the parallel computing model of heat conduction to simulate the local region correlations in high-resolution remote sensing images, and introduce RS-vHeat, an efficient multi-modal remote sensing foundation model. Specifically, RS-vHeat 1) applies the Heat Conduction Operator (HCO) with a complexity of $O(N^{1.5})$ and a global receptive field, reducing computational overhead while capturing remote sensing object structure information to guide heat diffusion; 2) learns the frequency distribution representations of various scenes through a self-supervised strategy based on frequency domain hierarchical masking and multi-domain reconstruction; 3) significantly improves efficiency and performance over state-of-the-art techniques across 4 tasks and 10 datasets. Compared to attention-based remote sensing foundation models, we reduce memory usage by 84\%, FLOPs by 24\% and improves throughput by 2.7 times. The code will be made publicly available.

In the era of smart cyber-physical grid, dynamic insecurity risk has become a significant concern due to the increasing integration of renewable energy sources and the inherent uncertainties in smart grid. Dynamic security assessment (DSA) has been adopted to hedge against such risks by estimating the stability of large-scale smart grids. Existing DSA approaches often involve complex high dimensional models which incur high communication and computational costs, hindering their practical adoption. In this paper, we address these limitations with the Quantum Federated Ensembled Variational Adaptive Learning (QFEVAL) approach for smart grid DSA. QFEVAL is designed to combine quantum machine learning and federated learning to handle the differential-algebraic equations that describe smart grid stability, providing an efficient way to deal with high-dimensional data and uncertainties. QFEVAL enables the training of the hybrid quantum-classical neural networks on distributed DSA datasets located at different nodes in smart grids, without requiring large numbers of parameters to be transmitted. QFEVAL accurately predicts the stability of the smart grid under various conditions, enabling the implementation of preventive stability control measures. Through extensive experiments, we demonstrate that QFEVAL achieves comparable performance to 9 state-of-the-art DSA approaches with more than 2 orders of magnitude fewer model parameter transmissions. QFEVAL paves the way for reliable, secure, and continuous electricity supply, offering a robust solution to the challenges of DSA in smart grids.

For more efficient generalization to unseen domains (classes), most Few-shot Segmentation (FSS) would directly exploit pre-trained encoders and only fine-tune the decoder, especially in the current era of large models. However, such fixed feature encoders tend to be class-agnostic, inevitably activating objects that are irrelevant to the target class. In contrast, humans can effortlessly focus on specific objects in the line of sight. This paper mimics the visual perception pattern of human beings and proposes a novel and powerful prompt-driven scheme, called "Prompt and Transfer" (PAT), which constructs a dynamic class-aware prompting paradigm to tune the encoder for focusing on the interested object (target class) in the current task. Three key points are elaborated to enhance the prompting: 1) Cross-modal linguistic information is introduced to initialize prompts for each task. 2) Semantic Prompt Transfer (SPT) that precisely transfers the class-specific semantics within the images to prompts. 3) Part Mask Generator (PMG) that works in conjunction with SPT to adaptively generate different but complementary part prompts for different individuals. Surprisingly, PAT achieves competitive performance on 4 different tasks including standard FSS, Cross-domain FSS (e.g., CV, medical, and remote sensing domains), Weak-label FSS, and Zero-shot Segmentation, setting new state-of-the-arts on 11 benchmarks.

Noticeable achievements have been made in entity-level perception tasks (e.g., object detection) in remote sensing (RS) image interpretation. But for RS images carrying rich content, individual perception cannot well obtain the interaction patterns between entities. The recognition of relationships between entities is the key to deeply understanding RS scenes. In this article, we propose a hypergraph-guided multimodal prototype network (HMPNet), which performs relation recognition by matching relation representations with multimodal predicate prototypes. To overcome the imbalance of modal information in the matching process, a multimodal calibration strategy is devised, taking into account the image subprototype and text subprototype, which makes prediction results more reliable. Meanwhile, to align image and text subprototypes and explore relevant semantic patterns, the multimodal hypergraph is constructed to efficiently capture the associations between heterogeneous prototypes. Experimental results show that the performance of our model can reach the state-of-the-art (SOTA) level on the RS scene graph generation (SGG) task.

Object detection and recognition (OD&R) based on deep learning is a hot topic in the application of synthetic aperture radar (SAR). These methodologies based on deep learning are inherently data-driven, which means that their performance is subjected to the corresponding datasets. Although existing datasets have included some common targets collected from real-valued intensity SAR images, there still exist some limitations in terms of quantity, categories, diversities, and data domain. Hence, it is urgent to establish a large-quantity benchmark for fine-grained OD&R on complex-valued SAR images, which contains rich signal-domain features well coupled with classical physical modeling. In addition, considering the unique imaging characteristics and diverse imaging conditions, some important attribute information, such as incidence and attitude angles, is necessary to be attached. In this article, we propose a novel benchmark dataset with more than 340k instances for fine-grained OD&R based on single-look complex (SLC) SAR images, which is named FAIR-CSAR. We collected complex-valued SAR images with a resolution of 1–5 m from 175 entire images of Gaofen-3 covering 32 cities and multiple sea areas worldwide. All instances in the FAIR-CSAR are annotated by oriented bounding boxes (OBBs), covering five major categories and 22 subcategories. Compared with existing datasets dedicated to OD&R, the FAIR-CSAR dataset has four particular advantages: 1) it contains complex-valued SAR images from various acquisition modes and polarization modes, including full-scale signal-domain features for object recognition; 2) it is much larger than other existing OD&R datasets in terms of quantity of instances; 3) it provides more fine-grained category annotation and more detailed attribute information; and 4) it provides more challenging images with some common imaging phenomena, such as speckle noise and azimuth ambiguities. To establish a baseline adapted for SLC SAR images, a multidomain feature extraction and fusion network (MDNet) is proposed as a novel framework to mine detailed information underlying various domains. A series of state-of-the-art (SOTA) algorithms are applied on the FAIR-CSAR to build the fine-grained OD&R benchmark. Experimental results indicate that FAIR-CSAR is closer to practical application and more challenging than existing datasets for SAR images.

The rapid development of Remote Sensing (RS) technology has generated a vast amount of heterogeneous time series data from various sources, including drone videos, satellite time-series images, and multi-object trajectories. Effectively processing and analyzing this multi-source heterogeneous data for accurate spatiotemporal prediction is crucial in fields such as environmental protection and disaster response. In this paper, we propose a universal predictive foundation model named Ringmo-SenseV2 to learn the general evolutionary patterns of RS elements from massive heterogeneous data. Ringmo- SenseV2 features a Mixture-of-Heterogeneous-Experts (MoHE) Transformer, which unifies the modeling of multi-source heterogeneous time-series data. Additionally, to better capture the complex dependencies across different spatiotemporal locations, we introduce a hypergraph translator, treating embeddings of different spatiotemporal locations as nodes and employing hypergraph convolution for information propagation. Furthermore, to enhance the model’s adaptability to different evolution speeds during pre-training, we implement the Adaptive tube Masking (AM) strategy, which controls prediction difficulty by adaptively setting mask proportions for sequences with varying evolution speeds. Extensive experiments demonstrate that Ringmo-SenseV2 exhibits outstanding performance across various RS prediction tasks. Further tests on scene graph generation for RS images showcase the model’s ability to extract image features, thereby enhancing image perception tasks.

With the development of large language models (LLMs) and remote sensing technology, visual language (VL) tasks in the field of remote sensing have attracted more and more research attention. Commonly used VL datasets currently usually focus on the overall scene of the image, lacking the description of instance-level details such as location, size, category and so on. More importantly, users are usually not allowed to directly intercept regions in the image to ask questions through these datasets. However, the instance-level question answering based on these information is of great significance for target extraction in practical applications. In this manuscript, we build an innovative and challenging dataset FAIR1M-GQA. It unlocks the ability of the model to learn directly from text input and text output both with region coordinates, which are directly linked to fine-grained objects in remote sensing images. We experiment our dataset to verify the feasibility of the relevant task and provide the benchmark results.

With the recent development of remote sensing (RS) technology, the amount of RS platforms has witnessed a substantial increase, and the capacity of Earth observation has been greatly enhanced. The interpretation of RS images has also gradually evolved from traditional centralized ground processing to on-orbit processing. However, the traditional single-platform on-orbit processing is limited to a single source of information, which results in the underutilization of the advantages of multiplatform observation in the current RS field, and restricts the accuracy of inference tasks. To tackle the aforementioned problem, we propose a multiplatform distributed collaborative inference network, which can combine the intermediate features from multiple platforms to improve the accuracy of inference tasks. First, we proposed the collaboration map generator, which generates the collaboration map for optimal collaborator selection autonomously. Second, a spatial feature compression (SFC) module is designed to compress the interplatform transmission features, adapting spatially sparse distribution characteristics of RS objects. Finally, a feature fusion module containing spatial priors is proposed to fuse the features collected from multiple platforms to obtain more precise inference results. We conducted extensive experiments on three public datasets and verified the effectiveness of the proposed framework. On the NWPU VHR-10 dataset, for example, the proposed method improves the detection accuracy by 13.7% and 10.3% under two experimental settings compared with a single platform and compresses the intermediate data transmission between platforms by more than 80%.

With well-pretrained foundation models (FMs), the performance of almost every remote sensing interpretation task has been boosted. The parameter volume of FMs increases with their continuously enhanced capabilities, leading to increased costs of fine-tuning. To apply FMs more effectively and efficiently, there are already some arts that introduce the parameter-efficient fine-tuning (PEFT) concept into remote sensing and achieve competitive performance with much lower parameter cost. However, the training efficiency of most PEFT frameworks may be not satisfactory. To make tuning FMs for remote sensing applications more efficient, we propose a training-efficient adapting (TEA) framework. Specifically, we attach a SIDE adapter network (SIDEAN) to the frozen powerful FMs and only update the SIDEAN to perform the downstream tasks. Moreover, to make TEA perceive remote sensing scenes from a macroscopic perspective and boost the performance, we propose a top-down guidance mechanism to inject macro scene information into the SIDEAN during adapting. TEA is also parameter-efficient, as SIDEAN is designed to be lightweight. We conduct extensive experiments to demonstrate the effectiveness and efficiency of TEA on ten widely adopted datasets covering four primary remote sensing tasks, e.g., object detection, orientated object detection, semantic segmentation, and scene classification. By training only 5.43% of the frozen FM parameters, TEA can save more than 57% of training memory footprint and up to 15% of time cost on average while achieving competitive performance on all datasets. Furthermore, TEA can surpass full fine-tuning on several datasets.

Remote sensing is stepping into the era of the foundation model, where the fine-tuning paradigm is widely adopted to transfer the profound knowledge of pretrained foundation models to downstream tasks. However, the full fine-tuning method would become inefficient in terms of training and storage, as the foundation models are getting larger and larger. Recently, a lot of deep learning research has proposed various parameter-efficient fine-tuning (PEFT) methods that perform well with a few trainable parameters. However, most of them focus on fine-tuning general foundation models without considering the special properties of remote sensing. In this article, we propose an adapter in remote sensing (AiRs) to fine-tune large foundation models for remote sensing downstream tasks by introducing the adapter-tuning framework. Specifically, we construct AiRs from two aspects: more expressive adaptation modules and a more efficient integration strategy. Specialized adaptation modules are applied to different functional layers in AiRs, which encode the inductive bias of remote sensing images and enhance the semantic concepts of geography. Moreover, AiRs establishes pathways between trainable modules with residual connections, which reduces training difficulty and improves performance. We conduct extensive experiments on object detection, semantic segmentation, and scene classification tasks. By training only 4.4% parameters of the pretrained backbone, AiRs surpasses the previous state-of-the-art (SOTA) PEFT competitors on all experimental datasets and outperforms the full fine-tuning on six out of ten datasets.

Recently, there has been a growing interest in few-shot incremental object detection (FSIOD). It learns new tasks with limited data while mitigating catastrophic forgetting on previous tasks. However, existing FSIOD methods experience parameter changes after training on new tasks, causing a parameter competition issue among tasks. Additionally, the background information differs among various tasks, and using a common background weight for all tasks results in the background shift. Constrained by these two issues, existing methods only alleviate catastrophic forgetting and cannot wholly prevent the performance decline on previous tasks. Especially for complex remote sensing images with messy background, the models trained on new tasks exhibit noticeable performance drops on previous tasks. In this paper, we propose a novel FSIOD method via dual-frequency prompt to address these challenges, named FSIOD-DFP. It can completely eliminate catastrophic forgetting while mitigating over-fitting. Specifically, a dual-frequency prompt generator is designed to tackle the parameter competition issue. It decouples the frequency components of images to produce prompts that modify the images to adapt to the base model trained on previous tasks. Compared to traditional prompts, our generator introduces fewer parameters to address over-fitting for limited data and allows freezing the base model to maintain the performance of previous data. Besides, a self-regularization loss is introduced to guide the prompt-modified images to leverage the knowledge of the base model effectively. Furthermore, we propose a task-decoupled detection head to address the background shift problem. It separates the detection heads for new and previous tasks to resolve the conflict in the background between different tasks. In FSIOD-DFP, only a prompt generator and a novel detection head are added and fine-tuned when learning a new task. Experiments on three remote sensing object detection datasets demonstrate that our method achieves state-of-the-art performance on both new and previous tasks in all few-shot incremental settings.

Research on multiview stereo (MVS) based on remote sensing images has promoted the development of large-scale urban 3-D reconstruction. However, remote sensing multiview image data suffer from the problems of occlusion and uneven brightness between views during acquisition, which leads to the problem of blurred details in depth estimation. To solve the above problem, we reexamine the deformable learning method in the MVS task and propose a novel paradigm based on view space and depth deformable learning (SDL-MVS), aiming to learn deformable interactions of features in different view spaces and deformably model the depth ranges and intervals to enable high accurate depth estimation. Specifically, to solve the problem of view noise caused by occlusion and uneven brightness, we propose a progressive space deformable sampling (PSS) mechanism, which performs deformable learning of sampling points in the 3-D frustum space and the 2-D image space in a progressive manner to embed source features to the reference feature adaptively. To further optimize the depth, we introduce depth hypothesis deformable discretization (DHD), which achieves precise positioning of the depth prior by adaptively adjusting the depth range hypothesis and performing deformable discretization of the depth interval hypothesis. Finally, our SDL-MVS achieves explicit modeling of occlusion and uneven brightness faced in MVS through the deformable learning paradigm of view space and depth, achieving accurate multiview depth estimation. Extensive experiments on LuoJia-MVS and WHU datasets show that our SDL-MVS reaches state-of-the-art performance. It is worth noting that our SDL-MVS achieves a mean absolute error (MAE) error of 0.086 and an accuracy of 98.9% for Acc$_{\lt 0.6\,\text {m}}$and 98.9% for Acc$_{\lt 3-\text {interval}}$on the LuoJia-MVS dataset under the premise of three views as input.

With the continuous improvement of remote sensing satellite technology for earth observation, satellite video object tracking has become a new research field. Tracking aerial objects in satellite video facilitates highly accurate positioning over a wide coverage area which plays a crucial role in remote sensing scenes such as military analysis and disaster prevention. However, for such a data-driven task, most previous satellite video datasets focus on ground object tracking and lack attention to high-speed aerial objects. In this regard, a high-resolution satellite video dataset for detecting and tracking high-speed aerial objects (AIR-HSAO) is built including satellite videos from over 20 regions worldwide, with detailed annotations for approximately 1400 airplane trajectories. Based on the AIR-HSAO, an end-to-end online multi-object tracking method called MGTrack is proposed for high-speed aerial objects. Specifically, the motion prediction module is proposed to implicitly model object motion and associate objects. Guided by the predicted offsets, the feature warping and fusion module aligns and fuses temporal features, fully exploiting the temporal correlations of the video sequence to obtain more accurate feature representations for high-speed objects. Additionally, a trajectory-aware size-enhanced loss is designed to handle the problem of discontinuous trajectories and small-scale objects. Experimental results on AIR-HSAO demonstrate that the proposed MGTrack improves the tracking accuracy by 2.3 MOTA and 1.6 IDF1 compared to the state-of-the-art tracking methods. The code is released at https://github.com/RenLibo-aircas/AIR-HSAT-dataset.

Sidelobes commonly disturb synthetic aperture radar (SAR) image understanding and interpretation. Traditional spatially variant filtering algorithms achieve a superior tradeoff between sidelobe suppression and resolution preservation by means of adaptively calculating filtering parameters under some specific restrictions, such as filter design restriction and minimum amplitude constraint (MAC). These restriction aims to obtain an efficient analytical solution for filters, which is easy to calculate under unsupervised conditions. However, the restriction scope is so narrow that the suppression performance achieved by these filters is limited. Also, since the unsupervised optimization based on MAC indiscriminately minimizes amplitude, the main-lobe loss is unavoidable. To further improve the performance, a spatially variant convolution neural network (SVNN) is proposed, which consists of two core modules. One is the spatially variant filter generation (SVFG) module, adaptively generating superior spatially variant filters under more relaxed restrictions. The other is a paralleled shifted convolution (PSC) module, converting the signal format to achieve a fast and parallel spatially variant filtering process. Benefiting from more relaxed filter restrictions, the novel network successfully achieves better performance on sidelobe suppression. In addition, with supervised optimization based on another more accurate restriction, namely, minimum error constraint (MEC), the proposed algorithm also achieves superior main-lobe maintenance. All of them are validated by comparative experiments based on satellite data from GaoFen-3 and TerraSAR-X, and our proposed method achieves state-of-the-art performance. The entire project is available athttps://github.com/suoyuxi/SVNN.

Synthetic aperture radar (SAR) possesses significant advantages in aircraft detection due to its all-day and all-weather monitoring capability, but some unique problems in SAR images decrease the performance of aircraft detection. The speckle effect and excessive dynamic range are the most common problems that interfere with the visual features in SAR images and deteriorate detection performance. However, there lacks a detection-oriented image enhancement algorithm to collaboratively solve these two problems. An adaptive image enhancement algorithm is proposed to improve the performance of aircraft detection in SAR images. The proposed image enhancement algorithm provides a pseudocolor image through speckle suppression and channel combination, which consists of the speckle noise suppression channel, strong scattering feature enhancement channel, and weak scattering feature enhancement channel. The speckle noise suppression is achieved by a despeckle network, and the radiational feature enhancement channels are derived from an adaptive quantization method based on the characteristics of amplitude distribution. By optimizing the quality of the input image, the proposed image enhancement algorithm improves the performance of aircraft detection. Experiments based on datasets acquired by GaoFen-3 satellites indicate that the proposed algorithms significantly improve the detection performance of various types of detectors. The source project is available athttps://github.com/suoyuxi/ChannelEnhancement.

Optical and synthetic aperture radar (SAR) image matching presents a formidable challenge due to their pronounced geometric and radiometric distinctions arising from multimodality. The distinct imaging mechanisms of optical and SAR sensors make it challenging to identify essentially homologous points in the physical sense, raising concerns about the accuracy and repeatability of correspondences in current feature matching methods. In this study, we introduce a detector-free feature matching algorithm specifically designed to match optical and SAR images through a two-step strategy. In the initial phase, our proposed method conducts pixelwise matching (PM) using downsampled feature descriptors, eliminating the necessity to identify repeatable keypoints. To mitigate complexity, we enforce a pseudo-epipolar constraint (PEC) to reduce computational costs by constraining the search range. Subsequently, refined matching is performed on the initial correspondences to rectify inaccuracies in the PM localization of the first step. Both matching steps are implemented on a graphics processing unit (GPU) to ensure high efficiency. The proposed algorithm attains an average matching accuracy of 2.39 pixels and operates with an efficiency of 1.09 s for 1108 image pairs, underscoring its superior comprehensive performance compared to various state-of-the-art algorithms, including handcrafted methods and deep learning networks.

As drone technology advances, using unmanned aerial vehicles for aerial surveys has become the dominant trend in modern low-altitude remote sensing. The surge in aerial video data necessitates accurate prediction for future scenarios and motion states of the interested target, particularly in applications like traffic management and disaster response. Existing video prediction methods focus solely on predicting future scenes (video frames), suffering from the neglect of explicitly modeling target’s motion states, which is crucial for aerial video interpretation. To address this issue, we introduce a novel task called Target-Aware Aerial Video Prediction, aiming to simultaneously predict future scenes and motion states of the target. Further, we design a model specifically for this task, named TAFormer, which provides a unified modeling approach for both video and target motion states. Specifically, we introduce Spatiotemporal Attention (STA), which decouples the learning of video dynamics into spatial static attention and temporal dynamic attention, effectively modeling the scene appearance and motion. Additionally, we design an Information Sharing Mechanism (ISM), which elegantly unifies the modeling of video and target motion by facilitating information interaction through two sets of messenger tokens. Moreover, to alleviate the difficulty of distinguishing targets in blurry predictions, we introduce Target-Sensitive Gaussian Loss (TSGL), enhancing the model’s sensitivity to both target’s position and content. Extensive experiments on UAV123VP and VisDroneVP (derived from single-object tracking datasets) demonstrate the exceptional performance of TAFormer in target-aware video prediction, showcasing its adaptability to the additional requirements of aerial video interpretation for target awareness.

Extrapolating future weather radar echoes from past observations is a complex task vital for precipitation nowcasting. The spatial morphology and temporal evolution of radar echoes exhibit a certain degree of correlation, yet they also possess independent characteristics. Existing methods learn unified spatial and temporal representations in a highly coupled feature space, emphasizing the correlation between spatial and temporal features but neglecting the explicit modeling of their independent characteristics, which may result in mutual interference between them. To effectively model the spatiotemporal dynamics of radar echoes, we propose a spatial-frequency-temporal correlation-decoupling transformer (SFTformer). The model leverages stacked multiple SFT-Blocks to not only mine the correlation of the spatiotemporal dynamics of echo cells but also avoid the mutual interference between the temporal modeling and the spatial morphology refinement by decoupling them. Furthermore, inspired by the practice that weather forecast experts effectively review historical echo evolution to make accurate predictions, SFTfomer incorporates a joint training paradigm for historical echo sequence reconstruction and future echo sequence prediction. Experimental results on the HKO-7 dataset and ChinaNorth-2021 dataset demonstrate the superior performance of SFTfomer in short-term (1 h), mid-term (2 h), and long-term (3 h) precipitation nowcasting.

Optical satellites are the most popular observation platforms for humans viewing Earth. Driven by rapidly developing multisource optical remote sensing technology, content-based remote sensing image retrieval (CBRSIR), which aims to retrieve images of interest using extracted visual features, faces new challenges derived from large data volumes, complex feature information, and various spatiotemporal resolutions. Most previous works delve into optical image representation and transformation to the semantic space of retrieval via supervised or unsupervised learning. These retrieval methods fail to fully leverage geospatial information, especially spatiotemporal features, which can improve the accuracy and efficiency to some extent. In this article, we propose a cross-modal contrastive learning method (CCLS2T) to maximize the mutual information of multisource remote sensing platforms for correlation-aware retrieval. Specifically, we develop an asymmetric dual-encoder architecture with a vision encoder that operates on multiscale visual inputs, and a lightweight text encoder that reconstructs spatiotemporal embeddings and adopts an intermediate contrastive objective on representations from unimodal encoders. Then, we add a hash layer to transform the deep fusion features into compact hash index codes. In addition, CCLS2T exploits the prompt template (R2STFT) for multisource remote sensing retrieval to address the text heterogeneity of metadata files and the hierarchical semantic tree (RSHST) to address the feature sparsification of semantic-aware indexing structures. The experimental results on three optical remote sensing datasets substantiate that the proposed CCLS2T can improve retrieval performance by 11.64% and 9.91% compared with many existing hash learning methods and server-side retrieval engines, respectively, in typical optical remote sensing retrieval scenarios.

Natural language moment localization aims to localize the target moment that matches a given natural language query in an untrimmed video. The key to this challenging task is to capture fine-grained video-language correlations to establish the alignment between the query and target moment. Most existing works establish a single-pass interaction schema to capture correlations between queries and moments. Considering the complex feature space of lengthy video and diverse information between frames, the weight distribution of information interaction flow is prone to dispersion or misalignment, which leads to redundant information flow affecting the final prediction. We address this issue by proposing a capsule-based approach to model the query-video interactions, termed the Multimodal, Multichannel, and Dual-step Capsule Network ( [Formula: see text]DCapsN), which is derived from the intuition that "multiple people viewing multiple times is better than one person viewing one time." First, we introduce a multimodal capsule network, replacing the single-pass interaction schema of "one person viewing one time" with the iterative interaction schema of "one person viewing multiple times," which cyclically updates cross-modal interactions and modifies potential redundant interactions via its routing-by-agreement. Then, considering that the conventional routing mechanism only learns a single iterative interaction schema, we further propose a multichannel dynamic routing mechanism to learn multiple iterative interaction schemas, where each channel performs independent routing iteration to collectively capture cross-modal correlations from multiple subspaces, that is, "multiple people viewing." Moreover, we design a dual-step capsule network structure based on the multimodal, multichannel capsule network, bringing together the query and query-guided key moments to jointly enhance the original video, so as to select the target moments according to the enhanced part. Experimental results on three public datasets demonstrate the superiority of our approach in comparison with state-of-the-art methods, and comprehensive ablation and visualization analysis validate the effectiveness of each component of the proposed model.

Few-shot object detection, expecting detectors to detect novel classes with a few instances, has made conspicuous progress. However, the prototypes extracted by existing meta-learning based methods still suffer from insufficient representative information and lack awareness of query images, which cannot be adaptively tailored to different query images. Firstly, only the support images are involved for extracting prototypes, resulting in scarce perceptual information of query images. Secondly, all pixels of all support images are treated equally when aggregating features into prototype vectors, thus the salient objects are overwhelmed by the cluttered background. In this paper, we propose an Information-Coupled Prototype Elaboration (ICPE) method to generate specific and representative prototypes for each query image. Concretely, a conditional information coupling module is introduced to couple information from the query branch to the support branch, strengthening the query-perceptual information in support features. Besides, we design a prototype dynamic aggregation module that dynamically adjusts intra-image and inter-image aggregation weights to highlight the salient information useful for detecting query images. Experimental results on both Pascal VOC and MS COCO demonstrate that our method achieves state-of-the-art performance in almost all settings. Code will be available at: https://github.com/lxn96/ICPE.

The construction of a basic model to extract generalized features from a large number of multimodal data is a new challenge in the field of remote sensing. Compared with natural scene images, When faced with a complex application scenario of remote sensing of multi-sensor acquisition, models that are suitable for a specific task are difficult to generalize to new scenarios. In this paper, we propose a model architecture based on the concepts of multi-domain representation and cross-domain fusion. By extracting strong generalization features from massive multi-modal data, a single foundation model can accomplish generalization interpretation for multiple downstream tasks. Experimental results show that the proposed model performs well on multiple downstream tasks, which validates the feasibility of the remote sensing cross-modal foundation model in the interpretation task.

Semi-supervised remote sensing semantic segmentation is an efficient way to increase the use of unlabeled data and cut labelling costs. The unlabeled-to-labeled data ratio is employed in more recent methods, which is very different from what is really used in practise. In this paper, we propose a semi-supervised semantic generative network for remote sensing images, introducing a self-supervised learning method to enhance the feature representation of the model when the data ratio is high. Specifically, we design a new branch for unlabeled data, which includes modules for both semantic reconstruction and appearance reconstruction. It can effectively alleviate the category confusion in complicated remote sensing image when there are few labeled data. Comprehensive experiments on the ISPRS POTSDAM dataset demonstrate that the proposed method achieves promising results.

Edge detection is a technique used to identify inherent structures within an image, and it is an essential requirement for synthetic aperture radar (SAR) applications. In particular, ratio-based edge detectors have been widely used in SAR image registration because of their ability to extract invariant features and reduce the effects of speckle noise. However, current edge detectors often struggle to accurately detect multi-scale objects and low-contrast structures. To address this issue, we present a robust multi-scale edge detector that uses a modified convolution kernel to improve the extensibility of edge features and aggregates multi-scale feature responses. We also propose a local scale estimation module to enhance edge responses in low-contrast areas and reduce noise effects. The experimental results demonstrate that our proposed method effectively preserves the integrity, continuity, and robustness of multi-scale and low-contrast structures. By incorporating our proposed edge detector into feature and template matching frameworks, we are able to significantly improve matching accuracy and outperform state-of-the-art SAR image registration methods.

Aircraft classification in synthetic aperture radar (SAR) images plays a considerable role in global region management and surveillance. Recently, deep learning has been applied to solve the classification problem and made significant progress. Due to the imaging variability at different angles and component scattering discreteness in SAR images, previous works have had difficulty in achieving desirable classification results. To address these issues, we study the positional and semantic relationship between the scattering points and propose an innovative scattering topology network (ST-Net) in this article. First, considering the diversity of imaging results caused by different target attitude angles, we extract and transform the scattering cluster centers to update the information of various categories. It can guide the model to strengthen the discriminative features and mitigate the impact of imaging variability on classification performance. Second, a novel scattering topology module (STM) is introduced to model the spatial relationships and semantic information interaction of discrete scattering points. In this process, the topology relations and scattering characteristics are enhanced for further accurate classification. Third, context attention excitation (CAE) is designed to capture significant global and semantic information, which is conducive to suppressing background interference and reducing category confusion. In conclusion, the ST-Net is presented with the SAR imaging mechanism and the topology geometric representation of aircraft. We construct the SAR aircraft category dataset (SAR-ACD) and conduct extensive experiments on it to show the effectiveness of ST-Net, which illustrates that our method achieves superior classification performance.

Aircraft detection is a challenging task for remote sensing images. The anchor-based methods are of high complexity, and the keypoint-based detectors suffer the grouping difficulty. Some line-based models relying on local features are hindered by adhesion and disintegrity problems. Moreover, those detection representations rarely take into account the sword-shaped component geometric semantics (e.g., fuselage and the wing) of the aircraft itself, leading to being less robust and unfavorable for downstream tasks, such as ones needing the detailed size and shape of aircraft. Accordingly, we model the sword-shaped component geometry and propose S2CGNet, a more robust appearance-based aircraft detector. The sword attenuation mask (SAM) module is devised to encode a “sword-shaped mask” for each aircraft while exploring more robustness via the geometric surface embedding. The SAM can provide clearer borders to separate different aircraft more precisely. Besides, to address the instance disintegrity problem and further boost the quality of SAM, we propose an instance aware graph (IAG) module to jointly optimize the parameters of the fuselage/wing detection heads. Experimental results show that the performance of S2CGNet can reach the state-of-the-art (SOTA) level. Specifically, it achieves 98.5% in terms of AP50 on the combined dataset of Aircraft-KP and NWPU VHR-10, boosting 3.8% than the baseline. Besides, S2CGNet boosts the quality of detection results greatly, e.g., it yields a significant improvement of 21.3% on AP75 compared to the baseline. Furthermore, the generalization comparisons on the FAIR1M dataset strongly demonstrate the robustness of our model surpasses other oriented detectors by a large margin.

Few-shot object detection (FSOD) has received numerous attention due to the difficulty and time-consuming of labeling objects. Recent researches achieve excellent performance in a natural scene by only using a few instances of novel classes to fine-tune the last prediction layer of the model well-trained on plentiful base data. However, compared with natural scene objects with a single direction and small size variety, the direction and size of the objects in remote sensing images (RSIs) vary greatly. The methods proposed for the natural scene cannot be directly applied to RSIs. In this article, we first propose a strong baseline for RSIs. It fine-tunes all detector components acting on high-level features and effectively improves the performance of novel classes. Further analyzing the results of the baseline, we find that the error for novel classes is mainly concentrated in classification. It misclassifies novel classes as confusable base classes or backgrounds due to the difficulty in extracting generalized information from limited instances. As is well-known, text-modal knowledge can highly summarize the generalized and unique characteristics of categories. Thus, we introduce text-modal descriptions for each category and propose an FSOD method guided by TExt-MOdal knowledge, called TEMO. Specifically, a text-modal knowledge extractor and a cross-modal assembly module are proposed to extract text features and fuse the text-modal features into visual-modal features. The fused features greatly reduce the classification confusion of novel classes. Furthermore, we introduce a mask strategy and a separation loss to avoid over-fitting and ambiguity of text-modal features. Experimental results on detection in optical remote sensing images (DIOR), Northwestern Polytechnical University (NWPU), and fine-grained object recognition in high-resolution remote sensing imagery (FAIR1M) illustrate that our TEMO achieves state-of-the-art performance in all settings.

Building 3D reconstruction from remote sensing images has a wide range of applications in smart cities, photogrammetry and other fields. Methods for automatic 3D urban building modeling typically employ multi-view images as input to algorithms to recover point clouds and 3D models of buildings. However, such models rely heavily on multi-view images of buildings, which are time-intensive and limit the applicability and practicality of the models. To solve these issues, we focus on designing an efficient DSM estimation-driven reconstruction framework (Building3D), which aims to reconstruct 3D building models from the input single-view remote sensing image. Existing DSM estimation networks suffer from the imbalance between local features and global features, which leads to over-smooth DSM estimates at instance boundaries. To address this issue, we propose a Semantic Flow Field-guided DSM Estimation (SFFDE) network, which utilizes the proposed concept of elevation semantic flow to achieve the registration of local and global features. First, in order to make the network semantics globally aware, we propose an Elevation Semantic Globalization (ESG) module to realize the semantic globalization of instances. Further, in order to alleviate the semantic span of global features and original local features, we propose a Local-to-Global Elevation Semantic Registration (L2G-ESR) module based on elevation semantic flow. Our Building3D is rooted in the SFFDE network for building elevation prediction, synchronized with a building extraction network for building masks, and then sequentially performs point cloud reconstruction and surface reconstruction (or CityGML model reconstruction). On this basis, our Building3D can optionally generate CityGML models or surface mesh models of the buildings. Extensive experiments on ISPRS Vaihingen and DFC2019 datasets on the DSM estimation task show that our SFFDE significantly improves upon state-of-the-art and δ1, δ2and δ3metrics of our SFFDE are improved to 0.595, 0.897 and 0.970. Furthermore, our Building3D achieves impressive results in the 3D point cloud and 3D model reconstruction process.

Deep learning approaches have contributed to the rapid development of remote sensing (RS) image interpretation. The most widely used training paradigm is to use ImageNet pretrained models to process RS data for specified tasks. However, there are issues such as domain gap between natural and RS scenes and the poor generalization capacity of RS models. It makes sense to develop a foundation model with general RS feature representation. Since a large amount of unlabeled data is available, the self-supervised method has more development significance than the fully supervised method in RS. However, most of the current self-supervised methods use contrastive learning, whose performance is sensitive to data augmentation, additional information, and selection of positive and negative pairs. In this article, we leverage the benefits of generative self-supervised learning (SSL) for RS images and propose an RS foundationmodel framework called RingMo, which consists of two parts. First, a large-scale dataset is constructed by collecting two million RS images from satellite and aerial platforms, covering multiple scenes and objects around the world. Second, we propose an RS foundation model training method designed for dense and small objects in complicated RS scenes. We show that the foundation model trained on our dataset with RingMo method achieves state-of-the-art (SOTA) on eight datasets across four downstream tasks, demonstrating the effectiveness of the proposed framework. Through in-depth exploration, we believe it is time for RS researchers to embrace generative SSL and leverage its general representation capabilities to speed up the development of RS applications.

Aliasing artifacts introduced by azimuth ambiguity seriously impact the interpretation of synthetic aperture radar images. To achieve parameter-free and fast azimuth ambiguity suppression, a novel deep learning model is designed to estimate the ambiguous signal intensity to total signal intensity ratio in the range-Doppler domain. This model does not depend on processing parameters and can be applied in any acquisition mode. The mean shift algorithm is applied to select less ambiguous subspectra according to the estimation result. The selected subspectra are restored to a full spectrum with an energy concentrated extrapolation method to preserve the resolution. The enhanced spectral selection and extrapolation algorithm overcomes the dependence on processing parameters, and experiments based on TerraSAR-X and Radarsat-2 images indicate that the proposed algorithm suppresses the azimuth ambiguity significantly.

As the number of aero and space remote sensing platforms increases, distributed observation and real-time terminal processing become mainstream in the future. However, most of the training methods for the multi-platform are still limited to centralized structures or independent training based on a single platform, which is inefficient or limited in accuracy. In order to solve this problem, we innovatively propose a distributed collaborative method (DCM) for remote sensing image classification training in this article. First, the proposed training method, which is based on one cloud and several terminals, can aggregate different parameters of the terminal network to the cloud to improve global accuracy. Second, a sample proximity network is designed to process the problem of data heterogeneity on different terminal networks, which further improves the accuracy during the model fusion on the cloud. Third, a multi-layer grouped concatenation module is applied after the model fusion to extract hierarchical features with different categories of remote sensing images. Experimental results on the challenging remote sensing image classification dataset FAIR1M show that the proposed training method has better collaborative learning ability than the centralized-based model or terminal-trained lightweight network under the heterogeneous data.

Multi-sensor remote sensing applications require the registration of optical and Synthetic Aperture Radar (SAR) images, which presents challenges due to significant radiometric and geometric differences resulting from distinct imaging mechanisms. Although various algorithms have been proposed, including hand-crafted features and deep learning networks, most of them focus on matching radiometric-invariant features while ignoring geometric differences. Furthermore, these algorithms often achieve promising results on datasets that use manually labeled ground truths that may be less reliable for high-resolution SAR images affected by speckle noise. To address these issues, we propose a robust global-to-local registration algorithm consisting of four modules: geocoding, global matching, local matching, and refinement. We generate a geometry-invariant mask in the geocoding module to help the local matching module focus on valid areas, introduce a fast global matching method to solve large offsets, and use matching confidence to guide subsequent local matching based on the accuracy of global matching. We propose a feature based on multi-directional anisotropic Gaussian derivatives (MAGD) and embed it into the confidence-aware local matching with the geometry-invariant mask to reduce the effect of geometric differences. Finally, we refine correspondence positions and remove outliers. We also build a high-accuracy evaluation dataset with hundreds of image pairs, where the ground truth is obtained by meta poles, which have clear and reliable structures in both optical and SAR images. Experimental results on this dataset demonstrate the superiority of our proposed algorithm compared to several state-of-the-art methods.

The rapid development of satellite platforms has yielded copious and diverse multi-source data for earth observation, greatly facilitating the growth of multimodal semantic segmentation (MSS) in remote sensing. However, MSS also suffers from numerous challenges: 1) Existing inherent defects in each modality due to the different imaging mechanisms. 2) Insufficient exploration of the intrinsic characteristics of modalities. 3) The existence of the huge semantic gap between heterogeneous data causes difficulties in feature fusion. The inability to effectively utilize the rich and diverse information provided by each modality and ignorance of the heterogeneity between modalities will hinder the feature enhancement, and further significantly impacts the semantic segmentation accuracy. Furthermore, neglecting the huge gap makes feature fusion challenging. In this study, we introduce a novel framework for multimodal semantic segmentation that effectively mitigates the aforementioned problems. Our approach employs a pseudo-siamese structure for feature extraction. Specifically, we propose a simple yet effective geometric topology structure modeling (GTSM) module to extract geometric relationships and texture information from optical data. Additionally, we present a modality intrinsic noise suppression (MINS) module to fully exploit radiation information and alleviate the effects of unique geometric distortions for SAR. Furthermore, we present an adaptive multimodal feature fusion (AMFF) module for fully fusing different modality features. Extensive experiments on both WHU-OPT-SAR and DFC23 datasets validate the robustness and effectiveness of the proposed Modality Characteristics-Guided Semantic Segmentation (MoCG) network compared to other state-of-the-art semantic segmentation methods, including multimodal and single-modal approaches. Our approach achieves the best performance on both datasets, resulting in mIoU/OA gains 69.1%/87.5% on WHU-OPT-SAR and 86.7%/97.3% on DFC23.

The proposal of Segment Anything Model (SAM) has created a new paradigm for deep learning-based semantic segmentation field, and has shown amazing generalization performance. However, we find it may fail or perform poorly on multimodal remote sensing scenarios, especially the Synthetic Aperture Radar (SAR) images. Besides, SAM does not provide category information of objects. In this paper, we propose a foundation model for multimodal remote sensing image segmentation called RingMo-SAM, which can not only segment anything in optical and SAR remote sensing data, but also identify object categories. First, a large-scale dataset containing millions of segmentation instances is constructed by collecting multiple open-source datasets in this field to train the model. Then, by constructing an instance-type and terrain-type category-decoupling mask decoder, the category-wise segmentation of various objects is achieved. In addition, a prompt encoder embedded with the characteristics of multimodal remote sensing data is designed. It not only supports multi-box prompts to improve the segmentation accuracy of multi-objects in complicated remote sensing scenes, but also supports SAR characteristics prompts to improve the segmentation performance on SAR images. Extensive experimental results on several datasets including iSAID, ISPRS Vaihingen, ISPRS Potsdam, AIR-PolSAR-Seg, etc. have demonstrated the effectiveness of our method.

Due to variable imaging conditions, the resolution degradation often occurs in aerial images, which in turn impairs the performance upper bound of semantic segmentation. To solve this problem, super-resolution is placed before semantic segmentation as a pre-processing sub-task in most existing methods. The above two sub-tasks often form a unidirectional open-loop structure for relatively independent optimization, which constrains the ultimate segmentation performance improvement. To break down information barriers among them and form a more compact overall optimization, we propose an effective learning method named as Pixel-Wise Mutual-Feed Learning (PW-MFL) for segmenting images with resolution degradation. The key is to build auxiliary bidirectional connections, which contribute to the mutual pixel-wise spatial and feature information guidance during training. The feed-forward connection is realized by the Self-Attention Context Correlation (SACC) module, which enhances the intra-class semantic features of pixel positions with poor super-resolution performance by the fusion of that with superior performance. The feed-back connection is achieved by the Semantic Weighted Mapping (SWM) module, which aims to activate and adjust the detailed features of super-resolution in incorrectly segmented pixel positions via the semantic feature information. In addition, the Pixel-Aware Optimization (PAO) strategy is proposed to give more attention to optimizing specific pixel positions based on spatial information. Extensive experiments are conducted on three representative remote sensing segmentation benchmarks, ISPRS Vaihingen, ISPRS Potsdam, and iSAID datasets. The state-of-the-art segmentation level in resolution-degraded aerial images is achieved through the proposed learning method.

Remote sensing spatiotemporal prediction aims to infer future trends from historical spatiotemporal data, e.g., videos and time series images, has a broad application prospect in many fields. The foundation model is a promising research direction for spatiotemporal information mining because of its robust feature extraction capability, and has made rapid progress in natural scenes. Nevertheless, due to the spatially multi-scale and temporally multi-scale properties in remote sensing data, these methods still encounter bottlenecks when applied to remote sensing. Therefore, we propose a foundation model for remote sensing spatiotemporal prediction via spatiotemporal evolution decoupling, abbreviated as RingMo-Sense. Considering spatial affinity, temporal continuity, and spatiotemporal interaction, we construct spatial, temporal, and spatiotemporal triple-branch prediction networks. Specifically, we use parameter-sharing and progressive joint training strategies to achieve stable long-range prediction and parameter reduction simultaneously. In addition, we build a remote sensing spatiotemporal dataset by collecting various remote sensing videos and time series images. The experimental results on six downstream spatiotemporal tasks demonstrate that the proposed model yields competitive performance.

Deep learning technology has promoted the object detection task in the remote sensing (RS) field to move toward better performance and more demanding requirements. Except for rigid body objects, component objects (COs) with more complex characteristics remain a detection challenge. Its “partial rules and overall disorder” characteristic limits the model learning ability to the structural features. And the internal noise and relatively sparse arrangement are not conducive to optimizing the model by the existing sample assignment strategies. We propose CODet to detect COs in RS scenes. It consists of a cross-hierarchy feature fusion module (CFM) and a noise-sparse sample assignment (NSA) strategy. CFM learns the potential representation and relative position relationship of components by fusing different level features. NSA redefines the optimization process of sample assignment. It aims to alleviate the problems of classification–localization misalignment (CLM) and the positive–negative sample imbalance (PNI) caused by the object’s internal noise and sparse arrangement. The method is verified on the proposed COD dataset of six categories of COs, reaching an average mAP/mAP50of 54.3/86.0. To be closer to the task requirements of the practical RS scene, we also propose a RS large-scale images inference framework. It includes a dataset (APRoI, labeled with COs and rigid body objects), a large-scale image inference strategy, and a set of evaluation metrics. With CODet as the core, the framework can effectively reduce the inference time by three to four times on images with an average of more than 100 million pixels.

Object detectors of remote sensing (RS) imagery with deep learning have become increasingly popular and rely heavily on extensive labeled data. The source-only detectors, which are trained on massive labeled data in a source domain, in some cases fail to get satisfactory performance on a target domain due to the domain shift. To alleviate the domain shift, popular approaches consider feature distribution alignment, but the target domain with massive unlabeled data is under-utilized. Some methods use the source-only model to generate pseudo labels for target domain data, but the variation of different remote sensing scenarios produces the domain shift, which is injected directly into the pseudo labels. Therefore, We propose the Dual-head rectification Domain Adaptation network (DualDA-Net) to alleviate the domain shift and exploit the potential of unlabeled target domain data. DualDA-Net cooperates the coarse-to-fine consistency alignment (CCA) with dual-head co-training (DHCT) to align the distribution and generate pseudo labels progressively. Specifically, the CCA focuses on source and target domain feature distribution alignment via coarse-to-fine consistency alignment on multi-level features. Moreover, the DHCT with dual detection heads is deployed in the teacher-student framework, where one of the heads complements the other with high-quality predictions to rectify the pseudo labels as supervision and alleviate the biased information. Sufficient experiments have been conducted on several domain adaptation settings. The experimental results demonstrate that our DualDA-Net achieves success in the target domain for cross domain object detection of RS imagery.

Multimodal remote sensing (RS) image segmentation aims to comprehensively utilize multiple RS modalities to assign pixel-level semantics to the studied scenes, which can provide a new perspective for global city understanding. Multimodal segmentation inevitably encounters the challenge of modeling intra- and inter-modal relationships, $i.e$ ., object diversity and modal gaps. However, the previous methods are usually designed for a single RS modality, limited by the noisy collection environment and poor discrimination information. Neuropsychology and neuroanatomy confirm that the human brain performs the guiding perception and integrative cognition of multimodal semantics through intuitive reasoning. Therefore, establishing a semantic understanding framework inspired by intuition to realize multimodal RS segmentation becomes the main motivation of this work. Drived by the superiority of hypergraphs in modeling high-order relationships, we propose an intuition-inspired hypergraph network ( $I^{2}HN$ ) for multimodal RS segmentation. Specifically, we present a hypergraph parser to imitate guiding perception to learn intra-modal object-wise relationships. It parses the input modality into irregular hypergraphs to mine semantic clues and generate robust mono-modal representations. In addition, we also design a hypergraph matcher to dynamically update the hypergraph structure from the explicit correspondence of visual concepts, similar to integrative cognition, to improve cross-modal compatibility when fusing multimodal features. Extensive experiments on two multimodal RS datasets show that the proposed $I^{2}HN$ outperforms the state-of-the-art models, achieving F1/mIoU accuracy 91.4%/82.9% on the ISPRS Vaihingen dataset, and 92.1%/84.2% on the MSAW dataset.

Multimodal abstractive summarization for videos is an emerging task that aims to generate a summary from multi-source information (i.e., video, audio transcript). The challenge is how to merge multimodal long sequences to capture rich semantic information without allowing possible noise from either lengthy modal sequence to degrade the other modality and thus hurt the entire model. To address the issues, we propose amultistagefusion network withforgetgate (MFFG), which selectively integrates multi-source information through the cross-fusion in encoding and hierarchical fusion in decoding between modalities, and design a fusion forget gate module to suppress the potential multimodal noise flow of multi-source long sequence. Meanwhile, considering that the source text in this task is lengthy and has the same distribution as the output summary text, we inherit the partial structure of the MFFG model and again propose its variant, single-stage fusion network with forget gate (SFFG), which simplifies the fusion schema, and leverages the long source text to enhance the representation of the target summary. Experimental results on How2 dataset and How2-300 dataset demonstrate the superiority of the two multimodal fusion methods. Further, we provide a version of ASR transcription data of How2 dataset to evaluate model performance under noisy scenarios, and experimental results show obvious advantages of our proposed models over prior systems.

Sarcasm is a sophisticated construct to express contempt or ridicule. It is well-studied in multiple disciplines (e.g., neuroanatomy and neuropsychology) but is still in its infancy in computational science (e.g., Twitter sarcasm detection). In contrast to previous methods that are usually geared toward a single discipline, we focus on the multidisciplinary cross-innovation, i.e., improving embryonic sarcasm detection in computational science by leveraging the advanced knowledge of sarcasm cognition in neuroanatomy and neuropsychology. In this work, we are oriented toward sarcasm detection in social media and correspondingly propose a multimodal, multi-interactive, and multihierarchical neural network ($M_{3}N_{2} $). We select Twitter, image, text in image, and image caption as the input of$M_{3}N_{2} $since the brain’s perception of sarcasm requires multiple modalities. To reasonably address the multimodalities, we introduce singlewise, pairwise, triplewise, and tetradwise modality interactions incorporating gate mechanism and guide attention (GA) to simulate the interactions and collaborations of involved regions in the brain while perceiving multiple modes. Specifically, we exploit a multihop process for each modality interaction to extract modal information multiple times using GA for obtaining multiperspective information. Also, we adopt a two-hierarchical structure leveraging self-attention accompanied by attention pooling to integrate multimodal semantic information from different levels mimicking the brain’s first- and second-order comprehensions of sarcasm. Experimental results show that$M_{3}N_{2} $achieves competitive performance in sarcasm detection and displays powerful generalization ability in multimodal sentiment analysis and emotion recognition.

Estimating height information from a single remote sensing image is a critical component for 3D perception. Recent methods formulate it as a dense height prediction task based on regression loss functions. However, the regression accuracy is limited by the infinite continuous solution space. In this paper, we propose the soft weighted ordinal (SWO) classification loss for height prediction model to convert the regression problem with infinite continuous values into the classification problem with finite discrete values. which greatly improves the accuracy of high estimation. Specifically, we first define the discrete height rule and introduce the distance penalty metric to transform the continuous ground truth height value to the soft probability distributions. This is then used as supervised information to optimize the pixel-wise classification model. Finally, we utilize soft weighted summation to generate continuous height values in the inference phase. The proposed SWO classification loss can be used directly with existing dense prediction structures whose performance can be strengthened by direct replacement of the loss functions. Comprehensive experiments on the IS-PRS Vaihingen dataset show that the proposed method has achieved promising results.

In recent years, the stereo positioning technology based on multiview spaceborne synthetic aperture radar (SAR) images has been widely applied in digital surface model extraction. In this letter, problems of the existing methods based on the range–Doppler (RD) model in a multiview stereo solution are pointed out. A robust stereo positioning solution for multiview spaceborne SAR images based on the RD model is proposed. In the proposed method, the traditional RD model is normalized to reduce the model errors caused by the different scales of the range equation and Doppler equation. A weighting strategy is also proposed to improve the positioning accuracy. This strategy is useful for multiview stereo positioning when using images of different satellites with orbital data of different accuracies. The experiments based on GaoFen-3 and TerraSAR-X satellite data sets validate the effectiveness of the proposed method.

With the rapid update and iteration of current aerial image data, the continual learning scenarios and catastrophic forgetting problem attracted increased attention, especially in the semantic segmentation task. However, the existing methods mainly focus on the class continual learning in a single task and are not satisfactory when extended to multiple tasks. In this article, we consider more realistic and complicated settings, namely task continual learning. We revisit the characteristics of semantic segmentation and knowledge distillation (KD) strategy, then propose a general and effective framework, named structured inheritance, to learn new tasks while retaining high performance on old tasks. Specifically, we present two structure-preserving penalties: pixel affinity structure loss and representation consistency structure loss. The former breaks the isolation of pixels and retains the pixel interactive information learned by the old tasks. At the same time, the latter protects high-frequency stationary information between sequence semantic segmentation tasks. Our approach does not need to add extra parameters nor does it need to access the data stream of the old tasks. Therefore, it can be applied in practical applications with strict computational burden, memory cost, and storage budget. Extensive continual learning experiments on four semantic segmentation datasets of Vaihingen, Potsdam, DeepGlobe, and Gaofen challenge semantic segmentation dataset (GCSS) prove the effectiveness of our proposed framework, which outperforms the current state-of-the-art methods and even exceeds the theoretical upper-bound performance of multitask learning. The code and models will be made publicly available.

The state-of-the-art deep neural networks have made a great breakthrough in remote sensing image classification. However, the heavy dependence on large-scale data sets limits the application of the deep learning to synthetic aperture radar (SAR) automatic target recognition (ATR) field where the target sample set is generally small. In this work, a metalearning framework named MSAR, consisting of a metalearner and a base-learner, is proposed to solve the sample restriction problem, which can learn a good initialization as well as a proper update strategy. After training, MSAR can implement fast adaptation with a few training images on new tasks. To the best of our knowledge, this is the first study to solve a few-shot SAR target classification via metalearning. In particular, the few-task problem is defined by analyzing the effect of available training classes on the performance of metalearning models. In order to reduce the metalearning difficulties caused by the few-task problem, three transfer-learning methods are employed, which can leverage the prior knowledge from the pretraining phase. Besides, we design a hard task mining method for effective metalearning. Based on the Moving and Stationary Target Acquisition and Recognition (MSTAR) data set, a specialized data set named NIST-SAR is devised to train and evaluate the proposed method. The experiments on NIST-SAR have shown that the proposed method yields better performances with the largest absolute improvements of 1.7% and 2.3% for 1-shot and 5-shot, respectively, over the next best, which indicates that the proposed method is promising and metalearning is a feasible solution for few-shot SAR ATR.

In recent years, deep convolutional neural networks (DCNNs) have made significant progress in cloud detection tasks, and the detection accuracy has been greatly improved. However, most existing CNN-based models have high computational complexity, which limits their practical application, especially for spaceborne optical remote sensing. In addition, most of the methods cannot make adaptive adjustments based on the structural information of the clouds, and blurred boundaries often occur in the detection results. In order to address these problems, this article proposes a lightweight network (DABNet) to achieve high-accuracy detection of complex clouds, not only a clearer boundary but also lower false-alarm rate. Specifically, a deformable context feature pyramid module is proposed to improve the adaptive modeling capability of multiscale features. Besides, a boundary-weighted loss function is designed to direct the network to focus on cloud boundary information and optimize the relevant detection results. The proposed method has been validated on two data sets: the public GF-1 WFV benchmark and our self-built GF-2 cloud detection data set with higher spatial resolution. The experimental results exhibit that DABNet achieves state-of-the-art performance while only using 4.12M parameters and 8.29G multiadds.

Recently, satellite video has become an emerging means of earth observation, providing the possibility of tracking moving objects. However, the existing multi-object trackers are commonly designed for natural scenes without considering the characteristics of remotely sensed data. In addition, most trackers are composed of two independent stages of detection and reidentification (ReID), which means that they cannot be mutually promoted. To this end, we propose an end-to-end online framework, which is called TGraM, for multi-object tracking in satellite videos. It models multi-object tracking as a graph information reasoning procedure from the multitask learning perspective. Specifically, a graph-based spatiotemporal reasoning module is presented to mine the potential high-order correlations between video frames. Furthermore, considering the inconsistency of optimization objectives between detection and ReID, a multitask gradient adversarial learning strategy is designed to regularize each task-specific network. In addition, aiming at the data scarcity in this field, a large-scale and high-resolution Jilin-1 satellite video dataset for multi-object tracking (AIR-MOT) is built for the experiments. Compared with state-of-the-art multi-object trackers, TGraM achieves efficient collaborative learning between detection and ReID, improving the tracking accuracy by 1.2 multiple object tracking accuracy. The code and dataset will be available online (https://github.com/HeQibin/TGraM).

Aircraft detection in synthetic aperture radar (SAR) images plays a significant role in dynamic monitoring and national security. Previous methods have difficulty in obtaining the desirable detection performance due to the interference of complex scenes and diversity of aircraft sizes. In order to solve these problems, we propose an innovative scattering feature relation network (SFR-Net) in this article. First, considering that the strong scattering points of the aircraft in SAR images are usually discrete, we leverage the proposed scattering point relation module to fulfill the analysis and correlation of scattering points. By enhancing the characteristics and relationships among the scattering points, this method is beneficial to guarantee the completeness of aircraft detection results. Second, we design a salient fusion module to adaptively aggregate the features from different layers of SFR-Net with rich semantic information and plentiful details, which can highlight the significant objects with different sizes and enhance the distinguishable features. Third, to reduce the false alarm and improve the localization accuracy, the contextual feature attention is presented to capture the global spatial and semantic information with a large receptive field. Overall, the SFR-Net is designed based on the SAR imaging mechanism and the scattering characteristics of aircrafts. The extensive experiments are conducted on the SAR aircraft detection dataset (AIRD) from the Gaofen-3 satellite to demonstrate the effectiveness of the SFR-Net and also illustrate that our method achieves state-of-the-art performance.

With the proposal of neural architecture search (NAS), automated network architecture design gradually becomes a new way in deep learning research. Due to its high capability regarding automated design, some pioneers have made an attempt to apply NAS in remote sensing and made some achievements, like 1-D/3-D Auto-convolutional neural network (CNN) and polarimetric synthetic aperture radar (PolSAR)-tailored Differentiable Architecture Search (PDAS). However, there are still some areas to be improved for existing NAS in remote-sensing field. In this article, we propose a random topology and random multiscale mapping (RTRMM) method to generate a multiscale and lightweight architecture for remote-sensing image recognition. First, a random topology generator generates the topology through random graph. Second, during the experiment, we find remote-sensing image features extracted by a multiscale network are more appropriate, compared with features extracted by a single-scale model. Nevertheless, the complexity inevitably increases with the introduction of a multiscale concept. Consequently, we design a variable search space consisting of decomposition convolution units under the guidance of mathematical analysis. The mapping of each neuron is then determined by a random multiscale mapping sampler. After that, we assemble the topology and mappings into blocks and construct three RTRMM models. Experiments on four scene classification datasets confirm the feature extraction capability and lightweight performance of RTRMM models. Moreover, we also observe that our approach achieves a better tradeoff between floating-point operations (FLOPs) and accuracy than some current well-behaved methods. Furthermore, the results on Vaihingen dataset verify the high feature-transfer capability.

Numerous deep-learning methods have been successfully applied to semantic segmentation and height estimation of remote-sensing imagery. It has also been proved that such framework can be reusable for multiple tasks to reduce computational resource overhead. However, there are still some technical limitations due to the semantic inconsistency between 3-D and 2-D features and strong interference of different objects with similar spectral-spatial properties. Previous works have sought to address these issues through hard parameter sharing or soft parameter sharing schemes. But due to unintentional integration, the specific information transmitted between multiple tasks is not clear or in a lot of redundancy. Furthermore, tuning the weights by hand between classification and regression loss function is challenging. In this paper, a novel multi-task learning method, termed ASSEH, is proposed to associatively segment semantics and estimate height from monocular remote-sensing imagery. First, considering semantic inconsistency across tasks, we design a task-specific distillation (TSD) module containing a set of task-specific gating units for each task at the cost of fewer parameters. The module allows for task-specific features to be tailored from backbone, whilst allowing for task-shared features to be transmitted. Second, we leverage the proposed cross-task propagation (CTP) module to construct and diffuse the local pattern graphlets at the common positions across tasks. Such a high-order recursive method can bridge two tasks explicitly to effectively settle semantic ambiguities caused by similar spectral characteristics with less computational burden and memory requirements. Third, a dynamic weighted geometric mean (DWGeoMean) strategy is introduced to dynamically learn the weights of each task and be more robust to the magnitude of the loss function. Finally, the results on ISPRS Vaihingen and Urban Semantic 3D data set well demonstrate that our ASSEH achieves the state-of-the-art performance.

Existing deep learning models usually assume that all data obeys independent identically distribution, which is unreasonable in remote sensing. Due to the differences in camera parameters, spectral ranges, resolutions, and so on, the images acquired by remote sensing sensors may be greatly diverse, causing models to face catastrophic forgetting when they are trained on new data only. Thus, incremental learning is introduced. An ideal incremental learning model should be expanded as the number of tasks increases, so as to have enough ability to adapt to the changes in data. However, existing approaches normally expand heavy modules for each task, making the holistic models cumbersome. In this article, a lightweight incremental learning approach (LIL) is proposed for remote sensing image scene classification. We replace the role of the feature extractor with extracting features of a single task instead of task-sharing features of all tasks to lighten the backbone. In addition, we propose a light feature transfer module (FTM) to realize the alignment of data distributions between different tasks in the feature domain. Furthermore, dual-constraint loss with knowledge distillation and adversarial learning is introduced to promote the mapping and alignment of data distributions at both the feature level and the semantic level. In LIL, only a tiny FTM and a classifier are added to the model when the model learns a new task. Experimental results show that our approach with a small number of parameters outperforms state-of-the-art approaches for incremental learning on both a single dataset and a sequence of multiple datasets.

Semantic segmentation in very-high-resolution (VHR) aerial images is one of the most challenging tasks in remote sensing image understanding. Most of the current approaches are based on deep convolutional neural networks (DCNNs). However, standard convolution with local receptive fields fails in modeling global dependencies. Prior research works have indicated that attention-based methods can capture long-range dependencies and further reconstruct the feature maps for better representation. Nevertheless, limited by the mere perspective of spatial and channel attention and huge computation complexity of self-attention (SA) mechanism, it is unlikely to model the effective semantic interdependencies between each pixel pair of remote sensing data with complex spectra. In this work, we propose a novel attention-based framework named hybrid multiple attention network (HMANet) to adaptively capture global correlations from the perspective of space, channel, and category in a more effective and efficient manner. Concretely, a class augmented attention (CAA) module embedded with a class channel attention (CCA) module can be used to compute category-based correlation and recalibrate the class-level information. In addition, we introduce a simple yet effective region shuffle attention (RSA) module to reduce feature redundant and improve the efficiency of SA mechanism via regionwise representations. Extensive experimental results on the ISPRS Vaihingen, Potsdam benchmark, and iSAID data set demonstrate the effectiveness and efficiency of our HMANet over other state-of-the-art methods.

Semantic segmentation in aerial imagery is still an important, yet challenging task due to the complex characteristics of remote-sensing data. The critical issues consist of: 1) extreme foreground–background imbalance; 2) large intra-class variance; and 3) arbitrary-oriented, dense, and small objects. The above challenges make it unlikely to model the effective global interdependencies of semantic heterogeneous regions. Besides, general semantic segmentation methods suffer from feature ambiguity due to the joint feature learning paradigm, leading to inferior detail information. In this article, we propose an improved semantic segmentation framework to tackle these problems via graph reasoning (GR) and disentangled learning. On the one hand, a simple, yet effective GR unit is introduced to implement coordinate-interaction space mapping and perform relation reasoning over the graph. It can be deployed on the feature pyramid network (FPN) to exploit cross-stage multi-scale information. On the other hand, we propose a so- called disentangled learning paradigm to explicitly model the foreground and boundary objects, instantiated as foreground prior estimation (FPE) and boundary alignment (BA). The indication of the intermediate feature can be effectively emphasized to enhance the discriminative abilities of the network. Extensive experiments over iSAID, ISPRS Vaihingen, and the general Cityscapes datasets demonstrate the effectiveness and efficiency of the proposed framework over other state-of-the-art semantic segmentation methods.

Recently, deep learning in synthetic aperture radar (SAR) automatic target recognition (ATR) has made significant progress, but the sample limitation problem in the SAR field is still obvious. Compared with the optical remote sensing images, the SAR images are insufficient, especially those containing the geospatial targets with certain target attitude angles (TAAs). To solve these problems, a novel few-shot learning framework named scattering characteristics analysis network (SCAN) is proposed in this article. First, a scattering extraction module (SEM) is designed to combine the target imaging mechanism with the network, which learns the number and distribution of the scattering points for each target type via explicit supervision. Besides, considering the imaging variability of SAR targets, a TAA-guided metalearning network consisting of an angle self-adaption classifier (ASC) and a frequency embedded module (FEM) is designed. ASC guides the network to focus on the positive sample pairs with different TAAs. FEM combines pulse cosine transform (PCT) with the network training process effectively to enrich frequency-domain information. In addition, a new dataset named SAR aircraft category dataset is constructed for the experiments. Compared with other few-shot SAR target classification approaches, our model efficiently integrates the scattering characteristics with the learning process, and the test accuracy for 5-way 1-shot has been improved by 4.74%. Finally, the experimental results are provided to demonstrate the validity of the proposed method.

Ship detection has broad applications in many areas, including fishery management, maritime rescue, and maritime monitoring. Recently, numerous detectors based on deep learning have been carried in ship detection in synthetic aperture radar (SAR) images. However, detecting the inshore ships faces enormous challenges because of the strong scattering interference of the inland area. In order to address such issues, a novel method named strong scattering points network for ship detection is proposed in this article. First, according to the SAR imaging mechanism, the ships usually appear strong scattering phenomenon in the SAR images. Therefore, the proposed method detects the strong scattering points on the ship and then aggregates their positions to obtain the ship’s arbitrary orientation box. Second, our method designs an embedding vector to cluster these points as an individual object to regress the oriented bounding box. Third, in order to distinguish the strong scattering points on land, a ship attention module is employed to extract the image texture features and representations of local features. It can suppress the false alarm caused by land interference in the detection process. Furthermore, to demonstrate the effectiveness of the proposed algorithm, this article introduces a new ship dataset for oriented ship detection named large-scale dataset for ship detection in SAR images (LDSD). Moreover, the public SAR ship detection dataset (SSDD) is utilized to verify the robustness and generalization ability of the detector. The experimental results on two datasets show that our method has a strong anti-interference ability in the inshore background and achieves state-of-the-art detection performance.

Geographic object segmentation is a fundamental yet challenging problem for remote sensing image interpretation. The prevalent paradigm to solve this problem is to train deep neural networks on massive labeled samples. Although remarkable achievements have been attained, these methods suffer from the severe dependence on the large-scale dataset and require a long training process with high computation burden. To address these issues, a deep metametric learning framework, named DMML-Net, consisting of the metametric learner and the base-metric learner, is proposed for few-shot geographic object segmentation. First, DMML-Net formulates the segmentation as the metric-based pixel classification and develops a deep feature pyramid comparison network as the architecture of the metric learner for multiscale metric learning. Benefiting from this design, the segmentation can be efficiently solved, as well as being robust to deal with the scale variations of geographic objects. Second, an affinity-based fusion mechanism is introduced to adaptively reweight and fuse the semantic information across samples, effectively calibrating the deviation of prototypes induced by the intraclass variations. Third, considering the impact of the large interclass distribution divergences, DMML-Net presents a metametric training paradigm to provide the metric model with flexible scalability for fast adaptation to novel tasks. After metatraining, DMML-Net can be applied for the few-shot segmentation tasks of novel geographic objects with only a few gradient steps on the small training set. Experimental results on two benchmark remote sensing datasets demonstrate the validity and the superiority of our method in low-shot conditions where there are only one to ten labeled samples.

Range sidelobe is very common in synthetic aperture radar (SAR) images, particularly when imaging scene includes strongly scattering targets such as ships or complex buildings. As a kind of interference, it may reduce the image quality and hinder the image interpretation. Hence, range sidelobe suppression is an important mission for SAR images. The main task of mitigating the sidelobe is how to achieve the most effective suppression with the minimal resolution loss and signal-to-noise ratio (SNR) loss. However, the widely recognized classic method, spatially variant apodization (SVA), still has a lot of residual sidelobe energy and other problems. This article proposes a novel suppression approach based on time-variant transmission of chaotic frequency modulation (CFM) signals. The key is to build an appropriate transmitted signal set, where the signals are generated by various chaotic initial states and the same special map with low mixing rate and uniform invariant probability density (IPD). Due to their beneficial autocorrelation properties, the proposed approach achieves superior performance in range sidelobe suppression and resolution preservation. More importantly, it maintains the energy of the signals and overcomes the SNR loss that occurs in some classic methods, such as spectral weighting (SW) and SVA. In addition, it is suitable for both vertical and squint side-looking mode and can well reconstruct the weakly scattering targets which are severely disturbed by range sidelobe. All of them are validated by comparative experiments.

Despite impressive progress in the past decade, accurate and efficient multiview synthetic aperture radar (SAR) image registration remains a challenging task due to complex imaging mechanisms and various imaging conditions. Especially, for rugged areas, SAR images obtained from the opposite-side view reflect different characteristics, making popular SAR image registration methods no longer applicable. To this end, we propose a geometry-aware image registration method by extracting inherent orientation features and concentrating on geometry-invariant areas. First, slant range images are terrain-corrected using a digital elevation model (DEM) to reduce large relative positioning errors caused by elevation. Second, the Gabor-ratio detector is introduced to obtain multiscale orientation features, which are more robust under various imaging conditions. Then, a geometry-aware mask is produced by intersecting the 3-D space ray with DEM, and thus, SAR images can be divided into three categories, layover, shadow, and geometry-invariant areas. The geometry-aware matching method, which focuses on geometry-invariant areas and masks out misleading caused by geometric and radiometric distortions, is proposed to realize accurate matching. The rational polynomial coefficients (RPCs) are refined to achieve relative correction. Extensive results on dozens of SAR images demonstrate the effectiveness and universality of the proposed algorithm by quantitative evaluation using man-made and natural corner reflectors. An analysis of the factors affecting registration accuracy is also discussed.

Object detection is a fundamental part of the interpretation of remote sensing imagery. The one-stage object detector has been adopted into this field because of its high computational efficiency. However, this detector suffers from the misalignment among predefined anchor, object, and feature extracted by standard convolution kernel both in spatial and scale. It limits the further improvement of performance, especially for the long-narrow and multiscale geospatial objects. In this article, the problem is defined asthe feature misalignmentproblem. To deal with this issue, an efficient feature aligned single-shot detector (ASSD) is proposed, which consists of two modules: a novel pseudo anchor proposal module (PAPM) and a flexible context-based feature alignment module (CFAM). The PAPM replaces the regular anchor group with the proposed core anchor and refines it to get aligned locations. It can tackle the spatial misalignment between anchors and their corresponding objects and alleviate the negative/positive imbalance problem. Then, the CFAM adaptively adjusts the sampling points of the convolution kernel and collects the context information according to the aligned core anchor. This plug-and-play module can effectively rectify the misalignment between kernel and objects and extract aligned and robust features. A series of comprehensive experiments are conducted on two large-scale public remote sensing object detection datasets. Experiment results suggest that the proposed method is effective to alleviate the misalignment problem. Compared with the baseline model, the detection accuracy is improved by 8.5% mAP and 11.0% mAP on the challenging benchmark for object detection in optical remote sensing image (DIOR) and a large-scale dataset for object detection in aerial image (DOTA) dataset, respectively. Our best-resulting model achieves the state-of-the-art performance, surpassing other one-stage detectors both on the two datasets at a high detection speed of 21 FPS.

Deep learning-based object detectors have been widely adopted in the field of remote sensing imagery interpretation. These detectors heavily depend on the expensive large-scale labeled datasets, while the scarce remote sensing datasets limit the performance. The domain adaptive object detection can alleviate this problem. However, it struggles with the confusing feature’s alignment, damaging the domain generalization performance, especially for the remote sensing scene with sparse objects and diverse backgrounds. For that reason, a semisynthetic data generator (SDG) is proposed to automatically generate the remote sensing dataset with low cost and replace the real-world training dataset, afeature aligned domain adaptive object detector(FADA) is proposed to enhance the domain adaptation among the cross-domain remote sensing images. The FADA contains two proposed modules in addition to the base detector: an adversarial-based foreground alignment (AFA) and a prototype-based confusing feature alignment (PCFA). The AFA aligns the cross-domain foreground feature by adversarial training (AT), and it can filter the noisy background feature that is not suitable to transfer. Then, the PCFA adaptively aligns the confusing background and foreground feature, further promoting the domain adaptation performance. Comprehensive experiments validate the effectiveness of the proposed method. Compared with the baseline model trained on the semisynthetic source dataset, our FADA improves the generalized performance on the real-world target dataset a large-scale Dataset for Object deTection in Aerial images (DOTA) by 15.7% average precision (AP) and achieves state-of-the-art results.

The resolution degradation poses a huge challenge for object detection (OD) in the aerial imagery. Existing methods utilize super resolution (SR) based on Generative Adversarial Network (GAN) to restore texture details in degraded images. However, constrained detection results are still acquired due to the object feature difference between restored and clear images. Therefore, we propose a simple-yet-effective learning method called Mutual-Feed Learning (MFL) to solve the problem in this paper. A closed-loop structure is designed via building the feedback connection based on the feedforward connection between the two tasks. It effectively delivers the object spatial and feature information from OD to SR, and provides restoration-enhanced images from SR to OD. Specifically, a Feedback of Region of Interest (FROI) module is introduced to realize a region-level discrimination under the guidance of object information. It guides the discrimination process of super resolution. Furthermore, a Multi-Scale Object Information (MSOI) module is developed to implement a feature-level restoration by narrowing differences in object-related features. It improves the generation process of super resolution. Then object detection can be performed in restoration-enhanced images to obtain more accurate results. Extensive experiments over NWPU VHR-10, COWC, and FAIR1M dataset show that the method can achieve state-of-the-art results.

Lightweight object detector is currently gaining more and more popularity in remote sensing. In general, it’s hard for lightweight detectors to achieve competitive performance compared to traditional deep models, while knowledge distillation is a promising training method to tackle the issue. Since the background is more complicated and the object size varies extremely in remote sensing images, it will deliver lots of noise and affect the training performance when directly applying the existing knowledge distillation methods. To tackle the above problems, we propose an Adaptive Reinforcement Supervision Distillation (ARSD) framework to promote the detection capability of the lightweight model. Firstly, we put forward a multiscale core features imitation (MCFI) module for transferring the knowledge of features, which can adaptively select the multiscale core features of objects for distillation and focus more on the features of small objects by an area-weighted strategy. In addition, a strict supervision regression distillation (SSRD) module is designed to select the optimal regression results for distillation, which facilitates the student to effectively imitate the more precise regression output of the teacher network. Massive experiments on the DOTA, DIOR, and NWPU VHR-10 datasets prove that ARSD outperforms the existing distillation SOTA methods. Moreover, the performance of lightweight model trained with our method transcends other classic heavy and lightweight detectors, which beneficiates the development of lightweight models.

Remote sensing (RS) cross-modal text–image retrieval has attracted extensive attention for its advantages of flexible input and efficient query. However, traditional methods ignore the characteristics of multiscale and redundant targets in RS image, leading to the degradation of retrieval accuracy. To cope with the problem of multiscale scarcity and target redundancy in RS multimodal retrieval task, we come up with a novel asymmetric multimodal feature matching network (AMFMN). Our model adapts to multiscale feature inputs, favors multisource retrieval methods, and can dynamically filter redundant features. AMFMN employs the multiscale visual self-attention (MVSA) module to extract the salient features of RS image and utilizes visual features to guide the text representation. Furthermore, to alleviate the positive samples ambiguity caused by the strong intraclass similarity in RS image, we propose a triplet loss function with dynamic variable margin based on prior similarity of sample pairs. Finally, unlike the traditional RS image-text dataset with coarse text and higher intraclass similarity, we construct a fine-grained and more challenging Remote sensing Image-Text Match dataset (RSITMD), which supports RS image retrieval through keywords and sentence separately and jointly. Experiments on four RS text–image datasets demonstrate that the proposed model can achieve state-of-the-art performance in cross-modal RS text–image retrieval task.

Remote sensing (RS) crossmodal text-image retrieval has become a research hotspot in recent years for its application in semantic localization. However, since multiple inferences on slices are demanded in semantic localization, designing a crossmodal retrieval model with less computation but well performance becomes an emergent and challenging task. In this article, considering the characteristics of multi-scale and target redundancy in RS, a concise but effective crossmodal retrieval model (LW-MCR) is designed. The proposed model incorporates multi-scale information and dynamically filters out redundant features when encoding RS image, while text features are obtained via lightweight group convolution. To improve the retrieval performance of LW-MCR, we come up with a novel hidden supervised optimization method based on knowledge distillation. This method enables the proposed model to acquire dark knowledge of the multi-level layers and representation layers in the teacher network, which significantly improves the accuracy of our lightweight model. Finally, on the basis of contrast learning, we present a method employing unlabeled data to boost the performance of RS retrieval model further. The experiment results on four RS image-text datasets demonstrate the efficiency of LW-MCR in RS crossmodal retrieval (RSCR) tasks. We have released some codes of the semantic localization and made it open to access athttps://github.com/xiaoyuan1996/retrievalSystem.

Cross-modal remote sensing text-image retrieval (RSCTIR) has recently become an urgent research hotspot due to its ability of enabling fast and flexible information extraction on remote sensing (RS) images. However, current RSCTIR methods mainly focus on global features of RS images, which leads to the neglect of local features that reflect target relationships and saliency. In this article, we first propose a novel RSCTIR framework based on global and local information (GaLR), and design a multi-level information dynamic fusion (MIDF) module to efficaciously integrate features of different levels. MIDF leverages local information to correct global information, utilizes global information to supplement local information, and uses the dynamic addition of the two to generate prominent visual representation. To alleviate the pressure of the redundant targets on the graph convolution network (GCN) and to improve the model’s attention on salient instances during modeling local features, the denoised representation matrix and the enhanced adjacency matrix (DREA) are devised to assist GCN in producing superior local representations. DREA not only filters out redundant features with high similarity, but also obtains more powerful local features by enhancing the features of prominent objects. Finally, to make full use of the information in the similarity matrix during inference, we come up with a plug-and-play multivariate rerank (MR) algorithm. The algorithm utilizes the$k$nearest neighbors of the retrieval results to perform a reverse search, and improves the performance by combining multiple components of bidirectional retrieval. Extensive experiments on public datasets strongly demonstrate the state-of-the-art performance of GaLR methods on the RSCTIR task. The code of GaLR method, MR algorithm, and corresponding files have been made available at:https://github.com/xiaoyuan1996/GaLR.

Deep convolutional neural networks (CNNs) have brought a tremendous increase in detection accuracy, but too cumbersome model makes them hard to deploy on low computation edge devices, such as satellites and unmanned aerial vehicles. A promising method to tackle this problem is knowledge distillation (KD), which makes models lightweight with satisfactory accuracy. For remote sensing images, the objects are usually environment-related and located in a cluttered scene. The features that objects’ semantic information relies on are tangled. However, existing distillation methods only imitate feature distribution derived from regions, including objects resulting in poor performance. Furthermore, masses of instances generated by teachers are blindly inherited, even if some of them are outliers. In this article, we propose a general and effective KD framework called dynamic knowledge distillation (DKD). First, our framework leverages the dynamic global distillation (GD) module to discover valuable regions from the foreground and background for multiscale features imitation, avoiding ignoring the potential geographical spatial relationship. Second, we propose a dynamic instance selection distillation (ISD) module to give students the ability of self-judgment through the magnitude of detection loss. Third, toward more accurate handling of hard samples in regression, a training-status-aware loss is tailored to guide students mine knowledge about objects with large aspect ratio or small size. Extensive experiments are conducted to show the effectiveness of DKD framework. The detection results on DOTA and NWPU VHR-10 dataset illustrate that our method is suitable for single-stage, two-stage and even anchor-free detectors. It shows the state-of-the-art performance. The code will be publicly available.

The encoder–decoder framework is prevalent in existing remote-sensing image captioning (RSIC) models. The appearance of attention mechanisms brings significant results. However, current attention-based caption models only build up the relationships between the local features without introducing the global visual feature and removing redundant feature components. It will cause caption models to generate descriptive sentences that are weakly related to the scene of images. To solve the problems, this article proposed a global visual feature-guided attention (GVFGA) mechanism. First, GVFGA introduces the global visual feature and fuses them with local visual features to build up their relationships between them. Second, an attention gate utilizing the global visual feature is proposed in GVFGA to filter out redundant feature components in the fused image features and provide more salient image features. In addition, to relieve the hidden state’s burden, a linguistic state (LS) is proposed to specifically provide textual features, making the hidden state only guiding visual–textual attention process. What’s more, to further refine the fusion of visual features and textual features, a LS-Guided Attention (LSGA) mechanism is proposed. It can also filter out the irrelevant information in the fused visual–textual feature with the help of an attention gate. The experimental results show that this proposed image captioning model can achieve better results on three RSIC datasets, UCM-Captions, Sydney-Captions, and RSICD datasets.

In recent years, image-level weakly supervised semantic segmentation (WSSS) has developed rapidly in natural scenes due to the easy availability of classification tags. However, limited to complex backgrounds, multi-category scenes, and dense small targets in remote sensing (RS) images, relatively little research has been conducted in this field. To alleviate the impact of the above problems in RS scenes, a self-supervised Siamese network based on an explicit pixel-level constraints framework is proposed, which greatly improves the quality of class activation maps and the positioning accuracy in multi-category RS scenes. Specifically, there are three novel devices in this paper to promote performance to a new level: (a) A pixel-soft classification loss is proposed, which realizes explicit constraints on pixels during the image-level training; (b) A pixel global awareness module, which captures high-level semantic context and low-level pixel spatial information, is constructed to improve the consistency and accuracy of RS object segmentation; (c) A dynamic multi-scale fusion module with a gating mechanism is devised, which enhances feature representation and improves the positioning accuracy of RS objects, particularly on small and dense objects. Experiments on two RS challenge datasets demonstrate that these proposed modules achieve new state-of-the-art results by only using image-level labels, which improve mIoU to 36.79% on iSAID and 45.43% on ISPRS in the WSSS task. To the best of our knowledge, this is the first work to perform image-level WSSS on multi-class RS scenes.

With the development of deep convolutional neural networks, detecting rotating objects in remote-sensing images is of great significance in various fields. Existing rotating object detectors most suffer the problem of ambiguous supervision caused by inappropriate rotating object representations. This problem may result in fuzzy object localization and further lead to misclassification. In this article, we propose an Automatic Organized Points Detector (AOPDet), which derives precise localization results by applying a novel rotating object representation called nonsequential corners representation. To achieve the proposed representation, an Automatic Organization Mechanism (AOM) technique is designed to guide the model to organize points to object corners automatically. An Automatic-Organized-Points-specific (AOP-specific) head structure is also designed and equipped in the model to better focus on the rotating object detection task. On public aerial datasets, experiments show that the AOPDet achieves 17.0 mAP higher than the compared baseline model, reaching the state-of-the-art (SOTA) level. Detailed ablation experiments and error analysis strongly reveal the effectiveness of the proposed model.

Due to the characteristics of vertical orthophoto imaging, the apparent structural features of the object in the remote sensing image are relatively stable, such as the cross-shaped structure of the aircraft, the rectangular structure of the vehicle, etc. Compared with the traditional visual features, using these features is conducive to improving the accuracy of object detection. However, there are few studies on such characteristics. In this paper, we systematically study the invariant structural features of remote sensing objects and propose a Graph Focusing Aggregation Network (GFA-Net) to represent the structural features of remote sensing objects. Among them, in view of the problem that traditional convolutional neural networks (CNNs) are sensitive to the changes in rotation, scale, and other factors, which makes it difficult to extract structural features, we propose the Graph Focusing Process (GFP) based on the idea of graph convolution. Analysis and experiments show that graph structure has significant advantages over Euclidean feature space under CNN in expressing such structural features. In order to realize the end-to-end efficient training of the above model, we design Graph Aggregation Network (GAN) to update the weight of nodes. We verify the effectiveness of our method on the proposed multi-task datasets ACSD and large-scale fine-grained remote sensing dataset FAIR1M. Experiments conducted on the object detection data sets of DOTA and HRSC2016 prove that the proposed method is superior to the current state-of-the-art method.

Object detection in aerial images is important for a wide range of applications. The most challenging dilemma in this task is the arbitrary orientation of objects, and many deep-learning-based methods are proposed to address this issue. In previous works on oriented object detection, the regression-based method for object localization has limited performance due to the shortage of spatial information. And the models suffer from the divergence of feature construction for object recognition and localization. In this article, we propose a novel architecture, i.e., point-based estimator to remedy these problems. To utilize the spatial information explicitly, the detector encodes an oriented object with a point-based representation and operates a fully convolutional network for point localization. To improve localization accuracy, the detector takes the manner of coarse-to-fine to lessen the quantization error in point localization. To avoid the discrepancy of feature construction, the detector decouples localization and recognition with individual pathways. In the pathway of object recognition, the instance-alignment block is involved to ensure the alignment between the feature map and oriented region. Overall, the point-based estimator can be easily embedded into the region-based detector and leads to significant improvement on oriented object detection. Extensive experiments have demonstrated the effectiveness of our point-based estimator. Compared with existing works, our method shows state-of-the-art performance on oriented object detection in aerial images.

Recently, deep-learning methods have been successfully applied to the ship detection in the synthetic aperture radar (SAR) images. It is still a great challenge to detect multiscale SAR ships due to the broad diversity of the scales and the strong interference of the inshore background. Most prevalent approaches are based on the anchor mechanism that uses the predefined anchors to search the possible regions containing objects. However, the anchor settings have a great impact on their detection performance as well as the generalization ability. Furthermore, considering the sparsity of the ships, most anchors are redundant and will lead to the computation increase. In this article, a novel detection method named feature balancing and refinement network (FBR-Net) is proposed. First, our method eliminates the effect of anchors by adopting a general anchor-free strategy that directly learns the encoded bounding boxes. Second, we leverage the proposed attention-guided balanced pyramid to balance semantically the multiple features across different levels. It can help the detector learn more information about the small-scale ships in complex scenes. Third, considering the SAR imaging mechanism, the interference near the ship boundary with the similar scattering power probably affects the localization accuracy because of feature misalignment. To tackle the localization issue, a feature-refinement module is proposed to refine the object features and guide the semantic enhancement. Finally, extensive experiments are conducted to show the effectiveness of our FBR-Net compared with the general anchor-free baseline. The detection results on the SAR ship detection dataset (SSDD) and AIR-SARShip-1.0 dataset illustrate that our method achieves the state-of-the-art performance.

The detection of garbage dumps is of great significance for environmental protection. Recently, deep learning algorithms have brought impressive improvements for regular object detection. Different from conventional objects, garbage dumps are more inconspicuous and irregular and have the problem of blurred boundaries. To solve these problems, we propose a shape robust anchor-free network (SRAF-Net) that consists of feature extraction, multitask detection, and postprocessing. First, our network leverages the context-based deformable (CBD) module to combine context attention and deformable convolution. The contextual information obtained by context attention enables the network to focus on objects with inconspicuous appearance, while the deformable convolution enhances the feature representation. Then, we propose a multitask detection head to regress irregular garbage dumps in a more accurate and efficient way. The anchor-based methods need to define some anchors with a fixed shape. However, our detection method is anchor-free that learns the shapes of objects from training data. The detection head adaptively generates various shapes of bounding boxes with their classification confidences and localization confidences. Weighted by the localization confidences, we merge bounding boxes during postprocessing, which alleviates the blurred boundaries. In addition, we build a new public data set named garbage dumps data set (GDD) to verify the effectiveness of our method. Extensive experiments on GDD indicate that our method surpasses the existing detection methods in terms of speed and accuracy for the garbage dumps detection task.

Synthetic aperture radar (SAR) image simulation can provide SAR target images under different scenes and imaging conditions at a low cost. These simulation images can be applied to SAR target recognition, image interpretation, 3-D reconstruction, and many other fields. With the accumulation of high-resolution SAR images of targets under different imaging conditions, the simulation process should be benefited from these real images. Accurate simulation parameters are one of the keys to obtain high-quality simulation images. However, it takes a lot of time, energy, and resources to get simulation parameters from actual target measurement or adjusting manually. It is difficult to derive the analytical form of the relation between a SAR image and its simulation parameter, so nowadays the abundant real SAR images can hardly help the SAR simulation. In this article, a framework is proposed to obtain the relationship between SAR images and simulation parameters by training the deep neural network (DNN), so as to extract the simulation parameters from the real SAR image. Two DNNs, convolutional neural network (CNN), and generative adversarial network (GAN) are used to implement this framework. By modifying the network structures and setting reasonable training data, our DNNs can learn the relationship between image and simulation parameters more effectively. Experimental results show that the DNNs can extract the simulation parameters from the real SAR image, which can further improve the similarity of the simulation image while automating the setting of simulation parameters. Compared with CNN, the simulation parameters extracted by GAN can achieve better results at multiple azimuth angles.

Recently, the deep convolutional neural network has brought great improvements in object detection. However, the balance between high accuracy and high speed has always been a challenging task in multiclass object detection for large-scale remote sensing imagery. One-stage methods are more widely used because of their high efficiency but are limited by their performances on small object detection. In this article, we propose a unified framework called feature-merged single-shot detection (FMSSD) network, which aggregates the context information both in multiple scales and the same scale feature maps. First, our network leverages the atrous spatial feature pyramid (ASFP) module to fuse the context information in multiscale features by using feature pyramid and multiple atrous rates. Second, we propose a novel area-weighted loss function to pay more attention to small objects, while the replaced original loss treats all objects equally. We believe that small objects should be given more weight than large objects because they lose more information during training. Specifically, a monotonic decreasing function about the area is designed to add weights on the loss function. Extensive experiments on the DOTA data set and NWPU VHR-10 data set demonstrate that our method achieves state-of-the-art detection accuracy with high efficiency. We also build a new large-scale data set called AIR-OBJ data set from Google Earth and show the detection results of small objects, which validates the effectiveness on large-scale remote sensing imagery.

Object detection has been a building block in computer vision. Though considerable progress has been made, there still exist challenges for objects with small size, arbitrary direction, and dense distribution. Apart from natural images, such issues are especially pronounced for aerial images of great importance. This paper presents a novel multi-category rotation detector for small, cluttered and rotated objects, namely SCRDet. Specifically, a sampling fusion network is devised which fuses multi-layer feature with effective anchor sampling, to improve the sensitivity to small objects. Meanwhile, the supervised pixel attention network and the channel attention network are jointly explored for small and cluttered object detection by suppressing the noise and highlighting the objects feature. For more accurate rotation estimation, the IoU constant factor is added to the smooth L1 loss to address the boundary problem for the rotating bounding box. Extensive experiments on two remote sensing public datasets DOTA, NWPU VHR-10 as well as natural image datasets COCO, VOC2007 and scene text data ICDAR2015 show the state-of-the-art performance of our detector. The code and models will be available at https://github.com/DetectionTeamUCAS.

Semantic labeling in high-resolution aerial images is important for its wide range of applications. In this paper, we propose an end-to-end dual multi-scale feature fusion network (DMFFNet) for high-resolution aerial multi-source images. DMFFNet aims to further improve the semantic labeling results of the region where the multispectral features are indistinguishable. Specifically, we design a channel fusion strengthen (CFS) module, which can fuse features adaptively by modelling interdependencies between channels. Furthermore, a multiscale context aggregation (MCA) module is utilized to obtain larger receptive field and more contextual information. The experiment results confirm the DMFFNet with CFS and MCA improve the semantic labeling performance by utilizing multi-source data.

In this paper, we achieve a semantic segmentation of aerial imagery based on the fusion of multi-modal data in an effective way. The multi-modal data contains a true orthophoto and the corresponding normalized Digital Surface Model (nDSM), which are stacked together before they are fed into a Convolutional Neural Network (CNN). Though the two modalities are fused at the early stage, their features are learned independently with group convolutions firstly and then the learned features of different modalities are fused at multiple scales with standard convolutions. Therefore, the multi-scale fusion of multi-modal features is completed in a single-branch convolutional network. In this way, the computational cost is reduced while the experimental results reveal that we can still get promising results.

The local climate zone (LCZ) classification divides the urban areas into 17 categories, which are composed of 10 manmade structures and 7 natural landscapes. Though originally designed for temperature study, LCZ classification can be used for studies on economy and population. In this paper, we achieve a LCZ classification with convolutional neural networks based on the multi-source remote sensing data, including the polarimetric synthetic aperture radar (PolSAR) data and the corresponding multi-spectral imagery (MSI). Through experiments we attempt to reveal the contributions of the SAR data and the MSI to the classification performance. Furthermore, we emphasize the crucial importance of the preprocessing on the training data to derive a balanced dataset. We are ranked second in the Tianchi competition rankings when we submit our results.

Object detection has been playing a significant role in the field of remote sensing for a long period while it is still full of challenges. The biggest one is how to detect multi-scale objects with high accuracy and fast speed in remote sensing images. One-stage object detectors have been achieving relatively high accuracy and efficiency with small memory footprint. However, they have a not very well performance on small objects. In this paper, we discuss the importance of the context information between feature maps in different scales which is helpful for detecting small objects. Especially, we propose a Feature-merged detection networks (MergeNet), which can be inserted into the one-stage detectors easily, to unify the multi-scale feature and context information effectively. Experiments on DOTA dataset demonstrate that our model can significantly improve the performance of the one-stage method.

Deep learning based object detection has achieved great success. However, these supervised learning methods are data-hungry and time-consuming. This restriction makes them unsuitable for limited data and urgent tasks, especially in the applications of remote sensing. Inspired by the ability of humans to quickly learn new visual concepts from very few examples, we propose a training-free, one-shot geospatial object detection framework for remote sensing images. It consists of (1) a feature extractor with remote sensing domain knowledge, (2) a multi-level feature fusion method, (3) a novel similarity metric method, and (4) a 2-stage object detection pipeline. Experiments on sewage treatment plant and airport detections show that proposed method has achieved a certain effect. Our method can serve as a baseline for training-free, one-shot geospatial object detection.

Image collection summarisation aims to represent a large‐scale multi‐modal collection with a small subset of images and tags, helping navigate a large image dataset. Most extant methods leverage the contributions of text‐to‐visual summaries, ignoring the visual contribution to the textual topic. When the tags are weakly labelled, the textual topic cannot accurately reflect the visual summary. To solve this, the authors propose a novel model, joint optimisation of convex non‐negative matrix factorisation, which incorporates images and tags in a beneficial way. The objective function contains visual and textual error functions, sharing the same indicator matrix, connecting different modal relations. Then, they propose an iterative algorithm to optimise the proposed model. Finally, they explore the effects of different visual feature representations (e.g. bag‐of‐words and deep learning) on multi‐modal collection summary. Our proposed method is then compared with state‐of‐the‐art algorithms using two multi‐modal datasets (i.e. MIRFlickr and NUS‐WIDE‐SCENE). Experimental results demonstrate the effectiveness of their proposed approach.

Semantic segmentation in high-resolution aerial images is a fundamental research problem in remote sensing field for its wide range of applications. However, it is difficult to distinguish regions with similar spectral features using only multispectral data. Recent research studies have indicated that the introduction of multisource information can effectively improve the robustness of segmentation method. In this letter, we use digital surface models (DSMs) information as a complementary feature to further improve the semantic segmentation results. To this end, we propose a lightweight and simple DSM fusion (DSMF) branch structure module. Compared with the existing feature extraction structures, proposed DSMF module is simple and can be easily applied to other networks. In addition, we investigate four fusion strategies based on DSMF module to explore the optimal feature fusion strategy and four end-to-end DSMFNets are designed according to the corresponding strategies. We evaluate our models on International Society for Photogrammetry and Remote Sensing Vaihingen data set and all DSMFNets achieve promising results. In particular, DSMFNet-1 achieves an overall accuracy of 91.5% on the test data set.

In this letter, we propose an unsupervised change detection method for synthetic aperture radar (SAR) images based on variational multivariate Gaussian mixture model (MGMM) and Shannon entropy. First, the difference features are generated from the Gabor wavelet transform of two SAR images. In variational inference framework, the variational MGMM is first introduced to implement accurate modeling for the data distribution of difference features and to output responsibilities. Subsequently, spatial information is explored on the responsibilities to yield thecontextual responsibilitiesfor improving the accuracy and reliability of change detection. Then,a posterioriprobabilities of the changed and unchanged classes are derived from thecontextual responsibilities, and Shannon entropy, being directly related to the classification error rate, is proposed to determine the optimal index integer. Finally, the binary change mask is achieved by separating the pixels into the changed and unchanged classes. The experiments on three pairs of SAR images for describing urban sprawl and water bodies demonstrate the effectiveness of the proposed method.

Object detection has been playing a significant role in the field of remote sensing for a long time but it is still full of challenges. In this paper, we propose a novel detection framework based on rotational region convolution neural network to cope with the problem of non-maximum suppression in dense objects detection. The bounding boxes obtained by adopting our method is the minimum bounding rectangle of object with less redundant regions. Furthermore, we find the head direction of the object through prediction. There are three important changes to our framework over traditional detection methods, representation and regression of rotational bounding box, head direction prediction and rotational non-maximal suppression. Experiments based on remote sensing images from Google Earth for Object detection show that our detection method based on rotational region CNN has a competitive performance.

Deeper architectures are proven to be beneficial for the classification performance obviously in computer vision field. Inspired by this, deep CNN s are expected to make progress in the SAR target classification problem as well. However, it is hard to train deeper CNNs for SAR images. Such CNNs have millions of parameters to be determined in the network (for example the VGGNet has more than 130 million parameters), hence large-scale dataset is indispensable when training a deep CNN. But there is no large-scale annotated SAR target dataset, and data acquisition and annotation is much more costly for SAR images. With inadequate data, the network is easy to be overfitting. Several methods based on deep learning have been proposed for SAR image classifications, but they cannot get rid of the aforementioned data limitation of labelled SAR images. To solve this problem, this paper proposes a microarchitecture called CompressUnit (CU). With CU, we design a deeper CNN. Compared with the network with the fewest parameters for SAR image classification in literature so far, our network is 2X deeper with only about 10% of parameters. In this way, we get a deeper network with much fewer parameters. This network is easier to be trained with limited SAR data and is more likely to get rid of overfitting.

In this paper, we focus on the use of multi-modal data to achieve a semantic segmentation of aerial imagery. Thereby, the multi-modal data is composed of a true orthophoto, the Digital Surface Model (DSM) and further representations derived from these. Taking data of different modalities separately and in combination as input to a Residual Shuffling Convolutional Neural Network (RSCNN), we analyze their value for the classification task given with a benchmark dataset. The derived results reveal an improvement if different types of geometric features extracted from the DSM are used in addition to the true orthophoto.

In this paper, a novel framework for aircraft detection in high resolution apron area in Synthetic Aperture Radar (SAR) images is proposed, which combines the strength of location regression based convolutional neural network (CNN) framework and the salient features of target in SAR images. Specifically, a Constant False Alarm Rate (CFAR) based target pre-locating algorithm is introduced, which can match the scale of target in SAR images more accurate compared to the existing region proposal method. In addition, in order to eliminate the fact of overfitting, we explore several strategies for SAR data augmentation, including translation, adding noise and rotation within a small range. Experiments are conducted on the data set acquired by the TerraSAR-X satellite in a resolution of 3.0 meters. The results show that the proposed detection framework could effectively obtain a more accurate detection result.

Road extraction from high-resolution remote sensing images has been applied in many domains, but it is still full of challenges. We focus on the problem of slender roads, proposing a new multiple feature pyramid network (MFPN), which is composed of an effective feature pyramid and the tailored pyramid pooling module based on PSPNet. These two designs can address the sparsity of roads in remote sensing images via using multi-level semantic features. Experiments on remote sensing images from Quick Bird show that our MFPN model achieves competitive performance, especially for slender roads.

Through the recent performance of convolutional neural networks in image processing tasks, we propose a deep fully convolutional network for remote sensing image inpainting. The proposed Dense-Add Net (Dense-Add Network) can alleviate the vanishing-gradient problem, strengthen feature reuse, and substantially reduce the memory usage. We apply residual learning to learn the mappings from corrupted image to recovered image directly; it will back-propagate gradient to the bottom layers and accelerate the training process. We train the proposed Dense-Add Net with a robust Charbonnier loss function which can achieve high-quality reconstruction. The experimental verify the efficacy of our proposed Dense-Add Net.

Semantic segmentation of aerial images refers to assigning one land cover category to each pixel. This is a challenging task due to the great differences in the appearances of ground objects. Many attempts have been made during the past decades. In recent years, convolutional neural networks (CNNs) have been introduced in the remote sensing field, and various solutions have been proposed to realize dense semantic labeling with CNNs. In this letter, we propose shuffling CNNs to realize semantic segmentation of aerial images in a periodic shuffling manner. This approach is a supplement to current methods for semantic segmentation of aerial images. We propose a naive version and a deeper version of this method, and both are adept at detecting small objects. Additionally, we propose a method called field-of-view (FoV) enhancement that can enhance the predictions. This method can be applied to various networks, and our experiments verify its effectiveness. The final results are further improved through an ensemble method that averages the score maps generated by the models at different checkpoints of the same network. We evaluate our models using the ISPRS Vaihingen and Potsdam data sets, and we acquire promising results using these two data sets.

Water-body segmentation is an important issue in remote sensing and image interpretation. Classic methods for counteracting this problem usually include the construction of index features by combining different spectra, however, these methods are essentially rule-based and fail to take advantage of context information. Additionally, as the quality of image resolution improves, these methods are proved to be inadequate. With the rise of convolutional neural networks (CNN), the level of research about segmentation has taken a huge leap, but the field is still facing an increasing demand for data and the problem of blurring boundaries. In this letter, a new segmentation network called restricted receptive field deconvolution network (RRF DeconvNet) is proposed, with which to extract water bodies from high-resolution remote sensing images. Compared with natural images, remote sensing images have a weaker pixel neighborhood relativity; in consideration of this challenge, an RRF DeconvNet compresses the redundant layers in the original DeconvNet and no longer relies on a pretrained model. In addition, to tackle the blurring boundaries that occur in CNN, a new loss function called edges weighting loss is proposed to train segmentation networks, which has been shown to significantly sharpen the segmentation boundaries in results. Experiments, based on Google Earth images for water-body segmentation, are presented in this letter to prove our method.

Cloud and cloud shadow detection in remote sensing imagery is important for its wide range of applications. Traditionally, the detection is usually based on the manually designed thresholds from multiband, which is complicated and of multistage. To simplify the process of cloud and cloud shadow detection and improve the performance, we propose a multilevel feature fused segmentation network (MFFSNet), which can be trained end-to-end without any hand-tuned parameters. Specifically, a fully convolutional network is proposed for cloud and cloud shadow features learning. Then, we utilize a novel pyramid pooling module to extract contextual relation between cloud and shadow. Furthermore, a special multilevel feature fused structure is designed to combine semantic information with spatial information from different levels, so that we can better handle the multiscale objects and produce detailed segmentation boundaries. Experiments show that the MFFSNet outperforms the state-of-the-art methods and achieves high accuracies of 98.69% and 98.92% for cloud and cloud shadow detection.

Aircraft type recognition in remote sensing images is a meaningful task. It remains challenging due to the difficulty of obtaining appropriate representation of aircrafts for recognition. To solve this problem, we propose a novel aircraft type recognition framework based on deep convolutional neural networks. First, an aircraft segmentation network is designed to obtain refined aircraft segmentation results which provide significant details to distinguish different aircrafts. Then, a keypoints' detection network is proposed to acquire aircrafts' directions and bounding boxes, which are used to align the segmentation results. A new multirotation refinement method is carefully designed to further improve the keypoints' precision. At last, we apply a template matching method to identify aircrafts, and the intersection over union is adopted to evaluate the similarity between segmentation results and templates. The proposed framework takes advantage of both shape and scale information of aircrafts for recognition. Experiments show that the proposed method outperforms the state-of-the-art methods and can achieve 95.6% accuracy on the challenging data set.

Recently, convolutional neural networks (CNN) have achieved impressive breakthroughs in single image superresolution. In particular, an efficient nonlinear mapping by increasing the depth and width of the network can be learned between the low-resolution input image and the high-resolution target image. However, this will lead to a substantial increase in network parameters, requiring the massive amount of training data to prevent overfitting. Besides, most CNN-based methods ignore the full use of different levels of features and, therefore, achieve relatively low performance. In this letter, we propose a deep convolutional network named densely connected residual networks (DRNet). Our proposed DRNet can reach very deep and wide while requiring fewer parameters. The significant performance improvement of our model is mainly due to the integration of dense skip connection and residual learning. In this way, DRNet mitigates the problems of overfitting, vanishing gradient, and training instability during training very deep and wide networks. Moreover, it can improve the propagation and reuse of features by creating direct connections from the previous layers to the subsequent layers. We evaluate the proposed method using images from four benchmark datasets and set a new state of the art.

Recently, low-rank embedding (LRE) has yielded satisfactory results in dimensionality reduction (DR), for which low-rank representation and projection learning are integrated into one model to generate robust low-dimensional features. However, LRE requires to convert samples into vectors even if the data naturally appear in high-order form. Furthermore, LRE fails to take the label information into consideration. To address these problems, this paper proposes a novel supervised DR method based on multilinear algebra, i.e., the algebra of tensors. By the motivation of extending LRE into tensor space and simultaneously combining the tensor discriminant analysis, we establish tensor low-rank discriminant embedding (TLRDE) model for hyperspectral image (HSI) DR. The model of TLRDE is solved by an alternative iteration algorithm, whose convergence is also mathematically proven. The proposed TLRDE method employs the tensor representation to preserve the intrinsic geometrical structure, uses low-rank reconstruction to uncover the potential relationship among the data points, and combines label information to enhance the discriminability of features. Moreover, the proposed TLRDE does not suffer from the small sample size problem. The experimental results on three real HSI data sets validate the effectiveness of our proposed TLRDE method.

This paper addresses the unsupervised classification problems for multilook Polarimetric synthetic aperture radar (PolSAR) images by proposing a patch-level spatially variant Wishart mixture model (SVWMM) with double constraints. We construct this model by jointly modeling the pixels in a patch (rather than an individual pixel) so as to effectively capture the local correlation in the PolSAR images. More importantly, a responsibility parameter is introduced to the proposed model, providing not only the possibility to represent the importance of different pixels within a patch but also the additional flexibility for incorporating the spatial information. As such, double constraints are further imposed by simultaneously utilizing the similarities of the neighboring pixels, respectively, defined on two different parameter spaces (i.e., the hyperparameter in the posterior distribution of mixing coefficients and the responsibility parameter). Furthermore, the variational inference algorithm is developed to achieve effective learning of the proposed SVWMM with the closed-form updates, facilitating the automatic determination of the cluster number. Experimental results on several PolSAR data sets from both airborne and spaceborne sensors demonstrate that the proposed method is effective and it enables better performances on unsupervised classification than the conventional methods.

In this paper, we propose a top-down building reconstruction technique based on layover modelling and MCMC method. Through representing the layover with parameterized geometrical models, the problem is converted into an optimization problem under the Bayesian scheme. The energy function consists of two parts: region part and edge part. In order to obtain global optima, simulated annealing algorithm with MCMC is used in the optimization stage. Two groups of transmission kernels which are responsible for model updates are designed according to the model. This method is tested both on simulated SAR image and HR TanDEM-X data. At this moment, only qualitative analysis for this method is provided. It proves the effectiveness of the presented method. Detailed quantitative evaluation will be added when we submit the final version of this paper.

Building extraction from remote sensing images is a longstanding topic in land use analysis and applications of remote sensing. Variations in shape and appearance of buildings, occlusions and other unpredictable factors increase the hardness of automatic building extraction. Numerous methods have been proposed during the last several decays, but most of these works are task oriented and lack of generalization. This paper applys deep learning to building extraction in a supervised manner. A deep deconvolution neural network with 27 Convolution/Deconvolution weight layers is designed to realize building extraction in pixel level. As such a deep network is prone to overfitting, a data augment method that suits pixel-wise prediction tasks in remote sensing is suggested. Moreover, an overall training and inferencing architecture is proposed. Our methods are finally applied to building extraction tasks and get competitive results with other methods published.

Automated annotation of urban areas from overhead imagery plays an essential role in many remote sensing applications. Generative Adversarial Nets (GANs) is one of the most effective ways to handle this problem. In this manuscript, two tricks were added in conditional GANs(cGANs) which learn the mapping from input image to output remote sensing image. All the experimental results demonstrated that cGANs was a reliable way to generate high-quality remote sensing images. What's more, when this method be applied to semantic segmentation and accurate classification was made by using ISPRS 2D semantic labelling challenge dataset.

Automatic inshore ship recognition, which includes target localization and type recognition, is an important and challenging task. However, existing ship recognition methods mainly focus on the classification of ship samples or clips. These methods rely deeply on the detection algorithm to complete localization and recognition in large scene images. In this letter, we present an integrated framework to automatically locate and recognize inshore ships in large scene satellite images. Different from traditional object recognition methods using two steps of detection-classification, the proposed framework could locate inshore ships and identify types without the detection step. Considering ship size is a useful feature, a novel multimodel method is proposed to utilize this feature. And an Euclidean-distance-based fusion strategy is used to combine candidates given by models. This fusion strategy could effectively separate side-by-side ships. To handle large scene images efficiently, scale-invariant feature transform registration is also integrated into the framework to utilize geographic information. All of these make the framework an end-to-end fashion which could automatically recognize inshore ships in large scene satellite images. Experiments on Quickbird images show that this framework could achieve the actual applied requirements.

With the development of deep learning, supervised learning has frequently been adopted to classify remotely sensed images using convolutional networks. However, due to the limited amount of labeled data available, supervised learning is often difficult to carry out. Therefore, we proposed an unsupervised model called multiple-layer feature-matching generative adversarial networks (MARTA GANs) to learn a representation using only unlabeled data. MARTA GANs consists of both a generative model G and a discriminative model D. We treat D as a feature extractor. To fit the complex properties of remote sensing data, we use a fusion layer to merge the mid-level and global features. G can produce numerous images that are similar to the training data; therefore, D can learn better representations of remotely sensed images using the training data provided by G. The classification results on two widely used remote sensing image databases show that the proposed method significantly improves the classification performance compared with other state-of-the-art methods.

In this letter, we propose a novel supervised change detection method based on a deep siamese convolutional network for optical aerial images. We train a siamese convolutional network using the weighted contrastive loss. The novelty of the method is that the siamese network is learned to extract features directly from the image pairs. Compared with hand-crafted features used by the conventional change detection method, the extracted features are more abstract and robust. Furthermore, because of the advantage of the weighted contrastive loss function, the features have a unique property: the feature vectors of the changed pixel pair are far away from each other, while the ones of the unchanged pixel pair are close. Therefore, we use the distance of the feature vectors to detect changes between the image pair. Simple threshold segmentation on the distance map can even obtain good performance. For improvement, we use a k-nearest neighbor approach to update the initial result. Experimental results show that the proposed method produces results comparable, even better, with the two state-of-the-art methods in terms of F-measure.

Detecting artificial targets, such as aircraft, in satellite images is significant in military and civil applications. Although the performance has improved with the use of more complicated features and better learning methods, effectively handling aircraft with variations of type, pose, and size is still very challenging. To solve this problem, we propose a multiscale sliding-window framework based on aggregate channel features, well-designed features that contain rich information. We also employ a fast feature pyramids algorithm to accelerate multiscale aircraft detection. In this framework, features are trained by Cascade AdaBoost including multiple rounds of bootstrapping that leads to improved overall accuracy. A two-step nonmaximum suppression algorithm is carefully designed based on a given set of detections. Our method shows a competitive performance on the QuickBird images of 0.6 m resolution.

Aircraft type recognition of remote sensing images is critical both in civil and military applications. In this letter, we propose a novel landmark-based aircraft recognition method which is highly accurate and efficient. First, we propose a new idea to address the aircraft type recognition problem by aircraft's landmark detection. Its advantages are two folds. On the one hand, it needs fewer labeled data and alleviates the work of human annotation. On the other hand, a trained model has strong expansibility because it can be used for any type of aircraft that not contained in training data set without retraining. Then, we use a variant of a convolutional neural network called vanilla network for all landmarks regression at the same time. Therefore, it can avoid bad local minimum effectively by encoding the geometric constraints among landmarks implicitly. To handle aircrafts in myriads of poses, rotation jittering is used for data augmentation in preprocessing and multicrop fusion is used in postprocessing. Thus, an 80% reduction in error rate could be reached. Finally, we use the landmark template matching to recognize the aircraft. Our method shows a competitive performance both in accuracy and efficiency.

In weakly supervised object detection, conventional methods treat object location in each image as a latent variable and use non-convex optimization to solve the latent variable. However, as the optimization objective is image-level instead of sample-level, the learning procedure tends to choose object parts as false positive samples. Furthermore, when multiple classes of objects appear in the same images, the models could invite class-correlations and lose discriminative capability. In this paper, we propose a simple but effective suppression strategy that mines hard negative samples in the learning procedure to ease the above problems. We propose using a spatial-voting strategy to help finding negative samples to suppress the impact of object parts. We also use regions from class-correlated images as negative samples to suppress the impact of class-correlations. Experiments show that our approach significantly improves the baseline by 6% and achieves state-of-the-art performance.

Scattering structure features of targets is of great importance for Synthetic Aperture Radar (SAR) image analysis. In this paper, a novel algorithm for aircraft recognition in high resolution apron area of SAR images is proposed. The algorithm combines the strength of gradient saliency map and scattering structure features to improve accuracy and efficiency. Specially, Constant False-Alarm Rate (CFAR) algorithm is carried out to segment images. Then, a new efficient object locating method based on directional local gradient map is proposed to detect aircraft targets. Then, the candidate slices as well as template slices are modeled using Gaussian Mixture Model (GMM), which will be treated as structure features. In the recognition stage, a novel similarity measurement algorithm based on Kullback-Leibler Divergence for GMM models is proposed for classification. We conduct experiments on the dataset with 3.0m resolution and the recognition results demonstrate the accuracy of our proposed method.

In this paper, a fully automatic building reconstruction method for high resolution interferometric synthetic aperture radar (InSAR) data is presented. This method is based on stochastic geometrical model. Firstly, a building detection procedure is implemented on the big image and the entire scene is divided into building clips. After that, the reconstruction process is utilized for each building clip. In the reconstruction process, a building in 3D space is projected to the image plane and then decomposed to feature regions including layover, corner line, roof and shadow. We explore the statistic properties of the each region, and include it in the posterior function, together with the edge term and the prior we defined. Finally, in order to overcome local optima, a group of special transmission kernels are designed. The experimental results on TanDEM-X data demonstrate the effectiveness of our method.

The edges of the shadow region are blurred in the SAR image due to the moving of the radar during data collection. This phenomenon becomes obvious in the High Resolution SAR images. Shadow enhancement is of great value for ATR especially when the scattering centers of the target itself are not clear. In this paper, an approach for shadow enhancement in the SAR images for targets with plat structures is presented. And experiments on the Mini-SAR data test the validity of the approach.

The objective of this paper is to propose a hierarchical topic evolution model (HTEM) that can organize time‐varying topics in a hierarchy and discover their evolutions with multiple timescales. In the proposed HTEM, topics near the root of the hierarchy are more abstract and also evolve in the longer timescales than those near the leaves. To achieve this goal, the distance‐dependent Chinese restaurant process (ddCRP) is extended to a new nested process that is able to simultaneously model the dependencies among data and the relationship between clusters. The HTEM is proposed based on the new process for time‐stamped documents, in which the timestamp is utilized to measure the dependencies among documents. Moreover, an efficient Gibbs sampler is developed for the proposed HTEM. Our experimental results on two popular real‐world data sets verify that the proposed HTEM can capture coherent topics and discover their hierarchical evolutions. It also outperforms the baseline model in terms of likelihood on held‐out data.

Object detection has been one of the hottest issues in the field of remote sensing image analysis. In this letter, an efficient object detection framework is proposed, which combines the strength of the unsupervised feature learning of deep belief networks (DBNs) and visual saliency. In particular, we propose an efficient coarse object locating method based on a saliency mechanism. The method could avoid an exhaustive search across the image and generate a small number of bounding boxes, which can locate the object quickly and precisely. After that, the trained DBN is used for feature extraction and classification on subimages. The feature learning of the DBN is operated by pretraining each layer of restricted Boltzmann machines (RBMs) using the general layerwise training algorithm. An unsupervised blockwise pretraining strategy is introduced to train the first layer of RBMs, which combines the raw pixels with a saliency map as inputs. This makes an RBM generate local and edge filters. The precise edge position information and pixel value information are more efficient to build a good model of images. Comparative experiments are conducted on the data set acquired by QuickBird with a 60-cm resolution. The results demonstrate the accuracy and efficiency of our method.

Layover in slant range synthetic aperture radar (SAR) images contains rich 3-D information of building geometry. In this letter, a coarse-to-fine method for building reconstruction from high-resolution (HR) SAR layover map using restricted parametric geometrical models is presented. First, we propose a new restricted parametric geometrical model for building layover and for corner line, respectively. Under the guidance of these models, a hierarchical coarse-to-fine layover estimation scheme is designed. Owning to the coarse-to-fine scheme, this method is resistant to various flaws of layover. At last, the building is reconstructed from the well-estimated layover. Experiments on HR TanDEM-X data demonstrate the effectiveness and precision of our method.

Detecting objects in aerial images is challenged by variance of object colors, aspect ratios, cluttered backgrounds, and in particular, undetermined orientations. In this paper, we propose to use Deep Convolutional Neural Network (DCNN) features from combined layers to perform orientation robust aerial object detection. We explore the inherent characteristics of DC-NN as well as relate the extracted features to the principle of disentangling feature learning. An image segmentation based approach is used to localize ROIs of various aspect ratios, and ROIs are further classified into positives or negatives using an SVM classifier trained on DCNN features. With experiments on two datasets collected from Google Earth, we demonstrate that the proposed aerial object detection approach is simple but effective.

Sentiment analysis mainly focuses on the study of one's opinions that express positive or negative sentiments. With the explosive growth of web documents, sentiment analysis is becoming a hot topic in both academic research and system design. Fine‐grained sentiment analysis is traditionally solved as a 2‐step strategy, which results in cascade errors. Although joint models, such as joint sentiment/topic and maximum entropy (MaxEnt)/latent Dirichlet allocation, are proposed to tackle this problem of sentiment analysis, they focus on the joint learning of both aspects and sentiments. Thus, they are not appropriate to solve the cascade errors for sentiment analysis at the sentence or subsentence level. In this article, we present a novel jointly fine‐grained sentiment analysis framework at the subsentence level with Markov logic. First, we divide the task into 2 separate stages (subjectivity classification and polarity classification). Then, the 2 separate stages are processed, respectively, with different feature sets, which are implemented by local formulas in Markov logic. Finally, global formulas in Markov logic are adopted to realize the interactions of the 2 separate stages. The joint inference of subjectivity and polarity helps prevent cascade errors. Experiments on a Chinese sentiment data set manifest that our joint model brings significant improvements.

Automatic aircraft recognition is a challenging task. Conventional methods always extract the overall shapes of aircraft at first and then represent the aircraft based on the extracted shape with different features for recognition. The major problem of these methods is that they have a high requirement on shape extraction, which is too idealistic for targets in satellite images. In this letter, we propose a new aircraft recognition approach that can recognize aircraft robustly without perfect extraction of silhouette or shape of aircraft as a precondition, and can deal with the situation of parts missing and shadow disturbance. Specifically, a direction estimation method is proposed first to align aircraft to a same direction. Then, a reconstruction-based similarity measure is proposed, which transforms the type recognition problem into a reconstruction problem. Finally, a jigsaw matching pursuit algorithm is proposed to solve the reconstruction problem. We use panchromatic Quickbird imagery for evaluation, and the experimental results illuminate that the proposed method is effective and accurate.

In this letter, we propose a novel method to reconstruct accurate building structures from high-resolution multiview aerial imagery, using layered contour fitting (LCF) with a density-based clustering algorithm. Initially, the complicated 3-D scene is reconstructed by a probabilistic volumetric modeling algorithm. Subsequently, the reconstructed 3-D scene model is projected into layer images based on the height information. At last, we combine an extended layered density-based clustering approach with a generative LCF approach to remove noise and extract accurate building contours in every layer image at the same time. The final accurate 3-D building model is generated from these contours in layer images with a smoothing operation. Experiments on the aerial image sets demonstrate effectiveness and precision of our method.

Image quality assessment (IQA) aims to devise perceptual models to predict the image quality consistently with human subjective evaluation. The representative metrics focus on measuring the image quality with low-level features. In this letter, we assumed that the distortion in specific regions containing semantically significant objects would be enhanced by HVS significantly. According to this hypothesis, a novel IQA metric based on a commonly used object-detecting feature, Speed Up Robust Features (SURF), was proposed. First, it determined the interest points which represented significant objects through the SURF features both on the reference image and distorted image. Then it computed the multilevel SURF descriptors differences between the reference image and the distorted one. Finally, all the difference results were combined with a suitable pooling strategy. Comparing with other nine state-of-the-art IQA models on three biggest IQA databases, SURF-SIM demonstrated its highly competitive prediction accuracy especially on complicated applications and excellent robustness across different distortion types.

In this letter, we present a new method to detect inshore ships using shape and context information. We first propose a new energy function based on an active contour model to segment water and land and minimize it with an iterative global optimization method. The proposed energy performs well on the different intensity distributions between water and land and produces a result that can be well used in shape and context analyses. In the segmented image, ships are detected with successive shape analysis, including shape analysis in the localization of ship head and region growing in computing the width and length of ship. Finally, to locate ships accurately and remove the false alarms, we unify them with a binary linear programming problem by utilizing the context information. Experiments on QuickBird images show the robustness and precision of our method.

In this letter, we propose a new detection framework based on robust invariant generalized Hough transform (RIGHT) to solve the problem of detecting inshore ships in high-resolution remote sensing imagery. The invariant generalized Hough transform is an effective shape extraction technique, but it is not adaptive to shape deformation well. In order to improve its adaptability, we use an iterative training method to learn a robust shape model automatically. The model could capture the shape variability of the target contained in the training data set, and every point in the model is equipped with an individual weight according to its importance, which greatly reduces the false-positive rate. Through the iteration process, the model performance is gradually improved by extending the shape model with these necessary weighted points. Experimental result demonstrates the precision, robustness, and effectiveness of our detection framework based on RIGHT.

In this letter, we propose a rotation invariant parts-based model to detect objects with complex shape in high-resolution remote sensing images. Specifically, the geospatial objects with complex shape are firstly divided into several main parts, and the structure information among parts is described and regulated in polar coordinates to achieve the rotation invariance on configuration. Meanwhile, the pose variance of each part relative to the object is also defined in our model. In encoding the features of the rotated parts and objects, a new rotation invariant feature is proposed by extending histogram oriented gradients. During the final detection step, a clustering method is introduced to locate the parts in objects, and that method can also be used to fuse the detection results. By this way, an efficient detection model is constructed and the experimental results demonstrate the robustness and precision of our proposed detection model.

For high-resolution remote sensing image classification tasks, multiple features are usually required for better performances since single visual feature is valid only in describing one pattern of images. In this letter, we propose a novel Semi-Supervised Manifold learning based Multigraph Fusion framework (SSM-MF), in which multiple features are combined to learn a low-dimensional subspace. The obtained subspace can effectively characterize the semantic information of the features and thus benefits classification. Our framework employs a semi-supervised manner by exploiting labeled and unlabeled data and therefore enjoy three advancements: 1) discriminative information and geometric information in labeled data and the structural information in unlabeled data can be jointly utilized to enhance manifold learning; 2) our framework explores the complementary of multiple features and meanwhile avoids the curse of dimensionality; and 3) our semi-supervised learning mode makes use of information in abundant unlabeled data in real-world applications. Experiments on a remote sensing image data set validate the effectiveness of our proposed method.

In this letter, we propose a semisupervised context-sensitive technique for change detection in high-resolution multitemporal remote sensing images. This is achieved by analyzing the posterior probability of probabilistic Gaussian process (GP) classifier within a Markov random field (MRF) model. In particular, the method consists of two steps: 1) A semisupervised initialization exploits both labeled and unlabeled data based on a probabilistic GP classifier, and 2) an MRF regularization aims at refining the posterior probability by employing the spatial context information. In particular, both edge information and high-order potential are utilized in MRF energy function formulation. Experimental results obtained on real remote sensing multitemporal imagery data sets confirm the effectiveness of the proposed approach.

Automatic aircraft recognition in high-resolution satellite images has many important applications. Due to the diversity and complexity of fore-/background, recognition using pixel-based methods usually does not perform well. In this letter, we propose a new method integrating the high-level information of a shape prior, which is considered as a coarse-to-fine process. In the coarse stage, the pose of an aircraft is roughly estimated by a single template matching with a defined score criterion. In the fine stage, we derive a parametric shape model by applying principal component analysis and kernel density function, which have good effects on both dimension reduction and sample space description; then, a new variational formulation combining region information and a shape prior is proposed to segment the object using a level set method. Finally, the parameters of the segmentation result are directly applied to verify aircraft type with two$k$-nearest neighbor steps. Experiments on QuickBird images demonstrate the robustness and accuracy of the proposed method.

High-resolution remote-sensing image classification is a challenging task. In this letter, we first propose a bag-of-features (BOF) model-based classification framework for high-resolution remote-sensing images via Earth mover's distance (EMD) to perform histogram matching. Compared with conventional BOF, EMD-based BOF is insensitive to vector quantization and can explore the relations among visual codes. In addition, such relations can be utilized as a key discriminative feature for image classification task. However, EMD is not practically utilized because of expensive computational cost. Motivated by Pele and Werman, we propose a faster approximate EMD (AEMD), and our AEMD-based BOF can inherit the advantages of EMD. Experimental results on a multicategory remote-sensing image data set demonstrate the effectiveness of our classification framework.

In this letter, we propose a novel framework for large-satellite-image annotation using multifeature joint sparse coding (MFJSC) with spatial relation constraint. The MFJSC model imposes anl1, 2-mixed-norm regularization on encoded coefficients of features. The regularization will encourage the coefficients to share a common sparsity pattern, which will preserve the cross-feature information and eliminate the constraint that they must have identical coefficients. Spatial dependences between patches of large images are useful for the annotation task but are usually ignored or insufficiently exploited in other methods. In this letter, we design a spatial-relation-constrained classifier to utilize the output of MFJSC and the spatial dependences to annotate images more precisely. Experiments on a data set of 21 land-use classes and QuickBird images show the discriminative power of MFJSC and the effectiveness of our annotation framework.

In this letter, a statistical edge detector based on the square successive difference of averages has been proposed and tested for SAR images. The operator employs the square successive of mean difference as the edge strength indicator for SAR images. It has been proved to be with constant false alarm rate and performs well in representation of many more region shapes. A postprocessing approach, including edge thinning and adaptive double-threshold processing, is proposed to refine the edge detection results. The performance of the proposed operator has been evaluated and compared with that of the Canny and ratio-of-average operators on simulated and real SAR images. The experimental results indicate that the operator achieves better performance in the detection rate and the localization accuracy, and the detected edges are more complete and longer than those by the other two operators.

In this letter, we propose a novel method based on bilateral filter (BF) scale-invariant feature transform (SIFT) (BFSIFT) to find feature matches for synthetic aperture radar (SAR) image registration. First, the anisotropic scale space of the image is constructed using BFs. The constructing process is noniterative and fast. Compared with the Gaussian scale space used in SIFT, more accurately located matches can be found in the anisotropic one. Then, keypoints are detected and described in the coarser scales using SIFT. At last, dual-matching strategy and random sample consensus are used to establish matches. The probability of correct matching is significantly increased by skipping the finest scale and by the dual-matching strategy. Experiments on various slant range images demonstrate the applicability of BFSIFT to find feature matches for SAR image registration.

In this letter, we propose a novel method to solve the problem of detecting geospatial objects present in high-resolution remote sensing images automatically. Each image is represented as a segmentation tree by applying a multiscale segmentation algorithm at first, and all of the tree nodes are described as coherent groups instead of binary classified values. The trees are matched to select the maximally matched subtrees, denoted as common subcategories. Then, we organize these subcategories to learn the embedded taxonomic semantics of objects categories, which allow categories to be defined recursively, and express both explicit and implicit spatial configuration of categories. Detection, recognition, and segmentation of the geospatial objects in a new image can be simultaneously conducted by using the learned taxonomic semantics. This procedure also provides a meaningful explanation for image understanding. Experiments for complex and compound objects demonstrate the precision, robustness, and effectiveness of the proposed method.

Many traditional target extraction methods encountered new challenges as the spatial resolution is increasing quickly. For the purpose of extracting buildings in that circumstance, a new method combing both the object-based approach and boosting algorithm is proposed in this paper. The method associates segmentation with recognition by constructing a hierarchical object network, which effectively improves the problem of detecting targets with a modifiable sliding window existed in other methods. And some useful features are selected automatically to train a validate classifier. Then the label confidence of each object is computed using contextual models to complete the extraction procedure. Competitive results for both multiform and complicated buildings demonstrate the precision, robustness and effectiveness of the proposed method.