Remote sensing instance segmentation aims to detect and assign pixel-level labels to each instance in remote sensing images, which holds critical engineering significance for both civil and military applications. While existing domain-specific methods have made progress, they still struggle with three persistent challenges: ineffective context modeling in cluttered backgrounds, information loss during multi-scale feature fusion, and blurred boundaries for densely clustered small objects. To address these limitations, we propose a novel remote sensing instance segmentation framework with three artificial intelligence (AI) methodological innovations, which comprises: a Context Perception Module (CPM) for context modeling, a Context Guided Multi-Scale Feature Aggregation (CGFA) method for multi-scale feature fusion, and a Multi-Path Region Proposal Extractor (MPRPE) with boundary-refined segmentation. The CPM leverages the selective state space model (Mamba) to capture long-range contextual information, effectively addressing the issue of cluttered backgrounds in remote sensing images. The CGFA replaces standard feature pyramid network architecture which is limited by direct summation or concatenation, preserving fine-grained spatial details with context guidance. The MPRPE and boundary-aware segmentation head mitigate the challenges of missed detection of small objects and blurred edge predictions, which arise from the clustered distribution of small objects and semantic ambiguity. Extensive experiments on the challenging iSAID and NWPU VHR-10 datasets validate the proposed method’s consistent improvements across metrics while demonstrating its practical engineering impact on remote sensing interpretation systems.

Large Language Models (LLMs) demonstrate various capabilities previously considered unique to humans. However, current evidence is insufficient to determine whether LLMs have developed spatial cognition, a fundamental aspect of human cognition underpinning logical-mathematical reasoning and various other skills. Previous studies on this topic have primarily concentrated on small-scale perceptions, leaving the spatial cognition within the context of GIScience largely unexamined. We introduce a benchmark that evaluates spatial cognition abilities across seven categories to systematically assess how well LLMs process and generate three types of spatial knowledge: landmark, route, and survey knowledge. Furthermore, we propose a tool-augmented approach named Hybrid Mind, which integrates LLMs with deterministic GIS algorithms to enhance their performance in spatial cognitive tasks. The core idea involves the implementation of a mental map builder that generates a quantitative map based on segmented qualitative constraints, overcoming LLMs’ fallacies in synthesizing spatial information. Our experimental results revealed that although LLMs exhibited potential for spatial cognition, their performance was poor across most spatial cognitive tasks, particularly in constructing route and survey knowledge. The leading model, GPT-4-turbo, correctly answered fewer than one-fourth of the questions. In contrast, the Hybrid Mind approach significantly improved performance, correctly solving 70.48% of the questions.

• Research highlight 1: We propose a novel cross-task hybrid training methodology for multimodal images, offering a simple yet unified solution that simultaneously addresses both image generation and fusion tasks. • Research highlight 2: Building upon mutual-supervised multimodal image pairs, we innovatively integrate single-modality self-supervision to develop a hybrid-supervised decoupling framework with a dedicated loss function, achieving robust separation of content-style representations. • Research highlight 3: Extensive experiments spanning on four modalities and seven popular datasets demonstrate our method’s consistent superiority and impressive cross-task capability. Ablation studies further reveal that our framework learns generalized representations transferable across different image processing tasks. Multimodal image fusion and cross-modal translation are fundamental yet challenging tasks in computer vision, with their performance directly impacting downstream applications. Existing approaches typically treat these tasks independently, developing specialized models that fail to exploit the intrinsic relationships between different modalities. This limitation not only restricts model generalizability but also hinders further performance improvements. In this paper, we propose a joint optimization framework for image generation and fusion. Specifically, we generalize multimodal image tasks as the fusion and transformation of cross-modal features, and design a hybrid task training strategy. At the data level, we introduce a self-supervised and mutual-supervised hybrid mechanism for content-style feature decoupling, which achieves superior feature separation through stepwise training on intra-modal and cross-modal data. At the model level, we construct a triple-branch decoupling head along with fusion and transformation modules to ensure synchronous and efficient execution of dual tasks. Our method not only breaks through the single task limitation of the model, but also innovatively introduces mixed supervision into multimodal processing. We conduct comprehensive experiments covering four modalities fusion tasks on seven popular datasets. Extensive experimental results demonstrate that our method achieves superior performance on two tasks as compared of the respective state-of-the-art methods, and show impressive cross-task generalization capability.

Remote sensing image building change detection aims to identify building changes that occur in remote sensing images of the same areas acquired at different times. In recent years, the development of deep learning has led to significant advancements in building change detection methods. However, these fully supervised methods require a large number of bi-temporal remote sensing images with pixel-wise change detection labels to train the model, which incurs substantial time and manpower for annotation. To address this issue, this study proposes a single-temporal semi-supervised joint learning framework for building change detection, called SemiJointNet. Firstly, to reduce annotation costs, SemiJointNet uses building extraction labels instead of change detection labels to train the change detection model. Furthermore, to improve the semantic understanding capability of the model, SemiJointNet employs a joint learning approach for building extraction and change detection tasks. Lastly, SemiJointNet introduces semi-supervised learning, reducing the need for labels from thousands to dozens. Experimental results on the WHU dataset demonstrate that the proposed SemiJointNet achieves excellent building change detection performance with only a few dozen labels.

Cross-view geo-localization aims to determine the geographic origin of street-view images by matching them with a repository of remote sensing (RS) images equipped with GPS tags. Due to the substantial dissimilarities in viewpoint and visual appearance between street-view and RS images, this task is highly challenging. Recently, many deep-learning-based methods have been proposed. The CNN-based cross-view geo-localization techniques often employ polar transform and fail to capture extensive spatial correlations. And traditional transformer-based methods are prone to losing fine-grained details as a result of downsampling. Addressing these challenges, we propose a novel multi-scale transformer architecture for cross-view geo-localization in this study. Our model leverages multi-scale feature extraction to bolster the accuracy of image matching. We have conducted experiments on an open benchmark, demonstrating our approach’s superiority over the state-of-the-art methods.

Semantic segmentation of remote sensing images has attracted much attention for its wide applications. While deep learning models have shown impressive performance in this task, challenges arise when applying them to data from other domains without fine-tuning, due to domain gaps. Domain adaptation has emerged as a solution to bridge this gap. Existing works mainly focus on unsupervised domain adaptation, which lags far behind fully supervised models. However, active domain adaptation methods focused on natural images face challenges when applied to remote sensing images due to pronounced domain gaps and error unawareness problems. In this work, we propose a novel error-aware domain adaptation framework for remote sensing imagery semantic segmentation, called EasySeg, via interactive learning and active learning. Firstly, we introduce a point-level labeling strategy, named "See-First-Ask-Later" (SFAL), combining both interactive and active learning manners, allowing obvious errors and information-rich pixels to be annotated easily and efficiently. Then, we introduce an interactive semantic segmentation network (ISS-Net), which can perform automatic semantic segmentation and interactive refinement. Based on the acquired target point-level labels, ISS-Net generates dense and accurate pseudo-labels to enhance domain adaptation performance through retraining with consistency regularization. Comprehensive experiments on two tasks demonstrate that our method outperforms the state-of-the-art active domain adaptation methods in terms of overall accuracy (OA), mean accuracy (MA), F1 score, and mean Intersection of Union (mIoU) with lower labeling costs, even surpassing some fully supervised models. The source code of EasySeg is freely available at https://github.com/Yangliangzhe/EasySeg.

Remote sensing (RS) image semantic segmentation has attracted much attention due to its wide applications. However, deep learning based RS image semantic segmentation methods usually require substantial manual pixel-wise annotations, which are expensive and hard to obtain in practice. Although existing semi-supervised RS semantic segmentation methods effectively reduce dependence on labeled data, they generally focus on information consistency between labeled and unlabeled images, but ignore the potential context information between different areas of the RS image. In fact, the objects contained in a RS image usually have some long-range dependence between each other, since trees are usually on both sides of a road, and the middle of two rows of houses is commonly a road. Therefore, we believe that the potential dependencies between different areas of the RS image should be beneficial for reducing the label dependence of RS semantic segmentation. Based on this point, we propose a novel semi-supervised RS image semantic segmentation network named SegMind, which is based on mean teacher (MT) architecture and adopts Masked Image Modeling (MIM) to enhance information interactions of different areas. Moreover, Contrastive Learning (CL) and entropy loss are introduced to SegMind framework to further improve the linear separability and prediction confidence of the proposed model. Experiments on three datasets have demonstrated the superiority of proposed method over the state-of-the-art methods. The code is available at https://github.com/lzh-ggs-ddu/SegMind.

Online maps play an essential role in modern life. The convenience of acquiring remote sensing images provides reliable geographic information sources for the compilation of online maps. Some existing works have used the idea of domain mapping to translate remote sensing images into maps directly, which is of great prospect for application. However, many of the current remote sensing image-to-map translation works are performed in an unsupervised manner that would lead to problems such as distortion and local detail inaccuracy. Although the fully-supervised method is effective, it requires plenty of paired as well as matched data for training. Paired remote sensing images and maps with consistent spatial locations can be easily accessed through online map services, whereas many pairs of samples in which some geographic element information is not accurately and completely matched. Supervised learning-based translation models are often confused by these unmatched data. Accurate and complete matched data has to be selected deliberately by humans, and the manual selection process is time-consuming and laborious, which brings new challenges. Therefore, we propose a novel remote sensing image-to-map translation model named Semi-MapGen based on semi-supervised generative adversarial networks (GAN), which requires only a small set of accurate and complete matched data and plenty of unpaired data. In this model, we apply a knowledge extension-based learning strategy that can improve the accuracy of translated maps. In addition, we design Expansion loss and Channel-wise loss to learn the information from massive unpaired data in an unsupervised manner. Qualitative and quantitative experiment results on three datasets demonstrate that the proposed model outperforms state-of-the-art semi-supervised and supervised methods.

Remote sensing (RS) images change detection (CD) aims to obtain change information of the target area between multi-temporal RS images. With the modernization of cities, building change detection (BCD) plays a pivotal role in land resource planning, smart city construction and natural disaster assessment, and it is a typical application field of change detection task. Recently, deep learning based methods have shown their superiority in RS image change detection. However, the performance of the existing supervised change detection methods relies heavily on a large amount of high quality annotated bi-temporal RS image as training data, which is usually hard to obtain in practice. To address this issue, a semi-supervised BCD method using a pseudo bi-temporal data generator with consistency regularization was proposed. This method only needs a very small amount of single-temporal RS images with building extraction labels as labeled data. Firstly, with the help of the pseudo bi-temporal data generator, the model can generate a large number of pseudo bi-temporal images with CD labels from a small number of single-temporal images and corresponding building extraction labels automatically, which greatly augments the labeled data set for CD model training. Then, we proposed an error-prone data enhancement fine-tuning strategy to improve the learning effect of the proposed model to these synthesized training data. Finally, we enhance the robustness of the model by forcing the model to make consistent predictions on the images before and after perturbations. Extensive experimental results demonstrate that our method can effectively improve the BCD performance of the model even if labeled data are scare, and outperforms the state-of-the-art methods.

With the boost of deep learning methods, road extraction has been widely used in city planning and autonomous driving. However, it is very challenging to extract roads around the thorny occlusion areas, even in high-resolution remote sensing images. Existing approaches regard road extraction as an isolated binary segmentation task and ignore the surroundings’ contextual information in the optical image itself, especially the potential dependence implied between roads and buildings. To address the occlusion problem, we proposed a spatial attention-based road extraction neural network using contextual relation between roads and buildings named MSACon to extract the roads more precisely. First, we employed an existing building extraction method to predict buildings in the optical images. Second, we calculated the signed distance map (SDM) based on the building extraction results (which may be inaccurate) as ambiguous auxiliary information to infer the optical images’ potential roads. Due to the color, lines, and texture between the optical images and the SDM are distinct, we then designed the two-branch encoder to extract features and integrated the cross-domain features into the road decoder by a spatial attention-based fusion mechanism. Experiments demonstrate that the proposed method achieves superior performance than other state-of-the-art approaches even with ambiguous auxiliary information. Furthermore, MSACon shows obvious advantages in finding inconspicuous roads in the optical images and eliminating noisy roads, especially when dealing with areas where buildings are located along the roads.

Remote sensing image registration is the basis of change detection, environmental monitoring, and image fusion. Under severe appearance differences, feature-based methods have difficulty in finding sufficient feature matches to solve the global transformation and tackling the local deformation caused by height undulations and building shadows. By contrast, nonrigid registration methods are more flexible than feature-based matching methods, while often ignoring the reversibility between images, resulting in misalignment and inconsistency. To this end, this article proposes a nonrigid bidirectional registration network (NBR-Net) to estimate the flow-based dense correspondence for remote sensing images. We first propose an external cyclic registration network to strengthen the registration reversibility and geometric consistency by registering Image A to Image B and then reversely registering back to Image A. Second, we design an internal iterative refinement strategy to optimize the rough predicted flow caused by large distortion and viewpoint difference. Extensive experiments demonstrate that our method shows a performance superior to the state-of-the-art models on the multitemporal satellite image dataset. Furthermore, we attempt to extend our method to heterogeneous remote sensing image registration, which is more common in the real world. Therefore, we test our pretrained model in a satellite and unmanned aerial vehicle (UAV) image registration task. Due to the cyclic registration mechanism and coarse-to-fine refinement strategy, the proposed approach obtains the best performance on two GPS-denied UAV image datasets. Our code will be released athttps://github.com/xuyingxiao/NBR-Net.

Knowledge graph is becoming an indispensable resource that offers structured information for numerous AI applications. However, the knowledge graph often suffers from its incompleteness. Building a complete, high-quality knowledge graph is time-consuming and requires significant human annotation efforts. In this paper, we study the Knowledge Graph Population task, which aims at extending the scale of structured knowledge, with a special focus on reducing data preparation and annotation efforts. Previous works mainly based on discriminative methods build classifiers and verify candidate triplets that are extracted from texts, which heavily rely on the quality of data collection and co-occurrance of entities in the text. However, such methods fail to generalize on entity pairs that are not highly co-occurred, and fail to discover entity pairs that are not co-occurred at all in the given text corpus. We introduce a generative perspective to approach this task and define each relationship by learning the data distribution that embodies the core common properties for relational reasoning. A generative modelKGGenis proposed, which samples from the learned data distribution for each relation and can generate triplets regardless of entity pair co-occurrence in the text corpus. To further improve the generation quality while alleviate human annotation efforts, adversarial learning is adopted to not only encourage generating high quality triplets, but also give model the ability to automatically assess the generation quality. Quantitative and qualitative experimental results conducted on two real-world generic knowledge graphs show that the proposed modelKGGengenerates novel and meaningful triplets with improved efficiency and less human annotation comparing with the state-of-the-art approaches.

With the appearance of satellite cluster nowadays, some new challenges have emerged in satellite data transmission scheduling. Current researches ignored the new features of the satellite cluster data transmission such as the periodicity of data transmission window collisions which could direct the future scheduling. Considering the characteristics of the problem, a data transmission conflict window model for satellite cluster is established and a novel algorithm based on genetic algorithm is proposed. In order to improve the searching efficiency, convergence rate and stability of our algorithm, a rote learning operator is designed, which can generate heuristic from past scheduling results and lead searching direction of our algorithm. Finally, some experiments are conducted to validate the correctness and practicability of our algorithm.

Satellite observation scheduling is a complex combinational optimization problem. Current researches usually adopt intelligent optimization methods to solve it, ignoring the similar historical scheduling cases. In order to improve algorithm performance, case-based learning method is introduced to the scheduling process. Considering the characteristic of the problem, a method of retrieving, matching and revising satellite observing scheduling historical cases is designed. Then, a novel algorithm based on case-based learning and a genetic algorithm is proposed. Finally, some experiments are conducted to validate the correctness and practicability of our algorithm.

The earth observing mission scheduling problem is an important real-world problem that impacts the opportunity of dealing with emergencies and the collection of hazard monitoring research data. The period of traditional data acquisition cycle for earth science research is often too long to receive some important observation data in time. Besides, with rapid developing of sensor web techniques, the envisioned future earth scientists and emergency workers would like to investigate the natural phenomenon by using large numbers of sensors based on different platforms, such as satellite, balloon, aircraft and ground-based. How to schedule these sensors that are frequency agile and capable of multi-scene observations for completing a hazard monitoring research mission is a difficult task. In this paper, we focus on solving two problems above. The active observation model based on hazard monitoring domain knowledge is proposed for reducing responsive time of abnormal phenomenon. And then information gain model is introduced for evaluating observation schedule. On this basis, multi-platform sensor optimization scheduling model is constructed. Simulation and analysis show that the proposed model can solve the problem effectively. Moreover, the normal requests are also taken into account during these events for maximizing the value of various sensors.

In this paper, satellite imaging scheduling of cooperative multiple centers, an example of multi-agent systems, is discussed with a proposal of a novel Cooperative CoEvolutionary Planning Algorithm (CCEPA). Considering the numbers of the multi-center and the characteristics of targets to be observed, CCEPA decomposes the tasks into smaller components and evolves multiple solutions in the form of cooperative subpopulations. At the same time, for such evolutionary algorithm based on scheduling, a novel fixed-length binary encoding mechanism for tasks assigned to each center is also proposed. Incorporated with various features like archiving, dynamic sharing, the CCEPA is capable of maintaining archive diversity in the evolution and distributing the solutions uniformly along the Pareto front. Simulation and analysis show that the proposed algorithm can solve the problem effectively.

Electromagnetic detection satellite (EDS) is a type of Earth observation Satellites (EOSs). The Information collected by EDSs is very important in some application domain, such as industry, science and military. The scheduling of EDSs is a complex combinatorial optimization problem. Current research mainly focuses on the scheduling of imaging satellites and SAR satellites, little work on the scheduling of EDSs for its specific requirements. Considering the specific constrains of EDSs, we established a MultiSatellites scheduling model and proposed a scheduling algorithm based on genetic algorithm. A hybrid algorithm incorporated with genetic algorithm and stochastic climbing algorithm was constructed to improve the scheduling algorithm. To deal with some specific constrains, a punish function method was introduced. We have conducted some experiments to validate correctness and practicability of our scheduling algorithms.