• Research highlight 1: We propose a novel cross-task hybrid training methodology for multimodal images, offering a simple yet unified solution that simultaneously addresses both image generation and fusion tasks. • Research highlight 2: Building upon mutual-supervised multimodal image pairs, we innovatively integrate single-modality self-supervision to develop a hybrid-supervised decoupling framework with a dedicated loss function, achieving robust separation of content-style representations. • Research highlight 3: Extensive experiments spanning on four modalities and seven popular datasets demonstrate our method’s consistent superiority and impressive cross-task capability. Ablation studies further reveal that our framework learns generalized representations transferable across different image processing tasks. Multimodal image fusion and cross-modal translation are fundamental yet challenging tasks in computer vision, with their performance directly impacting downstream applications. Existing approaches typically treat these tasks independently, developing specialized models that fail to exploit the intrinsic relationships between different modalities. This limitation not only restricts model generalizability but also hinders further performance improvements. In this paper, we propose a joint optimization framework for image generation and fusion. Specifically, we generalize multimodal image tasks as the fusion and transformation of cross-modal features, and design a hybrid task training strategy. At the data level, we introduce a self-supervised and mutual-supervised hybrid mechanism for content-style feature decoupling, which achieves superior feature separation through stepwise training on intra-modal and cross-modal data. At the model level, we construct a triple-branch decoupling head along with fusion and transformation modules to ensure synchronous and efficient execution of dual tasks. Our method not only breaks through the single task limitation of the model, but also innovatively introduces mixed supervision into multimodal processing. We conduct comprehensive experiments covering four modalities fusion tasks on seven popular datasets. Extensive experimental results demonstrate that our method achieves superior performance on two tasks as compared of the respective state-of-the-art methods, and show impressive cross-task generalization capability.

Remote sensing image building change detection aims to identify building changes that occur in remote sensing images of the same areas acquired at different times. In recent years, the development of deep learning has led to significant advancements in building change detection methods. However, these fully supervised methods require a large number of bi-temporal remote sensing images with pixel-wise change detection labels to train the model, which incurs substantial time and manpower for annotation. To address this issue, this study proposes a single-temporal semi-supervised joint learning framework for building change detection, called SemiJointNet. Firstly, to reduce annotation costs, SemiJointNet uses building extraction labels instead of change detection labels to train the change detection model. Furthermore, to improve the semantic understanding capability of the model, SemiJointNet employs a joint learning approach for building extraction and change detection tasks. Lastly, SemiJointNet introduces semi-supervised learning, reducing the need for labels from thousands to dozens. Experimental results on the WHU dataset demonstrate that the proposed SemiJointNet achieves excellent building change detection performance with only a few dozen labels.

Cross-view geo-localization aims to determine the geographic origin of street-view images by matching them with a repository of remote sensing (RS) images equipped with GPS tags. Due to the substantial dissimilarities in viewpoint and visual appearance between street-view and RS images, this task is highly challenging. Recently, many deep-learning-based methods have been proposed. The CNN-based cross-view geo-localization techniques often employ polar transform and fail to capture extensive spatial correlations. And traditional transformer-based methods are prone to losing fine-grained details as a result of downsampling. Addressing these challenges, we propose a novel multi-scale transformer architecture for cross-view geo-localization in this study. Our model leverages multi-scale feature extraction to bolster the accuracy of image matching. We have conducted experiments on an open benchmark, demonstrating our approach’s superiority over the state-of-the-art methods.

Semantic segmentation of remote sensing images has attracted much attention for its wide applications. While deep learning models have shown impressive performance in this task, challenges arise when applying them to data from other domains without fine-tuning, due to domain gaps. Domain adaptation has emerged as a solution to bridge this gap. Existing works mainly focus on unsupervised domain adaptation, which lags far behind fully supervised models. However, active domain adaptation methods focused on natural images face challenges when applied to remote sensing images due to pronounced domain gaps and error unawareness problems. In this work, we propose a novel error-aware domain adaptation framework for remote sensing imagery semantic segmentation, called EasySeg, via interactive learning and active learning. Firstly, we introduce a point-level labeling strategy, named "See-First-Ask-Later" (SFAL), combining both interactive and active learning manners, allowing obvious errors and information-rich pixels to be annotated easily and efficiently. Then, we introduce an interactive semantic segmentation network (ISS-Net), which can perform automatic semantic segmentation and interactive refinement. Based on the acquired target point-level labels, ISS-Net generates dense and accurate pseudo-labels to enhance domain adaptation performance through retraining with consistency regularization. Comprehensive experiments on two tasks demonstrate that our method outperforms the state-of-the-art active domain adaptation methods in terms of overall accuracy (OA), mean accuracy (MA), F1 score, and mean Intersection of Union (mIoU) with lower labeling costs, even surpassing some fully supervised models. The source code of EasySeg is freely available at https://github.com/Yangliangzhe/EasySeg.

Remote sensing (RS) image semantic segmentation has attracted much attention due to its wide applications. However, deep learning based RS image semantic segmentation methods usually require substantial manual pixel-wise annotations, which are expensive and hard to obtain in practice. Although existing semi-supervised RS semantic segmentation methods effectively reduce dependence on labeled data, they generally focus on information consistency between labeled and unlabeled images, but ignore the potential context information between different areas of the RS image. In fact, the objects contained in a RS image usually have some long-range dependence between each other, since trees are usually on both sides of a road, and the middle of two rows of houses is commonly a road. Therefore, we believe that the potential dependencies between different areas of the RS image should be beneficial for reducing the label dependence of RS semantic segmentation. Based on this point, we propose a novel semi-supervised RS image semantic segmentation network named SegMind, which is based on mean teacher (MT) architecture and adopts Masked Image Modeling (MIM) to enhance information interactions of different areas. Moreover, Contrastive Learning (CL) and entropy loss are introduced to SegMind framework to further improve the linear separability and prediction confidence of the proposed model. Experiments on three datasets have demonstrated the superiority of proposed method over the state-of-the-art methods. The code is available at https://github.com/lzh-ggs-ddu/SegMind.

Online maps play an essential role in modern life. The convenience of acquiring remote sensing images provides reliable geographic information sources for the compilation of online maps. Some existing works have used the idea of domain mapping to translate remote sensing images into maps directly, which is of great prospect for application. However, many of the current remote sensing image-to-map translation works are performed in an unsupervised manner that would lead to problems such as distortion and local detail inaccuracy. Although the fully-supervised method is effective, it requires plenty of paired as well as matched data for training. Paired remote sensing images and maps with consistent spatial locations can be easily accessed through online map services, whereas many pairs of samples in which some geographic element information is not accurately and completely matched. Supervised learning-based translation models are often confused by these unmatched data. Accurate and complete matched data has to be selected deliberately by humans, and the manual selection process is time-consuming and laborious, which brings new challenges. Therefore, we propose a novel remote sensing image-to-map translation model named Semi-MapGen based on semi-supervised generative adversarial networks (GAN), which requires only a small set of accurate and complete matched data and plenty of unpaired data. In this model, we apply a knowledge extension-based learning strategy that can improve the accuracy of translated maps. In addition, we design Expansion loss and Channel-wise loss to learn the information from massive unpaired data in an unsupervised manner. Qualitative and quantitative experiment results on three datasets demonstrate that the proposed model outperforms state-of-the-art semi-supervised and supervised methods.

Cellular Signal Strength (CSS), defined as the signal power received by mobile phones, is an important aspect of geographic information flow analysis, because the density of such information can reflect the urbanization variables such as population, gross domestic product, built-up area, electric power consumption, etc. Despite the importance, the real-time analysis of global CSS distribution remains a challenging problem due to the large data scale. In this article, a Display-driven Computing (DisDC) technique is designed and applied to provide efficient large scale interactive CSS visualization, generating results by calculating the value of each pixel that directly for display. Specifically, we present an efficient CSS measurement algorithm, which introduces spatial indexes and a corresponding query strategy; besides, an optimized parallel computing architecture is proposed to ensure the ability of real-time visualization. Experiments show that our approach obviously outperforms traditional methods and is capable of handling more than 40 million base stations in real-time. Moreover, an online demonstration is provided athttps://github.com/MemoryMmy/CSSMap.

With the boost of deep learning methods, road extraction has been widely used in city planning and autonomous driving. However, it is very challenging to extract roads around the thorny occlusion areas, even in high-resolution remote sensing images. Existing approaches regard road extraction as an isolated binary segmentation task and ignore the surroundings’ contextual information in the optical image itself, especially the potential dependence implied between roads and buildings. To address the occlusion problem, we proposed a spatial attention-based road extraction neural network using contextual relation between roads and buildings named MSACon to extract the roads more precisely. First, we employed an existing building extraction method to predict buildings in the optical images. Second, we calculated the signed distance map (SDM) based on the building extraction results (which may be inaccurate) as ambiguous auxiliary information to infer the optical images’ potential roads. Due to the color, lines, and texture between the optical images and the SDM are distinct, we then designed the two-branch encoder to extract features and integrated the cross-domain features into the road decoder by a spatial attention-based fusion mechanism. Experiments demonstrate that the proposed method achieves superior performance than other state-of-the-art approaches even with ambiguous auxiliary information. Furthermore, MSACon shows obvious advantages in finding inconspicuous roads in the optical images and eliminating noisy roads, especially when dealing with areas where buildings are located along the roads.

Remote sensing image registration is the basis of change detection, environmental monitoring, and image fusion. Under severe appearance differences, feature-based methods have difficulty in finding sufficient feature matches to solve the global transformation and tackling the local deformation caused by height undulations and building shadows. By contrast, nonrigid registration methods are more flexible than feature-based matching methods, while often ignoring the reversibility between images, resulting in misalignment and inconsistency. To this end, this article proposes a nonrigid bidirectional registration network (NBR-Net) to estimate the flow-based dense correspondence for remote sensing images. We first propose an external cyclic registration network to strengthen the registration reversibility and geometric consistency by registering Image A to Image B and then reversely registering back to Image A. Second, we design an internal iterative refinement strategy to optimize the rough predicted flow caused by large distortion and viewpoint difference. Extensive experiments demonstrate that our method shows a performance superior to the state-of-the-art models on the multitemporal satellite image dataset. Furthermore, we attempt to extend our method to heterogeneous remote sensing image registration, which is more common in the real world. Therefore, we test our pretrained model in a satellite and unmanned aerial vehicle (UAV) image registration task. Due to the cyclic registration mechanism and coarse-to-fine refinement strategy, the proposed approach obtains the best performance on two GPS-denied UAV image datasets. Our code will be released athttps://github.com/xuyingxiao/NBR-Net.

Knowledge graph is becoming an indispensable resource that offers structured information for numerous AI applications. However, the knowledge graph often suffers from its incompleteness. Building a complete, high-quality knowledge graph is time-consuming and requires significant human annotation efforts. In this paper, we study the Knowledge Graph Population task, which aims at extending the scale of structured knowledge, with a special focus on reducing data preparation and annotation efforts. Previous works mainly based on discriminative methods build classifiers and verify candidate triplets that are extracted from texts, which heavily rely on the quality of data collection and co-occurrance of entities in the text. However, such methods fail to generalize on entity pairs that are not highly co-occurred, and fail to discover entity pairs that are not co-occurred at all in the given text corpus. We introduce a generative perspective to approach this task and define each relationship by learning the data distribution that embodies the core common properties for relational reasoning. A generative modelKGGenis proposed, which samples from the learned data distribution for each relation and can generate triplets regardless of entity pair co-occurrence in the text corpus. To further improve the generation quality while alleviate human annotation efforts, adversarial learning is adopted to not only encourage generating high quality triplets, but also give model the ability to automatically assess the generation quality. Quantitative and qualitative experimental results conducted on two real-world generic knowledge graphs show that the proposed modelKGGengenerates novel and meaningful triplets with improved efficiency and less human annotation comparing with the state-of-the-art approaches.

We present DiSA, a Display-driven Spatial Analysis framework for interactive analysis of large-scale geographical vector data. DiSA calculates visualization of analysis results directly using a parallel per-pixel approach with efficient fine-grained spatial indexes. Compared with conventional object-based methods, DiSA can greatly reduce the computational complexity (from O(n) to O(log(n)) in some cases), making it less sensitive to data volumes. Experimental results verify that DiSA can provide analysis of billion-scale spatial objects in milliseconds. We demonstrate DiSA with various application scenarios including raw data exploration, spatial buffer and overlay analysis, and global cellular signal strength analysis. Users can explore 10 millions of spatial objects, adjust algorithm parameters, and always see the results in real-time on a personal computer.

With the appearance of satellite cluster nowadays, some new challenges have emerged in satellite data transmission scheduling. Current researches ignored the new features of the satellite cluster data transmission such as the periodicity of data transmission window collisions which could direct the future scheduling. Considering the characteristics of the problem, a data transmission conflict window model for satellite cluster is established and a novel algorithm based on genetic algorithm is proposed. In order to improve the searching efficiency, convergence rate and stability of our algorithm, a rote learning operator is designed, which can generate heuristic from past scheduling results and lead searching direction of our algorithm. Finally, some experiments are conducted to validate the correctness and practicability of our algorithm.

Satellite observation scheduling is a complex combinational optimization problem. Current researches usually adopt intelligent optimization methods to solve it, ignoring the similar historical scheduling cases. In order to improve algorithm performance, case-based learning method is introduced to the scheduling process. Considering the characteristic of the problem, a method of retrieving, matching and revising satellite observing scheduling historical cases is designed. Then, a novel algorithm based on case-based learning and a genetic algorithm is proposed. Finally, some experiments are conducted to validate the correctness and practicability of our algorithm.

The earth observing mission scheduling problem is an important real-world problem that impacts the opportunity of dealing with emergencies and the collection of hazard monitoring research data. The period of traditional data acquisition cycle for earth science research is often too long to receive some important observation data in time. Besides, with rapid developing of sensor web techniques, the envisioned future earth scientists and emergency workers would like to investigate the natural phenomenon by using large numbers of sensors based on different platforms, such as satellite, balloon, aircraft and ground-based. How to schedule these sensors that are frequency agile and capable of multi-scene observations for completing a hazard monitoring research mission is a difficult task. In this paper, we focus on solving two problems above. The active observation model based on hazard monitoring domain knowledge is proposed for reducing responsive time of abnormal phenomenon. And then information gain model is introduced for evaluating observation schedule. On this basis, multi-platform sensor optimization scheduling model is constructed. Simulation and analysis show that the proposed model can solve the problem effectively. Moreover, the normal requests are also taken into account during these events for maximizing the value of various sensors.

In this paper, satellite imaging scheduling of cooperative multiple centers, an example of multi-agent systems, is discussed with a proposal of a novel Cooperative CoEvolutionary Planning Algorithm (CCEPA). Considering the numbers of the multi-center and the characteristics of targets to be observed, CCEPA decomposes the tasks into smaller components and evolves multiple solutions in the form of cooperative subpopulations. At the same time, for such evolutionary algorithm based on scheduling, a novel fixed-length binary encoding mechanism for tasks assigned to each center is also proposed. Incorporated with various features like archiving, dynamic sharing, the CCEPA is capable of maintaining archive diversity in the evolution and distributing the solutions uniformly along the Pareto front. Simulation and analysis show that the proposed algorithm can solve the problem effectively.

Electromagnetic detection satellite (EDS) is a type of Earth observation Satellites (EOSs). The Information collected by EDSs is very important in some application domain, such as industry, science and military. The scheduling of EDSs is a complex combinatorial optimization problem. Current research mainly focuses on the scheduling of imaging satellites and SAR satellites, little work on the scheduling of EDSs for its specific requirements. Considering the specific constrains of EDSs, we established a MultiSatellites scheduling model and proposed a scheduling algorithm based on genetic algorithm. A hybrid algorithm incorporated with genetic algorithm and stochastic climbing algorithm was constructed to improve the scheduling algorithm. To deal with some specific constrains, a punish function method was introduced. We have conducted some experiments to validate correctness and practicability of our scheduling algorithms.

EOSs (Earth Observing Satellites) circle the earth to take shotswhich are requested by customers. To make replete use of resourcesof EOSs, it is required to deal with the problem of united imagingscheduling of EOSs in a given scheduling horizon, which is acomplicated multi-objective combinatorial optimization problem. Inthis paper, we construct a mathematical model for the problem byabstracting imaging constraints of different EOSs. Then we propose anovel multi-objective EOSs imaging scheduling method, which is basedon the Strength Pareto Evolutionary Algorithm 2. The specialencoding technique and imaging constraint control are applied toguarantee feasibility of solutions. The approach is tested upon fourreal application problems of CBERS EOSs series. From the results, itis confirmed that the proposed approach is effective in solvingmulti-objective EOSs imaging scheduling problems.